Compare commits
290
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6427afe6ae | ||
|
|
d92068be7b | ||
|
|
6a35e5a58c | ||
|
|
bf4d99843f | ||
|
|
702289ad7d | ||
|
|
6ee95a3f31 | ||
|
|
159a583cf0 | ||
|
|
62ff9a5bd3 | ||
|
|
16442ecd23 | ||
|
|
ffb7fa52ec | ||
|
|
fcf34727fe | ||
|
|
a5e351f4cb | ||
|
|
e3279f4b5b | ||
|
|
d52687d0c8 | ||
|
|
36d9b38a0d | ||
|
|
da6414c27c | ||
|
|
5dcb9dfbf6 | ||
|
|
1c90777124 | ||
|
|
ba07b43b18 | ||
|
|
1db93ffc64 | ||
|
|
ea42b00d0a | ||
|
|
198e4db0a3 | ||
|
|
a438cbdac3 | ||
|
|
5fa35362ac | ||
|
|
ee67f69ef4 | ||
|
|
a211aecdfa | ||
|
|
5fb8f06a91 | ||
|
|
ec7b6e0748 | ||
|
|
7015bf58ac | ||
|
|
aa9b5b99e4 | ||
|
|
61f61b5d06 | ||
|
|
ffd7592687 | ||
|
|
1bce3f33a8 | ||
|
|
8607d21110 | ||
|
|
050a2d08e8 | ||
|
|
baa3351552 | ||
|
|
187389b7d1 | ||
|
|
731027c009 | ||
|
|
fc4ab370d6 | ||
|
|
d66579c8f6 | ||
|
|
c4ff0478b7 | ||
|
|
1ce9ffe79f | ||
|
|
df2c3f63a5 | ||
|
|
a9bc0c1cc5 | ||
|
|
6d28683898 | ||
|
|
82c03a8d10 | ||
|
|
31c5aee72d | ||
|
|
05857f71f0 | ||
|
|
98e9e62fce | ||
|
|
0d5693cc2c | ||
|
|
1572d74a18 | ||
|
|
9fd7e6c458 | ||
|
|
5703c86cf6 | ||
|
|
d85d74d607 | ||
|
|
46efb7ae98 | ||
|
|
4279e4914b | ||
|
|
a0acaad86d | ||
|
|
08c814aa0a | ||
|
|
32fa6b40f5 | ||
|
|
8df3469813 | ||
|
|
e7869d27bd | ||
|
|
0b002ef39c | ||
|
|
101848242f | ||
|
|
5395b6a47f | ||
|
|
90e4a368dc | ||
|
|
d1b5210d83 | ||
|
|
1162f7fe25 | ||
|
|
becae7ba51 | ||
|
|
94fee5d849 | ||
|
|
5856555390 | ||
|
|
0ffdf4e581 | ||
|
|
dd5bf9cd8c | ||
|
|
0e9d4ab86a | ||
|
|
ee7abe0a8e | ||
|
|
f70973e222 | ||
|
|
812dc3b303 | ||
|
|
33fc322f4d | ||
|
|
bec1dd0082 | ||
|
|
a32f7e50af | ||
|
|
1111bd84d7 | ||
|
|
873cc4007b | ||
|
|
0073e52881 | ||
|
|
fcf3b6f4d5 | ||
|
|
43f3565617 | ||
|
|
addfdd6940 | ||
|
|
33a00d5656 | ||
|
|
f9f48fbbf7 | ||
|
|
b43b60b183 | ||
|
|
17c2e061df | ||
|
|
a40595fa08 | ||
|
|
6e1099a905 | ||
|
|
2e1ae061b5 | ||
|
|
03fc13b84b | ||
|
|
a062f181ce | ||
|
|
87f464cc5f | ||
|
|
5a71e8659f | ||
|
|
1e8273eb19 | ||
|
|
199c2f20bb | ||
|
|
94229d95eb | ||
|
|
27b2161379 | ||
|
|
bc271d4de0 | ||
|
|
b637fe106f | ||
|
|
033a5d384c | ||
|
|
89302aa3e0 | ||
|
|
3d693a8735 | ||
|
|
411f0680b8 | ||
|
|
8a7328c282 | ||
|
|
5d1e6d0b09 | ||
|
|
b163ed1fcc | ||
|
|
f52895a646 | ||
|
|
5d13f9c83d | ||
|
|
0aa8f62e0c | ||
|
|
ee70d5b451 | ||
|
|
3121e47245 | ||
|
|
261064ec63 | ||
|
|
3f35153dd5 | ||
|
|
027c41d125 | ||
|
|
9359542990 | ||
|
|
24d52a8380 | ||
|
|
e773bf4c34 | ||
|
|
256884e671 | ||
|
|
aa35ba1594 | ||
|
|
2eb65a17a3 | ||
|
|
68d83c5b24 | ||
|
|
8e7fac8bf1 | ||
|
|
fdc61054e3 | ||
|
|
aabaaf2bb4 | ||
|
|
9f5ac38662 | ||
|
|
3983188b8a | ||
|
|
15ac7fd8dc | ||
|
|
3a9ed9d3fd | ||
|
|
a4866ca899 | ||
|
|
58ad0742d8 | ||
|
|
90f7bcb209 | ||
|
|
72b0810c53 | ||
|
|
03c73e7698 | ||
|
|
64ae75a480 | ||
|
|
0fa4d14f54 | ||
|
|
837b5df2f7 | ||
|
|
9da45c68d6 | ||
|
|
ddf104f8aa | ||
|
|
3d939b5c77 | ||
|
|
c567410687 | ||
|
|
26bec28c60 | ||
|
|
cde00e1d5f | ||
|
|
c29accd1be | ||
|
|
47dc3bc8e5 | ||
|
|
423fde8534 | ||
|
|
56d0dcc4ff | ||
|
|
84c95c53af | ||
|
|
e4a5d9b1d0 | ||
|
|
1ce1c3d4e0 | ||
|
|
8f23d4b114 | ||
|
|
f9e129f734 | ||
|
|
e228355a52 | ||
|
|
cd1653bfe9 | ||
|
|
5852b35ab9 | ||
|
|
a4440c9acf | ||
|
|
9ffe9e97c9 | ||
|
|
d16b5294f6 | ||
|
|
315bd7118a | ||
|
|
e2afb3e144 | ||
|
|
450a4d8605 | ||
|
|
9dfa5ba2ac | ||
|
|
b7a33a1e26 | ||
|
|
ec2d73bc06 | ||
|
|
a361c86c3c | ||
|
|
12a311ba4c | ||
|
|
897626a9f6 | ||
|
|
a3b0e68ec5 | ||
|
|
2405d72fb0 | ||
|
|
c41deb55d2 | ||
|
|
d5e12c82aa | ||
|
|
220e3c72f5 | ||
|
|
4430cc1748 | ||
|
|
860d512e91 | ||
|
|
3a9b47918d | ||
|
|
3b0aff7585 | ||
|
|
495db6dc89 | ||
|
|
e269f9a185 | ||
|
|
546caa31c4 | ||
|
|
db8be15a15 | ||
|
|
9c8c4a3ee5 | ||
|
|
600a57aafb | ||
|
|
2c8616b650 | ||
|
|
69fa79c585 | ||
|
|
cabe873503 | ||
|
|
36936dc7a3 | ||
|
|
bfa08742f8 | ||
|
|
b5a74178c3 | ||
|
|
1a83ed28fa | ||
|
|
984194e765 | ||
|
|
af5c939d16 | ||
|
|
89b65dd4c0 | ||
|
|
75887d0bc5 | ||
|
|
e5f4af8cf2 | ||
|
|
e27f8b1b7d | ||
|
|
332c334767 | ||
|
|
47b0ca5c90 | ||
|
|
d1aa4254c9 | ||
|
|
f63eca13b3 | ||
|
|
e898a3ec44 | ||
|
|
9eb5576682 | ||
|
|
5118258ee3 | ||
|
|
cfa3ad1930 | ||
|
|
b622de4fe6 | ||
|
|
820ce8fc8c | ||
|
|
4bb6dd26ed | ||
|
|
b2cb481681 | ||
|
|
50d61398b6 | ||
|
|
7b4a664c50 | ||
|
|
9638aa02c5 | ||
|
|
b245f5e254 | ||
|
|
6bdd3f2bfa | ||
|
|
6385d52bc0 | ||
|
|
013dbce4a9 | ||
|
|
e75683c01f | ||
|
|
bd8f09d647 | ||
|
|
0bcfeb4e80 | ||
|
|
c2a37ab7f8 | ||
|
|
6330abce5f | ||
|
|
03266fd4a2 | ||
|
|
3f00d3b413 | ||
|
|
befbb1d978 | ||
|
|
2bbdc52440 | ||
|
|
fc7717f6b1 | ||
|
|
0871e6ec11 | ||
|
|
f6b1834ea9 | ||
|
|
407416e6d0 | ||
|
|
39916b26e9 | ||
|
|
0760280bc6 | ||
|
|
f583502fc9 | ||
|
|
4adc656b54 | ||
|
|
dea46c3c6c | ||
|
|
9073d3f2df | ||
|
|
2baf22ac43 | ||
|
|
80f18caf03 | ||
|
|
c86adbe3cc | ||
|
|
36d342f176 | ||
|
|
1ee62392b9 | ||
|
|
ecfc2c215e | ||
|
|
122405df8e | ||
|
|
acd5a0d80b | ||
|
|
f4347e2f14 | ||
|
|
b68103d198 | ||
|
|
f801b4b3e2 | ||
|
|
a1a10e9ed2 | ||
|
|
0ef6d3f574 | ||
|
|
fccac61e58 | ||
|
|
10fa7c76d7 | ||
|
|
a028337490 | ||
|
|
e3934e4449 | ||
|
|
c570c687f6 | ||
|
|
d1ab2dc0b4 | ||
|
|
9ad0ec35e0 | ||
|
|
e4327a3a5e | ||
|
|
ce486fd5d2 | ||
|
|
fdd909ec40 | ||
|
|
cb1954e7b5 | ||
|
|
1b5308c9cc | ||
|
|
cbcbba8099 | ||
|
|
046a990198 | ||
|
|
fbf5b1d04b | ||
|
|
3c6c16abdf | ||
|
|
e823cc1cc5 | ||
|
|
c192810fba | ||
|
|
345bbe0552 | ||
|
|
f3ffdae909 | ||
|
|
6ecd03694b | ||
|
|
a036bac47b | ||
|
|
aafeb5c9df | ||
|
|
573a94e102 | ||
|
|
f4e81074d3 | ||
|
|
a3cee17d48 | ||
|
|
35ce23fb76 | ||
|
|
6374c1eb60 | ||
|
|
7c0be6968e | ||
|
|
7f314eb399 | ||
|
|
5ad3641cbf | ||
|
|
0bc4b7774f | ||
|
|
6b1c80b442 | ||
|
|
8448219de1 | ||
|
|
00e98f1f92 | ||
|
|
0ac99f0d68 | ||
|
|
1ade18209d | ||
|
|
a9cd3de5be | ||
|
|
6e7e77acbe | ||
|
|
890c3ee3fc | ||
|
|
a6872ac099 | ||
|
|
778efaba8b |
@@ -9,6 +9,7 @@ works — plus the engineering practice that holds across everything Novox build
|
||||
| [`context.md`](context.md) | The environment — conditions, not aspirations |
|
||||
| [`effect.md`](effect.md) | What is different when the work is done |
|
||||
| [`how-we-build.md`](how-we-build.md) | The rules that hold across the mesh, each one earned. **The source of the mesh constitution** — the governed page the mesh injects into design sessions is derived from it. |
|
||||
| [`glossary.md`](glossary.md) | One name per thing — the authority on vocabulary, and the words that were retired |
|
||||
| [`repos.md`](repos.md) | Where implementation lives, and what each repository owns |
|
||||
| [`process/`](process/) | The playbooks — how work moves through this repository, for engineers and agents alike |
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ indistinguishable from one that cannot.
|
||||
| `numbering` | the number in the filename is the number in the heading | — |
|
||||
| `topics` | every record names a topic the index knows | — |
|
||||
| *(index.py)* | the written reading order matches what the records say | — |
|
||||
| `status-vs-code` | a to-be document naming specific code is not still `designed` | **ten documents**, several with a *What was built* section, describing lab-proven code |
|
||||
|
||||
## What is deliberately not checked
|
||||
|
||||
@@ -47,3 +48,10 @@ what to do.
|
||||
|
||||
State what incident it would have caught, and make it fail before you make it pass. A check
|
||||
whose failure has never been observed is a guess about its own correctness.
|
||||
|
||||
## cycle.py
|
||||
|
||||
The development cycle, checked ([ADR 0080](../../02-DECISIONS/0080-the-development-cycle-is-checked.md)):
|
||||
a to-be design names a decision, an in-progress/implemented design names its owning code, a
|
||||
located/fixed issue names its owner, a fixed/resolved issue says what fixed it, a graduated
|
||||
research overview says what it became. `python3 00-META/checks/cycle.py`
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,188 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The development cycle, checked.
|
||||
|
||||
The knowledge flow (00-META/process/00-overview.md) says work moves idea -> research ->
|
||||
decision -> to-be design -> code, and symptom -> issue -> diagnosis -> fix. Those are rules,
|
||||
and a rule states how it is checked (AGENTS.md) -- this is how. Everything here reads only
|
||||
frontmatter, because status lives in frontmatter and nowhere else.
|
||||
|
||||
What is enforced:
|
||||
|
||||
design every 03-DESIGN doc parses, carries `layer:` matching its directory, and a
|
||||
known `status:`. A TO-BE doc names at least one decision (`decisions:`) -- no
|
||||
design without a decision -- and once `in-progress` or `implemented` it names
|
||||
its owning code (`code:`) -- no development without a design that says where.
|
||||
issues a known `status:`; once `located`, `located-in:` names the owner;
|
||||
once `resolved`, `fixed-by:` says what fixed it (prose counts --
|
||||
"nothing, the capability existed" is an answer).
|
||||
research a known `status:`; a `graduated` overview says what it `became:`, and every
|
||||
target it names exists.
|
||||
decisions every accepted record is REACHABLE from the cycle: cited by a design doc's
|
||||
frontmatter, a research overview, an issue report, a 00-META document, or another
|
||||
record's extends/supersedes chain. A decision nothing points at is one nobody will
|
||||
find by following pointers -- which is how records go stale in people's heads.
|
||||
|
||||
Deliberately NOT enforced: `resolved` issues may leave `located-in` empty (a symptom that
|
||||
turned out not to be a defect has no owner), and as-is docs need no decisions (they
|
||||
describe what exists, not what was decided).
|
||||
|
||||
python3 00-META/checks/cycle.py
|
||||
"""
|
||||
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||
|
||||
DESIGN_STATUSES = {"proposed", "designed", "in-progress", "implemented", "abandoned"}
|
||||
ISSUE_STATUSES = {"open", "diagnosing", "located", "resolved", "wontfix"}
|
||||
RESEARCH_STATUSES = {"active", "graduated", "abandoned"}
|
||||
|
||||
|
||||
def rel(path):
|
||||
return os.path.relpath(path, ROOT)
|
||||
|
||||
|
||||
def frontmatter(path):
|
||||
"""The YAML block between the first two --- lines, as {key: raw-value-string}.
|
||||
|
||||
Minimal on purpose, like records.py: enough for the fields these checks read. A list
|
||||
value (block or inline) is joined into its items; a scalar stays a string.
|
||||
"""
|
||||
text = open(path, encoding="utf-8").read()
|
||||
m = re.match(r"^---\n(.*?)\n---", text, re.S)
|
||||
if not m:
|
||||
return None
|
||||
front, out, key = m.group(1), {}, None
|
||||
for line in front.split("\n"):
|
||||
item = re.match(r"^\s+-\s*(.+?)\s*$", line)
|
||||
if item and key:
|
||||
out[key].append(item.group(1))
|
||||
continue
|
||||
kv = re.match(r"^([A-Za-z-]+):\s*(.*)$", line)
|
||||
if not kv:
|
||||
continue
|
||||
key, value = kv.group(1), kv.group(2).strip()
|
||||
if value.startswith("[") and value.endswith("]"):
|
||||
out[key] = [v.strip() for v in value[1:-1].split(",") if v.strip()]
|
||||
elif value == "":
|
||||
out[key] = [] # a block list may follow; stays [] if nothing does
|
||||
else:
|
||||
out[key] = value
|
||||
return out
|
||||
|
||||
|
||||
def listy(front, key):
|
||||
v = front.get(key)
|
||||
if v is None:
|
||||
return []
|
||||
return v if isinstance(v, list) else ([v] if str(v).strip() else [])
|
||||
|
||||
|
||||
def main():
|
||||
failures = []
|
||||
|
||||
def bad(path, why):
|
||||
failures.append(" %s: %s" % (rel(path), why))
|
||||
|
||||
# ---- design ------------------------------------------------------------------------
|
||||
for layer, name in (("00-as-is", "as-is"), ("01-to-be", "to-be")):
|
||||
for path in sorted(glob.glob(os.path.join(ROOT, "03-DESIGN", layer, "*.md"))):
|
||||
if os.path.basename(path) == "README.md":
|
||||
continue
|
||||
front = frontmatter(path)
|
||||
if front is None:
|
||||
bad(path, "no frontmatter")
|
||||
continue
|
||||
if front.get("layer") != name:
|
||||
bad(path, "layer is %r; this directory is %s" % (front.get("layer"), name))
|
||||
status = front.get("status")
|
||||
if status not in DESIGN_STATUSES:
|
||||
bad(path, "status %r is not one of %s" % (status, sorted(DESIGN_STATUSES)))
|
||||
if name == "to-be":
|
||||
if not listy(front, "decisions"):
|
||||
bad(path, "names no decisions -- no design without a decision")
|
||||
if status in ("in-progress", "implemented") and not listy(front, "code"):
|
||||
bad(path, "status %s but code: names no owner -- no development "
|
||||
"without a design that says where" % status)
|
||||
|
||||
# ---- issues ------------------------------------------------------------------------
|
||||
for path in sorted(glob.glob(os.path.join(ROOT, "04-ISSUES", "*", "00-report.md"))):
|
||||
front = frontmatter(path)
|
||||
if front is None:
|
||||
bad(path, "no frontmatter")
|
||||
continue
|
||||
status = front.get("status")
|
||||
if status not in ISSUE_STATUSES:
|
||||
bad(path, "status %r is not one of %s" % (status, sorted(ISSUE_STATUSES)))
|
||||
if status in ("located", "resolved") and not listy(front, "located-in"):
|
||||
bad(path, "status %s but located-in is empty" % status)
|
||||
if status == "resolved" and not listy(front, "fixed-by"):
|
||||
bad(path, "status %s but fixed-by says nothing" % status)
|
||||
|
||||
# ---- research ----------------------------------------------------------------------
|
||||
for path in sorted(glob.glob(os.path.join(ROOT, "01-RESEARCH", "*", "00-overview.md"))):
|
||||
front = frontmatter(path)
|
||||
if front is None:
|
||||
bad(path, "no frontmatter")
|
||||
continue
|
||||
status = front.get("status")
|
||||
if status not in RESEARCH_STATUSES:
|
||||
bad(path, "status %r is not one of %s" % (status, sorted(RESEARCH_STATUSES)))
|
||||
if status == "graduated":
|
||||
became = listy(front, "became")
|
||||
if not became:
|
||||
bad(path, "graduated but became: names nothing")
|
||||
for target in became:
|
||||
if not os.path.exists(os.path.join(ROOT, target)):
|
||||
bad(path, "became names %s, which does not exist" % target)
|
||||
|
||||
# ---- decisions -------------------------------------------------------------------
|
||||
records = {}
|
||||
for path in sorted(glob.glob(os.path.join(ROOT, "02-DECISIONS", "[0-9]*.md"))):
|
||||
front = frontmatter(path)
|
||||
records[os.path.basename(path)] = (path, (front or {}).get("status"))
|
||||
|
||||
cited = set()
|
||||
sources = (glob.glob(os.path.join(ROOT, "03-DESIGN", "*", "*.md"))
|
||||
+ glob.glob(os.path.join(ROOT, "01-RESEARCH", "*", "00-overview.md"))
|
||||
+ glob.glob(os.path.join(ROOT, "04-ISSUES", "*", "00-report.md"))
|
||||
+ glob.glob(os.path.join(ROOT, "00-META", "**", "*.md"), recursive=True))
|
||||
for path in sources:
|
||||
text = open(path, encoding="utf-8").read()
|
||||
if os.sep + "03-DESIGN" + os.sep in path:
|
||||
# A design doc's governing citations live in frontmatter; a prose mention is
|
||||
# commentary, not a home.
|
||||
m = re.match(r"^---\n(.*?)\n---", text, re.S)
|
||||
text = m.group(1) if m else ""
|
||||
for m in re.finditer(r"([0-9]{4}-[^\s\)\],#]+\.md)", text):
|
||||
cited.add(os.path.basename(m.group(1)))
|
||||
for name in records:
|
||||
front = frontmatter(records[name][0]) or {}
|
||||
for key in ("extends", "supersedes", "superseded-by"):
|
||||
v = front.get(key)
|
||||
if isinstance(v, str) and v:
|
||||
cited.add(os.path.basename(v))
|
||||
for name, (path, status) in records.items():
|
||||
if status == "accepted" and name not in cited:
|
||||
bad(path, "an accepted decision nothing in the cycle cites -- give it a home in a "
|
||||
"design doc's decisions:, a research became:, an issue, or 00-META")
|
||||
|
||||
checked = (
|
||||
len(glob.glob(os.path.join(ROOT, "03-DESIGN", "0*", "*.md")))
|
||||
+ len(glob.glob(os.path.join(ROOT, "04-ISSUES", "*", "00-report.md")))
|
||||
+ len(glob.glob(os.path.join(ROOT, "01-RESEARCH", "*", "00-overview.md")))
|
||||
+ len(records)
|
||||
)
|
||||
if failures:
|
||||
print("cycle: %d document(s) break the development cycle:" % len(failures))
|
||||
print("\n".join(failures))
|
||||
return 1
|
||||
print("cycle: %d documents checked, the chain holds" % checked)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -284,6 +284,37 @@ def check_numbering(failures, records):
|
||||
)
|
||||
|
||||
|
||||
def check_status_against_code(failures):
|
||||
"""A design document naming specific code may not still call itself `designed`.
|
||||
|
||||
**Naming a file is a claim that the file implements this**, so the two fields have to agree.
|
||||
They drifted: ten to-be documents named working, lab-proven code — several with a *What was
|
||||
built* or *Raised, and observed* section — while still saying nothing had been built.
|
||||
|
||||
Deliberately weak, and that is the point of it being mechanical. It cannot tell whether the
|
||||
prose is true, only that a document has stopped claiming to be unbuilt once it points at
|
||||
something. `code: [mesh-controller]` — a repository with no path — is a plan and stays
|
||||
`designed`.
|
||||
"""
|
||||
for path in markdown_files():
|
||||
if not rel(path).startswith("03-DESIGN/01-to-be/") or path.endswith("README.md"):
|
||||
continue
|
||||
front = frontmatter(read(path))
|
||||
if front.get("status") != "designed":
|
||||
continue
|
||||
for entry in front.get("code") or []:
|
||||
named = re.sub(r"\s*\(.*\)$", "", entry).strip().split(None, 1)
|
||||
if len(named) > 1:
|
||||
failures.add(
|
||||
"status-vs-code",
|
||||
rel(path),
|
||||
f"`designed`, but names {named[1]!r} in {named[0]}. Naming a file claims "
|
||||
f"it implements this — use `in-progress`, or `implemented` once it is "
|
||||
f"defensible from that repository's main branch.",
|
||||
)
|
||||
break
|
||||
|
||||
|
||||
def main():
|
||||
failures = Failures()
|
||||
records = load_records()
|
||||
@@ -293,6 +324,7 @@ def main():
|
||||
check_supersession_symmetry(failures, records)
|
||||
check_numbering(failures, records)
|
||||
check_topics(failures, records)
|
||||
check_status_against_code(failures)
|
||||
print(f"records: {len(records)} decision records checked")
|
||||
return failures.report()
|
||||
|
||||
|
||||
+1
-1
@@ -30,7 +30,7 @@ named, and nothing should be designed around a particular one existing.
|
||||
- **A hosted model provider** supplies the thinking for non-human agents, drawn from a
|
||||
shared pool of subscriptions — which is why budget pacing is a first-class concern.
|
||||
- **Long-lived user services** rather than an orchestrator. No cluster scheduler, no cloud
|
||||
control plane.
|
||||
controller.
|
||||
|
||||
Defaults, not mandates. A second model provider is anticipated by design; nothing in the
|
||||
domain may assume one vendor's credential lifecycle.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Glossary — the words this repository uses, and the ones it stopped using
|
||||
|
||||
One name per thing. This page is the authority; where an older record says something else, that
|
||||
record is being superseded, not this page. It exists because the terms kept drifting in
|
||||
conversation — control plane / controller / master / hub for one thing, substrate / foundation for
|
||||
another — and a mesh you cannot name precisely is a mesh two people describe differently.
|
||||
|
||||
## The mesh and its machines
|
||||
|
||||
- **node** — a machine in the mesh. There are 0..n of them, and each runs the host agent. A node is
|
||||
just a machine that has joined; being one implies nothing about what it runs.
|
||||
- **control-node** — the one node that also holds the `mesh-controller` seat. There is exactly one
|
||||
per mesh. "control-node" is not a separate kind of machine — it is a node that additionally runs
|
||||
the controller (and, today, the foundation). Lose it and the other nodes keep running what they
|
||||
were last told; they simply cannot be told anything new.
|
||||
- ~~master / slave~~, ~~hub / peer~~ — not used. The relationship is *controller and nodes*, and no
|
||||
node is subordinate: a node applies declarations on its own and survives the control-node dying.
|
||||
|
||||
## What runs the mesh
|
||||
|
||||
- **controller** — the component that decides what each node should be, holds the mesh's records,
|
||||
and tells nodes over the broker. Replaces **"control plane"** (borrowed from networking's
|
||||
control-plane/data-plane, and opaque here).
|
||||
- **mesh-controller** — the module that runs the controller. It **claims** the `mesh-controller`
|
||||
seat at mesh scope, which is what makes it singular. Replaces the module name **`mesh-control`**.
|
||||
(The git repository has been renamed `mesh-control` -> `mesh-controller` on the forge; the module,
|
||||
container and image it produces are `mesh-controller`.)
|
||||
- **foundation** — the store and the broker, raised at genesis before any module system exists.
|
||||
Replaces **"substrate"** (a biology metaphor that landed for no one). The foundation is not a
|
||||
third thing beside the store and broker — it *is* those two, named together.
|
||||
- **store** — the one postgres server. It holds the controller's own context databases
|
||||
(`inventory`, `identity`, `licences` — a context owns its store, [ADR 0008](../02-DECISIONS/0008-a-context-owns-its-store.md))
|
||||
and every module's own database. One server, many databases — never one shared "mesh database".
|
||||
- **broker** — the one lavinmq message bus. It carries the mesh bus on the `/` vhost and a vhost per
|
||||
consumer that requires `amqp`.
|
||||
|
||||
## What the mesh stores and serves
|
||||
|
||||
- **package** — what code resolves when it is **compiled**: an npm/cargo/pypi dependency, by
|
||||
**version**. Served by the **package-registry** (gitea). Only a builder talks to it.
|
||||
- **artifact** — what the mesh delivers to a machine to **install and run**: an OCI image, by
|
||||
**digest**. Served by the **artifact-store** (distribution). Every node pulls from it.
|
||||
- These are two protocols, not one store being weak — see [ADR 0075](../02-DECISIONS/0075-two-stores-and-which-provides-what.md).
|
||||
|
||||
## How modules relate to the mesh
|
||||
|
||||
- **seat** — a named position at a scope (node / site / mesh) with a **capacity**. A capacity-1 seat
|
||||
is exclusive (one holder); a higher-capacity seat is a **bench** (several holders coexist).
|
||||
- **claim** — a module taking a spot on a seat. `claims: [{name, scope}]` in a manifest. A
|
||||
mesh-scoped exclusive claim is how the mesh says "there is one of me". A foundation seat is
|
||||
named after the server it guards: the `mesh-controller`, `postgres` and `lavinmq` modules claim
|
||||
the `mesh-controller`, `mesh-store` and `mesh-broker` seats ([ADR 0079](../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)).
|
||||
- **provision** — a service one module `provides` and others `require`; the mesh resolves a provider
|
||||
and wires the two with an endpoint and a credential. This is separate from seats: a provision is
|
||||
a service you offer, a seat is a slot you occupy.
|
||||
|
||||
## How this page is kept
|
||||
|
||||
A new name for an existing thing lands here first, in the same change that introduces it in code. A
|
||||
record under `02-DECISIONS/` keeps whatever word it was written with — those are immutable — so a
|
||||
term retired here may still appear there, and the mapping above is how to read it.
|
||||
@@ -1,6 +1,10 @@
|
||||
# Process — overview
|
||||
|
||||
How work moves through HQ, and who may do what. Every other document in this folder is a
|
||||
How work moves through HQ, and who may do what. In industry terms this is **spec-driven
|
||||
development, with provenance**: the decision is the why, the design doc is the spec, `code:`
|
||||
names the implementation, and the lab beds are the conformance tests — and unlike the common
|
||||
form, the chain itself is checked ([ADR 0080](../../02-DECISIONS/0080-the-development-cycle-is-checked.md),
|
||||
[0081](../../02-DECISIONS/0081-a-decision-nothing-cites-is-not-yet-in-the-chain.md)). Every other document in this folder is a
|
||||
playbook: trigger, who runs it, steps, outputs. Engineers and agents follow the same
|
||||
playbooks; agents must not act outside them.
|
||||
|
||||
@@ -49,6 +53,19 @@ the expensive half.
|
||||
| [03](03-issues.md) | Issues | Something is wrong — often with the owner unknown |
|
||||
| [04](04-build-handoff.md) | Build handoff | A design is ready to be built |
|
||||
| [05](05-constitution-sync.md) | Constitution sync | `how-we-build.md` changed a rule the mesh enforces |
|
||||
| [06](06-writing-a-module.md) | Writing a module | Something that runs today must run on the mesh |
|
||||
| [07](07-feature-branches.md) | Feature branches across repos | Work that changes code, in one repo or several at once |
|
||||
|
||||
## The cycle is checked
|
||||
|
||||
The flow above is a rule, and a rule states how it is checked:
|
||||
[`00-META/checks/cycle.py`](../checks/cycle.py) refuses a to-be design that names no
|
||||
decision, an `in-progress`/`implemented` design that names no owning code, an issue marked
|
||||
`located`/`fixed` with no owner or `fixed`/`resolved` with no fix, and a `graduated`
|
||||
research overview that does not say what it became. Run it with `records.py` and `index.py`
|
||||
before any HQ merge. What the checks cannot see — that code work actually started from a
|
||||
handoff — is held by playbooks [04](04-build-handoff.md) and [07](07-feature-branches.md):
|
||||
a feature branch exists because a design or an issue sent it.
|
||||
|
||||
## Status lives in frontmatter
|
||||
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Playbook 05 — Constitution sync
|
||||
|
||||
Implements [ADR 0021](../../02-DECISIONS/0021-hq-is-the-source-of-the-constitution.md): HQ is
|
||||
the source of the constitution, and the knowledge-base page is derived, never edited.
|
||||
|
||||
**Trigger.** [`how-we-build.md`](../how-we-build.md) changed a rule that the mesh enforces at
|
||||
runtime.
|
||||
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
# Playbook 06 — Writing a module
|
||||
|
||||
**Trigger.** Something that runs today must run on the mesh, or a new capability must be
|
||||
declarable.
|
||||
|
||||
**Who runs it.** Whoever is porting or writing it.
|
||||
|
||||
*Written 2026-09-01 from doing this for the first time end to end. Every step below exists
|
||||
because skipping it cost something.*
|
||||
|
||||
## Before anything: read what runs
|
||||
|
||||
**A module is written from the thing, not from memory of the thing.** For a port, that means its
|
||||
current compose file, its environment, and where its data actually sits. Assumptions about any of
|
||||
the three have been wrong every time they were not checked.
|
||||
|
||||
Three questions, answered from the machine:
|
||||
|
||||
| | why it decides something |
|
||||
|---|---|
|
||||
| **what containers, and how do they find each other?** | more than one means a `network`; names between them must match what the software is configured to dial |
|
||||
| **where is its data?** | a bind mount moves with a path; a named volume does not; an anonymous volume is already losing data on every redeploy |
|
||||
| **which values are secret, and which are merely settings?** | a secret goes in `own-secrets` or a grant; a setting goes in the manifest and may be overridden per node |
|
||||
|
||||
## The steps
|
||||
|
||||
1. **Name what it provides and requires**, if anything. A name is what a consumer is coupled to,
|
||||
not the role it plays ([ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md)):
|
||||
`postgres-database`, not `database`. Most modules provide nothing and require nothing — an
|
||||
application is usually a leaf.
|
||||
|
||||
2. **Declare capabilities, not dependencies, for facts about the machine.** `container-runtime`,
|
||||
`package-manager`, `seat`. A capability is detected and refused against; it is not something a
|
||||
module can install.
|
||||
|
||||
3. **Write the resources in the order they must happen.** They are applied in the order written
|
||||
and orphans are removed in reverse, so a `network` is written before the containers that join
|
||||
it and removed after them.
|
||||
|
||||
4. **Put every secret in a file, never in `env`.** A declaration travels over the broker in plain
|
||||
text: a password in `env` is a password the broker sees. **The mesh delivers parts; a module
|
||||
that needs them combined combines them.**
|
||||
|
||||
**A sealed file holds the password and nothing else** — no key, no `=`, no newline that means
|
||||
anything. So `env-file` must never point at one. It points at a file the module *declares*,
|
||||
whose content leaves a hole:
|
||||
|
||||
```
|
||||
own-secrets superuser → /var/lib/postgres/superuser.secret the password, alone
|
||||
a file /var/lib/postgres/superuser.env, mode 0600,
|
||||
content: POSTGRES_PASSWORD=${secret:superuser}
|
||||
the container env-file: [/var/lib/postgres/superuser.env]
|
||||
```
|
||||
|
||||
The host fills the hole on the machine, which is the only place both halves exist — the mesh
|
||||
discarded the value
|
||||
([credentials and their rotation](../../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)).
|
||||
A **provisioner** is the exception: it reads a password file, so it mounts the `.secret`
|
||||
directly.
|
||||
|
||||
Every example module in `mesh-controller` had this wrong and shipped: `own-secrets` pointing at a
|
||||
path *named* `.env`, mounted as `env-file`, holding a bare password. Docker reads that as a
|
||||
malformed line and the container starts **with no password set at all** — not a failure to
|
||||
start, a service running on the wrong credential. They parsed and they resolved. Two tests in
|
||||
`examples/modules` now refuse both halves of it.
|
||||
|
||||
Add `restart-on` naming the env file, or the container keeps the credential it started with
|
||||
through every rotation.
|
||||
|
||||
5. **Pin every image by digest.** A tag moves. The manifest in a repository names artifacts; the
|
||||
manifest the mesh holds names digests, and they are not the same document.
|
||||
|
||||
6. **Decide generate or accept.** A new module's credential is generated. **An adopted one keeps
|
||||
the credential it already has** — `secret accept` — because minting a new password for a
|
||||
database that already exists locks the application out of its own data.
|
||||
|
||||
7. **Add a provisioner only if the software cannot read a file.** A proxy that watches a
|
||||
directory needs nothing. PostgreSQL needs `CREATE ROLE`, an object store needs a bucket and a
|
||||
policy, an identity provider needs a realm and a client — those need a small program beside
|
||||
them. It reads what the mesh granted and reconciles; it does not decide anything.
|
||||
|
||||
8. **Prove it in the lab, against the real software.** Not that a container started — that the
|
||||
thing works: the credential authenticates, a wrong one is refused, the containers reach each
|
||||
other, the data survives a restart.
|
||||
|
||||
## What the first port actually cost
|
||||
|
||||
Six attempts, one real bug. Recorded because the ratio is the lesson: **the mesh was right every
|
||||
time and the scaffolding was not.**
|
||||
|
||||
- A shape existed in the language and no host implemented it, so every declaration carrying one
|
||||
was refused whole — correctly, and the host said exactly that. **Nobody was reading the host's
|
||||
log.** Read it first; it is the only place that says why a machine did nothing.
|
||||
- A blind find-and-replace renamed a provision in quotes and missed the same word bare.
|
||||
- A command was tested only for the invocations that should fail, so it rejected every real one
|
||||
and the suite stayed green.
|
||||
- A test asserted on a helper rather than on the code that calls it, three separate times. **A
|
||||
test that cannot fail when the behaviour is deleted is not defending the behaviour.**
|
||||
|
||||
## Rules
|
||||
|
||||
- **Read the host's log before theorising.** A declaration that was sent and not applied says so
|
||||
there and nowhere else.
|
||||
- **A failing test is kept, not skipped.** It is the reproduction.
|
||||
- **Never rotate during an adoption.** Rotation is a separate act, afterwards, deliberately.
|
||||
- **A data directory is never removed by the mesh** ([ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md)),
|
||||
and that protects against the mesh only — not against a disk or a mistaken command.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Playbook 07 — Feature branches across repos
|
||||
|
||||
**Trigger.** Work that changes code — in one code repo or in several at once (`mesh-sdk`,
|
||||
`mesh-controller`, `mesh-catalog`, `mesh-host`, `mesh-lab`, and `hq` when a decision rides along).
|
||||
|
||||
**Who runs it.** Anyone who writes code, engineers and agents alike. Agents follow it exactly —
|
||||
it is the guard against the failure it was written for.
|
||||
|
||||
## The failure it prevents
|
||||
|
||||
A feature was worked as a branch-and-MR per *unit of thought* — one per decision, one per
|
||||
stacked increment — and each MR was treated as finished when it was *opened*, not when it was
|
||||
*merged*. Across repos the same feature took a different branch name in each. The MRs piled up
|
||||
unmerged: one session left **sixteen** stacked intermediate MRs that had to be consolidated and
|
||||
closed by hand. An MR is a review checkpoint, not a scratchpad.
|
||||
|
||||
## The rule
|
||||
|
||||
One feature is **one branch name**, **one worktree per repo**, **one MR per repo**, opened
|
||||
**once, at the end**.
|
||||
|
||||
1. **Name the feature once.** `feat/<slug>`. The *same* branch name in every repo the feature
|
||||
touches — never a different name per repo, never a fresh branch per increment within the
|
||||
feature.
|
||||
2. **Isolate each repo.** One git worktree per touched repo under `.work/<slug>/<repo>`, branched
|
||||
off `main`:
|
||||
```
|
||||
git worktree add .work/<slug>/<repo> -b feat/<slug> origin/main
|
||||
```
|
||||
Parallel features never collide, and no shared checkout is edited.
|
||||
|
||||
**Add the untouched siblings the lab reads.** The lab beds find the other repositories by
|
||||
sibling path from the lab checkout (`../mesh-tools/module.json`, `../mesh-sdk`, …), the way
|
||||
the main layout has them. A `.work/<slug>/` directory holding only the touched repos fails a
|
||||
bed at once with `no manifest for mesh-tools at …/.work/<slug>/mesh-tools/module.json`, after
|
||||
genesis has already passed. Give the directory those repos as **detached worktrees on `main`**
|
||||
— never symlinks:
|
||||
```
|
||||
git worktree add --detach .work/<slug>/mesh-tools main
|
||||
```
|
||||
3. **Commit as you go — locally.** Increments land on the one branch. Nothing is pushed and no
|
||||
MR is opened mid-feature.
|
||||
4. **Finish, then publish.** When the whole feature is done — every repo, tests green — push
|
||||
every branch and open **one MR per touched repo**, together.
|
||||
5. **Merge promptly, once approved.** Every merge into `main` is notified and approved
|
||||
([ADR 0023](../../02-DECISIONS/0023-approval-is-the-checkpoint.md)); once it is, merge —
|
||||
do not leave it sitting. The branch is deleted on merge.
|
||||
6. **Leave nothing behind.** After the MRs merge, no `feat/<slug>` branch and no `.work/<slug>`
|
||||
worktree survive.
|
||||
|
||||
## What this is not
|
||||
|
||||
- **Not a licence to batch unbounded work.** A feature is a *bounded* unit; if it sprawls for
|
||||
days, end-of-feature bloat merely replaces per-increment bloat. Split it into features, each
|
||||
its own branch and MR.
|
||||
- **Not a second trunk.** Every repo branches off `main`. There is no longer an `initialization`
|
||||
trunk.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The end state is visible, and its absence is the smell:
|
||||
|
||||
- After a feature merges, `git branch -r | grep feat/<slug>` and `git worktree list` return
|
||||
nothing for it. A surviving branch or worktree means step 6 was skipped.
|
||||
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it
|
||||
cannot find is a missing sibling worktree, not a mesh fault.
|
||||
- More than one open MR in a repo that share no feature name, or a stack of MRs none of which is
|
||||
merged, is the failure this playbook exists to prevent — stop and consolidate before opening
|
||||
more.
|
||||
+8
-6
@@ -21,17 +21,19 @@ and a forge address is an operational detail (see [`README`](../README.md)).
|
||||
## What the mesh becomes
|
||||
|
||||
[ADR 0019](../02-DECISIONS/0019-how-this-repository-works.md) records the repositories the
|
||||
monorepo decomposes into. **`mesh-lab`, `mesh-host` and `mesh-control` exist so far** — the lab is built first
|
||||
([ADR 0016](../02-DECISIONS/0016-the-lab.md)); the rest are the
|
||||
target, not the present.
|
||||
monorepo decomposes into. **`mesh-host`, `mesh-controller`, `mesh-catalog`, `mesh-lab`, `mesh-sdk` and `mesh-tools` exist
|
||||
so far** — the lab was built first ([ADR 0016](../02-DECISIONS/0016-the-lab.md)). The tiered
|
||||
decomposition below is the planned shape; the repositories built to date do not map onto it
|
||||
one-for-one — `mesh-catalog`, `mesh-sdk` and `mesh-tools` exist where the table names
|
||||
`mesh-foundation` and `mesh-surfaces`, and reconciling the two is itself still ahead.
|
||||
|
||||
| Repository | Tier | Holds |
|
||||
|---|---|---|
|
||||
| `mesh-host` | 0 | **exists.** The node host — one statically linked binary, requiring nothing present ([ADR 0005](../02-DECISIONS/0005-the-node-host.md)) |
|
||||
| `mesh-substrate` | 1 | the four pinned services, as declarations |
|
||||
| `mesh-control` | 2 | **exists.** The control plane and its contexts — one of seven built ([ADR 0006](../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
|
||||
| `mesh-foundation` | 1 | the four pinned services, as declarations |
|
||||
| `mesh-controller` | 2 | **exists.** The controller and its contexts — one of seven built ([ADR 0006](../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
|
||||
| `mesh-surfaces` | 3 | tools, web, cli |
|
||||
| `mesh-sdk` | — | contracts shared across tiers |
|
||||
| `mesh-sdk` | — | the stable spine modules build against — the tool-serving harness, the messaging/event framework, the contracts and core primitives. Holds nothing per-module and nothing volatile ([ADR 0039](../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)). |
|
||||
| `mesh-lab` | — | **exists.** The lab — scenario lifecycle, networking, placement. Ships to nobody; runs on a workstation. |
|
||||
|
||||
Tier 4's shape is open, and deliberately so: see ADR 0019 and
|
||||
|
||||
@@ -3,7 +3,6 @@ status: graduated
|
||||
initiated: 2026-08-22
|
||||
touches: [03-DESIGN/00-as-is/05-runtime-and-installation.md]
|
||||
became:
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 03-DESIGN/01-to-be/05-the-node-host.md
|
||||
---
|
||||
|
||||
@@ -90,5 +90,5 @@ the catalogue where modules genuinely change together under one intent. The skel
|
||||
| One repository per tier, or per context? | Already open from ADR 0001 as "catalogue destination — one repository or many". The skeleton assumes per tier and does not settle it. |
|
||||
| ~~Does an unprivileged node earn a place in the inventory, or only a presence?~~ | **Answered 2026-08-25** by the operator: a node is a *managed machine inside the mesh*, not an unprivileged something — and a disconnected node is still a node, in a different situation. The question posed a class distinction; the answer is that there is none, and what varies is **state**. Recorded as [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md). |
|
||||
| ~~Does absorbing overlay, filtering, packages, supervision and the container runtime make the host too large?~~ | **Answered 2026-08-25** — [`host-size.md`](host-size.md). Measured: the absorption is smaller than the machinery that already applies state, and eight of ten adapters already carry no dependency. The risk is not size but direction, and it is two modules wide. The claim survives with its scope corrected — the host carries one concern, *apply declared state on this machine*, of which the six are instances. Recorded as [ADR 0005](../../02-DECISIONS/0005-the-node-host.md), designed in [`05-the-node-host.md`](../../03-DESIGN/01-to-be/05-the-node-host.md). |
|
||||
| ~~Four substrate services or five?~~ | **Answered conditionally**, which is the honest form — [`07-the-substrate.md`](../../03-DESIGN/01-to-be/07-the-substrate.md). The substrate is *what the control plane consumes and cannot grant itself*. The identity provider qualifies only if the control plane delegates authentication; if it authenticates natively it is an ordinary hosted service. The count follows from a decision not yet taken, and asserting four was asserting that decision. |
|
||||
| ~~Four substrate services or five?~~ | **Answered conditionally**, which is the honest form — [`07-the-foundation.md`](../../03-DESIGN/01-to-be/07-the-foundation.md). The substrate is *what the control plane consumes and cannot grant itself*. The identity provider qualifies only if the control plane delegates authentication; if it authenticates natively it is an ordinary hosted service. The count follows from a decision not yet taken, and asserting four was asserting that decision. |
|
||||
| Does `feature` survive? | The skeleton splits it in two and argues the conflation is what makes the delivery pipeline hard to reason about. Unproven. |
|
||||
|
||||
@@ -4,8 +4,8 @@ initiated: 2026-08-25
|
||||
became:
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0008-a-context-owns-its-store.md
|
||||
- 03-DESIGN/01-to-be/06-the-control-plane.md
|
||||
- 03-DESIGN/01-to-be/07-the-substrate.md
|
||||
- 03-DESIGN/01-to-be/06-the-controller.md
|
||||
- 03-DESIGN/01-to-be/07-the-foundation.md
|
||||
touches:
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: proposed
|
||||
status: accepted
|
||||
date: 2026-08-30
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0019-how-this-repository-works.md
|
||||
---
|
||||
|
||||
# 25. The design record is read where it is written, never copied to be found
|
||||
|
||||
## Context
|
||||
|
||||
**These documents cannot be found by searching the mesh's memory, and never could.** Checked on
|
||||
2026-08-23 and again on 2026-08-31, against both the symptom-indexed store and the structured
|
||||
archive, using a decision record's full title and a distinctive phrase from a design document: no
|
||||
result, no partial match, no stale copy.
|
||||
|
||||
That matters because of what was promised. The objection to giving this material its own
|
||||
repository was that the mesh already has a knowledge store, and a second one repeats the mistake
|
||||
that store was created to fix. **The answer offered was indexing rather than location** — that
|
||||
these documents would be returned beside everything else in a search, so where they were authored
|
||||
became a separate question. The indexing was never built.
|
||||
|
||||
**The claim has since stopped being load-bearing**, which is why this is a decision rather than an
|
||||
incident. [`README.md`](../README.md) names the gap in the place the claim used to sit, and
|
||||
[ADR 0019](0019-how-this-repository-works.md)'s reasoning rests on cadence, reviewers and scope —
|
||||
none of which depend on being searchable from elsewhere. What remained was an unbuilt capability
|
||||
and an open question, recorded as
|
||||
[`04-ISSUES/006`](../04-ISSUES/006-hq-is-not-indexed-into-the-knowledge-base/00-report.md).
|
||||
|
||||
**A signpost was added on 2026-08-31 and measured.** One entry in the mesh's memory naming what
|
||||
lives here and when to come looking. A search for *design records, decisions, repository* returns
|
||||
it; a search phrased the way somebody actually asks — *why is the mesh built this way* — returns
|
||||
nothing, because the store matches terms and not meaning. **Reachable is not the same as
|
||||
surfacing**, and the measurement is what established which one a signpost buys.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A one-way sync into the mesh's memory.** A job reads this repository on a schedule and writes
|
||||
the documents into the searchable store. It works with what exists today and needs nothing
|
||||
built first. **Rejected**, because it creates a second copy of every document, and the failure
|
||||
mode of a derived copy is the one this repository is least able to tolerate: *the copy that is
|
||||
searched quietly stops matching the copy that is edited*, and the enforced one wins. A design
|
||||
record that has silently diverged from the reasoning it claims to carry is worse than one that
|
||||
cannot be found — the first misleads, the second merely fails.
|
||||
|
||||
2. **Leave the signpost and close nothing.** Honest, free, and it keeps the gap visible.
|
||||
**Rejected as an end state**, though it is what stands until the option below exists. It
|
||||
answers only for a reader who already suspects these documents exist, which is precisely not
|
||||
the reader the mesh's memory is designed for.
|
||||
|
||||
3. **An agent reads this repository directly, and the search consults it.** Nothing is copied.
|
||||
**Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**The design record is read where it is written.** Retrieval is an agent reading this repository,
|
||||
not a copy living in a second store — and a search of the mesh's memory consults that agent, so
|
||||
what it knows appears beside ordinary results rather than only when it is asked.
|
||||
|
||||
Both halves are the decision. The first alone is merely a reader, and would leave this repository
|
||||
reachable but not surfacing — the state measured above. **The second half is what discharges the
|
||||
promise** that these documents are returned beside everything else.
|
||||
|
||||
**There is no copy, and that is the point.** No sync, no schedule, no reconciliation, and nothing
|
||||
that can drift, because there is only ever one of each document. It is also always current,
|
||||
including for work that is not yet committed.
|
||||
|
||||
**The direction of reading is one-way and stays that way.** The agent reads this repository and
|
||||
answers from it. Nothing flows back: this repository is public, the mesh is not, and a return path
|
||||
would be how installation-specific detail arrives into documents that must not carry it
|
||||
([`README.md`](../README.md)).
|
||||
|
||||
## Consequences
|
||||
|
||||
**This repository stops being a fourth knowledge system, properly.** The original objection was
|
||||
about adding a knowledge *system*. An agent with read access adds no store at all — which answers
|
||||
the objection more completely than the indexing that was promised, rather than merely as well.
|
||||
|
||||
**ADR 0019's promise is amended, not satisfied.** It said these documents would be *indexed*. They
|
||||
will not be. They will be *read*, and the search will ask. The commitment that survives is the one
|
||||
that mattered — that a searcher finds them without already suspecting they exist — and the
|
||||
mechanism behind it is different from the one named.
|
||||
|
||||
**It is gated on an agent that does not exist yet.** Until it does, the signpost is what stands,
|
||||
and this repository is reachable rather than surfacing. That is a known and stated gap, not a
|
||||
silent one — and the gap is now a build task with a decided shape rather than an open question.
|
||||
|
||||
**The search must degrade honestly.** When the agent cannot be reached, a search has to say that
|
||||
this material was not consulted. A result set that silently omits it looks identical to one where
|
||||
nothing matched, and *silence and success must never look alike*
|
||||
([ADR 0004](0004-a-node-and-how-it-joins.md)) — the rule this repository has now paid for twice.
|
||||
|
||||
**A rule states how it is checked, and this one is checkable.** The check is the measurement that
|
||||
produced this record: search the mesh's memory for a phrase that appears only in a design document
|
||||
here, and require it back. That check fails today, deliberately, and passing it is what closes
|
||||
`04-ISSUES/006`.
|
||||
|
||||
## References
|
||||
|
||||
- [`04-ISSUES/006`](../04-ISSUES/006-hq-is-not-indexed-into-the-knowledge-base/00-report.md) —
|
||||
the gap, the two measurements, and why closing it early was refused
|
||||
- [ADR 0019](0019-how-this-repository-works.md) — the promise this amends
|
||||
- [`README.md`](../README.md) — the objection, and the gap named where the claim used to sit
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
---
|
||||
|
||||
# 26. The mesh has a session of its own, and it is the node session's mechanism
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md) gives every node a session: one per node, permanent,
|
||||
remembering across callers, its system prompt the node's engram, reachable over the broker like
|
||||
everything else. **Any node can message any node**, and that is called the one part of the system
|
||||
that is genuinely a mesh — symmetric, with no centre.
|
||||
|
||||
**There is no way to address the mesh itself.** A question that spans machines — *what is running
|
||||
across all of this*, *which nodes are behind*, *why is it built this way* — has to be put to some
|
||||
node, which then asks the others. That works, and it makes a mesh-wide question **nobody's
|
||||
question**: every node answers it as a foreigner, from a position where the whole is not in view.
|
||||
|
||||
**Three things independently arrived at the same missing piece.**
|
||||
|
||||
[ADR 0025](0025-the-design-record-is-read-not-copied.md), taken hours before this one, commits to
|
||||
an agent that reads the design repository directly and answers into search. That agent has to
|
||||
exist, run somewhere, and be askable — and nothing in the record says what it is or where it
|
||||
lives.
|
||||
|
||||
[`14-model-access.md`](../03-DESIGN/01-to-be/14-model-access.md) records, as a gap deliberately
|
||||
not half-built: *this worker uses that licence is a binding to an agent, not to a node* — and the
|
||||
provisions model has no consumer identity other than a node. A session that must be assigned a
|
||||
licence is exactly that consumer, and node sessions are already one.
|
||||
|
||||
**And ADR 0004 never said how a session is set up.** It describes behaviour and stops: nothing
|
||||
states how a session starts, where its context lives, how the engram reaches it, or how a message
|
||||
off the broker becomes a prompt. There is no design document for it. That gap was invisible until
|
||||
something had to be built *like* a node session, because describing a second instance of a
|
||||
mechanism requires the mechanism to have been described once.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **No mesh session; keep relaying through a node.** Costs nothing and works today. **Rejected.**
|
||||
It leaves mesh-wide questions belonging to nobody, and it does not survive contact with
|
||||
ADR 0025 — that agent still needs a home, so the thing gets built anyway, unnamed, as an
|
||||
attachment to whichever node happened to host it.
|
||||
|
||||
2. **A new kind of agent, built separately.** Purpose-built for the whole mesh. **Rejected.** It
|
||||
would hold a session, a memory, a licence and broker plumbing — every one of which the node
|
||||
session already has. Two implementations of one mechanism drift, and the vocabulary collision
|
||||
that [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) exists to undo began exactly this
|
||||
way: two things that were nearly the same, built twice, until neither word meant one thing.
|
||||
|
||||
3. **The same mechanism, started in a different context.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh has one session, addressed as the mesh, and it is a node session in every respect but
|
||||
three.**
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **the context it starts in** | the mesh's, not a machine's — this is the whole of what makes it different |
|
||||
| **its engram** | the mesh's system prompt, as a node's engram is that node's |
|
||||
| **its licence binding** | assigned in its own right, not inherited from the machine it runs on |
|
||||
|
||||
Everything else is unchanged and deliberately so: it is permanent, it remembers, it is reachable
|
||||
over the broker, it holds its own tools, and switched off it still answers *I am switched off*
|
||||
rather than falling silent.
|
||||
|
||||
**It runs on the node that holds the control plane** — not for convenience, but because that node
|
||||
is already the one place excepted from *compromise of a node is compromise of that node*
|
||||
(ADR 0004). An agent able to reach everything, placed anywhere else, creates a **second** such
|
||||
place. Putting it where the authority already sits concentrates nothing new.
|
||||
|
||||
**It is an addition to per-node messaging and never a replacement.** Every node remains directly
|
||||
addressable. This is not a preference: ADR 0001 holds that losing the control plane costs *change,
|
||||
not operation*, and a mesh whose only conversational surface lives on that node would lose the
|
||||
ability to ask anything while every machine kept running perfectly. **The front door may not be
|
||||
the single point.**
|
||||
|
||||
**It is not an employee** ([ADR 0003](0003-agents-are-persistent-employees.md)). Nobody hires it,
|
||||
it holds no task queue, it is never drained or reassigned. What it does with work that belongs
|
||||
somewhere else is **dispatch it** — to node sessions, or to workers — which is what a node session
|
||||
already does when asked something it does not have.
|
||||
|
||||
**It is ADR 0025's reader.** The agent that reads the design repository and answers into search is
|
||||
this session, not a second one. One agent, one memory, one place to reach; two would both need
|
||||
that repository and would eventually disagree about what it says.
|
||||
|
||||
**"One per node" is about address, not about process count.** ADR 0004's rule — *two and nothing
|
||||
decides which replies* — forbids ambiguity in who answers when a **node** is addressed. The mesh
|
||||
session answers when the **mesh** is addressed. The control-plane node therefore hosts two
|
||||
sessions and no ambiguity, and stating this here is what stops it reading as a contradiction
|
||||
later.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The node session's setup must now be designed, and it never was.** This decision is expressed as
|
||||
*the same as a node session, elsewhere*, which is only meaningful once that mechanism is written
|
||||
down. The design document covering both is the immediate consequence of this record, not a
|
||||
follow-up to it.
|
||||
|
||||
**A consumer that is not a machine stops being deferrable.** The licence binding above is the gap
|
||||
`14-model-access.md` names, and it now has two consumers rather than a hypothetical one. Until it
|
||||
exists, a session's model access can only be expressed as *this module on this machine*, which
|
||||
cannot say *this node's session uses the personal licence and the mesh's uses the company one* —
|
||||
the thing the binding is for.
|
||||
|
||||
**Symmetry is preserved, and it is worth being precise about why.** ADR 0004's claim is about what
|
||||
a node can reach, and it is untouched: node-to-node messaging is unchanged, nothing is routed
|
||||
through the mesh session, and it is a participant rather than a hop. What arrives is a
|
||||
participant that happens to be the one a person usually addresses.
|
||||
|
||||
**Availability degrades to inconvenience rather than to silence** — but only because of the
|
||||
addition rule above. If that rule is ever relaxed, this consequence inverts, and it inverts
|
||||
quietly: everything keeps working and nobody can ask about it.
|
||||
|
||||
**The surface a person uses is not decided here.** That a board is a good place to talk to it is
|
||||
likely and is not this record's business; the session is reachable over the broker like everything
|
||||
else, and what puts a text box in front of it is a separate choice.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — the node session this extends
|
||||
- [ADR 0025](0025-the-design-record-is-read-not-copied.md) — the reader this session is
|
||||
- [ADR 0003](0003-agents-are-persistent-employees.md) — the vocabulary this is not
|
||||
- [`03-DESIGN/01-to-be/14-model-access.md`](../03-DESIGN/01-to-be/14-model-access.md) — *a
|
||||
consumer that is not a machine*, the gap this makes concrete
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 27. A provision names what the consumer is coupled to, not the role it plays
|
||||
|
||||
## Context
|
||||
|
||||
Provisions are named after roles. The catalogue and every test fixture built so far say:
|
||||
|
||||
```
|
||||
provides: database
|
||||
requires: database
|
||||
```
|
||||
|
||||
**Nothing distinguishes one engine from another.** A module requiring `database` is satisfied by
|
||||
any module providing `database`, so a module written against PostgreSQL can be matched to a
|
||||
provider of Microsoft SQL Server, resolve as satisfied, deploy, and fail on its first query.
|
||||
|
||||
**The mesh runs several engines** — PostgreSQL, Microsoft SQL Server, MariaDB, and others behind
|
||||
products that expose their own. This is not a hypothetical collision.
|
||||
|
||||
**The failure is in the direction that hides.** Resolution *succeeds*. Nothing is refused, nothing
|
||||
is logged, and the breakage surfaces later as an error inside an application, on a machine, with
|
||||
nothing connecting it back to a match made elsewhere by something that thought it had done its
|
||||
job. **A wrong answer delivered confidently costs more than a refusal**, and the whole point of
|
||||
refusing on ambiguity ([ADR 0009](0009-modules-and-the-graph.md)) was to not do this.
|
||||
|
||||
**How it got in:** every test written for the resolver had exactly one provider of each name, so no
|
||||
mismatch was expressible and none was caught. The fixtures agreed with the design. That is the same
|
||||
fault as [`04-ISSUES/005`](../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)'s
|
||||
imagined output and [`019`](../04-ISSUES/019-a-comment-asserting-a-fact-about-a-machine/00-report.md)'s
|
||||
unchecked comment, at the level of a name rather than a line.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep role names; let the operator assign correctly.** The mesh would refuse ambiguity when two
|
||||
providers exist, so a person picks. **Rejected.** It makes correctness depend on somebody
|
||||
knowing that the module they are assigning speaks a particular dialect — which is exactly the
|
||||
knowledge the provisioning model exists to remove. And with one provider of each name, nothing
|
||||
is ambiguous and nothing is asked.
|
||||
|
||||
2. **A role name plus a `flavour:` or `engine:` qualifier**, matched as a second field.
|
||||
**Rejected.** Two fields that must agree is a constraint the resolver has to enforce and a
|
||||
manifest author has to remember, to express something one field already can. The name is the
|
||||
contract; splitting it invites a requirement that names a role and forgets the qualifier, which
|
||||
then matches everything again.
|
||||
|
||||
3. **The name says what the consumer is coupled to.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**A provision is named for the thing a consumer's code is written against.**
|
||||
|
||||
```
|
||||
provides: postgres-database
|
||||
requires: postgres-database
|
||||
```
|
||||
|
||||
**The test is whether the consumer can tell the difference.** If swapping the provider would break
|
||||
the consumer, the name must say which provider — because a match that breaks the consumer is not a
|
||||
match. If the consumer genuinely cannot tell, a role name is correct and better.
|
||||
|
||||
| provision | | why |
|
||||
|---|---|---|
|
||||
| `postgres-database`, `mssql-database` | **specific** | applications are written against a dialect; a swap breaks them |
|
||||
| `route` | **role** | the consumer wants its name reachable and does not care what proxies it |
|
||||
| `resolver` | **role** | the consumer wants names to resolve |
|
||||
| `artifact-store` | **role** | the consumer fetches by digest over a protocol, and nothing else |
|
||||
|
||||
**`database` is not a provision and may not be provided.** There is no context in which an
|
||||
application talks to a generic database: it talks to PostgreSQL or it talks to SQL Server. A name
|
||||
that cannot be true of any real consumer should not be expressible.
|
||||
|
||||
**This is about coupling, not about products.** Two providers of `postgres-database` — a container
|
||||
on this node and a managed instance elsewhere — are interchangeable and *should* both match. What
|
||||
may not be interchangeable is what the consumer's queries are written in.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Every manifest that names a database changes.** Doing this now costs a rename across a handful of
|
||||
examples. Doing it after modules are migrated costs it across all of them, plus every deployment
|
||||
that resolved against the old name.
|
||||
|
||||
**Wrong requirements now fail loudly, and at the right moment.** A module requiring
|
||||
`postgres-database` where only `mssql-database` is provided is unsatisfiable, so it is **refused at
|
||||
resolution** with both names visible — rather than deployed and broken later. This is the property
|
||||
that was lost, restored.
|
||||
|
||||
**Generic role names are still right, and the rule says when.** This does not push specificity
|
||||
everywhere; it puts it exactly where a consumer is coupled. Naming `route` after a particular proxy
|
||||
would be the same error in the other direction, and would prevent a swap that genuinely changes
|
||||
nothing.
|
||||
|
||||
**It is checked, not merely stated** ([`00-META/how-we-build.md`](../00-META/how-we-build.md) §5).
|
||||
A manifest providing a name known to be engine-generic is refused, naming what to say instead.
|
||||
Without that, this record is a convention, and a convention is what the previous naming was.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0009](0009-modules-and-the-graph.md) — provisions, and refusing on ambiguity
|
||||
- [`03-DESIGN/01-to-be/07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md) — *the
|
||||
provisioning model uses databases, roles and schemas as PostgreSQL means them*, which is this
|
||||
record's point made about the substrate before it was made about modules
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 28. The substrate supplies the control plane and nothing else
|
||||
|
||||
*Corrects one row of [ADR 0006](0006-the-substrate-and-the-control-plane.md) and makes explicit
|
||||
something it left unsaid. The rest of that record stands.*
|
||||
|
||||
## Context
|
||||
|
||||
ADR 0006 defines the substrate by a circularity: **what the control plane needs in order to run,
|
||||
and cannot ask itself for, because it is not running yet.** Two questions, and both must be
|
||||
answered *yes* for something to be substrate.
|
||||
|
||||
Its membership table admits the object store on this line:
|
||||
|
||||
| role | product | |
|
||||
|---|---|---|
|
||||
| object store | **MinIO** | it cannot grant itself a bucket |
|
||||
|
||||
**That answers the second question and assumes the first.** It is true that a control plane cannot
|
||||
grant itself a bucket. Nothing establishes that it needs one.
|
||||
|
||||
**It does not.** Verified 2026-08-31 against `mesh-control`: no S3 client, no bucket, no object
|
||||
storage of any kind outside comments. Artifacts reach nodes as content-addressed blobs in the OCI
|
||||
registry, and the code records the decision and its reasoning:
|
||||
|
||||
> One store, and it is the registry the bootstrap already pulls from. An OCI registry is a
|
||||
> content-addressed blob store that happens to also understand images… The alternative considered
|
||||
> was a second store beside it — S3-shaped, buckets, signed URLs. It is the right answer for
|
||||
> objects that are *mutable*, or need per-reader access, or are not build output. None of that
|
||||
> describes a digest-pinned archive, and standing up a second service to hold one kind of
|
||||
> immutable blob means two things to run, two things to back up and two ways for an artifact to be
|
||||
> missing.
|
||||
|
||||
**The row is inherited from the system being replaced**, where an object store distributed module
|
||||
tarballs. Here nothing does, and the row was never re-tested against the definition it sits under.
|
||||
|
||||
**A second thing ADR 0006 never says:** whether a substrate service and a module of the same
|
||||
product are the same instance. It says the substrate is *not the control plane* and *not a place
|
||||
for logic*, and stops. The question is not idle — an application wanting a database, on a mesh
|
||||
whose substrate is already running PostgreSQL, has an obvious wrong answer available.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave the object store as substrate, unused.** Harmless-looking. **Rejected.** A membership
|
||||
list that includes something nothing needs is a list that has stopped being derived from its
|
||||
test, and the next member is admitted by precedent instead of argument. It also mandates that
|
||||
every mesh run a service no mesh uses.
|
||||
|
||||
2. **Applications share the substrate's instances.** One PostgreSQL, one of everything.
|
||||
**Rejected**, below.
|
||||
|
||||
3. **The substrate is exactly what the control plane consumes; everything else is a module.**
|
||||
**Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**The object store is not substrate.** It fails the first half of the test: the control plane does
|
||||
not need one. An object store is an ordinary module, required through the module graph like
|
||||
anything else, and a module wanting one depends on a module providing one.
|
||||
|
||||
**The substrate has four members, not five**: a relational store, a message bus, an image registry,
|
||||
and conditionally an identity provider. The registry stays — the control plane genuinely cannot
|
||||
deliver an artifact without somewhere to put it.
|
||||
|
||||
**A substrate service and a module of the same product are different instances, and are not
|
||||
shared.** The mesh's own PostgreSQL and a PostgreSQL a workload was given are two servers, two
|
||||
containers, two lifecycles.
|
||||
|
||||
Three reasons, and the first is the one that matters:
|
||||
|
||||
**The substrate is not in the module graph.** It is raised from the pinned bundle the host carries,
|
||||
before any mesh exists to declare it. A workload depending on it would depend on something the
|
||||
graph cannot see, cannot rotate a credential for, and cannot move — which is every property the
|
||||
provisioning model exists to provide.
|
||||
|
||||
**It would put workload data in the control plane's own store.** The mesh's contexts own their
|
||||
stores exclusively ([ADR 0008](0008-a-context-owns-its-store.md)). An application sharing that
|
||||
server can exhaust it, lock it, or fill its disk, and the failure is the control plane going down
|
||||
— which is the one failure that makes every other one harder to fix.
|
||||
|
||||
**They are bounded differently.** The substrate is sized, backed up and upgraded as part of
|
||||
bootstrapping a mesh. A workload's database follows the workload — moved with it, destroyed with
|
||||
it, restored with it.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Migrating an object store is ordinary module work**, not substrate work. It was previously going
|
||||
to be done as part of completing the substrate, which would have been the wrong shape and would
|
||||
have coupled every mesh to a service the mesh does not use.
|
||||
|
||||
**A mesh with no workload needing one runs no object store at all.** That is the correct outcome
|
||||
and was not previously available.
|
||||
|
||||
**Two PostgreSQL containers on a node that hosts both is expected**, not duplication to be
|
||||
optimised away. Anyone tidying them together should find this record first.
|
||||
|
||||
**"Substrate by role and ordinary by delivery" loses one of its two members.** ADR 0006 uses that
|
||||
phrase of the object store and the registry — things that are substrate but provisioned once a
|
||||
control plane exists. It now describes the registry alone.
|
||||
|
||||
**The definition is applied, not just stated.** Both halves of the circularity test are asked of
|
||||
each member, and *cannot grant itself one* is not sufficient on its own — it is true of almost any
|
||||
service, which is what made it possible to admit a member on that half alone.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — the definition, and the table this
|
||||
corrects one row of
|
||||
- [ADR 0008](0008-a-context-owns-its-store.md) — a context owns its store exclusively
|
||||
- `mesh-control internal/builder/registry.go` — where artifacts go, and why not S3
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 29. A network is a shape, because an action cannot be undone
|
||||
|
||||
## Context
|
||||
|
||||
**A module of several containers has no way to let them reach each other by name.** A container
|
||||
declaration carries a `network` field, and it only ever *joins* one that already exists — it was
|
||||
added so the control plane could reach the store and the broker on the machine it was raised on.
|
||||
Nothing in the vocabulary **creates** a network.
|
||||
|
||||
Without one, containers on a machine share the runtime's default bridge, which gives addresses and
|
||||
no name resolution between them. So a module that is several containers can only be written by
|
||||
publishing ports onto the machine and pointing its own parts at the host — which puts a module's
|
||||
private wiring on the machine's own address space, where anything else on the machine can reach it
|
||||
and any other module can collide with it.
|
||||
|
||||
**This is the gap a mail system meets and nothing else so far does**
|
||||
([`00-work-breakdown.md`](../03-DESIGN/01-to-be/00-work-breakdown.md) 3.3). It is being taken now
|
||||
rather than then, because 3.3 is the task most likely to send work back into the declaration
|
||||
language and the least useful place to discover it.
|
||||
|
||||
**Adding a shape is not a small change, and the host says so** — the vocabulary is asserted
|
||||
against a stated number, with the reason written into the failure: *every addition widens what a
|
||||
compromised control plane can express, so a change here is a decision.* The host applies what it
|
||||
is told; the only bound on a hostile control plane is what the language can say
|
||||
([ADR 0004](0004-a-node-and-how-it-joins.md)).
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **An `action` that creates the network.** The vocabulary already has one, the bundle already
|
||||
uses seven of them, and `docker network create` with a `verify` is exactly the shape an action
|
||||
takes. Nothing would need adding. **Rejected**, on removal:
|
||||
|
||||
> An action has no footprint the host can undo — it ran, and whatever it did belongs to
|
||||
> whatever it acted on.
|
||||
|
||||
A network made this way **leaks when the module is unassigned**, and the mesh cannot tell: the
|
||||
record says an action ran, and there is nothing to reverse. Unassigning a module would leave a
|
||||
network behind on every machine it was ever on, and the only way to find them would be to go
|
||||
and look. *A resource the mesh can create and never clean up is one it should not create.*
|
||||
|
||||
There is a second reason, and it is the one that generalises: an action is opaque. **The mesh
|
||||
cannot tell what an action did**, so a network created by one is not a thing the mesh knows
|
||||
about — it cannot be reported, counted, or reasoned about, and a module could not require one.
|
||||
|
||||
2. **Publish ports on the machine instead.** No new shape, and it works today. **Rejected.** It
|
||||
makes a module's internal wiring part of the machine's address space: two modules that each
|
||||
want a database on a fixed port collide, and anything else on the machine can reach what was
|
||||
meant to be private. It also makes the module's manifest depend on what else is installed,
|
||||
which is the thing provisioning exists to remove.
|
||||
|
||||
3. **`network` as a ninth shape.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**`network` joins the vocabulary, and the vocabulary is nine shapes.**
|
||||
|
||||
```
|
||||
{"id": "internal", "type": "network", "name": "mail"}
|
||||
```
|
||||
|
||||
**A name and nothing else.** Not a driver, a subnet, an address range or a gateway: every one of
|
||||
those is a thing a module would have to know about the machine it lands on, and a module that
|
||||
names a subnet is a module that collides with whatever else chose the same one. The runtime picks;
|
||||
the mesh names.
|
||||
|
||||
**It is created if absent and removed when no longer declared** — an ordinary shape, with the same
|
||||
lifecycle as a directory. That is the whole reason it is a shape.
|
||||
|
||||
**Declared before the containers that join it.** Resources are applied in the order the module
|
||||
wrote them, and orphans are removed in **reverse** — so a network written first is created first
|
||||
and removed last, after the containers attached to it are gone. This is not a new rule; it is the
|
||||
existing one, and it happens to be exactly right here. A network written *after* its containers
|
||||
would fail to remove while they still hold it, and that failure is reported rather than silent.
|
||||
|
||||
**What it does not do:** it does not reach across machines. A network is one machine's, like
|
||||
everything else the host applies. Modules on different machines reach each other over the private
|
||||
network the mesh already provides ([ADR 0007](0007-connectivity.md)), and a shape that tried to
|
||||
span machines would be a second overlay with a worse contract.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The vocabulary is nine, and the count moves with a record.** The test that asserts it names this
|
||||
one, so the next person to change it finds the argument rather than a number to edit.
|
||||
|
||||
**A compromised control plane can now create and destroy networks on a machine.** Stated plainly
|
||||
because that is the cost, and the bound is the point: it can create a named network and remove
|
||||
one, and it can do neither to anything it did not declare. It cannot inspect, attach to, or
|
||||
reroute what is already there — those would be different shapes, and are not being added.
|
||||
|
||||
**A multi-container module becomes expressible**, which unblocks 3.3 and, less obviously, makes
|
||||
several smaller modules simpler: anything that is a service plus a sidecar currently has to
|
||||
publish a port to talk to itself.
|
||||
|
||||
**Nothing is required to use it.** A module of one container declares no network and joins none,
|
||||
exactly as now. The substrate keeps using `host`, which is a runtime-provided network and not one
|
||||
the mesh creates.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0005](0005-the-node-host.md) — the host's vocabulary, and why each shape is a decision
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — what may be pushed is bounded by form, not by trust
|
||||
- [`03-DESIGN/01-to-be/00-work-breakdown.md`](../03-DESIGN/01-to-be/00-work-breakdown.md) — 1.3,
|
||||
and the mail system at 3.3 that this is for
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 30. Data outlives the mesh that declared it
|
||||
|
||||
## Context
|
||||
|
||||
**The conversion runs on live services holding real data**, and starts on the node that holds all
|
||||
of it. Identity, mail, everything. The requirement stated plainly: a data directory may be
|
||||
*moved*, and may never be *lost*.
|
||||
|
||||
**The host deleted them.** A directory that stopped being declared was an orphan, and an orphan
|
||||
directory was removed with `os.RemoveAll` — everything under it — while the report said
|
||||
`removed`. A module unassigned took its database's files with it, and nothing anywhere said what
|
||||
had been in there.
|
||||
|
||||
Reproduced before it was fixed: assign a module, let a service write into its directory, unassign
|
||||
the module, and the file is gone.
|
||||
|
||||
**A directory stops being declared for ordinary reasons**, which is what makes this sharp rather
|
||||
than theoretical. A module unassigned from a node. A manifest edited to move a data folder — the
|
||||
exact operation the conversion needs. A resource renamed. A typo. **Every one of those is a normal
|
||||
day's work, and every one of them was destructive.**
|
||||
|
||||
**The removal order was already right, and that is what makes a fix possible.** Everything the
|
||||
mesh puts inside a directory is itself a declared resource, and orphans are removed in reverse
|
||||
declaration order — so by the time a directory is reached, what the mesh wrote there is already
|
||||
gone. Anything still present was put there by something else.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A `keep` flag on the directory.** A module declares which of its directories hold data, and
|
||||
the host leaves those. **Rejected.** It is safe only when somebody remembered, and the failure
|
||||
of forgetting is total and silent. A rule that protects data only when it was asked to is not
|
||||
a rule about data, it is a rule about attentiveness — and this is the one place in the system
|
||||
where being wrong does not recover.
|
||||
|
||||
2. **Never remove a directory.** Simple and unarguably safe. **Rejected**, narrowly: every module
|
||||
ever assigned would leave its directories behind for ever, and a machine that accumulates
|
||||
things nobody can account for is one where nobody can tell what is still in use. The clean-up
|
||||
that is genuinely the mesh's is worth keeping.
|
||||
|
||||
3. **Remove a directory only when it is empty.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**A directory that still holds anything is kept, and the mesh says so.** An empty one is removed.
|
||||
|
||||
**This is the host's existing line applied to the one shape where getting it wrong is
|
||||
unrecoverable** — *it removes what it made and leaves what it merely configured*
|
||||
([ADR 0005](0005-the-node-host.md)). An empty directory is what the host made. A full one is not.
|
||||
|
||||
**No flag, no declaration, nothing to remember.** Emptiness is the test, and it is derived from
|
||||
the removal order rather than asserted: the mesh's own contents are gone by then, so what remains
|
||||
is by definition something nobody declared.
|
||||
|
||||
**It is reported, not silent.** The outcome is `kept`, naming how many items are inside and saying
|
||||
they are for a person to deal with. A directory quietly left behind is how a machine accumulates
|
||||
things nobody can account for — which is the objection to option 2, and it is answered by saying
|
||||
so rather than by deleting.
|
||||
|
||||
**Files are unchanged.** A declared file is the mesh's own — it wrote it, it owns it, and losing a
|
||||
configuration file is not the failure this is about. The distinction is deliberate: **directories
|
||||
hold what other things produced; files are what the mesh itself put there.**
|
||||
|
||||
## Consequences
|
||||
|
||||
**Moving a data directory is now safe by default.** The manifest changes, the old path stops being
|
||||
declared, and the data stays where it is until somebody has looked at it. That was the operation
|
||||
most likely to destroy something during the conversion, and it is now the operation that does the
|
||||
least.
|
||||
|
||||
**Unassigning a module leaves its data.** Correct, and it means unassignment is no longer a way to
|
||||
clean up — removing data is a person's act, done knowingly. Given what unassignment did before,
|
||||
that is the trade being made and it is the right way round.
|
||||
|
||||
**A machine can accumulate directories nobody removed.** Accepted, and mitigated by saying so
|
||||
every time rather than by a periodic sweep. A sweep would be the deletion this record exists to
|
||||
prevent, on a timer, with nobody watching.
|
||||
|
||||
**It is not a backup, and must not be mistaken for one.** This stops the mesh destroying data. It
|
||||
does nothing about a disk, a mistaken `rm`, or a service corrupting its own store. The conversion
|
||||
still needs backups taken and **restored** before anything is moved — a backup nobody has restored
|
||||
is a belief, not a copy.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0005](0005-the-node-host.md) — the host removes what it made
|
||||
- [`03-DESIGN/01-to-be/00-work-breakdown.md`](../03-DESIGN/01-to-be/00-work-breakdown.md) — the
|
||||
conversion this was found by planning
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 31. The control plane authenticates nobody, so identity is a module
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0006](0006-the-substrate-and-the-control-plane.md) left one member of the substrate
|
||||
conditional, and said exactly why:
|
||||
|
||||
| role | product | |
|
||||
|---|---|---|
|
||||
| identity provider | — | **conditional**: substrate only if the control plane delegates authentication, which is undecided |
|
||||
|
||||
[`07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md) carried it as an open question —
|
||||
*whether identity is the fifth* — noting it followed from a decision nobody had taken.
|
||||
|
||||
**The decision is taken: the control plane does not delegate authentication.** There is no mesh
|
||||
identity provider.
|
||||
|
||||
**Nothing in the mesh's own machinery ever needed one.** A node proves itself with a keypair it
|
||||
generated, over a broker account issued at enrolment
|
||||
([ADR 0004](0004-a-node-and-how-it-joins.md)). Declarations are verified by signature. None of
|
||||
that touches an identity provider, and the conditional was never about machines — it was only ever
|
||||
about whether a *person* signing in to a mesh surface would be authenticated by something else.
|
||||
|
||||
## Decision
|
||||
|
||||
**Identity is a module**, like the mail system and the forge. It runs *on* the mesh, not *of* it
|
||||
([ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md)) — a provider other modules require,
|
||||
which is the ordinary shape and needs nothing new to express.
|
||||
|
||||
**So the substrate is three, and no longer conditional**: a relational store, a message bus, and
|
||||
an image registry. Together with
|
||||
[ADR 0028](0028-the-substrate-supplies-the-control-plane-and-nothing-else.md), which removed the
|
||||
object store, the list is settled and every member is there for the same reason — the control
|
||||
plane needs it and cannot ask itself for it.
|
||||
|
||||
**A mesh that wants no identity provider runs none.** That is now expressible, and was not while
|
||||
it sat in the substrate as a maybe.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The last open question about substrate membership is closed.** Both halves of ADR 0006's test
|
||||
now have an answer for every candidate, and the answer for identity is *the control plane does not
|
||||
need it*.
|
||||
|
||||
**It does not settle how a person signs in to a mesh surface**, and that is deliberately left
|
||||
open. What is settled is that whatever answers it is not part of what must exist before the mesh
|
||||
does — so it can be decided late, changed, or replaced, which is precisely what being substrate
|
||||
would have prevented.
|
||||
|
||||
**It becomes a real test of the module graph.** An identity provider is a module that *other
|
||||
modules require* — the object store already consumes it — so it exercises the provider chain more
|
||||
seriously than anything ported so far, where the provider was written alongside its consumer.
|
||||
|
||||
**Ordering follows from it rather than from preference.** Anything requiring identity has to move
|
||||
after it, which is a dependency the graph can state rather than something a person has to
|
||||
remember.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — the conditional this closes
|
||||
- [ADR 0028](0028-the-substrate-supplies-the-control-plane-and-nothing-else.md) — the other member
|
||||
removed, and the test applied properly
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — how a node proves itself, which needs none of this
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: superseded
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md
|
||||
superseded-by: 02-DECISIONS/0034-the-local-account-owns-the-mesh.md
|
||||
---
|
||||
|
||||
# 32. The local account owns the mesh; a surface delegates to a module
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0031](0031-the-control-plane-authenticates-nobody.md) settled that the control plane
|
||||
authenticates nobody, and deliberately left one thing open: **how a person signing in to a mesh
|
||||
surface is authenticated.** This answers it, and answers a question 0031 did not ask — *who owns
|
||||
the mesh at all.*
|
||||
|
||||
**There was no answer, and the absence was invisible** because every operation so far has been run
|
||||
by the person sitting at the machine. Nothing had to say whether that was the design or the
|
||||
circumstance.
|
||||
|
||||
## Decision
|
||||
|
||||
**The account that installed the host owns the mesh on that node.** Authority is a local login,
|
||||
and there is nothing else to hold.
|
||||
|
||||
**No mesh user model.** No accounts, no roles, no grants, nothing to administer. A person with a
|
||||
shell on a node can do anything the mesh can do there, because that is already true and pretending
|
||||
otherwise would be a boundary that does not exist.
|
||||
|
||||
**This follows from what was already decided rather than adding to it.**
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md) says there is no authorisation between nodes — every
|
||||
node is the operator's own, so a message from one is a message from them, and *the mesh boundary
|
||||
is therefore the security boundary*. A user model inside that boundary would guard nothing: anyone
|
||||
who could be stopped by it could equally read the node's key off the disk.
|
||||
|
||||
**The board is different, and the difference is the network.** A surface reachable by a browser
|
||||
has to know who is asking, because the people reaching it are not, by construction, people with a
|
||||
shell on the machine. **So the board delegates to an OAuth provider** — which is a module.
|
||||
|
||||
## What this does not change
|
||||
|
||||
**The identity provider is still not substrate** (ADR 0031). A *surface* delegating
|
||||
authentication is not *the control plane* delegating it. The control plane runs, applies
|
||||
declarations and reaches nodes with no identity provider in existence; only the board needs one,
|
||||
and only to decide whose browser it is talking to.
|
||||
|
||||
The test is unchanged and still answers no: *does the control plane need it in order to run?*
|
||||
|
||||
## Consequences
|
||||
|
||||
**The board depends on a module, and says so.** An ordinary edge in the graph, which means the
|
||||
board cannot come up before the provider it authenticates against — stated as a dependency rather
|
||||
than discovered as an outage.
|
||||
|
||||
**Moving the identity provider takes the board with it.** During that module's own conversion the
|
||||
board is unavailable, and that is acceptable: it is a surface, nothing depends on it, and a brief
|
||||
interruption is the trade already accepted everywhere else. Nothing that keeps a service serving
|
||||
goes through it.
|
||||
|
||||
**Anyone with a shell on a node has full authority there.** Written down rather than left implied,
|
||||
because it is the sentence that decides who gets an account on a machine. The protection is the
|
||||
machine's own login, and the overlay that keeps the machine unreachable from outside
|
||||
([ADR 0007](0007-connectivity.md)).
|
||||
|
||||
**A node cannot be operated by somebody without a login on it.** Deliberate, and the cost of
|
||||
having no user model: there is no way to give a person authority over one node without giving them
|
||||
a shell there. If that is ever wanted, it is a new decision and not a gap in this one.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — the control plane authenticates
|
||||
nobody; this answers what it left open
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — no authorisation between nodes, and why the mesh
|
||||
boundary is the security boundary
|
||||
- [`03-DESIGN/01-to-be/11-a-board.md`](../03-DESIGN/01-to-be/11-a-board.md) — the surface this is
|
||||
about
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md
|
||||
---
|
||||
|
||||
# 33. The substrate is a store and a broker
|
||||
|
||||
## Context
|
||||
|
||||
Third correction to one table in one day, all found the same way: by asking whether **both** halves
|
||||
of the substrate test were actually answered for a given member, or only the second.
|
||||
|
||||
The test ([ADR 0006](0006-the-substrate-and-the-control-plane.md)) is *what the control plane needs
|
||||
in order to run, and cannot ask itself for, because it is not running yet.* ADR 0006 admits the
|
||||
image registry on this line:
|
||||
|
||||
| role | product | |
|
||||
|---|---|---|
|
||||
| image registry | **an OCI registry** | it cannot grant itself a repository |
|
||||
|
||||
**That is the second half again.** It is true that a control plane cannot grant itself a
|
||||
repository. Nothing establishes that it needs one *in order to run*.
|
||||
|
||||
**Counted rather than argued.** `substrate-first-node.lock` — the only bundle there is, and what a
|
||||
first node actually becomes — raises twelve resources, and no registry is among them:
|
||||
|
||||
```
|
||||
container runtime · the store · one database per context · the schemas
|
||||
· the broker's certificate · the broker · the control plane
|
||||
```
|
||||
|
||||
The registry arrives afterwards, as an ordinary module the mesh assigns. That is what the lab
|
||||
asserts, in those words: *the mesh runs its own artifact store.*
|
||||
|
||||
**ADR 0006 half-said this already**, calling the registry *substrate by role and ordinary by
|
||||
delivery, provisioned once there is a control plane to do it.* A member that is provisioned by the
|
||||
thing it supposedly precedes is not a member; the phrase was carrying a contradiction rather than
|
||||
resolving one.
|
||||
|
||||
**The registry is a closer call than the object store, and the difference is worth keeping.** The
|
||||
control plane never touches an object store at all — no client, no bucket, ever
|
||||
([ADR 0028](0028-the-substrate-supplies-the-control-plane-and-nothing-else.md)). It genuinely
|
||||
*uses* the registry: the builder pushes to it, hosts pull from it, and nothing reaches a machine
|
||||
without it. **So the registry is a real dependency of the mesh operating, and not of the control
|
||||
plane starting** — and it is the second that the word substrate means.
|
||||
|
||||
## Decision
|
||||
|
||||
**The substrate is two things: a relational store and a message bus.** Both are in the bundle,
|
||||
both must exist before the control plane's first instruction, and neither can be asked for.
|
||||
|
||||
**The registry is an ordinary module.** The mesh cannot deliver anything without one, and it
|
||||
installs one the way it installs everything else. The first node's chicken-and-egg is already
|
||||
solved and needs nothing from this list: it fetches upstream images directly, then runs a registry
|
||||
of the mesh's own.
|
||||
|
||||
**The test is applied to both columns, every time.** *Cannot grant itself one* is true of almost
|
||||
any service and settles nothing on its own. It is what admitted the object store, and then the
|
||||
registry, and both were removed by asking the other question.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The substrate is now exactly what the bundle raises**, which is the strongest form this list can
|
||||
take: it can be checked by counting rather than by reading an argument. A member that is not in
|
||||
the bundle is not substrate, and the two statements cannot drift apart.
|
||||
|
||||
**A mesh that builds nothing still needs a registry** — to receive anything at all — but it needs
|
||||
it as a module, on its own schedule, replaceable. That was already true and was obscured by the
|
||||
list.
|
||||
|
||||
**The word may now be doing too little work.** "Substrate" for *a database and a broker* is a term
|
||||
of art for two things everybody can name. Renaming is not taken here and is worth considering
|
||||
separately; what this record fixes is the membership, not the vocabulary.
|
||||
|
||||
**Three removals from one table in one day is itself the finding.** Each member was admitted on the
|
||||
half of the test that is easy to answer, and the design read plausibly throughout. The rule that
|
||||
comes out of it is not about substrates: **a test with two conditions is a test only when both are
|
||||
asked.**
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — the definition, and the table this
|
||||
corrects a second row of
|
||||
- [ADR 0028](0028-the-substrate-supplies-the-control-plane-and-nothing-else.md) — the object
|
||||
store, removed for the same reason
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity, which was conditional and
|
||||
is now a module
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
supersedes: 02-DECISIONS/0032-the-local-account-owns-the-mesh.md
|
||||
---
|
||||
|
||||
# 34. The local account owns the mesh, and a web application's login is not that
|
||||
|
||||
*Supersedes [ADR 0032](0032-the-local-account-owns-the-mesh.md), which decided the right thing and
|
||||
described it wrongly. The decision below is unchanged; what it said about the board was an
|
||||
invention.*
|
||||
|
||||
## Context
|
||||
|
||||
ADR 0032 answered *who owns the mesh* — the account that installed the host — and then framed the
|
||||
board as **a surface that delegates authentication**, a category it made up for the occasion. It
|
||||
does not need one.
|
||||
|
||||
**The board is a web application.** It has a login, provided by the identity module, in the way
|
||||
every web application has a login. That is a fact about an application, not a property of the
|
||||
mesh, and giving it a name in the mesh's vocabulary implied a relationship that is not there.
|
||||
|
||||
The cost of the invented category was not cosmetic. It made the identity module look like part of
|
||||
the mesh's own authority — something the mesh *depends on* to know who anybody is — when the truth
|
||||
is that the mesh knows nothing about people at all, and one of the applications running on it has
|
||||
a login.
|
||||
|
||||
## Decision
|
||||
|
||||
**The account that installed the host owns the mesh on that node.** Authority is a local login.
|
||||
There is nothing else to hold, no user model, no roles, and nothing to administer.
|
||||
|
||||
**This follows from what was already decided.**
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md) says there is no authorisation between nodes — every
|
||||
node is the operator's own, so a message from one is a message from them, and *the mesh boundary
|
||||
is the security boundary.* A user model inside that boundary would guard nothing: anyone it could
|
||||
stop could read the node's key off the disk.
|
||||
|
||||
**A web application's login is its own business.** The board authenticates its users through the
|
||||
identity module. So might anything else the mesh runs. **None of that is mesh authority**, and the
|
||||
mesh does not learn who anybody is from it.
|
||||
|
||||
## The line this draws, which is the reason to write it down
|
||||
|
||||
**Signing in to an application must not, on its own, become authority over the mesh.**
|
||||
|
||||
Today it cannot: the board reads and does not act
|
||||
([`11-a-board.md`](../03-DESIGN/01-to-be/11-a-board.md) — *not the way to change things*). Looking
|
||||
at a page tells you what is true and changes nothing.
|
||||
|
||||
**The moment the board can assign a module, whoever it lets in has mesh authority** — and it would
|
||||
arrive as a feature rather than as a decision. That is the failure this record exists to make
|
||||
visible, because it is the kind that is only obvious afterwards.
|
||||
|
||||
So: **a surface that can change the mesh is a change to who owns the mesh**, and is taken as one.
|
||||
Not forbidden — wanting to manage nodes from a browser is reasonable — but not something that
|
||||
turns up in a pull request titled *add assign button*.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The identity module is not special.** Not substrate ([ADR 0031](0031-the-control-plane-authenticates-nobody.md)),
|
||||
not part of the mesh's authority, and nothing about the mesh stops working when it is down. Some
|
||||
applications cannot be logged into, which is what it means for an application's login provider to
|
||||
be unavailable.
|
||||
|
||||
**Anyone with a shell on a node has full authority there.** Unchanged from ADR 0032, and still the
|
||||
sentence that decides who gets an account on a machine. The protection is the machine's own login
|
||||
and the overlay that keeps it unreachable from outside ([ADR 0007](0007-connectivity.md)).
|
||||
|
||||
**A node cannot be operated by somebody without a login on it.** The cost of having no user model.
|
||||
If that is ever wanted, the paragraph above says what it costs.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0032](0032-the-local-account-owns-the-mesh.md) — superseded; same decision, invented category
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — the mesh boundary is the security boundary
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — the control plane authenticates
|
||||
nobody
|
||||
@@ -0,0 +1,115 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0034-the-local-account-owns-the-mesh.md
|
||||
---
|
||||
|
||||
# 35. One implementation, several surfaces, and what that costs
|
||||
|
||||
## Context
|
||||
|
||||
The mesh is operated from a command line today. It needs to be operable from a browser and from a
|
||||
model's tools as well, and the three must not be three different systems.
|
||||
|
||||
**The pattern is already in the code and unnamed.** `board` serves HTTP by calling the same
|
||||
functions the CLI calls; it holds nothing and decides nothing. What follows makes that the rule
|
||||
rather than a property of one command.
|
||||
|
||||
**The board is a presentation layer over the control plane.** Not an application beside it holding
|
||||
a database credential — the thing that shows what the control plane knows, and asks it to do what
|
||||
a person asked for.
|
||||
|
||||
## Decision
|
||||
|
||||
**The logic lives once, in the context that owns it. A surface is an adapter with no decisions in
|
||||
it.**
|
||||
|
||||
| surface | for |
|
||||
|---|---|
|
||||
| **command line** | a person on a machine, and the recovery path below |
|
||||
| **HTTP** | the board, and anything else that speaks to the mesh over a network |
|
||||
| **model tools** | an agent asking the mesh to do something |
|
||||
|
||||
**Every surface refuses identically, because the refusal is not in the surface.** An assignment
|
||||
that cannot be satisfied is refused by the same resolution whichever way it arrived. The moment a
|
||||
surface can accept something another would reject, the mesh has two answers to one question and
|
||||
people learn which to trust.
|
||||
|
||||
**Reading and doing are both exposed.** The HTTP surface is not read-only: managing the mesh from
|
||||
a browser is the point. This takes the decision
|
||||
[ADR 0034](0034-the-local-account-owns-the-mesh.md) said had to be taken deliberately —
|
||||
**a browser login now carries authority over the mesh** — and takes it knowingly rather than
|
||||
letting it arrive with a feature.
|
||||
|
||||
**The networked surfaces authenticate through an OAuth2 identity provider.** Named by protocol
|
||||
rather than by product, like every other dependency the mesh takes — AMQP for the bus, S3 for an
|
||||
object store, OCI for the registry
|
||||
([ADR 0006](0006-the-substrate-and-the-control-plane.md)). What fills the role today is a module
|
||||
running Keycloak; what the control plane knows is that it validates a token against a provider
|
||||
speaking OAuth2, and replacing that provider is a migration rather than a redesign.
|
||||
|
||||
**The command line does not authenticate at all**: it is already behind the machine's own login,
|
||||
which is what owns the mesh (ADR 0034).
|
||||
|
||||
## What this is not: a kernel every module imports
|
||||
|
||||
**The shared library is the failure this project was started over**, and the difference has to be
|
||||
stated or it will be rebuilt. The old one is 155 files and 34,636 lines *containing code from
|
||||
every context* — work-domain logic sitting in the kernel every module imports, each piece landing
|
||||
there to avoid a cycle between two modules that both needed it.
|
||||
|
||||
**Shared surfaces are not a shared library.** What is shared here is that three adapters call the
|
||||
same functions. Those functions stay in the context that owns them — provisioning's logic in
|
||||
provisioning, identity's in identity — and no module imports another's. A surface may call many
|
||||
contexts; a context still may not reach into another's store
|
||||
([ADR 0008](0008-a-context-owns-its-store.md)).
|
||||
|
||||
The test, when something is about to be put "somewhere shared": *does this belong to a context, or
|
||||
does it only belong to the surface?* If it belongs to a context it goes there, even if two
|
||||
surfaces want it.
|
||||
|
||||
## The loop this creates, and the way out
|
||||
|
||||
**The control plane's networked surfaces will depend on a module the control plane assigns.**
|
||||
An identity provider is an ordinary module ([ADR 0031](0031-the-control-plane-authenticates-nobody.md)). When
|
||||
it is down, or being migrated, or misconfigured, the HTTP and tool surfaces cannot authenticate
|
||||
anybody — including the person trying to fix it.
|
||||
|
||||
**The command line is the way out, and it is why local ownership matters more rather than less.**
|
||||
It authenticates through nothing, needs no network, and is available on the machine to the account
|
||||
that owns the mesh. **A mesh must always be operable by somebody standing at it.**
|
||||
|
||||
So the rule: **no capability exists only behind an authenticated surface.** Anything the board can
|
||||
do, the command line can do. That is not a courtesy to CLI users; it is the recovery path, and a
|
||||
capability that exists only over HTTP is one that disappears exactly when identity does.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Identity is still not substrate**, and the test still answers no: the control plane runs, applies
|
||||
declarations and reaches nodes with no identity provider in existence
|
||||
([ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md)). What is unavailable without it is two
|
||||
surfaces, not the mesh.
|
||||
|
||||
**Whoever the identity provider admits has authority over the mesh.** That is now a real perimeter
|
||||
with real consequences, where before it guarded a page that only read. Who may log in, and to
|
||||
which realm, becomes a decision about the mesh rather than about an application.
|
||||
|
||||
**A surface must not grow an opinion.** The likely erosion is a validation added to the board
|
||||
because it was quicker there — and then the CLI accepts something the board rejects, or worse the
|
||||
reverse. Adapters hold no decisions.
|
||||
|
||||
**Three surfaces over one implementation is a cost paid three times if it is not one
|
||||
implementation.** The reason to write this down now is that the second surface is the cheapest
|
||||
moment to get it right, and the third is where the drift usually starts.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0034](0034-the-local-account-owns-the-mesh.md) — the local account owns the mesh, and the
|
||||
line this record deliberately crosses
|
||||
- [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) — the shared library this must not
|
||||
become
|
||||
- [ADR 0008](0008-a-context-owns-its-store.md) — a context owns its store, which a surface does
|
||||
not change
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0035-one-implementation-several-surfaces.md
|
||||
---
|
||||
|
||||
# 36. Bootstrap ends at a usable mesh, and the first credential comes from a person
|
||||
|
||||
## Context
|
||||
|
||||
Bootstrap currently ends when the control plane starts
|
||||
([`07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md)). That is a mesh that runs and
|
||||
cannot yet be used by anybody who is not standing at the machine: the networked surfaces need an
|
||||
OAuth2 identity provider ([ADR 0035](0035-one-implementation-several-surfaces.md)), the provider is
|
||||
a module, and no module has been assigned.
|
||||
|
||||
**So bootstrap should go further** — through the identity provider and the first login — and stop
|
||||
at a mesh somebody can actually use.
|
||||
|
||||
**One thing in the way, and it is not incidental.** The mesh has never held a readable secret. The
|
||||
sealing code says what it does and why:
|
||||
|
||||
> Make generates a secret and seals it to both ends, **keeping no readable copy.**
|
||||
|
||||
An initial administrator's credential is the first value a **person must read**. Everything else
|
||||
the mesh generates is something no human ever sees, and everything a human provides is something
|
||||
the mesh immediately stops being able to read.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **The mesh generates it and prints it once**, to the terminal of whoever ran the bootstrap.
|
||||
Convenient, and needs no prompt. **Rejected.** It would give the control plane a plaintext
|
||||
secret for the first time — briefly, and only to one terminal, but the capability would then
|
||||
exist. *An exception made for one case does not stay one*: the next credential that is awkward
|
||||
to supply gets printed too, and the property that a copy of the mesh's database is a copy of
|
||||
nothing stops being checkable by reading the code.
|
||||
|
||||
2. **No password: a one-time link that lets the operator set their own.** The nicest to use.
|
||||
**Rejected for now** — it needs a mechanism that does not exist, and the thing it improves is
|
||||
one prompt, once, on a new mesh.
|
||||
|
||||
3. **The operator supplies it.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**Bootstrap runs to a usable mesh**: the substrate, the control plane, the identity provider as an
|
||||
ordinary module, its realm and client provisioned, an administrator able to log in, and the
|
||||
networked surfaces available.
|
||||
|
||||
**The administrator's credential is supplied by the person doing the bootstrap**, on standard
|
||||
input and not echoed — the path that already exists for a model-access key. The mesh seals it and
|
||||
cannot read it afterwards.
|
||||
|
||||
**What is created is an account in the identity provider, not a user of the mesh.** The mesh still
|
||||
has no user model and gains none here ([ADR 0034](0034-the-local-account-owns-the-mesh.md)). What
|
||||
this produces is the first login for the applications that have one.
|
||||
|
||||
**The provisioning is ordinary.** A realm, a client and a first account are what an identity
|
||||
module's provisioner makes from what the mesh granted it — the same shape as a database and a
|
||||
bucket, which are built and proven.
|
||||
|
||||
**The surfaces arrive when their dependency does.** The command API is not started with the
|
||||
control plane and then broken until identity exists; it becomes available once it can authenticate,
|
||||
the way anything else waits for a provider.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The mesh still never holds a readable secret**, and that sentence needs no exception clause.
|
||||
That is the whole reason for the prompt.
|
||||
|
||||
**An unattended bootstrap is still possible, and the value still comes from outside.** Automation
|
||||
supplying the credential is the operator supplying it. What is refused is the *mesh inventing*
|
||||
one — so an unattended bootstrap with no credential provided produces a mesh with no
|
||||
administrator, which is correct rather than broken.
|
||||
|
||||
**Bootstrap gains an interactive step**, and it is the only one. Worth stating because a bootstrap
|
||||
that cannot run without a person is a real constraint on how a node is stood up, and this is
|
||||
deliberate rather than an oversight.
|
||||
|
||||
**The identity provider is still not substrate.** It is assigned by the control plane, so it comes
|
||||
after it, and a thing that comes after cannot be a thing that must exist before
|
||||
([ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md)). Bootstrap running through it does not
|
||||
move it: bootstrap is a sequence, the substrate is a dependency.
|
||||
|
||||
**And the recovery path is unchanged.** When the identity provider is broken later — which is the
|
||||
failure that matters, not the one at first start — the command line still works, because it
|
||||
authenticates through nothing (ADR 0035).
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0035](0035-one-implementation-several-surfaces.md) — the surfaces, and why the command line
|
||||
must keep working
|
||||
- [ADR 0034](0034-the-local-account-owns-the-mesh.md) — the local account owns the mesh; this adds
|
||||
no user model
|
||||
- [ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md) — what must exist before the control
|
||||
plane, which this does not change
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
topic: building it
|
||||
status: proposed
|
||||
date: 2026-09-01
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
rests-on: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 37. Where a module lives
|
||||
|
||||
## The question
|
||||
|
||||
The mesh's own module descriptions currently sit in `examples/` inside the control plane, beside
|
||||
the small programs that hand out logins. That was fine while there were three of them. It is
|
||||
wrong now, and the name is doing active harm: everything in `examples/` reads as a sketch, and one
|
||||
of them shipped naming a container image that nothing in the repository builds. A directory called
|
||||
*the catalogue* would have made *does this actually work* the obvious question to ask of it.
|
||||
|
||||
So: **one repository holding the modules we ship?** And if so, where does everything that is not
|
||||
ours go?
|
||||
|
||||
## What a module actually is, counted
|
||||
|
||||
The system being replaced has **126 modules** on its main branch. The shape of them is the whole
|
||||
argument, so it is measured rather than assumed:
|
||||
|
||||
| | count | what it is |
|
||||
|---|---|---|
|
||||
| **the module is software** | 47 | its own source tree lives inside the module — a daemon, a service, a library |
|
||||
| **helper scripts only** | 44 | no application of its own; scripts it runs at install time or offers to an agent |
|
||||
| **a description and nothing else** | 35 | a package to install and some files to write |
|
||||
|
||||
**Two thirds of modules contain code.** The largest is a shared library of 182 source files. A
|
||||
speech-capture module carries a complete daemon — audio capture, mixing, transcription, a model
|
||||
runner. Treating a module as *a description of something else* is true of barely a quarter of them.
|
||||
|
||||
That kills the simplest answer. A catalogue cannot be "a folder of manifests" when most modules
|
||||
are programs.
|
||||
|
||||
## The four kinds, which want different homes
|
||||
|
||||
**1. What the mesh is made of.** The control plane, the host, the shared library, the board.
|
||||
These are not modules that happen to be ours; they are the mesh, expressed as modules so it can
|
||||
install itself. They belong in the repositories that build them, which already exist.
|
||||
|
||||
**2. Something the world made, that we describe.** A forge, a mail system, an identity provider,
|
||||
a media server. Nobody upstream ships a description; somebody has to write one, and it is the same
|
||||
description for everybody who runs it. **This is what a catalogue is for.** It is also where the
|
||||
small programs that create accounts belong, because such a program is part of describing that
|
||||
service, not part of the mesh.
|
||||
|
||||
**3. Something we wrote, that runs somewhere.** An application, a site, a side project. The
|
||||
description belongs **with the code, at the root of its own repository**, because the two change in
|
||||
the same commit. A repository that gains an environment variable and a description that gains it
|
||||
elsewhere will drift, and there is no mechanism that could stop it. This is already how it works
|
||||
and it should stay that way.
|
||||
|
||||
**4. A package and some files.** A tool, a font, a shell. Thirty-five of these, and each is a few
|
||||
lines. The catalogue.
|
||||
|
||||
## The proposal
|
||||
|
||||
**A `mesh-catalog` repository** holding kinds 2 and 4: descriptions of software we did not write,
|
||||
and the programs that provision it. Not kind 1, which is the mesh itself. Not kind 3, which lives
|
||||
with its own code.
|
||||
|
||||
**The mesh's list of modules is not this repository.** It is a table in the control plane, filled
|
||||
by adding a description to a running mesh. The catalogue is a *source* to add from — one of
|
||||
several, and the mesh already records which: every module carries where it came from, the branch
|
||||
followed there, and the commit its description was read at. **Nothing needs inventing to support
|
||||
modules from anywhere**; a repository of our own is simply the source we curate.
|
||||
|
||||
**A description is checked by the tool, not by a test that imports the tool.** Today a test in the
|
||||
control plane parses the example manifests by reaching into the control plane's internals, and
|
||||
another reads the control plane's own build file to check every image a module names can be built.
|
||||
Two jobs tangled. A `module check` command on the control plane's binary would let the catalogue
|
||||
hold data validated from outside, and would give the same check to somebody describing their own
|
||||
application in their own repository — which is the case that matters most and currently has no
|
||||
check at all.
|
||||
|
||||
## What this costs, and the argument against
|
||||
|
||||
**It is early.** Ten modules exist, four of them ours. Moving ten files is a morning; moving a
|
||||
hundred is a week — but the hundred is not here yet, and splitting now adds a second repository to
|
||||
release across before there is anything to release.
|
||||
|
||||
The counter is that the tangle is already producing faults rather than merely threatening to. A
|
||||
manifest naming an unbuildable image, and a test reading a build file two directories up, are both
|
||||
symptoms of one repository doing two jobs. And the moment the first module is adopted on a real
|
||||
machine, the descriptions stop being examples and become the thing deployments come from. **That
|
||||
is the moment this becomes urgent, and it is close.**
|
||||
|
||||
## What it does not settle
|
||||
|
||||
**Where a provisioning program's image is published**, and how a description pins it. A description
|
||||
names an image by digest; the image is built from the catalogue; the catalogue must therefore both
|
||||
produce an image and refer to it, which is the same knot the bootstrap has and solves by writing
|
||||
the digest down after building.
|
||||
|
||||
**Whether kind 4 deserves a module at all.** Thirty-five descriptions that say *install this and
|
||||
write these files* may be better as one module with settings than as thirty-five modules. Left
|
||||
open deliberately; it is a question about the shape of the catalogue, not about whether to have one.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: proposed
|
||||
date: 2026-09-01
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
rests-on: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 38. The mesh assigns the port, and a module does not care
|
||||
|
||||
## The problem, as met
|
||||
|
||||
A database module cannot start on a machine that runs the control plane. The mesh keeps its own
|
||||
store there and holds 5432; the module publishes 5432. Nothing notices until a container runtime
|
||||
three layers down says `port is already allocated`
|
||||
([`028`](../04-ISSUES/028-two-things-want-one-port-and-nothing-says-so/00-report.md)).
|
||||
|
||||
A module cannot fix this by choosing better, because **a module cannot know what else is on the
|
||||
machine.** It is written once and assigned anywhere. Any number it picks is a guess about a
|
||||
machine it has never seen, and two modules guessing the same number is not a mistake either of
|
||||
them made.
|
||||
|
||||
## The number is written three times, and nothing makes them agree
|
||||
|
||||
Every module says its port in three places:
|
||||
|
||||
| where | for | example |
|
||||
|---|---|---|
|
||||
| `listens` | the rule set that lets traffic in | `{port: 5432, from: mesh}` |
|
||||
| `serves` | what a consumer must know to connect | `{port: 5432}` |
|
||||
| a container's `ports` | what the runtime publishes | `"5432:5432"` |
|
||||
|
||||
They agree today because one person wrote all three. Nothing checks it. A module whose `serves`
|
||||
said 5432 and whose container published 5433 would resolve, compose, apply, and hand every
|
||||
consumer a port that answers nothing.
|
||||
|
||||
## The decision
|
||||
|
||||
**The mesh assigns the machine-side port, and the module says only what it needs.** A module
|
||||
declares that a container port must be reachable and what it is for. Which number the machine uses
|
||||
is the mesh's to choose, because the mesh is the only thing that knows what else is there.
|
||||
|
||||
**One source, and the other two are derived.** `serves` carries the assigned port so a consumer is
|
||||
told where to connect without the module having written it down; the rule set is computed from the
|
||||
same assignment. Three copies become one fact.
|
||||
|
||||
**An assignment is made once and kept**, exactly as a credential is. A port that moved on every
|
||||
push would restart both ends each time and would hand consumers a number that was true when it was
|
||||
read.
|
||||
|
||||
## Some ports cannot move, and that is a claim
|
||||
|
||||
Mail is 25, submission is 587, IMAP over TLS is 993. A mail system on a strange port is not a mail
|
||||
system. So a module may say a port is **fixed by the protocol** rather than assigned.
|
||||
|
||||
**A fixed port is exactly a claim** — the thing the mesh already has for what is singular on a
|
||||
machine: one seat, one display server, one artifact store. Two modules wanting 25 on one machine is
|
||||
the same shape as two wanting the seat, and gets the same answer: the second is refused, by name,
|
||||
when it is assigned rather than when it is applied.
|
||||
|
||||
That is why this does not need a new mechanism so much as it needs the existing one pointed at
|
||||
ports.
|
||||
|
||||
## What follows
|
||||
|
||||
- **A module becomes portable in a way it was not.** Two databases on one machine stop being a
|
||||
collision and become two assignments.
|
||||
- **The substrate has to be visible.** The mesh cannot assign around its own store while it has
|
||||
never heard of it. What the bundle holds must be written down somewhere the assignment can read
|
||||
— which the bundle does not say today.
|
||||
- **A refusal can be useful.** *25 is held by the mail system on this machine* is a sentence a
|
||||
person can act on. `port is already allocated` is not.
|
||||
- **`serves` stops being written by hand**, which is a small vocabulary change with a large
|
||||
consequence: what a consumer is told is now derived from what actually happened.
|
||||
|
||||
## What this does not settle
|
||||
|
||||
**Whether a module should publish to the machine at all.** Assignment makes publishing safe; it
|
||||
does not make it necessary. Consumers could instead reach a provider on the module's own network by
|
||||
name, with nothing published — which would make the question moot for anything inside the mesh, and
|
||||
would still leave it for anything reached from outside.
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-03
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 39. What the SDK holds, and what it refuses
|
||||
|
||||
_Reconciliation note (2026-09-05): supersedes the earlier "repository structure" decision, which the consolidation folded; no standalone record remains to point at, so body references to it now point at the nearest surviving record, [ADR 0015](0015-applications-live-in-their-own-repository.md)._
|
||||
|
||||
## Context
|
||||
|
||||
The earlier "repository structure" decision (folded in consolidation; see the reconciliation note
|
||||
above, and [ADR 0015](0015-applications-live-in-their-own-repository.md) as the nearest survivor)
|
||||
named `mesh-sdk` "contracts shared across tiers:
|
||||
types, not behaviour." That line is superseded here, because it draws the boundary in the wrong
|
||||
place. The boundary that matters is not *types versus behaviour* — it is **how often the thing
|
||||
changes**.
|
||||
|
||||
The current SDK is the cautionary tale, and its failure is precise. `hal/sdk` holds all the
|
||||
code, including a per-module API client for every service (`clients/plex.ts`, `clients/gitea.ts`,
|
||||
…) and a per-module tool implementation for each (`tools/plex.ts`, …). Every module depends on
|
||||
the SDK, so **every edit to any of that per-module code rebuilds every module** — the cascade.
|
||||
The SDK is under constant maintenance precisely because it became the place all the volatile
|
||||
per-module logic accumulated.
|
||||
|
||||
The root cause is worth stating exactly, because the fix follows from it: the pressure was never
|
||||
to share a client *between* modules. It was to share a client between one module's *own features*
|
||||
— plex's tools, its health check and its hooks all wanted the same `PlexClient` — and the only
|
||||
place to share code across a module's features was the global SDK. So **intra-module sharing
|
||||
leaked out as inter-module coupling.**
|
||||
|
||||
## Decision
|
||||
|
||||
The SDK holds the **stable spine** that modules build against, and earns its place by rarely
|
||||
changing. The test for membership is change-frequency, not kind.
|
||||
|
||||
### What it holds
|
||||
|
||||
- The **tool-serving harness** — the worker and registration mechanism, and the tool-definition
|
||||
type. *How* a tool is declared and served is settled; it does not change when an individual
|
||||
tool does.
|
||||
- The **messaging and event framework** — the broker client, the event consumer, the envelope.
|
||||
- The **contracts** — the manifest, declaration, provision and link shapes.
|
||||
- **Core primitives** — sealing and crypto, semver, the shared resolution helpers.
|
||||
|
||||
These change rarely and deliberately. When one of them does change, a rebuild of everything is
|
||||
the *correct* outcome, because the contract every module shares has genuinely changed.
|
||||
|
||||
### What it must not hold — the more important half
|
||||
|
||||
- **A module's API client.** A Plex client, a Gitea client, a MinIO client belong in their
|
||||
module. They change when that service's API or the module's use of it changes, which is often,
|
||||
and which has nothing to do with any other module.
|
||||
- **A module's tool implementations.** Same reason, same place: in the module.
|
||||
- **Anything volatile** — anything that changes when one service's features change.
|
||||
|
||||
The rule, stated so it can be applied without re-deriving it:
|
||||
|
||||
> If editing a thing recompiles unrelated modules **and** it changes often, it does not belong
|
||||
> in the SDK.
|
||||
|
||||
Both conditions are load-bearing. A rare change that cascades is fine — that is a contract, and
|
||||
the cascade is correct. A frequent change that stays local is fine — that is a module minding its
|
||||
own business. Only **frequent *and* cascading** is the disease, and per-module clients and tools
|
||||
are its carriers.
|
||||
|
||||
### Where per-module shared code lives instead
|
||||
|
||||
Code shared among a module's *own* features lives **in the module**. The default is the plainest
|
||||
thing that works: an ordinary shared file the features import — `plex/client.ts`, imported by
|
||||
`plex/tools/`. Within one module, features are files importing sibling files; no package
|
||||
boundary, no ceremony.
|
||||
|
||||
A **module-local SDK** (a sub-package with its own version) is warranted only for the few modules
|
||||
whose shared surface is large enough to version on its own. It is the exception, not the shape.
|
||||
|
||||
Either form gives the property the global SDK could not: editing a module's shared code rebuilds
|
||||
**that module and nothing else**.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The cascade becomes **structurally impossible for module logic**. There is no longer an edge
|
||||
from one module's internals to another, so the only thing that can rebuild everything is a real
|
||||
change to a shared contract in the SDK — which is rare, and when it happens, is right.
|
||||
- The SDK is small and stable **by construction**, not by discipline. Its size is no longer a
|
||||
thing anyone has to police.
|
||||
- **Converting a module from the current system is partly a de-coupling, not just a move.** Its
|
||||
client and its tools are pulled *out* of the shared SDK and *into* the module. A conversion
|
||||
that copied `clients/plex.ts` into the SDK's replacement would rebuild the exact mistake.
|
||||
- The host still does not import the SDK. It depends on nothing
|
||||
([ADR 0005](0005-the-node-host.md)) and **mirrors** the contracts rather than
|
||||
importing them, exactly as its apply-shapes table already does deliberately. The SDK is shared
|
||||
by the tiers that *can* share code; the host is not one of them.
|
||||
|
||||
## References
|
||||
|
||||
- The earlier "repository structure" decision — named the repositories; its `mesh-sdk`
|
||||
description ("types, not behaviour") is superseded by this record (folded in consolidation;
|
||||
nearest survivor [ADR 0015](0015-applications-live-in-their-own-repository.md)).
|
||||
- [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) — the decomposition this serves: code
|
||||
belongs to the boundary that owns it.
|
||||
- [ADR 0005](0005-the-node-host.md) — why the host mirrors the contracts instead of
|
||||
importing the SDK.
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-03
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 40. What a module is
|
||||
|
||||
_Reconciliation note (2026-09-05): supersedes the earlier "grouped by domain" decision, which the consolidation folded into how-we-build.md; no standalone record remains to point at._
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0009](0009-modules-and-the-graph.md) settled that everything is a module, but never said what a
|
||||
module *is* beyond "a directory the mesh processes." That gap let the catalogue's breadth read as a
|
||||
smell: a module can carry a container, a built image, tools, a provisioner, migrations, health,
|
||||
config, seat claims, requires and provides — so much that the unit seemed ill-defined.
|
||||
The earlier "grouped by domain" decision (folded in consolidation; see the reconciliation note
|
||||
above) tried to organise modules by domain, which is the wrong axis. This record states what a module is, drawn from the cases that
|
||||
stress-tested it: the shell, i3-vs-sway, umami, and "database."
|
||||
|
||||
## Decision
|
||||
|
||||
**A module is one self-contained piece of software the mesh installs and manages** — everything
|
||||
needed to make that one thing real and integrable: what runs, the seats it claims, what it provides
|
||||
to other modules, what it requires from them, and what operates it.
|
||||
|
||||
The **software is the module's identity.** Capabilities, seats and provisioned resources are the
|
||||
**relationships *between* modules**, not what a module is — and that is what binds a module into one
|
||||
thing. umami is bound by *being umami*: its container runs umami, its provisioner creates umami sites,
|
||||
its tools query umami, its `requires` gets umami a database. Every feature serves the one software.
|
||||
|
||||
### The three relationships
|
||||
|
||||
1. **Shared seat** — several modules fulfil a capability and coexist; one may be default. bash, zsh
|
||||
and fish all join `shell`.
|
||||
2. **Exclusive seat** — modules contend for a single slot; one holds it. i3 (needs x11) and sway
|
||||
(needs wayland) contend for `display-session`.
|
||||
3. **Provide / require** — a provider ships the **provisioner** that creates instances of the
|
||||
resource it offers and returns sealed credentials; a consumer requires it and the mesh wires the
|
||||
credential in. Symmetric: umami requires a database *and* provides analytics.
|
||||
|
||||
### Interfaces are mesh-owned; providers adapt to them
|
||||
|
||||
The mesh **defines the interface** for a capability — the provider-neutral contract of what a
|
||||
consumer receives and how it integrates. Both sides conform: a provider's provisioner **adapts** its
|
||||
software's real API to the mesh contract; a consumer depends on the **interface**, never on a
|
||||
provider. Swap one provider for another and the consumer does not change.
|
||||
|
||||
### The naming rule — draw the interface at the consumer's real coupling
|
||||
|
||||
Name a `provides`/`requires` at the **widest boundary across which the consumer genuinely does not
|
||||
care which implementation serves it**:
|
||||
|
||||
- Where the consumer's coupling is thin — an analytics embed snippet and dashboard, opaque to it —
|
||||
the mesh defines a neutral interface (`analytics`) and providers (umami, amumi) adapt. Swappable
|
||||
across vendors.
|
||||
- Where the consumer **speaks a protocol** — a database's wire protocol and query dialect — the
|
||||
interface *is* the protocol: `postgres-database`, `mssql-database`, `mongodb-database`. Swappable
|
||||
only among protocol-compatible implementations, **never across**, because the application cannot
|
||||
cross it either. "database" is not a capability; the protocol is.
|
||||
- **Never false genericity.** A name must not promise a swap the contract cannot deliver
|
||||
([research 005](../01-RESEARCH/005-domain-grouping/analysis.md)).
|
||||
|
||||
This is [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md)'s rule made general — "names the protocol, not
|
||||
the product; a database names the engine because the app targets it" — with the reason stated: the
|
||||
contract sits where the coupling is.
|
||||
|
||||
### What is not a module
|
||||
|
||||
- A **library** (built against, never deployed — [ADR 0039](0039-what-the-sdk-holds-and-refuses.md)).
|
||||
- A **control-plane context** (the mesh itself — [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md)).
|
||||
|
||||
A **swappable machine mechanism** (a firewall — ufw, nftables) *is* a module implementing a
|
||||
capability. The host hardcodes no firewall, supervisor, package manager or runtime; it owns only the
|
||||
generic apply primitives and platform detection, so it runs where none of those exist — an Android
|
||||
phone has no ufw, systemd, pacman or Docker.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Supersedes the earlier "grouped by domain" decision** (folded in consolidation; see the
|
||||
reconciliation note above). Modules are
|
||||
organised by their relationships (seats, provisions), not grouped into domain folders.
|
||||
- **Refines [ADR 0009](0009-modules-and-the-graph.md).** Everything the mesh runs and integrates is
|
||||
a module — but a module is defined by the *software it delivers*, not by being a bucket of features.
|
||||
- The target is a **self-fulfilling mesh**: declared wants bound to swappable modules, provisioners
|
||||
wiring credentials, nothing hardcoded. The control plane's whole job is the binding.
|
||||
- Converting a module from the old system includes pulling its per-module code out of the shared SDK
|
||||
([ADR 0039](0039-what-the-sdk-holds-and-refuses.md)) and shipping its provisioner as an adapter to a
|
||||
mesh interface — a de-coupling, not just a move.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0009](0009-modules-and-the-graph.md) — everything is a module; this says what one is.
|
||||
- [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) — contexts are the mesh, not modules.
|
||||
- The earlier "grouped by domain" decision — superseded (folded in consolidation; see the note above).
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — protocol-not-product, generalised here.
|
||||
- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — per-module code lives in the module.
|
||||
- [research 011](../01-RESEARCH/011-the-module-graph/00-overview.md) — the graph of these relationships.
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-03
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0040-what-a-module-is.md
|
||||
---
|
||||
|
||||
# 41. Events are a relationship, the lighter sibling of provisioning
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0040](0040-what-a-module-is.md) names two relationships between modules — seats and
|
||||
provide/require (provisioning). A third is latent in the mesh and worth making first-class: the
|
||||
broker every node already runs ([ADR 0002](0002-nodes-communicate-over-a-broker.md)) can carry a
|
||||
module's activity as **events**, which any other module reacts to. A logger that writes an audit
|
||||
trail, a module that acts when another module acts, observability — all of it is one mechanism, and
|
||||
today it is ambient rather than declared.
|
||||
|
||||
## Decision
|
||||
|
||||
**A module emits events and consumes events, and both are declared** — parallel to `provides` /
|
||||
`requires`, so the mesh knows the event graph the same way it knows the provisioning graph.
|
||||
|
||||
### Events are provisioning's lighter sibling
|
||||
|
||||
| | provisioning | events |
|
||||
|---|---|---|
|
||||
| shape | **1:1**, a provider creates a resource *for* one consumer | **1:many**, a module emits, any number listen |
|
||||
| credential | yes — sealed, per consumer | none — it is broadcast |
|
||||
| machinery | a provisioner (the reconcile adapter) | nothing but the broker's topic routing |
|
||||
| declared as | `provides` / `requires` | `emits` / `consumes` |
|
||||
|
||||
Because an event is broadcast and credential-free, there is no provisioner and no per-consumer
|
||||
setup — only a subscription. That is why it is the *lighter* relationship, and why most
|
||||
inter-module reaction should be an event, not a provision.
|
||||
|
||||
### An event carries what an audit needs
|
||||
|
||||
Every event carries its **type** (a dotted topic key, so listeners match by prefix), its **source**
|
||||
module, the **node** it came from, and the **time**. A body follows. The metadata is not optional:
|
||||
a reaction may only need the body, but an audit trail needs to know who did what, where and when,
|
||||
and an event that cannot answer that is not auditable.
|
||||
|
||||
### The audit logger is just a consumer of everything
|
||||
|
||||
A logger that records the whole mesh's activity is **not a privileged component** — it is an
|
||||
ordinary module that consumes `#` (every event) and writes them down. It holds no special access;
|
||||
it only listens widely. That it falls out of the model with no new machinery is the check that the
|
||||
model is right.
|
||||
|
||||
### `consumes` is validated like `requires`
|
||||
|
||||
A `consumes` for an event that **nothing** `emits` is a dangling edge, and the mesh refuses it
|
||||
before deploy — the same rule that catches a `requires` for a resource nothing provides
|
||||
([research 011](../01-RESEARCH/011-the-module-graph/00-overview.md)). A listener waiting for an
|
||||
event that can never arrive is a silent failure, and this repository's whole discipline is against
|
||||
silent failure.
|
||||
|
||||
### One runtime serves all three
|
||||
|
||||
The per-node module runtime that serves a module's tools also wires its `consumes` (subscribe,
|
||||
dispatch to the handler) and lets its code `emit`. Tools are *invoked* (request/reply), resources
|
||||
are *provisioned* (1:1, credentialed), events are *emitted and consumed* (1:many, broadcast) —
|
||||
three relationships, one broker, one runtime, all declared on the manifest.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The mesh gains a declared **event graph** alongside the provisioning graph — visible, validated,
|
||||
reasoned over.
|
||||
- **Reaction becomes the default coordination**: a module acts on another's event without either
|
||||
knowing the other, and without a credentialed link. Coupling drops.
|
||||
- An **audit trail** is a module, not a platform feature — and can be swapped, extended or run more
|
||||
than once (a file logger and a queryable one) with no change to anything that emits.
|
||||
- The runtime must dispatch a module's event handlers as well as its tools; that generalisation is
|
||||
small (both arrive by importing the module's entrypoint) but it is real work.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0002](0002-nodes-communicate-over-a-broker.md) — the broker events ride.
|
||||
- [ADR 0040](0040-what-a-module-is.md) — the relationships this extends.
|
||||
- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — `emit`/`on` are stable sdk surface; the
|
||||
broker binding and the runtime are not.
|
||||
- [research 011](../01-RESEARCH/011-the-module-graph/00-overview.md) — the graph these edges join.
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-03
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0041-events-are-a-relationship.md
|
||||
---
|
||||
|
||||
# 42. The shape of an event on the wire
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0041](0041-events-are-a-relationship.md) made events a relationship — `emits`/`consumes`, the
|
||||
graph, the audit logger. It did not say what an event *is* on the broker: the exchanges, the
|
||||
routing keys, the headers, the queues and their configuration. That shape is a contract every
|
||||
emitter and consumer conforms to, exactly as [ADR 0010](0010-delivery.md)
|
||||
is for declarations — and it was being decided ad-hoc in code. This settles it, so the sdk and the
|
||||
runtime implement one contract and a module never reinvents it.
|
||||
|
||||
## Decision
|
||||
|
||||
### Two exchanges, kept apart
|
||||
|
||||
- **`mesh.events`** — a durable topic exchange. Every event rides it: module, mesh and node.
|
||||
- **`mesh.rpc`** — a durable topic exchange. Tool invocations (request/reply) ride it.
|
||||
|
||||
Kept separate because RPC is not an event: a `#` subscription on `mesh.events` is then a complete
|
||||
audit of what happened, with none of the invocation traffic.
|
||||
|
||||
### The routing key is the event type, namespaced by origin
|
||||
|
||||
Dotted and hierarchical — `<origin>.<name>.<event…>` — with three reserved origins:
|
||||
|
||||
- `module.<module>.<event>` — `module.umami.site.created`
|
||||
- `mesh.<context>.<event>` — `mesh.delivery.deployed`, `mesh.provisioning.granted`
|
||||
- `node.<node>.<event>` — `node.anchor.joined`, `node.anchor.unreachable`
|
||||
|
||||
Topic matching gives a consumer `node.*.joined`, `module.umami.#`, or `#`. The origin roots are
|
||||
reserved; everything after is the emitter's own namespace.
|
||||
|
||||
### Metadata in headers, payload in the body
|
||||
|
||||
An event's identity and provenance are AMQP **headers**, so a consumer — or the broker, or an
|
||||
audit tool — reads who/when/what without parsing the body, and the body is only the domain payload.
|
||||
|
||||
**Required headers**
|
||||
|
||||
| header | meaning |
|
||||
|---|---|
|
||||
| `x-event-id` | a unique id — for dedup and audit (delivery is at-least-once, below) |
|
||||
| `x-source` | the emitter: the module, context or node name |
|
||||
| `x-node` | the node it was emitted from |
|
||||
| `x-time` | emit time, RFC-3339 |
|
||||
| `content-type` | `application/json` |
|
||||
|
||||
**Optional headers**
|
||||
|
||||
| header | meaning |
|
||||
|---|---|
|
||||
| `x-causation-id` | the event or command that caused this one — tracing |
|
||||
| `x-schema` | a version of the body's shape, so a body evolves without silent misreads |
|
||||
|
||||
The routing key already carries the type; it is not duplicated as a header. An **unknown `x-`
|
||||
header is ignored, not refused** — unlike a declaration, an event is observed by parties that need
|
||||
not all understand every header, and refusing would couple every consumer to every emitter's
|
||||
additions.
|
||||
|
||||
### Messages are persistent
|
||||
|
||||
Events are published persistent (delivery-mode 2). An audit trail that loses events on a broker
|
||||
restart is not one, and the cost is disk the broker already spends on everything durable.
|
||||
|
||||
### Queues: one per consumer, durable, dead-lettered
|
||||
|
||||
- **A consumer's queue** is `<node>.<module>.events`, durable, bound to that module's consumed
|
||||
patterns. Durable so a restart does not drop what arrived while it was down. **Manual ack** after
|
||||
the handler succeeds — at-least-once.
|
||||
- **Prefetch** bounds in-flight work (default 32) so one slow consumer does not pull the whole
|
||||
backlog into memory.
|
||||
- **A dead-letter exchange** `mesh.events.dead` receives a message rejected past a redelivery limit,
|
||||
so a poison event is set aside for inspection rather than looping forever or vanishing silently.
|
||||
- **The audit logger's queue** `<node>.audit-logger.events`, bound to `#`, is the same shape —
|
||||
durable, persistent, dead-lettered — because completeness is its whole job.
|
||||
- **RPC reply queues** are exclusive, auto-delete and server-named; **RPC serve queues**
|
||||
`serve.<key>` are durable and shared, so several runtimes serving one tool key compete rather than
|
||||
each answer.
|
||||
|
||||
### At-least-once, and consumers are idempotent
|
||||
|
||||
A handler may see an event twice — a redelivery after a crash between doing the work and acking.
|
||||
Consumers must be idempotent, and `x-event-id` is what makes dedup possible. **Exactly-once is not
|
||||
offered**: it is a promise no broker keeps honestly, and saying so is better than pretending.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The event shape is a versioned, enforced contract, not conventions each module reinvents. The
|
||||
sdk's `emit`/`on` and the runtime's AMQP binding implement it; a module never sees an exchange or
|
||||
queue name.
|
||||
- Metadata-in-headers means the body is exactly the domain payload, and a consumer that only wants
|
||||
provenance never parses it.
|
||||
- Adding a header or an origin root widens the contract and is reviewed as one — the discipline
|
||||
[ADR 0010](0010-delivery.md) applies to the
|
||||
declaration vocabulary.
|
||||
- The sdk's first cut carried source/node/time in the *body*; this supersedes that — they move to
|
||||
headers. That is code to align, in `mesh-sdk` (`emit`/`on`) and `mesh-tools` (the binding, queue
|
||||
config, dead-letter).
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0041](0041-events-are-a-relationship.md) — events as a relationship; this is their wire shape.
|
||||
- [ADR 0010](0010-delivery.md) — the precedent: a wire
|
||||
contract, versioned, additions reviewed as security.
|
||||
- [ADR 0002](0002-nodes-communicate-over-a-broker.md) — the broker.
|
||||
- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — `emit`/`on` are stable sdk surface; the
|
||||
binding, queue config and dead-letter are the runtime's, not the sdk's.
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0041-events-are-a-relationship.md
|
||||
---
|
||||
|
||||
# 43. A module's broker account is scoped by what it emits and consumes
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0041](0041-events-are-a-relationship.md) made events a relationship — `emits` and `consumes`
|
||||
on the manifest. [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) gave them a wire shape — the
|
||||
`mesh.events` exchange, the durable per-consumer queue, the reserved routing-key origins. Neither
|
||||
said how a module *reaches* the broker: what account it holds, and what that account is allowed to
|
||||
do.
|
||||
|
||||
As the code stands, there is no answer. The mesh can provision a **node** account (at enrolment)
|
||||
and a **builder** account (scoped to the build queue), and it can *deliver* any module a sealed
|
||||
own-secret at a declared path — but it has no way to provision a broker **account** for a general
|
||||
module. A module that declares `own-secrets: {broker: …}` and nothing more receives thirty-two
|
||||
random bytes, not a credential. So on the broker, `emits` and `consumes` are enforced by nothing: a
|
||||
running module could bind any queue, consume any pattern, and publish under any origin, and the
|
||||
manifest that says otherwise would be describing a boundary no code draws — the exact shape of fault
|
||||
[04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) records, a scope
|
||||
declared in manifests and read by nothing.
|
||||
|
||||
This settles it, so a module's place on the bus is a thing the broker enforces rather than a thing
|
||||
the manifest merely claims.
|
||||
|
||||
## Decision
|
||||
|
||||
### A module gets a broker account when it is assigned, and its permissions are the manifest
|
||||
|
||||
When the mesh assigns a module to a node it provisions a broker account for that module on that node,
|
||||
sealed to the node ([ADR 0004](0004-a-node-and-how-it-joins.md)) and delivered as the
|
||||
module's `own-secrets` broker — `amqps://` with the mesh's fingerprint, the shape
|
||||
[ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) already carries. The account's permissions are
|
||||
derived from the manifest, and are exactly these:
|
||||
|
||||
- **What it consumes.** Read on `mesh.events`, and configure-and-read on its own queue
|
||||
`<node>.<module>.events` bound to the patterns in `consumes`. It cannot bind or read another
|
||||
module's queue. A module that consumes nothing gets no read on the events exchange at all.
|
||||
- **What it emits.** Write to `mesh.events`, restricted to routing keys under its own origin,
|
||||
`module.<name>.*`. It cannot publish as another module, and cannot publish under the reserved
|
||||
`mesh.*` or `node.*` origins — those belong to the mesh and the host (ADR 0042). A module that
|
||||
emits nothing gets no write.
|
||||
- **Nothing else.** The events account reaches `mesh.events` and that module's own queue, and no
|
||||
more. Tool serving and calling over `mesh.rpc` is a separate grant on the same principle — a
|
||||
module serves the tool keys it declares and calls the ones it is bound to — and is scoped the same
|
||||
way rather than folded in here.
|
||||
|
||||
### Consuming everything is a privilege, granted deliberately
|
||||
|
||||
`consumes: ["#"]` — the audit logger — is read across the whole bus: every module's events, the
|
||||
mesh's, every node's. That is not a pattern like any other; it is the power to see everything, and
|
||||
the account is where it becomes visible. The grant that lets one module read the entire bus is one
|
||||
the mesh issues on purpose and can be audited — the answer to *who can read everything* is a row, not
|
||||
a guess — rather than a breadth any manifest acquires by typing a single character. A `#` consume is
|
||||
a reviewed grant, not a default one.
|
||||
|
||||
### The account is how the declaration is enforced
|
||||
|
||||
Because the account can do only what `emits` and `consumes` name, the broker itself refuses a module
|
||||
that tries to consume a queue it did not declare or emit under an origin it does not own. That is what
|
||||
makes an event relationship a rule and not a comment — the discipline that a stated rule says how it
|
||||
is checked. A manifest that over-declares grants more than the module uses, which is visible and
|
||||
reviewable; one that under-declares makes the module fail closed at the broker, which is the safe
|
||||
direction to be wrong in.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The control plane gains a **generic module broker-account**, derived from the manifest. The
|
||||
builder stops being a special case: its access to the build queue becomes an ordinary expression of
|
||||
what it consumes and serves, not a bespoke account method. One rule, and the builder is an instance
|
||||
of it.
|
||||
- The runtime reads its credential from a file (the broker own-secret), `amqps://` verified against
|
||||
the mesh's fingerprint. The `guest` account is for raising the substrate, never for a module — a
|
||||
module documented as holding its own credential and handed the broker's administrative one is worse
|
||||
than one with no credential story at all.
|
||||
- `emits` and `consumes` stop being advisory. They are the module's authority on the bus, so the
|
||||
manifest is now a security boundary and is reviewed as one, the discipline
|
||||
[ADR 0010](0010-delivery.md) applies to the declaration
|
||||
vocabulary.
|
||||
- *Who can read the whole bus* becomes an answerable question, because `#` is a grant and not an
|
||||
accident.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0041](0041-events-are-a-relationship.md) — events are a relationship; this scopes the account
|
||||
by that relationship.
|
||||
- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) — the wire this account secures: the queue,
|
||||
the origins, the `amqps` credential shape.
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — the link is the security boundary; a
|
||||
module's account is sealed to its node the same way a node's is.
|
||||
- [ADR 0010](0010-delivery.md) — a declaration is owned
|
||||
and its additions reviewed; a module's broker permissions are that discipline applied to the bus.
|
||||
- [04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) — a scope declared
|
||||
in manifests and enforced by no code: the fault this decision closes for events.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0027-a-provision-names-what-the-consumer-is-coupled-to.md
|
||||
---
|
||||
|
||||
# 44. A public name is provisioned, not registered by hand
|
||||
|
||||
## Context
|
||||
|
||||
The mesh names and resolves its own machines internally: the overlay generates
|
||||
`<service>.<node>.<suffix>` wildcards, dnsmasq answers them (`wildcard-resolution`), and the mesh
|
||||
issues a certificate for each internal name. A service reachable at a *public* domain —
|
||||
`plex.example.com`, not `plex.anchor.internal` — needs three things that machinery does not give it:
|
||||
|
||||
- a **public DNS record** at a registrar or DNS provider, so the name resolves on the internet;
|
||||
- a **publicly-trusted certificate** for it, because the mesh's own authority is trusted by nobody
|
||||
outside the mesh;
|
||||
- and routing from that name to the module — which the reverse proxy already does: a module
|
||||
`requires` the `route` capability and the proxy provides it, routing by the host it was asked for.
|
||||
|
||||
The routing exists. The public DNS record does not: the mesh has no way to make a name resolve on
|
||||
the public internet, so today that is a step someone does by hand at a DNS provider, outside the
|
||||
mesh, remembered nowhere. A public name is therefore the one part of reaching a service that the
|
||||
declaration graph cannot grant or withdraw — which means it is created once and outlives whatever it
|
||||
was for, the shape of drift this project exists to remove.
|
||||
|
||||
## Decision
|
||||
|
||||
### A public name is a capability, requested like any other
|
||||
|
||||
A module reachable at a public host declares `requires: ["public-dns"]` and contributes the hostname
|
||||
it wants — beside `requires: ["route"]`, which exposes it through the proxy. The name is then
|
||||
provisioned on declaration ([ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md)): created
|
||||
when the module is assigned, removed when it is withdrawn, reconciled like every provision.
|
||||
|
||||
### The interface is neutral; the providers are the registrars
|
||||
|
||||
`public-dns` is drawn at the consumer's coupling: the consumer wants *a public name that resolves to
|
||||
me*, and does not care whether Cloudflare, Route 53 or a registrar's own API puts the record there.
|
||||
So the interface is neutral and the providers are provider-scoped — `cloudflare-dns`,
|
||||
`route53-dns`, `porkbun-dns` — each implementing the one `public-dns` contract, the same way a
|
||||
neutral database coupling is answered by `postgres-database` and `mssql-database`. A module names
|
||||
`public-dns`; it never names a registrar.
|
||||
|
||||
### The record points at the mesh's public ingress, not at the node
|
||||
|
||||
What the name resolves to is the address the reverse proxy answers on, not the consuming machine's.
|
||||
A public service is reachable only *through* the proxy — the proxy holds the `route` grant and routes
|
||||
by host to the module — so the public name must resolve to the proxy. `public-dns` and `route` are
|
||||
the two halves of one public exposure: the name, and what the name reaches.
|
||||
|
||||
### The record is a fact, not a secret
|
||||
|
||||
A DNS record is public by definition, so the grant returns the fully-qualified name and its TTL and
|
||||
nothing sealed. The only secret is the provider's own API credential, which is the provider module's
|
||||
own-secret and never leaves it — the module that wanted the name never sees it.
|
||||
|
||||
### Events
|
||||
|
||||
The provider emits `module.<provider>.record.created` and `module.<provider>.record.removed`
|
||||
([ADR 0041](0041-events-are-a-relationship.md)), so *which names the mesh publishes, and where* is a
|
||||
question answered from the event trail and the grants, not from a folder of records edited at a
|
||||
provider.
|
||||
|
||||
### The public certificate is the proxy's, and is named here only to pair it
|
||||
|
||||
A public name without a publicly-trusted certificate is reachable and not trusted — the same pairing
|
||||
the internal name and the mesh-issued certificate already have. Obtaining that certificate (ACME
|
||||
against the now-resolving public name) is the reverse proxy's to do, and its mechanism is its own
|
||||
decision; it is named here so the pairing is not forgotten, not resolved here.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A public name is created and torn down with the module, so it cannot outlive it, and the mesh can
|
||||
say which public names it publishes without anyone reading a registrar's dashboard.
|
||||
- Adding a registrar is adding a provider that answers `public-dns`; the modules that want names do
|
||||
not change.
|
||||
- Public exposure of a service is a trio of separate, declared, enforced relationships: the firewall
|
||||
opens the proxy's public port ([ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md)),
|
||||
`route` routes the host to the module, and `public-dns` makes the host resolve.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — a capability is provisioned on
|
||||
declaration; a public name is one.
|
||||
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the firewall, the other
|
||||
half of the reachability question this was asked with.
|
||||
- [ADR 0041](0041-events-are-a-relationship.md) — the provider's record events.
|
||||
- [ADR 0040](0040-what-a-module-is.md) — a provider and its interface; the neutral-interface,
|
||||
scoped-provider naming this follows.
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 45. A machine's firewall is the sum of what its modules listen on
|
||||
|
||||
## Context
|
||||
|
||||
The reverse proxy is a *provider*: a module `requires` the `route` capability and a running proxy
|
||||
provides it, routing traffic by name and reaching back to the consumer. A fair question follows —
|
||||
is the firewall the same shape? Should a module *register* a port with a firewall provider the way
|
||||
it requests a route?
|
||||
|
||||
It should not, and the difference is the point. A reverse proxy is a service another component
|
||||
performs; a firewall is a property of the machine — a packet filter the host applies to itself.
|
||||
Modelling it as a provider would invent a credential and a reach-back for something that has neither.
|
||||
|
||||
And the mesh already has the registration: a module declares `listens: [{ port, from }]` — the port
|
||||
it accepts connections on, and from where. That *is* how a service says it wants a port open. What is
|
||||
missing is not a model but enforcement. [04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md)
|
||||
records that a `scope:` key five manifests carry is read by no code: a manifest can appear to
|
||||
restrict a port and restrict nothing — the exact fault
|
||||
[how-we-build.md](../00-META/how-we-build.md) names, *an unenforced rule is indistinguishable from a
|
||||
wrong one*, made worse because the declaration reads as a restriction.
|
||||
|
||||
## Decision
|
||||
|
||||
### The firewall is derived and host-applied, not a provider
|
||||
|
||||
A machine's firewall is the sum of what the modules assigned to it declare they listen on, computed
|
||||
by the host and applied as one of its owned resources ([ADR 0005](0005-the-node-host.md):
|
||||
the host applies, it does not decide; [ADR 0010](0010-delivery.md):
|
||||
the declaration is owned resources). It is not a capability, not a per-consumer grant — opening a
|
||||
port is a declarative fact about a machine, so it is computed and applied, not requested and
|
||||
credentialed.
|
||||
|
||||
### `from` is the whole of public-versus-internal
|
||||
|
||||
The distinction the question is really about lives in `from`:
|
||||
|
||||
- `listens: [{ port: 5432, from: mesh }]` — open to the private overlay only.
|
||||
- `listens: [{ port: 443, from: anywhere }]` — open to the public internet.
|
||||
|
||||
A module registers a port on the firewall by listening on it and saying from where. There is no
|
||||
separate firewall capability, because the firewall is not a thing that reaches back or holds a
|
||||
secret; it is the machine's own filter over the ports its modules named.
|
||||
|
||||
### The host enforces it both ways, and unknown keys are refused
|
||||
|
||||
A port a module listens on is opened to exactly the scope it named; a port nothing declares is
|
||||
closed. And a key the firewall does not read — the `scope:` of issue 003 — is refused at the
|
||||
manifest, not accepted and ignored, so a declaration that reads as a restriction is one. This is the
|
||||
discipline [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md) applied to the
|
||||
broker account, applied here to the packet filter: the declaration is the enforcement, or it is a
|
||||
comment.
|
||||
|
||||
### A public service is exposed through the proxy, not by opening its own port
|
||||
|
||||
Reaching the public internet is normally not `from: anywhere` on the service's own port. The service
|
||||
listens `from: mesh` — only the proxy reaches it — and `requires: route`, so the sole machine with a
|
||||
public opening is the one running the reverse proxy, and the service is exposed by name through it.
|
||||
`from: anywhere` is the deliberate direct-exposure case, for a service that is its own front door.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Issue 003 is closed: the firewall is computed from `listens` and enforced, so a declared scope is
|
||||
real and an undeclared port is shut. Rejecting unknown manifest keys is the general fix, of which
|
||||
the `scope:` key was one instance.
|
||||
- The firewall and the reverse proxy stop being confused for one model: the firewall is the machine's
|
||||
filter (host-derived from `listens.from`); `route` is a name-router (a provider); the public DNS
|
||||
name is a third thing ([ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md)). A
|
||||
public service uses all three.
|
||||
- The modelling question is answered: a module registers a port by declaring `listens`, and reaches
|
||||
the public internet by name through `route` + `public-dns` — never by the firewall being a
|
||||
provider.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0005](0005-the-node-host.md) — the host applies; the firewall is one of
|
||||
the things it applies.
|
||||
- [ADR 0010](0010-delivery.md) — the firewall is a derived
|
||||
owned resource, not a grant.
|
||||
- [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md) — the same discipline:
|
||||
a declaration is enforced, or it is a comment.
|
||||
- [ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md) — the public name, the other
|
||||
half of the reachability question this was asked with.
|
||||
- [04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) — the unenforced
|
||||
`scope:` this closes.
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0027-a-provision-names-what-the-consumer-is-coupled-to.md
|
||||
---
|
||||
|
||||
# 46. A module's configuration is its assignment's, not its manifest's
|
||||
|
||||
## Context
|
||||
|
||||
A module is assigned to a node — `assign <node> <module>`, always to a machine; there is no
|
||||
assignment to the mesh. "Mesh" is a *scope*, not a place: a `provides` or a `claim` scoped `mesh`
|
||||
reaches the whole mesh, but the module still runs on a node. So the two kinds of thing a module can
|
||||
carry are the manifest (what the module *is*) and, separately, what it should do *here* — which
|
||||
differs by deployment and by node.
|
||||
|
||||
The mesh already has the second: **settings**. `settings set <module> [--node <node>]` — with a node
|
||||
it is that machine's, without it the whole mesh's — layered over what the module declares and applied
|
||||
at resolution, changeable without editing the module and without a rebuild. That is the surface a
|
||||
meshboard would edit.
|
||||
|
||||
But settings today reach only a module's **config-file content** (a mergeable file the module owns).
|
||||
Configuration that is not a file has been landing in the manifest instead, statically — a registrar's
|
||||
zone and domain, the address public names point at, and, most sharply, `listens.from`. That last one
|
||||
is the tell: whether a port is open to the private overlay or to the public internet is a
|
||||
*per-node deployment choice* — the same database internal on one machine and public on another — and
|
||||
a value fixed in the manifest is one value for every machine, so it cannot be. Static configuration in
|
||||
the manifest is configuration in the wrong place: it cannot vary per node, and it cannot change
|
||||
without a new module version.
|
||||
|
||||
## Decision
|
||||
|
||||
### The manifest is identity and defaults; the assignment's settings are the configuration
|
||||
|
||||
A module's manifest declares what it is — what it provides, requires and claims, the shape of its
|
||||
resources — and, for anything configurable, a **default**. The values that make a running instance
|
||||
*this* instance are settings, carried by the assignment: per-node, or mesh-wide when no node is named,
|
||||
applied over the defaults at resolution. Change one and the next reconcile carries it; nothing is
|
||||
edited on a machine and nothing is rebuilt.
|
||||
|
||||
### Settings drive the configurable fields the manifest marks, not only file content
|
||||
|
||||
Settings extend beyond a config file's content to the manifest fields a module declares settable —
|
||||
foremost:
|
||||
|
||||
- **`listens.from`**: a module declares its safe default (`from: mesh`), and a per-node setting
|
||||
raises or lowers it. postgres declares `listens: [{ port: 5432, from: mesh }]`; on the machine that
|
||||
should expose it, a setting makes that port `from: anywhere`. Same module, different exposure, and
|
||||
the firewall ([ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md)) is computed
|
||||
from the effective value, so the packet filter follows the setting.
|
||||
- **A provider's own configuration**: a registrar's zone, domain and the ingress its names point at
|
||||
([ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md)) are mesh-wide settings, not
|
||||
manifest constants — one mesh's Cloudflare zone is not another's, and the module description is the
|
||||
same for both.
|
||||
|
||||
### Unset is the default, and an unknown setting is refused
|
||||
|
||||
A field with no setting keeps the manifest's default, so a module runs correctly configured by nobody.
|
||||
A setting that matches no settable field — like a config value that reaches no file today — is named,
|
||||
not silently dropped, so a misspelled setting is found rather than believed (the discipline of
|
||||
`UnusedSettings`, and of [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md):
|
||||
a declaration is enforced or it is a comment).
|
||||
|
||||
## Consequences
|
||||
|
||||
- The postgres case works: one module, `from: mesh` by default, `from: anywhere` where a setting says
|
||||
so — internal on ace, public on novox, changeable live.
|
||||
- Provider modules stop carrying a mesh's specifics: `cloudflare-dns` describes *a Cloudflare
|
||||
registrar*, and *which* zone and ingress is a setting, so the same module serves every mesh.
|
||||
- Configuration becomes a thing a meshboard manages — set per node or mesh-wide, applied on the next
|
||||
reconcile — rather than a manifest edit and a rebuild ([ADR 0011](0011-managed-files-are-generated-never-edited.md):
|
||||
the way you change a managed thing is not by editing it).
|
||||
- What a manifest may not do is grow a value that differs per machine; if it differs per machine it is
|
||||
a setting, and the manifest holds only the default.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — what is provisioned on
|
||||
declaration; its per-instance values are the assignment's.
|
||||
- [ADR 0011](0011-managed-files-are-generated-never-edited.md) — a managed thing is changed through
|
||||
the mesh, not by editing it; settings are that, for configuration.
|
||||
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the firewall follows the
|
||||
effective `listens.from`, so making `from` a setting makes exposure a setting.
|
||||
- [ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md) — the provider whose zone and
|
||||
ingress are settings, not manifest constants.
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md
|
||||
---
|
||||
|
||||
# 47. A module runs its code as its own process, with its own account
|
||||
|
||||
## Context
|
||||
|
||||
A module is one self-contained thing ([ADR 0040](0040-what-a-module-is.md)), and it gets a broker
|
||||
account scoped to what it emits and consumes ([ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md)).
|
||||
The catalogue now gives modules **tools** and **events** — real code, in the module ([ADR 0039](0039-what-the-sdk-holds-and-refuses.md)) —
|
||||
but nothing has said what *runs* that code. The audit-logger showed one shape and was treated as an
|
||||
exception: a container running the tool runtime carrying the module's compiled code, holding the
|
||||
module's own scoped account. Every module with tools or events needs the same, and the tempting
|
||||
alternative does not work.
|
||||
|
||||
**A node-wide runtime that loaded every assigned module's code cannot hold a per-module account.** It
|
||||
would run under one account with the union of every module's permissions — able to emit as any of
|
||||
them and read any of their queues — which is exactly the isolation ADR 0043 exists to draw. So the
|
||||
runtime is per-module, not per-node, and treating the audit-logger as special left the other
|
||||
modules' code with nothing to run it: the conversion produced tools and events that, as it stands,
|
||||
never execute.
|
||||
|
||||
## Decision
|
||||
|
||||
### A module with tools or events runs a process of its own
|
||||
|
||||
A module that has tools or events runs a **runtime process** — a container, the tool runtime carrying
|
||||
that module's compiled code — assigned and started like the module it is, holding the single broker
|
||||
account the mesh scoped to it (ADR 0043). One module, one process, one account.
|
||||
|
||||
### It serves its tools, each on its own key
|
||||
|
||||
A tool is served on its own key (`serve.<tool>`), and a caller invokes a named tool. Only the module
|
||||
that serves it answers, and the module's account is scoped to exactly its tool keys — so one module
|
||||
cannot answer another's calls, the isolation ADR 0043 gives events extended to tools. This supersedes
|
||||
a single `tools.invoke` endpoint that dispatched by name: that shape assumed one runtime for the
|
||||
whole node, and per-module runtimes competing on one key would each be handed calls for tools they do
|
||||
not have.
|
||||
|
||||
### It runs its events in the same process, under the same account
|
||||
|
||||
Emitting under the module's own origin and consuming its own queue ([ADR 0042](0042-the-shape-of-an-event-on-the-wire.md))
|
||||
happen in that same process, with that same account — not a second one to scope and seal. A module's
|
||||
tool code, its event code and, for a provider, its provisioner are the one module's code and run as
|
||||
the one module's process.
|
||||
|
||||
### The runtime image is the tool runtime plus the module's code
|
||||
|
||||
Built from the module's source like any module image — the audit-logger's shape, made the rule, not
|
||||
the exception. The module declares a `container` for it carrying `MESH_BROKER_FILE` (its sealed
|
||||
credential, ADR 0043) and its compiled code. A module with **neither** tools nor events runs no such
|
||||
process: a plain service module — the plex *server*, dnsmasq the resolver — is its service and files
|
||||
and nothing more. A module that is both a service and code declares both containers: the service, and
|
||||
the runtime beside it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The catalogue's tools and events become runnable: each tools-or-events module gains a runtime
|
||||
container with its scoped credential, and the audit-logger stops being special. Until this, the
|
||||
converted modules held code with nothing to execute it.
|
||||
- A process, and a small image, per tools-or-events module. That is the cost of ADR 0043's isolation:
|
||||
one account per module means one process per module. It is paid deliberately — a shared runtime is
|
||||
cheaper and cannot be scoped, and a mesh where any module can emit as any other is not one worth the
|
||||
saving.
|
||||
- `serve.<tool>` per key replaces the single `tools.invoke` dispatch. The sdk's serving and a module's
|
||||
account scope both come to name tools individually.
|
||||
- **A provider's provisioner is a runtime process too.** It already runs as its own container; its
|
||||
events (`bucket.created`, `database.provisioned`) belong to *that* process and need the same
|
||||
credential. So a provisioner that emits carries `MESH_BROKER_FILE` and its scoped account like any
|
||||
runtime — or it does not emit. (This is the fix for provisioners that emit today with no broker
|
||||
bound: the emit is a runtime's, and the provisioner is a runtime.)
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0040](0040-what-a-module-is.md) — a module is one self-contained thing; its code runs as one
|
||||
process.
|
||||
- [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md) — the scoped account
|
||||
this process holds, and the isolation that makes it per-module.
|
||||
- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) — the events this process runs, and the
|
||||
`serve.<key>` queue tools now use.
|
||||
- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — the code lives in the module; this runs it.
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 48. A provider creates the credential the mesh minted, and seals nothing
|
||||
|
||||
## Context
|
||||
|
||||
A provider module stands up a per-consumer resource — a database, a cache bucket, an object
|
||||
store user — and the consumer must end up holding a credential that authenticates against it.
|
||||
Building the module runtime (ADR 0047: a module runs its own code as its own process under its
|
||||
own account), the provider's provisioner was run for the first time as a delivered thing, and
|
||||
it did not work. It reads a seal key from the environment that nothing sets, and it seals every
|
||||
credential it produces to that key with a symmetric passphrase.
|
||||
|
||||
Tracing the credential's path turned up something larger than a missing key. **The provisioner
|
||||
harness the whole catalogue is built on describes a credential flow the mesh does not have, and
|
||||
duplicates — incorrectly — one it does.**
|
||||
|
||||
What the sdk's `runProvisioner` does today:
|
||||
|
||||
- reads request files named `*.grant.json` — which nothing in the mesh writes;
|
||||
- calls an adapter whose `create` **generates its own password** and returns it;
|
||||
- seals that password with a symmetric key (`$MESH_SEAL_KEY`) and writes a `*.credential`
|
||||
file — which nothing in the mesh reads, and no consumer ever unseals.
|
||||
|
||||
What the mesh already does, and has wired end to end:
|
||||
|
||||
- The control plane mints one password per (consumer, provider) pair (`Inventory.SecretFor` →
|
||||
`secrets.Make`) and seals it to **both** node keys asymmetrically — a copy the consumer's
|
||||
host can open and a copy the provider's host can open. No shared symmetric key exists
|
||||
anywhere, on purpose: a key both ends hold is a key the mesh would have to distribute, which
|
||||
is the same problem one level down, and the control plane's own code refuses it.
|
||||
- The provider is handed, at the path its `receives` names, one contribution per consumer:
|
||||
the **login to create** (`As`, derived by the mesh so the two ends agree by construction),
|
||||
the consumer's address and requested values, and a **`Secret` file** holding that consumer's
|
||||
password sealed to the provider and unsealed onto the machine by its host.
|
||||
- The consumer is handed the *same* password, as plaintext its own host wrote by unsealing its
|
||||
copy and substituting it into a config file. The consumer never unseals anything itself and
|
||||
holds no key.
|
||||
|
||||
So the password a provider's provisioner invents is not even the password the consumer was
|
||||
given: a consumer authenticating with the mesh's password against a resource the provisioner
|
||||
created with its own would simply fail. The symmetric seal is not an incomplete feature to
|
||||
finish delivering a key for. It is a second, contradictory credential model bolted beside the
|
||||
real one, and it cannot be made to work without building the very thing the mesh was designed
|
||||
not to have.
|
||||
|
||||
This is a decision and not a patch because the harness is the **provider contract**. Every
|
||||
provider — the four that exist and the many a real mesh grows — is built on
|
||||
`runProvisioner(resource, adapter)`. Whatever it says a provider is, they all inherit; and
|
||||
changing it later is one migration per provider. It is cheaper and more honest to settle what
|
||||
a provider is now.
|
||||
|
||||
## Decision
|
||||
|
||||
**A provider is handed the credential; it does not make one, does not seal one, and does not
|
||||
hand one back.** The provisioner's only job is to make the mesh's grants true in its own
|
||||
software.
|
||||
|
||||
Concretely, for the sdk harness and the adapter contract:
|
||||
|
||||
- The harness reconciles the **contributions the mesh delivers** to the provider's `receives`
|
||||
path — the list of consumers, each with its login name (`As`), address, requested values,
|
||||
and the path to its unsealed password (`Secret`). It does not read `*.grant.json` and it
|
||||
does not write `*.credential`.
|
||||
- For each consumer present, the harness reads the password from that consumer's `Secret` file
|
||||
and calls the adapter to bring the resource into being under the given login. For each
|
||||
consumer no longer present — the mesh drops it from the contributions file when its consumer
|
||||
goes away — the harness calls the adapter to withdraw it.
|
||||
- The adapter shrinks to the per-software half and nothing else. It is given the login, the
|
||||
password, and the values, and it makes the resource exist or removes it. It generates no
|
||||
password, derives no name, seals nothing, and returns no credential:
|
||||
roughly `create({ as, password, values })` and `remove({ as })`, both returning nothing.
|
||||
- `$MESH_SEAL_KEY`, the symmetric `seal()`/`writeSealedCredential` path, and the `*.grant.json`
|
||||
/ `*.credential` files are removed from the provisioning path entirely. The credential
|
||||
reaches the consumer through the mesh's own asymmetric channel, which already crosses node
|
||||
boundaries and holds no shared secret.
|
||||
|
||||
Identity stays the mesh's to say. The login the provider creates is the name the mesh derived
|
||||
and gave the consumer to present; the provider never invents a name, because a name the
|
||||
consumer cannot learn is a name it cannot authenticate with.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A provider module becomes smaller and unable to be wrong in this way: with no password to
|
||||
generate and no key to seal to, the class of bug where the two ends hold different secrets
|
||||
cannot be written. A provider added after this inherits the corrected contract and has no
|
||||
seal to reintroduce.
|
||||
- The four current providers (redis, postgres, minio, umami) each lose their `generatePassword`
|
||||
+ seal code and gain a `create` that takes the password it is given. Their teardown becomes
|
||||
"withdraw the login named `As`".
|
||||
- The symmetric `seal()`/`unseal()` primitive loses its only caller and leaves — checked, not
|
||||
assumed: nothing else in the sdk or the catalogue called it, so it is removed with the
|
||||
provisioner it belonged to.
|
||||
- **How this is verified:** redis is assigned as a provider in the lab, the contributions and
|
||||
the unsealed password the mesh would deliver are put in its `receives` path, and a client
|
||||
authenticates as that consumer with the mesh's password and gets PONG — where a provider that
|
||||
invented its own password answers WRONGPASS — with `$MESH_SEAL_KEY` set nowhere and no
|
||||
`.credential` file written. Proven: `provider-uses-mesh-credential` is green.
|
||||
|
||||
**What this does not cover — credential provisions, not data provisions.** This decision is about a
|
||||
provision whose credential is a *secret the mesh mints* — a login and password (redis, postgres,
|
||||
minio). A provider that instead *generates* the thing the consumer needs, and that thing is not a
|
||||
secret — umami's `analytics`, where the consumer wants back a `siteId` umami assigned — does not fit,
|
||||
because a contract that returns nothing has no way to hand that data back. The seal-key fault was
|
||||
never umami's (it sealed no password; it returned a public id), so removing the seal does not break
|
||||
it further, and it still reconciles its sites off the mesh's contributions. But delivering
|
||||
provider-generated data back to a consumer is a *return path* the mesh does not have and this
|
||||
decision does not build — a separate shape, left to a separate decision.
|
||||
- Teardown beyond "remove the login" — data an object store leaves behind when a consumer
|
||||
leaves — is named by each provider's adapter, not by the harness, and is out of scope here
|
||||
except to say the contract must leave room for it.
|
||||
|
||||
## References
|
||||
|
||||
- [04-ISSUES/032](../04-ISSUES/032-provider-runtime-has-no-seal-key/00-report.md) — the
|
||||
observation and the cross-repo trace this decision rests on.
|
||||
- ADR 0047 (the module runtime) — what first ran a provider's provisioner as a delivered
|
||||
process and exposed this; link to be filled when 0047 lands on the trunk.
|
||||
- Control-plane mechanisms this relies on already existing: `mesh-control` —
|
||||
`internal/inventory/secrets.go` (`SecretFor`, `SecretsFrom`), `internal/secrets/seal.go`
|
||||
(`Make`, the two-blob asymmetric sealing), `cmd/mesh-control/plan.go` (`grantsFor`, the
|
||||
`Grant.Sealed = ForProvider` delivery), `internal/catalogue/declaration.go` (the `receives`
|
||||
contribution: `As`, `At`, `Values`, `Secret`).
|
||||
- The path being removed: `mesh-sdk` — `src/provisioner/index.ts` (`runProvisioner`, `sealKey`,
|
||||
`writeSealedCredential`) and the symmetric `src/primitives/index.ts` `seal()`/`unseal()`.
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 49. A consumer's identity is bounded by the tightest backend that must accept it
|
||||
|
||||
## Context
|
||||
|
||||
The mesh says who a consumer is, once, and hands the same name to the provider (to create) and the
|
||||
consumer (to present), so the two ends agree by construction rather than by two conventions (the
|
||||
principle behind `ConsumerIdentity`, 04-ISSUES/023). The name is `mesh_<node>_<module>`, cleaned to
|
||||
lower-case letters, digits and underscore.
|
||||
|
||||
Proving the provider contract per backend (ADR 0048) turned up 04-ISSUES/034: redis and postgres
|
||||
create that name verbatim, but **minio refuses it** — an S3 access key is capped at 20 characters,
|
||||
and `mesh_anchor_bucketuser` is 22. The provisioner then retries for ever, per consumer, and the
|
||||
consumer holding that same too-long name could never present it either.
|
||||
|
||||
Two things about the existing derivation decide most of this:
|
||||
|
||||
- **The charset is already right.** `[^a-z0-9_]` is deliberately conservative, and its own comment
|
||||
says it reaches "a PostgreSQL role, a MinIO access key, an LDAP uid and a Keycloak client without
|
||||
quoting." That much is true.
|
||||
- **The length is wrong.** `CheckIdentity` refuses names over `identityLimit = 63`, commented as
|
||||
"the shortest identifier limit among the systems these names reach: PostgreSQL's". It is not the
|
||||
shortest — S3's 20 is shorter — so the guard that was meant to catch exactly this lets it through,
|
||||
and the failure lands at provision time as a silent retry instead of at assignment as a refusal.
|
||||
|
||||
So this is a small wrong constant with a real cost attached: whatever bound we set, `mesh_` (5) plus
|
||||
a node name plus `_` plus a module name has to fit inside it.
|
||||
|
||||
## The options
|
||||
|
||||
**A — Bound the identity by the true minimum, and refuse early.** Lower `identityLimit` to the real
|
||||
shortest (20, S3's), so `CheckIdentity` refuses an over-long name *at assignment* with a clear
|
||||
message, the way it already refuses over-63 names. The derivation does not change; long names are
|
||||
simply rejected before anything is provisioned.
|
||||
- *For:* smallest change; keeps "the mesh says the identity once, verbatim" intact; the failure
|
||||
moves from a per-consumer provision-time retry to an up-front, legible refusal — which is what
|
||||
`CheckIdentity` exists to do.
|
||||
- *Against:* a hard budget. `mesh_` + node + `_` + module ≤ 20 means node + module ≤ 14 characters.
|
||||
`anchor` + `bucketuser` (16) is already over. It pushes the constraint onto how machines and
|
||||
modules are named, which is a real limitation on legible names.
|
||||
|
||||
**B — Keep the readable name when it fits, compact it when it does not.** Below the bound, the name
|
||||
is `mesh_<node>_<module>` as today; over it, the mesh substitutes a deterministic short form (e.g.
|
||||
`mesh_` + a truncated hash of node+module) — still one derivation, so both ends still agree.
|
||||
- *For:* no naming constraint; short backends always satisfied; the common case stays legible.
|
||||
- *Against:* some identities become opaque, and a provisioner tracing "whose login is this" loses
|
||||
the answer for exactly the consumers that overflowed. The mesh now owns a fallback format and its
|
||||
collision properties (a truncated hash is not free of collisions at 15 characters).
|
||||
|
||||
**C — Let each interface declare its identifier bounds, and derive within the tightest a consumer
|
||||
reaches.** `s3-bucket` states `identifier: { max: 20 }`; `postgres-database` states 63; the mesh
|
||||
derives a name that fits the **minimum** bound across the providers a given consumer is granted.
|
||||
- *For:* the most precise — each provision gets exactly the room it has, and a database consumer
|
||||
keeps long legible names while an S3 consumer gets a short one; the constraint lives where the
|
||||
fact does (on the interface).
|
||||
- *Against:* the most work, and a consumer of two interfaces with different bounds must satisfy the
|
||||
smaller — so its name shortens for both, reintroducing B's opacity in a narrower case. It also
|
||||
means one consumer can hold **different** identities per provision, which the "said once" model
|
||||
currently forbids.
|
||||
|
||||
**D — Let the provider generate a backend-valid identity and hand it back (rejected).** minio mints
|
||||
its own access key and returns it to the consumer. This is the data-provision return path this era
|
||||
keeps meeting — but it directly contradicts 023 and ADR 0048: the identity would no longer be the
|
||||
mesh's single derivation the two ends share, it would be a value one side invents and the other must
|
||||
be told. Listed for completeness; not recommended.
|
||||
|
||||
**E — A module (and a node) may declare a short slug; the identity is built from it.** The identity
|
||||
becomes `mesh_<node-slug|node-name>_<module-slug|module-name>`: where a slug is declared it is used,
|
||||
otherwise the cleaned name. A slug is a deliberately short, operator-chosen identifier — `kc` for
|
||||
keycloak, `wkstn` for a workstation. It is optional: short names (`anchor`, `redis`) need none.
|
||||
- *For:* this is the escape hatch B wanted to be, without the opacity. The name stays legible — a
|
||||
provisioner can read `mesh_wkstn_kc` and know who is asking — because a person chose it, not a
|
||||
hash function. And it makes an early refusal *palatable*: if even the slug-built identity overflows,
|
||||
the refusal points at the slug, a field made for exactly this, rather than at the machine's name.
|
||||
Both ends still derive it from one declared thing, so they agree by construction.
|
||||
- *Against:* a new optional manifest field, and someone must pick the slug — but only for names that
|
||||
would otherwise overflow, and picking a short legible identifier is a better job than being handed
|
||||
a hash.
|
||||
|
||||
## What implementing A revealed
|
||||
|
||||
A was tried first. At `identityLimit = 20`, the readable budget is `mesh_` (5) + node + `_` + module
|
||||
≤ 20, i.e. **node + module ≤ 14 characters** — far tighter than it looked. The catalogue's own
|
||||
existing tests use `workstation`+`keycloak` (25), which compacts to `mesh_dbbc02f8dde34d3`; common
|
||||
mesh names (`home-server`, `the-build-node`, `workstation`) blow the budget with any module. So B's
|
||||
compact fallback would fire for the *common* case, not the rare overflow — which inverts A+B: most
|
||||
identities would be opaque hashes. A alone (hard refusal at 20) would refuse most realistic names.
|
||||
This is what moved the recommendation to E: the problem is not the limit, it is that the *readable
|
||||
name* is the wrong source when it is long, and a slug is a better source than either a hash or a ban.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**E, over a per-consumer bound (start with the global minimum, 20).** Build the identity from an
|
||||
optional slug, keep it when it fits, and refuse at assignment with "declare or shorten `<module>`'s
|
||||
slug" when it does not — no hash, no lost legibility, and the fix is a first-class field. Set the
|
||||
bound to the true minimum (20) now; it needs no per-interface machinery to unblock S3, and a module
|
||||
that consumes S3 simply declares a short slug. Graduate to **C** (per-interface bounds) later if it
|
||||
turns out that non-S3 consumers are paying for S3's limit often enough to mind — E and C compose:
|
||||
slugs are the mechanism, per-interface bounds refine where the ceiling sits. **B is dropped**: a
|
||||
declared slug is a strictly better escape hatch than an opaque hash. **D stays rejected.**
|
||||
|
||||
## Consequences (of E)
|
||||
|
||||
- A module manifest gains an optional `slug`; a node may carry one too. `ConsumerIdentity` prefers
|
||||
the slug over the cleaned name for each half. `identityLimit` becomes 20 (the true minimum), and
|
||||
`CheckIdentity` refuses at `module add` / assignment — now with a message naming the slug to set.
|
||||
- The common case stays legible; only names that overflow the budget need a slug, and what they get
|
||||
is a name a person chose, not a hash.
|
||||
- Existing modules/nodes whose names overflow declare a slug once — a migration cost paid as a clear
|
||||
refusal with an obvious remedy, not a silent hash or a silent truncation.
|
||||
- minio (04-ISSUES/034) is unblocked: an S3 consumer declares a short slug and its access key fits.
|
||||
- **How it is checked:** the minio grant e2e — a consumer whose (slugged) identity fits reaches its
|
||||
bucket with the credential the mesh delivered — plus unit tests that a slug is preferred, that an
|
||||
un-sluggable over-long identity is refused (naming the slug), and that two consumers never collide.
|
||||
|
||||
## References
|
||||
|
||||
- [04-ISSUES/034](../04-ISSUES/034-mesh-login-exceeds-s3-access-key-limit/00-report.md) — the
|
||||
observation.
|
||||
- ADR 0048 — a provider creates the credential the mesh minted; the identity it creates it under is
|
||||
the one this decision bounds.
|
||||
- `mesh-control` `internal/catalogue/identity.go` — `ConsumerIdentity`, `identityUnusable`,
|
||||
`identityLimit`, `CheckIdentity` — where the constant and the check live.
|
||||
@@ -0,0 +1,206 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0024-model-access-is-a-provision.md
|
||||
---
|
||||
|
||||
# 50. Model access is vendor-agnostic, and a vendor is an adapter
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0024](0024-model-access-is-a-provision.md) settled that model access is a provision and that
|
||||
a licence is a named thing an operator uses. What shipped, and runs, is a single vendor: the mesh's
|
||||
"claude" feature. A read-only trace of that feature (2026-09-05, in the code workspace) was made to
|
||||
answer whether the model-access provision is Anthropic-shaped or genuinely general. The finding is
|
||||
that **the vendor-agnostic layer already largely exists**, and the Anthropic specifics are a thin
|
||||
band around it that a per-vendor adapter can hold.
|
||||
|
||||
**What is already general, with evidence.** `mesh-control internal/licences` models
|
||||
`licence(name, vendor, serves)` and `licence_holder(licence, node, module, sealed)`, and each
|
||||
holder's credential is sealed per-holder through `internal/secrets`. `serves` carries the
|
||||
non-secret facts (a base URL, a model) and is not vendor-specific. The `accept` verb
|
||||
([ADR 0024](0024-model-access-is-a-provision.md), and
|
||||
[`14-model-access`](../03-DESIGN/01-to-be/14-model-access.md)) already takes an operator-supplied
|
||||
value, seals it to each holder and discards the plaintext. A model the mesh runs itself answers
|
||||
`model-access` at node scope with no licence at all. None of that mentions Anthropic.
|
||||
|
||||
**What is Anthropic-specific.** The credential is not a static key: it is a subscription OAuth grant
|
||||
— an hourly access token plus a refresh token. That shape drags four things behind it that a static
|
||||
key does not need: **central rotation** (one manager node refreshes under a lease and publishes the
|
||||
new token), **delivery that strips the refresh token** so a consuming node holds only an access token,
|
||||
an **identity guard** that reads the credential to catch a mis-binding, and a **usage** reading with
|
||||
Anthropic's own `utilization%` semantics. Most vendors are a single static key, which the sealed-key
|
||||
model already handles and which needs none of these four.
|
||||
|
||||
**The tension at the centre of this.** A refreshable credential cannot be both *sealed so the mesh
|
||||
cannot read it* and *rotated centrally*. Central rotation means some node in the mesh holds the
|
||||
refresh token in readable form, because that is what refreshing requires. Per-holder sealing means no
|
||||
node but the holder can read the credential. For a static key the two never meet — there is nothing to
|
||||
rotate. For a refreshable grant they collide directly, and this record exists to say which gives way,
|
||||
and by how much.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep Anthropic special-cased in the core.** Leave the three binding columns and the `claude_*`
|
||||
schema, and add other vendors beside them the same way. **Rejected.** It is exactly what
|
||||
[ADR 0024](0024-model-access-is-a-provision.md) ruled against: a module that names a vendor cannot
|
||||
be moved onto another model without editing it, and moving it is the point. It also grows the core
|
||||
by one band per vendor, when the bands are the same shape.
|
||||
|
||||
2. **One provision, and refuse to hold any refresh token — re-seal only.** Make every credential
|
||||
purely sealed per-holder, including refreshable ones; let each holder refresh its own grant.
|
||||
**Rejected.** It throws away the hard half [ADR 0024](0024-model-access-is-a-provision.md) says
|
||||
already works — the lease, the single-refresher, the switch-on-exhaustion — and replaces it with N
|
||||
nodes each holding a refresh token, which is the very thing today's delivery strips on the stated
|
||||
ground that *a node never holds a refresh token*. A refresh token is the long-lived secret; spraying
|
||||
it across every holder is strictly worse than keeping one copy on one node.
|
||||
|
||||
3. **One provision, and abandon central rotation entirely** for refreshable vendors — treat the grant
|
||||
as opaque and let it expire. **Rejected.** For a subscription-seat vendor an expired access token is
|
||||
a dead licence; without rotation the feature that works today stops working. This is option 2's cost
|
||||
without option 2's autonomy.
|
||||
|
||||
4. **One vendor-blind provision, plus a per-vendor adapter, with a bounded carve-out for the
|
||||
refreshable case.** **Adopted**, below.
|
||||
|
||||
## Decision
|
||||
|
||||
**`model-access` is one consumer-facing, vendor-blind provision.** A consumer declares
|
||||
`requires: model-access`, and is coupled to *reaching a model* — a base URL, a model name, a key —
|
||||
and not to which vendor answers. That is the coupling the name is drawn at
|
||||
([ADR 0040](0040-what-a-module-is.md)'s rule: name the interface at the widest boundary across which
|
||||
the consumer does not care which implementation serves it). Where a consumer were genuinely coupled to
|
||||
a specific wire API it could not swap across, the same rule would split the name — but the consumers
|
||||
that exist reach their model through a CLI or SDK that hides the vendor, so `model-access` is the true
|
||||
coupling and stays one name. This extends [ADR 0024](0024-model-access-is-a-provision.md) and
|
||||
[ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) without changing them.
|
||||
|
||||
**A vendor is an adapter, keyed by the licence's `vendor` field.** The lifecycle a licence needs is
|
||||
vendor-specific and lives in a per-vendor adapter selected by `licence.vendor`, exactly as
|
||||
`public-dns` is one neutral interface answered by registrar-scoped providers —
|
||||
`cloudflare-dns`, `route53-dns` ([ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md)).
|
||||
A consumer names `model-access` and never a vendor, the same way a module names `public-dns` and never
|
||||
a registrar.
|
||||
|
||||
**The field is named `vendor`, not `provider`.** The inventory already uses "provider" for the
|
||||
provider-pin — *which node answers a brokered provision*. Reusing it for *which company sells this
|
||||
licence* would collide two unrelated facts on one word. `vendor` is the licence's, and is separate.
|
||||
|
||||
### The adapter's capabilities, all but one optional
|
||||
|
||||
An adapter declares:
|
||||
|
||||
- **`shape`** — `static-key` or `refreshable-grant`. This is the switch the carve-out below turns on.
|
||||
- **`accept(value) → sealed`** — take an operator-supplied credential and seal it to the holders, the
|
||||
`accept` verb [ADR 0024](0024-model-access-is-a-provision.md) already defines.
|
||||
- **`refresh(licence)`** — refreshable-grant only: the lease / rotate / publish machinery.
|
||||
- **`identity(credential) → account-id`** — the mis-binding guard, for a vendor whose credential
|
||||
carries an identity worth checking.
|
||||
- **`usage(licence) → normalised rows`** — the vendor's usage reading, mapped to the common shape below.
|
||||
- **`deliver`** — the credential *value* only; the destination path is the consumer's, not the
|
||||
adapter's.
|
||||
|
||||
**A static-key vendor implements almost nothing** — `shape: static-key`, `accept` is the generic
|
||||
seal, `deliver` is the value, and `refresh`, `identity` and `usage` are absent or trivial. The
|
||||
abstraction earns its keep by making the common vendor small, not the rare one clever.
|
||||
|
||||
### The carve-out — the one place the guarantee is relaxed, said plainly
|
||||
|
||||
The mesh's standing principle is that it cannot read what it stores: `accept` seals to the holders and
|
||||
discards the plaintext ([ADR 0024](0024-model-access-is-a-provision.md)), and a provider seals nothing
|
||||
because the credential travels the mesh's own asymmetric channel
|
||||
([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)). A `refreshable-grant`
|
||||
credential cannot honour that principle and be centrally rotated at the same time, and central rotation
|
||||
is the working half [ADR 0024](0024-model-access-is-a-provision.md) is explicit about keeping.
|
||||
|
||||
**So, for `refreshable-grant` vendors only:**
|
||||
|
||||
- the **manager node holds the refresh token encrypted at rest** — readable by that node, because
|
||||
rotation requires it. This is the bounded exception.
|
||||
- **access tokens are still sealed per-holder**, as every credential is; a holder reads its own and no
|
||||
other node reads it.
|
||||
- the **refresh token is stripped on delivery** — it never reaches a consuming node. *A node never
|
||||
holds a refresh token* stays true for every node but the one manager.
|
||||
|
||||
**Static-key vendors keep the full guarantee.** There is no token to rotate, so there is nothing to
|
||||
hold readably, so `accept` discards the plaintext and the carve-out never fires. The majority of
|
||||
vendors are static-key, and the majority therefore lose nothing.
|
||||
|
||||
The exception is stated rather than hidden because a relaxed guarantee that is not written down is
|
||||
indistinguishable from a broken one. It is bounded on three axes at once: **refreshable-grant vendors
|
||||
only, the refresh token only, the manager node only.**
|
||||
|
||||
### The settled details this record also fixes
|
||||
|
||||
- **Usage is normalised to `(licence, consumer, period, metric, value)` plus the raw response as
|
||||
jsonb.** The metric is vendor-defined — Anthropic's `utilization%` is one metric, a token count is
|
||||
another — and no common unit is forced across vendors. The raw response is kept so a reading can be
|
||||
re-derived if the normalisation is later found wrong.
|
||||
- **Binding is explicit per consumer, and an unchosen consumer is refused — no implicit fallback.**
|
||||
This is the direction the resolver already takes, and it is the safe one: a mesh with several ways
|
||||
to reach a model refuses a consumer that has not said which, naming the candidates and the command,
|
||||
rather than silently choosing one ([ADR 0024](0024-model-access-is-a-provision.md), and
|
||||
[`14-model-access`](../03-DESIGN/01-to-be/14-model-access.md)).
|
||||
- **Subscription-seat authentication lives entirely inside the adapter**, never in the generic core.
|
||||
So does an interactive `/login` — an adapter-specific "adopt" origin for a credential a person must
|
||||
produce in a browser; the generic `licence key <name>` covers the static-key case.
|
||||
- **Anthropic is the first `refreshable-grant` adapter**, carrying the OAuth refresh, the usage
|
||||
reading, the identity guard and access-token-only delivery. **`anthropic-api-key` is a
|
||||
`static-key` adapter for the same vendor's plain API keys**, and is the early second case that
|
||||
proves the abstraction is not a single vendor wearing a coat: it exercises the whole path with the
|
||||
carve-out switched off.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The carve-out is the mesh's one deliberate relaxation of "it cannot read what it stores."** It is
|
||||
bounded to refreshable-grant vendors, to the refresh token, and to the manager node; the static-key
|
||||
majority keep the full guarantee unchanged. This is the open risk the analysis carried here, and it
|
||||
is recorded as an exception rather than pretended away.
|
||||
- **Anthropic collapses from special case to adapter.** The three binding columns
|
||||
(`nodes.node_license`, `nodes.hal_claude_account`, `agents.claude_account`) become three ordinary
|
||||
consumers of `model-access`; the `claude_*` schema becomes the generic licence tables plus one
|
||||
adapter. What was hardcoded becomes data keyed by `vendor`.
|
||||
- **Adding a vendor is adding an adapter, and a static-key vendor is nearly free.** The modules that
|
||||
want a model do not change when a vendor is added — they named `model-access`, not a vendor.
|
||||
- **The refresh-token concentration is now a stated property to defend, not an accident.** The manager
|
||||
node is a place a long-lived secret lives readably, and losing it or compromising it is a bounded,
|
||||
named blast radius rather than a surprise.
|
||||
|
||||
### How each claim here is checked
|
||||
|
||||
- **Vendor-blind provision, static-key path.** A lab scenario binds an `anthropic-api-key` licence to
|
||||
a consumer; the consumer resolves, receives a key sealed to its node, and reaches a model — and the
|
||||
key is **nowhere in the control plane's database** nor in anything that crossed the broker. This is
|
||||
the `licence_holder` sealed-per-node check that [`14-model-access`](../03-DESIGN/01-to-be/14-model-access.md)
|
||||
already runs, now asserted for a second vendor.
|
||||
- **The carve-out is exactly as narrow as stated.** For a `refreshable-grant` licence, a test asserts
|
||||
the refresh token exists (encrypted) **only on the manager node**, is **absent from every holder's
|
||||
delivery**, and that the delivered credential is access-token-only — and that for a `static-key`
|
||||
licence no refresh token is stored anywhere.
|
||||
- **Adapter selection is keyed by `vendor`.** A scenario with two vendors on two licences verifies each
|
||||
licence's lifecycle runs its own adapter, and that a consumer naming `model-access` never names a
|
||||
vendor to get one.
|
||||
- **Refuse-if-unchosen.** Already checked in [`14-model-access`](../03-DESIGN/01-to-be/14-model-access.md):
|
||||
a consumer with more than one candidate is refused with the candidates and the command named.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0024](0024-model-access-is-a-provision.md) — model access is a provision, a licence is a named
|
||||
thing, and `accept`; this record generalises its single vendor and keeps its working central
|
||||
rotation.
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — a provision names the
|
||||
coupling; `model-access` is drawn at the consumer's.
|
||||
- [ADR 0040](0040-what-a-module-is.md) — the naming rule and the neutral-interface / scoped-provider
|
||||
shape a vendor adapter follows.
|
||||
- [ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md) — registrar-scoped `public-dns`
|
||||
providers, the precedent a `vendor`-scoped adapter mirrors.
|
||||
- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) — the mesh seals credentials
|
||||
and holds no readable copy; the carve-out here is the bounded, named exception to that for a
|
||||
refreshable grant.
|
||||
- [`03-DESIGN/01-to-be/14-model-access.md`](../03-DESIGN/01-to-be/14-model-access.md) — the design this
|
||||
record extends, amended to describe the adapter generalisation.
|
||||
- The read-only vendor-agnostic analysis, 2026-09-05 (code workspace) — the inventory and the decisions
|
||||
taken on the open questions this record encodes.
|
||||
@@ -0,0 +1,166 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0030-data-outlives-the-mesh-that-declared-it.md
|
||||
---
|
||||
|
||||
# 51. Shared data is the operator's, and a module is granted access to it
|
||||
|
||||
## Context
|
||||
|
||||
**Eight modules declared one filesystem as eight private ones.** The media stack — a library
|
||||
server, the acquisition managers for films, series, music and books, a subtitle fetcher and two
|
||||
download clients — shares directories on one machine: the download clients write into
|
||||
`/services/media/downloads` and the managers read it; the managers write into the libraries and
|
||||
the library server reads them. That sharing is the entire point of the stack. Yet each module
|
||||
declared every shared directory it touched as its own `directory` resource, with an owner and a
|
||||
mode. `/services/media/downloads` was written seven times, as seven private directories that
|
||||
happen to be the same path.
|
||||
|
||||
**The resolver refuses exactly that, and is right to.** Two modules declaring one path on one node
|
||||
are refused by name, with no exemption for identical content and no merge — because two owners of
|
||||
one path is the class of fault this repository keeps recording ([04-ISSUES/036](../04-ISSUES/036-six-modules-own-what-they-must-share/00-report.md)).
|
||||
So the stack as written refuses its own only sensible assignment: all of it on one machine,
|
||||
sharing one filesystem. It passed today only because no test co-resolves any two of the eight. The
|
||||
first machine assigned two of them together is where the refusal would have surfaced.
|
||||
|
||||
**The vocabulary had one word for two intentions, and this was already seen.**
|
||||
[04-ISSUES/026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md) found the
|
||||
same gap from the other side and named it precisely: two kinds of mount are spelled identically —
|
||||
*the directory my data lives in*, which the mesh creates and owns, and *a facility I was granted*,
|
||||
which already exists and the mesh only reaches. That issue deferred inventing a field to tell them
|
||||
apart, because doing so is a design decision and it declined to make one to get a check green. This
|
||||
is that decision.
|
||||
|
||||
**A `directory` resource is owned, on every axis.** The host creates it, sets its owner and mode,
|
||||
and removes it when it is empty and no longer declared — it *removes what it made and leaves what
|
||||
it merely configured* ([ADR 0005](0005-the-node-host.md), [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md)).
|
||||
A shared media library is none of that. It existed before the mesh, several modules read and write
|
||||
it at once, and losing it is the one failure that does not recover. It is not any module's
|
||||
resource; it is the operator's, and a module only needs to be let at it.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Make the stack one module with several containers**, the way the mail system already is. The
|
||||
issue raises it directly: is a set of modules that must share a filesystem really one module?
|
||||
**Rejected.** The eight are independently assignable and independently useful — a person may run
|
||||
the download client without the library server, or the film manager without the music one — and
|
||||
folding them into a single module to express a shared directory would make *what a module is*
|
||||
turn on an incidental filesystem contract. It also does not generalise: the next pipeline of
|
||||
modules handing files to each other on one machine (an ingest folder, a spool, a drop directory)
|
||||
would face the same wall and the same wrong remedy.
|
||||
|
||||
2. **One module owns the directories and the rest `require` them.** **Rejected**, and this is the
|
||||
heart of the decision. Nobody owns shared operator data. The library predates the mesh and
|
||||
outlives any one module, so making the library server or a manager its owner means unassigning
|
||||
that module orphans everyone else's access — and the owner would set the owner and mode of a
|
||||
tree it did not create. Ownership is the wrong relationship to model, because the true owner is
|
||||
not a module at all.
|
||||
|
||||
3. **A flag on a directory resource** — `external: true`, or an owner of `operator`. **Rejected.**
|
||||
It overloads one shape with a boolean that inverts every one of its semantics: created becomes
|
||||
*must already exist*, owned becomes *touch nothing*, removed-when-empty becomes *never removed*.
|
||||
That is the *two-kinds-spelled-identically* trap of 04-ISSUES/026 reintroduced with a single
|
||||
quiet field — a reviewer reading `type: directory` would have to check one boolean elsewhere to
|
||||
know whether the mesh owns the thing at all.
|
||||
|
||||
4. **A distinct `accesses` declaration, separate from resources.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**Shared, pre-existing data is operator-owned and external. The mesh does not create it, does not
|
||||
set its owner or mode, does not reconcile it and does not remove it.** A media library, a download
|
||||
spool, an ingest directory is the operator's, and the mesh is a guest in it.
|
||||
|
||||
**A module declares that it needs *access* to such a path, not that it owns a resource there.** The
|
||||
manifest field is `accesses`: a list of `{path, mode}`, where mode is `read` or `read-write` and
|
||||
absent narrows to `read` — the safe default, because the danger with an access is being given more
|
||||
than was meant, not less. A module's own configuration and state directories stay owned
|
||||
`directory` resources; only the shared, pre-existing paths become accesses.
|
||||
|
||||
**The host mounts an accessed path and owns nothing about it.** It reaches the machine as a new
|
||||
declaration shape, `access`, distinct from `directory`. The host confirms the path is present and
|
||||
does nothing else — no create, no chown, no mode, no removal.
|
||||
|
||||
**An accessed path absent at apply time is refused, clearly, not created.** The mesh does not own
|
||||
it, so conjuring it would be a lie the host then acts on — and specifically the lie 04-ISSUES/026
|
||||
records, where a bind mount whose source does not exist is made by the container runtime as root
|
||||
with the wrong ownership. The host says the operator must provide the path instead.
|
||||
|
||||
**Several modules accessing one path is normal, and never refused.** The duplicate-path refusal is
|
||||
about *ownership*, not *use*: it applies to resources a module owns and to those alone. An access
|
||||
is not a resource and never enters the check, so the eight-module stack co-resolves. What stays
|
||||
refused is genuine rivalry — two modules owning one path — and the new contradiction it exposes: a
|
||||
path one module owns while another merely accesses it, because that asserts both that the mesh owns
|
||||
the directory and that the operator does.
|
||||
|
||||
This is a decision and not a patch because it settles *what a module may say about a path it did
|
||||
not make*, which every co-located file-handoff in the catalogue now and later depends on — and
|
||||
because it draws the ownership line [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md)
|
||||
started: the host owns what it made and keeps what it merely configured, and this adds the third
|
||||
case it did not have a word for — what it neither made nor configured, and must not touch.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **The stack co-resolves.** A control-plane unit test assigns two modules that declare access to
|
||||
one path on one node and asserts no refusal — the exact case the resolver refuses when the same
|
||||
path is owned. The mirror test, two modules *owning* one path, still refuses, so the sharing
|
||||
vocabulary does not weaken the rule it sits beside.
|
||||
- **Ownership and access cannot both be claimed of one path.** A unit test asserts the resolver
|
||||
refuses a path one module owns and another accesses, naming both.
|
||||
- **Absent is refused, not created.** A host unit test applies an access to a path that does not
|
||||
exist and asserts a clear refusal that names the operator, and that nothing was created.
|
||||
- **Present is confirmed and nothing moves.** A host unit test applies an access to an existing
|
||||
directory and asserts the apply reports no change and disturbs nothing.
|
||||
- **Undeclaring never removes.** A host unit test drops a previously declared access and asserts
|
||||
the operator's directory and its contents are left exactly as they were — the data-loss failure
|
||||
[ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md) exists to prevent, on a directory the
|
||||
mesh never made.
|
||||
- **Every media manifest is corrected.** No `/services/media/*` path is an owned `directory`
|
||||
resource in any of the eight; each is an `accesses` entry, and each module's own config and state
|
||||
directories remain owned. Checked by the control-plane manifest parser, which now understands
|
||||
`accesses` and refuses a malformed one.
|
||||
|
||||
## Consequences
|
||||
|
||||
**A shared filesystem between co-located modules now has a vocabulary**, and it is not the media
|
||||
stack's alone: any pipeline handing files to a neighbour on one machine — an ingest directory, a
|
||||
spool, a drop folder — says *I access this operator path* rather than *I own this directory*, and
|
||||
several of them may say it of one path.
|
||||
|
||||
**Unassigning a module that reached shared data leaves the data.** Correct, and the same trade
|
||||
[ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md) made for owned directories: removing
|
||||
data is a person's act, done knowingly, not a side effect of unassignment.
|
||||
|
||||
**The operator must provision the shared paths before the stack is applied**, and a machine that
|
||||
lacks one is told plainly which. That is a real new obligation, and it is the right one: the mesh
|
||||
cannot own what predates it, so it cannot create it either, and saying so at apply time beats a
|
||||
directory conjured as root and a service that half-works.
|
||||
|
||||
**The host vocabulary grew by one shape**, which is a cost — every added shape widens what a
|
||||
compromised control plane can express ([ADR 0005](0005-the-node-host.md)). It is a narrow one: an
|
||||
`access` is confirmed by a stat and grants the host no new action. It earns its place by letting
|
||||
the host refuse to create what it must not own, which no existing shape could say.
|
||||
|
||||
**A path can be both owned and accessed only by refusal.** If a future manifest declares one path
|
||||
as an owned directory in one module and an access in another, the resolver refuses it rather than
|
||||
guessing which is meant — the two assertions about who owns the data cannot both hold.
|
||||
|
||||
## References
|
||||
|
||||
- [04-ISSUES/036](../04-ISSUES/036-six-modules-own-what-they-must-share/00-report.md) — six (in
|
||||
fact eight) modules own what they must share; the problem this resolves
|
||||
- [04-ISSUES/026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md) — the two
|
||||
kinds of mount spelled identically, which deferred this field to a decision
|
||||
- [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md) — data outlives the mesh; the host
|
||||
keeps what it did not make. This record adds the case it had no word for
|
||||
- [ADR 0005](0005-the-node-host.md) — the host removes what it made and leaves what it merely
|
||||
configured; the vocabulary is finite and every shape is a security decision
|
||||
- [ADR 0040](0040-what-a-module-is.md) — what a module is; an access is a new thing a module may
|
||||
say about the machine it lands on
|
||||
- mesh-control `feat/shared-data-access`, mesh-catalog `feat/media-access-not-ownership`,
|
||||
mesh-host `feat/mount-operator-owned` — the mechanism, the corrected manifests, and the host
|
||||
shape
|
||||
@@ -0,0 +1,198 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 52. An init step is a container run once to completion, gating what follows
|
||||
|
||||
## Context
|
||||
|
||||
**A module can declare things that exist; it cannot declare a step that runs.** The host owns a
|
||||
finite vocabulary of shapes — `file`, `directory`, `service`, `package`, `container`, `action` —
|
||||
and every one but `action` describes *state*: a thing that should be present, with content or a
|
||||
mode or an image, which the host reconciles toward ([ADR 0005](0005-the-node-host.md)). That is
|
||||
right for what it covers. But a real class of modules needs, once, to *run their own code at a
|
||||
point in their own lifecycle* — and the vocabulary has no word for it
|
||||
([04-ISSUES/037](../04-ISSUES/037-a-module-cannot-run-code-at-a-lifecycle-phase/00-report.md)).
|
||||
|
||||
**mosquitto is the sharp case, and it fails silently without this.** Its Dynamic Security plugin
|
||||
loads at broker start and refuses to come up unless `dynamic-security.json` already holds an admin
|
||||
client. Seeding that file is a step that must happen *after* the data directory exists and *before*
|
||||
the broker container starts. The manifest can declare the directory, the config file and the broker
|
||||
container; it cannot declare "seed this, once, before that container starts." Written as it is
|
||||
today, the broker starts against an unseeded store and the plugin aborts — and the next reconcile
|
||||
does not fix it, because nothing in the declaration ever seeds the file.
|
||||
|
||||
**It is not one module's defect.** The database providers need the same to run a first-boot
|
||||
migration, an extension enable, or a health gate before they are announced ready; today that works
|
||||
only where the *image* happens to seed itself from an environment variable, and anything the mesh
|
||||
must run once against the server has no home. This is the timing face of the same gap
|
||||
[04-ISSUES/035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md) records
|
||||
from the content side: the manifest needed *the file to exist before first start*, and had only
|
||||
*the file has this content, forever*.
|
||||
|
||||
**The obvious answer is the one that already went wrong.** An earlier mesh had exactly this as a
|
||||
feature — event-driven hooks that ran custom code at phases of build, publish and deploy. It was
|
||||
powerful and it was *complex to set up and flaky*, and that fragility, not the need, is the content
|
||||
of the issue. Whatever this becomes must not rebuild that engine.
|
||||
|
||||
**The ground has shifted since that engine, in a way that makes a much smaller answer possible.** A
|
||||
module with tools or events now runs a **process of its own** — a container carrying the module's
|
||||
compiled code, holding the single broker account the mesh scoped to it, isolated from every other
|
||||
module ([ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)). The
|
||||
code that must run at first boot is *already in that image, under that account*. So the mesh does
|
||||
not need a way to run a module's code — it has one. It needs a way to say **run this container to
|
||||
completion, and start the one that depends on it only after it has.**
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A host `run` shape — a command the host executes on the machine.** The direct reading of
|
||||
"run code at a lifecycle phase." **Rejected.** The host's vocabulary is finite and every added
|
||||
shape is a security decision, because it widens what a *compromised control plane* can express
|
||||
([ADR 0005](0005-the-node-host.md)). A general "run this command as the host" is the largest
|
||||
such widening there is: the blast radius is the whole machine, as root. The mesh already drew
|
||||
this exact line for `action` — it runs a command, and so it is *permitted from the bundle and
|
||||
refused from the link*, because the bundle arrives with the binary and the link is a separate
|
||||
party with an unbounded reach. A new host-command shape usable by an ordinary module would be an
|
||||
`action` from the link by another name, which is precisely what is refused.
|
||||
|
||||
2. **Per-phase lifecycle hooks on a module** — `pre-start`, `post-start`, `pre-remove`, and their
|
||||
build/publish cousins, each naming code the mesh runs at that phase. The general answer, and the
|
||||
old feature. **Rejected for now.** It is the flaky engine the issue warns against, and most of
|
||||
its phases have no present need. Deciding the full set of phases, where each one's code runs, and
|
||||
how each is made idempotent is a large design taken to buy capability nothing yet asks for. The
|
||||
three blocked modules all need one phase — *before a container starts* — and a mechanism narrow
|
||||
enough to be obviously correct beats a general one that is not.
|
||||
|
||||
3. **A distinct one-shot resource type** — a new shape, sibling to `container`, that names an image
|
||||
and runs it once. **Rejected.** It grows the host vocabulary by a whole shape (a `Type`, a
|
||||
struct, an applier, a place in every host's shape list) to express something a `container` almost
|
||||
already is. A one-shot *is* a container — a pinned image, an account, volumes, an environment —
|
||||
that happens to exit. Spending a new shape on the difference is the cost of option 1 in smaller
|
||||
type, for a capability the existing shape can carry with one modifier.
|
||||
|
||||
4. **A modifier on the existing `container` shape: this container runs once, to completion, and the
|
||||
host gates the apply on it.** **Adopted.** It reuses the shape the host already has, adds no new
|
||||
host action, and leans on two guarantees the host already gives — *apply in declared order,
|
||||
never sorted*, and *a failed step fails the apply* — to turn "before that container starts" into
|
||||
an emergent property of ordering rather than a dependency graph the host must resolve.
|
||||
|
||||
## Decision
|
||||
|
||||
**A run-once step is an ordinary `container`, marked to run to completion.** The manifest sets
|
||||
`run-once: true` on a container resource. Everything else about it is a container as before — a
|
||||
digest-pinned image ([ADR 0006](0006-the-substrate-and-the-control-plane.md)), volumes, an
|
||||
environment, and for a module's own code the same scoped account its runtime already holds
|
||||
([ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)). The mesh adds
|
||||
no way to run code; it marks a container the host must **run to completion and require to exit 0**,
|
||||
rather than start and leave running.
|
||||
|
||||
**The gate is declaration order, not a named dependency.** The host applies a declaration in the
|
||||
order it is given, does not sort, and does not resolve dependencies — ordering is a decision, and it
|
||||
is the control plane's ([ADR 0005](0005-the-node-host.md),
|
||||
[04-ISSUES/013](../04-ISSUES/013-a-file-arrives-after-the-service-that-needs-it/00-report.md)). A
|
||||
run-once step is placed *before* the container that depends on it, and **a failed run-once step
|
||||
halts the apply**, exactly as a failed `action` does — so everything the declaration places after
|
||||
it, the broker included, is never reached until the step has completed. "Before the broker starts"
|
||||
is therefore expressed by list position plus completion, and the host cross-references nothing.
|
||||
|
||||
**Completion is recorded, and a re-apply does not re-run it.** The host records what it applied only
|
||||
after the fact, as the digest of the declaration that produced it
|
||||
([ADR 0018](0018-a-picture-is-read-from-what-runs.md)) — a run-once step no differently. Because the
|
||||
step leaves nothing running to inspect, that persisted digest, not a live container, is the marker
|
||||
that it happened. On a later apply the host finds the digest already recorded for this exact
|
||||
declaration and does nothing; it re-runs only when the declaration's digest has changed, and a step
|
||||
that exited non-zero recorded nothing and so is retried next apply. This is the reconcilable,
|
||||
idempotent discipline the state shapes get for free, made explicit for a step.
|
||||
|
||||
This is a decision and not a patch because it settles **what a module may say about running its own
|
||||
code**, which the whole catalogue of providers — a seed before start, a first-boot migration, a
|
||||
health gate — now and later depends on, and because it draws the line the issue asked for: the
|
||||
narrowest sound mechanism that unblocks the three modules without rebuilding the hook engine whose
|
||||
fragility is the warning.
|
||||
|
||||
### The security bound, stated plainly
|
||||
|
||||
**The host gains no new action and no new shape.** `run-once` is a boolean modifier on the
|
||||
`container` shape that already exists. A run-once container is strictly *less* powerful than an
|
||||
`action`: it cannot run an arbitrary host command, only a digest-pinned image under an account the
|
||||
mesh scoped — which is exactly the capability `container` already grants from the link. A control
|
||||
plane that is compromised can express nothing through `run-once` it could not already express by
|
||||
declaring an ordinary `container`. The dangerous expansion of option 1 — a command the host runs on
|
||||
the machine — is not made.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **A run-once step runs to completion and its exit 0 is required.** A host unit test applies a
|
||||
run-once container whose image exits 0, asserts the host ran it to completion (not detached, not
|
||||
left running) and reported it done; a sibling test applies one that exits non-zero and asserts
|
||||
the apply fails, naming the step.
|
||||
- **A failed run-once step gates what follows.** A host unit test places a run-once container that
|
||||
exits non-zero before another container and asserts the second is never started and the apply is
|
||||
reported gated — the mirror of the existing test that a failed action stops what follows.
|
||||
- **It is not re-run once it has completed.** A host unit test applies a run-once step, then applies
|
||||
the identical declaration again with the first run's record present, and asserts the second apply
|
||||
runs nothing and reports the step unchanged.
|
||||
- **A changed declaration re-runs it.** A host unit test applies a run-once step, then applies one
|
||||
whose image or environment differs, and asserts it runs again — the digest moved, so the marker no
|
||||
longer matches.
|
||||
- **The vocabulary carries the field end to end.** A control-plane unit test resolves a module whose
|
||||
manifest marks a container `run-once` and asserts the rendered host declaration carries the field,
|
||||
in author order before the container it gates; the manifest parser refuses a `run-once` that is
|
||||
not boolean.
|
||||
- **mosquitto seeds before the broker.** mosquitto's manifest declares a run-once init container,
|
||||
before the `server` (broker) container, that writes the admin client into `dynamic-security.json`
|
||||
and exits — checked by the control-plane resolver, which now understands the field, and by the
|
||||
ordering of the rendered declaration. The end-to-end proof that it runs exactly once, at the right
|
||||
phase, and converges on re-apply is owed to a lab scenario ([04-ISSUES/037] open question), which
|
||||
this record does not close.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The three blocked modules gain a home for their step.** mosquitto seeds its dynsec admin before
|
||||
the broker; a provider that must migrate or health-gate at first boot declares a run-once step in
|
||||
its own runtime image, under its own account, before the container that depends on it.
|
||||
- **The general lifecycle hook is deferred, deliberately.** Only *before a container starts* is
|
||||
bought here. `post-start`, `pre-remove` and the build/publish phases remain unbuilt, and the day
|
||||
one is genuinely needed it is decided then, against a need, not speculatively — the same restraint
|
||||
that kept this from being the old engine.
|
||||
- **The seed-then-mutate file is safe if the step is written to be.** A run-once seed writes
|
||||
`dynamic-security.json` only when it is absent and never reconciles it, so what the running plugin
|
||||
grows in that file afterward is never wiped
|
||||
([04-ISSUES/035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md)).
|
||||
The host's marker guarantees the step is not re-run; the step's own code guarantees it does not
|
||||
clobber on the pass it does run.
|
||||
- **The host vocabulary did not grow, and that is the point.** The cost of a run-once step is one
|
||||
boolean and a completion path in the container applier, not a new shape and not a new action. The
|
||||
mesh expresses ordering and completion; the module runs its own code, where it already runs it.
|
||||
- **A run-once step that never converges is a stuck apply, loudly.** A step that exits non-zero
|
||||
every time halts the apply every time, and the container it gates never starts — which is the
|
||||
correct failure, reported, rather than a broker that half-starts against an unseeded store and a
|
||||
reconcile that reports success. It is failed forward, not failed silent.
|
||||
|
||||
## References
|
||||
|
||||
- [04-ISSUES/037](../04-ISSUES/037-a-module-cannot-run-code-at-a-lifecycle-phase/00-report.md) — a
|
||||
module cannot run its own code at a lifecycle phase; the gap this resolves, and the warning about
|
||||
the old hook engine
|
||||
- [04-ISSUES/035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md) — the
|
||||
content face of the same gap: a file needed before first start, that the running program then
|
||||
mutates
|
||||
- [04-ISSUES/013](../04-ISSUES/013-a-file-arrives-after-the-service-that-needs-it/00-report.md) — the
|
||||
order of a declaration is the control plane's, and the host applies it as given; the gate rests on
|
||||
this
|
||||
- [ADR 0005](0005-the-node-host.md) — the host's finite vocabulary, the ordered declaration it does
|
||||
not sort, and `action` as the shape a command already is and why it is refused from the link
|
||||
- [ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md) — a module runs
|
||||
its code as its own process under its own account; a run-once step is that process, run to
|
||||
completion
|
||||
- [ADR 0018](0018-a-picture-is-read-from-what-runs.md) — what was applied is recorded after it works;
|
||||
the completion marker is that record
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — a container is pinned by digest; a
|
||||
run-once container no differently
|
||||
- mesh-control `feat/lifecycle-run-once`, mesh-host `feat/apply-run-once`, mesh-catalog
|
||||
`feat/mosquitto-bootstrap` — the vocabulary, the apply support, and mosquitto's seeded broker
|
||||
@@ -0,0 +1,181 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-06
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0052-a-step-that-runs-once-before-a-container.md
|
||||
---
|
||||
|
||||
# 53. A scheduled step is a container run on a recurring schedule
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0052](0052-a-step-that-runs-once-before-a-container.md) gave the mesh a step that runs *once*;
|
||||
a real class of modules needs one that runs *again and again*.** kometa reconciles a media library
|
||||
against its lists on a timer; a ticketing integration polls its source for new work every few minutes;
|
||||
a backup, a cache warm, a metrics roll-up all recur. The host's vocabulary describes *state* — a file,
|
||||
a directory, a container that should be running — and 0052 added *a step that happens once and is
|
||||
done*. Neither says *this should happen every night at 3, forever*
|
||||
([04-ISSUES/037](../04-ISSUES/037-a-module-cannot-run-code-at-a-lifecycle-phase/00-report.md) named
|
||||
the lifecycle gap; 0052 closed the once-before-start face of it and left the recurring face open).
|
||||
|
||||
**The modules that need it already run their own code.** As with run-once, the ground has shifted
|
||||
since the old mesh's flaky hooks: a module with tools or events runs a **process of its own** — a
|
||||
container carrying the module's compiled code under the single scoped account the mesh gave it
|
||||
([ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)). The work that
|
||||
must recur is already in that image, under that account. The mesh does not need a way to run a
|
||||
module's code on a timer — it has the code and the account. It needs a way to say **run this
|
||||
container again on this cadence**.
|
||||
|
||||
**The shape of the answer is already decided, one modifier over.** 0052 rejected a host `run` command,
|
||||
per-phase hooks, and a distinct one-shot resource type, and adopted *a modifier on the `container`
|
||||
shape the host already has*, because a one-shot is a container that happens to exit. A scheduled step
|
||||
is the same container that happens to exit — run repeatedly. Reusing that shape keeps the host
|
||||
vocabulary flat and inherits 0052's security bound whole. The only thing 0052's `run-once` does not
|
||||
carry is *when to run it again*.
|
||||
|
||||
**One difference from run-once changes a rule, and it is the reason this is its own record.** A
|
||||
run-once step **gates the apply**: it is placed before the container that depends on it, and a failure
|
||||
halts everything after it, because "seed the store before the broker starts" is a correctness
|
||||
precondition ([ADR 0052](0052-a-step-that-runs-once-before-a-container.md)). A scheduled step is the
|
||||
opposite: it runs *after* the machine is up and converged, on its own clock, and a single failed run
|
||||
is an ordinary operational event — the next run comes anyway. A scheduled step that halted the apply,
|
||||
or that a failed run marked the node not-current over, would make a routine poll into a reason the
|
||||
whole machine reads as broken. So the gating rule 0052 established is exactly the rule this record
|
||||
must **not** inherit.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A host `cron`/`timer` shape — the host installs a system timer that runs a command.** The direct
|
||||
reading. **Rejected**, for the reason 0052 rejected a host `run` shape: it widens what a
|
||||
*compromised control plane* can express toward "run this command on the machine, forever," which is
|
||||
the largest widening there is, and it is `action`-from-the-link by another name
|
||||
([ADR 0005](0005-the-node-host.md)). A recurring command is worse than a one-off, because it
|
||||
persists.
|
||||
|
||||
2. **Per-phase lifecycle hooks** — `on-schedule` joining `pre-start`/`post-start` as named code the
|
||||
mesh runs. **Rejected for now**, as in 0052: it is the flaky hook engine the issue warns against,
|
||||
and the three modules that need this need one thing — *run this container on a cadence* — which a
|
||||
narrow modifier expresses without deciding a whole hook vocabulary.
|
||||
|
||||
3. **A distinct `scheduled` resource type**, sibling to `container`. **Rejected**, as 0052 rejected a
|
||||
distinct one-shot type: it spends a whole new host shape (a `Type`, a struct, an applier, a place
|
||||
in every host's shape list) on something a `container` already almost is — a scheduled task *is* a
|
||||
container (pinned image, account, volumes, environment) that runs on a clock.
|
||||
|
||||
4. **A modifier on the existing `container` shape: `schedule`, a cron expression the host runs the
|
||||
container on.** **Adopted.** It reuses the shape the host has, adds no new host action, and sits
|
||||
beside `run-once` as its recurring twin — the same container, exited, run again.
|
||||
|
||||
## Decision
|
||||
|
||||
**A scheduled step is an ordinary `container`, marked with a `schedule`.** The manifest sets
|
||||
`schedule: "<cron>"` on a container resource — a standard five-field cron expression. Everything else
|
||||
about it is a container as before: a digest-pinned image
|
||||
([ADR 0006](0006-the-substrate-and-the-control-plane.md)), volumes, an environment, and for a module's
|
||||
own code the same scoped account its runtime already holds
|
||||
([ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)). The mesh adds no
|
||||
way to run code; it marks a container the host must **run on that cadence, each time to completion**,
|
||||
rather than start once and leave running (a service) or run once and gate (a run-once step).
|
||||
|
||||
**A run is fired by the clock, not by the apply, and does not gate it.** Applying the declaration
|
||||
installs the schedule; it does not run the step. The machine converges — reports applied and current —
|
||||
as soon as the schedule is installed, exactly as it does for a service that is running. Thereafter the
|
||||
host fires the container when the cron expression is due. This is the deliberate inversion of 0052:
|
||||
a scheduled step is downstream of convergence, not a precondition of it.
|
||||
|
||||
**A failed run is recorded and the next run still comes; it never marks the node not-current.** A run
|
||||
that exits non-zero is logged against the module — visible, auditable — but it does not fail the apply,
|
||||
does not halt other resources, and does not flip the node's reported state. A poll that fails at 03:00
|
||||
and succeeds at 03:05 is the system working, not a machine that needs attention. A step that fails
|
||||
*every* time is a loud, repeating log entry, which is the correct signal for "this recurring job is
|
||||
broken" — distinct from "this machine did not converge."
|
||||
|
||||
**Runs do not stack.** If a run is still going when the next is due, the host skips the due run rather
|
||||
than starting a second copy, and logs the skip. A slow nightly job that occasionally overruns must not
|
||||
spawn a growing pile of concurrent containers competing for the same account and volumes — the failure
|
||||
mode that made the old timers dangerous.
|
||||
|
||||
**Each run is independent and idempotent by the module's own code.** The mesh guarantees only *the
|
||||
container is run on the cadence*; that a run does the right thing when the previous one half-finished
|
||||
is the module's contract, the same discipline a run-once seed owes
|
||||
([ADR 0052](0052-a-step-that-runs-once-before-a-container.md),
|
||||
[04-ISSUES/035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md)).
|
||||
|
||||
This is a decision and not a patch because it settles **what a module may say about running its own
|
||||
code on a cadence**, which every recurring provider job — a sync, a poll, a roll-up — now and later
|
||||
depends on, and because it draws the line 0052 left: the recurring twin of run-once, with the gating
|
||||
rule deliberately reversed so a routine job's failure is never a machine's failure.
|
||||
|
||||
### The security bound, stated plainly
|
||||
|
||||
**The host gains no new action and no new shape.** `schedule` is a string modifier on the `container`
|
||||
shape that already exists. A scheduled container is strictly *less* powerful than an `action`: it runs
|
||||
a digest-pinned image under an account the mesh scoped, which is exactly what `container` already
|
||||
grants from the link, and it cannot run an arbitrary host command. A compromised control plane can
|
||||
express nothing through `schedule` it could not already express by declaring a `container` — the cron
|
||||
string only says *how often*, not *what*. The dangerous expansion of option 1 — a command the host
|
||||
runs on the machine on a timer — is not made.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **A scheduled step runs when the schedule is due.** A host unit test installs a container with a
|
||||
schedule that is due immediately (or advances a injected clock to when it is due) and asserts the
|
||||
host ran it to completion; a sibling test with a schedule not yet due asserts it has not run.
|
||||
- **Installing it does not run it, and the node is current without a run.** A host unit test applies a
|
||||
scheduled container and asserts the apply reports current *before* any run has fired — the schedule
|
||||
is state that is present, not a step that gated.
|
||||
- **A failed run does not fail the apply or the node.** A host unit test fires a scheduled container
|
||||
that exits non-zero and asserts the failure is recorded against the module, the apply is not failed,
|
||||
and the node stays current — the mirror of the run-once test where a non-zero exit *does* halt.
|
||||
- **Runs do not stack.** A host unit test fires a scheduled container whose run outlasts its next due
|
||||
time and asserts the host skipped the due run and logged the skip, rather than starting a second
|
||||
container.
|
||||
- **The vocabulary carries the field end to end.** A control-plane unit test resolves a module whose
|
||||
manifest sets `schedule` on a container and asserts the rendered host declaration carries the field;
|
||||
the manifest parser refuses a `schedule` that is not a valid cron expression, and refuses a container
|
||||
that is both `run-once` and `schedule` (a step is one or the other, never both).
|
||||
- **A real module recurs in the lab.** A converted module declaring a scheduled step (kometa's library
|
||||
sync, or a poller) is assigned in a lab scenario, and the scenario asserts the scheduled container
|
||||
fires on its cadence and its account and volumes are the module's — the end-to-end proof this record
|
||||
owes, as 0052 owed its run-once lab proof.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The recurring providers gain a home for their cadence.** kometa reconciles on its schedule; a
|
||||
poller polls; a roll-up rolls up — each a scheduled step in its own runtime image, under its own
|
||||
account, on the cron it declares.
|
||||
- **The general lifecycle hook is still deferred.** Only *run once before* (0052) and *run on a
|
||||
cadence* (this) are bought. `post-start`, `pre-remove` and the build/publish phases remain unbuilt,
|
||||
decided when a real need arrives, not speculatively — the restraint that kept both from being the old
|
||||
engine.
|
||||
- **A recurring job's failure is loud but not fatal.** The node stays current while a scheduled step
|
||||
fails and retries; a step that fails forever is a repeating log entry, not a machine marked broken.
|
||||
This is the correct separation — a machine's convergence and a job's success are different questions —
|
||||
and it is why this could not simply be `run-once` without the schedule.
|
||||
- **The host vocabulary did not grow, again, and that is the point.** The cost of a scheduled step is
|
||||
one string field and a cron loop in the container applier, not a new shape and not a new action. The
|
||||
mesh expresses cadence; the module runs its own code, where it already runs it.
|
||||
- **`run-once` and `schedule` are exclusive and complete for now.** A container runs once and gates, or
|
||||
runs on a cadence and does not, or runs and stays up (a service). A manifest that asks for two of
|
||||
these at once is refused, because the three are distinct answers to "how does this container run."
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md) — the run-once step; this is its
|
||||
recurring twin, reusing the `container`-modifier shape and inheriting its security bound, and
|
||||
reversing its gating rule
|
||||
- [04-ISSUES/037](../04-ISSUES/037-a-module-cannot-run-code-at-a-lifecycle-phase/00-report.md) — a
|
||||
module cannot run its own code at a lifecycle phase; 0052 closed the once-before-start face, this
|
||||
closes the recurring face
|
||||
- [ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md) — a module runs
|
||||
its code as its own process under its own account; a scheduled step is that process, run on a cadence
|
||||
- [ADR 0005](0005-the-node-host.md) — the host's finite vocabulary, and why a host command (even on a
|
||||
timer) is refused from the link
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — a container is pinned by digest; a
|
||||
scheduled container no differently
|
||||
- [ADR 0018](0018-a-picture-is-read-from-what-runs.md) — what was applied is recorded after it works;
|
||||
the installed schedule is state, a fired run is an event
|
||||
- mesh-control `feat/schedule-container`, mesh-host `feat/apply-schedule`, and the converted module
|
||||
that first declares a scheduled step — the vocabulary, the apply support, and the recurring proof
|
||||
@@ -0,0 +1,178 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-06
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0050-model-access-is-vendor-agnostic.md
|
||||
---
|
||||
|
||||
# 54. Model usage is a vendor-neutral record, produced by the adapter, at two grains
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0050](0050-model-access-is-vendor-agnostic.md) fixed the *shape* of a usage reading and left
|
||||
its *home* open.** It decided that an adapter may expose `usage(licence) → normalised rows`, that a
|
||||
row is `(licence, consumer, period, metric, value)` plus the raw vendor response as jsonb, and that the
|
||||
metric is vendor-defined (Anthropic's `utilization%` is one metric, a token count another). It did not
|
||||
say where those rows are stored, how they get produced on a cadence, or whether the only grain is the
|
||||
whole licence — and the mesh's first real consumer needs all three answered.
|
||||
|
||||
**The predecessor mesh recorded usage at two grains, and both are wanted.** It polled the vendor for a
|
||||
licence-level reading (an account's `utilization%`), and it also attributed **per-session** token and
|
||||
cost — which model, how many input and output tokens, what it cost — parsed from the agent's own
|
||||
transcript, including the case where a long session switched the account it billed against mid-way. The
|
||||
licence-level reading answers "how close is this subscription to its cap"; the session-level reading
|
||||
answers "what did this piece of work cost, and against which account." A mesh that kept only the first
|
||||
could not bill a project or notice a runaway session; keeping only the second could not see a cap
|
||||
approaching. Both are load-bearing and neither subsumes the other.
|
||||
|
||||
**A session is already a consumer, so the second grain needs no second vocabulary.**
|
||||
[ADR 0026](0026-the-mesh-has-a-session-of-its-own.md) and design 15 establish that an agent session is
|
||||
a thing in its own right, and that **model access is bound to the session, not to the machine** — a
|
||||
session is a *consumer* of `model-access`, identified by `(node, module)` and its own id. That is
|
||||
exactly the `consumer` column ADR 0050 already put in the usage row. So the two grains are not two
|
||||
schemas; they are the same row at two consumer resolutions: the holding module for the licence grain,
|
||||
the session for the finer one. The design's own still-open worker-naming gap (design 14) is the same
|
||||
gap here and is left where it is — a session id distinguishes what `(node, module)` cannot.
|
||||
|
||||
**The pieces this needs already exist.** A periodic reading is a **scheduled step**
|
||||
([ADR 0053](0053-a-step-that-runs-on-a-schedule.md)) — the adapter's `usage()` poll is a container the
|
||||
mesh runs on a cadence, which is precisely what 0053 was built for. A durable audit of what happened is
|
||||
**an event the audit trail records** ([ADR 0041](0041-events-are-a-relationship.md),
|
||||
[ADR 0042](0042-the-shape-of-an-event-on-the-wire.md)); the `audit-logger` module already consumes every
|
||||
event. And a queryable current picture is **a context store**, the same shape the licences themselves
|
||||
live in ([ADR 0008](0008-a-context-owns-its-store.md)). Nothing new in kind is required; what is missing is
|
||||
the decision to point them at usage.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Licence grain only — a single `utilization%` poll, nothing per session.** Rejected: it cannot
|
||||
attribute cost to a piece of work or catch a session that is burning an account down, which is half
|
||||
of why usage is recorded at all.
|
||||
|
||||
2. **A bespoke `sessions` schema mirroring the old mesh's `*_sessions` / `*_session_account_usage`
|
||||
tables.** Rejected: it reintroduces a second, vendor-shaped vocabulary for something the mesh
|
||||
already names — a session is a consumer, and its usage is a usage row. A parallel schema would drift
|
||||
from the `model-access` vocabulary and force every reader to learn two.
|
||||
|
||||
3. **Store usage only as raw vendor blobs, normalise later.** Rejected as the *whole* answer (kept as a
|
||||
fallback within the chosen one): a reader that must parse Anthropic's response shape to answer "what
|
||||
did this cost" has the vendor coupling the whole feature exists to remove. The raw blob is kept
|
||||
beside the normalised row (0050 already requires this), not instead of it.
|
||||
|
||||
4. **One vendor-neutral usage record at two consumer grains, produced by the adapter, recorded as
|
||||
both an event and a queryable row.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**Model usage is one vendor-neutral record — `(licence, consumer, period, metric, value)` plus the raw
|
||||
response — recorded at two grains that differ only in the `consumer`.** At the **licence grain** the
|
||||
consumer is the holding module and the metric is the vendor's own account reading (Anthropic:
|
||||
`utilization%`). At the **session grain** the consumer is the agent session — `(node, module)` and its
|
||||
session id ([ADR 0026](0026-the-mesh-has-a-session-of-its-own.md)) — and the metrics are the ones a
|
||||
session bills: input tokens, output tokens, model, and cost. The row shape is 0050's, unchanged; the
|
||||
grain is which consumer the row is *for*.
|
||||
|
||||
**The adapter is the only thing that knows the vendor, and it produces both grains.** Reading an
|
||||
account's cap is the adapter's `usage(licence)` verb ([ADR 0050](0050-model-access-is-vendor-agnostic.md));
|
||||
attributing a session's cost is the adapter reading that vendor's transcript or usage API and emitting
|
||||
rows keyed to the session. The mesh defines the row and the plumbing; the adapter fills it from whatever
|
||||
the vendor exposes, and a static-key vendor that exposes nothing simply produces no rows — usage is an
|
||||
optional reading, not a requirement of holding a licence.
|
||||
|
||||
**A reading is taken on a schedule, not on a request.** The licence-grain poll is a **scheduled
|
||||
container** ([ADR 0053](0053-a-step-that-runs-on-a-schedule.md)) the adapter runs on a cadence; the
|
||||
session-grain rows are produced as sessions progress, from the transcript the session already writes.
|
||||
Neither blocks anything: a poll that fails is a logged, retried scheduled run (0053's rule), and a
|
||||
session whose cost cannot yet be attributed is a row not yet written, never a session refused.
|
||||
|
||||
**Usage is recorded two ways, for two audiences.** Each reading is **emitted as an event**
|
||||
([ADR 0041](0041-events-are-a-relationship.md)) — an immutable "this was observed at this time" that the
|
||||
`audit-logger` already records, so the history of what an account did is in the audit trail by default,
|
||||
under nobody's special arrangement. And the **current** picture — the latest reading per
|
||||
`(licence, consumer, period, metric)` — is upserted into a **usage context store**, so "how close is
|
||||
this cap" and "what has this project spent this month" are a query, not a fold over the event log. The
|
||||
event is the record of what happened; the store is the answer to what is true now.
|
||||
|
||||
**Usage is not a credential, and is recorded in the clear.** The one thing the mesh must not read is the
|
||||
sealed key ([ADR 0050](0050-model-access-is-vendor-agnostic.md), the refresh-token carve-out aside). A
|
||||
token count and a cost are not secrets; they are the operator's own operational facts, recorded openly
|
||||
so they can be queried, audited, and charged against. This is the deliberate opposite of the credential
|
||||
rule, and stating it prevents a later reader assuming usage inherits the key's secrecy and hiding it
|
||||
from the person who is paying.
|
||||
|
||||
This is a decision and not a patch because it settles **where a model's usage lives and at what grain**,
|
||||
which every reader — a bill, a cap alarm, a per-project report — depends on, and because it closes the
|
||||
half [ADR 0050](0050-model-access-is-vendor-agnostic.md) explicitly left open, reusing the session
|
||||
([ADR 0026](0026-the-mesh-has-a-session-of-its-own.md)), the schedule
|
||||
([ADR 0053](0053-a-step-that-runs-on-a-schedule.md)), the event ([ADR 0041](0041-events-are-a-relationship.md))
|
||||
and the store ([ADR 0008](0008-a-context-owns-its-store.md)) the mesh already has rather than inventing a
|
||||
vocabulary beside them.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **A usage row is vendor-neutral and carries the raw beside it.** A unit test constructs an Anthropic
|
||||
`utilization%` reading and a token/cost reading and asserts both render to
|
||||
`(licence, consumer, period, metric, value)` with the vendor response preserved in the raw column;
|
||||
a reader that answers "what did this cost" touches only the normalised columns.
|
||||
- **The two grains differ only in the consumer.** A unit test records a licence-grain row (consumer =
|
||||
the module) and a session-grain row (consumer = a session id) for one licence and asserts both are
|
||||
the same shape and both are returned when the licence's usage is asked for, distinguishable by
|
||||
consumer.
|
||||
- **A poll is a scheduled run and its failure is not fatal.** The adapter's usage container declares a
|
||||
`schedule` ([ADR 0053](0053-a-step-that-runs-on-a-schedule.md)); a host test (0053's) already proves a
|
||||
scheduled step runs on cadence, does not gate, and logs rather than fails on a non-zero run — the poll
|
||||
inherits this and adds nothing to check.
|
||||
- **Each reading is an event the audit trail records.** An integration check asserts a usage reading
|
||||
emits an event that the `audit-logger` receives (it consumes `#`), so the history is present without
|
||||
the usage module and the audit module knowing about each other beyond the event.
|
||||
- **The current picture is a query.** A store test upserts two readings for one
|
||||
`(licence, consumer, period, metric)` and asserts the later replaces the earlier, so "what is true
|
||||
now" is one row, while the event log keeps both.
|
||||
- **A static-key vendor with no usage reading records nothing, and that is fine.** A test resolves a
|
||||
`static-key` model-access consumer whose adapter has no `usage` verb and asserts the licence works and
|
||||
no usage rows or poll are required — usage is optional, holding a licence is not conditioned on it.
|
||||
- **Usage is readable in the clear; the key is not.** A test asserts a usage row is stored unsealed and
|
||||
is returned to an ordinary query, while the licence key remains sealed and absent from the same
|
||||
surfaces — the deliberate inversion of the credential rule.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A bill and a cap alarm are both queries.** "What did project X spend this month" reads the
|
||||
session-grain rows; "how close is account Y to its cap" reads the latest licence-grain metric — both
|
||||
from the usage store, neither a fold over events or a call to the vendor.
|
||||
- **The session becomes the unit of cost, which is what it already is.** Because a session is the
|
||||
consumer, attributing cost needs no new identity — and the worker-granularity gap
|
||||
([design 14](../03-DESIGN/01-to-be/14-model-access.md)) surfaces here exactly as it does for access,
|
||||
to be closed once, for both, when a session id is threaded through.
|
||||
- **The adapter carries the vendor's usage quirks alone.** Anthropic's `utilization%`, its transcript
|
||||
shape, a mid-session account switch — all live in the Anthropic adapter; the mesh, the store, and
|
||||
every reader see only rows. A second vendor adds a second adapter and no new table.
|
||||
- **Usage history is durable and tamper-evident by reuse, not by a new mechanism.** It rides the event
|
||||
trail the mesh already keeps, so an operator who wants the whole history has it, and one who wants the
|
||||
current number has the store — without usage owning either mechanism.
|
||||
- **The refresh-token carve-out is untouched by this.** Usage is read *from* an authenticated adapter;
|
||||
it neither holds nor exposes the credential, so the one place the mesh reads what it stores
|
||||
([ADR 0050](0050-model-access-is-vendor-agnostic.md)) is not widened by recording what that credential
|
||||
was spent on.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0050](0050-model-access-is-vendor-agnostic.md) — model access is vendor-agnostic; fixes the
|
||||
usage row shape and the `usage(licence)` adapter verb, and leaves its home open — which this closes
|
||||
- [ADR 0026](0026-the-mesh-has-a-session-of-its-own.md) — the mesh has a session of its own; a session
|
||||
is the consumer the finer grain attributes to
|
||||
- [ADR 0053](0053-a-step-that-runs-on-a-schedule.md) — a scheduled step; the licence-grain poll is one
|
||||
- [ADR 0041](0041-events-are-a-relationship.md) — events are a relationship; a usage reading is one, and
|
||||
the audit-logger records it
|
||||
- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) — the shape of an event on the wire; the form a
|
||||
usage reading takes to reach the audit trail
|
||||
- [ADR 0008](0008-a-context-owns-its-store.md) — a context owns its store; the current usage picture
|
||||
lives in one
|
||||
- [ADR 0024](0024-model-access-is-a-provision.md) — model access is a provision; usage is a reading of
|
||||
what that provision was used for
|
||||
- [03-DESIGN/01-to-be/14-model-access.md](../03-DESIGN/01-to-be/14-model-access.md) — the model-access
|
||||
design; this fills its usage section and shares its open worker-naming gap
|
||||
- [03-DESIGN/01-to-be/15-the-agent-session.md](../03-DESIGN/01-to-be/15-the-agent-session.md) — the agent
|
||||
session; the consumer the session grain is keyed to
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-07
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0050-model-access-is-vendor-agnostic.md
|
||||
---
|
||||
|
||||
# 55. Model access is answered by a licence, or by a node that hosts the model
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0024](0024-model-access-is-a-provision.md) made model access a provision, and
|
||||
[ADR 0050](0050-model-access-is-vendor-agnostic.md) fixed what answers it: a record — a licence — that
|
||||
an adapter turns into a sealed vendor credential.** A consumer requires `model-access`, is put on a
|
||||
licence, and is delivered a key (a static API key, or an access token a manager refreshes). Every
|
||||
answer so far has been a credential to reach a vendor's API across the internet.
|
||||
|
||||
**But a model need not come from a vendor. A node in the mesh can host one.** An operator with a GPU
|
||||
runs Ollama or vLLM, which serves an OpenAI-compatible API on that node. A consumer that wants that
|
||||
model does not need a vendor credential — it needs the model server's **endpoint**: the base URL and
|
||||
the model name, and a key only if the server is configured to want one. This is model access answered
|
||||
by a **node**, not by a record.
|
||||
|
||||
**The mesh already knows how a node answers a provision — it is the ordinary provider/consumer path.**
|
||||
A provider `provides` a provision at a scope, `serves` its connection facts, and the mesh fills the
|
||||
consumer's bound facts with the provider's `at`/`port` and each served fact — exactly how a Postgres
|
||||
consumer learns where its database is. Nothing reserved `model-access` to records: a node offering
|
||||
`provides: ["model-access"]` resolves through this path, and the resolver already **prefers a local
|
||||
answer over a licence** — its own comment names the case, "a model the mesh runs itself." So the
|
||||
capability exists; what is missing is the decision to use it, and the statement of what a node-answer
|
||||
delivers and where it stops.
|
||||
|
||||
**A node-answer and a record-answer are the same provision with two shapes of answer.** This is the
|
||||
same move [ADR 0050](0050-model-access-is-vendor-agnostic.md) already made for shapes within the
|
||||
vendor path (static-key vs refreshable-grant): one provision, more than one way it is answered. A
|
||||
consumer written against `model-access` should not care whether the model behind it is a vendor's or
|
||||
the mesh's own — it asks for model access and is given what reaches a model.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A separate provision for the local case (`local-model`, `model-endpoint`).** A node answers that;
|
||||
`model-access` stays record-only. Rejected: it splits "where my model comes from" into two
|
||||
provisions a consumer must choose between in its manifest, when the mesh already models a
|
||||
record-answer and a node-answer to **one** provision. A consumer would have to know, at authoring
|
||||
time, whether its model will be a vendor's or the mesh's — the exact coupling the provision was
|
||||
meant to remove. It is the safer implementation (see the limitation below) but the worse interface.
|
||||
|
||||
2. **Model access answered by either a licence or a node, under the one provision.** A consumer
|
||||
requires `model-access`; the operator answers it with a licence (a vendor) or by assigning a
|
||||
node that hosts a model. Adopted: one interface, and the answer is an operator's deployment choice,
|
||||
not a consumer's authoring choice.
|
||||
|
||||
## Decision
|
||||
|
||||
**Model access is one provision answered two ways: by a licence (a record, turned into a sealed vendor
|
||||
credential by an adapter — [ADR 0050](0050-model-access-is-vendor-agnostic.md)) or by a node that hosts
|
||||
the model (a provider that serves an endpoint).** A consumer requires `model-access` and is delivered
|
||||
whichever the operator assigned; it does not name the kind.
|
||||
|
||||
**A node-answer delivers an endpoint, not a credential.** The provider `provides: ["model-access"]`
|
||||
and `serves` its connection facts — the port it listens on and the model it runs — and the mesh fills
|
||||
the consumer's bound facts with the provider node's `at`, the served `port`, and the served `model`,
|
||||
the same way every provider consumer learns where its provider is. The consumer assembles a base URL
|
||||
(`http://<at>:<port>/v1`) and points an OpenAI-compatible client at it. If the local server wants a
|
||||
key, the provider mints one the ordinary way (a per-consumer secret, sealed and host-unsealed); if it
|
||||
does not — the common Ollama case — the consumer lists `model-access` under `binds` and **not** under
|
||||
`secrets`, and no key is delivered. Secret delivery and fact delivery are already independent, so a
|
||||
keyless endpoint is expressed by asking for the facts and not a secret.
|
||||
|
||||
**A node-answer uses no adapter.** The adapter registry ([ADR 0050](0050-model-access-is-vendor-agnostic.md))
|
||||
is the vendor-credential machinery — accept-and-seal, refresh, usage. A node-hosted model has no vendor
|
||||
secret to seal; its endpoint is served, and its key (if any) is minted like any provider's. The adapter
|
||||
is consulted only for the record/vendor answer. So the vendor-agnostic decision is untouched, and the
|
||||
node-answer adds no vendor logic anywhere.
|
||||
|
||||
**The resolver prefers a local answer.** When a node's own set answers `model-access` — a model the
|
||||
mesh runs itself — a licence for it is not consulted. This is already the resolver's behaviour and is
|
||||
made a decision here: a mesh that runs a model uses it, and a licence is the answer for a consumer that
|
||||
has no local model, not a competitor to one that does.
|
||||
|
||||
**One limitation, stated so it is not found as a bug.** The local-preference above is exact for a
|
||||
**node-scope** provider co-located with its consumer, and for any node-answer in a mesh that holds no
|
||||
`model-access` licence. It is *not* yet exact for a **mesh-scope** model server — one node serving the
|
||||
model to others — **while a licence for `model-access` also exists in the same mesh**: the record pass
|
||||
that turns a licence into an answer keys on same-node satisfaction and would still demand the licence be
|
||||
used, double-answering. Until that pass is taught to stand down when a brokered node need already
|
||||
answers, a mesh-scope local model and a vendor licence must not both answer `model-access` in one mesh.
|
||||
A node-scope local model has no such constraint. This is named because an unstated limitation is
|
||||
indistinguishable from a bug, and costs more.
|
||||
|
||||
This is a decision and not a patch because it settles **what may answer model access** — a question
|
||||
every model-access consumer's meaning depends on — and because it lets the mesh's own hosted models sit
|
||||
behind the same provision as the vendors', which is what makes "the mesh can run its own model" a
|
||||
deployment choice rather than a second interface to build against.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **A node answers model access without a licence, and is preferred over one.** A resolver test
|
||||
assigns a consumer and a module that `provides: ["model-access"]` at node scope on the one node, with
|
||||
a licence also present, and asserts no resolved need is answered by the record — the local model
|
||||
answers and the licence is ignored. (This test exists; the decision adopts what it proves.)
|
||||
- **The consumer is delivered an endpoint, not a credential.** A mesh bed assigns a model-server
|
||||
provider and a consumer that binds `model-access` and does not list it under `secrets`, and asserts
|
||||
the consumer's config carries `OPENAI_BASE_URL` built from the provider's served `at`/`port`, and
|
||||
that no key file was delivered to its secret path.
|
||||
- **The local endpoint is reachable through what the consumer was given.** The bed makes a request to
|
||||
the base URL the consumer wrote and asserts the model server answers — the wiring, not a model's
|
||||
output, is what is proven (the server may be a stub; a real model is not needed to prove the mesh
|
||||
routed the consumer to it).
|
||||
- **A node-answer consults no adapter.** A node-answered `model-access` need is resolved with the
|
||||
vendor registry never read — asserted by the absence of any vendor on a node-answered need and the
|
||||
ordinary served-facts delivery.
|
||||
- **The scope limitation holds where stated.** The node-scope case is what the bed and the resolver
|
||||
test exercise; the mesh-scope-plus-licence collision is recorded here and left for the resolver
|
||||
change that reconciles the two answer passes, not worked around in a module.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The mesh can run its own model, and a consumer reaches it through the same `model-access` it uses
|
||||
for a vendor.** One interface, two answers; a consumer moves between a vendor and a local model by an
|
||||
operator reassigning its provision, not by a code change.
|
||||
- **A local model is keyless by default and keyed by the ordinary path when it must be.** Nothing new
|
||||
is invented for the local server's credential: it either has none, or mints one the way every
|
||||
provider does.
|
||||
- **The vendor path is untouched.** Adapters, the refresh carve-out, and usage
|
||||
([ADR 0054](0054-model-usage-is-recorded-at-two-grains.md)) are the record answer's business; a
|
||||
node-answer neither uses nor changes them. A local model that exposes usage would serve it as facts,
|
||||
not as an adapter's usage verb.
|
||||
- **The two answer passes meet in one place, and must be reconciled there.** The mesh-scope limitation
|
||||
is the single point where a node-answer and a record-answer to the one provision can collide; it is
|
||||
named, and its fix is a resolver change, not a per-module workaround.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0024](0024-model-access-is-a-provision.md) — model access is a provision; this decides a node
|
||||
may answer it, not only a record
|
||||
- [ADR 0050](0050-model-access-is-vendor-agnostic.md) — model access is vendor-agnostic; the record
|
||||
answer and its adapters, which the node answer sits beside and does not use
|
||||
- [ADR 0054](0054-model-usage-is-recorded-at-two-grains.md) — model usage; a vendor's business on the
|
||||
record path, served as facts (if at all) on the node path
|
||||
- [ADR 0008](0008-a-context-owns-its-store.md) — a context owns its store; a licence is a record in one,
|
||||
a node-answer needs none
|
||||
- [03-DESIGN/01-to-be/14-model-access.md](../03-DESIGN/01-to-be/14-model-access.md) — the model-access
|
||||
design, which this extends with the node answer
|
||||
@@ -0,0 +1,115 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-09
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0007-connectivity.md
|
||||
---
|
||||
|
||||
# 66. Public routing is name-agnostic, its names are resolved inside the mesh, and an internal authority can certify them
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0007](0007-connectivity.md) and [connectivity §3](../03-DESIGN/01-to-be/08-connectivity.md)
|
||||
made a public route a grant: a workload that must be reachable requires a route, the proxy provides
|
||||
it, the consumer contributes the name it wants and the port it listens on.** What was never pinned
|
||||
is **what that name is** — and building a whole mesh in the lab showed the gap costs more than it
|
||||
looks.
|
||||
|
||||
**The catalogue shipped each route as a full domain.** A module that needed a public name carried
|
||||
that name, in full, as a literal in its manifest. Running the same catalogue against a different
|
||||
domain — a lab standing in for production, or a second operator's mesh — meant overriding that
|
||||
literal on every routed module, per node. The mesh was, in effect, carrying a **map of names to
|
||||
services**: the one thing it should never hold, because a name is the operator's choice (one runs
|
||||
the forge at `git`, another at `code`) and the domain is the node's, and neither is the mesh's to
|
||||
know.
|
||||
|
||||
**And a second gap surfaced the moment an internal issuer tried to certify those names.**
|
||||
[Connectivity §5](../03-DESIGN/01-to-be/08-connectivity.md) already states the issuer must be
|
||||
configurable and that the lab runs its own ACME authority. With that authority wired to the proxy,
|
||||
issuance still could not complete: the authority accepted the order and offered a challenge, then
|
||||
**could not connect to the validation target.** Nothing inside the mesh resolved the public route
|
||||
name. The mesh publishes each `<node>.internal` name into every container, but not the public names
|
||||
the proxy serves — so a validator living in the mesh had no address to reach, and a name the mesh
|
||||
cannot resolve is a name it cannot have certified.
|
||||
|
||||
**The two are one problem.** A name the mesh can *compose* from parts it is given, and *propagate*
|
||||
to whoever needs to resolve it, is exactly a name it can also have *certified* — and the reverse:
|
||||
without the composition and the propagation, neither the routing nor the certificate is the
|
||||
operator's to move between meshes.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. Keep the full domain in the manifest, override per node.** The status quo. It works, and it is
|
||||
wrong in the specific way this repository cares about: the catalogue holds a domain map, lab and
|
||||
production differ by an override on every routed module rather than one fact, and a module manifest
|
||||
names something — the public domain — that belongs to the node, not the module. An unowned name in
|
||||
the wrong place is the shape of a leak.
|
||||
|
||||
**2. A module declares a label; the node declares its public domain; the mesh composes.** The route
|
||||
contribution carries a subdomain the operator chose, the node carries its public domain as
|
||||
node-level configuration, and the mesh joins `<label>.<public-domain>` and grants exactly that. The
|
||||
mesh interprets nothing. Lab-versus-production becomes one node setting. Chosen.
|
||||
|
||||
**3. For certification, issue only from a publicly reachable node against a public authority.**
|
||||
This is already true for public meshes and stays true. It is not an option for a lab or an
|
||||
internal-only mesh: there is no public authority to answer, and no public reachability to validate
|
||||
against. An internal issuer is required there — and an internal issuer must be able to *validate*,
|
||||
which it cannot do unless the routed name resolves and is reachable **inside** the mesh. So the
|
||||
resolution gap is not optional to close; it is what makes an internal authority possible at all.
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh core holds no map of hostnames, subdomains or domains.** A route contribution carries a
|
||||
**label** (the subdomain) chosen by the module's operator. A node contributes its **public domain**
|
||||
as node-level configuration. The mesh composes `<label>.<public-domain>`, grants exactly that name,
|
||||
and never interprets what it means. One operator's forge at `git.example.tld` and another's at
|
||||
`code.other.example` are the same module with two facts supplied around it.
|
||||
|
||||
**When the proxy is granted a name, the mesh publishes that name → the node that serves it into
|
||||
internal resolution, mesh-wide** — the same mechanism, and the same "given by the mesh, not chosen
|
||||
by a module," that already writes `<node>.internal` into every declared container
|
||||
([connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md)). The mesh propagates the names it was
|
||||
told to serve. It still knows nothing about what any of them mean.
|
||||
|
||||
**An internal authority certifies those names by the same path a public one would.** The proxy is
|
||||
pointed at whichever issuer the mesh names — a public ACME authority, or an internal one — and
|
||||
trusts that issuer's root; nothing else about issuance changes. The internal authority validates by
|
||||
reaching the routed name, which the clause above has just made resolvable inside the mesh. So the
|
||||
three are one decision: **compose the name, propagate it, certify it** — each is meaningless without
|
||||
the one before it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Lab-versus-production is one node-level `public-domain` setting**, not an override on every
|
||||
routed module. The same catalogue runs against any domain.
|
||||
- **The manifest layer needs composition it does not yet have.** Today a route name is stored as a
|
||||
literal, with no interpolation of a node's domain into a module's label. Until that exists, the
|
||||
composed name is produced by a per-node settings override — a stopgap that reproduces option 1's
|
||||
per-module cost and is explicitly *not* the design.
|
||||
- **An internal issuer depends on route-name resolution.** Its challenge validates against the
|
||||
routed name; without that name in internal resolution, issuance for it cannot complete inside the
|
||||
mesh. The lab found this as a live failure, not a theory.
|
||||
- **Nothing about the public path changes.** A publicly reachable node issuing a public name from a
|
||||
public authority is untouched; this widens the same shape to names and meshes that are not public.
|
||||
|
||||
**How each is checked** — an unenforced rule is indistinguishable from a wrong one:
|
||||
|
||||
- **Name-agnostic:** the same catalogue resolves against two different public domains by changing
|
||||
one node setting and nothing else; and no module manifest contains a full public domain. A
|
||||
manifest that pins an FQDN is the smell the check looks for.
|
||||
- **Resolution:** a request to a routed name, made from inside the mesh, reaches the workload that
|
||||
serves it — and, the sharper check, issuance for that name against the internal authority
|
||||
completes, which it cannot unless the validator resolved and reached the target.
|
||||
- **Internal authority:** a TLS handshake to a routed name verifies against the internal root and
|
||||
nothing else, the same shape §5 already uses for internal node-to-node names.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0007 — connectivity](0007-connectivity.md), which made a route a grant and named exposure,
|
||||
resolution and certificates as one context.
|
||||
- [ADR 0009 — modules and the graph](0009-modules-and-the-graph.md), the provide/require/contribute
|
||||
vocabulary a route and an authority both use.
|
||||
- [Connectivity design §2, §3, §5](../03-DESIGN/01-to-be/08-connectivity.md), amended alongside this
|
||||
record.
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-10
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 67. Genesis is a pivot: a temporary control plane installs the registry that makes it permanent
|
||||
|
||||
## Context
|
||||
|
||||
**The mesh builds its own modules into its own registry, and there is no public registry for them
|
||||
— that is the point, not an omission.** A module is cloned from the forge, built, and published
|
||||
where the mesh can move, replace and back it up. Nothing about that arrangement wants a copy of the
|
||||
mesh's code hosted by somebody else.
|
||||
|
||||
**Every image must be pinned by digest** ([ADR 0006](0006-the-substrate-and-the-control-plane.md)),
|
||||
and the reasoning is exactly right: a bundle is applied where no mesh exists to check anything
|
||||
against anything, so what it names must be exact.
|
||||
|
||||
Those two sentences are individually correct and together they close a door. The digest a pin means
|
||||
is a *manifest* digest, and a manifest digest is **assigned by a registry when something is pushed
|
||||
to it**. The control plane's image is built from source and pushed nowhere, so it has no such
|
||||
digest, so it cannot be named — and a registry cannot be installed without a control plane to
|
||||
install it. That is not a pin. It is a dependency the pinning rule created by accident.
|
||||
|
||||
**It went unnoticed because the lab hid it.** The lab raised a disposable registry, stocked it from
|
||||
a workstation, and rewrote every image reference to point at it — so the lab bootstrapped along a
|
||||
path no real machine has. A first node in the lab always worked, and a first node anywhere else had
|
||||
no path at all. Every bootstrap fault found this year was found late for the same reason: **the
|
||||
install procedure existed only as a test fixture**, and a fixture is free to invent what it needs.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. Publish the mesh's own images to a public registry.** Rejected on the premise: there is no
|
||||
public registry for the mesh's modules and there is not meant to be. It would also make raising a
|
||||
mesh depend on somebody continuing to host its code, which is the dependency the whole arrangement
|
||||
exists to remove.
|
||||
|
||||
**2. Build the control plane from source on the first machine.** Rejected. A bare machine would
|
||||
need a toolchain and a working tree — and worse, the source lives in a forge **that runs on the
|
||||
mesh**. A total rebuild would then need the mesh it is rebuilding. Acceptable for adding a node to
|
||||
a healthy mesh; useless for the case that matters.
|
||||
|
||||
**3. Keep a disposable registry as an install step.** Rejected. It exists in no production, and
|
||||
concealing this problem is precisely what it has been doing.
|
||||
|
||||
**4. Pivot through a temporary control plane.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**An image may be named by the digest of its own configuration.** A bare `sha256:…` names an image
|
||||
the machine already holds — content-addressed, immutable, unforgeable, and requiring nothing to
|
||||
have served it. It satisfies what the pinning rule asks for; the rule simply never contemplated an
|
||||
image that no registry had ever seen. It is legal exactly where nothing could have served one.
|
||||
|
||||
**Genesis is a pivot**, in this order:
|
||||
|
||||
1. the installer **carries the control-plane image** and loads it onto the machine
|
||||
2. a **temporary** control plane is raised from it, named by that image's own digest
|
||||
3. the **registry module is installed** — its image is upstream and it is never built, which
|
||||
[`04-ISSUES/029`](../04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md)
|
||||
already settled: a module that provides the artifact store cannot be delivered through it
|
||||
4. the control-plane image is **pushed into the mesh's own registry**, which assigns it a manifest
|
||||
digest — the first one it has ever had
|
||||
5. the control plane is **reinstalled as an ordinary module** pinned to that digest
|
||||
|
||||
**The host performs the replacement, not the control plane.** Tier 0 outlives tier 2: the control
|
||||
plane composes a declaration naming the registry-pinned image, and the host applies it and recreates
|
||||
the container. Nothing is asked to replace itself while running, and the control plane is stateless
|
||||
— what it knows is in the store.
|
||||
|
||||
**The installer is tier 0, and a separate program from the host.** Bootstrapping is by hand and
|
||||
changes the machine, which is tier 0's definition. But the host states that it *connects to nothing
|
||||
and listens on nothing*, and that claim is what makes the one thing running forever on every
|
||||
machine auditable. An installer connects to plenty. Same tier, same delivery, different program.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The control plane stops being a special case.** It becomes an ordinary module with an ordinary
|
||||
image in the mesh's own registry — so the mesh can build and roll out **its own upgrades**, which
|
||||
is what a mesh that runs itself was always reaching for.
|
||||
- **The bundle's job shrinks** to raising a temporary control plane exactly once.
|
||||
- **The source builds the installer; it does not run it.** Cloning moves to a release machine, where
|
||||
a forge being available is an ordinary working assumption, and leaves the disaster-recovery path
|
||||
where it very much is not.
|
||||
- **The lab's disposable registry is deleted.** The lab bootstraps by running the same program a
|
||||
bare machine runs — the only arrangement in which the installer cannot quietly drift out of truth
|
||||
again.
|
||||
- **Two public images remain at genesis** — the store and the broker. An air-gapped install would
|
||||
embed those too, at a much larger artifact; that is a build variant, not a different design.
|
||||
- **A control-plane module manifest must exist**, and did not.
|
||||
- **The registry may require nothing.**
|
||||
[`04-ISSUES/029`](../04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md)
|
||||
states that a module providing the artifact store may not *build* artifacts, because there is
|
||||
nowhere to put them until it runs. The pivot shows that is the narrow case of a wider rule: **it
|
||||
may not require anything the store is needed to deliver.** Found the hard way — an unrelated
|
||||
change gave the registry a public name and, with it, a route requirement. At genesis nothing
|
||||
provides a route, and nothing can, because the routing stack needs images and images need the
|
||||
store. The same cycle, re-entered through a door the existing wording did not cover.
|
||||
- **The handover is the sharp edge.** For one moment the bundle and the module both describe the
|
||||
same container, and the host tracks what it owns. If a safe handover is not expressible with what
|
||||
exists, the install **stops before it** and says what is missing. A machine left without a control
|
||||
plane cannot be fixed remotely, so a partial install that halts cleanly is the better outcome.
|
||||
|
||||
**How each is checked** — an unenforced rule is indistinguishable from a wrong one:
|
||||
|
||||
- **Naming by its own digest:** a machine that can reach no registry at all raises a control plane.
|
||||
- **The pivot completed:** after installing, the running control plane's image is pinned by a digest
|
||||
**the mesh's own registry assigned** — not by an image id. If it is still the image id, the pivot
|
||||
did not happen and the mesh cannot upgrade itself.
|
||||
- **No fiction left in the lab:** the scenario declares no registry machine, and the bed bootstraps
|
||||
through the installer rather than around it.
|
||||
- **The handover:** a machine whose control plane has been replaced still has one, and it answers.
|
||||
- **The registry requires nothing:** its manifest is resolvable on a mesh that has no other module
|
||||
in it. A requirement added to it later is caught where it is written, rather than by a genesis
|
||||
that cannot complete — which is how this one was found.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0006 — the substrate and the control plane](0006-the-substrate-and-the-control-plane.md),
|
||||
which pinned images by digest and named what a first node fetches.
|
||||
- [ADR 0005 — the node host](0005-the-node-host.md), which makes tier 0 the one thing installed by
|
||||
hand and the only thing that changes a machine — the property this keeps true by shipping the
|
||||
installer beside the host rather than inside it.
|
||||
- [`04-ISSUES/029`](../04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md),
|
||||
the same cycle one layer down, and the rule that a registry module is named and never built.
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
topic: building it
|
||||
status: proposed
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0016-the-lab.md
|
||||
---
|
||||
|
||||
# 68. The lab takes requests, one at a time, and runs each from its own copy
|
||||
|
||||
## Context
|
||||
|
||||
**The lab is exclusive hardware, and today a person holds it.** Raising a scenario takes over
|
||||
addresses and names on the workstation for as long as it stands, and only one scenario can stand
|
||||
at a time. So a run is not merely slow — it occupies the machine and the person who started it,
|
||||
who then waits rather than works.
|
||||
|
||||
**Running it in the background against the working copy is worse than waiting.** The obvious fix
|
||||
is to start a run and carry on editing. But a run reads the working copy as it goes: binaries are
|
||||
rebuilt from it, manifests are read out of it, and the bed's own code is loaded from it. Edit
|
||||
while it runs and the result describes a state that never existed — a mixture of what was there
|
||||
when each file happened to be read. A green result obtained that way is not evidence, and a red
|
||||
one costs a day to disbelieve.
|
||||
|
||||
**Nothing today records what was asked for.** A run is a command line in somebody's terminal. What
|
||||
commit it exercised, what it was trying to find out, and what it answered all live in scrollback,
|
||||
which is why the same question gets re-run rather than looked up.
|
||||
|
||||
**Most of the parts already exist.** The lab writes a receipt of its last run. The mesh already
|
||||
carries messages between nodes and can notify a person. The machine already runs work on a
|
||||
schedule. What is missing is the thing in the middle.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. Leave it as it is — a person drives the lab and waits.** Rejected. It is the loop
|
||||
[ADR 0010](0010-delivery.md) removed everywhere else, kept here by habit rather than by argument,
|
||||
and the cost compounds: because a run is expensive to start and blocks the person, fewer are run,
|
||||
so faults are found later and in larger batches.
|
||||
|
||||
**2. Run in the background against the working copy.** Rejected on the reasoning above. The
|
||||
failure is silent, which is the kind this repository exists to refuse.
|
||||
|
||||
**3. Put the lab behind the ordinary build pipeline.** Rejected for now. The pipeline builds
|
||||
artifacts and does not own a machine that can raise virtual machines; giving it one makes the
|
||||
pipeline's slowest job the lab's, and couples every push to hardware only one machine has. This
|
||||
may become right later; it is not the smallest thing that works.
|
||||
|
||||
**4. A queue in front of the lab, and an isolated copy behind it.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**The lab accepts requests rather than commands.** A request is recorded, queued, and answered.
|
||||
The person who made it is told when it is answered and does not wait.
|
||||
|
||||
**A request names a bed and a commit, and nothing else.** This is the load-bearing restriction. A
|
||||
request may say *run this bed, at this version of these repositories*. It may not say what to
|
||||
install, on which machine, or with which settings — because a request that could say those things
|
||||
would be a second way of installing a mesh, and the whole reason the installer exists is that the
|
||||
lab already was one ([ADR 0067](0067-genesis-is-a-pivot.md)). The bed decides what is installed;
|
||||
the request only decides which bed and which version.
|
||||
|
||||
**Requests are released one at a time.** The hardware admits one standing scenario, so the queue
|
||||
enforces what the hardware already requires, rather than leaving it to whoever remembers.
|
||||
|
||||
**Every run happens in a copy the lab owns.** The lab checks the requested commit out into its own
|
||||
path and builds and runs from there. A working copy is never read by a run. This is what makes the
|
||||
queue safe to use while work continues, and without it the rest of this record is not worth
|
||||
having.
|
||||
|
||||
**The lab is reached through tools, not only a command line.** A command line is available only
|
||||
to whoever is sitting at the machine, which is the constraint this record exists to remove. The
|
||||
lab answers three questions to anything that can reach the mesh — *what is standing now*, *what is
|
||||
queued or running*, and *what did this request answer* — and accepts a request and a cancellation.
|
||||
An agent can therefore start a run, stop attending to it, and come back; and somebody who did not
|
||||
start a run can still see it, which is the difference between a shared lab and a private one.
|
||||
|
||||
**The restriction holds at every door.** A tool submits a bed and a commit, exactly as a command
|
||||
line does. A tool that could name a module, a node or a setting would reintroduce the second
|
||||
installer through a different entrance, and the entrance is not what made it dangerous.
|
||||
|
||||
**Every run leaves a record that outlives the terminal**: what was asked, which commit, when it
|
||||
ran, what it answered, and where its output went. A question already answered is looked up rather
|
||||
than re-run.
|
||||
|
||||
## Consequences
|
||||
|
||||
Work continues while the lab runs, which is the point. A second session may edit freely, because
|
||||
nothing it edits is what the lab is reading.
|
||||
|
||||
A request is reproducible by construction: it names a commit, so the same request can be asked
|
||||
again and compared. Today two runs of "the same thing" are only as alike as the tree happened to be.
|
||||
|
||||
The lab gains a second copy of every repository it exercises, costing disk and needing to be kept
|
||||
from drifting into a place people edit by hand.
|
||||
|
||||
Anything that can reach the mesh can now see what the lab is doing, including an agent working on
|
||||
something else. That is the intended gain and also the obvious hazard: a thing that is easy to ask
|
||||
is easy to ask too often, and the hardware still admits one scenario at a time.
|
||||
|
||||
The queue becomes a thing that can fail — stuck, backed up, or lost — and a queue nobody watches
|
||||
is worse than no queue, because it absorbs requests silently.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A run never reads a working copy | The runner is given a path it owns and no other; a run started while a working copy is deliberately dirtied produces a result matching the commit, not the edits. |
|
||||
| One scenario stands at a time | A second request submitted while one runs is observed to wait, not to raise. |
|
||||
| A request cannot say what to install | The request format admits a bed and a commit only. A request naming a module, a node or a setting is refused, and the refusal is exercised. |
|
||||
| A request is answered | Every queued request reaches a terminal state with a record. A request that vanishes is a failure of the queue, not a quiet nothing. |
|
||||
| The lab can be asked from elsewhere | What is standing is asked from a session that did not raise it, and the answer matches the machine. A lab that only answers its own caller has not left the terminal. |
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 69. A module is a repository and a path within it
|
||||
|
||||
## Context
|
||||
|
||||
**The builder clones one repository and reads `module.json` at its root.** The to-be design says
|
||||
so in as many words — *"one file at the root"* — and the code implements it: clone, read the root
|
||||
manifest, build what it declares.
|
||||
|
||||
**Nothing that exists is shaped that way.** The catalogue holds sixty-seven modules, each in its
|
||||
own directory, and has no manifest at its root. None of the five code repositories has one either.
|
||||
So today the builder cannot be asked to build any module that exists: pointed at the catalogue it
|
||||
finds no manifest, and pointed at a module's source it finds no manifest.
|
||||
|
||||
**The system being replaced already works the other way**, and has for years: a monorepo with one
|
||||
directory per piece of software, and the coordinator builds a module from a repository and a path
|
||||
inside it. The root-only assumption is not a simplification of that — it is a different model that
|
||||
was never reconciled with it.
|
||||
|
||||
**And it splits what a build needs into two places.** The control plane's manifest sits in the
|
||||
catalogue; the source it describes sits in the control plane's own repository. A build must read
|
||||
one tree, so under the root-only model neither location can be built from.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. One repository per module.** Rejected. Sixty-seven repositories for sixty-seven modules, most
|
||||
of which are a single manifest naming a public image, and every one needing its own creation,
|
||||
permissions and lifecycle. It also contradicts [ADR 0015](0015-applications-live-in-their-own-repository.md),
|
||||
which put *applications* in their own repositories precisely because modules do not need one.
|
||||
|
||||
**2. Keep manifests in the catalogue and source elsewhere, and have a build fetch both.** Rejected.
|
||||
A build would clone two trees whose versions can disagree, so "what commit is this module?" stops
|
||||
having one answer — and that question is the whole basis of knowing when to rebuild.
|
||||
|
||||
**3. A module is a repository and a path within it.** Chosen. It is what the current system does,
|
||||
what the catalogue already looks like, and it keeps a module's description beside the thing it
|
||||
describes.
|
||||
|
||||
## Decision
|
||||
|
||||
**A module is named by a repository and a path within it.** The path holds `module.json`, and
|
||||
everything that manifest declares is produced from that path. A module whose path is the root is
|
||||
the ordinary case of this, not a separate one.
|
||||
|
||||
**A module's manifest lives beside its source.** Where a module has code, its directory holds both,
|
||||
so one commit answers "what is this module, and what is it made of". Where a module has no source —
|
||||
a manifest naming a public image — the directory holds only the manifest, and there is nothing to
|
||||
build.
|
||||
|
||||
**This moves the core modules.** The control plane and the builder are built from the control
|
||||
plane's repository, so their manifests belong in that repository at their own paths, not in the
|
||||
catalogue. The catalogue keeps the modules whose source it holds, and the modules that are only a
|
||||
manifest.
|
||||
|
||||
**One commit, one module version.** Because a module is one path in one repository, the commit that
|
||||
built it identifies it exactly, and "the source has moved ahead of what the mesh holds" stays a
|
||||
question with a yes or no answer.
|
||||
|
||||
## Consequences
|
||||
|
||||
The builder gains a path alongside the repository and the ref. A build is `repository, path, ref`,
|
||||
and the manifest it returns is the module the mesh records.
|
||||
|
||||
The catalogue stops being the place every manifest lives, and becomes the place manifests live
|
||||
*when their module has no other home*. That is a smaller claim than it sounds: most of the
|
||||
sixty-seven stay exactly where they are.
|
||||
|
||||
Two repositories change shape — the control plane's gains manifests for the modules built from it.
|
||||
Nothing else moves.
|
||||
|
||||
A repository can hold modules that are built and modules that are not, and no rule distinguishes
|
||||
them beyond whether their manifest declares anything to build.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A module is buildable from its repository and path | The builder is asked for a module by repository and path, and returns a manifest whose artifacts are pinned to digests the mesh's registry assigned. |
|
||||
| A manifest sits beside what it describes | A module declaring something to build, whose path holds no source to build it from, is refused at build time rather than producing an empty result. |
|
||||
| One commit identifies one module | Two builds of the same repository, path and commit produce the same digests. |
|
||||
| The core modules are built like any other | The control plane is rebuilt from its own repository and path, and the running mesh is upgraded to it — the same path an ordinary module takes. |
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0067-genesis-is-a-pivot.md
|
||||
---
|
||||
|
||||
# 70. The catalogue owns the module graph, and genesis builds rather than carries
|
||||
|
||||
## Context
|
||||
|
||||
**The module graph has no owner.** What modules exist, what each requires and provides, what each
|
||||
claims, what each is made of — all of it lives inside the control plane because that is where it
|
||||
was first written, not because anything decided it belonged there.
|
||||
|
||||
**The control plane's own test says it does not belong there.**
|
||||
[`06-the-control-plane`](../03-DESIGN/01-to-be/06-the-controller.md) defines the tier as
|
||||
*everything that needs to know about more than one node*, and states the corollary plainly:
|
||||
anything a single machine could answer alone is not the control plane's. What a module is, and
|
||||
what it needs, requires no knowledge of any node whatsoever.
|
||||
|
||||
**And nothing can query it.** The graph is the thing that answers *what must be rebuilt when this
|
||||
changes*, *what would break if this were removed*, and *what can be installed here* — and today it
|
||||
is reachable only as control-plane internals. [ADR 0009](0009-modules-and-the-graph.md) already
|
||||
decided that a build edge is derived, that an artifact is stale when anything it was built against
|
||||
moved, and that the rebuild set is therefore computable. None of that has anywhere to live.
|
||||
|
||||
**Genesis currently carries an image, and cannot produce a builder at all.**
|
||||
[ADR 0067](0067-genesis-is-a-pivot.md) has the installer carry the control plane's image. That
|
||||
works, but it leaves the builder with no route onto a fresh mesh — it cannot be fetched from the
|
||||
public internet, because the mesh builds it, and the installer carries one image only. So a raised
|
||||
mesh cannot build anything, including the modules it is made of.
|
||||
|
||||
## Decision
|
||||
|
||||
**The catalogue is a core module, beside the control plane and the builder, and it owns the module
|
||||
graph.** Modules, what they require and provide, what they claim, their dependencies and their
|
||||
build edges, assignments and configuration — the catalogue holds them and serves tools over them:
|
||||
install a module, query what is available and what it needs, query the graph, query and update
|
||||
settings.
|
||||
|
||||
**It is one per mesh**, expressed the way the control plane already expresses it — a claim scoped
|
||||
to the mesh, not a new mechanism.
|
||||
|
||||
**The control plane consumes it.** Resolving what a module requires into an actual binding, and
|
||||
composing what a machine should be, both need to know what modules are. So the dependency runs
|
||||
from the control plane to the catalogue, which is the opposite of what the tiers suggest and is
|
||||
therefore written down here rather than left to be inferred.
|
||||
|
||||
**Genesis builds the core modules rather than carrying them.** The installer ships an *init
|
||||
builder* — the one thing carried — which is started, clones the source, and builds the control
|
||||
plane, the catalogue and the builder. The installer then raises a temporary control plane, which
|
||||
installs the catalogue, registers the permanent control plane and the builder, and assigns all
|
||||
three to the first machine. The temporary control plane stops; the installer verifies that the
|
||||
permanent one answers and can query the graph. The builder then sees a catalogue in its initial
|
||||
state and builds the core modules into it.
|
||||
|
||||
**So exactly one thing is carried, and it is a builder rather than a result.** That is the
|
||||
difference from [ADR 0067](0067-genesis-is-a-pivot.md), which carried the control plane's image:
|
||||
carrying a builder produces every core module on the machine, including the builder itself, so
|
||||
there is no component left without a route.
|
||||
|
||||
## Consequences
|
||||
|
||||
The catalogue joins the small set of things that cannot arrive through the ordinary path, because
|
||||
it cannot be installed by something that needs it in order to install anything. It arrives the same
|
||||
way everything else does under this record — built by the init builder before the mesh can install
|
||||
anything — so the set is answered by one mechanism rather than three special cases.
|
||||
|
||||
The rebuild fan-out gains a home. *What was this built against* and *what must rebuild now* are
|
||||
questions about the graph, and the graph now has an owner to hold the edges and answer them.
|
||||
|
||||
The catalogue holds state, so it owns a store in the substrate's database, the same way the control
|
||||
plane's contexts do. That is the mesh's own store and not the `postgres` module, which is a
|
||||
provider other modules consume.
|
||||
|
||||
A mesh without a catalogue cannot resolve anything, where previously it merely lacked an interface.
|
||||
That is the cost of ownership over surfacing, and it is deliberate: one owner beats two copies.
|
||||
|
||||
## Open, and to be settled before this is built
|
||||
|
||||
**Where the init builder clones from.** [ADR 0067](0067-genesis-is-a-pivot.md) rejected building
|
||||
from source at genesis partly because the source lives in a forge that runs on the mesh, so a
|
||||
total rebuild would need the mesh it is rebuilding. Carrying a builder answers the toolchain half
|
||||
of that objection and not this half. Genesis must therefore name a source that exists before the
|
||||
mesh does.
|
||||
|
||||
**What the init builder publishes into.** Building produces artifacts that must be pinned by a
|
||||
digest a registry assigned, and today the registry is installed after the machine has joined. If
|
||||
the core modules are built first, the registry has to exist first, so the substrate's order needs
|
||||
restating rather than assumed.
|
||||
|
||||
**Where the line falls between the catalogue and the control plane.** Claims and assignments need
|
||||
to know about every node, which by the control plane's own test is its work. Whether the catalogue
|
||||
holds them and asks, or the control plane holds them and the catalogue surfaces them, is not
|
||||
settled here.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The catalogue owns the graph | The control plane answers *what does this module require* by asking the catalogue, and a mesh whose catalogue is stopped cannot resolve — observed, not assumed. |
|
||||
| One per mesh | A second catalogue assigned anywhere in the mesh is refused by the claim, and the refusal is exercised. |
|
||||
| Genesis builds rather than carries | The installer carries exactly one artifact, and after installing, every core module is pinned to a digest the mesh's own registry assigned. |
|
||||
| The builder has a route | A mesh raised by the installer, with no hand-placed image, can build a module. |
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0070-the-catalogue-owns-the-module-graph.md
|
||||
---
|
||||
|
||||
# 71. Genesis clones from a mesh, and checks what it got
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0070](0070-the-catalogue-owns-the-module-graph.md) has the init builder clone the source,
|
||||
and does not say from where.** [ADR 0067](0067-genesis-is-a-pivot.md) had already rejected building
|
||||
at genesis partly for that reason: the forge holding the source runs *on* the mesh, so a total
|
||||
rebuild would need the mesh it is rebuilding.
|
||||
|
||||
That objection is real but narrower than it reads. It only binds when the mesh being raised and the
|
||||
mesh holding the source are the same one, which is true exactly once.
|
||||
|
||||
## Decision
|
||||
|
||||
**Genesis clones from a mesh's forge, reached by name.** Any mesh that holds the source can serve
|
||||
it. The first mesh is not structurally special — it is simply the only one that existed when there
|
||||
was nothing else to clone from.
|
||||
|
||||
**If the mesh serving the source is lost, the name moves to another mesh that holds a copy.**
|
||||
Recovery is a name pointing somewhere else, not a backup being restored. This is what makes the
|
||||
source's survival a property of there being more than one mesh, rather than a property of somebody
|
||||
having remembered to take a copy. A mesh that has installed from that name holds the source
|
||||
afterwards, so every installation adds a place the name could point.
|
||||
|
||||
**Genesis names a commit and checks what it got.** It does not clone whatever a branch happens to
|
||||
point at. The forge a mesh installs from is the trust anchor for everything that mesh will ever
|
||||
run, and a branch is a moving target that somebody else controls.
|
||||
|
||||
*This is not hypothetical.* On 2026-09-11 the forge that would serve this role was running a
|
||||
cryptominer, and its git operations were being tampered with in flight — output injected into the
|
||||
protocol stream by a hook that fired on every fetch. Nothing was altered: the repositories were
|
||||
verified against local copies and found byte-identical. But a mesh installing from that name during
|
||||
those hours had no way to establish that for itself, and would have had none.
|
||||
|
||||
## Consequences
|
||||
|
||||
The init builder needs a name it can resolve and a commit it can verify, and nothing else. It does
|
||||
not need to know which mesh answers.
|
||||
|
||||
Whoever operates the mesh that name points at carries a responsibility to everyone installing from
|
||||
it, and should know that. It is not merely a convenience host.
|
||||
|
||||
A mesh that cannot reach any forge cannot be raised. That is a real limit and it is accepted: the
|
||||
alternative is carrying the whole source in the installer, which makes the installer a release
|
||||
artifact that goes stale rather than a program that fetches what it was told to.
|
||||
|
||||
## Open — what relationship a mesh keeps afterwards
|
||||
|
||||
**Not decided, and named here so it is not decided by accident** by whoever writes the init
|
||||
builder. Two shapes, and they are meaningfully different:
|
||||
|
||||
**A snapshot, and then independence.** A mesh installs once, mirrors the source into its own forge,
|
||||
and has no upstream afterwards. It is fully self-hosted, in the sense that nothing it needs lives
|
||||
anywhere else. Updates are then something an operator does deliberately, by pulling changes in —
|
||||
tooling for which is possible and is not a priority.
|
||||
|
||||
**A continuing upstream for core modules**, the way a distribution serves packages and a separate
|
||||
collection serves everything else. A mesh keeps looking at the origin for the modules that make a
|
||||
mesh a mesh, and holds its own for the rest.
|
||||
|
||||
The first is more obviously aligned with the rest of this design, which is arranged so nothing a
|
||||
mesh needs depends on somebody else continuing to host it. The second is more convenient and makes
|
||||
a security problem in one forge everybody's problem. Neither is chosen here.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| Genesis needs only a name and a commit | A mesh is raised with the name pointed at a different mesh than the last time, and the result is identical. |
|
||||
| What was cloned is what was asked for | Genesis is pointed at a commit and refuses a forge serving different content under it, rather than building what it received. |
|
||||
| Losing the serving mesh is survivable | The name is repointed at a mesh that installed from it earlier, and a raise succeeds. |
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0070-the-catalogue-owns-the-module-graph.md
|
||||
---
|
||||
|
||||
# 72. Two graphs, and a build chain that orders itself
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0070](0070-the-catalogue-owns-the-module-graph.md) gave the catalogue the module graph and
|
||||
said the control plane consumes it — that resolving a requirement and composing what a machine
|
||||
should be *"both need to know what modules are, so the dependency runs from the control plane to
|
||||
the catalogue."*
|
||||
|
||||
**That paragraph is wrong, and this record corrects it.** It was written before the two graphs had
|
||||
been told apart, and it creates a dependency that does not need to exist: a catalogue that is down
|
||||
would leave the control plane unable to compose the declaration that would repair it.
|
||||
|
||||
## Decision
|
||||
|
||||
**There are two graphs, with different owners, and they meet only when something is installed.**
|
||||
|
||||
| Graph | Owner | What it links | Answers |
|
||||
|---|---|---|---|
|
||||
| the module graph | the **catalogue** | module-versions to each other | what was this built against · what must rebuild now · what does this need |
|
||||
| the runtime graph | the **control plane** | module-versions to nodes | what runs where · who consumes this provision · what breaks if this machine goes |
|
||||
|
||||
The catalogue does not know nodes exist. The control plane holds module-versions and nodes, along
|
||||
with capabilities and claims, because deciding whether a machine qualifies needs every node.
|
||||
|
||||
**So the control plane never asks the catalogue anything.** It holds what it needs to compose a
|
||||
declaration. A catalogue that is down stops new installs and stops the rebuild fan-out, and does not
|
||||
touch anything already running or the ability to repair it.
|
||||
|
||||
**Everything between them travels as events over the broker**, like all module-to-module
|
||||
communication. The builder finishes and announces that a module was built. The catalogue registers
|
||||
it, places it among what it depends on, and announces that a module was upgraded. The control plane
|
||||
reacts to *that*, not to build output — a semantic fact rather than an artifact.
|
||||
|
||||
**Nothing is lost if a receiver is down.** A consuming module's queue is durable with a dead-letter
|
||||
exchange, declared by the mesh rather than by the module, so an event waits for a consumer that is
|
||||
not there.
|
||||
|
||||
**Build order is not computed. It emerges from the chain.** The builder never consults the graph: it
|
||||
builds what it is asked for, one at a time. The catalogue asks for the next build after the previous
|
||||
registration, so *"do not start this until that is registered"* holds by construction rather than by
|
||||
a schedule somebody maintains.
|
||||
|
||||
**The catalogue's rule is a condition, not an order:** ask for a module to be rebuilt once everything
|
||||
it was built against is current. That handles a chain and a diamond with one rule, where an
|
||||
order-based approach needs to know the shape in advance.
|
||||
|
||||
**Two refusals belong to the catalogue.** A cycle, because the chain would never settle. And a
|
||||
rebuild whose artifacts are identical to what it replaced, which is not an upgrade and must not be
|
||||
announced as one — or a single change ripples outward forever through modules that did not change.
|
||||
|
||||
**Genesis does none of this.** The init builder has a fixed, short list — control plane, catalogue,
|
||||
builder — in a written order, because there is no catalogue yet to ask.
|
||||
|
||||
## Consequences
|
||||
|
||||
The builder stays simple, and independent of the catalogue. That is what makes genesis possible at
|
||||
all: the thing that builds the catalogue cannot require the catalogue.
|
||||
|
||||
The two sides can be briefly out of step — the catalogue may hold a module-version a moment before
|
||||
the control plane knows of it. Assigning in that instant fails, and should say why rather than
|
||||
report that no such module exists.
|
||||
|
||||
A module's declared events stop being documentation and become its permissions: an account is
|
||||
scoped from what a module emits and consumes, so a builder that announces what it built is granted
|
||||
what it needs by the ordinary mechanism rather than by a special case.
|
||||
|
||||
## Open
|
||||
|
||||
**Whether an upgrade is applied or merely noticed.** Today the system this replaces deploys
|
||||
automatically, and that is a defensible default for core modules on a mesh its operator runs. But
|
||||
the design as it stands does the opposite: it records that the source moved ahead, makes it visible,
|
||||
and waits to be told. This must become a setting with a chosen default rather than inherited
|
||||
behaviour — and the choice matters most on the day a bad commit reaches something that carries mail.
|
||||
|
||||
**Whether a module assigned to several machines upgrades on all of them at once.** Doing so makes
|
||||
one bad commit simultaneous everywhere. Doing one machine and pausing turns it into one casualty.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-13
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0070-the-catalogue-owns-the-module-graph.md
|
||||
---
|
||||
|
||||
# 73. The installer carries a builder, and the registry stays where it is
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0070](0070-the-catalogue-owns-the-module-graph.md) decided that genesis builds rather than
|
||||
carries, and [ADR 0071](0071-where-genesis-gets-its-source.md) settled where it clones from. Two
|
||||
questions were left open, and the design record names them as the one gap that stops a fresh mesh
|
||||
from being able to produce anything at all: **how the builder arrives**, and **what it publishes
|
||||
into**.
|
||||
|
||||
Today the installer carries the control plane's image inside itself. That works, and it is why
|
||||
genesis needs no registry: nothing is ever fetched, because the one image that matters is already
|
||||
present. The cost is that the mesh which results holds an artifact it did not make, cannot rebuild,
|
||||
and knows nothing about — no version, no source, no edges. That is the same shape as the fault
|
||||
[issue 044](../04-ISSUES/044-the-runtime-every-module-builds-on-cannot-be-built-by-the-mesh/00-report.md)
|
||||
recorded for the shared runtime, and fixing it there while shipping it here on every new mesh would
|
||||
be a strange place to stop.
|
||||
|
||||
## Decision
|
||||
|
||||
**The installer carries a builder, and nothing else.** One artifact, not a growing set. It clones
|
||||
the source at a named commit, checks what it got ([ADR 0071](0071-where-genesis-gets-its-source.md)),
|
||||
and produces the control plane from the same repository and path that any later rebuild of it would
|
||||
use. What raises the mesh is therefore the same thing that will maintain it, and there is no second
|
||||
mechanism kept in step with the first.
|
||||
|
||||
**The registry does not move, and the argument for moving it does not survive being made.**
|
||||
|
||||
It was put this way: a produced image has to be put somewhere before anything can fetch it, so the
|
||||
registry must now precede the control plane, and
|
||||
[ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md)'s answer to that question has to flip.
|
||||
|
||||
It does not, because the premise is false. **The thing that builds the image and the machine that
|
||||
runs it are the same machine.** A built image is already in that machine's container runtime, and
|
||||
the temporary control plane names it exactly as it names a carried one — by the digest of its own
|
||||
configuration, a local identity that requires nothing to have served it. Building changes where the
|
||||
bytes came from. It does not change where they are.
|
||||
|
||||
| | is it substrate? | must it precede the control plane? |
|
||||
|---|---|---|
|
||||
| the store | yes | yes — there is nowhere else to put the control plane's state |
|
||||
| the broker | yes | yes — the control plane reaches a machine only over it |
|
||||
| the image registry | yes — it cannot grant itself a repository | **still no** — the first machine neither fetches the control plane nor needs to, whether the image was carried in or made here |
|
||||
|
||||
So the registry stays where [ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md) put it:
|
||||
substrate by role, ordinary by delivery, installed by the temporary control plane as its first act.
|
||||
The bundle carries two services and a control plane, as it did. **What publishes into the registry
|
||||
is unchanged too** — the existing step that pushes the control plane's image into it, which is the
|
||||
moment that image first receives a digest assigned by something other than itself. It now pushes
|
||||
something this mesh built rather than something it was handed.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Genesis gains one step and changes no others.** A build happens before the image is loaded. The
|
||||
pivot described in [ADR 0067](0067-genesis-is-a-pivot.md) survives exactly as written, because the
|
||||
step it pivots on never cared where the image came from.
|
||||
|
||||
**A fresh mesh can produce from the moment it exists.** The builder is present before the control
|
||||
plane is, so the core modules, the catalogue and the builder's own module can be built in the
|
||||
ordinary way rather than waiting for somebody to carry them in. The paragraphs in
|
||||
[`17-raising-a-mesh`](../03-DESIGN/01-to-be/17-raising-a-mesh.md) that describe this were describing
|
||||
something that could not start; they can start now.
|
||||
|
||||
**Genesis needs more of the outside world.** Carrying an image needed nothing but the installer.
|
||||
Building one needs the source, and whatever the build itself reaches for. This is a real cost and
|
||||
is not waved away: it makes genesis fail in more ways, all of them at a step that says what it was
|
||||
doing. It is accepted because the alternative is a mesh that cannot rebuild its own control plane,
|
||||
which fails in exactly one way, silently, later, and for ever.
|
||||
|
||||
**A pre-built bundle remains possible and is not this.** Nothing here forbids delivering artifacts
|
||||
rather than building them; it fixes where they may come from. A bundle of pre-built core modules is
|
||||
an **export of a mesh that built them**, carrying what the catalogue knows about each alongside the
|
||||
artifact itself — so that loading one leaves the graph in the state building would have left it. A
|
||||
bundle that carries images without that is the thing this decision rejects, whoever ships it.
|
||||
|
||||
## What this does not decide
|
||||
|
||||
**Whether the builder's own module is carried or built.** It builds everything else; what installs
|
||||
*it* as an ordinary module afterwards, so that it too can be upgraded, is the same closed-list
|
||||
question [`12-a-module-repository`](../03-DESIGN/01-to-be/12-a-module-repository.md) already holds,
|
||||
and is unchanged by this.
|
||||
|
||||
**How a machine authenticates to a registry that asks it to.** Genesis raises its own and reaches it
|
||||
over the loopback, so this remains a joining problem
|
||||
([issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md)).
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-15
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0039-what-the-sdk-holds-and-refuses.md
|
||||
---
|
||||
|
||||
# 74. The mesh defines a module protocol; an SDK is an implementation of it
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0039](0039-what-the-sdk-holds-and-refuses.md) says what the SDK holds: the tool-serving
|
||||
harness, the messaging and event framework, the contracts, and core primitives. It settles what
|
||||
belongs in *an* SDK. It does not say what happens when there is more than one.
|
||||
|
||||
There is already more than one. **The contracts are expressed twice** — as Go types in the control
|
||||
plane and the host, and as TypeScript types in the SDK — and nobody has felt it because both live
|
||||
in one repository and one head.
|
||||
|
||||
**A correction, made after inspecting the wire rather than the types** (2026-09-16). This record
|
||||
first claimed the two implementations already disagreed — `resource` vs `Provision`, `consumer`
|
||||
meaning the module in one and the node in the other, headers declared on one side and emitted by
|
||||
neither. **On inspection the live wire agrees**, and the claim was wrong:
|
||||
|
||||
- The grant types that disagreed (`Grant`, `Interface`, `Credential` in the SDK's `contracts`)
|
||||
were **dead** — exported and imported by nothing. The live provisioning wire is the contributions
|
||||
file, whose shape (`as`, `secret`, `node`, `at`, `values`) is the same on both sides. Those dead
|
||||
types have been removed.
|
||||
- The envelope agrees too: Go emits all five required headers, and `x-causation-id`/`x-schema` are
|
||||
**optional** — the SDK sets them when a handler has a causation or a schema, and a bare event
|
||||
carrying neither is correct, not a drift.
|
||||
|
||||
So the danger was never live disagreement. It was **dead types that contradicted the live wire**,
|
||||
which read as the contract and were not — and are exactly what led this record to assert a drift
|
||||
that inspection did not find. That is a sharper reason for the decision below, not a weaker one: a
|
||||
type is only as good as its being the wire, and the way to guarantee that is to specify the wire and
|
||||
check implementations against it, rather than to trust a hand-kept type to still describe it.
|
||||
|
||||
A failure of this kind does not announce itself. Two implementations that disagree about an
|
||||
envelope do not fail to compile — they ignore each other's messages, and a mesh where a module
|
||||
stops reacting looks exactly like a mesh where nothing happened.
|
||||
|
||||
## The question this settles
|
||||
|
||||
A module may be written in any language the mesh can build
|
||||
([`18-building-a-module`](../03-DESIGN/01-to-be/18-building-a-module.md)). Every language needs an
|
||||
SDK. What is an SDK *of*?
|
||||
|
||||
Two answers were available, and the obvious one is wrong.
|
||||
|
||||
**Shared types, generated.** Write the shapes once — a schema, an IDL — and generate Go, TypeScript,
|
||||
Rust. It is the familiar answer and it solves the smaller half of the problem. The shapes are not
|
||||
where the difficulty is.
|
||||
|
||||
**A specified wire, with a conformance suite.** The shapes are a consequence; what an SDK must get
|
||||
right is *behaviour*.
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh defines a module protocol. An SDK is an implementation of that protocol in one
|
||||
language, and nothing more.**
|
||||
|
||||
That is the whole of what an SDK is. Not a library a language happens to have, not a convenience
|
||||
layer, not a place for helpers to accumulate — an implementation of a specified protocol, finished
|
||||
when it implements it and correct when it agrees with every other implementation.
|
||||
|
||||
### The protocol is split per capability
|
||||
|
||||
**A module does not use all of it, so an SDK need not implement all of it.** A module that only
|
||||
consumes events uses the event capability. One that serves tools uses the tool capability. A
|
||||
provider uses provisioning. Nothing about consuming an event requires knowing how a grant is
|
||||
answered.
|
||||
|
||||
So the protocol is a floor plus capabilities:
|
||||
|
||||
| part | what it covers | who needs it |
|
||||
|---|---|---|
|
||||
| **connection** — the floor | reading the sealed credential, pinning the certificate fingerprint, taking identity from the credential rather than the environment | everything |
|
||||
| **events** | the envelope and its headers, the durable per-consumer queue, binding, at-least-once with dedup on `x-event-id` | a module that emits or consumes |
|
||||
| **tools** | registration, the shared durable `serve.<key>` queue, request and reply | a module with a surface |
|
||||
| **provisioning** | a grant in, a credential out, and what each carries | a module that provides something |
|
||||
|
||||
**This is the same shape the host already has.** A host declares which resource kinds it can apply,
|
||||
and a partial host — one that can write files and run things but not manage users or containers —
|
||||
is a real thing rather than a broken one ([ADR 0005](0005-the-node-host.md)). An SDK that implements
|
||||
the floor and events is exactly as legitimate, and a module written against it is a module that
|
||||
does events.
|
||||
|
||||
**So a language arrives in pieces rather than all at once.** A Rust SDK implementing connection and
|
||||
events is useful the day it exists; tools and provisioning follow when something needs them. The
|
||||
alternative — a language is unsupported until it is entirely supported — is what makes adding one a
|
||||
project rather than a contribution.
|
||||
|
||||
**And what a language can be used for is then a fact the mesh can state**, rather than something an
|
||||
author discovers by writing a module that cannot be built: the toolchain list says which languages
|
||||
exist, and the conformance results say what each can do.
|
||||
|
||||
### What the specification covers
|
||||
|
||||
Per capability, what two implementations can disagree about:
|
||||
|
||||
- **the exchanges and queues** — which exchanges exist, that a consumer's queue is durable and
|
||||
named `<node>.<module>.events`, that a tool is served from a shared durable `serve.<key>`
|
||||
- **the envelope** — every header, which are required, what an unknown `x-` header means, and that
|
||||
ignoring one is correct rather than lax
|
||||
- **identity** — that a module's node and module name come from its sealed credential and not from
|
||||
its environment, so what it emits matches what the mesh authorised
|
||||
- **delivery** — at-least-once, and that dedup is on `x-event-id`, which only the emitter can make
|
||||
- **the credential** — the sealed document's fields, and that a connection pins a certificate
|
||||
fingerprint rather than trusting an authority
|
||||
- **provisioning** — a grant in, a credential out, and what each carries
|
||||
- **the vocabulary** — that `consumer` is one thing, named once
|
||||
|
||||
### Conformance is per capability
|
||||
|
||||
**A suite per part, and an SDK claims the parts it passes.** A monolithic pass/fail would make a
|
||||
partial implementation indistinguishable from a broken one, which is the distinction this is built
|
||||
on.
|
||||
|
||||
**And the suite is executable, not prose.** A specification nobody can run is a document two
|
||||
implementations drift from while both believe they conform. Conformance is a set of fixtures — an
|
||||
emitted event, a served tool call, a grant and its answer — that every SDK must produce and consume
|
||||
byte-for-byte.
|
||||
|
||||
**The existing two implementations are the first two to be made to pass it.** Not a future language:
|
||||
the drift above is present, and a suite that only new SDKs must satisfy would leave the disagreement
|
||||
that already exists in place while certifying everything added afterwards against it.
|
||||
|
||||
## Why not generated types
|
||||
|
||||
Generation makes the shapes agree and leaves everything that matters unspecified. Two SDKs
|
||||
generated from one schema can still name their queues differently, take identity from the
|
||||
environment, dedup on the wrong field, or omit a header the other requires — and every one of those
|
||||
is a mesh that runs and quietly does not work.
|
||||
|
||||
It also makes the contract into whatever the generator supports, which is a decision nobody made
|
||||
about a boundary everything else depends on.
|
||||
|
||||
**The shapes are worth generating once the wire is specified.** That is a convenience, and it comes
|
||||
second.
|
||||
|
||||
## Consequences
|
||||
|
||||
**A language is a commitment, and now a divisible one.** Adding one means implementing the protocol
|
||||
and passing the suites for the parts it claims. That is more work than transliterating types, and
|
||||
it is the work that was always there — the difference is that it can be finished, and finished in
|
||||
pieces, rather than believed.
|
||||
|
||||
**Versioning becomes possible.** `x-schema` exists for it and is never written. A specified envelope
|
||||
with a version on the body is what lets a mesh hold a module built against an older SDK, which is
|
||||
the ordinary state of any mesh that has been running for a while.
|
||||
|
||||
**The two current implementations agree on the live wire** — inspection showed it. What was wrong
|
||||
was a set of dead types beside the wire, now removed. The suite's job here is therefore prevention:
|
||||
to keep that agreement true as the wire changes, and to hold a new language's SDK to it, rather than
|
||||
to repair a break that exists today.
|
||||
|
||||
**This does not make the mesh polyglot by itself**, and should not be reported as though it does. It
|
||||
makes polyglot possible to do correctly. A Rust SDK is still a Rust SDK.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| One vocabulary | A word means one thing across implementations, checked by the fixtures using it. |
|
||||
| The wire is what is specified | Both existing SDKs run the conformance suite in their own test suites, and a change to one that breaks a fixture fails there rather than in a mesh. |
|
||||
| A new SDK is a passing SDK | A language is not listed as buildable for a capability until its SDK passes that capability's suite; the toolchain list and the conformance results name the same set. |
|
||||
| A partial SDK is a real thing | An SDK implementing the floor and one capability passes, is listed for that capability, and a module using another is refused with the reason — rather than failing at runtime in a language nobody said was finished. |
|
||||
| An unknown header is ignored | A fixture carries one, and every implementation accepts it. |
|
||||
| Identity comes from the credential | A fixture sets an environment that disagrees with the credential, and the emitted event carries the credential's. |
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-15
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0014-no-npm-workspace.md
|
||||
---
|
||||
|
||||
# 75. An artifact store is a provision; a package registry is a different one
|
||||
|
||||
## Context
|
||||
|
||||
Two questions have been circling, and they turn out to be one question asked twice.
|
||||
|
||||
**"Should gitea be the mesh's registry?"** It serves OCI images and a dozen package ecosystems, it
|
||||
is already needed — genesis clones from one — and the mesh's own registry has neither
|
||||
authentication ([issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md))
|
||||
nor a transport a runtime will accept over a network
|
||||
([issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)). Gitea
|
||||
has both.
|
||||
|
||||
**"Where does the SDK come from?"** [ADR 0014](0014-no-npm-workspace.md) already answers it — each
|
||||
module consumes its dependencies, the mesh's own shared library included, *from the private
|
||||
registry* — and nothing installs one, so today it comes from a git URL, which is
|
||||
[issue 053](../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md).
|
||||
|
||||
**The framing that dissolves both:** `artifact-store` is already a provision, and `registry`
|
||||
already provides it. So "should gitea be the registry" is not a question about replacing a
|
||||
component. It is a question about **a second provider of an existing provision** — which this mesh
|
||||
has a mechanism for, and uses for certificate authorities and VPNs already.
|
||||
|
||||
## Decision
|
||||
|
||||
**Two provisions, because they are two jobs.**
|
||||
|
||||
| provision | is | for |
|
||||
|---|---|---|
|
||||
| `artifact-store` | content-addressed blobs, pinned by digest, no versions, no ranges | what the **mesh** delivers to **machines** |
|
||||
| `package-registry` | an ecosystem's own registry — npm, cargo, PyPI, Go | what **code** resolves when it is compiled |
|
||||
|
||||
They are not the same store with different clients. One is addressed by digest and immutable by
|
||||
construction; the other is addressed by name and version, and resolves ranges. Conflating them is
|
||||
how a mesh that pins everything ends up rebuilding one commit into two different things.
|
||||
|
||||
**`registry` remains the provider genesis installs.** Not because it is better, but because of what
|
||||
it is: a directory and one container, no database, no control plane, installable at step 8 of an
|
||||
install where neither exists yet. Gitea needs a store and provisioning, which means a control plane,
|
||||
which means the pivot has already happened — and the pivot needs somewhere to publish to.
|
||||
|
||||
**Gitea also provides `artifact-store`, and a mesh may choose it.** Two providers of one provision
|
||||
is a thing the mesh understands: it refuses, names both, and choosing is assigning the one you want.
|
||||
|
||||
**And it does not claim `the-artifact-store`.** That claim is node-scoped, so a module holding it
|
||||
cannot share a machine with another that does — and a machine running gitea for git and packages
|
||||
*alongside* a registry serving artifacts is an ordinary arrangement, not a conflict. They are
|
||||
different ports doing different jobs.
|
||||
|
||||
The exclusivity that matters is mesh-wide and is already expressed: `provides` at mesh scope means
|
||||
two providers are two answers, and the resolver refuses until one is assigned. Forbidding
|
||||
co-residence adds nothing to that and forbids something reasonable. **Whether `registry` should
|
||||
still hold that claim is left open here** — it may be protecting something about the port or the
|
||||
data directory that is not written down, and removing a claim is not a thing to do from the outside
|
||||
of a manifest.
|
||||
A mesh that assigns gitea gets authentication and TLS for its artifacts — which is to say, **issues
|
||||
042 and 048 are answered by choosing a provider that already solved them**, rather than by
|
||||
reimplementing accounts and certificates in a registry that has none.
|
||||
|
||||
**Gitea provides `package-registry`.** That is ADR 0014's private registry, and it is one service
|
||||
rather than one per ecosystem. `verdaccio` may provide it too, for npm alone, and is then a choice
|
||||
somebody makes rather than the answer.
|
||||
|
||||
**The registry is not removed at the end of installing.** A mesh that never runs gitea still has an
|
||||
artifact store. Retiring it is a migration a mesh performs, not a step an installation ends with.
|
||||
|
||||
## Why not simply gitea, from the start
|
||||
|
||||
Because genesis would need a control plane before the thing that stores the control plane's image,
|
||||
and that is circular rather than merely awkward. It would also make one of the three things the
|
||||
build loop cannot produce for itself into a stateful application with a database — the pivot is the
|
||||
hardest part of this design already.
|
||||
|
||||
And it puts every artifact in the service that is also the trust anchor for everything the mesh will
|
||||
ever run ([ADR 0071](0071-where-genesis-gets-its-source.md)), which records that forge serving a
|
||||
cryptominer with tampered git operations. Two blast radii are better than one.
|
||||
|
||||
## Moving from one provider to the other is a designed act
|
||||
|
||||
**Not a removal.** Every image a machine runs is pinned to a digest at a named store, the control
|
||||
plane's own included. Changing the provider means:
|
||||
|
||||
1. gitea installed, reachable, and holding an account the builder may publish with
|
||||
2. every artifact mirrored
|
||||
3. every declaration re-pinned, the control plane's **last**, because it is what performs the others
|
||||
4. **every machine verified to have converged and to be able to pull from the new store**
|
||||
5. only then the old provider unassigned, and its volume kept ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md))
|
||||
|
||||
Step 4 is the one that is easy to skip and the only thing between this and a mesh that cannot
|
||||
restart its own control plane. A machine that reboots mid-migration pulls from a store that no
|
||||
longer exists, and a local image cache hides that until exactly the moment it matters.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The bootstrap is unchanged**, which is the point of keeping the small provider.
|
||||
|
||||
**042 and 048 gain a second answer.** They can be fixed in the registry, or dissolved by choosing a
|
||||
provider that already has accounts and TLS. The second is less work and more service.
|
||||
|
||||
**ADR 0014 becomes satisfiable.** There is a provision for the private registry, something that
|
||||
provides it, and a module may depend on it — so the SDK can be published and consumed rather than
|
||||
cloned, and issue 053 has somewhere to go.
|
||||
|
||||
**A mesh can be minimal or complete, and both are legitimate.** One with the small registry and no
|
||||
gitea builds and runs modules and cannot serve packages. That is a real configuration, not a broken
|
||||
one — the same way a partial host is real.
|
||||
|
||||
**And the bootstrap still has no package registry.** The first build of the shared base happens
|
||||
before anything has installed one. That is the same pivot as everything else and it is **not solved
|
||||
here**: it is named, so the next person does not discover it.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| Genesis needs no database | The installer raises a mesh of one on a machine with nothing, and the artifact store it installs has no store of its own. |
|
||||
| Two providers are a choice, not a conflict | A mesh holding both is asked to resolve `artifact-store` and refuses, naming both, until one is assigned. |
|
||||
| Providers may share a machine | A node is assigned both gitea and a registry, and both run — only one of them answers `artifact-store`. |
|
||||
| The two stores are not interchangeable | A module depending on `package-registry` is not satisfied by `artifact-store`, and the refusal says why. |
|
||||
| A migration is verified before it is finished | The old provider cannot be unassigned while any machine's declaration still names it. |
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-16
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0075-two-stores-and-which-provides-what.md
|
||||
---
|
||||
|
||||
# 76. The SDK is a published package, and the toolchain resolves it by version
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0014](0014-no-npm-workspace.md) decided a module consumes its dependencies — the mesh's own
|
||||
shared library included — from the private registry. [ADR 0075](0075-two-stores-and-which-provides-what.md)
|
||||
decided the private registry is a `package-registry` provision, and that gitea provides it. What
|
||||
neither settled, and what [issue 053](../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md)
|
||||
left open, is the one build where the rule cannot simply be obeyed: **the first one.**
|
||||
|
||||
The TypeScript toolchain image is built *from* the SDK — it carries the SDK so that every module
|
||||
compiled inside it resolves the shared library without each build fetching it. So the thing that
|
||||
compiles TypeScript and the thing that contains the SDK were the same object, and that object
|
||||
cannot be what builds the SDK. Stated as a question — "how does the SDK reach the registry before
|
||||
the toolchain exists, when the toolchain is what builds it?" — it reads as a paradox.
|
||||
|
||||
It is not one. The paradox exists only because the toolchain *bakes a git-cloned copy* of the SDK.
|
||||
The SDK itself is plain TypeScript: it needs `node` and `tsc` and nothing the mesh makes. A public
|
||||
base image can build it. The circularity is a property of the workaround, not of the SDK.
|
||||
|
||||
## Decision
|
||||
|
||||
**The SDK is an ordinary published package in the mesh's `package-registry`, consumed by version.**
|
||||
The git URL in the toolchain's manifest and the sibling-path lock beside it — the two halves of
|
||||
issue 053 — are both removed. A build resolves the SDK the way it resolves any dependency, with a
|
||||
lock that agrees with its manifest, so `npm ci` is the command and reproducibility is by
|
||||
construction rather than by the machine the build ran on.
|
||||
|
||||
**The SDK is built with a public base image, not with the mesh's toolchain.** It is *not* one of
|
||||
the components the loop cannot build — the control plane, the registry, the builder, the catalogue
|
||||
([`12-a-module-repository`](../03-DESIGN/01-to-be/12-a-module-repository.md)), which arrive by
|
||||
carrying an init builder because they are the loop's own machinery. The SDK is machinery for
|
||||
nothing; it is an ordinary dependency the loop builds and publishes like any other. The only
|
||||
constraint is narrow: it cannot be compiled *in the mesh toolchain*, because that toolchain is built
|
||||
from it. So it is compiled on a public base image instead — which needs nothing the mesh makes — and
|
||||
published before the toolchain that consumes it. It is not carried, because building it does not
|
||||
wait on a mesh existing first.
|
||||
|
||||
**The toolchain base stays, thinned.** mesh-tools remains the image bundles are compiled in and the
|
||||
one place the SDK is resolved — but it `npm ci`s the SDK by version from the registry instead of
|
||||
baking a copy cloned from a git URL. Bundles keep borrowing its resolved dependencies; what changes
|
||||
is that the version they borrow is named and honest. This was the shape chosen over dropping the
|
||||
shared base entirely and having every bundle resolve the SDK itself: one resolution point, one
|
||||
place to be right about the version.
|
||||
|
||||
**Genesis orders the publish before the first compile.** The package-registry provider is a public
|
||||
image (gitea), so it comes up needing no toolchain; the SDK is published into it; only then is the
|
||||
toolchain built, so the first `npm ci` has a registry to read from. Nothing in that chain is
|
||||
circular, because the only thing that needed the toolchain — baking the SDK — is gone.
|
||||
|
||||
## Consequences
|
||||
|
||||
Each language's toolchain repeats the shape: its own SDK, built from that language's public base
|
||||
image, published to the same registry, resolved by version with that ecosystem's lockfile-honest
|
||||
install (`npm ci`, `cargo` against a vendored or registry source, `pip` against a pinned set). The
|
||||
warning in issue 053 — that whatever the TypeScript repository does the others will copy — is
|
||||
answered by making the copied thing the correct one.
|
||||
|
||||
A change to the SDK is publish-then-consume, exactly as [ADR 0014](0014-no-npm-workspace.md) already
|
||||
priced it: publish the new SDK version, then bump the toolchain (and any module pinning it directly)
|
||||
to consume it. There is no shortcut that resolves an unpublished SDK, which is the property that was
|
||||
missing.
|
||||
|
||||
mesh-tools is no longer an SDK carrier in the sense that mattered — it does not contain a copy
|
||||
whose provenance is a branch head somebody force-pushes. It contains a version.
|
||||
|
||||
A mesh with no package-registry cannot build TypeScript. This is accepted and is not new: it is the
|
||||
same shape as a mesh that cannot reach a forge being unable to be raised
|
||||
([ADR 0071](0071-where-genesis-gets-its-source.md)). Installing brings the registry up first.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The SDK a build compiles against is named, not cloned from a branch | The toolchain manifest pins `@novox/mesh-sdk` to a version, and the build runs `npm ci`, which refuses a lock that disagrees with the manifest. Issue 053's two checks become this one. |
|
||||
| The SDK builds without the mesh's own toolchain | The SDK's build recipe names a public base image. A recipe that named the mesh toolchain would reintroduce the cycle and is refused in review. |
|
||||
| The registry is up before the first compile | The genesis bed asserts the package-registry answers, and the SDK is published, before the base build runs. A base build that ran first would fail its `npm ci` with no registry, which is the positive control. |
|
||||
| A second language repeats the shape, not a new one | When a second SDK is added, its recipe is compared to this one: public base, publish by version, lockfile-honest install. |
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-16
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 77. The parts are named controller, foundation, node — not control plane, substrate, master
|
||||
|
||||
## Context
|
||||
|
||||
The words drifted. In conversation and in code the same thing was called *control plane*,
|
||||
*controller*, *master*, and *hub*; the store-and-broker pair was called *substrate* and
|
||||
*foundation*; a machine was a *node*, a *worker-node*, a *peer*, a *slave*. A mesh named
|
||||
differently by two people is a mesh they describe differently, and the drift was worst on the
|
||||
parts talked about most.
|
||||
|
||||
Three of the terms carried wrong ideas. *Control plane* is borrowed from networking's
|
||||
control-plane/data-plane split and means nothing here. *Master/slave* and *hub/peer* imply a
|
||||
subordinate — but no node is: a node applies its own declaration and keeps running when the
|
||||
control-node dies, so it is as much its own machine as any other. *Substrate* is a biology
|
||||
metaphor that landed for no one.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep the inherited words.** Rejected: they are the source of the drift, and two of them
|
||||
(control plane, substrate) are metaphors that teach the wrong shape to anyone reading them cold.
|
||||
2. **master / slave, or hub / peer, for the nodes.** Rejected: both name a hierarchy the mesh does
|
||||
not have. The control-node owns no other node; lose it and the rest keep running what they were
|
||||
last told.
|
||||
3. **controller / foundation / node + control-node.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
The component that decides what each node should be, holds the mesh's records, and tells nodes is
|
||||
the **controller** — the module `mesh-controller`, which claims the mesh-scoped `the-controller`
|
||||
seat. The store and broker raised at genesis are the **foundation**. Machines are **nodes**;
|
||||
there are 0..n of them, and exactly one — the one running the controller — is the **control-node**.
|
||||
|
||||
Retired: *control plane*, *substrate*, *master/slave*, *hub/peer*, *worker-node*.
|
||||
[`00-META/glossary.md`](../00-META/glossary.md) is the authority, and a new name for an existing
|
||||
thing lands there in the change that introduces it in code.
|
||||
|
||||
## Consequences
|
||||
|
||||
`mesh-control` became `mesh-controller` across the module, container, image, binary, `cmd/` dir and
|
||||
the git repository; `substrate` became `foundation` in the embedded base bundles, the default
|
||||
template and the example lock; the seat `the-control-plane` became `the-controller`. The 03-DESIGN
|
||||
prose and 00-META follow the new words.
|
||||
|
||||
What got harder: the records under `02-DECISIONS/` are immutable, so they keep the words they were
|
||||
written with — this record included, whose own title names what it retires. A term retired here
|
||||
still appears there, and the glossary is how to read it. The git repository on the forge was
|
||||
renamed `mesh-control` → `mesh-controller`.
|
||||
|
||||
## References
|
||||
|
||||
- [`00-META/glossary.md`](../00-META/glossary.md) — one name per thing, and the words retired.
|
||||
- The rename shipped across all six code repositories and hq (main).
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-16
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0033-the-substrate-is-a-store-and-a-broker.md
|
||||
---
|
||||
|
||||
# 78. The store and the broker are ordinary modules
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md) settled that the foundation is a store
|
||||
and a broker, raised at genesis; [ADR 0006](0006-the-substrate-and-the-control-plane.md) settled
|
||||
that the controller cannot grant itself either, because it consumes them and is not running yet to
|
||||
ask. Both were raised as bundle resources — plumbing, with no record in the mesh's module graph.
|
||||
|
||||
That left two costs, named in [issue 051](../04-ISSUES/051-the-mesh-cannot-update-what-it-depends-on/00-report.md).
|
||||
The foundation's own store and broker could not be upgraded — nothing owned them as modules. And a
|
||||
mesh that wanted a database or a queue for its modules installed the `postgres`/`lavinmq` modules,
|
||||
each of which raised a **second** server: a mesh ran two postgres and two brokers.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave them as bundle-only plumbing.** Rejected: they cannot be upgraded, and the second
|
||||
server stays. The floor keeps a permanent specialty in it.
|
||||
2. **The control-plane pivot verbatim — raise a temporary one, install the module, retire the
|
||||
temporary** ([ADR 0067](0067-genesis-is-a-pivot.md)). Rejected for a *stateful* server: it means
|
||||
a handover with real downtime, tearing down the store the controller is mid-read of.
|
||||
3. **Adopt in place.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
The foundation's store and broker are **adopted in place** as the ordinary `postgres` and `lavinmq`
|
||||
modules. Genesis still raises them first (nothing else can — ADR 0006), then each module declares a
|
||||
container with the **same name, image and spec** the foundation raised, so the applier — which keys
|
||||
on the container name and compares a spec digest — reconciles it rather than raising a second. The
|
||||
credentials are the foundation's, made at genesis and carried in through `secret accept`, because
|
||||
the mesh cannot invent a credential that already made the databases. The servers bind mesh-wide so
|
||||
a consumer on any node can reach the one shared server.
|
||||
|
||||
A mesh runs **one postgres and one lavinmq**, and each is upgradeable through a stated window: the
|
||||
store's is a connection-pool reconnect; the broker's is the harder case of recreating the bus the
|
||||
push travels over, so the mesh reconnects to the one that returns.
|
||||
|
||||
## Consequences
|
||||
|
||||
The twelve-module floor has no specialty left in it — the store and broker are moments in a
|
||||
module's life, not a separate kind of thing. Two follow-ups are tracked:
|
||||
[issue 054](../04-ISSUES/054-the-adopted-store-and-broker-are-open-before-the-filter/00-report.md)
|
||||
(the adopted servers bind `0.0.0.0` before the packet filter is installed) and
|
||||
[issue 055](../04-ISSUES/055-the-adopted-store-and-broker-may-be-reachable-on-one-node-only/00-report.md)
|
||||
(whether a consumer on another node reaches them over the overlay).
|
||||
|
||||
What got harder: a foundation upgrade recreates the very server the controller reads from, or the
|
||||
bus the instruction to upgrade travels over — a window that a stateless module upgrade does not have.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 051](../04-ISSUES/051-the-mesh-cannot-update-what-it-depends-on/00-report.md) — the gap this closes.
|
||||
- [`03-DESIGN/01-to-be/07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md) — the amended design.
|
||||
- Shipped across mesh-host, mesh-catalog and mesh-lab (main); proven 22/22 in the one-node lab.
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-17
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0078-the-store-and-broker-are-modules.md
|
||||
---
|
||||
|
||||
# 79. The foundation seats are named after their servers
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0078](0078-the-store-and-broker-are-modules.md) settled that the store and broker are the
|
||||
ordinary `postgres` and `lavinmq` modules, adopted in place on the control-node, and that a mesh
|
||||
runs **one postgres and one lavinmq**. But that singularity held only by convention: genesis
|
||||
assigns them to the control-node alone. [Issue 056](../04-ISSUES/056-an-adopted-module-assigned-to-a-second-node-raises-a-second-server/00-report.md)
|
||||
recorded the gap — a second `assign` to another node finds no container of that name there and
|
||||
raises a *second* server, holding none of the first's data, and nothing refuses it.
|
||||
|
||||
The mesh already has the mechanism for "there is one of me": a mesh-scoped exclusive **seat**.
|
||||
[ADR 0077](0077-the-controller-and-the-foundation.md) gave the controller one, which it named
|
||||
`the-controller`. Nothing gave the store and broker theirs.
|
||||
|
||||
## Decision
|
||||
|
||||
Each foundation module claims a mesh-scoped seat named **after the server it guards**:
|
||||
`mesh-controller`, `mesh-store`, `mesh-broker`. So the `postgres` module claims `mesh-store` and
|
||||
the `lavinmq` module claims `mesh-broker`; the resolver refuses a second holder mesh-wide, the same
|
||||
way it keeps one hub and one controller. A second `assign` is now a refusal at resolution — *one
|
||||
per mesh* — not a silent second server.
|
||||
|
||||
The controller's seat, which [ADR 0077](0077-the-controller-and-the-foundation.md) named
|
||||
`the-controller`, is renamed `mesh-controller` under this same rule, so all three foundation seats
|
||||
follow one convention: the seat is the server. (The module `mesh-controller` and its seat now share
|
||||
a name, which is the point — there is one of that server, and the seat says so.)
|
||||
|
||||
This does not tie a foundation module to the control-node — a mesh-scoped seat forbids a *second*
|
||||
holder, not a wrong single one. Adoption still requires the container to already be running where
|
||||
the module lands, which genesis arranges; the seat closes the "two servers" gap, and the
|
||||
control-node convention remains what puts the one holder in the right place.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- The resolver's `checkClaims` refuses two holders of a mesh-scoped seat
|
||||
(`mesh-controller/internal/catalogue/resolve.go`).
|
||||
- `TestAFoundationModuleCannotBeRaisedOnASecondNode` asserts each foundation module's second
|
||||
assignment is refused with *one per mesh*.
|
||||
- The `postgres`, `lavinmq` and `mesh-controller` manifests declare the seat.
|
||||
|
||||
## Consequences
|
||||
|
||||
"One store, one broker, one controller" is now a property the mesh enforces rather than a
|
||||
convention it hopes for. The silent operational edge that remains — a cross-node consumer
|
||||
provisioned only when the provider is pushed again — is separate, and tracked as
|
||||
[issue 057](../04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/00-report.md).
|
||||
|
||||
## References
|
||||
|
||||
- [issue 056](../04-ISSUES/056-an-adopted-module-assigned-to-a-second-node-raises-a-second-server/00-report.md) — the gap this closes.
|
||||
- [ADR 0077](0077-the-controller-and-the-foundation.md) — named the controller's seat, here renamed.
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — asserted one postgres and one lavinmq, here enforced.
|
||||
- [`00-META/glossary.md`](../00-META/glossary.md) — seat and claim.
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-09-17
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0019-how-this-repository-works.md
|
||||
---
|
||||
|
||||
# 80. The development cycle is checked, not trusted
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0019](0019-how-this-repository-works.md) made this repository the source of truth, and the
|
||||
process overview drew the flow work must follow: an idea or a symptom, a decision, a to-be design,
|
||||
a build in a code repository, an as-is update on shipping. The playbooks describe every step, and
|
||||
frontmatter carries every status.
|
||||
|
||||
But the flow itself was enforced by nothing. A design could appear citing no decision; a design
|
||||
could sit `in-progress` naming no code; an issue could be `fixed` by nobody knows what. Each is
|
||||
indistinguishable from correct work until somebody reads carefully — and the whole point of the
|
||||
playbooks is that nobody should have to hold this repository in their head. A session that starts
|
||||
cold (or an agent after a context clear) must be able to *find* the chain by following frontmatter
|
||||
pointers, which only works if the pointers are reliably there.
|
||||
|
||||
## Decision
|
||||
|
||||
The development cycle is enforced mechanically, to the extent frontmatter can carry it:
|
||||
|
||||
- **No design without a decision** — every to-be design names at least one record in `decisions:`.
|
||||
- **No development without a design that says where** — an `in-progress` or `implemented` design
|
||||
names its owning code in `code:`.
|
||||
- **No owner-less diagnosis, no fix-less fix** — an issue marked `located` or `fixed` names
|
||||
`located-in:`; one marked `fixed` or `resolved` says `fixed-by:` (prose counts — "nothing, the
|
||||
capability existed" is an answer).
|
||||
- **No silent graduation** — a `graduated` research overview says what it `became:`, and the
|
||||
targets exist.
|
||||
|
||||
[`00-META/checks/cycle.py`](../00-META/checks/cycle.py) refuses violations, beside `records.py`
|
||||
and `index.py`; all three run before any merge here. What frontmatter cannot see — that code work
|
||||
actually started from a handoff — remains held by playbooks 04 and 07: a feature branch exists
|
||||
because a design or an issue sent it, and a merge is a human checkpoint.
|
||||
|
||||
## Consequences
|
||||
|
||||
A `/clear` costs little: [`AGENTS.md`](../AGENTS.md) now carries the cycle and a where-to-look
|
||||
table, and the chain a fresh session needs is guaranteed present in frontmatter rather than
|
||||
reconstructed from memory. The checks are the floor, not the ceiling — they verify pointers exist,
|
||||
not that their content is true; reading remains the job.
|
||||
|
||||
## References
|
||||
|
||||
- [`00-META/process/00-overview.md`](../00-META/process/00-overview.md) — the flow, and its new
|
||||
"The cycle is checked" section.
|
||||
- [ADR 0019](0019-how-this-repository-works.md) — the repository this disciplines.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-09-17
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0080-the-development-cycle-is-checked.md
|
||||
---
|
||||
|
||||
# 81. A decision nothing cites is not yet in the chain
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0080](0080-the-development-cycle-is-checked.md) made the development cycle checked — but its
|
||||
checks covered designs, issues and research, not the decisions themselves. Measuring showed why
|
||||
that matters: 19 of 70 records were cited by nothing — no design doc's `decisions:`, no research
|
||||
`became:`, no issue, no other record. Among them sat load-bearing decisions (the credential flow,
|
||||
the module-runtime cluster), and the cost had already been paid once in practice: a stale premise
|
||||
about an orphaned decision survived in working memory precisely because no pointer led to the
|
||||
record that had settled it.
|
||||
|
||||
## Decision
|
||||
|
||||
Every **accepted** decision must be reachable from the cycle: cited by a design doc's frontmatter
|
||||
(`decisions:` — a governing citation, not a prose mention), a research overview, an issue report,
|
||||
a `00-META` document, or another record's `extends`/`supersedes` chain.
|
||||
[`cycle.py`](../00-META/checks/cycle.py) refuses orphans. Proposed records are exempt — a record
|
||||
under consideration has no home yet — and superseded records are reachable through their
|
||||
supersession chain by construction.
|
||||
|
||||
The 19 orphans were given true homes in the same change: the module-runtime cluster
|
||||
(0044–0049, 0053–0055) into the connectivity, controller, protocol, writing and model-access
|
||||
designs; the build decisions (0072, 0076) into the building design; 0021 into the playbook that
|
||||
implements it; the process records were already reachable once `00-META` counted as a source.
|
||||
|
||||
Taken together with 0080, the practice has a name the industry will recognise:
|
||||
**spec-driven development, with provenance** — a decision is the *why*, the design doc is the
|
||||
spec, `code:` names the implementation, and the lab beds are the conformance tests. What the
|
||||
common form leaves implicit, the cycle makes checked: the spec itself must trace to a decision,
|
||||
and the decision must be findable from the work it governs.
|
||||
|
||||
## Consequences
|
||||
|
||||
Following pointers now reaches every accepted decision, so a cleared session (or a person) can
|
||||
trust the frontmatter graph as the whole map. The check is reachability, not truth: a citation
|
||||
placed wrongly still lies, and reading remains the job.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0080](0080-the-development-cycle-is-checked.md) — the cycle this completes.
|
||||
- [`00-META/process/00-overview.md`](../00-META/process/00-overview.md) — the flow.
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-17
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0072-two-graphs-and-the-build-chain.md
|
||||
---
|
||||
|
||||
# 82. The registry is reached by name, and the overlay is its security
|
||||
|
||||
## Context
|
||||
|
||||
Delivery ends at a node pulling an image, and for every node but the one that built it, that step
|
||||
has never worked. Three facts conspired, each recorded separately:
|
||||
|
||||
- Artifact references are written under `127.0.0.1:5000` — a deliberate parking (the catalogue
|
||||
says so in the commit that reverted the mesh-reachable name: *"the runtime refused it: 'http:
|
||||
server gave HTTP response to HTTPS client' … the binding expression returns then"*), so a joined
|
||||
node is told to pull from its own loopback
|
||||
([issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)).
|
||||
- A container runtime treats any non-loopback registry as HTTPS, and the mesh's registry serves
|
||||
plain HTTP; nothing the mesh writes tells any runtime otherwise — the one place that file
|
||||
existed was the lab's, which is why this never failed in a bed (issue 048).
|
||||
- Nothing gives a node an account for the registry
|
||||
([issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md)) —
|
||||
and nothing has ever said whether one is required.
|
||||
|
||||
The tempting fix is TLS from the mesh's own authority. The machinery even exists — the mesh CA
|
||||
issues leaves for `<node>.internal` names, and a `certificate:` manifest field delivers them. But
|
||||
the registry is reached **over the overlay**, and the overlay is WireGuard: every byte is already
|
||||
encrypted and already authenticated to a peer the mesh admitted. The mesh's own code has carried
|
||||
this position for weeks: *"the registry is reached over the mesh's own private network … a second
|
||||
layer inside it would be certificates to issue and rotate for no property the first does not
|
||||
have."* TLS inside the tunnel would also re-order genesis (a certificate needs the network, the
|
||||
registry precedes it) and put CA handling into three clients (the runtime, the archive fetcher,
|
||||
the builder) — cost with no new property.
|
||||
|
||||
## Decision
|
||||
|
||||
**The overlay is the registry's transport security, and the mesh writes the trust it means.**
|
||||
|
||||
1. **References name the registry by its mesh name.** The builder publishes to
|
||||
`<provider>.internal:<port>` (the binding's `at` and `port`) — the parked one-line change
|
||||
lands. Genesis still publishes to loopback on the first machine, before any overlay exists;
|
||||
references minted at genesis stay loopback and are valid where they matter — on that machine.
|
||||
A rebuild re-pins to the mesh name.
|
||||
2. **Every node on the private network is told the mesh's registry speaks plain HTTP.** The
|
||||
controller injects, into every such node's declaration, a merged `/etc/docker/daemon.json`
|
||||
naming `<provider>.internal:<port>` under `insecure-registries`, and a service resource that
|
||||
restarts the runtime when that file changes — the `/etc/hosts` pattern for the content, the
|
||||
nftables pattern for the reload. No module author is involved; being on the network is what
|
||||
grants the trust, because being on the network is what the trust *is*.
|
||||
3. **No accounts (issue 042), recorded as the position it always was.** Reading and pushing
|
||||
require presence on the overlay and nothing else. The boundary is enforced, not assumed: the
|
||||
registry's `listens` is `from: mesh`, the firewall derives from it, and the overlay admits only
|
||||
peers holding mesh-issued keys. An operator's *external* registry credential is an ordinary
|
||||
operator-supplied secret (`secret accept`), owned by whichever module names that registry.
|
||||
Per-node accounts return as a decision, not a patch, if the boundary assumption ever changes —
|
||||
the shape would be the existing provision flow.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- The no-fake lab bed: a joined node pulls a mesh-built image by the registry's `.internal` name
|
||||
over the overlay — an address outside every range the lab's own runtime configuration trusts,
|
||||
so the mesh-written trust is what makes it work or nothing does.
|
||||
- The firewall half is generated from `listens` and visible in the node's ruleset (`from: mesh`).
|
||||
- The plaintext-inside-tunnel position holds exactly as long as the registry is unreachable off
|
||||
the overlay; the `listens` stanza and the derived ruleset are the check.
|
||||
|
||||
## Consequences
|
||||
|
||||
A runtime restart when the trust first lands on a node — at joining, before workloads, where it
|
||||
is free. The registry's HTTP is exposed to whatever stands on the overlay, which is the stated
|
||||
boundary; a mesh that wants defence in depth inside its own tunnel reopens this record rather
|
||||
than bolting certificates on quietly. Issues 042 and 048 close on this record.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md),
|
||||
[issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md).
|
||||
- [ADR 0072](0072-two-graphs-and-the-build-chain.md) — the build chain this completes.
|
||||
- [ADR 0029](0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md) — the overlay as a
|
||||
boundary.
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-18
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0010-delivery.md
|
||||
---
|
||||
|
||||
# 83. One push leaves the mesh consistent
|
||||
|
||||
## Context
|
||||
|
||||
A provision is minted while composing the *consumer's* node; the provider's grant list is a pure
|
||||
read of secrets already issued from it. So assigning a cross-node consumer and pushing its node
|
||||
produced a consumer that retried forever against a provider that had never heard of it, until the
|
||||
provider's node was pushed a second time — an action with no signal to take, documented nowhere,
|
||||
and invisible whenever consumer and provider share a machine
|
||||
([issue 057](../04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/00-report.md)).
|
||||
|
||||
Two remedies were on the table: **cascade** — a push also delivers to the machines its compose
|
||||
changed — or **report** — a push says "now push the provider" and leaves the act to the operator.
|
||||
|
||||
## Decision
|
||||
|
||||
A push finishes what it starts: after composing and sending the named node, the controller flushes
|
||||
every *other* machine that is now behind — whose declaration differs from what it was last sent —
|
||||
by name, in the push's own output, converging over a bounded number of rounds (a flushed send may
|
||||
itself mint).
|
||||
|
||||
Behind is measured against what a machine was last *sent*, not against a before/after snapshot of
|
||||
this push. The mint that makes a provider behind happens when the consumer is assigned or its
|
||||
account issued — before `push` runs at all — so by push time the provider already differs from
|
||||
what it holds, with no in-command delta to detect. The only durable signal is "what it should be"
|
||||
versus "what it last received", which is the same comparison `push --behind` already makes.
|
||||
|
||||
A machine behind for an unrelated reason is flushed by this too, and that is correct rather than a
|
||||
cost: a named push that knew a machine was behind and left it so would be the very silence this
|
||||
decision removes. The narrower reading — flush only what this push provably changed — was
|
||||
rejected because it cannot see a mint that a prior command performed, which is precisely the 057
|
||||
case.
|
||||
|
||||
Reporting alone was rejected because it converts a derived fact the controller already holds into
|
||||
an operator obligation, and an obligation enforced by nothing is issue 057 restated. The
|
||||
declaration is computed from the whole mesh; delivering a mesh that is knowingly inconsistent and
|
||||
merely saying so would make "push succeeded" mean less than it says.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One push is sufficient for a cross-node consumer: the provider's grants arrive from the same
|
||||
act that minted the provision. The undocumented rule "push the provider node too" ceases to
|
||||
exist rather than becoming documentation.
|
||||
- A named push delivers to every machine that is behind, not only the one named — each named in
|
||||
the output, never silent. `push --behind` remains the way to reconcile the mesh without naming
|
||||
a node; a named push now carries the same guarantee for the machines its work touched and any
|
||||
others already waiting.
|
||||
- How this is checked: the built-store-cross-node bed registers a cross-node consumer, pushes
|
||||
only the consumer's node, and asserts the provider minted its vhost — the workaround push is
|
||||
removed, so a regression fails the bed.
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-20
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
|
||||
---
|
||||
|
||||
# 84. Which provider serves a consumer, when the mesh runs more than one
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) settled what a provision
|
||||
is *named* for — the thing the consumer's code is coupled to, so `postgres-database` and
|
||||
`mssql-database` are different provisions and a wrong match is refused at resolution. It said one
|
||||
thing more, in passing, and left it: *"Two providers of `postgres-database` — a container on this
|
||||
node and a managed instance elsewhere — are interchangeable and should both match."* Naming was
|
||||
decided; **which of several providers serves a given consumer was not.**
|
||||
|
||||
The mesh assumes there is only one to choose. Provisions are mesh-scoped: the adopted store is a
|
||||
single `mesh-store` a consumer on any node reaches, and `scope: "mesh"` is written into the
|
||||
provision definitions. **That assumption is false on day one, and was always meant to be.**
|
||||
Node-specific services delivered to the mesh is the plan, not an edge case:
|
||||
|
||||
- Both control-capable nodes already run their **own** general-purpose relational store (the same
|
||||
engine, one instance each), serving that node's own applications.
|
||||
- Each runs its **own** SQL server, its **own** cache, its **own** object store. One node alone
|
||||
runs six separate relational-store instances, each raised by the module that needed it.
|
||||
- The one provision that is currently single — the identity provider, one instance on one node —
|
||||
already authenticates applications whose home is a **different** node.
|
||||
|
||||
So several providers of one provision name genuinely coexist, and they are **not** interchangeable
|
||||
the way 0027's aside supposed. They differ by node, by the data they hold, and by locality. A
|
||||
consumer bound to the wrong one reads the wrong database, or takes a cross-node hop it did not
|
||||
need, or cannot be moved without silently rebinding. The model has no field in which to say which
|
||||
one. This is 0027's own fault — *a match that resolves and is wrong* — one level up: 0027 refused
|
||||
the wrong **dialect**; nothing refuses, or even asks about, the wrong **instance**.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep `scope: "mesh"` — one provider per provision, mesh-wide.** Rejected: it is false on day
|
||||
one, and making it true would force every node's applications onto one node's server — the
|
||||
exact opposite of node-specific services delivered to the mesh, and a single point of failure
|
||||
the topology was built to avoid.
|
||||
2. **Resolve to any provider of the name (0027's "both match").** Rejected: when providers hold
|
||||
different data and live on different nodes they are not interchangeable, and picking one
|
||||
arbitrarily is a wrong-instance match — the confidently-wrong answer 0027 exists to prevent,
|
||||
restated at the level of the instance rather than the dialect.
|
||||
3. **Always require the consumer to name the provider explicitly.** Rejected: needless ceremony in
|
||||
the common case, where the consumer wants the provider on its own node; and a field every
|
||||
manifest must carry is a field an author forgets, which then matches everything again — the
|
||||
failure 0027 warned about for qualifiers.
|
||||
4. **A provision is node-scoped; the consumer selects the provider, defaulting to co-location.**
|
||||
Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A provision is served by a provider identified by its node, and the consumer selects which one.**
|
||||
A provider is a (node, module) pair, not a mesh-wide singleton. A consumer's binding resolves to a
|
||||
specific provider, and the selection is part of the assignment
|
||||
([ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md)), not the
|
||||
manifest.
|
||||
|
||||
**The default is co-location.** A consumer that names no provider is served by the provider of
|
||||
that provision on **its own node**. This is the common case and needs nothing said. A mesh with
|
||||
one provider of a kind is just the case where co-location and "the only one" coincide — expressed
|
||||
as *there happens to be one*, not as a scope.
|
||||
|
||||
**A consumer coupled to a provider's data names it.** Where two consumers must share one database,
|
||||
or a consumer must reach a provider on another node, the assignment names that provider — because
|
||||
that coupling is exactly what may not be guessed, and naming it is what makes a later move safe.
|
||||
|
||||
**A module need not consume a provider at all.** It may carry its **own** instance inside its own
|
||||
composition — on its own module network, publishing no host port, **not** declared as a provision —
|
||||
when a genuine engine fork or a pinned server version makes the shared provider unusable. Such an
|
||||
instance is invisible to resolution and can be bound by nothing else. The rule is *share by
|
||||
default; embed only when a fork or a version forces it* — most of the per-module stores that exist
|
||||
today are vanilla engines on stale pins that a consolidation onto the node's provider would absorb.
|
||||
|
||||
## Consequences
|
||||
|
||||
The mesh can carry its real topology **deliberately** rather than by the accident of which
|
||||
provider happened to be the single one. A consumer's data-coupling becomes a stated fact, which is
|
||||
what lets a provider be moved without a consumer silently following the wrong one — provided a
|
||||
provider keeps its identity across a relocation, which is a follow-up this record opens rather than
|
||||
closes. Rotation ([13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)) addresses a
|
||||
specific provider's holders, not "the provision's".
|
||||
|
||||
What got harder: an assignment now may carry a provider selection, and a wrong one is a new way to
|
||||
misconfigure. It is mitigated the way 0027 mitigated its own: the co-location default removes the
|
||||
choice in the common case, and genuine ambiguity — several providers, none named, none co-located —
|
||||
is refused with the candidates named, never resolved by picking.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — the naming this extends;
|
||||
its "both match" aside is the gap closed here.
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity is exactly such a
|
||||
provider, and already serves consumers on another node.
|
||||
- [ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md) — where the
|
||||
selection lives.
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — the adopted store, whose mesh-wide
|
||||
binding is the single-provider assumption this record replaces.
|
||||
- [issue 067](../04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md) — the
|
||||
gap, and the day-one evidence.
|
||||
- [`03-DESIGN/01-to-be/23-choosing-a-provider.md`](../03-DESIGN/01-to-be/23-choosing-a-provider.md)
|
||||
— the design.
|
||||
@@ -0,0 +1,153 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-20
|
||||
amended: 2026-09-20
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md
|
||||
---
|
||||
|
||||
# 85. A secret is a provision, and the vault is the module that provides it
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0031](0031-the-control-plane-authenticates-nobody.md) decided that identity *runs on the
|
||||
mesh, not of it* — a module other modules require, rather than a privileged part of the
|
||||
controller. [ADR 0078](0078-the-store-and-broker-are-modules.md) did the same for the store and
|
||||
the broker: the twelve-module floor has no specialty left in it. **Secrets are the exception that
|
||||
survived.** No module owns a secret.
|
||||
|
||||
Secret handling is smeared across three built-in parts of the runtime:
|
||||
|
||||
- the **controller mints** one credential per consumer↔provider pair and seals it to both node
|
||||
keys ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md));
|
||||
- the **mesh generates local secrets** — *"generated secrets are the mesh's, never authored"*
|
||||
(as-is, `06-configuration-and-secrets.md`) — for a value a single module needs for its own use;
|
||||
- the **synchroniser injects** both into a node's generated files.
|
||||
|
||||
Nothing is the owner of "a secret" the way the store module is the owner of "a database", and the
|
||||
cost is recorded rather than hypothetical. The as-is design names the weakness in its own words:
|
||||
*"Rotation is not a mesh operation… there is no mechanism that rotates one and informs everything
|
||||
holding it."* A `rotate` command has since been built and proven, but it reaches **only** the
|
||||
provisioned pairs; a secret a module generates for its own fully-local use — the password of a
|
||||
version-pinned embedded store ([ADR 0084](0084-which-provider-serves-a-consumer.md)), an internal
|
||||
token — is minted by the mesh and then has no operation that can remake it. Three species of
|
||||
secret, and only the first has an owner:
|
||||
|
||||
| species | minted by | rotates? |
|
||||
|---|---|---|
|
||||
| a provisioned credential (a database login) | controller, sealed to nodes (0048) | yes — `rotate`, per pair |
|
||||
| a module's own local secret | the mesh, as a generated value | **no owner, no rotation** |
|
||||
| an operator-delivered secret (an external key) | a person, sealed in (`secret accept`) | no rotation, no audit |
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave it a property of the controller.** Rejected: it is the smear above — no owner, local
|
||||
secrets that cannot be rotated, operator secrets that cannot be audited — and it is exactly the
|
||||
specialty 0078 removed for the store and broker, kept here for no reason anyone recorded.
|
||||
2. **A dedicated vault built into the foundation, not a module.** Rejected: it reintroduces a
|
||||
privileged built-in, the thing 0031 and 0078 went out of their way to remove, and a mesh that
|
||||
wants none would still carry it.
|
||||
3. **Fold all minting, the controller's provisioning credentials included, into the vault.**
|
||||
Rejected: the controller must mint in order to **deliver** any provision — the vault's own
|
||||
credential among them — so making the vault mint the credential of its own delivery is the
|
||||
store/broker chicken-and-egg for no gain. The provisioned-pair credential already has an owner
|
||||
and a rotation ([13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)); this record
|
||||
does not disturb it.
|
||||
4. **A secret is a provision; the vault is an ordinary module that provides it.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**Secret-holding is a module, parallel to identity.** A module that needs a secret **for its own
|
||||
use** — a local service's password, an internal token, an external key it was handed — requires a
|
||||
`secret` provision from a vault provider, exactly as it requires a database from the store. The
|
||||
vault generates the value (or holds one it was given), and because the credential belongs to the
|
||||
consumer↔vault pair it **rotates, backs up and is audited through the same per-pair machinery**
|
||||
[13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md) already defines. That is what
|
||||
gives the second and third species the rotation and audit they lack. A module's own secret stops
|
||||
being a generated value that nothing owns and becomes an ordinary provision with a provider.
|
||||
|
||||
**The controller's minting of provisioning-pair credentials is unchanged** ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)):
|
||||
it is how every provision, the vault's own included, is delivered. The vault does not mint the
|
||||
mesh's delivery credentials; it provides secrets to modules, and is itself provisioned the ordinary
|
||||
way.
|
||||
|
||||
**The vault is node-scoped like every provider** ([ADR 0084](0084-which-provider-serves-a-consumer.md)):
|
||||
each node its own, selected the same way, so a module's own secret is held by the vault on the
|
||||
module's node. **A mesh that wants no vault runs none** — a module requiring no secret needs
|
||||
nothing, which is the same test 0031 applied to identity.
|
||||
|
||||
## Consequences
|
||||
|
||||
The gap that opened this — a generated local secret with no rotation — **closes without new
|
||||
machinery**: rotating such a secret is the vault's provisioner remaking a pair credential, the
|
||||
operation 13 already specifies. Backup, audit and break-glass gain an owner — the vault module —
|
||||
and become things a design specifies rather than absences. The as-is sentence *"rotation is not a
|
||||
mesh operation"* is already false for provisioned pairs and, once the vault ships, for local
|
||||
secrets too; the as-is document is updated when it does, not before.
|
||||
|
||||
What got harder: a break-glass path — recovering a secret when the sealed delivery path is
|
||||
unavailable — must not reintroduce a key that one place holds, which is the property
|
||||
[ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) was built to preserve; how
|
||||
the vault offers recovery without it is left to the design as an open question. And a module that
|
||||
today bakes a password into its own composition must instead require it from the vault — a
|
||||
migration taken module by module, not a flag day.
|
||||
|
||||
## Amendment — 2026-09-20, before anything shipped
|
||||
|
||||
Recorded on the record itself rather than as a supersession, by its decider, on the day it was
|
||||
accepted and before any code was merged against the sentences that change. The original text above
|
||||
is left as written; this section says what it got wrong and what stands instead.
|
||||
|
||||
**What it got wrong.** The decision treated the vault as a provider like the store — optional, and
|
||||
one per node — and left the mesh's own root secrets outside it: the store's superuser, the broker's
|
||||
administrator, the controller's contexts, sealed to a node key and nothing else. Those are the
|
||||
secrets with no rotation and no recovery, and they are the ones a vault exists for. At genesis they
|
||||
are not even secret: the foundation raises its store and broker with fixed, well-known credentials
|
||||
and carries those into the mesh. Leaving that floor in place gave module secrets an owner and the
|
||||
root secrets none.
|
||||
|
||||
**What stands instead.**
|
||||
|
||||
- **The vault is a foundation module.** It is installed at genesis as part of the foundation
|
||||
([ADR 0078](0078-the-store-and-broker-are-modules.md) is the precedent: a foundation piece is
|
||||
still an ordinary module), not assigned later by a mesh that happens to want one. *"A mesh that
|
||||
wants no vault runs none"* is withdrawn. A mesh has root secrets, so a mesh has a vault.
|
||||
- **One per mesh, on the control-node.** *"The vault is node-scoped like every provider"* is
|
||||
withdrawn. Node scoping ([ADR 0084](0084-which-provider-serves-a-consumer.md)) exists because a
|
||||
store holds data a consumer is coupled to; the vault holds nothing a consumer is coupled to, and a
|
||||
second one would be a second place to lose. Consumers on other nodes reach it as they reach
|
||||
identity.
|
||||
- **The vault holds the mesh's root secrets under an operator-held key.** The controller mints and
|
||||
delivers exactly as before; in addition, every secret a module holds for itself is sealed a second
|
||||
time, to an **operator sealing key** whose private half never enters the mesh. The vault keeps
|
||||
those operator-sealed copies on its own disk, outside the store, and can hand them out — they are
|
||||
ciphertext to everything but the operator. This is the break-glass path the original text left
|
||||
open, and it does **not** reintroduce a key one place holds: the mesh holds blobs it cannot open,
|
||||
and the operator holds a key with nothing to open until given a blob. Recovery needs both.
|
||||
- **Genesis mints real root secrets and seals them to the operator key first**, so the fixed
|
||||
credentials the foundation is raised with are replaced before the mesh is handed over.
|
||||
|
||||
**Unchanged.** A module's own secret is a `secret` provision the controller mints and the vault
|
||||
records; the provisioned-pair path of [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)
|
||||
is untouched; the vault stores no plaintext, ever.
|
||||
|
||||
**What it costs.** An operator key is a thing a person must keep, and a mesh whose operator key is
|
||||
lost has root secrets that can be rotated but not recovered — the same standing as today, stated.
|
||||
Sealing every own secret twice is a column and a call. Genesis grows a step. A module's
|
||||
vault-provided secret (a pair credential) is not yet sealed to the operator key; that is the next
|
||||
increment, not this one.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity is a module; this is the
|
||||
same move for secrets.
|
||||
- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) — the provisioning-credential
|
||||
path, left unchanged.
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — the de-specialisation this completes.
|
||||
- [ADR 0084](0084-which-provider-serves-a-consumer.md) — the node-scoping the vault obeys.
|
||||
- [issue 068](../04-ISSUES/068-secrets-have-no-owning-module/00-report.md) — the gap.
|
||||
- [`03-DESIGN/01-to-be/24-the-secrets-vault.md`](../03-DESIGN/01-to-be/24-the-secrets-vault.md) —
|
||||
the design. The source mesh's `secret_locate` / `secret_backup` / `secret_verify` /
|
||||
`secret_breakglass` subsystem is the prior art it draws on.
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0085-a-secret-is-a-provision.md
|
||||
---
|
||||
|
||||
# 86. A secret reaches a process as a file, and an exception is declared
|
||||
|
||||
## Context
|
||||
|
||||
The mesh seals a secret to the machine that uses it and discards the plaintext; the host unseals it
|
||||
into a file at 0600 ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)). Then
|
||||
the module hands it to its container through an env-file, and the runtime puts it where anything
|
||||
on the machine that can talk to the runtime can read it: `docker inspect` prints it, and
|
||||
`/proc/<pid>/environ` holds it for the life of the process
|
||||
([issue 041](../04-ISSUES/041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md)).
|
||||
|
||||
Two facts made this a decision rather than a fix. **It is a supported shape**: placing
|
||||
`${secret:…}` in a file a container reads as its environment is what playbook 06 shows, and 36
|
||||
containers in 25 catalogue modules do it, the controller's own among them. And **nothing says which
|
||||
secrets are exposed**: a reader cannot tell from a manifest whether a credential is protected from
|
||||
`inspect` or not, because the manifest looks the same either way. The vault
|
||||
([ADR 0085](0085-a-secret-is-a-provision.md)) rests on the seal this undoes.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave it: the environment is where configuration goes.** Rejected — it gives the sealed value
|
||||
back to a routine operation, and the care taken to seal it is then theatre.
|
||||
2. **Refuse every secret in an environment.** Rejected — some software reads its configuration
|
||||
from the environment and nothing else, and a rule the catalogue cannot obey is a rule that gets
|
||||
switched off.
|
||||
3. **A secret reaches a process as a file; an environment exception is declared, with a reason,
|
||||
and refused otherwise.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A secret reaches a process as a file.** A module mounts the file the host wrote and points the
|
||||
program at it; the mesh's own programs accept a `_FILE` twin for every variable that carries a
|
||||
credential, the way the controller's store connections already did.
|
||||
|
||||
**A container that reads a secret from its environment says so.** The manifest key
|
||||
`secrets-in-environment` on the container carries the reason. It is catalogue-level — the host
|
||||
never sees it — and it exists so a reader can tell from the manifest which secrets are exposed
|
||||
that way and why.
|
||||
|
||||
**Everything else is refused.** A `${secret:…}` placeholder inside a container's `env` is never
|
||||
filled and is refused outright. A file carrying a secret that a container names in `env-file` is
|
||||
refused unless the container declares the exception.
|
||||
|
||||
## Consequences
|
||||
|
||||
The property *a sealed credential is readable only where it is used* becomes a manifest-level
|
||||
fact: true where no exception is declared, stated where one is. The controller reads all six of
|
||||
its credentials from files; the 35 other containers are marked with their reason and convert one
|
||||
by one where their software accepts a path, which is per-module work under
|
||||
[issue 041](../04-ISSUES/041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md).
|
||||
|
||||
What got harder: a module author meets one more refusal, and the reason they write is only as
|
||||
honest as they are. What is not decided: the reach of the exposure on a node — which identities can
|
||||
talk to the runtime — which decides whether a declared exception is a hardening item or something
|
||||
sharper.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The catalogue engine refuses at composition, before anything reaches a machine, and its tests
|
||||
refuse both shapes and prove the reason is stripped from what is sent.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 041](../04-ISSUES/041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md)
|
||||
- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md), [ADR 0085](0085-a-secret-is-a-provision.md)
|
||||
- [`03-DESIGN/01-to-be/13-credentials-and-their-rotation.md`](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0010-delivery.md
|
||||
---
|
||||
|
||||
# 87. A seeded file is created once, and what grows in it is not the mesh's
|
||||
|
||||
## Context
|
||||
|
||||
A declaration is complete for what the host owns, and the host reconciles what is declared
|
||||
([ADR 0010](0010-delivery.md)): a file with this content, held to it. That is the only thing a
|
||||
manifest could say about a file, and it is the wrong thing for a file a module needs to **exist
|
||||
before first start** and something else then legitimately writes into — an access list a
|
||||
provisioner appends consumers to and the program persists back, a bootstrap configuration a
|
||||
program rewrites. Every reconcile restored the seed behind the running program, erased what had
|
||||
grown in it, and reported success
|
||||
([issue 035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md)).
|
||||
|
||||
A run-once step ([ADR 0052](0052-a-step-that-runs-once-before-a-container.md)) can write a seed
|
||||
only if absent, and that closed the instance for a broker whose seed is a program's job. It left
|
||||
the general case: a plain file the mesh writes and never overwrites.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Two owners never share a file: the provisioner owns it, and first-start ordering is solved
|
||||
another way.** Rejected as the only answer — some software refuses to start without the file,
|
||||
and a module that must ship a program merely to write an empty file has been made to write a
|
||||
program to say one word.
|
||||
2. **A create-once semantic on a file.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A file resource may say `create-once`. The host writes it when it is absent and, when it is
|
||||
present, leaves it entirely alone — content, mode and owner — and reports it as **kept**, not
|
||||
corrected. What is in the file then is somebody else's work the mesh asked for. The mesh removes
|
||||
nothing it did not create ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md)); it now
|
||||
also does not overwrite what it created once and handed over.
|
||||
|
||||
On the security question ADR 0010 asks of every new resource behaviour: this **narrows** what a
|
||||
declaration can do to a machine. A create-once file gives a compromised control plane one fewer
|
||||
way to change a machine repeatedly — it can seed, once, and never again.
|
||||
|
||||
## Consequences
|
||||
|
||||
A module says which of its files are seeds, and the difference is visible in the manifest rather
|
||||
than in whether the file happened to be revisited. A later change to a seed's declared content
|
||||
does not reach a machine that already has the file; that is the meaning of a seed, and a module
|
||||
that needs the new content ships it as a run-once step that migrates the existing file.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The host's apply tests: a seed is created, grown into by hand, reconciled, and the growth survives
|
||||
with the outcome `kept`. The vault bed declares one on a real node, grows it, pushes again, and
|
||||
reads it back.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md)
|
||||
- [ADR 0010](0010-delivery.md), [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md), [ADR 0052](0052-a-step-that-runs-once-before-a-container.md)
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0078-the-store-and-broker-are-modules.md
|
||||
---
|
||||
|
||||
# 88. The foundation filters before anything listens
|
||||
|
||||
## Context
|
||||
|
||||
Adopting the store and broker as ordinary modules
|
||||
([ADR 0078](0078-the-store-and-broker-are-modules.md)) needs them reachable by consumers across
|
||||
the mesh, so genesis raises them bound to every interface. The packet filter that decides who may
|
||||
reach them is a module too, installed a dozen steps later. Between the two, a control-node facing
|
||||
the network has its store and its bus open to anyone who can reach the machine, for the length
|
||||
of the install ([issue 054](../04-ISSUES/054-the-adopted-store-and-broker-are-open-before-the-filter/00-report.md)).
|
||||
The design's rule — what a port is reachable from is decided by the filter — is enforced by
|
||||
nothing for that window.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Accept the window**: the machine is mid-bootstrap and the exposure matches what the
|
||||
pre-adoption modules had in steady state. Rejected — the whole point of deriving the filter
|
||||
was to stop accepting that.
|
||||
2. **Bind narrowly at genesis and widen once the filter exists.** Rejected — a bind change is a
|
||||
recreate of the mesh's store during install, and adoption in place needs the same spec.
|
||||
3. **The foundation carries a filter of its own, applied before the store.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
The foundation bundle installs the packet filter and loads a base ruleset **before the store and
|
||||
broker are raised**: drop by default; keep loopback, replies, ping and ssh; keep the mesh's own
|
||||
ports a node must reach before it is on the private network — the bus it enrols over and the
|
||||
registry it pulls from; and let the container runtime's own networks through the forward chain so
|
||||
containers keep working. It is written into the **same table** the filter module later derives, so
|
||||
that module replaces it wholesale the moment it can compute one from what the mesh knows, and
|
||||
nothing of the base survives to contradict it.
|
||||
|
||||
## Consequences
|
||||
|
||||
From its first resource a machine being made into a mesh refuses what it will refuse when
|
||||
finished; the window closes. What got harder: the base ruleset is static and names two ports the
|
||||
mesh's derived one also names — a change to which ports the foundation needs is now made in two
|
||||
places, and the bundle's own test says which.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The installer's bundle test asserts the filter and its load precede the store and broker and that
|
||||
the rules name ssh, the bus and the registry and not the store's or broker's client ports. The
|
||||
genesis bed probes the machine from outside throughout the install: the store's port is never
|
||||
reachable, while the bus becomes reachable.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 054](../04-ISSUES/054-the-adopted-store-and-broker-are-open-before-the-filter/00-report.md), issue 047
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md)
|
||||
- [`03-DESIGN/01-to-be/07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md)
|
||||
@@ -0,0 +1,69 @@
|
||||
---
|
||||
topic: checking it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0016-the-lab.md
|
||||
---
|
||||
|
||||
# 89. A bed reads the catalogue it proves
|
||||
|
||||
## Context
|
||||
|
||||
A lab bed installs a catalogue module and asserts what the mesh does with it; "proven in the
|
||||
lab" is the standard a module must meet before it ships. The beds built the manifests they
|
||||
install inline — a literal copied from the catalogue when each bed was written, and never since.
|
||||
Six modules were converted to file-delivered secrets ([ADR 0086](0086-a-secret-reaches-a-process-as-a-file.md))
|
||||
and not one bed ran the converted shape; each ran its copy, and the copies still delivered
|
||||
secrets the way the catalogue engine now refuses
|
||||
([issue 073](../04-ISSUES/073-beds-carry-copies-of-catalogue-manifests/00-report.md)). A run's
|
||||
receipt named the commits of the host, the controller and the lab, and said nothing about the
|
||||
catalogue, so a catalogue change and a proven catalogue change were indistinguishable.
|
||||
|
||||
The end-to-end design already has the rule this breaks — *the run rebuilds what it tests; an
|
||||
artifact rebuilt from memory is one rebuilt sometimes* — for binaries and images. A manifest is
|
||||
an artifact too.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep the copies and check them against the catalogue** — a test that diffs each literal
|
||||
against the module's manifest, ignoring what the lab must rewrite. Rejected: it keeps two
|
||||
sources of truth and adds a third thing that can drift, the list of what to ignore.
|
||||
2. **Read the catalogue, rewriting only what the lab must.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A bed that installs a catalogue module reads that module's manifest from the catalogue checkout
|
||||
the run was pointed at. It may rewrite what the lab must and nothing else: a build artifact
|
||||
becomes the image the machine holds, an image is pinned to what the machine holds, a host port
|
||||
is remapped where one machine carries colliding modules, and an address may point at a stand-in
|
||||
the bed raises in place of an upstream. Everything else is the catalogue's, verbatim.
|
||||
|
||||
A bed that needs less than the catalogue declares — no upstream server, a secret in the
|
||||
environment, a requirement edge removed — is not testing that module. It is a mesh test, and it
|
||||
carries a name of its own (see [issue 074](../04-ISSUES/074-a-mesh-test-wears-a-catalogue-modules-name/00-report.md)).
|
||||
|
||||
The run's receipt names the catalogue's commit alongside the other repositories', so a receipt
|
||||
taken before a manifest changed says so.
|
||||
|
||||
## Consequences
|
||||
|
||||
A catalogue change is proven by the beds that install the module, or it is not proven, and the
|
||||
receipt says which. What got harder: a bed can no longer trim a module to the shape it finds
|
||||
convenient; it meets the module's declared requirements or gives its fixture another name.
|
||||
The beds that still carry a copy are declared, each with its reason, and the declared list
|
||||
only shrinks.
|
||||
|
||||
## How it is checked
|
||||
|
||||
A unit test in the lab refuses an inline manifest literal that names a catalogue module unless
|
||||
the bed is declared, with its reason, in the test's own list; a declaration for a bed that no
|
||||
longer carries the copy is refused too, so the list cannot outlive the debt. The receipt test
|
||||
asserts the catalogue is claimed whenever the run is pointed at one.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 073](../04-ISSUES/073-beds-carry-copies-of-catalogue-manifests/00-report.md), [issue 074](../04-ISSUES/074-a-mesh-test-wears-a-catalogue-modules-name/00-report.md)
|
||||
- [ADR 0016](0016-the-lab.md), [ADR 0086](0086-a-secret-reaches-a-process-as-a-file.md)
|
||||
- [`03-DESIGN/01-to-be/01-end-to-end-testing.md`](../03-DESIGN/01-to-be/01-end-to-end-testing.md)
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0010-delivery.md
|
||||
---
|
||||
|
||||
# 90. A failure that repeats is said to be stuck
|
||||
|
||||
## Context
|
||||
|
||||
Delivery is a comparison, not a one-shot ([ADR 0010](0010-delivery.md)): a node re-applies the
|
||||
declaration it holds on a steady interval and reports each time. That is right for a failure that
|
||||
goes away by itself — the overlay not up yet, a registry briefly unreachable — and it makes a
|
||||
failure that will never go away look exactly the same. A resource nothing can ever apply is
|
||||
attempted, fails, is reported, and is attempted again every few minutes, indefinitely; the mesh
|
||||
keeps one report per machine, replaced, so each attempt arrives as "failed" at a fresh time.
|
||||
Nothing distinguished "failed once, will succeed when its dependency arrives" from "failed
|
||||
identically for ever", and nothing escalated the second
|
||||
([issue 065](../04-ISSUES/065-a-permanently-failing-resource-is-retried-for-ever-with-no-escalation/00-report.md)).
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **The host gives up** after some number of attempts. Rejected: the host does not know whether
|
||||
a failure is permanent — that a registry has not answered three times is not evidence it never
|
||||
will — and a host that stops trying is a node that must be pushed to again by hand.
|
||||
2. **A duration** since the failure was first seen. Rejected as the signal: a laptop shut for a
|
||||
week has had one attempt, and a week is not evidence of anything.
|
||||
3. **The controller counts identical reports**, and says when there are enough of them. Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
The mesh keeps, beside each machine's last report, when the current failure was first reported
|
||||
and how many reports in a row have said it — the same outcome, the same refusal, the same failed
|
||||
resources by id. Not by the host's words: an error carrying a duration or a counter would read as
|
||||
new on every report, and the resource looping on it is exactly what this is for. A report that
|
||||
says something different starts the count again; a clean apply clears it. **Three identical reports in a row make a machine stuck**: `status` says
|
||||
so beside the failure, with the count and the time it began, and the machine-readable status
|
||||
carries the same three facts. The host keeps retrying; being stuck is a statement about the
|
||||
mesh's knowledge, not an instruction to the machine.
|
||||
|
||||
The controller counts rather than the host, because only it sees every node: one stuck machine
|
||||
and a mesh-wide fault are different situations, and the host cannot tell them apart.
|
||||
|
||||
## Consequences
|
||||
|
||||
A resource that will never apply is visible from `status` after three reconcile intervals, to
|
||||
anyone who looks, without being asked for. What got harder: nothing on the machine changes — a
|
||||
gating failure that stops what follows still stops it, and this only makes the wait visible.
|
||||
Whether a stuck machine should also be raised as an event, and what a gating failure should do,
|
||||
stay open in the issue's own questions.
|
||||
|
||||
## How it is checked
|
||||
|
||||
An inventory test records the same failure three times and asserts the count and the unchanged
|
||||
start; the same resource failing in other words, and asserts the count went on; a different
|
||||
failure, and asserts it restarted; a clean apply, and asserts both cleared. The status command's own test asserts a stuck machine is said to be one.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 065](../04-ISSUES/065-a-permanently-failing-resource-is-retried-for-ever-with-no-escalation/00-report.md)
|
||||
- [ADR 0010](0010-delivery.md)
|
||||
- [`03-DESIGN/01-to-be/10-delivery.md`](../03-DESIGN/01-to-be/10-delivery.md)
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0051-shared-data-is-the-operators.md
|
||||
---
|
||||
|
||||
# 91. A mount is declared, and there are three things it can be
|
||||
|
||||
## Context
|
||||
|
||||
A container's bind mount whose source does not exist is created by the container runtime, as
|
||||
root, with whatever mode it picks. So `owner` and `mode` — which exist so a module can say who
|
||||
its data belongs to — never reach the directories that hold data, and the rule that keeps a
|
||||
directory when a module goes away ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md))
|
||||
does not cover them, because the mesh has never heard of them
|
||||
([issue 026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md)). Fourteen
|
||||
such mounts were declared by hand; a check that every mount is declared was then written and
|
||||
withdrawn, because it refused the builder: the builder mounts the container runtime's socket,
|
||||
which is not its data, already exists, and belongs to the machine. Declaring it as the module's
|
||||
own directory would be a lie the host would act on.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Declare the socket as a directory anyway.** Rejected: the host would create, own and
|
||||
protect a path that is the machine's.
|
||||
2. **A new manifest field** naming machine paths a module may mount. Rejected: the manifest
|
||||
already says the module needs the container runtime, and a second field would say the same
|
||||
thing in paths.
|
||||
3. **Three declarations, one for each kind of path a mount can be.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A container may not mount a path the module never declared, and a path is declared in one of
|
||||
three ways, which are the three things a path can be:
|
||||
|
||||
- **the module's own** — a directory or file resource, or where a secret, a grant or a
|
||||
contribution lands. Created and owned by the mesh for this module, kept when the module goes;
|
||||
- **the operator's** — an `accesses` entry ([ADR 0051](0051-shared-data-is-the-operators.md)):
|
||||
pre-existing, shared, granted for use, never owned;
|
||||
- **the machine's** — a facility a declared capability grants. `container-runtime` grants its
|
||||
socket. The path exists, the machine owns it, and the capability is the declaration.
|
||||
|
||||
A mount under a declared directory is declared. The check runs where the manifest is parsed,
|
||||
and names the path and the three remedies.
|
||||
|
||||
## Consequences
|
||||
|
||||
Every directory that holds a module's data is one the mesh created with the module's owner and
|
||||
mode, and one ADR 0030 protects. What got harder: a manifest borrowed from a compose file no
|
||||
longer passes on the strength of its volume lines; each must say what kind of path it mounts.
|
||||
The table of what a capability grants is small and in the catalogue's parser; a new capability
|
||||
that grants a path adds a row.
|
||||
|
||||
## How it is checked
|
||||
|
||||
Manifest tests refuse an undeclared mount, accept one under a declared directory, accept one
|
||||
the module accesses, and accept the runtime's socket with the capability and refuse it without.
|
||||
A test parses every manifest in the catalogue beside the checkout and fails on any that breaks
|
||||
the rule, so the catalogue cannot drift back.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md)
|
||||
- [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md), [ADR 0051](0051-shared-data-is-the-operators.md)
|
||||
- [`03-DESIGN/01-to-be/18-building-a-module.md`](../03-DESIGN/01-to-be/18-building-a-module.md)
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0085-a-secret-is-a-provision.md
|
||||
---
|
||||
|
||||
# 92. An operator delivers a pair credential, and the mesh never replaces it
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0085](0085-a-secret-is-a-provision.md) names three species of secret and gives the vault
|
||||
two of them: a module's own secret, which the mesh mints, and an operator-delivered secret — a
|
||||
credential for something outside the mesh, which only a person can supply. Under 0085 a secret
|
||||
from the vault is a pair credential between the consumer and the vault. The controller's one
|
||||
command that takes a value from a person wrote only a module's own secret, sealed to one node.
|
||||
Nothing could put a value into a pair, so the third species had no entry
|
||||
([issue 070](../04-ISSUES/070-an-operator-cannot-deliver-a-pair-credential/00-report.md)), and
|
||||
an operator's credential could be held only as an own secret — un-audited, un-rotatable, the
|
||||
gap 0085 opened to close.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A new verb on the pair.** Rejected: `secret accept` already means "a value a person
|
||||
supplied, sealed on the way in, plaintext discarded"; a second verb would mean the same.
|
||||
2. **`secret accept` grows a provider end.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
`secret accept <node> <module> <name> --provider <node>` seals the supplied value to the
|
||||
consumer's node, to the provider's node, and to the operator's key when the mesh has one, and
|
||||
records the pair as `accepted`. Every pair credential now says where it came from: `made` or
|
||||
`accepted`.
|
||||
|
||||
An accepted pair is never replaced by a made one. When a sealing key at either end changes, the
|
||||
mesh cannot re-seal a value it does not hold, so the read is refused and names the remedy —
|
||||
accept it again. `rotate` refuses an accepted pair for the same reason: the mesh cannot make its
|
||||
replacement, and deleting it would have the next read mint one, delivered and reported as
|
||||
applied while failing to authenticate somewhere else entirely. Rotating an accepted credential
|
||||
is accepting a new value.
|
||||
|
||||
## Consequences
|
||||
|
||||
A credential for something outside the mesh lives in the vault's ledger with the others, sealed
|
||||
to both ends and recoverable by the operator. What got harder: a mesh whose node keys change
|
||||
cannot heal an accepted pair by itself; a person is asked. That is the honest shape — the value
|
||||
was never the mesh's to make.
|
||||
|
||||
## How it is checked
|
||||
|
||||
An inventory test accepts a value into a pair, reads it back twice unchanged with origin
|
||||
`accepted`, asserts `rotate` refuses it naming the remedy while a made pair still rotates;
|
||||
another changes a node's key and asserts the read is refused, then accepts again and reads.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 070](../04-ISSUES/070-an-operator-cannot-deliver-a-pair-credential/00-report.md), [issue 069](../04-ISSUES/069-one-secret-provision-yields-one-value/00-report.md)
|
||||
- [ADR 0085](0085-a-secret-is-a-provision.md)
|
||||
- [`03-DESIGN/01-to-be/24-the-secrets-vault.md`](../03-DESIGN/01-to-be/24-the-secrets-vault.md)
|
||||
@@ -84,6 +84,10 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0001** — [The mesh brokers capabilities; nodes host; agents think](0001-mesh-brokers-nodes-host-agents-think.md)
|
||||
- **0002** — [Nodes communicate over a message broker, not over HTTP](0002-nodes-communicate-over-a-broker.md)
|
||||
- **0003** — [An agent is a persistent employee, not an instance of a pool](0003-agents-are-persistent-employees.md)
|
||||
- **0077** — [The parts are named controller, foundation, node — not control plane, substrate, master](0077-the-controller-and-the-foundation.md)
|
||||
- **0083** — [One push leaves the mesh consistent](0083-one-push-leaves-the-mesh-consistent.md)
|
||||
- **0088** — [The foundation filters before anything listens](0088-the-foundation-filters-before-anything-listens.md)
|
||||
- **0090** — [A failure that repeats is said to be stuck](0090-a-failure-that-repeats-is-said-to-be-stuck.md)
|
||||
|
||||
### Its tiers, from the bottom up
|
||||
|
||||
@@ -92,11 +96,53 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0006** — [The substrate and the control plane](0006-the-substrate-and-the-control-plane.md)
|
||||
- **0007** — [Connectivity](0007-connectivity.md)
|
||||
- **0008** — [A context owns its store, exclusively](0008-a-context-owns-its-store.md)
|
||||
- **0028** — [The substrate supplies the control plane and nothing else](0028-the-substrate-supplies-the-control-plane-and-nothing-else.md)
|
||||
- **0029** — [A network is a shape, because an action cannot be undone](0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md)
|
||||
- **0030** — [Data outlives the mesh that declared it](0030-data-outlives-the-mesh-that-declared-it.md)
|
||||
- **0031** — [The control plane authenticates nobody, so identity is a module](0031-the-control-plane-authenticates-nobody.md)
|
||||
- **0033** — [The substrate is a store and a broker](0033-the-substrate-is-a-store-and-a-broker.md)
|
||||
- **0036** — [Bootstrap ends at a usable mesh, and the first credential comes from a person](0036-bootstrap-ends-at-a-usable-mesh.md)
|
||||
- **0066** — [Public routing is name-agnostic, its names are resolved inside the mesh, and an internal authority can certify them](0066-public-routing-is-name-agnostic.md)
|
||||
- **0067** — [Genesis is a pivot: a temporary control plane installs the registry that makes it permanent](0067-genesis-is-a-pivot.md)
|
||||
- **0070** — [The catalogue owns the module graph, and genesis builds rather than carries](0070-the-catalogue-owns-the-module-graph.md)
|
||||
- **0071** — [Genesis clones from a mesh, and checks what it got](0071-where-genesis-gets-its-source.md)
|
||||
- **0072** — [Two graphs, and a build chain that orders itself](0072-two-graphs-and-the-build-chain.md)
|
||||
- **0073** — [The installer carries a builder, and the registry stays where it is](0073-the-installer-carries-a-builder.md)
|
||||
- **0074** — [The mesh defines a module protocol; an SDK is an implementation of it](0074-the-wire-is-specified-not-the-types.md)
|
||||
- **0075** — [An artifact store is a provision; a package registry is a different one](0075-two-stores-and-which-provides-what.md)
|
||||
- **0078** — [The store and the broker are ordinary modules](0078-the-store-and-broker-are-modules.md)
|
||||
- **0079** — [The foundation seats are named after their servers](0079-the-foundation-seats-are-named-after-their-servers.md)
|
||||
- **0092** — [An operator delivers a pair credential, and the mesh never replaces it](0092-an-operator-delivers-a-pair-credential.md)
|
||||
|
||||
### What runs on them, and how it gets there
|
||||
|
||||
- **0009** — [Modules and the graph](0009-modules-and-the-graph.md)
|
||||
- **0010** — [Delivery](0010-delivery.md)
|
||||
- **0024** — [Model access is a provision, and a licence is a thing with a name](0024-model-access-is-a-provision.md)
|
||||
- **0026** — [The mesh has a session of its own, and it is the node session's mechanism](0026-the-mesh-has-a-session-of-its-own.md)
|
||||
- **0027** — [A provision names what the consumer is coupled to, not the role it plays](0027-a-provision-names-what-the-consumer-is-coupled-to.md)
|
||||
- **0035** — [One implementation, several surfaces, and what that costs](0035-one-implementation-several-surfaces.md)
|
||||
- **0038** — [The mesh assigns the port, and a module does not care](0038-the-mesh-assigns-the-port.md) *(proposed)*
|
||||
- **0040** — [What a module is](0040-what-a-module-is.md)
|
||||
- **0041** — [Events are a relationship, the lighter sibling of provisioning](0041-events-are-a-relationship.md)
|
||||
- **0042** — [The shape of an event on the wire](0042-the-shape-of-an-event-on-the-wire.md)
|
||||
- **0043** — [A module's broker account is scoped by what it emits and consumes](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md)
|
||||
- **0044** — [A public name is provisioned, not registered by hand](0044-a-public-name-is-provisioned-like-any-capability.md)
|
||||
- **0045** — [A machine's firewall is the sum of what its modules listen on](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md)
|
||||
- **0046** — [A module's configuration is its assignment's, not its manifest's](0046-a-module-configuration-is-its-assignments-not-its-manifest.md)
|
||||
- **0047** — [A module runs its code as its own process, with its own account](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)
|
||||
- **0048** — [A provider creates the credential the mesh minted, and seals nothing](0048-a-provider-creates-the-credential-the-mesh-minted.md)
|
||||
- **0049** — [A consumer's identity is bounded by the tightest backend that must accept it](0049-a-consumers-identity-fits-the-tightest-backend.md)
|
||||
- **0050** — [Model access is vendor-agnostic, and a vendor is an adapter](0050-model-access-is-vendor-agnostic.md)
|
||||
- **0051** — [Shared data is the operator's, and a module is granted access to it](0051-shared-data-is-the-operators.md)
|
||||
- **0052** — [An init step is a container run once to completion, gating what follows](0052-a-step-that-runs-once-before-a-container.md)
|
||||
- **0053** — [A scheduled step is a container run on a recurring schedule](0053-a-step-that-runs-on-a-schedule.md)
|
||||
- **0054** — [Model usage is a vendor-neutral record, produced by the adapter, at two grains](0054-model-usage-is-recorded-at-two-grains.md)
|
||||
- **0055** — [Model access is answered by a licence, or by a node that hosts the model](0055-model-access-is-answered-by-a-licence-or-a-node.md)
|
||||
- **0084** — [Which provider serves a consumer, when the mesh runs more than one](0084-which-provider-serves-a-consumer.md)
|
||||
- **0085** — [A secret is a provision, and the vault is the module that provides it](0085-a-secret-is-a-provision.md)
|
||||
- **0087** — [A seeded file is created once, and what grows in it is not the mesh's](0087-a-seeded-file-is-created-once.md)
|
||||
- **0091** — [A mount is declared, and there are three things it can be](0091-a-mount-is-declared-three-ways.md)
|
||||
|
||||
### How it is built
|
||||
|
||||
@@ -106,11 +152,19 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0014** — [No workspace — each module is a standalone package consuming published dependencies](0014-no-npm-workspace.md)
|
||||
- **0015** — [Applications live in their own repository; the monorepo is for the mesh](0015-applications-live-in-their-own-repository.md)
|
||||
- **0016** — [The lab](0016-the-lab.md)
|
||||
- **0037** — [Where a module lives](0037-where-a-module-lives.md) *(proposed)*
|
||||
- **0039** — [What the SDK holds, and what it refuses](0039-what-the-sdk-holds-and-refuses.md)
|
||||
- **0068** — [The lab takes requests, one at a time, and runs each from its own copy](0068-the-lab-takes-requests.md) *(proposed)*
|
||||
- **0069** — [A module is a repository and a path within it](0069-a-module-is-a-repository-and-a-path.md)
|
||||
- **0076** — [The SDK is a published package, and the toolchain resolves it by version](0076-the-sdk-is-a-published-package.md)
|
||||
- **0082** — [The registry is reached by name, and the overlay is its security](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)
|
||||
- **0086** — [A secret reaches a process as a file, and an exception is declared](0086-a-secret-reaches-a-process-as-a-file.md)
|
||||
|
||||
### How it is checked
|
||||
|
||||
- **0017** — [A test defends a decision](0017-a-test-defends-a-decision.md)
|
||||
- **0018** — [A picture of a system is read from the system, never from what asked for it](0018-a-picture-is-read-from-what-runs.md)
|
||||
- **0089** — [A bed reads the catalogue it proves](0089-a-bed-reads-the-catalogue-it-proves.md)
|
||||
|
||||
### How we work
|
||||
|
||||
@@ -119,5 +173,10 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0021** — [HQ is the source of the mesh constitution](0021-hq-is-the-source-of-the-constitution.md)
|
||||
- **0022** — [The constitution absorbs what is already enforced](0022-the-constitution-absorbs-what-is-enforced.md)
|
||||
- **0023** — [The approval is the checkpoint, not the second pair of hands](0023-approval-is-the-checkpoint.md)
|
||||
- **0025** — [The design record is read where it is written, never copied to be found](0025-the-design-record-is-read-not-copied.md)
|
||||
- **0032** — [The local account owns the mesh; a surface delegates to a module](0032-the-local-account-owns-the-mesh.md) *(superseded)*
|
||||
- **0034** — [The local account owns the mesh, and a web application's login is not that](0034-the-local-account-owns-the-mesh.md)
|
||||
- **0080** — [The development cycle is checked, not trusted](0080-the-development-cycle-is-checked.md)
|
||||
- **0081** — [A decision nothing cites is not yet in the chain](0081-a-decision-nothing-cites-is-not-yet-in-the-chain.md)
|
||||
|
||||
<!-- index:end -->
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
---
|
||||
layer: as-is
|
||||
status: implemented
|
||||
code: [hal]
|
||||
updated: 2026-08-23
|
||||
code: [hal, mesh-controller, mesh-catalog, mesh-host]
|
||||
updated: 2026-09-21
|
||||
decisions:
|
||||
- 02-DECISIONS/0011-managed-files-are-generated-never-edited.md
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0085-a-secret-is-a-provision.md
|
||||
---
|
||||
|
||||
# Configuration and secrets
|
||||
@@ -73,9 +74,31 @@ when the file is created, so regeneration left the previous mode in place. The c
|
||||
worth remembering beyond the instance: a permission set at creation is not a permission
|
||||
maintained.
|
||||
|
||||
**Rotation is not a mesh operation.** Secrets can be generated and granted; there is no
|
||||
mechanism that rotates one and informs everything holding it. Where a rotation has been done,
|
||||
it has been done by hand, and doing it wrong has taken services down.
|
||||
**Rotation was not a mesh operation** in the mesh being replaced, and doing it by hand took
|
||||
services down. On the mesh that exists now it is: `rotate <provision>` discards a pair credential
|
||||
and delivers both ends in one send, and a module's own secret is such a pair credential when the
|
||||
module takes it from the vault — which, at the time of writing, one module does (redis).
|
||||
|
||||
## The vault, as it runs
|
||||
|
||||
Since 2026-09-21 the mesh runs `mesh-vault`, a foundation module installed at genesis beside the
|
||||
adopted store and broker. It provides `secret`: a module that requires one receives a pair
|
||||
credential the controller minted, and the vault's ledger records the holder and the value's
|
||||
fingerprint, notices a rotation, and answers over the mesh by fingerprint only. It holds no value.
|
||||
|
||||
Genesis makes the store's superuser and the broker's administrator rather than copying the
|
||||
template's, keeps them at the paths the store and broker modules declare as their own secrets, and
|
||||
makes an **operator sealing key** before the first secret is accepted: its private half is a file
|
||||
beside the produced bundle, which the operator carries off the machine, and its public half is
|
||||
what the mesh records. Every secret a module holds for itself and every pair credential is sealed
|
||||
to that key as well as to its node. The export of those copies is written beside the key at the
|
||||
end of genesis and kept by the vault on its own disk; the operator recovers any secret from it, off
|
||||
the mesh, with `secret recover`. Secrets made before the key existed, or sealed to a replaced key,
|
||||
are listed as such rather than passed off as recoverable. The produced bundle and the installer's
|
||||
transcript carry no credential in the clear.
|
||||
|
||||
Two things this does not yet do: a module with several own secrets cannot take them all from the
|
||||
vault (issue 069), and an operator cannot hand a value into a pair credential (issue 070).
|
||||
|
||||
## Node-level and mesh-level values
|
||||
|
||||
|
||||
@@ -1,158 +1,430 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: [hal]
|
||||
updated: 2026-08-23
|
||||
decisions: [02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md]
|
||||
code: []
|
||||
updated: 2026-09-01
|
||||
decisions:
|
||||
- 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0016-the-lab.md
|
||||
- 02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md
|
||||
---
|
||||
|
||||
# Work breakdown — the decomposition
|
||||
# Work breakdown — replacing what provisions the mesh
|
||||
|
||||
How [ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md) gets built, in what order, and where a human must look.
|
||||
*Rewritten 2026-08-31. The previous version planned a decomposition of the existing system in
|
||||
place: extract contexts, convert modules to declared features, shrink its shared library. That is
|
||||
not what is being done — a replacement is being built beside it, and the old plan's Phase 0 was
|
||||
the only part that survived contact with it. So the document that was supposed to say what happens
|
||||
next had been describing work on a system being retired.*
|
||||
|
||||
Ordering is not preference. Each phase removes a constraint the next one needs gone.
|
||||
## The goal, in one sentence
|
||||
|
||||
---
|
||||
**Modules move to the new mesh one at a time, until the old registry can be switched off.**
|
||||
|
||||
Everything below is ordered by what that requires. Nothing here is a rewrite of the old system;
|
||||
its modules are the input.
|
||||
|
||||
## Phase 0 — a mesh that runs — **done**
|
||||
|
||||
Not *the code exists*. Twenty-two assertions on real machines in the lab, each confirmed to fail
|
||||
when the behaviour is removed ([ADR 0016](../../02-DECISIONS/0016-the-lab.md),
|
||||
[ADR 0017](../../02-DECISIONS/0017-a-test-defends-a-decision.md)).
|
||||
|
||||
| what is proven | |
|
||||
|---|---|
|
||||
| **a mesh comes into being** | a bare machine becomes one; others join with nothing but a token |
|
||||
| **credentials** | delivered to both ends with the mesh holding neither; rotated so the old one stops working |
|
||||
| **declarations survive reality** | a stopped machine is waited for; one that fell behind catches up unnamed; unassigning takes away exactly what it should; what the mesh says nothing about is left alone |
|
||||
| **failure is legible** | a machine that cannot do what it was told is named, with why |
|
||||
| **the mesh runs itself** | its own artifact store, and a builder that is a module the mesh assigns |
|
||||
| **names and reachability** | internal names, wildcards under a machine, containers reaching other machines, certificates the mesh issued, filtering that matches exactly what was declared |
|
||||
| **delivery** | a new commit reaches a machine already running the old one |
|
||||
| **model access** | answered by a record, with a key the mesh cannot read |
|
||||
|
||||
**What Phase 0 does not prove, and it is the important sentence in this document:** every module
|
||||
exercised above was written to test the mechanism. **No module from the existing system has ever
|
||||
run on this.** The vocabulary was shaped by the things used to test it — the same fault as a
|
||||
fixture agreeing with the code it checks
|
||||
([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)), at the
|
||||
scale of a design.
|
||||
|
||||
## Phase 1 — the vocabulary a real module needs
|
||||
|
||||
Found by taking real modules and asking what they would require. Each is a gap in what can be
|
||||
*expressed*, not a defect in what is built.
|
||||
|
||||
| # | task | done when |
|
||||
|---|---|---|
|
||||
| ~~1.1~~ | ~~An **object-store provision**~~ — **done 2026-08-31**, and it needed no change to the mesh: see below | seven assertions against a real store |
|
||||
| ~~1.2~~ | ~~**A session as a consumer of a licence**~~ — **done 2026-08-31**, and it also needed no change: see below | two sessions on one machine, different licences, each its own key |
|
||||
| ~~1.3~~ | ~~A **network** shape, and ordering within a module~~ — **done 2026-08-31** | the shape is created and removed; ordering was already there, and is now asserted |
|
||||
| 1.4 | **Public certificate issuance** — **built; one gap** | ordering, the challenge and issuance are proven against a real authority; **collecting the issued certificate is not** ([`04-ISSUES/020`](../../04-ISSUES/020-a-certificate-is-issued-and-never-collected/00-report.md)) |
|
||||
|
||||
**1.3 and 1.4 block later ones** and are listed now so they are not met as surprises. 1.3 is what
|
||||
a mail system needs and nothing else so far does.
|
||||
|
||||
**Phase 1 is closed with 1.4 partly open**, deliberately. Two of its four items needed no code at
|
||||
all; the network shape was built; and certificates are configured correctly, order correctly, and
|
||||
are issued correctly — the client does not collect what the authority issued, against a server
|
||||
that exists to be a test server. That is filed rather than chased, because the remainder may say
|
||||
nothing about a real authority and the next thing to learn comes from moving a module rather than
|
||||
from a fourth lab run.
|
||||
|
||||
**Checkpoint:** each is demonstrated in the lab before the module needing it is attempted.
|
||||
|
||||
### 1.2, and the same surprise twice
|
||||
|
||||
**A binding is per module per machine, and the two sessions are two modules** — the same mechanism
|
||||
in different context roots, and a context root is what a module delivers. So `(node, module)`
|
||||
already names them apart, and nothing needed adding.
|
||||
[`14-model-access.md`](14-model-access.md) had called per-module-per-machine *a step toward it and
|
||||
not it*, which is true of a **worker** — many run on one machine from one module — and not true of
|
||||
a session, of which there is one per node and one for the mesh.
|
||||
|
||||
### 1.3, and the first one that needed building
|
||||
|
||||
**Ordering was already there** — the apply loop sorts nothing, so a module says *this before that*
|
||||
by writing it first. Untested until now, and the kind of property a later change breaks silently.
|
||||
Worth separating from readiness: a container started is not a container ready, and nothing waits.
|
||||
What needs something *usable* retries, which is what both provisioners do and is the better answer
|
||||
anyway, because a dependency can restart long after everything was applied.
|
||||
|
||||
**The network was a real gap, and the first thing in Phase 1 that needed a decision.** Adding a
|
||||
shape widens what a compromised controller can express, so
|
||||
[ADR 0029](../../02-DECISIONS/0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md)
|
||||
records why this one is worth it: an `action` could create a network and **nothing could ever
|
||||
remove it**, because an action leaves no footprint the host can undo. The vocabulary is nine.
|
||||
|
||||
**Three tasks in a row that were already possible.** Both were written from the design rather than
|
||||
from the code, which is the review's finding arriving in the plan: *a claim here is counted, not
|
||||
reasoned.* The remaining Phase 1 items should be checked against the code before being started,
|
||||
not after.
|
||||
|
||||
### 1.1, and what it turned out to be
|
||||
|
||||
*Done 2026-08-31. Worth recording because the task was not the one written down.*
|
||||
|
||||
**The controller special-cases nothing.** `provides`, `requires`, `contributes` and `grants`
|
||||
are name-agnostic — asking for a bucket needed no change to the mesh at all. What was missing was
|
||||
a provider, and the last step where something on the machine turns a delivered secret into a key
|
||||
that works. So "add an object-store provision" was never mesh work.
|
||||
|
||||
The provision is `s3-bucket`: a consumer's code is written against the S3 API and swapping one
|
||||
store for another does not break it, so by
|
||||
[ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md) the name
|
||||
says the protocol. A database is the other case, and names the engine.
|
||||
|
||||
**One assertion here that a database does not need.** One PostgreSQL server holds separate
|
||||
databases and the product enforces the boundary; one object store holds every bucket behind one
|
||||
endpoint, so *a consumer cannot reach another consumer's bucket* is a policy somebody wrote — and
|
||||
a policy granting everything would pass every other test. **What is asserted is what the policy
|
||||
does not say.**
|
||||
|
||||
## Data is the constraint, and it outranks the order below
|
||||
|
||||
*2026-08-31.* The modules being converted run live services — identity, mail — and **the data must
|
||||
survive every step**. A data folder may move; it may never be lost.
|
||||
|
||||
**One thing was found by asking this and is fixed**
|
||||
([ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md)): the host deleted
|
||||
a directory and everything under it when the directory stopped being declared, which happens when
|
||||
a module is unassigned or a manifest is edited to move a data folder — the exact operation this
|
||||
plan needs. A directory holding anything the mesh did not put there is now kept and reported.
|
||||
|
||||
**That is not a backup and must not be read as one.** It stops the mesh destroying data. It does
|
||||
nothing about a disk, a mistaken command, or a service corrupting its own store.
|
||||
|
||||
**So the rule for every step below:** the data is copied, the copy is verified by reading it back
|
||||
through the service that owns it, and only then does anything point at the new location. Never
|
||||
moved and then checked. **A backup nobody has restored is a belief, not a copy.**
|
||||
|
||||
## A module is adopted with the credentials it already has
|
||||
|
||||
*2026-08-31.* **Nothing is rotated during the conversion.** A service being adopted keeps the
|
||||
password it is already using, because minting a new one is how a running service stops being able
|
||||
to reach its own database in the middle of a migration.
|
||||
|
||||
The mesh has both paths and this needs the second:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **generate** | a new secret, sealed to both ends. What a *new* module gets |
|
||||
| **accept** | a value supplied from outside, sealed, plaintext discarded. **What an adopted module gets** |
|
||||
|
||||
**Rotation is a separate act, afterwards, once everything works.** The machinery for it is built
|
||||
and proven — a credential moving at both ends with the old one ceasing to work — and it is exactly
|
||||
the sort of thing to do deliberately on a quiet afternoon rather than as a side effect of moving a
|
||||
service between systems.
|
||||
|
||||
**So there is a step before any of this: read the current environment out of the old system**, because
|
||||
adoption means supplying those values and they live in its files today.
|
||||
|
||||
**And there is a failure worse than losing data, which is likelier.** A database image consumes its
|
||||
password environment variable **only when its data directory is empty**. Everything here keeps its
|
||||
data on a persistent directory, so the role holds whatever password it was created with, for ever.
|
||||
Regenerate that variable and the application moves on while the database does not — permanently,
|
||||
because nothing reconciles it. Eight modules are in that state today, working only because nobody
|
||||
has regenerated their credential since their data directory was created.
|
||||
|
||||
*Where the detail lives:* this is operational and names machines, so it is in the mesh's own
|
||||
knowledge base rather than here — `migration/where-service-data-lives`, which surveys where every
|
||||
service's data actually sits and what each stop or removal would cost, and
|
||||
`troubleshooting/db-password-frozen-at-first-init` for the lockout itself. **This document says the
|
||||
rule; those say the specifics.**
|
||||
|
||||
*Corrected 2026-08-31 — an earlier version of this paragraph made that sound more dangerous than it
|
||||
is.* A sealed secret is not unreadable; it is sealed **to the node**, which holds the private half
|
||||
and writes the plaintext into the module's own file. The value is there, on the machine, as an
|
||||
ordinary file. What does not exist is a way to ask *the mesh* what a secret is, and there is no
|
||||
reveal command, because a mesh that can reveal a secret is a mesh that holds one.
|
||||
|
||||
## Where it starts, and what that costs
|
||||
|
||||
**On the node holding all the production data**, because that is where the services being
|
||||
converted actually are.
|
||||
|
||||
Recorded plainly rather than argued with: this is the highest-risk order available. Everything
|
||||
proven so far was proven on machines that could be destroyed and raised again, and the first real
|
||||
exercise of the conversion will be on the one machine where a mistake is not recoverable. Nothing
|
||||
about the lab work transfers automatically — a scenario proves the mechanism, not the state on
|
||||
that machine.
|
||||
|
||||
**What makes it survivable is preparation rather than caution**: a restored backup before the
|
||||
first step, one service at a time, and the previous arrangement left standing until the new one
|
||||
has been read back. None of that is slower than the alternative, because the alternative includes
|
||||
losing something.
|
||||
|
||||
## How the two systems hand over
|
||||
|
||||
*2026-08-31.* **The old system's brain is switched off; its services keep running.**
|
||||
|
||||
Not a migration and not a period of dual control. The old controller — provisioning, the
|
||||
coordinator, the pipeline, the things that *decide* and *write* — is stopped. Every workload it
|
||||
was managing goes on running exactly as it is, because nothing is managing it. Then the new mesh
|
||||
takes ownership of them one at a time.
|
||||
|
||||
**Nothing is ever unassigned in the old system.** Unassigning is how it removes things, and
|
||||
removing is how data is lost. The old system is never asked to take anything away; it is asked to
|
||||
stop having opinions.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **stopped, and disabled** | provisioning, the coordinator, environment and configuration sync, the pipeline — anything that decides or writes a file |
|
||||
| **left alone entirely** | the units running the actual services: identity, mail, databases, the forge. They keep serving throughout |
|
||||
| **never used** | unassign, remove, delete — any operation whose job is to take something away |
|
||||
|
||||
**Disabled, not merely stopped**, and this is the part that is easy to get wrong: those units are
|
||||
enabled, so stopping them lasts until the machine reboots. A reboot mid-conversion would bring the
|
||||
old controller back and it would resume regenerating managed files underneath the new one —
|
||||
which is the one situation where two systems really would be fighting over the same machine.
|
||||
|
||||
**A service left running with nothing managing it is the safe state.** It has its data, its
|
||||
configuration is already on disk, and nothing is going to change either. That is the whole trick:
|
||||
the risk in a conversion is in the *managing*, not in the *running*.
|
||||
|
||||
**A brief interruption is acceptable. Losing data is not.** Where those two trade against each
|
||||
other, the interruption wins every time — a service can be restarted, and there is no operation
|
||||
that un-deletes a mail spool.
|
||||
|
||||
**The new host cannot remove what it did not put there.** Orphans are per-origin, so it only ever
|
||||
removes resources it recorded itself. Services it has never been told about are not orphans to
|
||||
it — they are simply not its business, which is what makes taking ownership one module at a time
|
||||
safe.
|
||||
|
||||
## Phase 2 — the first real module
|
||||
|
||||
| # | task | done when |
|
||||
|---|---|---|
|
||||
| 2.1 | Port an **object store** module | it runs on the new mesh, serves a bucket to another module, and its credential rotates |
|
||||
| 2.2 | Copy the data, and read it back through the service that owns it | the new location answers with what the old one holds |
|
||||
| 2.3 | Point one dependent at it, old arrangement left standing | something real reads and writes through the new mesh's copy |
|
||||
|
||||
**Checkpoint, and it is a human one:** it runs for a week before anything else moves. The point of
|
||||
going first is to find what Phase 1 missed, and a week is roughly how long that takes to show.
|
||||
|
||||
## Phase 3 — the modules that prove the shape
|
||||
|
||||
Each exercises something the first one does not.
|
||||
|
||||
| # | task | proves |
|
||||
|---|---|---|
|
||||
| 3.1 | An **identity provider** | a module that is itself a provider — the provides/requires chain, with consumers requiring it |
|
||||
| 3.2 | A **forge** | a port claim against the machine's own daemon, and a module wanting both a database and an object store |
|
||||
| 3.3 | A **mail system** | several containers as one module, a private network between them, and names that are not one-per-node |
|
||||
|
||||
**3.3 is the hardest thing in this document** and is deliberately last. If the declaration
|
||||
language turns out to be insufficient, it says so here.
|
||||
|
||||
### Where Phase 3 actually stands — *2026-09-01*
|
||||
|
||||
All three have manifests. All three parse, resolve and plan. **None of them can start**, and the
|
||||
two reasons are both filed rather than guessed at.
|
||||
|
||||
The **vocabulary held**. Nothing in 3.1–3.3 turned out to need a new shape: the identity provider,
|
||||
the forge and the mail system are all expressible with what exists, including the mail system's
|
||||
several containers on a private network — which was the one expected to break it. That is the
|
||||
question this phase was designed to answer, and the answer is yes.
|
||||
|
||||
What did not hold was underneath the vocabulary:
|
||||
|
||||
- **[`022`](../../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md)**
|
||||
(fixed) — a credential belonged to a machine, so a node running several modules against one
|
||||
database could not be planned. The refusal was loud on the provider and silent on the consumer.
|
||||
- **[`023`](../../04-ISSUES/023-a-consumer-cannot-build-a-connection-string/00-report.md)** (open)
|
||||
— a consumer gets its password and still cannot connect: the user name is invented by the
|
||||
provisioner and recorded nowhere, and the bound values cannot reach a configuration file.
|
||||
|
||||
A third fault was in the manifests themselves rather than the design: each declared a secret at a
|
||||
path named `.env` and read it as one, when a sealed file holds a password and nothing else. They
|
||||
parsed and resolved and could never have worked, which is what a manifest checked only by a parser
|
||||
buys. Two tests now refuse both halves of it.
|
||||
|
||||
### 3.1 needs a program, not a decision — *2026-09-01*
|
||||
|
||||
023 is fixed, and with it the two design faults are gone. What stands between 3.1 and a running
|
||||
identity provider is now one concrete thing: **the realm provisioner does not exist.**
|
||||
|
||||
Its manifest named an image — `mesh-provision-keycloak` — that nothing builds and no program
|
||||
backs. That has been removed rather than left standing, because a manifest describing a program
|
||||
nobody wrote is the same mistake as the credential files that could never be read: it parses, it
|
||||
resolves, and it could never work.
|
||||
|
||||
So Keycloak's manifest now says what is true today — a server the mesh runs, with its database
|
||||
and its admin credential, both reaching it in a shape it can read. It no longer claims to provide
|
||||
`oidc-client`, which means a consumer asking for one is **refused by name at plan time** rather
|
||||
than resolving cleanly and waiting for a client nothing will create.
|
||||
|
||||
The provisioner is the same shape as the two that exist: it reads what the mesh granted and
|
||||
reconciles a realm and a client per consumer. **It should be written against a real Keycloak in
|
||||
the lab**, not from the API documentation — the object store's took three corrections that only a
|
||||
running server produced.
|
||||
|
||||
The forge (3.2) and the mail system (3.3) need no provisioner and are not blocked on this.
|
||||
|
||||
### And they could not have run anyway — *2026-09-01*
|
||||
|
||||
Every one of the five named a container image that does not exist: sixty-four zeros where a digest
|
||||
belongs, eighteen times over
|
||||
([`025`](../../04-ISSUES/025-a-module-must-pin-a-digest-and-nothing-produces-one/00-report.md)).
|
||||
They parsed, resolved and composed into a declaration a host accepts, and every one would have
|
||||
stopped on the machine at the moment of fetching.
|
||||
|
||||
Nothing caught it because nothing could. A host checks the *shape* of a reference and no more —
|
||||
verifying a digest exists means reaching a registry, which is the one thing a host must never have
|
||||
to do. The refusal now sits where a declaration is composed instead, which is the last moment
|
||||
before a machine sees one.
|
||||
|
||||
Twelve are pinned to real images. Two further faults surfaced only by pinning for real: the mail
|
||||
system's seven images named repositories that **do not exist**, because it publishes to a
|
||||
different registry than assumed, and one of the seven had been renamed upstream.
|
||||
|
||||
**The forge now runs**, on a database another module provides, with a password it did not choose
|
||||
and a connection string it could not have written. That is the first of these descriptions to be
|
||||
started rather than planned, and it exercises everything the credential work added.
|
||||
|
||||
What is still missing is the mechanism: nothing turns a tag into a digest as part of the mesh's
|
||||
own work, so it was done by hand. Asking a registry takes about a second and pulls nothing, which
|
||||
removes the main argument for leaving it undone.
|
||||
|
||||
## The conversion is done by hand, and that is a decision
|
||||
|
||||
*2026-08-31.* **Moving from the current system to this one is a person at a command line, working
|
||||
through it.** Not a migration program, not a converter, not a period of dual-writing.
|
||||
|
||||
**What that removes from this plan is larger than what it adds.** Nothing below needs an importer,
|
||||
a translation layer, a compatibility shim, or a mechanism for keeping two systems agreeing while
|
||||
both are live — and every one of those is a thing somebody would otherwise reasonably build, use
|
||||
once, and maintain for a year. The modules are the input; a person reads what one does today and
|
||||
writes what it declares tomorrow.
|
||||
|
||||
**It also changes what "safe" means for the system being retired.** A fix to it has to be safe on
|
||||
its own, because there is no careful rollout to sequence it into: the thing is being switched off
|
||||
by hand, not managed into retirement. A change needing three steps in the right order is a change
|
||||
that will be half-applied.
|
||||
|
||||
**And it is why the checkpoints below are weeks rather than gates.** Nothing enforces the order —
|
||||
a person does — so the value of the sequence is entirely in what each step teaches before the next
|
||||
one starts.
|
||||
|
||||
## Phase 4 — switch the old registry off
|
||||
|
||||
| # | task | done when |
|
||||
|---|---|---|
|
||||
| 4.1 | Move the remainder, by hand, a module at a time | nothing is assigned in the old system that is not assigned in the new one |
|
||||
| 4.2 | The old one authoritative for nothing | a change to any module goes through the new mesh only |
|
||||
| 4.3 | Switch it off | it is stopped, and nothing notices |
|
||||
|
||||
**4.3 is a day's work and the phases above it are not.** Naming it as a phase is what stops it
|
||||
being mistaken for the goal.
|
||||
|
||||
## Sequencing
|
||||
|
||||
- **1 before 2.** Attempting a module without the vocabulary it needs produces a workaround, and a
|
||||
workaround in a manifest is a design decision taken by whoever was in a hurry.
|
||||
- **2 before 3, with the week.** Moving three modules before running one is how three modules
|
||||
acquire the same defect.
|
||||
- **3.3 last.** It is the only one that may send work back into the declaration language.
|
||||
- **4 cannot start early, and there is no partial credit.** A registry still authoritative for one
|
||||
module is still running.
|
||||
|
||||
## How this list is kept true
|
||||
|
||||
*This section exists because the document it replaces was wrong for weeks and nothing said so.*
|
||||
|
||||
**A claim here is counted, not reasoned.** The review of 2026-08-31 found a bundle described as
|
||||
carrying two images that carries three, a bootstrap described as needing six shapes that uses
|
||||
four, and ten documents calling themselves `designed` while naming lab-proven code. Each was
|
||||
produced by describing the system from its design instead of reading it.
|
||||
|
||||
**A phase is done when the lab says so**, and the lab keeps a receipt of when it last ran and
|
||||
against which commits. A phase marked done here whose assertions have not run is a claim about the
|
||||
past.
|
||||
|
||||
**What is not proven gets said.** Phase 0 is done and its limitation is written into it. A list
|
||||
that records only progress becomes a list nobody believes.
|
||||
|
||||
## Rules of engagement
|
||||
|
||||
These exist so the work can run largely unattended without accumulating the kind of
|
||||
damage this refactor is meant to remove.
|
||||
Unchanged from the previous version: they were about how work is done rather than what the work
|
||||
is.
|
||||
|
||||
### Autonomous by default
|
||||
|
||||
An agent may, without asking:
|
||||
|
||||
- read anything, measure anything, query any database read-only
|
||||
- create branches, write code and tests, open pull requests
|
||||
- run the test suite and typechecks
|
||||
- write and update `hq/` documents
|
||||
Read anything, measure anything, query read-only. Create branches, write code and tests, run the
|
||||
suites, and write or update documents here.
|
||||
|
||||
### Always stop and ask
|
||||
|
||||
- **destroying or overwriting data** — dropping a table, deleting a provision, rotating a
|
||||
live credential, removing a module from a node
|
||||
- **destroying or overwriting data** — dropping a table, deleting a provision, rotating a live
|
||||
credential, removing a module from a node
|
||||
- **merging anything** — every merge is a human checkpoint, without exception
|
||||
- **a decision the ADRs do not already answer** — record the question in the relevant
|
||||
research effort rather than picking and moving on
|
||||
- **any change to `hq/00-META`** — it is stable by nature
|
||||
- **anything touching a machine outside the lab**, including a configuration change that restarts
|
||||
something people are using
|
||||
- **a decision the records do not already answer** — record the question rather than picking and
|
||||
moving on
|
||||
- **any change to [`00-META`](../../00-META/)** — it is stable by nature
|
||||
|
||||
### Definition of done for every task
|
||||
|
||||
1. tests written **and failing first**, then passing
|
||||
2. typecheck clean in every package the change touches
|
||||
3. the local mesh (Phase 0) comes up, and the behaviour is demonstrated in it
|
||||
4. `hq/` updated if the task changed or answered anything documented
|
||||
5. deployed, and **delivery verified on every node** — not "the pipeline was green"
|
||||
3. the behaviour demonstrated **in the lab, on real machines** — not asserted
|
||||
4. documents here updated if the task changed or answered anything recorded
|
||||
5. delivered, and the **effect** verified — not that a pipeline was green
|
||||
|
||||
### Non-negotiables carried from the current system
|
||||
### Non-negotiables
|
||||
|
||||
- **Never edit mesh-managed files on disk.** Use the owning tool.
|
||||
- **Never write to production databases directly.** Migrations for schema, application
|
||||
code for data.
|
||||
- **Every schema change ships twice** — consolidated schema *and* an incremental
|
||||
migration.
|
||||
- **Expand, then contract.** Add the new shape, migrate, verify, and only then remove the
|
||||
old one — never in a single step.
|
||||
- **A green pipeline proves transport, not effect.** Verify the effect.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — A mesh that runs locally *(prerequisite)*
|
||||
|
||||
Nothing else starts until this exists. Every fault this refactor addresses was found in
|
||||
production because there was nowhere else to find it.
|
||||
|
||||
| # | task | done when |
|
||||
|---|---|---|
|
||||
| 0.1 | Container image for a node runtime | a node process starts in a container and registers |
|
||||
| 0.2 | Compose topology: broker, registry DB, object store, *n* nodes | `up` yields a mesh that elects a provider node and settles |
|
||||
| 0.3 | Seed a minimal mesh: nodes, one module, one provision | a module deploys end-to-end with no external service |
|
||||
| 0.4 | Run the pipeline inside it | a push-equivalent produces a cascade and a deployed artifact |
|
||||
| 0.5 | Fixtures for the failure modes already known | credential rotation reaching a running session; a provider deploy rotating a shared credential; a migration that ships nothing — each reproducible on demand |
|
||||
|
||||
**Checkpoint:** a human confirms the local mesh reproduces at least one bug from
|
||||
2026-08-22 before any decomposition begins.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Make the model expressible
|
||||
|
||||
The decomposition is impossible while a feature is a singleton per module.
|
||||
|
||||
| # | task | done when |
|
||||
|---|---|---|
|
||||
| 1.1 | Decision record — named features, per-node opt-in (next free number) | accepted |
|
||||
| 1.2 | Manifest: declared `features:` with type + directory | a module declares two of one kind and both build |
|
||||
| 1.3 | Selection: `always` / `optional` | a node installs a subset; artifacts stay selection-blind |
|
||||
| 1.4 | `requires:` moves onto the feature | a schema feature's database is not provisioned where the feature is not installed |
|
||||
| 1.5 | Assignment carries the opted-in feature set | opting a node in requires no rebuild |
|
||||
|
||||
**Checkpoint:** one existing module converted to declared features, deployed, verified —
|
||||
before any others follow.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Draw the boundary the domain already has
|
||||
|
||||
Cheapest first, and each one proves the extraction pattern before the expensive ones.
|
||||
|
||||
| # | task | extracted from | risk |
|
||||
|---|---|---|---|
|
||||
| 2.1 | `hal/knowledge` — one store, review workflow ported | hippocampus + noxflow `knowledge_*` | low — additive |
|
||||
| 2.2 | `hal/stream` — the record; notifications and messaging as views | axon, synapse, notifications, meetings, conversations | medium |
|
||||
| 2.3 | `hal/agents` — identity, licence, runs, memory, thoughts | noxflow agents, `hal/thoughts` | **high** — touches credentials |
|
||||
| 2.4 | `hal/work` — what remains of noxflow | noxflow tasks | medium |
|
||||
| 2.5 | `hal/ai` — one module per provider, each providing *a model provider* | `hal/claude*` | medium |
|
||||
|
||||
Each extraction is expand-then-contract: new context alongside, dual-write, verify, cut
|
||||
over, remove. **Never a move commit.**
|
||||
|
||||
**Checkpoint:** after 2.1, a human confirms the extraction pattern before 2.2 begins.
|
||||
After 2.3, a human confirms credentials still reach every agent on every node.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Reclaim the kernel
|
||||
|
||||
Only possible once domains have modules to own their code.
|
||||
|
||||
| # | task | done when |
|
||||
|---|---|---|
|
||||
| 3.1 | Move work-domain code out of `hal/sdk` | `workflow-engine.ts`, `task-commands.ts` live in `hal/work` |
|
||||
| 3.2 | Move provider code out | `claude-credentials.ts` lives in `hal/ai` |
|
||||
| 3.3 | Move delivery code out | feature handlers, artifact manager, build executor live in `hal/delivery` |
|
||||
| 3.4 | Decide the residue | ADR: what `hal/sdk` keeps (open question 4) |
|
||||
|
||||
**Measure:** `hal/sdk` line count, tracked per task. Today: **34,636** across **155**
|
||||
files.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Separate what the mesh runs from the mesh
|
||||
|
||||
| # | task | done when |
|
||||
|---|---|---|
|
||||
| 4.1 | Decide the destination (open question 3) | ADR accepted |
|
||||
| 4.2 | Cross-repository dependency resolution proven | a catalogue module builds against a published `@hal/*` |
|
||||
| 4.3 | Move the 91 catalogue modules | this repository contains only mesh contexts |
|
||||
|
||||
**Checkpoint:** move one application first and run it for a week before the rest follow.
|
||||
|
||||
---
|
||||
|
||||
## Sequencing constraints
|
||||
|
||||
- **0 before everything.** Unverifiable refactors are how this list got long.
|
||||
- **1 before 2.** Extracting into contexts without per-node features recreates the module
|
||||
count inside the new names.
|
||||
- **2 before 3.** A domain can only own its shared code once the domain has a module.
|
||||
- **2.3 after 2.1 and 2.2.** Agents touch credentials; do it once the pattern is proven on
|
||||
cheaper contexts.
|
||||
- **4 last.** It is the only phase that is pure movement, so it is the only one safe to
|
||||
defer indefinitely.
|
||||
- **Never edit mesh-managed files on disk.** Use the thing that owns the file.
|
||||
- **Never write to a production database directly.** Migrations for schema, application code for
|
||||
data.
|
||||
- **Every schema change ships twice** — consolidated schema *and* an incremental migration.
|
||||
- **Expand, then contract.** Add the new shape, migrate, verify, and only then remove the old one.
|
||||
- **A green pipeline proves transport, not effect.**
|
||||
|
||||
## What "done" looks like
|
||||
|
||||
Eight contexts. `hal/sdk` holding only what is genuinely cross-cutting. A mesh that stands
|
||||
up on a laptop. A module count that grows only when the domain does.
|
||||
The old registry is off. Every module runs on the new mesh, declared rather than scripted. A
|
||||
machine that fails says what it could not do. And the number of modules grows when the work does,
|
||||
not when the platform needs somewhere to put something.
|
||||
|
||||
@@ -2,11 +2,11 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-lab]
|
||||
updated: 2026-08-23
|
||||
updated: 2026-09-21
|
||||
decisions:
|
||||
- 02-DECISIONS/0016-the-lab.md
|
||||
- 02-DECISIONS/0016-the-lab.md
|
||||
- 02-DECISIONS/0019-how-this-repository-works.md
|
||||
- 02-DECISIONS/0089-a-bed-reads-the-catalogue-it-proves.md
|
||||
---
|
||||
|
||||
# End-to-end testing
|
||||
@@ -41,13 +41,13 @@ not the first one built** ([ADR 0016](../../02-DECISIONS/0016-the-lab.md)).
|
||||
|
||||
| | **Bootstrap scenario** | **Full scenario** |
|
||||
|---|---|---|
|
||||
| Contains | virtual machines, the host binary, a pinned substrate bundle | a complete mesh: forge, coordinator, delivery, modules |
|
||||
| Contains | virtual machines, the host binary, a pinned foundation bundle | a complete mesh: forge, coordinator, delivery, modules |
|
||||
| Verdict from | what the host reports about the state it reconciled | a pipeline result ending in verify |
|
||||
| Exercises | the node host and the substrate | the control plane and everything above it |
|
||||
| Exercises | the node host and the foundation | the controller and everything above it |
|
||||
| Exists to | **develop the mesh** | **test what runs on it** |
|
||||
|
||||
The bootstrap scenario is a **strict subset**: same virtualisation, same networking, same
|
||||
lifecycle — it simply stops before a control plane exists. Everything from *"Where this sits in
|
||||
lifecycle — it simply stops before a controller exists. Everything from *"Where this sits in
|
||||
the way work happens"* onward describes the full scenario, and applies once there is a
|
||||
coordinator to describe.
|
||||
|
||||
@@ -337,11 +337,14 @@ credential rotation reaches every consumer, that delivery to an absent node is r
|
||||
pending rather than done, that a returning node catches up. These are fewer and change
|
||||
rarely, but they are where the known production faults get encoded so they stay fixed.
|
||||
|
||||
The known faults become mesh tests that fail today. That is the Phase 0 checkpoint.
|
||||
The known faults become mesh tests that fail today. Phase 0 of
|
||||
[`00-work-breakdown.md`](00-work-breakdown.md) is now complete on this basis — twenty-two
|
||||
assertions on real machines — and what it does **not** cover is recorded there: no module
|
||||
from the existing system has run against any of it yet.
|
||||
|
||||
---
|
||||
|
||||
## The substrate
|
||||
## The foundation
|
||||
|
||||
### A node is a system container
|
||||
|
||||
@@ -390,6 +393,65 @@ Everything a node itself does is real, because a node is a real machine.
|
||||
|
||||
---
|
||||
|
||||
## A suite too expensive to run on every push says when it last ran
|
||||
|
||||
*Written 2026-08-31, from resolving [04-ISSUES/005](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md).*
|
||||
|
||||
This suite needs a machine with a hypervisor. It therefore cannot run on every push, and a suite
|
||||
that does not run on every push runs **when somebody remembers**. Remembering is not a mechanism,
|
||||
and the harness this one replaces proves it: it had not built for two and a half months, nothing
|
||||
said so, and the coverage was assumed rather than checked.
|
||||
|
||||
**The danger is not that the suite breaks. It is that nobody notices it stopped running** — and
|
||||
that danger belongs to *this* design, not to the harness it retired.
|
||||
|
||||
So three rules, each held by a test:
|
||||
|
||||
**A run leaves a receipt** — when, what passed, what it ran, and the commit each repository was
|
||||
at. Kept **outside version control**: the question is *has this machine run it*, and a receipt in
|
||||
git would be a claim about everybody's machine made by whoever committed last.
|
||||
|
||||
**A receipt says why it does not count.** Old, failed, taken against commits the repositories have
|
||||
moved past, or a run that never raised a machine. Something can be asked, and answers non-zero.
|
||||
**A receipt that says nothing about something is not a receipt that clears it** — including a
|
||||
receipt written before it recorded a given fact, which claims nothing rather than everything.
|
||||
|
||||
**The run rebuilds what it tests.** The suite consumes artifacts from other repositories, and an
|
||||
artifact rebuilt from memory is one rebuilt sometimes. A stale binary reporting success against
|
||||
rules that have since changed is the same fault wearing different clothes.
|
||||
|
||||
**The general rule, which outlives this suite:** *silence and success must never look alike.*
|
||||
It is the same rule the host follows about a service that does not exist
|
||||
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) — absence must be distinguishable
|
||||
from a failure to answer — applied to coverage instead of to a machine.
|
||||
|
||||
## A bed reads the catalogue it proves
|
||||
|
||||
*Written 2026-09-21, from resolving [04-ISSUES/073](../../04-ISSUES/073-beds-carry-copies-of-catalogue-manifests/00-report.md);
|
||||
decided in [ADR 0089](../../02-DECISIONS/0089-a-bed-reads-the-catalogue-it-proves.md).*
|
||||
|
||||
The rule above — the run rebuilds what it tests — was held for binaries and images and not for
|
||||
manifests. Beds built the manifests they install inline, as literals copied from the catalogue
|
||||
when each bed was written; the copies did not move when the catalogue did, and a catalogue change
|
||||
was proven by no bed at all, while every bed stayed green against its copy.
|
||||
|
||||
**A bed that installs a catalogue module reads that module's manifest from the catalogue the run
|
||||
was pointed at.** It rewrites what the lab must — a build artifact becomes the image the machine
|
||||
holds, an image is pinned, a host port is remapped where one machine carries colliding modules,
|
||||
an address may point at a stand-in the bed raises — and nothing else.
|
||||
|
||||
**A bed that needs less than the module declares is not testing that module.** No upstream
|
||||
server, a secret in the environment, a requirement edge cut so no second provider is needed:
|
||||
that is a mesh test, and it carries a fixture with a name of its own, never a catalogue module's.
|
||||
|
||||
**The receipt names the catalogue's commit** with the others', so a run taken before a manifest
|
||||
changed says so — the same rule as for the binaries, for the same reason.
|
||||
|
||||
*How it is checked:* a unit test in the lab refuses an inline manifest literal that names a
|
||||
catalogue module unless the bed is declared, with its reason, in the test's own list, and refuses
|
||||
a declaration for a copy that is gone; the receipt test asserts the catalogue is claimed whenever
|
||||
the run is pointed at one.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Bringing a node into being is part of the framework.** A test creates its own nodes — one
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-lab]
|
||||
updated: 2026-08-25
|
||||
updated: 2026-08-28
|
||||
decisions:
|
||||
- 02-DECISIONS/0016-the-lab.md
|
||||
- 02-DECISIONS/0016-the-lab.md
|
||||
@@ -205,7 +205,7 @@ machines:
|
||||
|
||||
place:
|
||||
all: [host]
|
||||
anchor: [substrate]
|
||||
anchor: [foundation]
|
||||
|
||||
snapshot: raised
|
||||
```
|
||||
@@ -334,12 +334,12 @@ than a fork.
|
||||
# bootstrap — tiers 0 and 1
|
||||
place:
|
||||
all: [host]
|
||||
anchor: [substrate]
|
||||
anchor: [foundation]
|
||||
|
||||
# full — adds a control plane, a forge, and a module under test
|
||||
# full — adds a controller, a forge, and a module under test
|
||||
place:
|
||||
all: [host]
|
||||
anchor: [substrate, control, forge]
|
||||
anchor: [foundation, control, forge]
|
||||
module: a-web-service
|
||||
assert:
|
||||
- the service answers on its published name
|
||||
@@ -524,7 +524,7 @@ machines:
|
||||
|
||||
place:
|
||||
all: [host]
|
||||
anchor: [substrate]
|
||||
anchor: [foundation]
|
||||
|
||||
snapshot: raised
|
||||
```
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-lab]
|
||||
updated: 2026-08-25
|
||||
updated: 2026-08-28
|
||||
decisions:
|
||||
- 02-DECISIONS/0016-the-lab.md
|
||||
- 02-DECISIONS/0016-the-lab.md
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
status: in-progress
|
||||
code: [mesh-lab]
|
||||
updated: 2026-08-24
|
||||
updated: 2026-09-11
|
||||
decisions:
|
||||
- 02-DECISIONS/0016-the-lab.md
|
||||
- 02-DECISIONS/0010-delivery.md
|
||||
@@ -48,6 +48,7 @@ present, but that the machine can actually do the work:
|
||||
| **the pool the lab will use is on that driver** | a pool exists, and is the slow kind — the failure that has no symptom |
|
||||
| hardware virtualisation is present | machines will be emulated and unusably slow |
|
||||
| an image can be fetched or is cached | the first raise will fail late instead of early |
|
||||
| **a machine on an uplink reaches something real**, by fetching it — not by reading a route or a policy | forwarding is being dropped by something else on the workstation; names still resolve, and every pull hangs |
|
||||
|
||||
Each check states **why it matters**, in the terms of what it costs. *"The pool uses the `dir`
|
||||
driver"* means nothing to someone who does not already know it means seventy-six times slower
|
||||
@@ -101,7 +102,22 @@ In order, on a machine with nothing:
|
||||
existed. Observed directly: a shell whose process tree predated the grant could not reach
|
||||
the daemon while a fresh lookup showed the membership present. The bootstrap has to say so,
|
||||
or the first thing a person meets is a permission error that looks like a broken install.
|
||||
5. **Verification**, as above, before anything is raised.
|
||||
5. **A path out to the internet for the machines that need one.** A scenario's own segments are
|
||||
isolated on purpose, but a machine that fetches what it starts from is attached to an uplink
|
||||
the daemon translates. That is the daemon's business and it does it — and then a **container
|
||||
runtime on the same workstation sets the kernel's forwarding policy to drop**, which the
|
||||
daemon's own accept rules do not override, because both are consulted and a drop anywhere is
|
||||
the answer.
|
||||
|
||||
The result is the sharpest instance of *available is not adequate* yet: the machines get
|
||||
addresses, they resolve names — the daemon's resolver is on the bridge, so that half works —
|
||||
and every packet to anything real is discarded. Nothing is misconfigured, nothing logs, and
|
||||
the failure presents as *every image pull hangs*. A workstation that runs containers is the
|
||||
ordinary case, so this is a prerequisite rather than a quirk: forwarding must be permitted for
|
||||
the lab's own bridges, and it must be **verified by reaching something**, never by reading a
|
||||
setting.
|
||||
|
||||
6. **Verification**, as above, before anything is raised.
|
||||
|
||||
## Open
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-host]
|
||||
updated: 2026-08-30
|
||||
updated: 2026-09-21
|
||||
decisions:
|
||||
- 02-DECISIONS/0019-how-this-repository-works.md
|
||||
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
@@ -47,10 +47,10 @@ mesh database, and it has no listening surface.
|
||||
|---|---|
|
||||
| `apply` | reconciling declared state on this machine |
|
||||
| `store` | local state, authoritative while disconnected |
|
||||
| `link` | the single outbound connection to the control plane |
|
||||
| `link` | the single outbound connection to the controller |
|
||||
| `profile` | what this machine can be asked to do |
|
||||
| `inventory` | what this machine is and has |
|
||||
| `substrate.lock` | the pinned tier-1 descriptor, appliable with no mesh present |
|
||||
| `foundation.lock` | the pinned tier-1 descriptor, appliable with no mesh present |
|
||||
|
||||
### apply
|
||||
|
||||
@@ -74,7 +74,7 @@ the machine in whatever state it reached, and nothing must claim otherwise.
|
||||
|
||||
### store
|
||||
|
||||
Local, and **authoritative while disconnected**. Not a cache of the control plane — the record
|
||||
Local, and **authoritative while disconnected**. Not a cache of the controller — the record
|
||||
of what this node has applied and what it currently holds.
|
||||
|
||||
This is structural rather than convenient: if disconnection is an ordinary situation rather
|
||||
@@ -84,7 +84,7 @@ not come back and ask what it is.
|
||||
|
||||
### link
|
||||
|
||||
The node's one connection to the control plane, and its security boundary
|
||||
The node's one connection to the controller, and its security boundary
|
||||
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
|
||||
|
||||
It is the broker connection that already exists
|
||||
@@ -137,11 +137,11 @@ One behaviour, two sources
|
||||
|
||||
| Situation | Source |
|
||||
|---|---|
|
||||
| no mesh reachable | `substrate.lock` — the pinned bundle the host carries |
|
||||
| mesh reachable | the control plane, over the link |
|
||||
| no mesh reachable | `foundation.lock` — the pinned bundle the host carries |
|
||||
| mesh reachable | the controller, over the link |
|
||||
|
||||
**The first node is not a different kind of node.** It is a node whose mesh is not up yet. It
|
||||
applies the bundle it carries, the control plane comes up on top of it, and from that moment it
|
||||
applies the bundle it carries, the controller comes up on top of it, and from that moment it
|
||||
takes declarations like every other node. Its specialness is temporary and self-erasing.
|
||||
|
||||
**A joining node does the minimum to be reachable and nothing else** — an identity, an address,
|
||||
@@ -155,7 +155,7 @@ Settled by [ADR 0005](../../02-DECISIONS/0005-the-node-host.md).
|
||||
**JSON**, because the host has no dependencies to spend and the standard library carries no
|
||||
YAML. **An ordered list of typed resources**, each with a stable identity — the order is stated
|
||||
rather than derived, because deriving it would be the host deciding the thing most likely to
|
||||
differ from what the control plane intended.
|
||||
differ from what the controller intended.
|
||||
|
||||
**Unknown is refused, never skipped.** An unknown version, type or field refuses the whole
|
||||
declaration. A host that skipped what it did not understand would apply most of it and report
|
||||
@@ -173,14 +173,17 @@ without one, applying the bundle it carries, has nothing to check against.
|
||||
Staged so each stage is verifiable in the lab before the next exists.
|
||||
|
||||
**1 — profile and inventory.** The host runs on a machine, detects what it can do, and reports
|
||||
what it is. No control plane, no declarations, no network. Verifiable immediately: the lab's
|
||||
what it is. No controller, no declarations, no network. Verifiable immediately: the lab's
|
||||
`place:` gains its first implementation, and a raised scenario finally contains something.
|
||||
|
||||
**2 — apply, from the bundle.** The host applies `substrate.lock` with no mesh present. This is
|
||||
**2 — apply, from the bundle.** The host applies `foundation.lock` with no mesh present. This is
|
||||
the first node's path, and it is the claim the skeleton's Move 1 rests on and has never proved:
|
||||
that one host can raise the substrate alone.
|
||||
that one host can raise the foundation alone.
|
||||
|
||||
Raising the substrate needs six shapes in the host's vocabulary, and **all six are built**:
|
||||
Raising the foundation uses **four** shapes — `package`, `container`, `service`, `action` —
|
||||
counted from the bundle that exists rather than reasoned about. `file` and `directory` are listed
|
||||
below because they are the cheapest to be sure of and a foundation that needed them would find them
|
||||
ready; the current bundle simply does not. **All of them are built:**
|
||||
|
||||
| | | |
|
||||
|---|---|---|
|
||||
@@ -190,6 +193,28 @@ Raising the substrate needs six shapes in the host's vocabulary, and **all six a
|
||||
| `container` | **built** | pinned by digest ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)); identified by a label carrying a digest of the declaration that made it, because a runtime normalises what it is given and that is indistinguishable from drift |
|
||||
| `action` | **built** | bundle-only ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)); verify is mandatory and is the idempotency check as well as the read-back |
|
||||
|
||||
**A service says what it must reflect, and that is declared state rather than a command.**
|
||||
`restart-on` names files whose change means the unit must be restarted — because a running service
|
||||
does not re-read its configuration, and replacing a file, finding the service already running and
|
||||
doing nothing leaves a machine behaving the way it did before while every check passes. A *command*
|
||||
to restart would be an action, and the link may not carry one, so this is the shape that rule
|
||||
leaves rather than a way around it.
|
||||
|
||||
**It may name a file another module put there**, written `<module>.<id>`. The case that needed it:
|
||||
a resolver restarting when the mesh rewrites the names, which are computed by the mesh and belong
|
||||
to its module rather than to the daemon's. Without it the daemon serves the names it started with
|
||||
for ever — every machine that joined afterwards unreachable by name, and every check passing. An
|
||||
unqualified name still means *my own*, so the common case reads as it always did.
|
||||
|
||||
**An action's verify is the definition of what the action is for**, and the action's own idea of
|
||||
being finished must be the same one. *Written 2026-08-31, after this went wrong.* If an action
|
||||
waits on one test and its verify reads back another, the two can disagree — and then the action
|
||||
succeeds into a state its own verify rejects. The host says so accurately and uselessly: *the
|
||||
action ran without error and its own verify still fails.* It is intermittent, it reads as a slow
|
||||
machine, and the remedy people reach for is a longer timeout, which cannot help.
|
||||
[04-ISSUES/017](../../04-ISSUES/017-an-action-succeeded-into-a-state-its-verify-rejects/00-report.md)
|
||||
is that, in the one action the whole bootstrap depends on.
|
||||
|
||||
**The parser enforces the boundary rather than the caller remembering it.** `Parse` refuses an
|
||||
action and is what the link uses; `ParseTrusted` permits one and is what the bundle uses. The
|
||||
safe path is the default and the permissive one has to be named.
|
||||
@@ -200,7 +225,7 @@ container runtime. All three were instead verified against a real machine — a
|
||||
labelled, replaced when its declaration changed, exec'd into and removed; an action that exits
|
||||
zero and satisfies nothing failing the apply. That is lab-installation work
|
||||
([`04-lab-installation.md`](04-lab-installation.md)) rather than a constraint on the design, but
|
||||
until it is done the substrate bootstrap has no end-to-end test.
|
||||
until it is done the foundation bootstrap has no end-to-end test.
|
||||
|
||||
**3 — link and store.** The node connects, receives declarations, and holds what it applied.
|
||||
|
||||
@@ -228,9 +253,67 @@ Each decision above owes a test:
|
||||
| 0036 — disconnection is a situation | a node cut off and returned reconciles without being re-adopted |
|
||||
| 0008 — a failed step fails the apply | an apply with a failing step reports failure |
|
||||
|
||||
## What a machine says about itself, and what the mesh keeps of it
|
||||
|
||||
*2026-08-31, from finding that half of it was being discarded.*
|
||||
|
||||
**A capability is detected and never assumed** ([ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)),
|
||||
so the only account of what a machine can do is the one the machine gave. That account has two
|
||||
halves and the mesh was keeping one:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **the yes or no** | gates an assignment — *this machine has no seat, and nothing can be installed to fix that* |
|
||||
| **the detail** | carries a value — `seat: card1-DP-1`, an architecture, an amount of memory |
|
||||
|
||||
They are **one fact read two ways**: *can this run here* and *what should it be configured as*. A
|
||||
module that must not be assigned without an OLED panel and one that dims itself differently on one
|
||||
are reading the same line. Keeping only the first read makes the second unanswerable, and there is
|
||||
nowhere else to get it — the detector is the only thing that looked.
|
||||
|
||||
**The reason an absent capability is absent goes the same way, and it is the half a person needs
|
||||
most.** *This machine has no container runtime* is the answer; *docker is not installed* is why.
|
||||
The first is the mesh's to say and the second is only the machine's.
|
||||
|
||||
### The eight, and what each one is evidence of
|
||||
|
||||
*Written 2026-08-31 from `internal/profile/detectors.go`, because the set was implemented and
|
||||
enumerated in no document. A vocabulary a module writes against, that exists only in code, is one
|
||||
nobody can write against without reading the code.*
|
||||
|
||||
| capability | what a detection proves |
|
||||
|---|---|
|
||||
| `container-runtime` | a runtime is **running**, not installed |
|
||||
| `package-manager` | the machine's own package manager works |
|
||||
| `service-manager` | an init that can be asked for state — including *degraded*, which reports on stdout and exits non-zero |
|
||||
| `firewall` | a filter this host can write rules into |
|
||||
| `overlay` | the private network can be joined |
|
||||
| `graphical-session` | a display server **is running** — state |
|
||||
| `seat` | hardware where one **could** run — and assignment needs this one, not the row above |
|
||||
| `privileged` | the host can change the machine |
|
||||
|
||||
**`seat` and `graphical-session` are the pair worth reading twice**, because collapsing them is
|
||||
the obvious economy and it is wrong in both directions: a machine with a seat and no session can
|
||||
be given a display server, and a machine with a session running is not thereby able to host a
|
||||
second one.
|
||||
|
||||
**A detection runs something that only succeeds if the thing is *functioning*, never `--version`.**
|
||||
A version string proves a binary is on disk, which
|
||||
[`04-ISSUES/007`](../../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md)
|
||||
records as false in the way that matters: the package was installed and the daemon was not
|
||||
running.
|
||||
|
||||
**Never reported and reported nothing stay different.** One machine has not run the host yet; the
|
||||
other ran it and can do nothing. Both refuse everything that requires a capability, and the
|
||||
remedies are not remotely alike.
|
||||
|
||||
*Checked by recording a profile with a present capability carrying a value and an absent one
|
||||
carrying its reason, and requiring both to survive — and by requiring a machine that never
|
||||
reported to be distinguishable from one that reported an empty list.*
|
||||
|
||||
## Open
|
||||
|
||||
- **Whether one host can raise the substrate alone.** Move 1 assumes it. Stage 2 tests it, and
|
||||
- **Whether one host can raise the foundation alone.** Move 1 assumes it. Stage 2 tests it, and
|
||||
if it is false the tier boundary moves.
|
||||
- **What the host carries versus what it finds.** It manages `wg`, `nft`, `pacman`, `docker`;
|
||||
it does not contain them, and how it obtains one it lacks is undecided —
|
||||
@@ -246,16 +329,18 @@ Each decision above owes a test:
|
||||
|
||||
## What was added to the vocabulary, and why each cost was worth paying
|
||||
|
||||
*Written 2026-08-30. Every addition widens what a compromised control plane can express, so the
|
||||
*Written 2026-08-30. Every addition widens what a compromised controller can express, so the
|
||||
count is asserted by a test and a change to it is a decision rather than a convenience.*
|
||||
|
||||
Six shapes raised the substrate. Two more exist because most of what a person installs is not a
|
||||
Four shapes raise the foundation. Five more exist because most of what a person installs is not a
|
||||
service:
|
||||
|
||||
| | why |
|
||||
|---|---|
|
||||
| **file**, **directory** | the foundation needs neither, and almost everything else does |
|
||||
| **user** | a shell, a terminal, a chat client, a desktop are a package plus configuration **in somebody's home**. A mesh with no user owns `/etc` and nothing anybody looks at |
|
||||
| **archive** | a theme is hundreds of files. Inlining them makes every declaration enormous and rewrites all of them when one changes |
|
||||
| **network** | a module of several containers has to let them reach each other by name, and doing it with an action would create something nothing could ever remove ([ADR 0029](../../02-DECISIONS/0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md)) |
|
||||
|
||||
And `file` gained two fields: `bytes`, because a wallpaper is not a string, and `owner`, because a
|
||||
file in a home belongs to somebody.
|
||||
@@ -282,4 +367,17 @@ of the reason to manage a machine.
|
||||
silently incomplete.
|
||||
|
||||
**A partial host does archives and refuses users**: an archive needs a filesystem and a way to
|
||||
fetch; a user needs a user database the host is allowed to write.
|
||||
fetch; a user needs a user database the host is allowed to write.
|
||||
|
||||
## A machine becomes the last thing it was told
|
||||
|
||||
Every declaration is complete, so applying an old one is never wrong, only wasted — and under a
|
||||
flurry of pushes a machine spent minutes becoming things the mesh had moved past
|
||||
([issue 031](../../04-ISSUES/031-a-machine-becomes-each-thing-it-was-told-in-turn/00-report.md)).
|
||||
So the host looks at what is already waiting before it applies anything: it holds a small window
|
||||
of unacknowledged declarations, applies the newest, and sets the rest aside — each **reported as
|
||||
superseded**, naming the one applied instead, because silence would read as a machine that
|
||||
ignored an instruction and "applied" would be a lie. Applying stays one at a time; only seeing
|
||||
is not. **Checked** by the link's unit tests on the drain, and by the genesis bed's settle wait,
|
||||
which counts on a node catching up to the newest declaration rather than the oldest.
|
||||
|
||||
|
||||
+27
-26
@@ -1,18 +1,19 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-control
|
||||
updated: 2026-08-29
|
||||
- mesh-controller
|
||||
updated: 2026-08-31
|
||||
decisions:
|
||||
- 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
- 02-DECISIONS/0008-a-context-owns-its-store.md
|
||||
- 02-DECISIONS/0019-how-this-repository-works.md
|
||||
- 02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md
|
||||
---
|
||||
|
||||
# The control plane
|
||||
# The controller
|
||||
|
||||
Tier 2. The term appears seventy-nine times across this repository and was defined nowhere,
|
||||
which is `how-we-build` §5 failing on this repository's own vocabulary.
|
||||
@@ -22,7 +23,7 @@ This document defines it. It does **not** design the contexts inside it; those a
|
||||
|
||||
## The definition
|
||||
|
||||
> **The control plane is everything that needs to know about more than one node.**
|
||||
> **The controller is everything that needs to know about more than one node.**
|
||||
|
||||
That is the whole test, and it is not arbitrary — it follows from
|
||||
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md). The host applies and
|
||||
@@ -32,11 +33,11 @@ exactly there:
|
||||
| Question | Whose |
|
||||
|---|---|
|
||||
| write this file, with this content, with this mode | the **host** — one machine |
|
||||
| which nodes should run the store | the **control plane** — needs every node |
|
||||
| which nodes should run the store | the **controller** — needs every node |
|
||||
| is this unit running | the **host** — one machine |
|
||||
| which peers belong in this node's overlay | the **control plane** — needs every node |
|
||||
| what does this machine have installed | the **host** reports; the control plane **records** |
|
||||
| has this node been unreachable for a week | the **control plane** — nobody else is watching |
|
||||
| which peers belong in this node's overlay | the **controller** — needs every node |
|
||||
| what does this machine have installed | the **host** reports; the controller **records** |
|
||||
| has this node been unreachable for a week | the **controller** — nobody else is watching |
|
||||
|
||||
A useful consequence: **anything a single machine could answer alone is not the control
|
||||
plane's.** If it needs no second node, putting it here is a mistake, and the tier rule will not
|
||||
@@ -67,7 +68,7 @@ something infrastructure*. `ai` is folded into `config`: a provider licence is a
|
||||
**`record` is an open question rather than an eighth entry.** Contexts integrate through it
|
||||
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), which makes it load-bearing,
|
||||
and [research 006](../../01-RESEARCH/006-mesh-from-scratch/skeleton.md) leaves *where it lives*
|
||||
unresolved — putting it in the substrate risks recreating the circularity the tier design just
|
||||
unresolved — putting it in the foundation risks recreating the circularity the tier design just
|
||||
removed. Listing it here would settle by naming what has not been settled by arguing.
|
||||
|
||||
**One of the seven is built.** `inventory` owns a database of that name and holds the node records;
|
||||
@@ -89,7 +90,7 @@ in front of them.
|
||||
The question this answers: **can a node write to the registry database?** No — and not "only
|
||||
through one node", which is the weaker arrangement it might be mistaken for.
|
||||
|
||||
> **No node holds a credential to any control-plane store, for writing or for reading.**
|
||||
> **No node holds a credential to any controller store, for writing or for reading.**
|
||||
|
||||
That is not a new rule here; it is four already taken, and it is worth seeing them together
|
||||
because each one alone reads like a detail:
|
||||
@@ -118,11 +119,11 @@ Reads work the same way in reverse — a node is *told*, in declarations. It nev
|
||||
|
||||
### Who actually consumes, and who writes
|
||||
|
||||
**The control plane is the consumer. There is one of it, and the context that owns the data does
|
||||
**The controller is the consumer. There is one of it, and the context that owns the data does
|
||||
the write.**
|
||||
|
||||
```
|
||||
node ──► broker ──► the control plane, consuming
|
||||
node ──► broker ──► the controller, consuming
|
||||
├─ a node reported what it applied ─► inventory writes the registry
|
||||
├─ a node reported health ─► observability writes its own store
|
||||
└─ a grant was requested ─► provisioning writes its own store
|
||||
@@ -139,9 +140,9 @@ the store it exclusively owns
|
||||
receiving half of what it expects* — which has happened, between a module's daemon and its
|
||||
capability server. With one consumer that class of fault cannot arise.
|
||||
|
||||
**And the broker is the buffer while the control plane is down.** Nodes go on publishing;
|
||||
messages queue; the control plane drains them when it returns. That is what makes
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)'s single control plane
|
||||
**And the broker is the buffer while the controller is down.** Nodes go on publishing;
|
||||
messages queue; the controller drains them when it returns. That is what makes
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)'s single controller
|
||||
tolerable — an outage delays the mesh's *knowledge* rather than losing it.
|
||||
|
||||
**With one consequence that must be bounded before it is discovered:** a queue with no limit
|
||||
@@ -177,7 +178,7 @@ volume genuinely argues against a relational store.
|
||||
node except through the host.
|
||||
- **Not a surface.** Tier 3 is how people and agents reach it. It has one interface; the
|
||||
surfaces are what speak to that interface.
|
||||
- **Not the substrate.** It *runs on* tier 1 — PostgreSQL, LavinMQ, MinIO, an OCI registry
|
||||
- **Not the foundation.** It *runs on* tier 1 — PostgreSQL, LavinMQ, an OCI registry
|
||||
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) — and cannot start without
|
||||
them, which is what makes them a lower tier.
|
||||
- **Not privileged on a node.** It has no more access to a machine than the declaration
|
||||
@@ -185,19 +186,19 @@ volume genuinely argues against a relational store.
|
||||
|
||||
## It is also a consumer
|
||||
|
||||
The property that makes tier 2 unlike the others: **the control plane has requirements of its
|
||||
The property that makes tier 2 unlike the others: **the controller has requirements of its
|
||||
own.** It needs a PostgreSQL database, an AMQP virtual host, and a bucket — the same things any
|
||||
module needs, granted the same way.
|
||||
|
||||
That is the circularity the tiers exist to resolve rather than hide: the control plane cannot
|
||||
That is the circularity the tiers exist to resolve rather than hide: the controller cannot
|
||||
provision its own database, because it is not running yet. So its **store** is raised from the
|
||||
bundle the host carries, before there is a control plane to ask
|
||||
bundle the host carries, before there is a controller to ask
|
||||
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md),
|
||||
[research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md)).
|
||||
|
||||
Its virtual host and its bucket are **not** in the bundle — by the time they are wanted there is
|
||||
a control plane to grant them. Whether the bus must come first is
|
||||
[open](07-the-substrate.md#open), and it turns on whether these contexts talk to each other over
|
||||
a controller to grant them. Whether the bus must come first is
|
||||
[open](07-the-foundation.md#open), and it turns on whether these contexts talk to each other over
|
||||
it.
|
||||
|
||||
## Where it runs
|
||||
@@ -209,10 +210,10 @@ hosts, assigned to nodes by the same mechanism as everything else.
|
||||
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The node is assigned,
|
||||
never elected — no promotion, no quorum, no split brain.
|
||||
|
||||
That is sound rather than merely cheap, because the design already tolerates the control plane
|
||||
That is sound rather than merely cheap, because the design already tolerates the controller
|
||||
being absent by construction: a node reconciles from **its own** store
|
||||
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) and
|
||||
never needed to ask anybody to hold the state it was last given. So the control plane being down
|
||||
never needed to ask anybody to hold the state it was last given. So the controller being down
|
||||
is not a new failure mode — it is
|
||||
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s ordinary disconnected
|
||||
situation, happening to every node at once. **What is lost is change, not operation.**
|
||||
@@ -226,13 +227,13 @@ every public name.
|
||||
- ~~**The contexts themselves.**~~ **Decided** — seven, by
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md).
|
||||
What remains open is narrower and named there: **where the record lives**, which research 006
|
||||
leaves unresolved because the substrate is the one place it must not go.
|
||||
leaves unresolved because the foundation is the one place it must not go.
|
||||
- **How far it may be split.** One deployable today. Splitting a context out costs the single
|
||||
interface a surface depends on
|
||||
([research 011](../../01-RESEARCH/011-the-module-graph/00-overview.md)).
|
||||
- ~~**How many run, and what a node does without one.**~~ **Resolved** by
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md). What remains is
|
||||
measurement: nothing reports how long the control plane has been unreachable, or how close a
|
||||
measurement: nothing reports how long the controller has been unreachable, or how close a
|
||||
certificate is to expiry — both needed for restore-not-failover to be a plan rather than a
|
||||
hope.
|
||||
- **What the interface is.** One interface is stated; its shape, and whether it is request,
|
||||
@@ -0,0 +1,304 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-host examples/foundation-first-node.lock
|
||||
- mesh-host internal/apply
|
||||
- mesh-host internal/bootstrap/phase3.go
|
||||
- mesh-host internal/bootstrap/rootsecrets.go
|
||||
- mesh-host internal/bootstrap/operator.go
|
||||
- mesh-catalog modules/mesh-vault
|
||||
- mesh-catalog modules/postgres
|
||||
- mesh-catalog modules/lavinmq
|
||||
- mesh-lab test/integration/mesh.test.ts (a bare machine becomes a mesh)
|
||||
updated: 2026-09-21
|
||||
decisions:
|
||||
- 02-DECISIONS/0088-the-foundation-filters-before-anything-listens.md
|
||||
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
- 02-DECISIONS/0078-the-store-and-broker-are-modules.md
|
||||
- 02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md
|
||||
- 02-DECISIONS/0077-the-controller-and-the-foundation.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
- 02-DECISIONS/0007-connectivity.md
|
||||
- 02-DECISIONS/0008-a-context-owns-its-store.md
|
||||
- 02-DECISIONS/0019-how-this-repository-works.md
|
||||
---
|
||||
|
||||
# The foundation
|
||||
|
||||
**One store, one broker — enforced, not conventional.** Each foundation module claims a
|
||||
mesh-scoped seat named after its server (`mesh-store`, `mesh-broker`, `mesh-controller` —
|
||||
[ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)), so
|
||||
assigning a second holder anywhere in the mesh is refused at resolution rather than silently
|
||||
raising a second server.
|
||||
|
||||
Tier 1. Defined the same way [the controller](06-the-controller.md) is, because the same
|
||||
gap applied: the word was load-bearing and unpinned.
|
||||
|
||||
## The definition
|
||||
|
||||
> **The foundation is what the controller consumes and cannot grant itself.**
|
||||
|
||||
Every module that needs a database asks the controller's provisioning for one. The control
|
||||
plane needs a database too — and it cannot ask itself, because it is not running yet. That
|
||||
circularity is not an awkwardness to work around; it *is* the definition. Anything on the wrong
|
||||
side of it must be raised some other way, and the other way is the bundle the host carries
|
||||
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
|
||||
|
||||
The test, applied:
|
||||
|
||||
| | controller needs it | can it grant itself one? | |
|
||||
|---|---|---|---|
|
||||
| a relational store — **PostgreSQL** | its own state lives there | no — provisioning needs the store | **foundation** |
|
||||
| a message bus — **LavinMQ** | it reaches nodes over it ([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)) | no — it cannot grant itself a virtual host | **foundation** |
|
||||
| ~~an object store~~ | ~~artifacts and blobs it delivers~~ | — | **not foundation** — [ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md) |
|
||||
| ~~an image registry~~ | ~~images it delivers to nodes~~ | — | **not foundation** — needed to operate, not to start ([ADR 0033](../../02-DECISIONS/0033-the-substrate-is-a-store-and-a-broker.md)) |
|
||||
| ~~an identity provider~~ | ~~only if it delegates authentication~~ | — | **not foundation** — it delegates to nothing ([ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md)) |
|
||||
| ingress — **Traefik** | not to start; only to be reached by name | — it grants itself one afterwards | **not foundation** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) |
|
||||
| anything else the mesh hosts | no | — | not foundation |
|
||||
|
||||
*The object-store row was wrong, and how it was wrong is worth keeping.* It answered *can it
|
||||
grant itself one* — no, it cannot grant itself a bucket — while assuming the first column. **The
|
||||
controller does not need an object store**: it has no S3 client and never has, and artifacts
|
||||
reach nodes as content-addressed blobs in the registry. The row was inherited from the system being
|
||||
replaced, where an object store distributed module tarballs, and was never re-tested against the
|
||||
definition above it. *Both columns must be answered, and the second is true of almost any service.*
|
||||
|
||||
**An object store is an ordinary module**, required through the module graph by whatever wants one.
|
||||
A mesh with no workload needing one runs none.
|
||||
|
||||
**The role and the product are both written**, here and everywhere
|
||||
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The role is what the argument
|
||||
turns on — the test above works on roles, and would give the same answers for a different store.
|
||||
The product is what actually gets installed and pinned, and a design that names only the role
|
||||
does not record that the choice was ever made.
|
||||
|
||||
The dependency is on the **protocol**, not the product: AMQP for the bus, S3 for the object
|
||||
store, the OCI protocol for the registry. That is what keeps the naming safe rather than a
|
||||
commitment that cannot be revisited — replacing one is a foundation migration, not a redesign.
|
||||
The store is the exception, and the exception matters: the provisioning model uses databases,
|
||||
roles and schemas as PostgreSQL means them, so it is the one member that is not a swap.
|
||||
|
||||
## What that resolves
|
||||
|
||||
**Four or five?** [Research 006](../../01-RESEARCH/006-mesh-from-scratch/00-overview.md) asks
|
||||
whether the identity provider is a foundation service, and the test answers it *conditionally* —
|
||||
which is the honest answer rather than a number.
|
||||
|
||||
- If the controller **delegates** authentication, it cannot serve anybody before the provider
|
||||
exists, and it cannot grant itself a client. **Foundation.**
|
||||
- If it **authenticates natively**, the provider is an ordinary hosted service like any other.
|
||||
**Not foundation.**
|
||||
|
||||
So the count follows from a design decision that has not been taken, and the record should say
|
||||
that rather than assert four.
|
||||
|
||||
**Why not "important infrastructure".** An identity provider, a mail server and an analytics
|
||||
service are all infrastructure by any ordinary reading, and none of them are foundation — the
|
||||
controller starts and runs without them. *Important* is not the test; *the controller
|
||||
cannot obtain it* is.
|
||||
|
||||
## What the foundation is not
|
||||
|
||||
- **Not tier 0.** The host raises the foundation; it is not part of it. The host carries the
|
||||
declaration that brings the foundation up, and depends on nothing.
|
||||
- **Not the controller.** These are services with no knowledge of the mesh. A store does not
|
||||
know what a node is.
|
||||
- **Not a place for logic.** The skeleton is explicit: tier 1 is *declarations only, no logic of
|
||||
its own.* A foundation service is an upstream image, pinned, with configuration.
|
||||
- **Not privileged.** The foundation is provisioned *from* by the controller and grants
|
||||
nothing on its own initiative.
|
||||
- **Not the mesh's supply of anything**
|
||||
([ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md)).
|
||||
A foundation service and a module of the same product are **different instances**. The mesh's own
|
||||
PostgreSQL and a PostgreSQL a workload was given are two servers, and a node hosting both runs
|
||||
two containers — expected, not duplication to be tidied away.
|
||||
|
||||
The foundation is raised from the bundle before any mesh exists, so **it is not in the module
|
||||
graph**: a workload depending on it would depend on something the graph cannot see, cannot rotate
|
||||
a credential for, and cannot move. It would also put workload data in the store the controller
|
||||
keeps its own state in, where a workload that fills a disk takes down the one thing needed to fix
|
||||
it.
|
||||
|
||||
## The pinned bundle
|
||||
|
||||
`foundation.lock` holds **what must exist before the controller runs** — which is a smaller
|
||||
set than the foundation, and the difference is easy to miss. It is the only place in the mesh
|
||||
where versions are pinned by hand rather than resolved.
|
||||
|
||||
Being foundation and being in the bundle are two different questions:
|
||||
|
||||
| | is it foundation? | must it precede the controller? |
|
||||
|---|---|---|
|
||||
| PostgreSQL | yes — the controller's own state lives in it | **yes** — there is nowhere to put that state otherwise |
|
||||
| LavinMQ | yes — it cannot grant itself a virtual host | **yes** — the controller reaches a node only over the link, and the link is the broker ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
|
||||
| the OCI registry | yes — it cannot grant itself a repository | no — the first node fetches upstream ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
|
||||
|
||||
The registry is **foundation by role and ordinary by delivery**: by the time it is wanted there is
|
||||
a controller, and it provisions it the way it provisions anything. That keeps the bundle small
|
||||
enough for the review [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) requires — one
|
||||
foundation image until
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) established that the
|
||||
broker has to precede the controller, and two since.
|
||||
|
||||
*Corrected 2026-08-31, from counting what the bundle holds rather than reasoning about it.* **It
|
||||
carries three images, not two** — PostgreSQL, LavinMQ, and the controller itself, which the
|
||||
sentence above had overlooked by counting only foundation services. The controller is what the
|
||||
foundation exists to start, and it is in the bundle for the same reason they are: there is nothing
|
||||
to fetch it with yet. It also carries seven actions, a package and a service.
|
||||
|
||||
**Why pinned:** the bundle is applied when no mesh exists, so nothing can resolve a version, ask
|
||||
a registry, or check a constraint. What the host carries must already be exact.
|
||||
|
||||
**Why references and not payload:** the bundle names images by **digest** and the host fetches
|
||||
them ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). A first node is
|
||||
a real machine with a network; the sealed case is the lab, and the lab places images itself.
|
||||
|
||||
Reproducibility comes from pinning the identity of a thing rather than carrying its bytes, which
|
||||
is what keeps the bundle small enough for a person to read and check.
|
||||
|
||||
## Filtered from its first resource
|
||||
|
||||
The bundle installs the packet filter and loads a base ruleset before the store and broker come
|
||||
up ([ADR 0088](../../02-DECISIONS/0088-the-foundation-filters-before-anything-listens.md)): drop by
|
||||
default, keep loopback, replies, ping, ssh, the bus and the registry, and the container runtime's
|
||||
networks through the forward chain. It is written into the same table the filter module derives,
|
||||
so that module replaces it wholesale once it can. Until it does, the machine admits nothing else —
|
||||
not the overlay hub's port, which is derived from the hub's endpoint — so the filter module is
|
||||
assigned to the control-node before a hub is placed there, as genesis does; a lab bed that raises
|
||||
the foundation without genesis must do the same, and says so by waiting for the hub's port in the
|
||||
ruleset the machine loaded. **Checked** by the installer's bundle test (order and rules) and by
|
||||
the genesis bed, which probes the machine from outside for the length of the install: the store's
|
||||
port never answers, the bus's does.
|
||||
|
||||
## Raising it
|
||||
|
||||
The order, from [research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md):
|
||||
|
||||
```
|
||||
0 a container runtime exists detected — docker or podman — or installed
|
||||
1 PostgreSQL runs pulled by digest, from the bundle
|
||||
2 a database per context is created an action, run locally — one today, `inventory`
|
||||
3 each context's schema is applied an action, against its own database
|
||||
4 LavinMQ runs pulled by digest, from the bundle
|
||||
5 a virtual host, a credential, and actions, run locally
|
||||
a self-signed certificate
|
||||
6 the controller starts and only now is there a mesh
|
||||
7 the registry, and everything else the ordinary path
|
||||
are provisioned
|
||||
```
|
||||
|
||||
**Steps 4 and 5 are why the bundle is not one image**
|
||||
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The controller cannot
|
||||
provision the broker, because provisioning means telling a host, and telling a host happens over
|
||||
the broker. The first node does not escape this by being local: it enrols the ordinary way, by
|
||||
dialling the broker at the address in its token.
|
||||
|
||||
**Step 2 is one database per context and not one called `mesh`.** A context is granted only what it
|
||||
exclusively owns ([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), *the mesh
|
||||
database* names a thing that will not exist
|
||||
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)), and a separate
|
||||
database is a boundary a cross-context join cannot casually cross where a separate schema is not.
|
||||
|
||||
Only PostgreSQL is raised from the bundle, for the reason in *The pinned bundle* above — the
|
||||
rest of the foundation is wanted only once there is a controller to provision it.
|
||||
|
||||
**Step 0 is easy to leave out and it is where several things meet.** A foundation service is a
|
||||
container, so a container runtime must be working before anything else happens — and a runtime
|
||||
is a *package*, not a container.
|
||||
|
||||
**Which runtime is detected, not chosen**
|
||||
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)): a machine that
|
||||
already has one keeps it. On a machine with none, the controller names the package, because
|
||||
what it is called differs per system. It is:
|
||||
|
||||
- what the host's capability detection already reports, and the first use of that report by
|
||||
something other than a person;
|
||||
- **adopted rather than installed** when the machine already has one with configuration somebody
|
||||
chose ([research 012](../../01-RESEARCH/012-the-minimum-viable-node/00-overview.md));
|
||||
- a package, which needs the machine's own package manager and a network — both permitted by
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md).
|
||||
|
||||
So the bootstrap uses four shapes: **package**, **container**, **service** and **action** —
|
||||
*counted from `foundation-first-node.lock`, which is the only bundle there is*. It had said six,
|
||||
adding `file` and `directory`, which this bootstrap never asks for.
|
||||
|
||||
All four are built, as are the host's other five
|
||||
([`05-the-node-host.md`](05-the-node-host.md) stage 2), so nothing in this bootstrap is blocked
|
||||
on the host any longer — which is the claim that mattered, and it was true either way.
|
||||
|
||||
**Steps 2 and 3 happen before there is a mesh to do them**, which is why provisioning is part of
|
||||
the bootstrap rather than a service consumers use later. They are **actions** the bundle
|
||||
declares and the host runs
|
||||
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) — so the
|
||||
host's vocabulary grows by one shape rather than by one resource type per foundation service.
|
||||
|
||||
## Open
|
||||
|
||||
- ~~**Whether identity is the fifth.**~~ **Closed 2026-08-31** by
|
||||
[ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md): the control
|
||||
plane delegates authentication to nothing, so identity is an ordinary module. With the object
|
||||
store gone ([ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md))
|
||||
the foundation is three, and no member is conditional.
|
||||
- ~~**Whether the bus must precede the controller.**~~ **Resolved** by
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) — it must, and the question as
|
||||
posed here could not have answered it. This asked whether the controller's contexts talk to
|
||||
each other over the bus; they do not, being one process, which under this framing would have
|
||||
kept LavinMQ out of the bundle. What decides it is how the controller reaches a *node*, which
|
||||
is only ever over the link.
|
||||
- **What issues the broker's certificate at bootstrap.** New, and created by the row above. A
|
||||
token pins the fingerprint a host must expect before it sends anything
|
||||
([`09-the-node-lifecycle.md`](09-the-node-lifecycle.md)), so the broker needs a certificate at a
|
||||
moment when there is no mesh to issue one and no public name to obtain one for. Self-signed and
|
||||
pinned is the shape that fits; how it is later replaced by the certificates in
|
||||
[`08-connectivity.md`](08-connectivity.md) is not decided.
|
||||
- **How a context added later gets its database.** By then there is a controller — but one
|
||||
holding a credential that can create databases holds more than what it exclusively owns
|
||||
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)).
|
||||
- ~~**Whether the host can do step 2.**~~ **Resolved** by
|
||||
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md). A service
|
||||
running on this machine is part of this machine, so the scope was never in question — the real
|
||||
question was whether the host must learn what a database is, and it must not. The bundle
|
||||
declares an **action**; the host runs it and verifies it, and what a database means stays with
|
||||
the module that provides one.
|
||||
- **Whether one host can raise all three.** The claim under stage 2 of
|
||||
[the node host](05-the-node-host.md), never proved. If it is false, the tier boundary moves.
|
||||
- ~~**The vault as the fourth piece.**~~ **Closed 2026-09-21** by
|
||||
[ADR 0085](../../02-DECISIONS/0085-a-secret-is-a-provision.md) as amended: genesis makes the
|
||||
operator key before anything is minted, raises the store and broker with credentials it made
|
||||
rather than the template's, adopts both as modules with those credentials, installs
|
||||
`mesh-vault` beside them, and writes the operator-sealed export next to the key
|
||||
([24](24-the-secrets-vault.md), [issue 071](../../04-ISSUES/071-the-foundation-is-raised-with-fixed-credentials/00-report.md)).
|
||||
Proven by the one-node genesis bed's root-secrets step.
|
||||
- **How the foundation is updated once a mesh exists.** Pinned by hand at bootstrap; afterwards
|
||||
the controller could deliver it like anything else, and nothing says whether it does.
|
||||
|
||||
## Raised, and observed
|
||||
|
||||
*Written 2026-08-30, the first time a bare machine became a running mesh and something joined it.*
|
||||
|
||||
**It works, and what that means precisely:** a machine with a container runtime and nothing else
|
||||
applied the bundle its host carries and ended with a store, a database per context, those
|
||||
contexts' schemas, a broker holding a certificate it generated itself, and the controller
|
||||
serving on top of them. Eleven resources, one command, no mesh to ask anything of.
|
||||
|
||||
**Then it joined itself.** The same machine took a token, checked the broker against the
|
||||
fingerprint pinned in it, generated three keypairs, and enrolled — which is
|
||||
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *the first node is a node whose
|
||||
mesh is not up yet*, observed rather than argued. Its specialness lasted one command.
|
||||
|
||||
**And a credential crossed.** With a second node recorded, the machine was declared the provider
|
||||
of a database and pushed to over the broker. What arrived and what did not is the whole of the
|
||||
[secrets argument](../../02-DECISIONS/0009-modules-and-the-graph.md), measured on a real machine:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| the password, in plain text | **on the machine only**, one file, mode 0600 |
|
||||
| in the declaration that crossed the broker | absent |
|
||||
| in the controller's database | absent |
|
||||
| in what the node reported back | absent |
|
||||
|
||||
**One fault, and it was in the joining.** The token did not say what the mesh calls the machine,
|
||||
so enrolment needed a flag its own help said it did not — and failed at the broker with an empty
|
||||
username. Recorded in ADR 0004 as the fifth thing a token carries.
|
||||
@@ -1,231 +0,0 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-08-30
|
||||
decisions:
|
||||
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
- 02-DECISIONS/0007-connectivity.md
|
||||
- 02-DECISIONS/0008-a-context-owns-its-store.md
|
||||
- 02-DECISIONS/0019-how-this-repository-works.md
|
||||
---
|
||||
|
||||
# The substrate
|
||||
|
||||
Tier 1. Defined the same way [the control plane](06-the-control-plane.md) is, because the same
|
||||
gap applied: the word was load-bearing and unpinned.
|
||||
|
||||
## The definition
|
||||
|
||||
> **The substrate is what the control plane consumes and cannot grant itself.**
|
||||
|
||||
Every module that needs a database asks the control plane's provisioning for one. The control
|
||||
plane needs a database too — and it cannot ask itself, because it is not running yet. That
|
||||
circularity is not an awkwardness to work around; it *is* the definition. Anything on the wrong
|
||||
side of it must be raised some other way, and the other way is the bundle the host carries
|
||||
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
|
||||
|
||||
The test, applied:
|
||||
|
||||
| | control plane needs it | can it grant itself one? | |
|
||||
|---|---|---|---|
|
||||
| a relational store — **PostgreSQL** | its own state lives there | no — provisioning needs the store | **substrate** |
|
||||
| a message bus — **LavinMQ** | it reaches nodes over it ([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)) | no — it cannot grant itself a virtual host | **substrate** |
|
||||
| an object store — **MinIO** | artifacts and blobs it delivers | no — it needs a bucket to hold them | **substrate** |
|
||||
| an image registry — **the OCI registry** | images it delivers to nodes | no — it needs a repository | **substrate** |
|
||||
| an identity provider | only if it delegates authentication | — | **conditional, below** |
|
||||
| ingress — **Traefik** | not to start; only to be reached by name | — it grants itself one afterwards | **not substrate** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) |
|
||||
| anything else the mesh hosts | no | — | not substrate |
|
||||
|
||||
**The role and the product are both written**, here and everywhere
|
||||
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The role is what the argument
|
||||
turns on — the test above works on roles, and would give the same answers for a different store.
|
||||
The product is what actually gets installed and pinned, and a design that names only the role
|
||||
does not record that the choice was ever made.
|
||||
|
||||
The dependency is on the **protocol**, not the product: AMQP for the bus, S3 for the object
|
||||
store, the OCI protocol for the registry. That is what keeps the naming safe rather than a
|
||||
commitment that cannot be revisited — replacing one is a substrate migration, not a redesign.
|
||||
The store is the exception, and the exception matters: the provisioning model uses databases,
|
||||
roles and schemas as PostgreSQL means them, so it is the one member that is not a swap.
|
||||
|
||||
## What that resolves
|
||||
|
||||
**Four or five?** [Research 006](../../01-RESEARCH/006-mesh-from-scratch/00-overview.md) asks
|
||||
whether the identity provider is a substrate service, and the test answers it *conditionally* —
|
||||
which is the honest answer rather than a number.
|
||||
|
||||
- If the control plane **delegates** authentication, it cannot serve anybody before the provider
|
||||
exists, and it cannot grant itself a client. **Substrate.**
|
||||
- If it **authenticates natively**, the provider is an ordinary hosted service like any other.
|
||||
**Not substrate.**
|
||||
|
||||
So the count follows from a design decision that has not been taken, and the record should say
|
||||
that rather than assert four.
|
||||
|
||||
**Why not "important infrastructure".** An identity provider, a mail server and an analytics
|
||||
service are all infrastructure by any ordinary reading, and none of them are substrate — the
|
||||
control plane starts and runs without them. *Important* is not the test; *the control plane
|
||||
cannot obtain it* is.
|
||||
|
||||
## What the substrate is not
|
||||
|
||||
- **Not tier 0.** The host raises the substrate; it is not part of it. The host carries the
|
||||
declaration that brings the substrate up, and depends on nothing.
|
||||
- **Not the control plane.** These are services with no knowledge of the mesh. A store does not
|
||||
know what a node is.
|
||||
- **Not a place for logic.** The skeleton is explicit: tier 1 is *declarations only, no logic of
|
||||
its own.* A substrate service is an upstream image, pinned, with configuration.
|
||||
- **Not privileged.** The substrate is provisioned *from* by the control plane and grants
|
||||
nothing on its own initiative.
|
||||
|
||||
## The pinned bundle
|
||||
|
||||
`substrate.lock` holds **what must exist before the control plane runs** — which is a smaller
|
||||
set than the substrate, and the difference is easy to miss. It is the only place in the mesh
|
||||
where versions are pinned by hand rather than resolved.
|
||||
|
||||
Being substrate and being in the bundle are two different questions:
|
||||
|
||||
| | is it substrate? | must it precede the control plane? |
|
||||
|---|---|---|
|
||||
| PostgreSQL | yes — the control plane's own state lives in it | **yes** — there is nowhere to put that state otherwise |
|
||||
| LavinMQ | yes — it cannot grant itself a virtual host | **yes** — the control plane reaches a node only over the link, and the link is the broker ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
|
||||
| MinIO | yes — it cannot grant itself a bucket | no — nothing is delivered before the mesh exists |
|
||||
| the OCI registry | yes — it cannot grant itself a repository | no — the first node fetches upstream ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
|
||||
|
||||
The two on the bottom rows are **substrate by role and ordinary by delivery**: by the time they
|
||||
are wanted there is a control plane, and it provisions them the way it provisions anything.
|
||||
That keeps the bundle to two images rather than four, which is what makes it small enough for the
|
||||
review [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) requires. It was one until
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) established that the broker has
|
||||
to precede the control plane.
|
||||
|
||||
**Why pinned:** the bundle is applied when no mesh exists, so nothing can resolve a version, ask
|
||||
a registry, or check a constraint. What the host carries must already be exact.
|
||||
|
||||
**Why references and not payload:** the bundle names images by **digest** and the host fetches
|
||||
them ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). A first node is
|
||||
a real machine with a network; the sealed case is the lab, and the lab places images itself.
|
||||
|
||||
Reproducibility comes from pinning the identity of a thing rather than carrying its bytes, which
|
||||
is what keeps the bundle small enough for a person to read and check.
|
||||
|
||||
## Raising it
|
||||
|
||||
The order, from [research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md):
|
||||
|
||||
```
|
||||
0 a container runtime exists detected — docker or podman — or installed
|
||||
1 PostgreSQL runs pulled by digest, from the bundle
|
||||
2 a database per context is created an action, run locally — one today, `inventory`
|
||||
3 each context's schema is applied an action, against its own database
|
||||
4 LavinMQ runs pulled by digest, from the bundle
|
||||
5 a virtual host, a credential, and actions, run locally
|
||||
a self-signed certificate
|
||||
6 the control plane starts and only now is there a mesh
|
||||
7 MinIO, the registry, and everything the ordinary path
|
||||
else are provisioned
|
||||
```
|
||||
|
||||
**Steps 4 and 5 are why the bundle is not one image**
|
||||
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The control plane cannot
|
||||
provision the broker, because provisioning means telling a host, and telling a host happens over
|
||||
the broker. The first node does not escape this by being local: it enrols the ordinary way, by
|
||||
dialling the broker at the address in its token.
|
||||
|
||||
**Step 2 is one database per context and not one called `mesh`.** A context is granted only what it
|
||||
exclusively owns ([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), *the mesh
|
||||
database* names a thing that will not exist
|
||||
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)), and a separate
|
||||
database is a boundary a cross-context join cannot casually cross where a separate schema is not.
|
||||
|
||||
Only PostgreSQL is raised from the bundle, for the reason in *The pinned bundle* above — the
|
||||
rest of the substrate is wanted only once there is a control plane to provision it.
|
||||
|
||||
**Step 0 is easy to leave out and it is where several things meet.** A substrate service is a
|
||||
container, so a container runtime must be working before anything else happens — and a runtime
|
||||
is a *package*, not a container.
|
||||
|
||||
**Which runtime is detected, not chosen**
|
||||
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)): a machine that
|
||||
already has one keeps it. On a machine with none, the control plane names the package, because
|
||||
what it is called differs per system. It is:
|
||||
|
||||
- what the host's capability detection already reports, and the first use of that report by
|
||||
something other than a person;
|
||||
- **adopted rather than installed** when the machine already has one with configuration somebody
|
||||
chose ([research 012](../../01-RESEARCH/012-the-minimum-viable-node/00-overview.md));
|
||||
- a package, which needs the machine's own package manager and a network — both permitted by
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md).
|
||||
|
||||
So the host's bootstrap vocabulary is six shapes: **package**, **container**, **file**,
|
||||
**directory**, **service**, and **action**. **All six are built**
|
||||
([`05-the-node-host.md`](05-the-node-host.md) stage 2), so nothing in this bootstrap is
|
||||
blocked on the host any longer.
|
||||
|
||||
**Steps 2 and 3 happen before there is a mesh to do them**, which is why provisioning is part of
|
||||
the bootstrap rather than a service consumers use later. They are **actions** the bundle
|
||||
declares and the host runs
|
||||
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) — so the
|
||||
host's vocabulary grows by one shape rather than by one resource type per substrate service.
|
||||
|
||||
## Open
|
||||
|
||||
- **Whether identity is the fifth.** Above; it follows from a decision not yet taken.
|
||||
- ~~**Whether the bus must precede the control plane.**~~ **Resolved** by
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) — it must, and the question as
|
||||
posed here could not have answered it. This asked whether the control plane's contexts talk to
|
||||
each other over the bus; they do not, being one process, which under this framing would have
|
||||
kept LavinMQ out of the bundle. What decides it is how the control plane reaches a *node*, which
|
||||
is only ever over the link.
|
||||
- **What issues the broker's certificate at bootstrap.** New, and created by the row above. A
|
||||
token pins the fingerprint a host must expect before it sends anything
|
||||
([`09-the-node-lifecycle.md`](09-the-node-lifecycle.md)), so the broker needs a certificate at a
|
||||
moment when there is no mesh to issue one and no public name to obtain one for. Self-signed and
|
||||
pinned is the shape that fits; how it is later replaced by the certificates in
|
||||
[`08-connectivity.md`](08-connectivity.md) is not decided.
|
||||
- **How a context added later gets its database.** By then there is a control plane — but one
|
||||
holding a credential that can create databases holds more than what it exclusively owns
|
||||
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)).
|
||||
- ~~**Whether the host can do step 2.**~~ **Resolved** by
|
||||
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md). A service
|
||||
running on this machine is part of this machine, so the scope was never in question — the real
|
||||
question was whether the host must learn what a database is, and it must not. The bundle
|
||||
declares an **action**; the host runs it and verifies it, and what a database means stays with
|
||||
the module that provides one.
|
||||
- **Whether one host can raise all four.** The claim under stage 2 of
|
||||
[the node host](05-the-node-host.md), never proved. If it is false, the tier boundary moves.
|
||||
- **How the substrate is updated once a mesh exists.** Pinned by hand at bootstrap; afterwards
|
||||
the control plane could deliver it like anything else, and nothing says whether it does.
|
||||
|
||||
## Raised, and observed
|
||||
|
||||
*Written 2026-08-30, the first time a bare machine became a running mesh and something joined it.*
|
||||
|
||||
**It works, and what that means precisely:** a machine with a container runtime and nothing else
|
||||
applied the bundle its host carries and ended with a store, a database per context, those
|
||||
contexts' schemas, a broker holding a certificate it generated itself, and the control plane
|
||||
serving on top of them. Eleven resources, one command, no mesh to ask anything of.
|
||||
|
||||
**Then it joined itself.** The same machine took a token, checked the broker against the
|
||||
fingerprint pinned in it, generated three keypairs, and enrolled — which is
|
||||
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *the first node is a node whose
|
||||
mesh is not up yet*, observed rather than argued. Its specialness lasted one command.
|
||||
|
||||
**And a credential crossed.** With a second node recorded, the machine was declared the provider
|
||||
of a database and pushed to over the broker. What arrived and what did not is the whole of the
|
||||
[secrets argument](../../02-DECISIONS/0009-modules-and-the-graph.md), measured on a real machine:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| the password, in plain text | **on the machine only**, one file, mode 0600 |
|
||||
| in the declaration that crossed the broker | absent |
|
||||
| in the control plane's database | absent |
|
||||
| in what the node reported back | absent |
|
||||
|
||||
**One fault, and it was in the joining.** The token did not say what the mesh calls the machine,
|
||||
so enrolment needed a flag its own help said it did not — and failed at the broker with an empty
|
||||
username. Recorded in ADR 0004 as the fifth thing a token carries.
|
||||
@@ -1,8 +1,13 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-08-29
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller internal/catalogue/filtering.go
|
||||
- mesh-controller examples/route-proxy
|
||||
- mesh-controller internal/identity/authority.go
|
||||
- mesh-host internal/identity/serving.go
|
||||
- mesh-host internal/apply (the service that reflects a rule set)
|
||||
updated: 2026-09-09
|
||||
decisions:
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
@@ -11,29 +16,33 @@ decisions:
|
||||
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
- 02-DECISIONS/0007-connectivity.md
|
||||
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
- 02-DECISIONS/0066-public-routing-is-name-agnostic.md
|
||||
- 02-DECISIONS/0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md
|
||||
- 02-DECISIONS/0044-a-public-name-is-provisioned-like-any-capability.md
|
||||
- 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md
|
||||
---
|
||||
|
||||
# Connectivity
|
||||
|
||||
One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most
|
||||
One of [the controller's](06-the-controller.md) ten contexts, and the one with the most
|
||||
moving parts: **overlay, resolution, exposure, filtering, certificates.**
|
||||
|
||||
It is written as a whole because the five are one design. They share inputs, they must agree, and
|
||||
every one of them today is computed in a different place by a different module from a different
|
||||
copy of the same facts.
|
||||
|
||||
## Why it is control-plane work
|
||||
## Why it is controller work
|
||||
|
||||
Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one
|
||||
Apply [the test](06-the-controller.md) — *everything that needs to know about more than one
|
||||
node* — to each responsibility:
|
||||
|
||||
| | needs to know | whose |
|
||||
|---|---|---|
|
||||
| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane |
|
||||
| **resolution** — which name is which node | **every node** | control plane |
|
||||
| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | control plane |
|
||||
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies |
|
||||
| **certificates** — who may present which name | which name belongs to which node | control plane |
|
||||
| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | controller |
|
||||
| **resolution** — which name is which node | **every node** | controller |
|
||||
| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | controller |
|
||||
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | controller decides, host applies |
|
||||
| **certificates** — who may present which name | which name belongs to which node | controller |
|
||||
|
||||
**Not one of the five can be answered by a machine on its own.** That is the whole reason this is
|
||||
a context rather than a set of node-local modules — and it is exactly what the current
|
||||
@@ -51,7 +60,7 @@ exist; WireGuard, the resolver and the proxy are all *a container or a package,
|
||||
|
||||
It is also what removes the last two upward dependencies.
|
||||
[Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules
|
||||
opening a direct connection to the control plane's database — `wireguard` and `traefik` — and
|
||||
opening a direct connection to the controller's database — `wireguard` and `traefik` — and
|
||||
they are the reason every node permanently holds a credential to it
|
||||
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Both are connectivity
|
||||
modules. **Closing this context closes that set.**
|
||||
@@ -68,7 +77,7 @@ wanted the exception.
|
||||
|
||||
**What made it look unavoidable:** a peer list cannot be written in a manifest. It is derived from
|
||||
every other machine, so it differs on each one and changes when any of them changes. So the
|
||||
manifest says its resources are **computed** — it names something in the control plane that works
|
||||
manifest says its resources are **computed** — it names something in the controller that works
|
||||
them out per node — and it is a module in every other respect: assigned, resolved, configured by
|
||||
settings, and absent from a machine nobody gave it to.
|
||||
|
||||
@@ -102,7 +111,7 @@ claim, and the collision is refused by name.
|
||||
**And the proxy's half, which was the other module reaching into the database.** A web application
|
||||
requiring a reverse proxy has to say *which name, which port*, and there was nowhere to put it —
|
||||
`requires` says a thing must exist and never said what to do with it. A module now contributes to
|
||||
a requirement, the control plane collects every contribution on a node, and the provider is given
|
||||
a requirement, the controller collects every contribution on a node, and the provider is given
|
||||
them as a file at a path it named. It reloads when that file changes, by the same `restart-on` the
|
||||
private network needed when a peer list changed under a running interface.
|
||||
|
||||
@@ -171,7 +180,7 @@ which of those it may dial, and which must dial it.
|
||||
**Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the
|
||||
public key is published to the mesh. This is already true and it is already right — it is
|
||||
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *a node holds its own
|
||||
identity* applied to the overlay, and it means the control plane computes a graph it cannot
|
||||
identity* applied to the overlay, and it means the controller computes a graph it cannot
|
||||
itself impersonate.
|
||||
|
||||
**Shape: a hub, with direct peering between co-located nodes.**
|
||||
@@ -211,7 +220,7 @@ files were right, the services were up, and every node reported success.
|
||||
document's own warning, arriving in its implementation: *a more specific route to a dead
|
||||
endpoint blackholes; it does not fall back to the general one.*
|
||||
- **The container runtime closes the door the overlay needs.** Docker sets the FORWARD policy to
|
||||
DROP, so a hub with `ip_forward` enabled still carries nothing between its spokes. The substrate
|
||||
DROP, so a hub with `ip_forward` enabled still carries nothing between its spokes. The foundation
|
||||
at tier 1 silently breaks the network at tier 2, and nothing in either tier's state says so. The
|
||||
hub inserts its own rule above those chains and removes it on the way down.
|
||||
|
||||
@@ -245,6 +254,120 @@ name and overlay address.
|
||||
database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)
|
||||
nothing needs a name before the link, and a fallback nothing needs is a path nothing tests.
|
||||
|
||||
### Names, and what a container can see
|
||||
|
||||
*2026-08-31, from a container that could not resolve a name every machine could.*
|
||||
|
||||
Internal names are `<node>.internal` — the suffix is the one IANA reserved in 2024, so a name that
|
||||
leaks into a public resolver fails rather than reaching a stranger's machine. They are computed
|
||||
centrally, because a name set needs every node at once, and written to each machine's hosts file.
|
||||
|
||||
**A file rather than a resolver**, and the reasoning holds: it works on every Linux, needs no
|
||||
package, and has no failure mode of its own. The stated trigger for a daemon was *names that are
|
||||
not one-per-node* — service names, wildcards.
|
||||
|
||||
**But a container does not inherit the machine's names.** It gets its own hosts file holding only
|
||||
its own hostname. So every name the mesh wrote was invisible to the majority of things that need
|
||||
one — and *on the machine it always worked*, which is exactly what made it easy to miss. It was
|
||||
found by a database client on one node failing to resolve another node, on a mesh where both names
|
||||
were correct and present on both machines.
|
||||
|
||||
**So the mesh gives its names to the containers it declares**, written into each container's own
|
||||
hosts file by the runtime. That extends the file decision rather than overturning it. Given by the
|
||||
mesh and not chosen by a module: a module that listed the machines would go stale the day one
|
||||
joins, and a module that did not would be one whose containers cannot reach anything by name.
|
||||
|
||||
**The boundary, which is deliberate and worth stating:** *declared* containers. A container
|
||||
somebody starts by hand is not the mesh's to configure, and reaching into every container on a
|
||||
machine — declared or not — is what a nameserver in `resolv.conf` would be for.
|
||||
|
||||
### The resolver, built
|
||||
|
||||
*2026-08-31.* **A service is reached at `<service>.<node>.internal`** — the first label is the
|
||||
service, the rest is the node — so what resolves is *anything under a node's name*, going to that
|
||||
node. What routes it once it arrives is a proxy's, and stays separate.
|
||||
|
||||
**The mesh writes the data and runs no daemon.** One wildcard per machine, from the same set that
|
||||
writes the hosts file. A resolver is third-party software and runs *on* the mesh rather than being
|
||||
*of* it: the mesh has no business shipping one, choosing which one, or knowing its configuration
|
||||
language. Swapping dnsmasq for unbound changes that module and nothing in the controller.
|
||||
|
||||
**Two roles, two claims, because they are different things.** systemd-resolved cannot answer a
|
||||
wildcard at all — it routes the mesh's suffix to something that can. Treating serving and asking
|
||||
as one role produces a module that cannot work.
|
||||
|
||||
| | claims | |
|
||||
|---|---|---|
|
||||
| serving | `the-dns-port` | answers the wildcards |
|
||||
| asking | `the-resolver-configuration` | decides what the machine asks |
|
||||
|
||||
So *which* resolver is not a mesh-wide decision. One machine can use what systemd already owns and
|
||||
another can run dnsmasq, and two of either on one machine is refused rather than fought over.
|
||||
|
||||
**Two things a resolver must not do**, both found by a machine rather than by reasoning:
|
||||
|
||||
- **Take an address something else holds.** systemd-resolved holds `127.0.0.53` *and* `127.0.0.54`.
|
||||
- **Read `resolv.conf` for its upstreams.** Whatever points a machine at the mesh writes the
|
||||
resolver's own address there, so it becomes its own upstream and every query it cannot answer
|
||||
loops until its receive queue fills. It needs no upstream: only the mesh's suffix is routed to
|
||||
it.
|
||||
|
||||
*Checked on two machines, through the path an application takes — nsswitch, files, then DNS —
|
||||
because the module deciding what the machine asks is half of what is being tested and only that
|
||||
path goes through it.*
|
||||
|
||||
**That is now the second reason to want a resolver**, and it is a different one from the trigger
|
||||
above:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| names that are not one-per-node | a service named under a machine — `postgres.novox.internal` |
|
||||
| containers the mesh did not declare | anything a person or another tool starts on a node |
|
||||
|
||||
### Which resolver is not a question the mesh answers
|
||||
|
||||
*Written 2026-08-31, after treating it as open when it had been decided two days earlier.*
|
||||
|
||||
**A resolver takes over `/etc/resolv.conf`, which is a singular resource, so it is a claim** —
|
||||
[ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md) lists it in the table beside the seat
|
||||
and pid 1. Choosing between resolved, dnsmasq and unbound is **assigning a module**, per machine,
|
||||
and two of them cannot both be assigned there:
|
||||
|
||||
> `resolved-config and dnsmasq both claim "/etc/resolv.conf", and only one thing may hold it per node`
|
||||
|
||||
So there is nothing global to settle and nothing for the mesh to guess. One machine can use what
|
||||
systemd already owns and another can run dnsmasq, and neither has to know about the other.
|
||||
|
||||
**What the mesh contributes is the part only it can know**: which machines exist and where. That
|
||||
is `mesh-resolver`, which writes one file and holds no claim, because writing a file takes nothing
|
||||
over. A daemon module requires that data and claims the resolver — so swapping the daemon changes
|
||||
that module and nothing else.
|
||||
|
||||
**This was recorded on 2026-08-29 and reopened as an unanswered question on the 31st.** Which is
|
||||
the argument for the table in ADR 0009 being a table: the pattern is only obvious once seen, and
|
||||
the cost of not seeing it is inventing a mechanism that already exists.
|
||||
|
||||
### And the public names a proxy serves must resolve in the mesh too
|
||||
|
||||
*2026-09-09, found by an internal certificate authority that could not issue.* The mesh writes every
|
||||
`<node>.internal` name into every declared container and treats the public names a proxy serves as a
|
||||
separate matter — *what routes it once it arrives is a proxy's, and stays separate*, above. That
|
||||
holds for a client dialling by internal name. It does not hold for anything **inside** the mesh that
|
||||
must reach a public name, and the first such thing to appear was the internal issuer of §5.
|
||||
|
||||
**An issuer validates by connecting to the name it is certifying.** Asked for a certificate for a
|
||||
routed public name, the internal authority accepted the order, offered a challenge, and then could
|
||||
not connect: nothing in the mesh resolved that name, so the challenge had no target. A name the mesh
|
||||
can reach from the outside but cannot resolve from the inside is a name it cannot certify with an
|
||||
authority of its own.
|
||||
|
||||
**So a granted route is published into internal resolution as well** — the routed name to the node
|
||||
that serves it, mesh-wide, by the same mechanism that writes the node names. It is *given by the
|
||||
mesh, not chosen by a module*, for the same reason the node names are: a module listing the routes
|
||||
would go stale the day one changes. The mesh propagates the names it was told to serve and still
|
||||
knows nothing about what they mean
|
||||
([ADR 0066](../../02-DECISIONS/0066-public-routing-is-name-agnostic.md)).
|
||||
|
||||
## 3 — Exposure
|
||||
|
||||
Settled by [ADR 0007](../../02-DECISIONS/0007-connectivity.md); summarised here because
|
||||
@@ -257,7 +380,60 @@ vocabulary — the mirror of a database grant, where the consumer supplies a tar
|
||||
name rather than supplying nothing and receiving credentials.
|
||||
|
||||
**A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is
|
||||
the case is a mesh-level fact, which is the fourth reason exposure is control-plane work.
|
||||
the case is a mesh-level fact, which is the fourth reason exposure is controller work.
|
||||
|
||||
### What was built
|
||||
|
||||
*2026-08-31.* Nothing new in the vocabulary, which was the claim and is now the fact: a route is a
|
||||
provision, a proxy provides it, and a module that must be reachable requires it. The consumer
|
||||
contributes the name it wants and the port it listens on; the proxy receives every consumer that
|
||||
asked; the consumer is told what the provider serves, which is how it knows its own name.
|
||||
|
||||
**One field was missing, and it is the one anything reaching back needs.** A contribution now
|
||||
carries **where the mesh says that machine is**. A database is reached *by* its consumer, so the
|
||||
mesh never had to tell a provider where anybody was; a proxy is the other direction — it is told
|
||||
to send traffic to a consumer and has to open a connection. Without it every provider implementing
|
||||
a provision would have to know how the mesh names machines, which is a convention leaking into
|
||||
every module.
|
||||
|
||||
**Exposure and filtering are different questions and a module answers both.** A workload says what
|
||||
it listens on and who may reach it; separately, it says it wants a route. A module that asked for a
|
||||
route and not for the port is unreachable by the proxy it just asked for — which the lab
|
||||
demonstrates, because the machine is already filtering by the time this runs.
|
||||
|
||||
**Withdrawal, which was open above.** The file the proxy is given is the whole truth about who has
|
||||
a route, so a proxy replaces its table rather than merging. Merging would keep serving a name whose
|
||||
module was unassigned — and *a stale public name pointing at nothing fails more visibly than a
|
||||
stale grant* is the reason it must not survive, not a reason to tolerate it.
|
||||
|
||||
**A name a proxy does not serve is refused by saying which it does.** A route that was withdrawn
|
||||
and a name that never existed are different things, and a bare 404 makes an operator go and read
|
||||
the mesh to tell them apart.
|
||||
|
||||
*Checked in the lab by a request to the name reaching the workload across the private network and
|
||||
returning the workload's own answer, then by unassigning the module and requiring the same request
|
||||
to stop working.*
|
||||
|
||||
### The name is a label, not a domain
|
||||
|
||||
*2026-09-09, from running the whole mesh in the lab.* A route contribution carried its public name
|
||||
in full — the forge as `git.example.tld`, spelt out in the module. Pointing the same catalogue at a
|
||||
different domain — a lab standing in for production, or a second operator's mesh — meant rewriting
|
||||
that name on every routed module. The mesh was holding a **map of names to services**, which is the
|
||||
one thing it must not: a public name is two facts owned by two different places, and neither is the
|
||||
module's manifest.
|
||||
|
||||
**A module contributes a label; the node contributes its public domain; the mesh composes.** The
|
||||
operator chooses where the forge lives — `git`, or `code` — and that is the module's to say. The
|
||||
domain is the node's, set once. The mesh joins them and grants `<label>.<public-domain>`,
|
||||
interpreting neither half. Moving a mesh to another domain is one node setting, not an edit per
|
||||
module.
|
||||
|
||||
**Today the name is still a literal, and that is the gap.** There is no interpolation of a node's
|
||||
domain into a module's label, so the composition is done by a per-node override — which reproduces
|
||||
exactly the per-module cost it is meant to remove. The design is the composition; the override is a
|
||||
stopgap until the manifest layer can carry a label and a domain separately
|
||||
([ADR 0066](../../02-DECISIONS/0066-public-routing-is-name-agnostic.md)).
|
||||
|
||||
## 4 — Filtering
|
||||
|
||||
@@ -275,6 +451,85 @@ from a wrong one, and costs more, because people believe it.*
|
||||
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and
|
||||
the one manifests lack. `scope:` survived because nothing rejected it.
|
||||
|
||||
### What was built
|
||||
|
||||
*2026-08-31. Everything above was the intention; this is what exists, and how each part is
|
||||
checked. [04-ISSUES/003](../../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) is
|
||||
resolved by it.*
|
||||
|
||||
**A module says what it listens on**, as a port, a protocol and a source — `mesh`, `anywhere`, or
|
||||
`machine`. The source is required and there is no default, which is the whole of *a rule names its
|
||||
source*: a manifest that omitted it would read as a restriction and be none. *Checked by a manifest
|
||||
with a port and no source being refused, and by one naming a source the mesh cannot render being
|
||||
refused as well — the second is what stops a source becoming a comment.*
|
||||
|
||||
**The set is derived per node**, from every module assigned to it, not from the module asking for
|
||||
it. Where two modules want the same port, the wider source wins and both are still named, because
|
||||
removing one of them must not read as a reason to close a port the other needs. *Checked by
|
||||
rendering a node whose firewall module has no ports of its own and asserting another module's port
|
||||
is in the result; and by giving one port two modules and one source each, and asserting the
|
||||
narrower rule disappears while both names survive.*
|
||||
|
||||
**What is not declared is closed.** The rule set drops by default. *Checked by naming the input
|
||||
chain in the assertion rather than the policy alone — the first version of that test passed while
|
||||
input accepted everything, because another chain in the same file also said `policy drop`.*
|
||||
|
||||
**From the mesh means the machines the mesh has**, as their addresses on the private network, not
|
||||
as a subnet. A subnet is a guess that stays wrong quietly; the address set shrinks when a node
|
||||
leaves and nobody edits anything. A machine that asks for `mesh` where the mesh knows no addresses
|
||||
is **closed and told so in the file** — widening it would open a port nobody asked to open, and
|
||||
dropping it silently would close one somebody did.
|
||||
|
||||
**Three things it deliberately does not do**, each of which looked right and would have broken
|
||||
something:
|
||||
|
||||
| | why not |
|
||||
|---|---|
|
||||
| decide what the machine **forwards** | the container runtime writes its own forwarding rules and a second policy is consulted alongside them, so a drop here stops every container on the node — the controller included. Nothing in a manifest says what a machine routes, so there is nothing to derive it from either |
|
||||
| **flush the ruleset** when loading | that empties every table on the machine, the runtime's among them. Only the mesh's own table is replaced, and it is declared empty first so the replacement works on a machine loading one for the first time |
|
||||
| carry a **command to load itself** | the link may not carry an action ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). A service is declared to reflect the file instead, so replacing it restarts what loads it — the shape that rule leaves, used here for the first time for its real purpose |
|
||||
|
||||
**A module that wants a rule set brings the unit that loads it.** Found the hard way: the unit a
|
||||
distribution packages for nftables runs, applies the rules and exits, so it is neither running nor
|
||||
stopped — and a host asked for a service that is "running" reports, quite correctly, that it is
|
||||
stopped. Every packet was filtered exactly as declared and the machine was marked as not doing what
|
||||
it was told.
|
||||
|
||||
**The vocabulary has no word for "ran, did its job, and exited"**, and that is a real gap rather
|
||||
than a wording problem: the whole class of configuration-applying units — packet filters, sysctl,
|
||||
tmpfiles — is shaped that way. Until there is one, a module ships a unit that stays, which is also
|
||||
the better shape: how a machine enforces rules is a fact about the machine, and the mesh has no
|
||||
business depending on what a distribution happens to package.
|
||||
|
||||
**One rule is derived from the overlay's shape rather than from what is assigned: a hub's own
|
||||
listening port.** A hub accepts inbound connections from every node at other sites; a machine that
|
||||
is not a hub dials out and needs nothing open, because a reply to a flow it started is already
|
||||
accepted. The two want different rules on an *identical module*, so `listens` — a static field —
|
||||
cannot say it. The machine a static answer gets wrong is the one facing the public internet, which
|
||||
is the machine that most needs filtering.
|
||||
|
||||
*Recorded as a gap on 2026-08-31 and closed the same day.* **A computed module now contributes
|
||||
listens the way it contributes resources.** The port comes from the endpoint, which is where the
|
||||
interface takes its `ListenPort` from — one source, so a rule set cannot open a port the interface
|
||||
is not on. It is open to *everywhere* deliberately: a node at another site is not on the private
|
||||
network until this port lets it on, so restricting it to the mesh would be a rule that can never
|
||||
be satisfied by the thing it exists for.
|
||||
|
||||
**A generator that cannot say what a machine opens is refused, not read as silence.** Closing a
|
||||
port on the evidence of a failure to look is how a machine is severed by a fault somewhere else —
|
||||
and the machine it would sever is the hub, whose only route to being repaired is the network it
|
||||
just closed.
|
||||
|
||||
*Checked by filtering the hub and then requiring the mesh to keep working: a declaration still
|
||||
reaches the other machine, and the other machine still reaches the hub. A rule file that looks
|
||||
right and a mesh that has stopped are exactly what that guards against.*
|
||||
|
||||
**And it is enforced, which is what separates this from `scope:`.** Checked on two real machines:
|
||||
two ports opened, one declared, and from the other machine the declared one answers and the
|
||||
undeclared one does not — then the module is removed and the port closes with nobody editing a
|
||||
rule. *A rule set that is written but never loaded passes every check that reads the file, which
|
||||
is why the check reads packets.*
|
||||
|
||||
## 5 — Certificates
|
||||
|
||||
**Two authorities, kept separate on purpose.**
|
||||
@@ -297,11 +552,57 @@ defaults to the public authority's *production* endpoint. Two consequences, and
|
||||
worse than the lab problem that found it — every certificate experiment on a real node consumes
|
||||
production issuance quota, and a retry loop can exhaust it for a week.
|
||||
|
||||
**The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the
|
||||
**The mesh CA is not a bootstrap concern.** A joining node verifies the controller against the
|
||||
fingerprint in its token ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)),
|
||||
so nothing needs the CA before membership. It certifies internal names afterwards, and that is
|
||||
all it does.
|
||||
|
||||
### What was built
|
||||
|
||||
*2026-08-31.*
|
||||
|
||||
**A node generates a fourth key**, and the reason is the one the other three already give: *a key
|
||||
used for two purposes is one rotation away from breaking the other.* The identity key would work
|
||||
for TLS and reusing it would mean rotating a node's identity every time its certificate is
|
||||
replaced. The private half never leaves the machine; the mesh is told the public half at
|
||||
enrolment.
|
||||
|
||||
**So there is no certificate request and nothing to seal.** The mesh signs a statement binding a
|
||||
public key to a name it alone assigns, which is the whole of what a certificate authority does.
|
||||
It issues rather than stores: the node's key does not change, so signing again produces an equally
|
||||
valid certificate and there is nothing to keep in step.
|
||||
|
||||
**A machine with no name inside the mesh is refused**, not given a certificate for nothing. A
|
||||
certificate for a name nothing resolves is a certificate nothing can check.
|
||||
|
||||
**And the key is stored in the format a server reads** — PKCS#8 PEM, not the host's own encoding.
|
||||
That is not an implementation detail of whoever writes the file: the file exists *because
|
||||
something else reads it*, so the format is the interface
|
||||
([04-ISSUES/014](../../04-ISSUES/014-a-key-that-is-present-and-unusable/00-report.md)).
|
||||
|
||||
*Checked by a real handshake between two machines: one serves on its internal name with the key it
|
||||
generated, the other verifies against the mesh's authority and nothing else. Every cheaper check
|
||||
passed while the server could not start — the key was present, the certificate was valid, and
|
||||
nothing read either the way a server would.*
|
||||
|
||||
### An internal issuer, pointed at and trusted
|
||||
|
||||
*2026-09-09, from wiring one to the proxy in the lab.* §5 above asks for two things this build
|
||||
leaned on: the issuer must be configurable, and the lab runs its own ACME authority rather than
|
||||
collapsing the split. Wiring the proxy to that authority is the whole of it — the proxy is told
|
||||
**which** issuer to use and given that issuer's **root** to trust, and every other step of issuance
|
||||
is unchanged. **The same code path certifies against an internal authority as against a public one;
|
||||
only the issuer differs.** That is what makes trusted certificates possible for a mesh whose names
|
||||
the public internet cannot resolve.
|
||||
|
||||
**And it does not work until the routed name resolves inside the mesh** — the §2 finding above,
|
||||
arriving here because this is what needed it. The authority's challenge reaches the routed name only
|
||||
once that name is in internal resolution; a public authority is handed that dependency by public
|
||||
DNS, and an internal one has to be handed it by the mesh. *Checked by a handshake to a routed name
|
||||
that verifies against the internal root and nothing else — which cannot succeed unless the issuer
|
||||
first reached the name to certify it*
|
||||
([ADR 0066](../../02-DECISIONS/0066-public-routing-is-name-agnostic.md)).
|
||||
|
||||
## What this removes
|
||||
|
||||
The list is worth having in one place, because it is most of the argument:
|
||||
@@ -328,9 +629,19 @@ The list is worth having in one place, because it is most of the argument:
|
||||
- **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from
|
||||
an address, but no procedure exists, and a graph delivered node by node has an ordering problem
|
||||
while it is half-applied.
|
||||
- **Revoking a route** when a module is unassigned ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)).
|
||||
A stale public name pointing at nothing fails more visibly than a stale grant.
|
||||
- ~~**Revoking a route** when a module is unassigned.~~ **Resolved** 2026-08-31 — see §3. The file
|
||||
a proxy is given is the whole truth about who has a route, so a route does not outlive the module
|
||||
that asked for it.
|
||||
- **IPv6.** [ADR 0007](../../02-DECISIONS/0007-connectivity.md) makes
|
||||
it expressible; nothing here says the overlay or the resolver handle it.
|
||||
- **Reporting declared-versus-observed.** ADR 0007 makes the disagreement detectable and does not
|
||||
say who looks or what they are told.
|
||||
- **Composing a route name from a label and a node's domain.**
|
||||
[ADR 0066](../../02-DECISIONS/0066-public-routing-is-name-agnostic.md) makes the public name the
|
||||
operator's to move between meshes, but the manifest layer still stores it as a literal — so today
|
||||
the composition is a per-node override rather than the design. The interpolation that would let a
|
||||
module carry a label and a node carry the domain, and the mesh join them, does not yet exist.
|
||||
- **Publishing route names into internal resolution.** The same ADR requires a granted route to be
|
||||
resolvable inside the mesh, not only routable from outside it; the mechanism that writes
|
||||
`<node>.internal` into containers does not yet also write the routed names, which is why an
|
||||
internal issuer cannot currently validate one without a hand-placed entry.
|
||||
|
||||
@@ -1,8 +1,14 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-08-27
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-host internal/link/run.go
|
||||
- mesh-host internal/link/enrol.go
|
||||
- mesh-host packaging/nox-mesh-host-resume.service
|
||||
- mesh-host packaging/nox-mesh-host-network.sh
|
||||
- mesh-controller internal/token
|
||||
- mesh-controller internal/inventory/nodes.go
|
||||
updated: 2026-08-31
|
||||
decisions:
|
||||
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
@@ -136,12 +142,12 @@ nox-mesh-host enrol --token <one-time token>
|
||||
|
||||
The token carries **four** things and is carried by a person
|
||||
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)): the broker's
|
||||
address, the fingerprint to expect, **the control plane's signing identity**, and the right to
|
||||
address, the fingerprint to expect, **the controller's signing identity**, and the right to
|
||||
join once.
|
||||
|
||||
**The fourth is the one this document listed three of.** A node connects to the broker and takes
|
||||
instruction from the control plane behind it, and those are two different identities. Pinning only
|
||||
the broker would make the control plane's authority *transitive* — a compromised broker could then
|
||||
instruction from the controller behind it, and those are two different identities. Pinning only
|
||||
the broker would make the controller's authority *transitive* — a compromised broker could then
|
||||
forge declarations, which, since the host applies whatever the link delivers, is the whole machine.
|
||||
So the transport is verified once at connect, and **each declaration is verified by its signature,
|
||||
every time**.
|
||||
@@ -152,10 +158,10 @@ What happens, in order:
|
||||
2. it checks the broker's certificate against the pinned fingerprint — *before* sending anything;
|
||||
3. it presents the one-time secret **and its own public key**, which the mesh records;
|
||||
4. it reports its `profile` and `inventory` upward;
|
||||
5. the control plane decides what this machine should be, and sends a declaration;
|
||||
5. the controller decides what this machine should be, and sends a declaration;
|
||||
6. the host applies it, reads back, and reports.
|
||||
|
||||
**Step 4 is the one that is easy to miss and is what makes step 5 possible.** The control plane
|
||||
**Step 4 is the one that is easy to miss and is what makes step 5 possible.** The controller
|
||||
cannot decide what a machine should run without knowing what it *can* run — a graphical session,
|
||||
a container runtime, an architecture. The profile is not a diagnostic; it is the input.
|
||||
|
||||
@@ -203,7 +209,7 @@ channel, not about network reachability.** What it forbids is a listening thing
|
||||
instructions and changes the machine. A node being reachable on the overlay — the whole purpose
|
||||
of the overlay — is untouched by it, and so is a person opening a shell on it.
|
||||
|
||||
The distinction is *who can tell this machine what to be*: only the control plane, only over the
|
||||
The distinction is *who can tell this machine what to be*: only the controller, only over the
|
||||
link the node opened, only in declarations of known shape.
|
||||
|
||||
---
|
||||
@@ -213,18 +219,18 @@ link the node opened, only in declarations of known shape.
|
||||
The same path, with the mesh built in the middle of it.
|
||||
|
||||
```
|
||||
# 1 — raise the substrate and the control plane from the carried bundle
|
||||
# 1 — raise the foundation and the controller from the carried bundle
|
||||
nox-mesh-host reconcile
|
||||
|
||||
# 2 — the control plane now exists, and issues the first token
|
||||
mesh-control token issue
|
||||
# 2 — the controller now exists, and issues the first token
|
||||
mesh-controller token issue
|
||||
|
||||
# 3 — the machine joins the mesh it just raised
|
||||
nox-mesh-host enrol --token <token>
|
||||
```
|
||||
|
||||
Step 1 is the bootstrap from [`07-the-substrate.md`](07-the-substrate.md): a container runtime,
|
||||
then PostgreSQL, then the database, then the schema, then the control plane. It needs no identity
|
||||
Step 1 is the bootstrap from [`07-the-foundation.md`](07-the-foundation.md): a container runtime,
|
||||
then PostgreSQL, then the database, then the schema, then the controller. It needs no identity
|
||||
because nothing is being asked of anyone — the host is applying a declaration it already
|
||||
carries, to the machine it is already on.
|
||||
|
||||
@@ -232,7 +238,7 @@ carries, to the machine it is already on.
|
||||
the bootstrap script never had. Its specialness lasted two commands.
|
||||
|
||||
**And enrolment is exercised on node one.** The path every other node depends on is walked
|
||||
immediately, against a control plane on the same machine, rather than being written and first
|
||||
immediately, against a controller on the same machine, rather than being written and first
|
||||
used months later on node two.
|
||||
|
||||
---
|
||||
@@ -259,7 +265,7 @@ closes — an authoritative local store, reconcile on start, *last heard from* r
|
||||
alarm — is what an episodic host needs, at a shorter period.
|
||||
|
||||
**It cannot be the first node**, and that is not a limitation to work around. Every step of
|
||||
raising a substrate is a `package`, a `container` or an `action` against one, and a partial host
|
||||
raising a foundation is a `package`, a `container` or an `action` against one, and a partial host
|
||||
refuses the first two. So `mesh-host-android bundle` returns a file that says so rather than an
|
||||
empty placeholder waiting to be filled in.
|
||||
|
||||
@@ -344,7 +350,7 @@ rest. The rule that exists to stop the host lying about what it did also makes i
|
||||
|
||||
## Updating what the node holds
|
||||
|
||||
An ordinary declaration. Someone assigns a module; the control plane recomputes what that node
|
||||
An ordinary declaration. Someone assigns a module; the controller recomputes what that node
|
||||
should be and sends it; the host applies the difference and removes what is no longer declared.
|
||||
|
||||
**Removal is not symmetric, and the asymmetry is the design:**
|
||||
@@ -404,7 +410,7 @@ one binary that has always been the same binary.
|
||||
|
||||
Two cases, and they are genuinely different.
|
||||
|
||||
**Graceful.** The control plane sends a final declaration that names nothing. The host removes
|
||||
**Graceful.** The controller sends a final declaration that names nothing. The host removes
|
||||
what it owns by the table above, reports, and drops its identity. The machine keeps the host
|
||||
installed and is back to `hosted`. Nothing is left behind that anybody has to remember.
|
||||
|
||||
@@ -542,8 +548,8 @@ rather than left to be worked out.
|
||||
**A token is issued *for* a node record**, and that is where a re-enrolment is decided.
|
||||
|
||||
```
|
||||
mesh-control token issue --node workstation # this machine is that node again
|
||||
mesh-control token issue --new # a machine the mesh has not seen
|
||||
mesh-controller token issue --node workstation # this machine is that node again
|
||||
mesh-controller token issue --new # a machine the mesh has not seen
|
||||
```
|
||||
|
||||
The host does not need to know which it is. It presents a token and receives an identity; what
|
||||
@@ -623,7 +629,7 @@ keeps cataloguing.
|
||||
|
||||
### Where the enrolment token comes from
|
||||
|
||||
**`mesh-control token issue` prints it once**, to the person running it. Single-use, and it
|
||||
**`mesh-controller token issue` prints it once**, to the person running it. Single-use, and it
|
||||
expires whether used or not ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
|
||||
|
||||
It is carried by hand — read off a screen, pasted into a terminal. That is the design rather than
|
||||
@@ -632,7 +638,7 @@ lets a node verify a mesh it has never spoken to
|
||||
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). A token emailed,
|
||||
committed, or dropped in shared storage has lost the only property that makes it worth carrying.
|
||||
|
||||
**On the first node it comes from the control plane that was raised two commands ago**, which is
|
||||
**On the first node it comes from the controller that was raised two commands ago**, which is
|
||||
the same command against a mesh that is one machine old.
|
||||
|
||||
---
|
||||
@@ -647,3 +653,44 @@ the same command against a mesh that is one machine old.
|
||||
- **How a previous declaration is retained and chosen**, which is what rollback of anything else
|
||||
would use ([ADR 0010](../../02-DECISIONS/0010-delivery.md)).
|
||||
- **A node returning after months** applies a very large jump in one go. Correct, and untested.
|
||||
|
||||
## Going away and coming back
|
||||
|
||||
*2026-08-31, from being asked whether a machine that drops off has to be adopted again.*
|
||||
|
||||
**It does not, and nothing about it expires.** A disconnected node is the same node in a different
|
||||
situation ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) — reachability is state,
|
||||
not class. The node holds its own identity and the mesh holds the public half; there is no lease,
|
||||
no timeout, and nothing that lapses while a machine is shut. A laptop closed for a week comes back
|
||||
and reconnects, and a declaration sent while it was away is waiting for it.
|
||||
|
||||
**The only thing that forces re-enrolment is a machine losing its own key** — a reinstall, an image
|
||||
re-cloned. That is deliberate: the mesh then reports what it can no longer seal to rather than
|
||||
delivering blobs the machine cannot open.
|
||||
|
||||
### The gap that was left, and why it mattered
|
||||
|
||||
**A machine that suspends does not know it has been disconnected, and neither does anything else.**
|
||||
After a resume the socket looks perfectly healthy from inside the process: no error, no close,
|
||||
because nothing has tried to send anything yet. Heartbeats discover it twenty or thirty seconds
|
||||
later.
|
||||
|
||||
For those twenty or thirty seconds the node believes it is in the mesh and is not — and *absence
|
||||
must never be indistinguishable from a failure to answer* is the rule this whole design is built
|
||||
on. It recovered on its own, which is why this was a quality gap rather than a fault. It was still
|
||||
the machine waiting to be told something it already knew.
|
||||
|
||||
**So the machine says so.** Waking, and changing network, both rouse the host.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **it ends the current attempt** | not the wait after it — the process is not waiting, it is sitting inside a connection that will never return |
|
||||
| **by signal, not by anything listening** | a socket for this would be a control surface on every machine, in exchange for saving twenty seconds, and the security argument rests on there not being one |
|
||||
| **two rouses at once are one** | a machine suspending and resuming repeatedly must not build a backlog of reconnections to work through |
|
||||
| **the backoff is not reset** | being roused says the machine changed, not that whatever refused the connection has stopped. A laptop woken on a network with no route would otherwise retry at full speed for as long as somebody keeps opening the lid |
|
||||
| **`down` does not rouse** | the link is already gone, reconnecting will fail, and the backoff exists for exactly that |
|
||||
|
||||
*Checked by holding a link that never returns on its own — which is precisely what a suspended
|
||||
connection is — rousing it, and requiring the attempt to end and another to begin. And by running
|
||||
the dispatcher against every event a network manager emits, requiring it to act on the ones that
|
||||
change where packets go and on no others.*
|
||||
|
||||
@@ -1,9 +1,14 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-08-28
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller internal/builder
|
||||
- mesh-controller cmd/mesh-controller (build, build --behind, push, status)
|
||||
- mesh-controller internal/inventory/builds.go
|
||||
updated: 2026-09-21
|
||||
decisions:
|
||||
- 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
|
||||
- 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
|
||||
- 02-DECISIONS/0010-delivery.md
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
@@ -75,7 +80,7 @@ whenever anybody writes something reusable, which is constantly.
|
||||
|
||||
## Delivery is a comparison, not a pipeline
|
||||
|
||||
The control plane holds two facts and builds the difference:
|
||||
The controller holds two facts and builds the difference:
|
||||
|
||||
```
|
||||
what source exists ─┐
|
||||
@@ -91,7 +96,7 @@ That is the same shape the host uses on a machine, one layer up:
|
||||
|
||||
| | reconciles | against |
|
||||
|---|---|---|
|
||||
| the control plane | artifacts | source |
|
||||
| the controller | artifacts | source |
|
||||
| the host | machine state | declarations |
|
||||
|
||||
**There is no pipeline as a state machine.** No stage list something can be omitted from, and no
|
||||
@@ -175,6 +180,49 @@ Not aspirations — things without which the above does not work:
|
||||
same digest, a cascade would stop at the first module whose output did not move. Without them,
|
||||
one core-library commit redeploys the fleet with no behavioural change.
|
||||
- **How a module publishes its own types**, which differs per language.
|
||||
- **How the control plane upgrades itself.** It declares its own new version and the host applies
|
||||
- **How the controller upgrades itself.** It declares its own new version and the host applies
|
||||
it — but if the new one is broken, the thing that would fix it is the thing that is broken. The
|
||||
host has a launcher for exactly this; the control plane has nothing.
|
||||
host has a launcher for exactly this; the controller has nothing.
|
||||
|
||||
## What "behind" means, and what it used to mean
|
||||
|
||||
*2026-08-31.*
|
||||
|
||||
The risk this record names is losing **did my change go out?** — answerable today by opening a
|
||||
pipeline, and something has to replace it or the comparison is worse to live with whatever its
|
||||
other properties.
|
||||
|
||||
**It was answerable only for the machines that broke.** `push --behind` meant *failed or refused*,
|
||||
so a machine that applied cleanly and whose declaration has since changed was not behind. For every
|
||||
machine that worked, the answer was silence — and silence meant both *your change is running there*
|
||||
and *your change has not been sent*, which is the question unanswered rather than answered.
|
||||
|
||||
**So the mesh records a digest of what it last sent each machine.** A digest rather than the
|
||||
declaration: what a machine should be is recomputable at any moment, and a stored copy would be a
|
||||
second account of it, able to disagree with the first. What cannot be recomputed is what was
|
||||
*actually sent*.
|
||||
|
||||
**Recorded after the send.** A digest kept for something that failed to send would make the machine
|
||||
look current for a declaration it never received — the failure mode this is meant to remove,
|
||||
arrived at from the other side.
|
||||
|
||||
**Three situations, kept apart**, because they read differently to whoever is looking even where
|
||||
the remedy is the same push:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **out of date** | it was sent something, and the mesh would now send something else |
|
||||
| **never told** | nobody has ever asked this machine to be anything |
|
||||
| **not worked out** | the mesh cannot say what it should be — not reported here at all, because saying "waiting" about it would invent a comparison. `plan` is where that is answered |
|
||||
|
||||
**`status` says it and `push --behind` acts on it**, and both because the alternative is a flag that
|
||||
knows something the person reading the status does not.
|
||||
|
||||
**A failure that repeats is said to be stuck**
|
||||
([ADR 0090](../../02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md)). A machine
|
||||
re-applies on its interval and reports each time, so a resource nothing can ever apply arrives as
|
||||
the same failure over and over, at a fresh time each time. The mesh keeps, beside the last report,
|
||||
when the current failure began and how many reports in a row have said it — the same resources by
|
||||
id, whatever the words; three make the machine stuck, and `status` says so beside the failure. The host keeps trying — stuck is what the mesh
|
||||
knows, not what the machine is told. *How it is checked:* an inventory test counts three identical
|
||||
reports, a different one, and a clean apply; the status test asserts the word appears.
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-08-30
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller cmd/mesh-controller/board.go
|
||||
- mesh-controller cmd/mesh-controller/readable.go
|
||||
updated: 2026-08-31
|
||||
decisions:
|
||||
- 02-DECISIONS/0008-a-context-owns-its-store.md
|
||||
- 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
|
||||
@@ -39,14 +41,47 @@ it is enough to freeze it.** A board that reads the provisioning tables directly
|
||||
breaks when provisioning changes its tables, and the change then gets weighed against the board.
|
||||
|
||||
**So a board reads through interfaces and holds nothing.** Everything on the mesh page above is
|
||||
already answerable by asking the control plane — what nodes exist, what each resolves to, what it
|
||||
already answerable by asking the controller — what nodes exist, what each resolves to, what it
|
||||
takes from elsewhere, which module came from which commit. A board that asks those questions is a
|
||||
client. A board that queries `inventory` is a second control plane with a worse contract.
|
||||
client. A board that queries `inventory` is a second controller with a worse contract.
|
||||
|
||||
**It stores nothing of its own.** No cache that can disagree, no table of "what the mesh looked
|
||||
like last time". If a question is slow to answer, the answer belongs in the context that owns it,
|
||||
where everything else asking gets it too.
|
||||
|
||||
## Where it is reachable from, which decides everything else about it
|
||||
|
||||
*Written 2026-08-31. It was assumed throughout and stated nowhere, which is the wrong way round
|
||||
for the most consequential fact about this component.*
|
||||
|
||||
**The board is published on a public name.** Not reachable only over the private network — on the
|
||||
internet, behind the reverse proxy, like any other published workload.
|
||||
|
||||
**So its login is a perimeter, not defence in depth.** A board on the overlay alone would sit
|
||||
inside the boundary that [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) already
|
||||
calls the security boundary, and a login there would guard a room whose door is inside the
|
||||
building. This one faces everybody.
|
||||
|
||||
**Which makes the identity provider the mesh's outermost gate.** The board is a presentation layer
|
||||
over the controller and the controller's networked surfaces can change the mesh
|
||||
([ADR 0035](../../02-DECISIONS/0035-one-implementation-several-surfaces.md)), so **whoever that
|
||||
provider admits can assign modules, from anywhere.** Said flatly because it is easy to arrive at
|
||||
one reasonable step at a time and then be surprised by.
|
||||
|
||||
Three things follow, and none of them are the board's own design:
|
||||
|
||||
- **Who may log in, and how, is a decision about the mesh** rather than about an application. A
|
||||
realm that lets somebody in has let them into the mesh.
|
||||
- **A public name needs a public certificate**, from an authority the world trusts rather than the
|
||||
mesh's own — which is why that exists at all.
|
||||
- **The provider failing is not only "the board is down".** Its configuration going wrong in the
|
||||
other direction — a realm that admits too much — is a mesh-wide exposure with no local symptom.
|
||||
|
||||
**The command line is unaffected and is the reason this is tolerable.** It authenticates through
|
||||
nothing, needs no network, and answers to the machine's own login — so the mesh remains operable
|
||||
by somebody standing at it whatever happens to the gate. *That is the property to protect if the
|
||||
rest of this is ever traded away.*
|
||||
|
||||
## What it is not
|
||||
|
||||
- **Not the way to change things.** Reading is the whole of it to begin with. Every action the
|
||||
@@ -72,3 +107,36 @@ page that led with the third would bury the first:
|
||||
as it was and what is wrong is in what was sent; failed means it is in a state nobody declared and
|
||||
what is wrong is on the machine. They are fixed in different places, so a page that said "error"
|
||||
for both would send half its readers to the wrong one.
|
||||
|
||||
## What was built
|
||||
|
||||
*2026-08-31.*
|
||||
|
||||
**One reading, three ways of saying it.** The questions are asked once, by one function, and
|
||||
answered as a person's `status`, as its JSON, and as this page. Three implementations of *which
|
||||
machine is not doing what it was told* would be three chances to disagree about it — and the
|
||||
disagreement would surface as two people looking at two screens arguing about which machine is
|
||||
broken.
|
||||
|
||||
**It holds nothing and changes nothing.** Every request reads the mesh now. There is no cache to
|
||||
go stale, no table of what the mesh looked like last time, and no button: every action a board
|
||||
could offer already exists as a command, and one that did something no command does would be a
|
||||
second implementation of a decision.
|
||||
|
||||
**It never touches a context's store.** That is the whole constraint above, kept: the board is a
|
||||
client of the same functions the commands use, so provisioning can change its tables without the
|
||||
change being weighed against a page.
|
||||
|
||||
**A board that cannot read the mesh says so.** An empty page says *nothing is wrong* in the one
|
||||
situation where nobody can know that, so the failure is rendered instead — and it says explicitly
|
||||
that it is a statement about the page rather than about the mesh.
|
||||
|
||||
**A machine's own words are shown, and are not markup.** They are the whole reason the page is
|
||||
useful — a board that said only *failed* would send a person to ask the thing they opened the
|
||||
board to avoid asking. They are also the only text on the page that nobody in this repository
|
||||
wrote, which is why the escaping is a test rather than an assumption.
|
||||
|
||||
*Checked by giving a machine a declaration it cannot apply and requiring the page to name that
|
||||
machine, say `failed` rather than `error`, and quote what the host said — then by comparing the
|
||||
page's own JSON against the command's, because two answers to "which machine is broken" would be
|
||||
worse than either alone.*
|
||||
|
||||
@@ -1,11 +1,15 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-control internal/builder
|
||||
- mesh-control internal/catalogue/build.go
|
||||
updated: 2026-08-30
|
||||
- mesh-catalog modules/builder
|
||||
- mesh-controller internal/builder
|
||||
- mesh-controller internal/catalogue/build.go
|
||||
- mesh-controller internal/inventory/secrets.go
|
||||
- mesh-controller cmd/mesh-builder
|
||||
updated: 2026-09-12
|
||||
decisions:
|
||||
- 02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0010-delivery.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
@@ -37,13 +41,25 @@ resolution, and assigning it brings both. So the module count does not return, b
|
||||
that made it return — *a module is expensive, so put several things in one* — is gone. A module
|
||||
here is cheap: a manifest and, usually, nothing else.
|
||||
|
||||
## One file at the root
|
||||
## One file, at the module's own path
|
||||
|
||||
`module.json`, and a convention somebody can look for beats a setting somebody has to find. It
|
||||
says what the module is, what it provides and requires, what it claims, what capabilities it
|
||||
needs, what it puts on a machine — and, if anything must be produced from the source, what to
|
||||
build.
|
||||
|
||||
**A module is a repository and a path within it** ([ADR 0069](../../02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md)).
|
||||
The manifest sits at that path, beside the source it describes, and everything it declares is
|
||||
produced from there. A module whose path is the repository root is the ordinary case of this and
|
||||
not a separate shape.
|
||||
|
||||
*Corrected 2026-09-12. This section previously said the manifest was at the root, full stop, which
|
||||
made a repository hold exactly one module. Nothing that exists is shaped that way: the catalogue
|
||||
holds sixty-seven modules in their own directories, the system being replaced has always built a
|
||||
module from a repository and a path, and the root-only reading left every existing module
|
||||
unbuildable — pointed at the catalogue the builder finds no manifest, pointed at a module's source
|
||||
it finds no manifest either.*
|
||||
|
||||
## The manifest in the repository is not the manifest the mesh holds
|
||||
|
||||
A resource names an artifact:
|
||||
@@ -69,10 +85,10 @@ the worst possible moment.
|
||||
|
||||
## The builder runs on a node
|
||||
|
||||
**Not in the control plane, and this is the same boundary as everywhere else.** Building needs a
|
||||
container runtime and a working tree; what the control plane may send a machine is bounded by the
|
||||
**Not in the controller, and this is the same boundary as everywhere else.** Building needs a
|
||||
container runtime and a working tree; what the controller may send a machine is bounded by the
|
||||
declaration language ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and *run this build*
|
||||
is not in it. The alternative — the control plane holding a container socket — would make it the
|
||||
is not in it. The alternative — the controller holding a container socket — would make it the
|
||||
one component that can do anything on any machine, which is the property the whole design is
|
||||
arranged to avoid.
|
||||
|
||||
@@ -80,7 +96,7 @@ So the builder is a program a machine runs, given work over the broker like anyt
|
||||
its own credential and nothing more.
|
||||
|
||||
**A build is work, not state**, and that is why it does not travel as a declaration. Everything
|
||||
else the control plane sends a node is *what you should be*, reconciled forever. A build happens
|
||||
else the controller sends a node is *what you should be*, reconciled forever. A build happens
|
||||
once and is finished; as a declaration it would either rebuild on every reconcile or carry "and I
|
||||
already did this" — state about an event rather than about a machine.
|
||||
|
||||
@@ -108,6 +124,77 @@ Three properties of the builder that are decisions:
|
||||
is not running, and those want completely different responses — the same rule the host follows
|
||||
about a service that does not exist
|
||||
|
||||
### And it is a module the mesh assigns
|
||||
|
||||
*2026-08-31. Written after `builder issue --node`, which is the part that makes the sentence
|
||||
"holding its own credential" true rather than aspirational.*
|
||||
|
||||
A build machine is a machine that runs the builder, and there is exactly one honest way to say
|
||||
which machines those are: **assign it**. So the builder is a module like any other — an image, a
|
||||
container, a working directory, and a claim so a machine does not end up running two.
|
||||
|
||||
The one thing that could not be a module in the ordinary way is the credential. It is not
|
||||
generated, because the broker has to have been told about it, and it is not written in a manifest,
|
||||
because a manifest is public and the same file goes to every machine that ever runs it. So the
|
||||
mesh **creates the account, seals the URL to the machine that will use it, and discards the
|
||||
plaintext** — the "given, not generated" case above, and its first user.
|
||||
|
||||
Nothing is printed. A credential shown on a terminal is a credential in a scrollback buffer, and
|
||||
the copy that matters would then exist in two places, one of which nobody is guarding.
|
||||
|
||||
**What this replaces:** a builder started by hand with whatever credential was to hand, which in
|
||||
practice meant the broker's administrative account. *A program documented as holding its own
|
||||
credential and given somebody else's is worse than one with no story at all* — the documentation
|
||||
is what stops anybody checking.
|
||||
|
||||
*Checked in the lab by assigning it and then asking the mesh to build a module: the credential
|
||||
file arrives readable only by that machine, names the scoped account rather than the broker's own,
|
||||
and the build completes — which is the only proof the credential authenticates, because a
|
||||
container that is up holding a credential it cannot use looks identical from outside.*
|
||||
|
||||
**And it is told what to check the broker against**, not only who to connect as. A mesh's broker
|
||||
presents a certificate of the mesh's own, which is in no public trust store, so a URL alone reaches
|
||||
only a broker somebody else vouches for — which is no mesh broker at all. The credential carries
|
||||
the fingerprint beside the URL: **the same two facts a node's token carries**
|
||||
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)), for the same reason, arriving by
|
||||
a path other than the thing being trusted.
|
||||
|
||||
**A builder that is a module cannot see the machine's filesystem.** It runs in a container, so a
|
||||
local path exists for the machine and not for it. That is not a limitation to work around — it is
|
||||
the arrangement working: a build machine shares the runtime it was given rather than the machine
|
||||
it sits on. **A module is cloned from the forge over a URL**, and "build this directory" is a
|
||||
convenience for a builder somebody started by hand.
|
||||
|
||||
### And the loop is closed
|
||||
|
||||
*2026-08-31.* [ADR 0010](../../02-DECISIONS/0010-delivery.md) replaced a pipeline with a comparison
|
||||
and named the risk: **losing the question "did my change go out?"**. The mesh could already answer
|
||||
which modules were behind their source — and then a person read that list and retyped each
|
||||
repository, which is a person being the loop, and the loop is the thing the pipeline was doing
|
||||
before it was taken away.
|
||||
|
||||
`build --behind` is the other half, and it is the mirror of `push --behind`: the mesh knows what is
|
||||
stale, so it builds it. The two forms are deliberately not combined — naming a repository and
|
||||
asking which need building are different requests, and guessing which was meant would sometimes
|
||||
build something nobody named.
|
||||
|
||||
**One failing does not stop the others**, for the same reason one broken module no longer blocks a
|
||||
machine's whole declaration: a mesh where one bad repository holds back nine good ones is a mesh
|
||||
where nobody dares add the tenth.
|
||||
|
||||
**Each is built from its own recorded ref**, not from the commit the mesh happened to notice.
|
||||
Pinning to that would quietly turn a tracked branch into a pin — a change of meaning nobody asked
|
||||
for, arrived at by an implementation detail.
|
||||
|
||||
**Building is not delivering, and the two stay separate.** A machine keeps running what it has
|
||||
until it is told otherwise; the mesh changing its mind is not a machine acting on it, and
|
||||
collapsing the two is how a mesh comes to report success for something that has not happened.
|
||||
|
||||
*Checked end to end: a commit, a build, a catalogue entry, and a machine that ends up running what
|
||||
the source says — with both halves that make the answer trustworthy. It is still running the old
|
||||
one until it is pushed, and it stops being reported as behind once it has caught up, because a
|
||||
status that says "behind" for ever is one nobody reads.*
|
||||
|
||||
## What is kept
|
||||
|
||||
**Every result, including the failures.** A failed build that leaves no trace is indistinguishable
|
||||
@@ -116,7 +203,7 @@ at something. A build that failed before it knew what it was building keeps the
|
||||
is what a person goes and looks at.
|
||||
|
||||
Recording is idempotent on the correlation, because a result arrives twice — once as the answer to
|
||||
whoever asked and once on the exchange, where the control plane is also listening. Two rows would
|
||||
whoever asked and once on the exchange, where the controller is also listening. Two rows would
|
||||
show one build as two, and which is real is not answerable afterwards.
|
||||
|
||||
That is what a builds view reads, and until it existed there was nothing to read: a result was
|
||||
@@ -138,9 +225,16 @@ answered to the asker and kept nowhere.
|
||||
| kind | is |
|
||||
|---|---|
|
||||
| **image** | built from a Dockerfile in this repository |
|
||||
| **bundle** | this module's own code, compiled by the toolchain its language implies, then packed |
|
||||
| **archive** | a directory in this repository, packed |
|
||||
| **upstream** | an image somebody else built, mirrored into the mesh's own registry |
|
||||
|
||||
**The second was added later and is why most modules now need no Dockerfile.** An archive packs a
|
||||
directory as it stands, so shipping compiled output meant compiling somewhere first — and that
|
||||
meant every module repeating a recipe that is easy to get wrong in ways that fail elsewhere. The
|
||||
whole surface a module has, and a module that exercises all of it, are in
|
||||
[`18-building-a-module`](18-building-a-module.md).
|
||||
|
||||
**The third exists because a module usually runs software it did not write.** A database module
|
||||
ships configuration and a provisioner and does not build a database. Naming the upstream reference
|
||||
directly would need every machine to reach a public registry, and would pin to a tag its owner can
|
||||
@@ -157,7 +251,8 @@ A database has a superuser password, a broker an administrator, a registry an ac
|
||||
them is *for* anybody** — they are not the credential a consumer is given, and the mechanism that
|
||||
hands those out has a consumer in the middle of it.
|
||||
|
||||
So a module says what it needs and where to put it, and the mesh generates one **per node**,
|
||||
So a module says what it needs and where to put it — `own-secrets`, keyed by a name of the
|
||||
module's choosing — and the mesh generates one **per node**,
|
||||
seals it to that machine and reads it no more than it reads any other secret. Per node
|
||||
deliberately: a module running on three machines has three passwords, where one in the manifest
|
||||
would put the same secret on every machine that ever runs it, in a file anybody can read, for ever.
|
||||
@@ -167,6 +262,55 @@ Remade when the machine's sealing key changes. **Declared and not made is refuse
|
||||
module whose own credential is silently absent starts, fails to authenticate, and the reason is
|
||||
three layers from the machine reporting it.
|
||||
|
||||
### It is named for whose it is, not how secret it is
|
||||
|
||||
*2026-08-31, from an audit asking whether the manifest format was becoming hard to hold in the
|
||||
head.*
|
||||
|
||||
The field was called `needs`, beside `secrets` — which is where a **provision's** credential lands
|
||||
on a consumer. Both were name-to-path, both held something secret, and the names distinguished
|
||||
them not at all. **Reaching for the wrong one parsed cleanly and failed somewhere else entirely**,
|
||||
which is the shape of fault this whole design exists to prevent, sitting in the manifest format.
|
||||
|
||||
The axis that separates them is not how secret they are — both are — but **whose**:
|
||||
|
||||
| | keyed by | belongs to |
|
||||
|---|---|---|
|
||||
| `secrets` | the provision it is for | the relationship with another machine |
|
||||
| `own-secrets` | a name the module chose | this module, and nobody else |
|
||||
|
||||
**A manifest using the old name is told the new one** rather than refused with "unknown field".
|
||||
Whoever wrote it knew what they meant, and the mesh knows what it is called now.
|
||||
|
||||
### Some of them the mesh cannot make
|
||||
|
||||
*2026-08-31, from making the builder a module — the first thing to hold one.*
|
||||
|
||||
A generated secret is the mesh's, and remaking it costs nothing: **nothing else ever knew the old
|
||||
one.** That is the assumption the paragraph above rests on, and it is not true of every secret a
|
||||
module needs.
|
||||
|
||||
A broker account's password exists because **the broker was told about it**. A licence key exists
|
||||
because somebody bought it. The mesh's job with these is to carry the value to the machine that
|
||||
will use it and then be unable to read it — the same sealing, from the other direction: **given,
|
||||
not generated.**
|
||||
|
||||
Treating the two alike is wrong in exactly one place, and it is the place nobody looks. When a
|
||||
machine rejoins it has a new sealing key, and everything sealed to the old one is remade. Remaking
|
||||
a *given* secret puts thirty-two random bytes where a working credential was, and every visible
|
||||
signal says it worked: the mesh sealed a secret, the machine applied it, the file is there with
|
||||
the right permissions. What fails is a program authenticating to something else, hours later,
|
||||
with an error that names neither the mesh nor the secret.
|
||||
|
||||
So **where the value came from is recorded, and a given secret is never regenerated.** A rejoined
|
||||
machine asking for one is refused, naming the remedy — issue it again — because the remedy is a
|
||||
command somebody runs and no amount of pushing will produce a password the broker has never heard
|
||||
of.
|
||||
|
||||
*Checked by taking a given secret, changing the machine's sealing key, and asserting the mesh
|
||||
refuses rather than answers; and by asserting that two ordinary pushes hand back the same value,
|
||||
without which the refusal would be a secret that never survives at all.*
|
||||
|
||||
## What one assignment gets you
|
||||
|
||||
A database module, written to see whether it could be:
|
||||
@@ -203,3 +347,82 @@ access, or are not build output. None of that describes a digest-pinned archive,
|
||||
second service for one kind of immutable blob is two things to run, two to back up, and two ways
|
||||
for an artifact to be missing. **Overturnable without touching anything else**: a manifest carries
|
||||
a URL and a digest, and neither says what served it.
|
||||
|
||||
### And the mesh runs it
|
||||
|
||||
*2026-08-31.* Which registry is a **provision**, mesh-scoped: a build machine requires
|
||||
`artifact-store` and is told where it is, the same way an application is told where its database
|
||||
is. Nothing is configured with an address.
|
||||
|
||||
This closes the last thing the mesh depended on and did not run. The registry a bootstrap pulls
|
||||
from belongs to whoever raised the machine; from the moment the mesh has one of its own, an
|
||||
artifact's home is somewhere the mesh can move, replace and back up.
|
||||
|
||||
**The chicken and egg is the bootstrap's, resolved the same way.** A registry module is an
|
||||
`upstream` artifact — mirrored from a registry that already exists into the one being started. The
|
||||
first copy comes from outside, exactly once, and every copy after it is the mesh's.
|
||||
|
||||
*Checked in the lab by assigning it and then asking for `/v2/` — on the machine, and from a second
|
||||
machine across the private network, because a mesh-scoped provision that only answers locally is
|
||||
not one. A container that is running is not a registry that replies, and this project has paid for
|
||||
that distinction once already.*
|
||||
|
||||
## The four the loop cannot build
|
||||
|
||||
*2026-09-12. Written down because it keeps being rediscovered as if it were new, once per
|
||||
component. It is one rule; the instances are listed below. Revised the same day, when the catalogue
|
||||
gained an owner and turned out to be a fourth — which is the argument for stating the rule rather
|
||||
than the list.*
|
||||
|
||||
**Anything the build loop needs in order to run cannot be delivered by the build loop.** It arrives
|
||||
from outside exactly once, and from then on it is an ordinary module, upgraded like one. What
|
||||
"from outside" means is *not* the same for all three, and saying so matters, because two of them
|
||||
have a route and one does not:
|
||||
|
||||
| What | Why it cannot come through the loop | How it arrives |
|
||||
|---|---|---|
|
||||
| The controller | It is what installs modules. Nothing can install it before it runs. | **Carried inside the installer** and published once there is a registry ([ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md)) |
|
||||
| The registry | It *is* where artifacts are delivered from. A store cannot be delivered through itself. | **Pulled from the public internet** — its image is an ordinary public one, never built ([`04-ISSUES/029`](../../04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md)) |
|
||||
| The builder | It is what builds. Nothing builds it before it runs. | Built at genesis by the init builder ([ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md)) |
|
||||
| The catalogue | It owns the module graph, and nothing can be resolved or installed without it. | Built at genesis by the init builder ([ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md)) |
|
||||
|
||||
**How they arrive is settled and not yet built.** The installer carries an init builder, which
|
||||
clones the source and builds the controller, the catalogue and the builder before a mesh exists
|
||||
to install anything. Two things about that are open and named in
|
||||
[ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md): where the init builder
|
||||
clones from, given the forge normally runs on the mesh it would be rebuilding, and what it
|
||||
publishes into, given the registry is installed later in the order today.
|
||||
|
||||
**There is a fourth, and the list was closed too early.** The catalogue owns the module graph
|
||||
([ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md)), so it cannot be
|
||||
installed by something that needs it in order to install anything — the same shape as the other
|
||||
three. This section previously said the list was closed at three, which was written before the
|
||||
catalogue had an owner and is corrected here rather than left to be reasoned from.
|
||||
|
||||
**And the answer for all four is now one mechanism, not four special cases.** Genesis carries an
|
||||
*init builder* and builds the core modules on the machine — controller, catalogue and builder —
|
||||
rather than carrying a finished image of any of them. So the question is no longer "how does this
|
||||
one get here first?" asked once per component; it is answered once, by the thing that is carried
|
||||
being a builder rather than a result.
|
||||
|
||||
Everything outside those four is either upstream — a third-party image pulled by digest — or built
|
||||
by the builder from a repository and a path, and published to the registry.
|
||||
|
||||
**One of those built things has an ordering constraint worth naming, because it looks like a fifth
|
||||
member of the list and is not.** The SDK the toolchain compiles against is built by the ordinary
|
||||
builder and published like anything else — but it cannot be compiled *in the mesh toolchain*, since
|
||||
that toolchain is built from it, and it is published to the *package* registry rather than the
|
||||
artifact store. So it is compiled on a public base image and published before the toolchain that
|
||||
consumes it ([ADR 0076](../../02-DECISIONS/0076-the-sdk-is-a-published-package.md)). It is not
|
||||
carried and it is not machinery; it is a dependency with a sequence, which is why it belongs here as
|
||||
a footnote to the rule rather than a row in the table.
|
||||
|
||||
**A carried artifact is not a differently-pinned artifact.** Once published it is named by a digest
|
||||
the mesh's registry assigned, exactly like everything the builder produces. A reader cannot tell
|
||||
from a running mesh which of its images were carried, and that is the point: carrying is how the
|
||||
first copy arrives, not what it permanently is.
|
||||
|
||||
*Checked by the thing already checked at genesis: after installing, the running controller is
|
||||
pinned to a digest the mesh's own registry assigned, and not to the id of the image the installer
|
||||
carried. The same check applies to the builder and to the registry, and it is the same check —
|
||||
an image id where a registry digest belongs means the pivot did not finish.*
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller internal/inventory/secrets.go
|
||||
- mesh-controller cmd/mesh-controller/rotate.go
|
||||
- mesh-controller examples/postgres-provisioner
|
||||
updated: 2026-09-21
|
||||
decisions:
|
||||
- 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0085-a-secret-is-a-provision.md
|
||||
- 02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md
|
||||
---
|
||||
|
||||
# 13 — Credentials, and moving them
|
||||
|
||||
*Written 2026-08-31, when rotation was built. The delivery half was already proven; this is the
|
||||
half [ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md) records as
|
||||
unowned, and it was **measurably false** in the system being replaced.*
|
||||
|
||||
## What went wrong before, precisely
|
||||
|
||||
`provision_ensure`, documented as *NEVER rotates an existing secret*, minted a new password on
|
||||
every adoption and updated **only the provider's row**. Consumers on three nodes held dead
|
||||
credentials for two days. Two rows for one provision were written 216 ms apart, so at most one
|
||||
could have matched the live role. The mesh reported success throughout.
|
||||
|
||||
Three separate faults, and it is worth naming them apart because they have different fixes:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **one credential, many holders** | rotating it was necessarily a fan-out, and nothing enumerated who held it |
|
||||
| **the record moved and the consumers did not** | the change and the delivery were different acts, and only the first happened |
|
||||
| **nothing said so** | the mesh could not tell a rotated credential from a working one, so nobody looked |
|
||||
|
||||
## What replaces it
|
||||
|
||||
**Every pair has its own credential.** A provision between one consumer and one provider is one
|
||||
password, made once and kept. So rotating a credential touches one role and leaves every other
|
||||
consumer alone — and *who holds this* is a query rather than an assumption. That alone removes the
|
||||
first fault: there is no shared secret to fan out.
|
||||
|
||||
**A consumer is a module on a machine, not a machine.** This was written as though a pair were two
|
||||
machines, and built that way, and it was wrong in a way that only shows on a real node
|
||||
([`022`](../../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md)):
|
||||
a machine running several services against one database server had one credential between them.
|
||||
The provider refused to plan at all, and the consuming node did not refuse — it gave the first
|
||||
module a credential and the rest nothing.
|
||||
|
||||
Two modules on one node are as separate as two on different nodes. They are different containers,
|
||||
with different data, and one login opening both is the thing this page exists to prevent. It is
|
||||
also what makes withdrawal possible: one role per machine cannot express *this module no longer
|
||||
has a login and the others still do*.
|
||||
|
||||
**The change and the delivery are one command.** `rotate` discards the credential and sends both
|
||||
ends, and it does the sending itself. Leaving that to whoever remembers is the second fault
|
||||
exactly, and the interval in which it goes wrong is unbounded — two days, in the recorded case.
|
||||
|
||||
**It is all-or-nothing.** If any affected machine cannot be resolved, nothing is sent and the old
|
||||
credential keeps working. A mesh that has not rotated is far better than one that has
|
||||
half-rotated, and the difference is that the first is obvious.
|
||||
|
||||
**The window is stated rather than hidden.** A role's password changes on the provider and the file
|
||||
changes on the consumer, and those cannot be simultaneous. So there is an interval in which a
|
||||
consumer cannot authenticate, and the honest thing is to make it as short as the broker allows and
|
||||
to say it exists. `status` names who is still behind.
|
||||
|
||||
## The provider makes it true, and the mesh cannot
|
||||
|
||||
The mesh generated the password, sealed it to the machine that must accept it, and **discarded the
|
||||
plaintext** — so it cannot tell a database to start accepting it. Something on that machine reads
|
||||
what the host wrote and makes it true.
|
||||
|
||||
That something is part of the module, not part of the controller. **The controller decides
|
||||
and never touches a machine; a provisioner runs on the machine and touches it.** Two files, because
|
||||
the mesh could not compose a document containing a value it does not have:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| a manifest | every consumer, what it asked for, and where its credential is |
|
||||
| one file per consumer | that consumer's password, alone |
|
||||
|
||||
**Both, or neither works.** A provisioner given the passwords and not the manifest finds a
|
||||
directory of unexplained secrets and reports that nothing has been granted — which is true, and
|
||||
reads exactly like a credential that was never delivered. That has now happened once, here.
|
||||
|
||||
**The password a provisioner uses is itself a file the mesh wrote.** Passing it through the
|
||||
environment needs a person in the middle of the one path that exists so there is not one, and puts
|
||||
a superuser password where `docker inspect` prints it.
|
||||
|
||||
## A secret reaches a process as a file
|
||||
|
||||
The care above ends at the container's door if the module then hands the value to the runtime as
|
||||
an environment variable: `docker inspect` prints it, and the process's `/proc` entry holds it for
|
||||
anything on the machine that can talk to the runtime. So a secret reaches a process **as a file**
|
||||
([ADR 0086](../../02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md)): the module mounts
|
||||
the file the host wrote and points the program at it, and the mesh's own programs take a `_FILE`
|
||||
twin for every variable that carries a credential. Software that reads only its environment is
|
||||
not forbidden; it is **declared**, on the container, with a reason, so the manifest says which
|
||||
secrets are exposed that way. **Checked** at composition: the catalogue engine refuses a
|
||||
`${secret:…}` in a container's `env` outright, and a secret-carrying file named in `env-file`
|
||||
unless the container carries the declaration.
|
||||
|
||||
## How it is checked
|
||||
|
||||
Not by comparing two files. **Two ends holding a matching string proves they agree, not that either
|
||||
is right** — the recorded fault produced two ends that agreed with each other and not with the
|
||||
database.
|
||||
|
||||
So the check is three logins against a real PostgreSQL, from the consumer's own machine, over the
|
||||
private network:
|
||||
|
||||
1. the delivered credential authenticates
|
||||
2. after rotation, the new one authenticates
|
||||
3. **the one that was rotated away does not**
|
||||
|
||||
The third is what makes it a rotation rather than an addition. Without it the check passes against
|
||||
a provider that added a password and removed nothing.
|
||||
|
||||
**Not over loopback.** `pg_hba` trusts anything there, so every password looks correct — a
|
||||
deliberately wrong one returned a row for an afternoon before that was noticed.
|
||||
|
||||
|
||||
## The secret that is not a pair
|
||||
|
||||
Everything on this page is about the credential *between a consumer and a provider* — the login
|
||||
one module uses against another. A mesh also holds secrets that are not that: a value a single
|
||||
module needs for **its own** use, and a value only an operator can supply. Those had no owner, and
|
||||
so no rotation — the gap this page's machinery could not reach because there was no pair to rotate.
|
||||
|
||||
[ADR 0085](../../02-DECISIONS/0085-a-secret-is-a-provision.md) gives them one. Such a secret is
|
||||
provided by a **vault** module ([24](24-the-secrets-vault.md)): a module requires a `secret`
|
||||
provision, and the credential of that consumer↔vault pair is an ordinary pair credential — so it
|
||||
rotates, and is queried for who holds it, through exactly the machinery described above, unchanged.
|
||||
The rotation a module's own secret lacks is not a second mechanism; it is this one, pointed at a
|
||||
secret the vault provides. What this page proves for a database password holds, by construction,
|
||||
for a secret from the vault.
|
||||
@@ -0,0 +1,186 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller internal/licences
|
||||
- mesh-controller cmd/mesh-controller/licence.go
|
||||
updated: 2026-09-05
|
||||
decisions:
|
||||
- 02-DECISIONS/0024-model-access-is-a-provision.md
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0054-model-usage-is-recorded-at-two-grains.md
|
||||
- 02-DECISIONS/0055-model-access-is-answered-by-a-licence-or-a-node.md
|
||||
---
|
||||
|
||||
# 14 — Model access
|
||||
|
||||
*[ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md) decided it and listed four
|
||||
things the mesh did not have. Written 2026-08-31, when two of them were built. **The other two are
|
||||
still gaps and are still written as gaps** — the record's own warning is that pretending otherwise
|
||||
is how a plan becomes a surprise.*
|
||||
|
||||
## What was built
|
||||
|
||||
**A licence is a record, and the first provision no machine answers.** Everything else the mesh
|
||||
brokers is answered by something running on a node. A hosted model is on nobody's machine and is
|
||||
reached over the public internet, so the rule that refuses two ends sharing no private network —
|
||||
correct everywhere else — must not apply to it. A machine on no private network at all can hold a
|
||||
licence, and that is not a special case to remember: it falls out of the answer not being a
|
||||
machine.
|
||||
|
||||
**The name is the operator's.** *The personal account*, *the organisation's account*. Those are
|
||||
names a person uses, and the mesh uses them too, because the whole point is saying **which one** a
|
||||
consumer uses — and an anonymous credential hanging off a provider cannot be said. Many to many,
|
||||
so deliberately **not a claim**: two machines sharing an account is the ordinary case rather than
|
||||
a collision.
|
||||
|
||||
**One provision name for all of them.** A module requires `model-access`, never `anthropic`. A
|
||||
module that named a provider could not be moved onto a model the mesh runs itself without editing
|
||||
it — and moving it is the point.
|
||||
|
||||
**A model in a machine's own set answers it locally**, and no record is consulted. That is what
|
||||
makes *the mesh's own model* an ordinary answer rather than a parallel arrangement.
|
||||
|
||||
### Accept: taking a value the mesh did not make
|
||||
|
||||
Every other credential here the mesh generated, sealed to both ends and discarded. An API key
|
||||
arrives from a person, and the missing verb was *accept*: **take a value, seal it to each holder,
|
||||
discard the plaintext.** A mesh that kept operator-supplied keys readably is the arrangement this
|
||||
project measured and rejected.
|
||||
|
||||
**It seals to the holders that exist at that moment**, and this has a consequence that must be
|
||||
said out loud rather than discovered:
|
||||
|
||||
> A consumer put on a licence *after* the key was supplied has no key, and **the mesh cannot make
|
||||
> one** — it discarded the only copy.
|
||||
|
||||
So that state is reported at every point a person could meet it: when the consumer is put on the
|
||||
licence, in `licence list`, and — decisively — **the declaration is refused** rather than written
|
||||
without the file. A machine that resolves cleanly and receives nothing fails later, somewhere that
|
||||
names neither the licence nor the mesh.
|
||||
|
||||
**A key is read from a file or standard input, never an argument.** A key on a command line is a
|
||||
key in shell history and in every process listing taken while it ran. It is never echoed back:
|
||||
what is stored is unreadable by whoever holds it, the controller included, and printing it
|
||||
would put the one copy that matters on a terminal.
|
||||
|
||||
## Refusing is felt, and that is the design working
|
||||
|
||||
ADR 0024 predicted it: *a mesh holding three ways to reach a model refuses every consumer that has
|
||||
not said which — which is correct and is a great deal of saying-which the first time.*
|
||||
|
||||
It is correct, and correct is not the same as usable. So the refusal names **the candidates and
|
||||
the exact command**. The difference between a mesh that refuses helpfully and one that merely
|
||||
refuses is whether anybody can act on it without going and reading something else.
|
||||
|
||||
## Still gaps
|
||||
|
||||
Unchanged from [ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md), and deliberately
|
||||
not half-built:
|
||||
|
||||
**A consumer that is not a machine.** *This worker uses that licence* is a binding to an agent, not
|
||||
to a node. What is delivered still lands on a machine; what is **chosen** is chosen per agent, and
|
||||
the provisions model has no consumer identity other than a node. What exists today is per module
|
||||
per machine, which is a step toward it and is not it.
|
||||
|
||||
*2026-08-31: this gap now has named consumers rather than hypothetical ones.*
|
||||
[ADR 0026](../../02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md) puts two sessions on the
|
||||
controller node — the node's own and the mesh's — each bound in its own right. See
|
||||
[`15-the-agent-session.md`](15-the-agent-session.md).
|
||||
|
||||
**And for sessions the gap is already closed, which was not obvious.** A binding is per module per
|
||||
machine, and this document called that *a step toward it and not it* — reasoning that a machine
|
||||
cannot name an agent. It cannot; but the two sessions are **two modules**, because they are the
|
||||
same mechanism started in different context roots and a context root is what a module delivers.
|
||||
So `(node, module)` tells them apart, and asking for a licence per session needed no new consumer
|
||||
identity. Checked rather than argued: two sessions on one machine hold different licences, each is
|
||||
given its own key, and releasing one leaves the other.
|
||||
|
||||
**What is still open is the rest of the gap, and it is the harder half.** A *worker* is not one
|
||||
per machine — many can run on one, from one module — so `(node, module)` cannot name them apart
|
||||
and this reasoning does not extend to them. That belongs with
|
||||
[ADR 0003](../../02-DECISIONS/0003-agents-are-persistent-employees.md), which is unbuilt, and it
|
||||
is the reason this section stays open rather than being struck out.
|
||||
|
||||
**Switching is a reaction, not a declaration.** A licence that hits its limit and must be swapped is
|
||||
a response to something observed. Expressing it as a declaration would make the declaration mean
|
||||
*whatever is working right now*, which is not a thing anybody declared. It belongs with
|
||||
observability, changing a binding — and the binding is then declared as usual. **Saying this
|
||||
plainly is what stops the declaration language growing a conditional**, and nothing built here
|
||||
grew one.
|
||||
|
||||
## The vendor-agnostic generalisation
|
||||
|
||||
*[ADR 0050](../../02-DECISIONS/0050-model-access-is-vendor-agnostic.md) is accepted; this section
|
||||
describes what it decides. The section above stands as what is built today.*
|
||||
|
||||
What runs is one vendor — the mesh's Anthropic feature. A read-only trace asked whether the
|
||||
`model-access` provision is Anthropic-shaped or genuinely general, and found that the general layer
|
||||
already exists: a licence is a record with a `vendor` and a non-secret `serves`, a holder's
|
||||
credential is sealed per holder, `accept` takes an operator-supplied value and discards the
|
||||
plaintext, and a locally-run model answers at node scope with no licence. None of that names
|
||||
Anthropic. What is Anthropic's is a thin band: the credential is a subscription OAuth grant — an
|
||||
hourly access token and a refresh token — and that shape alone drags central rotation, a
|
||||
refresh-token-stripping delivery, an identity guard and a `utilization%` usage reading behind it.
|
||||
|
||||
**A vendor is an adapter.** The vendor-specific lifecycle moves into a per-vendor adapter selected by
|
||||
the licence's `vendor` field — the same shape as a registrar-scoped `public-dns` provider behind the
|
||||
neutral `public-dns` interface ([ADR 0044](../../02-DECISIONS/0044-a-public-name-is-provisioned-like-any-capability.md)).
|
||||
A consumer still requires `model-access` and never names a vendor; the interface is drawn at the
|
||||
consumer's real coupling — *reach a model* — per [ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md).
|
||||
The field is `vendor` rather than `provider`, because "provider" already means *which node answers a
|
||||
brokered provision* and the two facts must not share a word.
|
||||
|
||||
The adapter declares a `shape` — `static-key` or `refreshable-grant` — and, optionally, the verbs a
|
||||
vendor happens to need: `accept` a supplied key, `refresh` a grant, read an `identity` off the
|
||||
credential to catch a mis-binding, report `usage`, and `deliver` the value. **A static-key vendor
|
||||
implements almost nothing** — a supplied key, sealed to its holders, delivered. The abstraction is
|
||||
built so the common vendor is small and the rare one carries its own weight.
|
||||
|
||||
```
|
||||
requires: model-access (the consumer, vendor-blind)
|
||||
│
|
||||
┌──────┴───────┐
|
||||
│ a licence │ vendor: … serves: base URL, model
|
||||
└──────┬───────┘
|
||||
selected by │ vendor
|
||||
┌─────────────────┼──────────────────────────┐
|
||||
▼ ▼ ▼
|
||||
anthropic-api-key anthropic (another vendor)
|
||||
shape: static-key shape: refreshable-grant
|
||||
accept, deliver accept, refresh, identity,
|
||||
usage, deliver
|
||||
```
|
||||
|
||||
**The crux is one relaxation, stated plainly.** A refreshable credential cannot be both sealed so the
|
||||
mesh cannot read it *and* rotated centrally — rotation needs a readable refresh token, and the working
|
||||
central rotation is the half [ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md) keeps
|
||||
on purpose. So for `refreshable-grant` vendors only, the **manager node holds the refresh token
|
||||
encrypted at rest** — a bounded, declared exception. Access tokens stay sealed per holder, and the
|
||||
refresh token is stripped on delivery, so *a node never holds a refresh token* remains true for every
|
||||
node but the one manager. **Static-key vendors keep the full guarantee**: there is nothing to rotate,
|
||||
so `accept` discards the plaintext and the carve-out never fires — and static-key is the majority. The
|
||||
exception is written down because a relaxed guarantee that is not stated is indistinguishable from a
|
||||
broken one, and it is narrow on three axes at once: refreshable-grant only, the refresh token only,
|
||||
the manager node only.
|
||||
|
||||
Usage is normalised to `(licence, consumer, period, metric, value)` with the raw response kept beside
|
||||
it; the metric is vendor-defined, so no false common unit is forced. Anthropic becomes the first
|
||||
`refreshable-grant` adapter, and `anthropic-api-key` — the same vendor's plain keys — is the
|
||||
`static-key` case that proves the abstraction is more than one vendor in disguise.
|
||||
|
||||
## How it is checked
|
||||
|
||||
In the lab, on real machines, in the order a person would meet it: a consumer is refused with both
|
||||
candidates named; put on one and still refused because no key exists; the key is given on standard
|
||||
input and not echoed; the public half arrives saying it came from a record rather than a machine;
|
||||
the key arrives readable only by that machine — and it is **nowhere in the controller's own
|
||||
database**, nor in anything that crossed the broker.
|
||||
|
||||
For the generalisation ([ADR 0050](../../02-DECISIONS/0050-model-access-is-vendor-agnostic.md)): a
|
||||
second vendor — `anthropic-api-key`, static-key — is bound to a consumer and exercises the whole path
|
||||
with the carve-out switched off, its key sealed per node and absent from the controller's database.
|
||||
For a `refreshable-grant` licence, the refresh token is asserted to exist (encrypted) **only on the
|
||||
manager node**, to be **absent from every holder's delivery**, and the delivered credential to be
|
||||
access-token-only; a `static-key` licence stores no refresh token anywhere. A scenario with two
|
||||
vendors confirms each licence's lifecycle runs its own adapter, selected by `vendor`.
|
||||
@@ -0,0 +1,226 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: [mesh-controller, mesh-host]
|
||||
updated: 2026-08-31
|
||||
decisions:
|
||||
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
- 02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md
|
||||
- 02-DECISIONS/0024-model-access-is-a-provision.md
|
||||
- 02-DECISIONS/0025-the-design-record-is-read-not-copied.md
|
||||
---
|
||||
|
||||
# The agent session
|
||||
|
||||
**One mechanism, started twice.** A node's session and the mesh's session are the same thing
|
||||
pointed at different context. This document describes the mechanism; where the two differ it says
|
||||
so, and the differences are few enough to list here:
|
||||
|
||||
| | node session | mesh session |
|
||||
|---|---|---|
|
||||
| **address** | the node's name | the mesh |
|
||||
| **context root** | the node's | the mesh's |
|
||||
| **engram** | that node's | the mesh's |
|
||||
| **licence** | bound in its own right | bound in its own right |
|
||||
| **runs on** | that node | the node holding the controller |
|
||||
| **how many** | one per node | one |
|
||||
|
||||
Everything below applies to both unless it says otherwise.
|
||||
|
||||
## What a session is, restated for what is being built
|
||||
|
||||
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) settled the behaviour: permanent,
|
||||
remembering across callers, its own tools, reachable over the broker, and — switched off — still
|
||||
answering *I am switched off* rather than falling silent.
|
||||
|
||||
**Nothing said how one is set up**, and that gap is what this document closes. It was invisible
|
||||
until [ADR 0026](../../02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md) needed to describe
|
||||
a second instance, because *the same as that one, elsewhere* means nothing until the first has
|
||||
been written down.
|
||||
|
||||
## The context root is the whole of the difference
|
||||
|
||||
**A session is defined by the directory it starts in.** That directory holds the engram, the
|
||||
session's tools, and whatever standing instruction it works under. Two sessions differing only in
|
||||
their root are two different agents, and nothing else has to differ to make them so.
|
||||
|
||||
This is deliberately a **small** definition. The alternative — a session type, with the mesh
|
||||
session as a distinct kind — would mean two implementations of one mechanism, and
|
||||
[ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md) records what that
|
||||
costs: two things nearly the same, built twice, until neither word means one thing.
|
||||
|
||||
**The root is delivered as declared state, not carried by the session.** It is files on a machine,
|
||||
which is precisely what the host applies
|
||||
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). A session's context therefore changes the
|
||||
way anything else changes — the mesh declares it, the host writes it — and there is no second
|
||||
mechanism for shipping an engram.
|
||||
|
||||
**Changing the engram is changing a file.** So it is versioned, reviewable, and rolled back like
|
||||
any other declared state; and a node whose engram was changed reports having applied it, the same
|
||||
as it reports anything else.
|
||||
|
||||
## Where each one runs
|
||||
|
||||
**A node's session runs on that node**, and cannot be moved. Moved, one machine is answering as
|
||||
another (ADR 0004).
|
||||
|
||||
**The mesh's session runs on the node holding the controller.** The reasoning is in ADR 0026
|
||||
and is worth carrying here because it is easy to get backwards: this is not *the important agent
|
||||
goes on the important machine*. It is that the controller node is already the one place
|
||||
excepted from *compromise of a node is compromise of that node*, and an agent able to reach
|
||||
everything, placed anywhere else, would create a second such place.
|
||||
|
||||
## Messages
|
||||
|
||||
**A session is reached over the broker**
|
||||
([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)). There is no second
|
||||
transport, nothing is dialled at a session, and the mesh session is not a hop: node-to-node
|
||||
messages continue to travel directly, and nothing is routed through it.
|
||||
|
||||
**A message becomes a prompt; a reply travels back the same way.** Whoever asked — a person at a
|
||||
surface, another session, a worker — is a caller, and the session remembers what each of them
|
||||
asked, together, over time.
|
||||
|
||||
**Being asked something it does not have, a session may ask another.** How it does so is its own
|
||||
business and follows from its engram rather than from a message format: it may say who wants to
|
||||
know, or simply ask. A person relaying a question makes the same choice.
|
||||
|
||||
**A switched-off session answers.** The queue is still read and the state is the reply, with no
|
||||
model involved. This is the same rule the host follows about a service that does not exist, and
|
||||
it is the rule this repository has now paid for three times: **absence must never be
|
||||
indistinguishable from a failure to answer**
|
||||
([005](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md),
|
||||
[008](../../04-ISSUES/008-the-documented-node-rescue-does-not-exist/00-report.md)).
|
||||
|
||||
## Model access is bound to the session, not to the machine
|
||||
|
||||
**A session is a consumer in its own right.** This is the gap
|
||||
[`14-model-access.md`](14-model-access.md) records as *a consumer that is not a machine*, and it
|
||||
is the one part of this design that cannot be built from what exists: the provisions model has no
|
||||
consumer identity other than a node, so today a binding can only say *this module on this
|
||||
machine*.
|
||||
|
||||
That is not sufficient here, and the shortfall is concrete rather than theoretical:
|
||||
|
||||
- the controller node hosts **two** sessions, which must be able to hold **different**
|
||||
licences — a per-machine binding cannot express it at all;
|
||||
- *this node's session uses the personal licence, the mesh's uses the company one* is the ordinary
|
||||
case, not an exotic one.
|
||||
|
||||
**What is delivered still lands on a machine. What is chosen is chosen per session.** Delivery
|
||||
follows the session's context root, which is where its credentials belong — the same rule ADR 0001
|
||||
states for an agent's config directory, with the root standing in for it.
|
||||
|
||||
**Switching a licence remains a reaction, not a declaration** (ADR 0024). A session that exhausts
|
||||
a licence is observed and its binding changed; the binding is then declared as usual. Nothing here
|
||||
grows a conditional in the declaration language.
|
||||
|
||||
## What it remembers, and where
|
||||
|
||||
**A session's memory lives in its context root**, beside its engram and its tools. That is the
|
||||
same rule as everything else here rather than a new one: the root is the whole of what makes one
|
||||
session a different agent from another, and memory is part of what makes it *that* agent.
|
||||
|
||||
**The mesh session's memory is its own.** It is not assembled from the node sessions on demand,
|
||||
and it is not a view over theirs. What the mesh has been asked, and what it worked out, is held
|
||||
in the mesh's root — not in the root of the node that happens to host it.
|
||||
|
||||
**That distinction is the point of putting it there.** The controller node runs two sessions
|
||||
on one machine. If memory belonged to the machine rather than to the root, they would share it,
|
||||
and the mesh's recollection of a fortnight of questions would be indistinguishable from that
|
||||
node's own — which is the collision ADR 0026 exists to avoid, arriving through the back door.
|
||||
|
||||
**The root holds two kinds of thing, and confusing them destroys the memory.** The engram and the
|
||||
tools are **declared**: the mesh says what they are and the host writes them, so editing one on
|
||||
the machine survives until the next heartbeat and no longer
|
||||
([ADR 0011](../../02-DECISIONS/0011-managed-files-are-generated-never-edited.md)). The memory is
|
||||
**written by the session itself** and is declared by nobody — the mesh does not get to say what a
|
||||
session remembers, and a mechanism that regenerates the root wholesale would erase a fortnight of
|
||||
it on the next pass, silently, while reporting success.
|
||||
|
||||
So the root is not uniformly managed, and **which parts are must be explicit rather than
|
||||
inferred**. A session's memory is its own output, kept across restarts, backed up as data rather
|
||||
than reproduced from a declaration — because there is nothing to reproduce it from.
|
||||
|
||||
## What the mesh session knows
|
||||
|
||||
**It holds the design record by reading it**
|
||||
([ADR 0025](../../02-DECISIONS/0025-the-design-record-is-read-not-copied.md)), not by holding a
|
||||
copy. It is that reader; there is not a second agent for it.
|
||||
|
||||
**It answers into a symptom search**, so what it knows appears beside ordinary results rather than
|
||||
only when it is asked. **And when it cannot be reached, the search says so.** A result set that
|
||||
silently omits this material looks identical to one where nothing matched — the same rule as the
|
||||
switched-off session above, at a different layer.
|
||||
|
||||
**Reading is one-way.** It reads the repository and answers from it; nothing flows back. The
|
||||
repository is public and the mesh is not, and a return path is how installation-specific detail
|
||||
arrives into documents that must not carry it.
|
||||
|
||||
## What it does with work
|
||||
|
||||
**It dispatches; it does not hold a queue.** Asked for something that belongs elsewhere, it asks a
|
||||
node session or hands the work to a worker. It is not an employee
|
||||
([ADR 0003](../../02-DECISIONS/0003-agents-are-persistent-employees.md)): nobody hires it, it is
|
||||
never drained, and it is not reassigned.
|
||||
|
||||
The distinction is the one ADR 0001 was written to keep: a session comes with the thing it belongs
|
||||
to and goes when that thing goes; a worker is hired, holds tasks, and moves. Built from the same
|
||||
parts, run on entirely different terms.
|
||||
|
||||
## The surface is separate
|
||||
|
||||
That a person usually reaches the mesh session through a board is likely and is not settled here.
|
||||
The session is reachable over the broker like everything else; what puts a text box in front of it
|
||||
is a different design, and the session does not know which surface asked.
|
||||
|
||||
**Several sessions open at once is a property of the surface, not of the sessions.** A board
|
||||
showing the mesh's session beside one per node, switchable, leaves *one per node, permanent*
|
||||
untouched: each is still one conversation remembering all its callers together. Only *concurrent
|
||||
conversations with the same session* would touch ADR 0004, and that is not what is wanted.
|
||||
|
||||
Such a surface is a **client** and holds nothing — it sends prompts and shows what the sessions
|
||||
themselves remember, which is what keeps it inside `11-a-board.md`'s rule that the board is not a
|
||||
second implementation of anything. Prior art to draw the interaction from: impire's *soulstream*
|
||||
(already cited in [ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md)) and
|
||||
`herdrdev/herdr`. **Not yet designed, and not on the path to replacing provisioning** — recorded
|
||||
here so the shape is not rediscovered.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Two sessions on one node, and no ambiguity.** The controller node hosts its own node session
|
||||
and the mesh session. ADR 0004's *one per node* forbids ambiguity about who answers when a **node**
|
||||
is addressed; these answer to different addresses.
|
||||
|
||||
**A new node gets a session by being declared, not by being set up.** The context root is declared
|
||||
state, so a joining node's session arrives the way its packages and services do.
|
||||
|
||||
**The mesh session is a single point of convenience, and must never become one of reach.** Every
|
||||
node stays directly addressable. If that ever stops being true, the failure is quiet in the worst
|
||||
way: every machine keeps working and nobody can ask about it.
|
||||
|
||||
## How it is checked
|
||||
|
||||
**A test names the decision it defends** ([ADR 0017](../../02-DECISIONS/0017-a-test-defends-a-decision.md)),
|
||||
and these run in the lab on real machines:
|
||||
|
||||
| Check | Defends |
|
||||
|---|---|
|
||||
| a node is asked something and its session answers | ADR 0004 |
|
||||
| the mesh is asked something and the mesh session answers, on the controller node | ADR 0026 |
|
||||
| both sessions on that node answer, to their own addresses, without ambiguity | ADR 0026 |
|
||||
| a session switched off replies saying so, rather than timing out | ADR 0004 |
|
||||
| a session whose engram was changed reports having applied it, like any declared file | ADR 0005 |
|
||||
| the two sessions on one node hold different licences, and each uses its own | ADR 0024 |
|
||||
| a node is messaged directly while the mesh session is stopped, and answers | ADR 0001 |
|
||||
| a search consulting an unreachable mesh session says it was not consulted | ADR 0025 |
|
||||
|
||||
The last two are the ones worth writing first. Both defend properties that are invisible while
|
||||
everything works, and both describe a mesh that looks entirely healthy at the moment it has
|
||||
stopped telling the truth.
|
||||
|
||||
## Deliberately not decided
|
||||
|
||||
**How a person's identity reaches a session.** Callers are distinguished, but who a caller *is*,
|
||||
and whether a session should act differently for different people, is the human-agent question
|
||||
ADR 0001 leaves open and this does not close.
|
||||
@@ -0,0 +1,176 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-09-01
|
||||
decisions:
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
|
||||
---
|
||||
|
||||
# What a module must be able to say
|
||||
|
||||
**Measured, not guessed.** 127 manifests in the system being replaced were read and every key
|
||||
counted, then set against what the new manifest can express. This document is the coverage
|
||||
checklist: what is already sayable, what is deliberately not, and what is missing.
|
||||
|
||||
*Surveyed 2026-09-01. Counts are modules, not occurrences, unless stated.*
|
||||
|
||||
## Already sayable
|
||||
|
||||
| what it says | used by | how it is said here |
|
||||
|---|---|---|
|
||||
| **depends on another module** | 65 | `requires` naming a module. A requirement naming a module means *that module*, not anything providing the name |
|
||||
| **system packages** | 28 | the `package` shape |
|
||||
| **a container** | 48 | the `container` shape, pinned by digest |
|
||||
| **systemd units** | 17 | a `process`, which the mesh writes the unit for. *Was: a `file` for the unit and a `service` for its state — which made every module author write unit syntax, and is why `process` exists ([`18`](18-building-a-module.md))* |
|
||||
| **how to reach it** | 17 | `serves`, with the mesh adding which machine and where |
|
||||
| **a public name** | 11 | requiring `route` and contributing the name |
|
||||
| **ports it opens** | 11 | `listens`, from which filtering is computed |
|
||||
| **data directories, and who owns them** | 38 | `directory` with `owner`; never removed while holding anything ([ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md)) |
|
||||
| **what it provides and requires** | 15 + 2 | `provides` / `requires`, named for what the consumer is coupled to |
|
||||
| **restart when something changes** | 11 | `restart-on` |
|
||||
| **a generated credential** | 20 | `own-secrets`, sealed to the machine |
|
||||
| **a value that differs per node** | 43 | settings, and `computed` for what only the mesh knows |
|
||||
| **images built from source** | 2 | `build.artifacts` — an `image` from a Dockerfile, or a `bundle`, which names a language and lets the mesh choose the toolchain ([`18`](18-building-a-module.md)) |
|
||||
|
||||
## Deliberately not sayable
|
||||
|
||||
**Stage hooks — 36 modules.** Arbitrary code at install, configure and start. **The link may not
|
||||
carry an action** ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)): what may be pushed is
|
||||
bounded by form, and a command to run is not a form. A module needing setup logic ships a program
|
||||
that reads what the mesh delivered and reconciles — which is what the provisioners are, and they
|
||||
are ~350 lines each including the reasoning.
|
||||
|
||||
**Flavours — 6 modules.** Variants of one module. Retired in favour of claims
|
||||
([ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)): two display servers are two
|
||||
modules that both claim the seat, and adding a third changes nothing anywhere else. What is lost
|
||||
is `extends` chains, which were doing inheritance and are better as separate modules.
|
||||
|
||||
## Missing, and what each would take
|
||||
|
||||
Ordered by how many modules need it. The largest entry turned out not to be a gap at all, which
|
||||
is left in place rather than deleted: the first framing of it was wrong in an instructive way, and
|
||||
a checklist that quietly loses its biggest item reads as though nobody looked.
|
||||
|
||||
### Tool servers — 56 modules — *not a gap*
|
||||
|
||||
Over half the modules ship a `tools/` directory that becomes tools an agent can call on that node.
|
||||
This was written up as the largest single gap. **It is expressible with what exists**, and the
|
||||
first framing of it was wrong in a way worth keeping: *a module provides `tools`, the session
|
||||
requires them* does not work, because a requirement has exactly one answer and 56 modules offering
|
||||
tools would be 56 answers to one question.
|
||||
|
||||
Turn it around and it fits exactly. The session **provides** `tool-host`; every module offering
|
||||
tools **requires** it and **contributes** where its tools are. Many-to-one is what `contributes`
|
||||
has always been, and the session receives all of them in one file:
|
||||
|
||||
```
|
||||
given: { from: gitea, values: { at: … } }
|
||||
{ from: minio, values: { at: … } }
|
||||
{ from: umami, values: { at: … } }
|
||||
```
|
||||
|
||||
Verified by resolving it, not by reading the code. It also only became possible today: until
|
||||
[`022`](../../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md) was
|
||||
fixed, several modules on one node requiring the same thing was refused outright.
|
||||
|
||||
What remains is not vocabulary but a decision about **what a tool server is** — a container the
|
||||
module already runs, and what the session does with the list. That is work, not a missing shape.
|
||||
|
||||
### Schema migrations — 14 modules
|
||||
|
||||
A module with a database needs its schema brought up to date before it runs. The mesh does this
|
||||
for its own contexts and has no way for a *module* to declare it. The provisioner pattern covers
|
||||
it — a program that runs migrations and exits — but nothing expresses *this must happen before
|
||||
that starts*, which is the actual requirement.
|
||||
|
||||
### Configuration merging — 18 modules, 134 files
|
||||
|
||||
Files assembled from a module's default plus per-node overrides, with a strategy (`replace`,
|
||||
`merge`) and a format (`toml`, `yaml`, `json`). Settings already merge into a file's content; what
|
||||
is missing is format-aware merging.
|
||||
|
||||
**And it should stay missing.** A mechanism that understands TOML will be asked for YAML, then
|
||||
INI — which is how the arrangement being replaced became something nobody could hold in their
|
||||
head. The module knows its own format because it wrote the rest of the file.
|
||||
|
||||
### Health checks — 7 modules
|
||||
|
||||
`{type: port|url, expect: …}`. The mesh knows whether a container is running, which is not the
|
||||
same as whether it answers — a distinction this project has paid for twice already.
|
||||
|
||||
An `action` carries a `verify` and is exactly this shape. **It is not available to a module**: the
|
||||
link may not carry a command to run ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and a
|
||||
module's resources reach a machine over the link. So a health check needs a way to say *ask this
|
||||
and expect that* without saying *run this* — closer to a `listens` entry than to an action.
|
||||
|
||||
**This is the gap most worth closing**, because *running* and *answering* being conflated is a
|
||||
class of fault, not an inconvenience.
|
||||
|
||||
### Theme knobs — 3 modules, 101 values
|
||||
|
||||
`{theme: {kind: color|font|string, label}}` — declared so a ricing tool can offer them. Settings
|
||||
already carry the value; what is missing is the **metadata** saying a value is presentable and
|
||||
what kind it is. Small, self-contained, and only interesting once something presents them.
|
||||
|
||||
### Event routing — 2 modules
|
||||
|
||||
`{routing-key: tool}`, generating a consumer. Two modules; wait for a third before deciding.
|
||||
|
||||
### Publishing a package — 6 modules
|
||||
|
||||
Modules published to a registry and consumed as libraries. This is a *build* output the mesh does
|
||||
not deliver to a node, so it may not belong here at all.
|
||||
|
||||
## What checking the coverage found
|
||||
|
||||
Two faults, both surfaced by asking what a *real* node looks like rather than what a test does.
|
||||
Neither is about the vocabulary; both are about the machinery under it.
|
||||
|
||||
**A credential belonged to a machine, not to a module**
|
||||
([`022`](../../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md),
|
||||
fixed). A node running three services against one database could not be planned at all — and on
|
||||
the consumer's side did not refuse, it just gave two of the three no credential. Every scenario
|
||||
written to date had one consumer per node, which is the natural shape of a small test and not the
|
||||
shape of a machine.
|
||||
|
||||
**A consumer could not build a connection string**
|
||||
([`023`](../../04-ISSUES/023-a-consumer-cannot-build-a-connection-string/00-report.md), fixed). It
|
||||
had its password in the right shape and the host, port and user name were out of reach: the user
|
||||
name was invented by the provisioner and recorded nowhere, and the bound values sat in a JSON
|
||||
document that an application reading `KEY=value` cannot use.
|
||||
|
||||
The asymmetry was backwards, which is what made it worth stating. **The secret is the hard case** —
|
||||
the mesh must not be able to read it — and the secret was the part that already arrived. The host
|
||||
and port are ordinary facts held in the clear, and they were the ones stuck. Both halves came from
|
||||
the same thing: the mesh knew something and did not say it.
|
||||
|
||||
## What the survey found that is not about coverage
|
||||
|
||||
**Declaration and reality had drifted in the system being replaced.** Several live provisions are
|
||||
brokered by modules whose manifests declare nothing — a speech-to-text engine served to a consumer
|
||||
on another node, an object-store bucket held by a module whose manifest mentions only its
|
||||
database. **A manifest that does not have to be true stops being true**, which is the argument for
|
||||
resolution refusing rather than warning.
|
||||
|
||||
**Two derivations of the same fact.** A module's kind was computed in two places from different
|
||||
evidence — one from the manifest, one from what is on disk — producing different labels for the
|
||||
same module. There is one derivation here, and there should stay one.
|
||||
|
||||
**A live listing returned credentials in plaintext.** Not a coverage question, but the reason
|
||||
sealing is worth its inconvenience.
|
||||
|
||||
**An image store is a module, and was written up here as something the mesh does.** It was
|
||||
considered for the foundation and removed, because the test is not *can it grant itself one* —
|
||||
nearly anything passes that — but whether the controller needs it before it can give its first
|
||||
instruction. It does not. So a registry somebody runs for their own images is the same module as
|
||||
the one the mesh runs for its own: it offers a place to push, and claims that role once per
|
||||
machine.
|
||||
|
||||
**A rule was enforced only at the far end.** A module may not declare an action, and the host
|
||||
refused one correctly — but the controller accepted it into the catalogue, resolved it and
|
||||
pushed it, so the refusal arrived on a machine with nothing tying it back to the manifest. The
|
||||
rule held; it was just unusable, which is the same shape as the network shape that cost five
|
||||
failing tests before anyone read the host's log. It is now refused where it is written.
|
||||
@@ -0,0 +1,209 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-host cmd/mesh-bootstrap
|
||||
- mesh-host internal/bootstrap
|
||||
- mesh-lab test/integration/whole-mesh-full.test.ts
|
||||
updated: 2026-09-21
|
||||
decisions:
|
||||
- 02-DECISIONS/0067-genesis-is-a-pivot.md
|
||||
- 02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md
|
||||
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
- 02-DECISIONS/0005-the-node-host.md
|
||||
- 02-DECISIONS/0010-delivery.md
|
||||
- 02-DECISIONS/0036-bootstrap-ends-at-a-usable-mesh.md
|
||||
---
|
||||
|
||||
# Raising a mesh
|
||||
|
||||
How a mesh comes into existence on machines that have none, and how a machine joins one that
|
||||
already exists. This is the procedure an operator runs. It is not a description of the lab, and
|
||||
the lab does not have one of its own.
|
||||
|
||||
## Two moments, and only two
|
||||
|
||||
A mesh is raised once and joined many times, and the two are not variations of each other.
|
||||
|
||||
**Genesis** happens on one machine, when there is no mesh. Nothing can be asked, nothing can be
|
||||
granted, and nothing has been published anywhere. It is the only moment at which the ordinary
|
||||
rules cannot all hold at once, and it is resolved by a pivot
|
||||
([ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md)).
|
||||
|
||||
**Joining** happens on every machine after the first, and it is ordinary. A mesh exists, so it can
|
||||
be asked for a token and told what the machine should be. Joining installs the host and nothing
|
||||
else: no temporary anything, no foundation raised by hand, no registry.
|
||||
|
||||
Confusing the two is what produced a procedure that only ever worked in a fixture. A bed that
|
||||
raises four machines the same way has not tested genesis at all — it has tested joining, four
|
||||
times, with the first one hand-fed.
|
||||
|
||||
## What changed, and what did not
|
||||
|
||||
*2026-09-13.* The installer carries a builder now, and builds the controller it raises. Three
|
||||
records settle it: [ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md) that
|
||||
genesis builds rather than carries, [ADR 0071](../../02-DECISIONS/0071-where-genesis-gets-its-source.md)
|
||||
where it clones from, and [ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md) how
|
||||
the builder arrives — which also records an argument that failed. It was put that a produced image
|
||||
must be published before anything can fetch it, so the registry would have to come up before the
|
||||
controller. It does not: the machine that builds the image is the machine that runs it, and a
|
||||
local image is named by the digest of its own configuration exactly as a carried one is. **Building
|
||||
changes where the bytes came from, not where they are.**
|
||||
|
||||
So the pivot is unchanged, the registry is where it was, and one step was added before the bundle is
|
||||
written. What follows describes the program that exists.
|
||||
|
||||
## Genesis
|
||||
|
||||
The installer is a single program carrying **the builder** inside it — not the controller
|
||||
([ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md)). What cannot be fetched is
|
||||
the thing that does the fetching, so that is what is carried; everything else is made here.
|
||||
|
||||
It proceeds in one direction, and every step is safe to run again.
|
||||
|
||||
**First it refuses to start if the machine is not ready.** A container runtime, the ability to
|
||||
write where it must write, the host binary where it expects it — and a repository and a commit to
|
||||
build from, because an installer told nothing would raise a store and a broker and then have
|
||||
nothing to raise a controller from. A machine that is not ready is told what is missing rather
|
||||
than half-changed.
|
||||
|
||||
**Then it loads the carried builder and builds the controller with it**, from a repository on a
|
||||
mesh that already exists and a named commit ([ADR 0071](../../02-DECISIONS/0071-where-genesis-gets-its-source.md)).
|
||||
This is the same repository and path every later rebuild of the controller will use, so what
|
||||
raises the mesh is the same thing that will maintain it.
|
||||
|
||||
**Then it describes what the machine will become.** The image it just made is named by the digest
|
||||
of its own configuration — content-addressed and unforgeable, and requiring nothing to have served
|
||||
it. That is legal precisely where nothing could have served one, and it is why building here needs
|
||||
no registry: the machine that made the image is the machine that will run it.
|
||||
|
||||
**Then it raises the foundation and a temporary controller, and waits for that controller to
|
||||
answer.** At this point the machine is a mesh of one node with nothing joined to it.
|
||||
|
||||
**Then the machine joins the mesh it is itself running.** It enrols, and an agent runs on it. Being
|
||||
heard from once is not the same as an agent running, and the installer checks both, because
|
||||
enrolling is itself the thing that makes a mesh hear from a machine.
|
||||
|
||||
**Then it installs a registry**, so the mesh has somewhere to keep its own images.
|
||||
|
||||
**Then it publishes the controller's image to that registry**, which is the moment the image
|
||||
first receives a digest assigned by something other than itself. This is the carrying step, and it
|
||||
is the same step for all three things the build loop cannot produce for itself — the controller,
|
||||
the registry, and the builder. The rule and its closed list are in
|
||||
[`12-a-module-repository`](12-a-module-repository.md#the-three-the-loop-cannot-build-and-there-are-only-three).
|
||||
|
||||
**Then it installs the controller again, as an ordinary module pinned to that digest, and drops
|
||||
the temporary one.** The pivot is complete: what raised the mesh is gone, and what runs is a module
|
||||
like any other. From here the mesh can build and roll out its own upgrades, including to the thing
|
||||
that runs it.
|
||||
|
||||
## After the pivot, and still part of installing
|
||||
|
||||
Genesis ends with a mesh of one that runs, and that is not the same as a mesh that works. What it
|
||||
has is a controller, a store, a queue and a registry. What it cannot yet do is **produce
|
||||
anything** — and almost every module in the catalogue is waiting to be produced, because a manifest
|
||||
names what its artifacts are and nothing has made them.
|
||||
|
||||
So installing continues:
|
||||
|
||||
**The builder arrives, and installing is what brings it.** It is a module like any other and is
|
||||
assigned to a machine like any other, but it cannot be built by the thing it is — see
|
||||
[`12-a-module-repository`](12-a-module-repository.md#the-three-the-loop-cannot-build-and-there-are-only-three).
|
||||
So it is carried, and it is already here: it is what built the controller. The last step of
|
||||
installing publishes it into the mesh's own registry, installs it as an ordinary module pinned to
|
||||
that digest, and issues it a broker account — the same two acts the controller went through,
|
||||
plus the one thing only a builder needs. The account is issued before the machine is sent anything,
|
||||
because a builder that arrives without its credential starts, finds nothing it may read, and waits,
|
||||
which looks exactly like a builder with no work.
|
||||
|
||||
**The core modules are built.** Each is named by a repository and a path
|
||||
([ADR 0069](../../02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md)), and the builder is
|
||||
asked for each in turn: it clones, reads the manifest at that path, produces what it declares,
|
||||
publishes each artifact into the mesh's own registry, and hands back the module with its artifacts
|
||||
pinned and the commit recorded. The mesh records that, and from then on the module is described by
|
||||
something it made rather than by a placeholder.
|
||||
|
||||
**The controller is built like the rest.** It was carried in and published once, which got the
|
||||
mesh running; building it from its own repository and path is what makes it upgradeable. The first
|
||||
time that happens is the moment the mesh stops depending on the installer for anything.
|
||||
|
||||
**And then the catalogue.** Every module with source of its own is built the same way. Until this
|
||||
has happened a mesh can install only what is public or carried, which is the foundation and little
|
||||
else.
|
||||
|
||||
Only after all of that is the ordinary loop available: change a module's source, the mesh notices
|
||||
its own copy is older than the source, rebuilds it, and rolls it out. That loop is what makes
|
||||
moving services across one at a time possible, so it is part of installing rather than something
|
||||
that comes later.
|
||||
|
||||
What remains after *that* belongs to somebody else: adding machines, and deciding what they run.
|
||||
|
||||
## Joining
|
||||
|
||||
A machine joins with the host binary and a token. It does not raise a foundation, does not install a
|
||||
registry, and is never enrolled twice. The mesh already knows how to tell a machine what to be;
|
||||
joining is the point at which a machine starts listening.
|
||||
|
||||
## Where the line falls
|
||||
|
||||
The installer owns everything that is the mesh's own. The lab owns only what is the lab's: raising
|
||||
virtual machines, giving them addresses that resolve nowhere, and injecting faults.
|
||||
|
||||
**The lab runs the installer. It does not describe installation.** This is the whole point. A
|
||||
second description kept in step with the first is the arrangement that already failed — the fixture
|
||||
invented a registry that exists in no production, and hid two separate faults for as long as it
|
||||
existed. Anything the lab must do that the installer does not is either a lab concern or a hole in
|
||||
the installer, and saying which is a decision, not a convenience.
|
||||
|
||||
## What is not yet true
|
||||
|
||||
Stated plainly, because a document that implies otherwise is worse than none.
|
||||
|
||||
**The installer is not packaged.** It builds from source. An operator raising a first machine still
|
||||
needs a toolchain and a working tree, which is most of the burden the installer exists to remove.
|
||||
|
||||
**A mesh cannot say how it was raised.** Nothing afterwards can contradict a claim that a machine
|
||||
was brought up the supported way, so the rule that it must be is, today, unenforced.
|
||||
|
||||
**Nothing asks for the core modules yet.** Installing ends with a builder that works — a raised
|
||||
machine will build the shared base and a module on top of it when asked — and nothing asks. The
|
||||
paragraphs above describing the catalogue being built are a thing somebody now types, rather than a
|
||||
thing that cannot happen.
|
||||
|
||||
**A module's declaration still has to be copied onto the machine by hand.** The installer reads the
|
||||
registry's and the builder's manifests from a checkout somebody put there. The controller's is no
|
||||
longer among them: it lives in the controller's own repository
|
||||
([ADR 0069](../../02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md)), the build the
|
||||
installer runs reports it with the artifact resolved, and genesis registers that
|
||||
([issue 072](../../04-ISSUES/072-the-controllers-manifest-exists-twice/00-report.md)). The
|
||||
builder's manifest belongs in the same repository and is still read from the catalogue; that is
|
||||
the last of this.
|
||||
|
||||
**A machine has no account for a registry that asks for one.** The mesh grants a consumer a
|
||||
credential for a database; it does not yet do so for the store its own images live in. Genesis
|
||||
avoids the question by carrying the image it needs, which makes this a joining problem and a
|
||||
pulling problem, not a genesis one.
|
||||
|
||||
**The SDK still comes from a git URL, and the ordering that fixes it is decided but not built.**
|
||||
[ADR 0076](../../02-DECISIONS/0076-the-sdk-is-a-published-package.md) settles that the package
|
||||
registry (gitea) comes up and the SDK is published into it *before* the base toolchain is built, so
|
||||
the toolchain resolves the SDK by version rather than cloning it — closing
|
||||
[issue 053](../../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md). The
|
||||
builder already knows how to be handed a package-registry credential and inject it into a build; what
|
||||
is not yet wired is the genesis step that raises gitea and publishes the SDK ahead of the base, and
|
||||
the toolchain's own manifest still names the SDK by a git URL. Until both land, the base build clones
|
||||
the SDK inside `docker build`, which is slow and names a branch head rather than a version.
|
||||
|
||||
## How these rules are checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| Genesis works on a machine that is not the lab | The bed raises its first machine by running the installer, not around it. A bed that stops doing so fails its own acceptance check. |
|
||||
| The other machines join, and are not re-raised | The bed gives them the host binary and a token only. A second enrolment of the first machine is a failure, not a no-op. |
|
||||
| Every step may be run again | The installer is re-run against a raised machine and must change nothing and report why. |
|
||||
| An image is named exactly | A machine refuses a bundle naming an image by tag. The refusal is exercised, not assumed. |
|
||||
| The installer is what installed this | **Nothing.** See above. |
|
||||
| The builder can arrive on a fresh mesh | The installer carries it and installs it as its last step, and the genesis bed raises a machine by running the installer. A bed that raises one any other way fails its own acceptance check. |
|
||||
| The controller a mesh runs is one it built | The genesis bed asserts the running controller is pinned to a digest this mesh's own registry serves, for an image built from a named repository and commit — not one the installer carried. |
|
||||
| A core module is built rather than only carried | The controller is rebuilt from its own repository and path, and the running mesh is upgraded to the result — the same path any module takes. |
|
||||
| Installing produced a mesh that can produce | A module with source of its own is asked for and comes back pinned to a digest this mesh's registry assigned, not to a placeholder. **Done by hand on a raised machine, not yet by a bed** — it built the shared base and then a module naming that base. Nothing automated asserts it, which makes this the weakest check on this page. |
|
||||
@@ -0,0 +1,240 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code:
|
||||
- mesh-controller cmd/mesh-builder
|
||||
- mesh-controller internal/builder
|
||||
- mesh-catalog modules/builder
|
||||
updated: 2026-09-21
|
||||
decisions:
|
||||
- 02-DECISIONS/0091-a-mount-is-declared-three-ways.md
|
||||
- 02-DECISIONS/0087-a-seeded-file-is-created-once.md
|
||||
- 02-DECISIONS/0040-what-a-module-is.md
|
||||
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
|
||||
- 02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md
|
||||
- 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
- 02-DECISIONS/0072-two-graphs-and-the-build-chain.md
|
||||
- 02-DECISIONS/0076-the-sdk-is-a-published-package.md
|
||||
---
|
||||
|
||||
# Building a module
|
||||
|
||||
[`12-a-module-repository`](12-a-module-repository.md) says what a module may build — an image, an
|
||||
archive, an upstream mirror — and where the result goes. This says **how a build is modelled**, and
|
||||
why the current model does not fit what a module is.
|
||||
|
||||
## The domain, in one sentence
|
||||
|
||||
Turning a module's source into artifacts the mesh can pin, publish and deliver — and saying
|
||||
truthfully what was produced and what it was produced against.
|
||||
|
||||
Everything else is somebody else's: *what* to build is the controller's, *what a build means* is
|
||||
the catalogue's ([ADR 0072](../../02-DECISIONS/0072-two-graphs-and-the-build-chain.md)), *where an artifact runs* is the
|
||||
controller's again. The builder's whole responsibility is the middle.
|
||||
|
||||
## The language
|
||||
|
||||
| term | is |
|
||||
|---|---|
|
||||
| **source** | a repository, a path within it, and a ref — resolved to one commit ([ADR 0069](../../02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md)) |
|
||||
| **recipe** | how *one* artifact is produced from that source |
|
||||
| **toolchain** | what a recipe runs inside — a compiler, a runtime, the SDK |
|
||||
| **artifact** | what a recipe produced, named by the digest of its content |
|
||||
| **publication** | putting an artifact where machines can fetch it, by that digest |
|
||||
| **announcement** | telling the mesh what was built, and every artifact it stood on |
|
||||
| **build** | one request and its outcome, correlated, recorded whether it worked or not |
|
||||
|
||||
## What the model is today, and where it does not fit
|
||||
|
||||
**A toolchain is not modelled at all** — it arrives as two build arguments the module's own
|
||||
Dockerfile declares and the mesh fills in. **A language is not a concept.** And a recipe is
|
||||
effectively singular: producing anything *compiled* means writing a Dockerfile.
|
||||
|
||||
**Archives already work, and that is the corrected half of this.** An earlier draft of this
|
||||
document said the builder refused them. It does not: an archive is packed deterministically,
|
||||
hashed, published by digest, fetched by the machine and unpacked. Only the *local* builder used at
|
||||
genesis refuses one, deliberately — an archive is bytes that mean nothing until something serves
|
||||
them, and there is no registry yet.
|
||||
|
||||
**What an archive cannot do is compile.** `from` names a directory and the directory is packed as
|
||||
it stands, so shipping compiled output means compiling somewhere first — which means a Dockerfile,
|
||||
which is the burden this is about. The gap is not the artifact kind. It is that **no recipe both
|
||||
builds and packs**.
|
||||
|
||||
The cost is not theoretical. To add a module that carries its own code today, an author writes a
|
||||
Dockerfile that: declares two `ARG` bases with no defaults; compiles under a specific working
|
||||
directory so the SDK resolves upward; invokes the compiler *by absolute path*, because the usual
|
||||
`node_modules/.bin` entry is a symlink that the base image's own assembly resolves away; copies the
|
||||
output into a second stage; and sets an environment variable naming the compiled entrypoints. Every
|
||||
module repeats it. Miss any step and the failure arrives somewhere else — as a crash loop, a
|
||||
placeholder digest, a module that builds and does nothing.
|
||||
|
||||
The evidence that this is too hard is in the catalogue: **most modules are not converted**, and the
|
||||
two converted during one session were each wrong twice before they were right, against a working
|
||||
example sitting open in the next window.
|
||||
|
||||
**A module is not a container** ([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)) — it is
|
||||
one piece of software and everything that makes it real: its provisioner, its tools, its hooks, its
|
||||
scheduled steps, its event consumers. A build model whose only output is a container image is
|
||||
modelling one of ten resource kinds and calling it the module.
|
||||
|
||||
## The model
|
||||
|
||||
**Recipe becomes explicit, and there is more than one kind.**
|
||||
|
||||
| recipe | produces | from |
|
||||
|---|---|---|
|
||||
| `image` | an image | a Dockerfile, when the software genuinely needs one |
|
||||
| `archive` | a bundle, fetched by digest and unpacked | a directory, packed as it stands — **today** |
|
||||
| `bundle` | the same, fetched and unpacked | this module's source, *compiled* by a toolchain and then packed — **the missing one** |
|
||||
| `upstream` | a mirror | somebody else's pinned reference |
|
||||
|
||||
**Toolchain becomes explicit, and is derived rather than written.** A module says what it is written
|
||||
in; the builder knows what that implies. The two base images stop being something an author names
|
||||
and become something a toolchain *is*.
|
||||
|
||||
```
|
||||
module says: language: typescript
|
||||
builder knows: compile in the typescript toolchain, bundle, produce an archive
|
||||
```
|
||||
|
||||
A Dockerfile remains available and stops being compulsory. It is the right answer for software that
|
||||
needs a particular base, and the wrong answer for "compile my module's code", which is the same
|
||||
operation every time.
|
||||
|
||||
**Why an archive and not always an image.** An archive is a content-addressed blob fetched over
|
||||
plain HTTP and verified by its own digest, so it needs no registry account and no trusted transport
|
||||
— it verifies itself. A container image is refused by a runtime over plain HTTP as *policy*, which
|
||||
is why [issue 048](../../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)
|
||||
exists. Modules that are the mesh's own code do not need a container's isolation from the mesh; they
|
||||
need to run. Third-party software still arrives as an image, because that is how its author ships it.
|
||||
|
||||
## The invariants
|
||||
|
||||
1. **An artifact is named by the digest of its content.** Not by a tag, not by a path, never by a
|
||||
placeholder. A recipe that cannot produce a digest has produced nothing.
|
||||
2. **A build announces exactly what it made and everything it stood on.** The second half is what
|
||||
makes build edges derived rather than declared ([ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)).
|
||||
3. **A failed build produces nothing and announces nothing.** A partial artifact under a digest the
|
||||
mesh will later trust is worse than no artifact.
|
||||
4. **A recipe with nowhere to publish refuses before it builds, not after.** An archive is bytes that
|
||||
mean nothing until something serves them; producing one with no destination has produced nothing
|
||||
usable, and saying so beats returning a path no other machine can read.
|
||||
5. **A build is recorded whether or not it worked**, with everything it was told — the resolved
|
||||
manifest, the path, what it stood on. A record that keeps only the outcome cannot be replayed to
|
||||
anything that missed it ([issue 050](../../04-ISSUES/050-the-catalogue-knows-nothing-built-before-it/00-report.md)).
|
||||
|
||||
## What this costs, stated before it is chosen
|
||||
|
||||
**Every language is permanent.** It needs an SDK — broker client, sealed-credential reading, the
|
||||
event envelope, tool serving — a toolchain image, and a bundler the builder understands. And the
|
||||
spine changes rarely but cascades when it does
|
||||
([ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)): with one language a contract
|
||||
change is one edit; with four it is four that must land together, and a mesh whose SDKs disagree
|
||||
about the envelope fails by ignoring messages rather than by failing to compile.
|
||||
|
||||
**So the contracts have to stop being expressed twice before they are expressed four times.** They
|
||||
are already: the manifest, declaration and link shapes exist as Go structs in the controller and
|
||||
as TypeScript types in the SDK, kept in step by hand. Nobody has felt it because both live in one
|
||||
repository. A second *language* makes that drift; a specified envelope and schema that every SDK
|
||||
implements makes a second language an implementation rather than a translation.
|
||||
|
||||
**Order matters, then.** Language-neutral contracts, then a second language. The other way round
|
||||
makes the drift worse while hiding it.
|
||||
|
||||
## How these rules are checked
|
||||
|
||||
| rule | checked by |
|
||||
|---|---|
|
||||
| A module with its own code needs no Dockerfile | A module declaring only a language builds, and its artifact is pinned to a digest the mesh's registry assigned. |
|
||||
| An archive is produced and delivered | A module declaring an archive is built, and the machine assigned it has the unpacked files — verified on the machine, not in the build's own output. |
|
||||
| A Dockerfile still works | A module that declares one builds exactly as before, because software that needs a particular base has not stopped existing. |
|
||||
| Nothing is announced that was not made | A build made to fail announces nothing, and the catalogue's graph is unchanged afterwards. |
|
||||
| A build keeps what it was told | A catalogue started after a build asks for what it missed and receives the manifest and the artifacts stood on, not a summary. |
|
||||
| The toolchain is the mesh's, not the author's | Changing the toolchain makes every module built against it stale, and each names what moved. |
|
||||
|
||||
## The whole surface, in one place
|
||||
|
||||
**A reference, and it is kept true by a test rather than by care.** A table like this goes stale the
|
||||
day somebody adds a field, so `mesh-catalog/modules/showcase` is a module that uses all of it, and
|
||||
`TestTheShowcaseModuleIsAValidManifest` fails when it stops doing so. Read the module when this
|
||||
disagrees with it.
|
||||
|
||||
### What a module declares
|
||||
|
||||
| field | is |
|
||||
|---|---|
|
||||
| `module`, `version`, `slug` | its identity; the slug is the short name generated names are built from |
|
||||
| `capabilities` | what a machine must have for this to run there |
|
||||
| `provides` | **the shared seat** — several modules may fill one capability and coexist |
|
||||
| `claims` | **the exclusive seat** — two modules claiming one thing in a scope cannot both be assigned there |
|
||||
| `serves` | what a consumer must know to connect. The mesh fills in the assigned port ([ADR 0038](../../02-DECISIONS/0038-the-mesh-assigns-the-port.md)) |
|
||||
| `requires` | what must be provided by something on the same node |
|
||||
| `binds` | where the mesh writes what a requirement resolved to |
|
||||
| `secrets` | where the mesh seals the credential for a requirement |
|
||||
| `own-secrets` | secrets that are the module's own — a superuser, a broker account |
|
||||
| `emits` / `consumes` | the event graph: 1:many, broadcast, no credential |
|
||||
| `listens` | the port **its own software** uses, and from where. Filtering is computed from these |
|
||||
| `contributes` / `receives` | values one module adds to another's configuration, and the other half |
|
||||
| `accesses` | operator-owned paths it may use and must not own ([ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md)) |
|
||||
| `certificate` | a certificate for a name it serves |
|
||||
| `grants` | credentials it must create for its consumers |
|
||||
| `filtering` | rules beyond its own ports |
|
||||
| `computed` | marks a module the controller generates rather than an author writing |
|
||||
| `build.artifacts` | what it produces |
|
||||
|
||||
**A container mounts only what the manifest declares**
|
||||
([ADR 0091](../../02-DECISIONS/0091-a-mount-is-declared-three-ways.md)). A bind mount the module
|
||||
never declared is created by the container runtime as root, so the module's owner and mode never
|
||||
reach its data and the rule that keeps data when a module goes away does not cover it. A path is
|
||||
declared in one of three ways, for the three things a path can be: the module's own (a directory
|
||||
or file resource, or where a secret, grant or contribution lands), the operator's (an `accesses`
|
||||
entry), or the machine's (a facility a declared capability grants — `container-runtime` grants its
|
||||
socket). *How it is checked:* the parser refuses an undeclared mount naming the path and the three
|
||||
remedies, and a test parses every manifest in the catalogue beside the checkout.
|
||||
|
||||
### What it builds
|
||||
|
||||
| kind | is |
|
||||
|---|---|
|
||||
| `bundle` | its own code, compiled by the toolchain its language implies, then packed |
|
||||
| `archive` | a directory, packed as it stands |
|
||||
| `image` | built from a Dockerfile — for software that needs a particular base |
|
||||
| `upstream` | somebody else's image, mirrored and pinned by a digest this mesh assigned |
|
||||
|
||||
### What it puts on a machine
|
||||
|
||||
| resource | is | a module may |
|
||||
|---|---|---|
|
||||
| `directory` | a directory with a mode and an owner | ✅ |
|
||||
| `file` | literal content, with `${bound:…}` and `${secret:…}` filled in | ✅ |
|
||||
| `user` | a login | ✅ |
|
||||
| `access` | a pre-existing path it may use and must not own | ✅ |
|
||||
| `archive` | files fetched by digest and unpacked | ✅ |
|
||||
| `package` | a package that must be present | ✅ |
|
||||
| `network` | a named container network | ✅ |
|
||||
| `container` | an image, in three modes | ✅ |
|
||||
| `process` | **its own code**, in three modes | ✅ |
|
||||
| `service` | an **existing** unit put into a state | for software shipping its own unit |
|
||||
| `action` | a command to run | ❌ **refused** — [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
|
||||
|
||||
A `file` may also say `create-once`: written when absent, left alone when present, reported as
|
||||
kept — a seed a program then owns ([ADR 0087](../../02-DECISIONS/0087-a-seeded-file-is-created-once.md)).
|
||||
Checked by the host's apply tests and by the vault bed, which grows into one and pushes again.
|
||||
|
||||
**The two at the bottom are the interesting rows.** `action` is refused outright: the link may not
|
||||
carry a command, so a module needing something done ships a program that reconciles — which is what
|
||||
a run-once `process` is. `service` installs no unit by design, which is right for software that
|
||||
ships one and wrong for code the mesh built, which has no unit until the mesh writes it.
|
||||
|
||||
### How its code runs, and what that code can be
|
||||
|
||||
| mode | is | | shape | loaded by |
|
||||
|---|---|---|---|---|
|
||||
| *(default)* | a unit restarted when it exits | | tools | a tool host, over the broker |
|
||||
| `run-once` | run to completion; what follows is gated on it | | event consumer | the same host, reacting |
|
||||
| `schedule` | a timer; a missed fire happens when the machine returns | | provisioner | invoked when a consumer is granted |
|
||||
| | | | a process | the machine's supervisor |
|
||||
|
||||
**Tools, hooks and consumers are not further modes**, which is the test of whether three is the
|
||||
right number: they are loaded by a tool host, and a tool host is a process that stays up.
|
||||
@@ -0,0 +1,187 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code:
|
||||
- mesh-sdk src
|
||||
- mesh-tools src/broker-amqp.ts
|
||||
- mesh-controller internal/link
|
||||
updated: 2026-09-15
|
||||
decisions:
|
||||
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
|
||||
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
|
||||
- 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md
|
||||
- 02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md
|
||||
- 02-DECISIONS/0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md
|
||||
---
|
||||
|
||||
# The module protocol
|
||||
|
||||
**What a module's code and the mesh say to each other.** An SDK is an implementation of this in one
|
||||
language and nothing more ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md)).
|
||||
|
||||
This is a specification, so it says what is required rather than how anything is arranged. Where it
|
||||
describes current behaviour that is *not yet* specified-and-conformed, it says so.
|
||||
|
||||
## The shape of it
|
||||
|
||||
A **floor** every implementation needs, and three **capabilities** that are independent of each
|
||||
other. An SDK implements the floor plus whatever capabilities it claims; a module is refused at
|
||||
build time if it uses a capability its language's SDK does not implement.
|
||||
|
||||
| part | a module uses it to |
|
||||
|---|---|
|
||||
| **connection** | reach the broker as itself |
|
||||
| **events** | emit, and react to what others emit |
|
||||
| **tools** | answer questions asked of it |
|
||||
| **provisioning** | give a consumer an instance of what it provides |
|
||||
|
||||
---
|
||||
|
||||
## The floor: connection
|
||||
|
||||
### The credential
|
||||
|
||||
A module is given a **sealed credential** as a file, and told where by its declaration. The
|
||||
document:
|
||||
|
||||
| field | is | required |
|
||||
|---|---|---|
|
||||
| `url` | an `amqps://` URL carrying the account's user and password | yes |
|
||||
| `fingerprint` | sha256 of the certificate the broker must present | yes for a scoped account |
|
||||
| `node` | the machine this account was issued for | yes for a scoped account |
|
||||
| `module` | the module this account was issued for | yes for a scoped account |
|
||||
|
||||
A plain string rather than a document is a **bootstrap URL** — unscoped, for the moment before a
|
||||
mesh can issue anything. An implementation accepts both and must not treat the second as ordinary.
|
||||
|
||||
### Connecting
|
||||
|
||||
- The connection **pins the fingerprint**. It does not trust a certificate authority, and it does
|
||||
not skip verification. A broker presenting a different certificate is refused, whatever else is
|
||||
true of it.
|
||||
- A scoped account **does not declare exchanges**. The foundation owns them; an account that may
|
||||
declare one is an account that may create a parallel mesh by typo.
|
||||
- An implementation **declares its own queue** and nothing else.
|
||||
|
||||
### Identity
|
||||
|
||||
**A module's node and module name come from its credential, never from its environment.**
|
||||
|
||||
This is not a convenience. It is what makes what a module emits match what the mesh authorised: an
|
||||
environment variable can be set by anything on the machine, and a module that took its identity
|
||||
from one could emit events attributing them to another module. Where an environment variable and
|
||||
the credential disagree, the credential wins and the variable is overwritten.
|
||||
|
||||
---
|
||||
|
||||
## Capability: events
|
||||
|
||||
### The exchanges
|
||||
|
||||
| exchange | carries |
|
||||
|---|---|
|
||||
| `mesh.events` | every event |
|
||||
| `mesh.events.dead` | what could not be handled |
|
||||
|
||||
### The queue
|
||||
|
||||
One **durable** queue per consumer, named `<node>.<module>.events`, with as many bindings as the
|
||||
module has patterns. Durable because an event emitted while a module is restarting is exactly the
|
||||
one that must not be lost.
|
||||
|
||||
**A message matching two bindings is delivered once**, so an implementation must match the routing
|
||||
key against its own patterns locally to decide which handlers run. An implementation that ran every
|
||||
handler whose exchange binding matched would run the wrong one.
|
||||
|
||||
### The envelope
|
||||
|
||||
Headers ride as AMQP headers. The body is JSON.
|
||||
|
||||
| header | is | required |
|
||||
|---|---|---|
|
||||
| `x-event-id` | a unique id, made by the emitter | yes |
|
||||
| `x-source` | the module, context or node that emitted it | yes |
|
||||
| `x-node` | the machine it was emitted from | yes |
|
||||
| `x-time` | emit time, RFC-3339 | yes |
|
||||
| `content-type` | always `application/json` | yes |
|
||||
| `x-causation-id` | the event or command that caused this one | no |
|
||||
| `x-schema` | a version of the body's shape | no |
|
||||
|
||||
**An unknown `x-` header is ignored, never refused.** An event is observed by parties that need not
|
||||
all understand every header, and an implementation that refused one would make adding a header a
|
||||
breaking change for everybody.
|
||||
|
||||
### Delivery
|
||||
|
||||
At-least-once. **Deduplication is on `x-event-id`**, which only the emitter can produce — a
|
||||
consumer cannot tell a redelivery from a second event any other way.
|
||||
|
||||
### What is true, checked (2026-09-16)
|
||||
|
||||
Go emits all five required headers; the SDK requires exactly those. `x-causation-id` and `x-schema`
|
||||
are **optional** — the SDK sets them when a handler has a causation or a schema, and reads them
|
||||
back; a bare event carrying neither is correct. So the envelope agrees across the two
|
||||
implementations. `x-schema` is available for versioning a body's shape and is set by whoever has a
|
||||
version to declare.
|
||||
|
||||
---
|
||||
|
||||
## Capability: tools
|
||||
|
||||
A module's tools are its operator-facing surface.
|
||||
|
||||
- A tool is served from a **shared durable queue**, `serve.<key>`. Shared, so several runtimes
|
||||
serving one tool compete for a call rather than each answering it.
|
||||
- A call is request and reply. The reply returns through the RPC exchange `mesh.rpc`, keyed by the
|
||||
caller's own reply queue — **not** through the default exchange, which would let a caller publish
|
||||
into any queue on the broker.
|
||||
- A caller needs a **reply queue**, and that is what a module's scoped account may not declare
|
||||
([issue 049](../../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md)).
|
||||
So a module may serve tools and may not call them, and nothing today issues an account to anything
|
||||
that wants to ask.
|
||||
|
||||
### Not yet true
|
||||
|
||||
The caller's half has no account. Until that is settled, the only thing that can ask a module a
|
||||
question is the foundation's bootstrap admin, which is not a protocol so much as a way in.
|
||||
|
||||
---
|
||||
|
||||
## Capability: provisioning
|
||||
|
||||
A provider ships the provisioner that creates instances of what it offers
|
||||
([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)).
|
||||
|
||||
| direction | carries |
|
||||
|---|---|
|
||||
| **grant, in** | the provision, who is asking — **a module on a machine**, not a machine — and what the consumer contributed |
|
||||
| **credential, out** | the fields the provision promises a consumer |
|
||||
|
||||
**Who is asking is one thing with two parts.** A machine routinely runs several modules wanting the
|
||||
same provision, so a grant addressed to a node alone does not name a consumer, and withdrawing one
|
||||
would take another's away.
|
||||
|
||||
### Checked, and it agrees (2026-09-16)
|
||||
|
||||
This looked like the sharpest disagreement and was not one. The live wire is the contributions file
|
||||
— `as`, `secret`, `node`, `at`, `values` — and it is the same on both sides. The types that
|
||||
disagreed (`Grant`, `Interface` in the SDK's `contracts`) were dead: exported, imported by nothing,
|
||||
describing fields the wire does not carry. They have been removed. The lesson kept: a type beside
|
||||
the wire that has drifted from it is worse than none, which is why the wire is specified and
|
||||
implementations are checked against it rather than trusted to still match a hand-kept shape.
|
||||
|
||||
---
|
||||
|
||||
## How an implementation is checked
|
||||
|
||||
Per capability, against fixtures rather than prose — a specification nobody can run is a document
|
||||
two implementations drift from while both believe they conform.
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The floor is the floor | Every implementation reads the same credential fixture, and refuses one whose fingerprint does not match what the broker presents. |
|
||||
| Identity comes from the credential | A fixture sets an environment that disagrees with the credential; the emitted event carries the credential's. |
|
||||
| The envelope is the envelope | An emitted event is compared header by header against a fixture; a missing required header fails, an unknown `x-` header is accepted. |
|
||||
| Delivery is at-least-once | A fixture delivered twice is handled once. |
|
||||
| A grant names a consumer | A grant fixture is read by every implementation and yields the same module and the same node. |
|
||||
| A partial SDK is legitimate | An implementation claiming the floor and events passes those suites and is listed for them; a module using tools in that language is refused at build time with the reason. |
|
||||
@@ -0,0 +1,180 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code:
|
||||
- mesh-catalog modules/showcase
|
||||
- mesh-controller internal/builder
|
||||
- mesh-sdk src
|
||||
updated: 2026-09-15
|
||||
decisions:
|
||||
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
|
||||
- 02-DECISIONS/0040-what-a-module-is.md
|
||||
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
|
||||
- 02-DECISIONS/0053-a-step-that-runs-on-a-schedule.md
|
||||
---
|
||||
|
||||
# Writing a module
|
||||
|
||||
A worked guide. One module, four capabilities, four languages, and the packages it publishes.
|
||||
|
||||
The reference for *what can be said* is [`18-building-a-module`](18-building-a-module.md); the
|
||||
reference for *what the code and the mesh say to each other* is
|
||||
[`19-the-module-protocol`](19-the-module-protocol.md). This is how you actually write one.
|
||||
|
||||
## First: what is in the SDK, exactly
|
||||
|
||||
**The protocol, and nothing else** ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md)).
|
||||
An SDK is an implementation of the module protocol in one language. If something is not in the
|
||||
protocol it does not belong in an SDK, and that rule is what stops it becoming the 34,000-line
|
||||
shared library this design exists to avoid ([ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)).
|
||||
|
||||
Split the way the protocol is split:
|
||||
|
||||
| the SDK gives you | so that you can |
|
||||
|---|---|
|
||||
| **connection** — reads the sealed credential, pins the certificate, takes your identity from it | reach the broker as *this module on this machine*, and not be able to claim otherwise |
|
||||
| **events** — emit, subscribe, the envelope, dedup | react to what happens in the mesh |
|
||||
| **tools** — register and serve | be asked questions |
|
||||
| **provisioning** — receive a grant, return a credential | give a consumer an instance of what you provide |
|
||||
|
||||
**What it does not give you, deliberately:**
|
||||
|
||||
- **No configuration loader.** Configuration arrives as files the mesh wrote and environment the
|
||||
mesh set. Reading a file is not a thing an SDK needs to teach.
|
||||
- **No API clients.** A Plex client belongs in the Plex module. It changes when Plex changes,
|
||||
which has nothing to do with any other module — *frequent **and** cascading is the disease.*
|
||||
- **No storage, no HTTP framework, no logging library.** Use the language's.
|
||||
|
||||
If you find yourself wanting to add something to the SDK, the test is ADR 0039's: **does editing it
|
||||
recompile unrelated modules, and does it change often?** Both, and it does not belong.
|
||||
|
||||
## A module with four capabilities, in four languages
|
||||
|
||||
A module is **one piece of software** ([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)) and
|
||||
may still be written in several languages — each artifact names its own, compiles alone and is
|
||||
packed alone.
|
||||
|
||||
```
|
||||
showcase/
|
||||
module.json
|
||||
events/ ← TypeScript: reacts to what the mesh does
|
||||
tools/ ← Go: answers questions
|
||||
provisioner/ ← Rust: grants instances of what it provides
|
||||
ingest/ ← Python: a scheduled job
|
||||
```
|
||||
|
||||
### The manifest
|
||||
|
||||
```json
|
||||
{
|
||||
"module": "showcase",
|
||||
"build": { "artifacts": [
|
||||
{ "name": "events", "kind": "bundle", "language": "typescript",
|
||||
"entrypoints": ["index.js"] },
|
||||
{ "name": "tools", "kind": "bundle", "language": "go",
|
||||
"entrypoints": ["tools"] },
|
||||
{ "name": "provisioner", "kind": "bundle", "language": "rust",
|
||||
"entrypoints": ["provisioner"] },
|
||||
{ "name": "ingest", "kind": "bundle", "language": "python",
|
||||
"entrypoints": ["ingest.py"] }
|
||||
]},
|
||||
"resources": [
|
||||
{ "id": "events", "type": "process", "name": "showcase-events",
|
||||
"artifact": "events", "run": ["node", "index.js"] },
|
||||
{ "id": "tools", "type": "process", "name": "showcase-tools",
|
||||
"artifact": "tools", "run": ["./tools"] },
|
||||
{ "id": "grants", "type": "process", "name": "showcase-grants",
|
||||
"artifact": "provisioner", "run": ["./provisioner"] },
|
||||
{ "id": "ingest", "type": "process", "name": "showcase-ingest",
|
||||
"artifact": "ingest", "run": ["python", "ingest.py"],
|
||||
"schedule": "0 3 * * *" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Four artifacts, four toolchains, four processes, one module.** Nothing here says how any of them
|
||||
is hosted — that is the mesh's, and it is why these are `process` rather than four containers.
|
||||
|
||||
### Step 1 — say what it is, in each language's own terms
|
||||
|
||||
Each part is an ordinary project in its language, depending on the mesh SDK for that language the
|
||||
way it would depend on anything:
|
||||
|
||||
```
|
||||
events/package.json "@novox/mesh-sdk": "^1.2.0"
|
||||
tools/go.mod require novox.example/mesh-sdk v1.2.0
|
||||
provisioner/Cargo.toml mesh-sdk = "1.2"
|
||||
ingest/pyproject.toml dependencies = ["mesh-sdk~=1.2"]
|
||||
```
|
||||
|
||||
**Resolved from the mesh's own package registries**, which are the ordinary registries for each
|
||||
ecosystem, hosted by the mesh. An author does nothing unusual: `npm install`, `go mod tidy`,
|
||||
`cargo build`, `pip install` all work, and a developer's laptop resolves exactly what a build does.
|
||||
|
||||
### Step 2 — write each part against its capability
|
||||
|
||||
Each uses only the part of the protocol it needs. The event consumer never learns what a grant is.
|
||||
|
||||
```
|
||||
events/index.ts on("module.builder.built", …) → the events capability
|
||||
tools/main.go tool("showcase_state", …) → the tools capability
|
||||
provisioner/main.rs grant → credential → the provisioning capability
|
||||
ingest/ingest.py emit("module.showcase.ingested", …) → events, emitting only
|
||||
```
|
||||
|
||||
### Step 3 — the mesh does the rest
|
||||
|
||||
You do not write a Dockerfile, a unit file, or a port number. The builder picks the toolchain from
|
||||
the language, compiles each artifact alone, and publishes it. The controller assigns the machine
|
||||
and the ports; the host writes the units.
|
||||
|
||||
### What it costs you to use four languages
|
||||
|
||||
**Honestly: four sets of dependencies to keep current, and four SDKs that must agree.** The mesh
|
||||
makes it possible, not free. A module in one language is simpler, and the reason to use four is
|
||||
that one of them genuinely suits a part better — not that you can.
|
||||
|
||||
## Publishing a package is a capability
|
||||
|
||||
A module may publish libraries as well as run code. **The SDK is not special; it is simply the first
|
||||
module that did this**, and its own consumers are the mesh's modules.
|
||||
|
||||
```json
|
||||
{
|
||||
"module": "plex",
|
||||
"build": { "artifacts": [
|
||||
{ "name": "server", "kind": "upstream", "from": "plexinc/pms-docker@sha256:…" },
|
||||
{ "name": "client-ts", "kind": "package", "language": "typescript", "from": "clients/typescript" },
|
||||
{ "name": "client-rust", "kind": "package", "language": "rust", "from": "clients/rust" },
|
||||
{ "name": "client-py", "kind": "package", "language": "python", "from": "clients/python" }
|
||||
]}
|
||||
}
|
||||
```
|
||||
|
||||
A `package` artifact is built and **published to the mesh's registry for that ecosystem**, under the
|
||||
version its own project file declares. Another module then depends on it the ordinary way:
|
||||
|
||||
```
|
||||
"@novox/plex-client": "^2.0.0"
|
||||
```
|
||||
|
||||
**Why this belongs to modules rather than being a separate thing.** A client for a piece of software
|
||||
changes when that software changes, and the module that owns the software is the only thing that
|
||||
knows. Putting the Plex client anywhere else is the shared-library disease with extra steps.
|
||||
|
||||
### What this leaves open
|
||||
|
||||
- **Which private registry.** That there *is* one is settled —
|
||||
[ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md) says each module consumes its dependencies
|
||||
from the private registry, the mesh's own shared library included. Which software serves it is
|
||||
not: the catalogue holds `verdaccio`, and a git host usually serves package registries too.
|
||||
- **And the bootstrap does not have one.** ADR 0014 assumes a registry exists; on a fresh mesh
|
||||
nothing has installed one when the first SDK is built. That is the same pivot as everything else
|
||||
and it has not been designed.
|
||||
- **Who may publish.** A builder pushing a package needs an account on that registry, which is a
|
||||
credential in the bootstrap path and does not exist yet.
|
||||
- **Versions and ranges.** Everything else the mesh delivers is pinned by digest, and a range is
|
||||
resolved at build time from whatever the registry holds. A lock file records what was chosen, and
|
||||
the builder records that as the build edge — but *a mesh that can rebuild a commit and get a
|
||||
different library* is a real change from how everything else here works, and it should be a
|
||||
decision rather than a consequence.
|
||||
@@ -0,0 +1,189 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code:
|
||||
- mesh-host internal/bootstrap
|
||||
- mesh-host cmd/mesh-bootstrap
|
||||
- mesh-lab test/integration/one-node-mesh.test.ts
|
||||
updated: 2026-09-21
|
||||
decisions:
|
||||
- 02-DECISIONS/0067-genesis-is-a-pivot.md
|
||||
- 02-DECISIONS/0073-the-installer-carries-a-builder.md
|
||||
- 02-DECISIONS/0071-where-genesis-gets-its-source.md
|
||||
- 02-DECISIONS/0014-no-npm-workspace.md
|
||||
---
|
||||
|
||||
# The installation, in full
|
||||
|
||||
Every step from a machine with nothing to a mesh that maintains itself. Written out explicitly
|
||||
because it is the procedure everything else depends on, and because the parts that do not exist
|
||||
yet are easier to see beside the parts that do.
|
||||
|
||||
[`17-raising-a-mesh`](17-raising-a-mesh.md) argues *why* it is shaped this way. This says *what
|
||||
happens*, in order, with each step's name as the installer prints it.
|
||||
|
||||
## What must be true before anything starts
|
||||
|
||||
| | why |
|
||||
|---|---|
|
||||
| a container runtime | the foundation is containers, and the installer refuses without one |
|
||||
| the host binary, where the installer expects it | it is what the machine becomes |
|
||||
| a repository and a commit to build from | the installer carries a builder, not a controller, so it must be told what to make ([ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md)) |
|
||||
| a way out to the internet | the store, the broker and the registry are pulled from it |
|
||||
| a reachable git host, and a commit it serves | what is cloned is the trust anchor for everything this mesh will ever run ([ADR 0071](../../02-DECISIONS/0071-where-genesis-gets-its-source.md)) |
|
||||
| the address other machines will reach this one on | a token carries it verbatim; the installer refuses to guess |
|
||||
|
||||
## Phase one — a machine becomes a mesh of one
|
||||
|
||||
Twelve steps, run by one program, each safe to run again.
|
||||
|
||||
| # | step | what happens | true afterwards |
|
||||
|---|---|---|---|
|
||||
| 1 | `preflight` | everything above is checked | the machine is not half-changed by a missing prerequisite |
|
||||
| 2 | `load` | the carried builder image is loaded | the mesh has the one thing it cannot fetch — the thing that does the fetching |
|
||||
| 3 | `build` | the builder clones the named repository at the named commit and **builds the controller** | what will run is something this mesh made and can make again |
|
||||
| 4 | `bundle` | the foundation template is written out, with the built controller's id in place of the placeholder | the machine has a description of what it will become |
|
||||
| 5 | `apply` | store, broker, schemas, and a **temporary** controller are raised | a mesh of one exists and answers |
|
||||
| 6 | `verify` | the controller is asked, rather than assumed | it replies, and says it has no machines |
|
||||
| 7 | `enrol` | the machine joins the mesh it is itself running | the mesh has one node, and an agent runs on it |
|
||||
| 8 | `registry` | the mesh's own artifact store is installed | there is somewhere to keep what this mesh makes |
|
||||
| 9 | `publish` | the controller's image is pushed into it | the image has a digest something other than itself assigned |
|
||||
| 10 | `controller` | it is installed again, as an ordinary module pinned to that digest | what runs is a module like any other |
|
||||
| 11 | `retire` | the temporary controller is dropped | **the pivot is complete** — what raised the mesh is gone |
|
||||
| 12 | `builder` | the carried builder is published, installed as a module, and issued a broker account | the mesh can produce |
|
||||
|
||||
**Steps 8 to 11 are the pivot** ([ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md)). Before
|
||||
them the controller is something the installer put there; after them it is something the mesh
|
||||
holds a record of and can upgrade. The account in step 12 is issued *before* the machine is sent
|
||||
anything, because a builder that arrives without its credential starts, finds nothing it may read,
|
||||
and waits — which looks exactly like a builder with no work.
|
||||
|
||||
## Phase two — a mesh of one becomes a mesh that works
|
||||
|
||||
**Genesis ends with a mesh that runs, which is not the same as a mesh that works.** It has a control
|
||||
plane, a store, a broker, a registry and a builder. It holds no module graph, has no private
|
||||
network, no packet filter, and cannot resolve a name. Calling that "installed" is what let the
|
||||
catalogue be missing from a test for weeks without anything complaining.
|
||||
|
||||
| # | step | what happens | why it is here |
|
||||
|---|---|---|---|
|
||||
| 13 | the shared base is built | the toolchain and runtime every module with code of its own stands on | nothing else with code can be built until it exists |
|
||||
| 14 | a store module is built and run | a database **provider**, which the foundation's store is not | the foundation's store is the controller's own memory, and offers nothing to anything |
|
||||
| 15 | the catalogue is built and run | the module graph | without it the mesh cannot say what it holds, what a change reaches, or what must be rebuilt |
|
||||
| 16 | the catalogue asks for what it missed | the builds made before it existed are replayed | on a fresh mesh those are always the base, the store and the catalogue itself ([issue 050](../../04-ISSUES/050-the-catalogue-knows-nothing-built-before-it/00-report.md)) |
|
||||
| 17 | the controller is rebuilt from its own repository | and rolled out through the module path | the moment the mesh stops depending on the installer for anything |
|
||||
| 18 | `networking` is assigned **and the node placed** | a private network, and names | assigning installs the module; placing says where this machine is on it. Both, or the names file is written empty |
|
||||
| 19 | the packet filter is assigned | rules generated from what modules declared | until this, every rule the mesh computes has never been applied to anything |
|
||||
|
||||
## Phase three — machines arrive
|
||||
|
||||
Only now. A machine joining a mesh that cannot build anything proves that enrolment works, which
|
||||
was never the doubtful part.
|
||||
|
||||
| # | step | what happens |
|
||||
|---|---|---|
|
||||
| 20 | a node record is made and a token issued | one-time, carrying the broker's address and the fingerprint to pin |
|
||||
| 21 | the machine enrols and runs its agent | being heard from once is not an agent running; both are checked |
|
||||
| 22 | it is placed on the private network | or nothing can bind a consumer on it to a provider elsewhere |
|
||||
| 23 | modules are assigned to it | and it pulls what it needs from the mesh's registry |
|
||||
|
||||
## What is not yet true
|
||||
|
||||
Stated plainly, because a procedure that implies otherwise is worse than none.
|
||||
|
||||
**Steps 13 to 19 are not the installer's.** They are things somebody types. The installer ends at
|
||||
step 12, and everything that turns a running mesh into a working one is manual — which is why a
|
||||
test had to be written to find out they were missing.
|
||||
|
||||
**Step 23 does not work for a second machine.** It has no account on the registry
|
||||
([issue 042](../../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md)) and
|
||||
no reason to trust a registry serving plain HTTP over the network
|
||||
([issue 048](../../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)).
|
||||
Both are invisible on a mesh of one, where the registry is loopback.
|
||||
|
||||
**There is no private package registry**, and [ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md)
|
||||
assumes one: a module consumes its dependencies, the mesh's own shared library included, from it.
|
||||
At step 13 nothing has installed one, so the first build of the shared base resolves the SDK some
|
||||
other way — today by a git URL, which is
|
||||
[issue 053](../../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md).
|
||||
|
||||
**The installer does not install the host's unit, though one exists.** `mesh-host/packaging/` ships
|
||||
`nox-mesh-host.service` and two companions; the installer declines to place them because a unit file
|
||||
is a packaging decision. So an install that does nothing further leaves a machine whose containers
|
||||
come back after a reboot and whose agent does not — it runs the right things and can no longer be
|
||||
told anything.
|
||||
|
||||
**This is a gap in packaging, not in the mesh**, and the distinction matters: the lab's
|
||||
`--host-in-background` says in its own help that it does not survive a reboot, so a lab run failing
|
||||
this is the lab being honest rather than the mesh being broken. What is missing is the step that
|
||||
puts the shipped unit on the machine.
|
||||
|
||||
**The foundation's credentials are the installer's, not the template's.** The template still
|
||||
carries a fixed store password and the broker image's default administrator, because it is applied
|
||||
raw by beds that raise no mesh; the installer replaces both with values it makes once and keeps at
|
||||
the paths the store and broker modules declare, rewrites the produced bundle to use them, and
|
||||
writes that bundle at 0600. After enrolment and before the first secret is accepted it makes the
|
||||
operator key beside the bundle and gives the mesh its public half; the run ends with the
|
||||
operator-sealed export beside the key ([24](24-the-secrets-vault.md),
|
||||
[issue 071](../../04-ISSUES/071-the-foundation-is-raised-with-fixed-credentials/00-report.md)).
|
||||
What is not yet true: nothing rotates those two credentials afterwards, and the operator key is a
|
||||
file a person must carry off the machine — the installer says so and cannot check it.
|
||||
|
||||
**Nothing asserts a mesh was installed this way.** A claim that a machine was brought up by this
|
||||
procedure cannot be contradicted by anything afterwards.
|
||||
|
||||
## What a finished mesh holds
|
||||
|
||||
**Twelve, and after the pivot none of them is a specialty.** Every row is a module the mesh built,
|
||||
holds a version of, and can upgrade — which is the whole claim, and is not true today for the first
|
||||
two.
|
||||
|
||||
| # | module | provides | note |
|
||||
|---|---|---|---|
|
||||
| 1 | `postgres` | `postgres-database` | **the controller's own records and every module's.** One server, not two — it *claims* the mesh-scoped `mesh-store` seat, so a second is refused |
|
||||
| 2 | `lavinmq` | `amqp` | the broker every node dials, and what modules are granted vhosts on — *claims* `mesh-broker`, one per mesh |
|
||||
| 3 | `mesh-controller` | *claims* `mesh-controller` | decides what runs where |
|
||||
| 4 | `distribution` | `artifact-store` | what the mesh built, pinned by digest — the module is the software (Distribution), the provision is the job |
|
||||
| 5 | `builder` | — | turns source into artifacts |
|
||||
| 6 | `mesh-tools` | build inputs | the base everything with code compiles against. **Runs nowhere** |
|
||||
| 7 | `mesh-catalog` | the module graph | what is held, what a change reaches, what must be rebuilt |
|
||||
| 8 | `networking` | `private-network`, naming | requirements only — assigning it brings `mesh-wireguard` and `mesh-names` |
|
||||
| 9 | `dnsmasq` + one of `resolved-split-dns` / `resolv-conf` | `wildcard-resolution` | names that actually resolve, on top of `mesh-resolver`'s data |
|
||||
| 10 | `step-ca` | `acme-ca` | certificates for `.internal` |
|
||||
| 11 | `firewall` | *claims* `the-packet-filter` | rules generated from what modules declared |
|
||||
| 12 | `gitea` | `package-registry` | where a module's dependencies come from ([ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md)), and the git host |
|
||||
|
||||
### Why it is twelve and not thirteen
|
||||
|
||||
**The foundation's store and the `postgres` module are the same module.** They were two rows while the
|
||||
foundation was a different *kind* of thing: a store raised from a bundle cannot provide
|
||||
`postgres-database`, so anything wanting a database needed a second server. That is visible on any
|
||||
mesh built today — `mesh-store` and `postgres`, two containers, **the same image**.
|
||||
|
||||
The naming rule settles which name survives
|
||||
([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)):
|
||||
|
||||
> Where the consumer **speaks a protocol** — a database's wire protocol and query dialect — the
|
||||
> interface *is* the protocol. **"database" is not a capability; the protocol is.** Never false
|
||||
> genericity: a name must not promise a swap the contract cannot deliver.
|
||||
|
||||
So there is no `store` module. The controller is coupled to postgres — its own queries use
|
||||
`distinct on` and `on conflict`, which are not portable — and calling it `store` would advertise a
|
||||
swap that would fail the first time somebody tried it.
|
||||
|
||||
**The same applies to the broker**, with a different outcome: `amqp` *is* a protocol and more than
|
||||
one implementation speaks it, so `amqp` is a legitimate provision and `lavinmq` is one provider of
|
||||
it. The foundation's broker and the `lavinmq` module collapse the same way.
|
||||
|
||||
### What this costs, and it is the last specialty
|
||||
|
||||
Adopting the store and the broker as modules is [issue 051](../../04-ISSUES/051-the-mesh-cannot-update-what-it-depends-on/00-report.md),
|
||||
and it is the only part of this that has not been designed. The two hard parts:
|
||||
|
||||
- **upgrading a store the controller is reading from** — a rollout where the thing being replaced
|
||||
is the thing holding the record of the rollout
|
||||
- **upgrading a broker over the broker** — the instruction arrives on what it replaces, so the
|
||||
machine must finish without being able to report progress
|
||||
|
||||
Both are operations with windows, and a machine rebooting inside one is an operator's problem during
|
||||
an operation rather than a reason not to do it.
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-host
|
||||
- mesh-controller
|
||||
- mesh-catalog
|
||||
- mesh-lab
|
||||
updated: 2026-09-15
|
||||
decisions:
|
||||
- 02-DECISIONS/0067-genesis-is-a-pivot.md
|
||||
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
|
||||
- 02-DECISIONS/0075-two-stores-and-which-provides-what.md
|
||||
- 02-DECISIONS/0014-no-npm-workspace.md
|
||||
---
|
||||
|
||||
# The work ahead
|
||||
|
||||
Everything decided this cycle and not yet built, in the order its dependencies allow. Each phase
|
||||
ends at something provable on a running mesh, because a phase that ends at a claim is a phase that
|
||||
went missing without anything complaining.
|
||||
|
||||
## How this is built, and when it is run
|
||||
|
||||
**Implemented as code with unit tests, committed per change, and run in the lab ONCE the pieces that
|
||||
would change the outcome are in place.** The mistake this corrects: repeatedly running a 20-minute
|
||||
lab against a mesh mid-transformation, debugging paths the next phase deletes. The base-build hang,
|
||||
for instance, is almost certainly the SDK resolving from a git URL inside a docker build (issue
|
||||
053) — which Phase 2 removes. Debugging it on the current shape is debugging deprecated code.
|
||||
|
||||
So the lab run is the acceptance test at the end of the assembled work, not the tool for finding
|
||||
each bug. Where a fault can be reasoned out of the code path, it is — reading, not running.
|
||||
|
||||
## Phase 0 — folded in
|
||||
|
||||
The installer's own regressions (stale carried builder, a diagnostic on the parsed stdout) are
|
||||
fixed and committed. Whether it reaches a full green run is answered by the final lab run below,
|
||||
after the phases that change its build path are in — not before.
|
||||
|
||||
## Phase 1 — the protocol is one thing, and correct *(mostly done: the drift was dead types)*
|
||||
|
||||
**Why here.** The Go controller and the TypeScript SDK disagree about what a grant carries
|
||||
(`consumer` is the module in one, the node in the other). That is exercised by the installer's own
|
||||
provisioning — the catalogue's database — so it belongs before more is built on it.
|
||||
|
||||
- [x] 1.1 the drift was not live — inspection showed the wire agrees (contributions file; envelope
|
||||
required headers). The dead types that disagreed are removed, ADR 0074 and doc 19 corrected
|
||||
- [ ] 1.2 a conformance fixture for the two live cross-language contracts — the event envelope and
|
||||
the contributions file — checked in both suites, as **prevention** rather than repair
|
||||
- [ ] 1.3 (deferred) a full per-capability suite when a third language is actually added; not
|
||||
needed to keep two honest
|
||||
|
||||
**Done when.** A fixture pins the envelope and the contributions file, and a change to either side
|
||||
that breaks agreement fails a test rather than a mesh.
|
||||
|
||||
## Phase 2 — the private package registry, and the SDK in it
|
||||
|
||||
**Why here.** ADR 0014 says a module consumes its dependencies, the SDK included, from the private
|
||||
registry. Nothing installs one, so the SDK comes from a git URL (issue 053). Needs the installer
|
||||
(Phase 0) and gitea.
|
||||
|
||||
- [ ] 2.1 gitea provides `package-registry` in full — the endpoints, an account the builder may
|
||||
publish with
|
||||
- [ ] 2.2 the builder publishes the SDK there on build, by version
|
||||
- [ ] 2.3 modules consume it by version; the git URL and the sibling-path lock are gone
|
||||
- [ ] 2.4 a second language's SDK published the same way, proving the path is not TypeScript-only
|
||||
|
||||
**Done when.** A module builds against the SDK resolved from the mesh's own registry, and issue 053
|
||||
closes.
|
||||
|
||||
**Where it stands (2026-09-16).** The decision the bootstrap turned on is settled and recorded
|
||||
([ADR 0076](../../02-DECISIONS/0076-the-sdk-is-a-published-package.md)): the SDK is a published
|
||||
package, built on a public base image and published before the toolchain that consumes it; mesh-tools
|
||||
stays the thin toolchain base but `npm ci`s the SDK by version. On the code, the builder now resolves
|
||||
a package-registry credential — from a binding the mesh writes or from the environment for a hand-run
|
||||
or bootstrap build — and injects it into an image build as a buildkit secret, never a layer, so a
|
||||
token is not baked into the toolchain image. Unit-tested. Still ahead: gitea serving the registry in
|
||||
full with a provisioner that mints tokens (2.1), the SDK built and published by the mesh (2.2), the
|
||||
mesh-tools manifest flipped off the git URL to `npm ci` by version (2.3), and the genesis step that
|
||||
raises gitea and publishes the SDK before the base build. Those close together in one lab run.
|
||||
|
||||
## Phase 3 — nothing is special after installation
|
||||
|
||||
**Why last.** The hardest and riskiest, and it needs everything above: an installer that completes,
|
||||
a protocol that agrees, a registry to publish to. Issue 051.
|
||||
|
||||
- [x] 3.1 the store is adopted — raised at genesis, then held as the `postgres` module; one server,
|
||||
the controller's records and the database of each module that asks for one. *(Adopted in place: the module's
|
||||
server names the same container and the same pinned upstream image the foundation runs, so the
|
||||
applier reconciles it rather than raising a second postgres. Proven 22/22 in the one-node lab —
|
||||
the catalogue, a `postgres-database` consumer, gets its database from it. Follow-up: the store
|
||||
binds `0.0.0.0` from genesis so a mesh consumer can reach it, but the packet filter is
|
||||
installed later — a brief pre-filter window where `mesh-store` is open before `from: mesh`
|
||||
clamps it; bring the filter up earlier or bind narrower at genesis.)*
|
||||
- [x] 3.2 the broker is adopted — one `lavinmq`, the `/` vhost for the mesh bus, a vhost per
|
||||
consumer that requires `amqp`; the second server gone. *(Adopted in place like the store; the
|
||||
module names mesh-broker with the foundation's TLS spec, the provisioner runs host-networked
|
||||
as lavinmq's default guest, amqp-ping reaches it. Proven 22/22.)*
|
||||
- [x] 3.3 an upgrade of each, proven: a store the controller reads from, a broker over the
|
||||
broker, each with a stated window. *(Lab steps S1/S2: move each module's source, build, roll
|
||||
out, then a full push applies the server change and recreates the container. The store's
|
||||
window is a pool reconnect; the broker's is longer — recreating the bus the push travels over,
|
||||
so the mesh reconnects to the one that returns. Data survives on the named volumes.)*
|
||||
- [x] 3.4 `status` can say the foundation is behind its source, which today it cannot form.
|
||||
*(Given by the adoption: postgres/lavinmq are ordinary modules with a source now, so
|
||||
`module list`/`status` reports them behind or current like any other — the question could
|
||||
not form when they were bundle containers.)*
|
||||
|
||||
**Done when.** A mesh built from bare metal runs one postgres and one lavinmq, and can upgrade
|
||||
either — so the twelve-module floor has no specialty left in it.
|
||||
|
||||
## The through-line
|
||||
|
||||
Each phase leaves the mesh more able to describe and rebuild itself, and each ends at a run rather
|
||||
than a paragraph. The order is dependency, not preference: the installer carries everything, the
|
||||
protocol is what everything speaks, the registry is what everything is built from, and adoption is
|
||||
what makes the last two things ordinary.
|
||||
|
||||
## The one lab run
|
||||
|
||||
After Phases 1–3 are implemented and unit-tested and committed: rebuild the images, **verify each
|
||||
carries its change**, and run the one-node installer once. All 22 checks green, twice, is the
|
||||
acceptance test for the whole cycle — not a debugging loop, a proof that the assembled thing works.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user