ADR 0237, to-be 45 Phase 5: a change is judged against the mesh that runs before it merges

The operator approved Phase 5. Decides what the design left open: the build seat runs the
merge check, the facts live in the artifact store, the merge gate composes the mesh as it is
and with the change and judges only what the change adds, a replay lives where its incident
is, and the controller's tests run a bus of their own at the mesh's release. A core issue now
resolves only with a replay or a stated reason, checked by cycle.py.
This commit is contained in:
jochen
2026-10-06 21:10:54 +02:00
parent be0754ec7f
commit d245df7dea
11 changed files with 244 additions and 3 deletions
+4 -1
View File
@@ -56,4 +56,7 @@ a to-be design names a decision, an in-progress/implemented design names its own
located/fixed issue names its owner, a fixed/resolved issue says what fixed it, a graduated
research overview says what it became, and no two issue records share a number (issue 155 — the
number is how a record is cited, and `main` lags every open pull request, so two people reading it
allocate the same one). `python3 00-META/checks/cycle.py`
allocate the same one). And a core issue — one opened from 2026-10-07 whose `located-in` names a core
repository — resolves only with `replay:` (an id in mesh-lab's replays register) or `replay-none:` saying
why none is possible ([ADR 0237](../../02-DECISIONS/0237-a-change-is-judged-against-the-mesh-that-runs-before-it-merges-on-the-build-seat.md));
it failed on a resolved core issue carrying neither before it passed. `python3 00-META/checks/cycle.py`
+25
View File
@@ -16,6 +16,11 @@ What is enforced:
once `resolved`, `fixed-by:` says what fixed it (prose counts --
"nothing, the capability existed" is an answer). And no two records share a
number -- the number is how a record is cited.
replays a core issue -- one opened from 2026-10-07 whose `located-in:` names a core repository --
resolves only with `replay:` (the replay's id in mesh-lab's replays register) or
`replay-none:` (why no replay is possible): ADR 0237, to-be 45 §9. A replay is what
proves a fix fails before and passes after; without one, a fixed class comes back
through a different door (research 031: four such chains in one week).
research a known `status:`; a `graduated` overview says what it `became:`, and every
target it names exists.
decisions every accepted record is REACHABLE from the cycle: cited by a design doc's
@@ -41,6 +46,16 @@ DESIGN_STATUSES = {"proposed", "designed", "in-progress", "implemented", "abando
ISSUE_STATUSES = {"open", "diagnosing", "located", "resolved", "wontfix"}
RESEARCH_STATUSES = {"active", "graduated", "abandoned"}
# The core, as to-be 45 names it, by where its code lives: the controller, the node-engine, the node
# tools and the console, the SDK's loops, and the catalogue's bus, forge (the announcer of merges) and
# build agent. A `located-in:` entry naming any of these makes an issue a core issue.
CORE = re.compile(r"\b(mesh-controller|mesh-host|mesh-tools|mesh-sdk|node-tools|"
r"mesh-catalog\s+modules/(nats|gitea|build-agent))\b")
# The day the rule began (ADR 0237): issues opened before it are not held to it.
REPLAYS_FROM = "2026-10-07"
REPLAY_ID = re.compile(r"^R\d+\b")
def rel(path):
return os.path.relpath(path, ROOT)
@@ -159,6 +174,16 @@ def main():
bad(path, "status %s but located-in is empty" % status)
if status == "resolved" and not listy(front, "fixed-by"):
bad(path, "status %s but fixed-by says nothing" % status)
# A core issue resolves with its replay, or with why none is possible (ADR 0237).
replay = " ".join(listy(front, "replay"))
if replay and not REPLAY_ID.match(replay):
bad(path, "replay: %r names no replay -- an id from mesh-lab's replays register, R<issue>" % replay)
core = any(CORE.search(entry) for entry in listy(front, "located-in"))
opened = str(front.get("opened") or "")
if (status == "resolved" and core and opened >= REPLAYS_FROM and not replay
and not " ".join(listy(front, "replay-none")).strip()):
bad(path, "a core issue resolved with no replay: name it in `replay:` (mesh-lab replays "
"register.go) or say in `replay-none:` why none is possible (ADR 0237)")
# ---- research ----------------------------------------------------------------------
for path in sorted(glob.glob(os.path.join(ROOT, "01-RESEARCH", "*", "00-overview.md"))):
+5 -1
View File
@@ -42,7 +42,11 @@ incident someone must **clear**.
2. Investigate in `01-diagnosis.md` in the same folder — the trail, dated, including what was
ruled out. Move `status:` to `diagnosing`, then `located` once the owner is known.
3. Resolve. Set `status: resolved`, fill `fixed-by:`, and if the root cause was a design gap,
run playbook [02](02-graduation.md) and fill `amended-design:`.
run playbook [02](02-graduation.md) and fill `amended-design:`. **A core issue** — its `located-in`
names the controller, the node-engine, the node tools, the SDK, or the catalogue's bus, forge or build
agent — resolves with `replay:`, the id of its replay in mesh-lab's replays register, proved to fail on
the commit before the fix and pass on it, or with `replay-none:` saying why none is possible
([ADR 0237](../../02-DECISIONS/0237-a-change-is-judged-against-the-mesh-that-runs-before-it-merges-on-the-build-seat.md); `cycle.py` checks it).
## Rules
@@ -0,0 +1,162 @@
---
topic: the mesh
status: accepted
date: 2026-10-06
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
---
# 237. A change is judged against the mesh that runs, before it merges, on the build seat
## Context
**The operator approved starting Phase 5** of [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md)
on 2026-10-06: the facts snapshot, the merge gate, versions tested as run, and the replays of
[ADR 0227](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) rule 9. The
design says what they are; building them left decisions it does not make — where a check runs on a mesh
that has no CI, what a snapshot must hold for a check to compose a machine faithfully, where a replay lives
when its incident is in one repository's logic and when it is in what the mesh runs, and how a test suite
that failed in parallel can be one a gate is allowed to read.
The evidence is research 031's class (h), *checks that pass in CI and fail live*, and the incidents of the
window since:
- **236**: a manifest passed `module check` and was refused whole by the node-engine on the first machine
it was assigned to. **263**: a real machine's name made a consumer's identity 26 characters against a
bound of 20, and the provider's whole machine could not be pushed. **262**: an answer glibc forgave was
final to musl. Each check was right about the world it was given; none was given the mesh's.
- **278**: a file of a module no machine held read as shared code, and a merge rebuilt 103 modules. Its
width could have been read before the merge; nobody was shown it.
- **The controller's suite failed when its packages ran in parallel** against one shared bus, and one test
hung under the race detector: the live tests assert, read and remove the mesh's own objects by their
fixed names, so two packages at once were one deleting what the other read. A red suite read as noise.
- **The bus tests ran a release the mesh did not run** (2.10's instructions in the tests' own comments
after the bus had moved to 2.11), and the 2.10 release that skipped messages (266) had been tested by
nobody's tests.
## Options weighed
**Where a check runs.**
1. *An outside CI.* Rejected: a second system to run, watch and keep in step, with no access to the mesh's
facts, its artifact store or its toolchains — the very things a check must be fed.
2. *The live controller composes the change.* Rejected as the gate: a change to the controller is judged by
the controller it changes, so the running one cannot judge it; and a composition run inside the serving
controller is load and risk on the control node for every pull request.
3. **The build seat, as one more kind of work on its queue** — chosen. The build machine already holds the
repositories, a container runtime, the artifact store and the mesh's Go toolchain; the controller already
asks it for work and watches every ask (S6).
**Where the facts are kept.** A key-value bucket on the bus would need a grant for every reader and a bus
that carries a document of a megabyte. **The artifact store, under `facts:latest`** — the design's choice —
is read by any machine of the mesh over plain HTTP, keeps one version, and is where the build seat already
reads and writes.
**Where a replay lives.** All in mesh-lab would put a test of the controller's planning in a repository that
cannot import it. All in their own repositories would leave the bus's release and the resolver's answers —
what the mesh *runs*, not what it wrote — with no home. **Both, by kind** — chosen.
**What the tests' bus is.** A shared bus per run needs serialised packages and still leaks between tests.
**A server per test, linked in at the release the mesh runs** — chosen.
## Decision
1. **The facts snapshot.** The controller (lease holder) composes it every ten minutes and keeps it in the
artifact store as `facts:latest` when its content moved or the one kept is a day old; the manifest it
replaced is deleted by digest, so the nightly collector takes it and the store keeps one. It holds every
machine — under a stable pseudonym of the same length as its name — with its roles in words, system,
C library, architecture, builds, reported capabilities, assignments, pins, settings and the names (never
the values) of secrets a person gave; every seat and holder; every module the mesh holds, as its manifest,
with its source and build edges; the bus's, store's and node-engine's versions as they run; and how each
machine's declaration composes today. No secret, no address: a key or value reading as a secret is
withheld, an address becomes one from a documentation range, and machine, site, account and domain
names are replaced wherever they appear. `facts`, `facts compose`, `facts export` read and write it by
hand. **S14** raises `facts-stale` past two days.
2. **The merge gate is the controller's `merge-gate`.** It raises two throwaway stores from the snapshot
through the controller's own records — the mesh as it is, and with the change's manifests in place of the
mesh's — composes every machine twice in each (the first time making the credentials a push makes), and
runs the node-engine's validator over each body. **What composes in the first and not in the second is
the change's**, named by the machine's roles and the module; what was already broken is said and fails
nothing. It also fails: a manifest the judging controller cannot read; a consumer the change leaves out
of a grant or a credential it leaves bound elsewhere; a module a machine runs removed from its source;
two definitions of one module name; a stored setting the changed definition cannot keep; and a module no
machine runs yet that the node-engine would refuse on the first machine that could run it (tried there
and taken back). It **warns** when a merge would rebuild more than twelve modules, and says when shared
code is why — the width read with the same rule the merge handler uses, the changed directories that
hold a definition read from the tree as the forge's announcer reads them.
3. **A catalogue change is judged by the controller the mesh runs** (version skew caught: a field only a
newer controller reads fails the pull request, not the registration after it); while the running one
predates the gate, by the controller's main, said. A controller change is judged by itself. A node-engine
change is judged by the running controller built with the change's validator in place of the one it
vendors.
4. **The check runs on the build seat.** The forge's announcer — the one that announces merges — announces
each new head of an open pull request (`pull.updated`) and marks it pending; the controller, for a
repository it builds a module from into that branch, asks the build seat a check: the head, and beside it
the controller the mesh runs and its main, the catalogue, the node-engine it runs, and mesh-lab. The
builder reads the snapshot, raises a throwaway store and bus **of the versions the snapshot says run**,
builds the judge, and runs the repository's own `merge-check.sh` — or the gate alone for a repository
with none — **in the mesh's Go toolchain, in a container of its own with no container runtime socket**:
a pull request is code nobody has approved yet. Then mesh-lab's replays from its main — reviewed code —
with the socket, against that bus and the change's own catalogue. The verdict is pass, warning, fail, or
**error, never read as a pass**, said by the controller as `checked`; the forge's holder sets it as the
head commit's status `mesh/merge-gate` and, when it is not a pass, comments with the check's own account.
Nothing a check does is recorded or registered. Whether the status is required to merge is the
operator's setting on the forge.
5. **A replay lives where its incident is.** One of a component's logic is a test in that repository,
named `TestReplay<issue>`, written only with what the component had before the fix, so it can be laid
over the older commit. One of what the mesh runs — the bus server's release, the resolver the catalogue
configures under musl and glibc — is in mesh-lab's `replays/`. mesh-lab's `register.go` names every
replay with its issue and fix, and `replays/cmd/prove` runs each on the commit before its fix (it must
fail, or, for a check that did not exist, not build) and on its fix (it must pass).
6. **A core issue resolves with its replay or a stated reason.** An issue opened from 2026-10-07 whose
`located-in` names a core repository — mesh-controller, mesh-host, mesh-tools, mesh-sdk, or the
catalogue's bus, forge or build-agent module — cannot be `resolved` without `replay:` (a register id) or
`replay-none:` (why none is possible). The issues the replays were written from carry theirs.
7. **The tests' bus is a server of their own, of the release the mesh runs.** The controller's suite links
the bus server in at the version go.mod pins and starts one per test; a test holds that pin to the
catalogue's bus image and, given a snapshot, to the release the mesh runs. The suite runs its packages in
parallel and under the race detector; a test reading timing, not state, is rewritten to read state. A
person may still point a run at a bus of their own with `MESH_TEST_NATS_EXTERNAL=1`.
## Consequences
- **A pull request to the core or the catalogue waits minutes for its verdict**, and the controller's
merge check runs the whole suite. That is the cost ADR 0227 accepted against the hours each of 202, 236
and 263 cost.
- **The gate is as faithful as the snapshot.** What the snapshot does not carry — a machine's adopted
state's detail, its tunnels, ports chosen by hand — a machine may compose differently in the gate than on
the mesh; then the gate says the machine composes on the mesh and not as raised, and judges it by what the
change adds. A gap that hides a real failure is found by the live probe D1, which stays.
- **The build agent pulls the toolchain, store and bus images for every check**, and mesh-lab's replays
pull the resolver and two C libraries' images from the public registry.
- **A bus upgrade is a pin moved in two places**: the catalogue's image and the controller's go.mod, which
a test holds equal. That is the rule, not a cost of it.
- **The forge's holder stays TypeScript** for this change; porting it to Go is its own piece of work.
## How it is checked
| Rule | Checked by |
|---|---|
| the snapshot carries no secret, address or name, and every machine's length | the controller's test raising a mesh with secrets, settings holding a password, an address and a machine's name, a credential made, and asserting none is in the snapshot; the scrubber's own tests |
| the snapshot is kept, one version, and said when stale | the artifact store's live test against the store's registry (put, read back by tag, the replaced one collected); S14's suppression in the generated signals test |
| the gate fails 236, 263, version skew, a removal, and says 278's width | the controller's gate tests, one per incident, against a raised mesh |
| a check runs the repository's script beside the mesh's versions, leaves nothing, and is never a pass when it cannot run | the builder's live tests against a container runtime and a registry |
| the verdict reaches the pull request, an error never as a success | the forge module's tests |
| 236, 262, 263, 266 (and 273) fail before their fix and pass after | `replays/cmd/prove` in mesh-lab |
| a core issue resolves only with a replay or a reason | `00-META/checks/cycle.py` |
| the tests' bus is the mesh's release, and the suite is deterministic | `internal/testbus`'s tests; the suite run three times in parallel under the race detector |
## References
- [ADR 0227](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) rule 9,
[to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §9 and Phase 5.
- [ADR 0225](0225-a-consumers-identity-is-bounded-by-the-provision-it-requires.md),
[ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md),
[ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md).
- Issues [236](../04-ISSUES/236-the-catalogue-check-passes-a-manifest-the-host-refuses/00-report.md),
[262](../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md),
[263](../04-ISSUES/263-every-consumer-pays-for-the-tightest-backends-name-limit/00-report.md),
[266](../04-ISSUES/266-a-merge-on-the-bus-was-never-handed-to-the-controller/00-report.md),
[273](../04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md),
[278](../04-ISSUES/278-a-module-held-by-no-machine-was-read-as-shared-code/00-report.md).
+1
View File
@@ -205,6 +205,7 @@ python3 00-META/checks/index.py fail if stale
- **0231** — [A healer acts on what observation raised, and only observation says it worked](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)
- **0234** — [The mesh holds a conversation with its operator, over channels that are seats, and an answer that performs an action is authorised by the controller](0234-the-mesh-holds-a-conversation-with-its-operator.md)
- **0236** — [A build is judged on its first machine and put back by something other than itself, and so it rolls out unattended](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)
- **0237** — [A change is judged against the mesh that runs, before it merges, on the build seat](0237-a-change-is-judged-against-the-mesh-that-runs-before-it-merges-on-the-build-seat.md)
### Its tiers, from the bottom up
@@ -4,6 +4,7 @@ status: in-progress
code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
updated: 2026-10-06
decisions:
- 02-DECISIONS/0237-a-change-is-judged-against-the-mesh-that-runs-before-it-merges-on-the-build-seat.md
- 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md
- 02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md
- 02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md
@@ -72,6 +73,8 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
| stream definitions and bus permissions | controller | the bus | — |
| builds and their outcomes | the build seat's holder | its own state | the controller asks |
| a merge announced | **one** announcer per forge (the hook, or the poll when the hook is absent — never both) | the bus | — |
| a pull request's head announced | the forge's announcer, the one that announces its merges | the bus | the controller asks the build seat to check it |
| a pull request's merge check | the build seat's holder that ran it, said as the controller's `checked` | the bus | the forge's holder sets it as the pull request's status |
| a provider's standing | the provider | the provider's events | the controller keeps the newest word as a condition |
| a consumer's retirement: active, retired (when, why), deleted | the provider | its own backend's mark; said on `provisioner.retirement` | the controller asks the provider's tools; a person approves, rejects and deletes through the controller's verbs |
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
@@ -191,7 +194,7 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
| S11 | node tools heartbeat | node tools | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — |
| S14 | facts snapshot exported | controller | daily | 2 days (Phase 5) | `facts-stale` | warning | — |
| S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | `facts-stale` | warning | — |
| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — |
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
@@ -528,6 +531,32 @@ A new core issue resolves with its replay added, or with a stated reason none is
mesh stays the test bed ([ADR 0149](../../02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md)): a
replay covers what must not be done to it on purpose, and every rule keeps a live check.
**As built** ([ADR 0237](../../02-DECISIONS/0237-a-change-is-judged-against-the-mesh-that-runs-before-it-merges-on-the-build-seat.md)):
- **The snapshot** is composed every ten minutes and kept as `facts:latest` when it moved or is a day
old; the one it replaced is let go of, so the store keeps one. It carries, beyond the list above, each
machine's roles in words, its reported capabilities, pins, settings, the names of secrets a person gave,
and how its declaration composes today; every module the mesh holds as its manifest, with its source
and build edges. `facts`, `facts compose`, `facts export`.
- **The gate is the controller's `merge-gate`**: the mesh as the snapshot says it is and the mesh with the
change, each raised in a throwaway store through the controller's own records and every machine composed
twice and validated; what composes in the first and not the second is the change's. It also fails a
manifest the judging controller cannot read (version skew), a consumer left out of a grant, a module
removed while a machine runs it, two definitions of one name, a stored setting the change cannot keep,
and a new module the node-engine would refuse on the first machine that could run it; it warns when a
merge would rebuild more than twelve modules, saying when shared code is why (issue 278).
- **It runs on the build seat**, asked by the controller when the forge's announcer says a pull request's
head moved: the repository's own `merge-check.sh` in the mesh's Go toolchain with no container runtime
socket, beside a throwaway store and bus of the versions the snapshot says run, then mesh-lab's replays.
The verdict is the head commit's status `mesh/merge-gate`, an error never a pass. A catalogue change is
judged by the controller the mesh runs; a node-engine change by it built with the change's validator.
- **Replays live where their incident is**: a component's logic in its repository as `TestReplay<issue>`,
what the mesh runs (the bus's release, the resolver under musl and glibc) in mesh-lab's `replays/`, all
named in its register and proved by `replays/cmd/prove`. A core issue opened from 2026-10-07 resolves only
with `replay:` or `replay-none:` (`cycle.py`).
- **The controller's tests run a bus of their own per test**, linked in at the release the mesh runs and held
to it by a test, so the suite runs in parallel and under the race detector.
---
## 10. The phases
@@ -661,6 +690,18 @@ a build whose migration cannot be undone; the next three core rollouts' verdicts
**Done when:** the replays of 236, 262, 263 and 266 fail on the commit before their fix and pass after;
a new core issue cannot resolve without a replay or a stated reason.
**Built 2026-10-06** ([ADR 0237](../../02-DECISIONS/0237-a-change-is-judged-against-the-mesh-that-runs-before-it-merges-on-the-build-seat.md),
the operator's "start Phase 5"), on branches, not yet merged: in mesh-controller the facts snapshot and
S14, `merge-gate`, the check asked of and run by the build seat, `merge-check.sh`, the replays of 263 and 273,
and a bus per test at the mesh's release; in mesh-catalog the forge's announcer of pull requests' heads and
the verdict as their status, and `merge-check.sh`; in mesh-host `merge-check.sh` and the grants in the
installer's user list; in mesh-lab the replays of 262 and 266, the register and the prover; in hq the
replay rule in `cycle.py`. The prover ran the replays of **236, 262, 263, 266 and 273: each fails on the
commit before its fix** (236's check did not exist there) **and passes on it**. **Not yet:** R4, R5 and the
rest of the window's incidents as replays; the merge gate's first runs against the live mesh's snapshot;
the status required to merge, which is the operator's setting on the forge; mesh-tools' and mesh-sdk's
`merge-check.sh`.
## What is not decided here
- The bus as a cluster of three, to upgrade it live.
@@ -3,6 +3,7 @@ status: open
opened: 2026-10-04
located-in: []
fixed-by:
replay: R236
amended-design:
---
@@ -3,6 +3,7 @@ status: resolved
opened: 2026-10-05
located-in: [mesh-catalog]
fixed-by: novox/mesh-catalog#71 (20603b6), novox/mesh-controller#65 (df9231c), novox/mesh-catalog#73 (b4b86c1)
replay: R262
amended-design:
---
@@ -3,6 +3,7 @@ status: located
opened: 2026-10-06
located-in: [mesh-controller internal/catalogue, mesh-controller cmd/mesh-controller, mesh-catalog modules]
fixed-by:
replay: R263
amended-design:
---
@@ -3,6 +3,7 @@ status: located
opened: 2026-10-06
located-in: [mesh-catalog, mesh-controller]
fixed-by:
replay: R266
amended-design:
---
@@ -3,6 +3,7 @@ status: located
opened: 2026-10-06
located-in: [mesh-controller]
fixed-by: novox/mesh-controller PR #86
replay: R273
amended-design:
---