jschoubben
b4607dfc03
Decided after measuring what renumbering actually costs: 96 references in code comments across two repositories, none of which would have failed to compile. They would have pointed at the wrong reasoning, which is worse than a broken link because nothing reports it. So a number identifies a record and never changes. It cannot also be a position -- a position moves when the set changes, and an identity that moves is not one. The reading order moves into an index generated from each record's `topic:`. Six topics, in the order somebody learns the system. The index is WRITTEN rather than only generated on demand, which reverses what this repository previously said. The reason it said otherwise is that a hand-written index drifts -- but a reader looking at the folder on a forge sees the folder, not a command, and the drift objection is answered by checking rather than by refusing to write one. That is §5's own rule: a rule states how it is checked. Two checks, both confirmed to bite. index.py fails when the written order no longer matches the records. records.py fails when a record has no topic or one nobody defined -- the quiet failure being a record that vanishes from the order rather than appearing in the wrong place.
93 lines
4.8 KiB
Markdown
93 lines
4.8 KiB
Markdown
---
|
||
topic: checking it
|
||
status: accepted
|
||
date: 2026-08-24
|
||
deciders: jochen
|
||
reconstructed: false
|
||
---
|
||
|
||
# 17. A test defends a decision
|
||
|
||
## Context
|
||
|
||
[`how-we-build.md`](../00-META/how-we-build.md) §5 already says that **if a document states a
|
||
rule about the mesh, it says how the rule is verified**, on the grounds that an unenforced rule
|
||
is indistinguishable from a wrong one and costs more, because people believe it.
|
||
|
||
That rule is applied to prose and to acceptance criteria. It has never been applied to
|
||
**decisions**, and it should be — a decision record states something that must be true, which
|
||
is the same kind of claim.
|
||
|
||
The gap was found by review. The lab reached 2,128 lines with 1,072 of them untested, and
|
||
**no stated rule was broken.** There is no testing posture in `how-we-build.md` at all: no
|
||
expectation, no gate, no definition of done. Every decision the lab embodies — the underlay
|
||
boundary, the closed address space, routers as scenery, waiting for usable rather than for a
|
||
call to return — was verified by hand, by running scenarios and reading output, and none of
|
||
that survives the terminal it was run in.
|
||
|
||
Which is the fault the mesh already has catalogued at scale: an end-to-end harness that has
|
||
not built since 2026-06-04, and nothing said so
|
||
([`04-ISSUES/005`](../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)). Coverage
|
||
assumed rather than checked.
|
||
|
||
## Considered options
|
||
|
||
1. **A coverage percentage.** Rejected. It measures how much code a test touched, not whether
|
||
anything important is defended, and it is satisfied by tests that assert nothing. A number
|
||
would have been met by testing the parser harder while the hypervisor integration stayed
|
||
unasserted.
|
||
2. **Test-driven development as a hard rule.** Rejected, and not because it is wrong in general.
|
||
Half the lab's implementation was discovery: that the hypervisor CLI reads a definition from
|
||
stdin and hangs, that it assigns a MAC without recording it, that a stock image's boot-time
|
||
networking flushes a static address. A test written first against undiscovered behaviour
|
||
asserts a guess.
|
||
3. **A test defends a decision.** Chosen.
|
||
|
||
## Decision
|
||
|
||
**Every decision record states something that must be true. A test asserts it.**
|
||
|
||
A decision with no test is a decision that will quietly stop being true, and nobody will find
|
||
out from a document. Concretely:
|
||
|
||
- Where a decision is about **structure or logic** — what a declaration may say, what is
|
||
refused, how a name is derived — the test is a unit test, and it is **written first**, because
|
||
the behaviour is knowable before the code.
|
||
- Where a decision is about **behaviour against a real system** — a hypervisor, a broker, a
|
||
daemon — the test runs against the real thing, and is written **alongside**, because the
|
||
behaviour is discovered rather than known.
|
||
- **Mocking the boundary is forbidden.** A test that fakes a hypervisor asserts that the fake
|
||
behaves as expected, which is the shape of test this whole effort exists to stop shipping.
|
||
- **The gate is blocking, and green is the definition of done.** A change that has not run its
|
||
tests is not finished, whatever its diff looks like.
|
||
|
||
A test names the decision it defends. Not as ceremony: it is what makes the pairing checkable,
|
||
so a decision without one can be *found* rather than noticed.
|
||
|
||
## Consequences
|
||
|
||
- The question *"which tests matter"* has an answer that is not a number. The decisions are the
|
||
list, and they are already written down.
|
||
- **A new decision costs a test.** That is the intended friction — a decision nobody will assert
|
||
is one worth reconsidering.
|
||
- Integration tests need real infrastructure and are slow. That cost is accepted: a fast test
|
||
suite that mocks the boundary would tell us nothing about the boundary, which is where every
|
||
interesting fault in this session actually was.
|
||
- Some decisions are not mechanically assertable — *the mesh brokers capabilities; nodes host;
|
||
agents think* is a shape, not a predicate. Those should say so in the record rather than being
|
||
quietly exempt, so the exemption is visible.
|
||
- Records 0001–0033 were made before this rule. They are not retroactively invalid, but each
|
||
should acquire a test or an explicit note that it cannot have one, and until then this rule
|
||
is aspirational for them — which is exactly the state §5 warns about, recorded rather than
|
||
hidden.
|
||
|
||
## References
|
||
|
||
- [`how-we-build.md`](../00-META/how-we-build.md) §5 — the rule this extends from prose to
|
||
decisions.
|
||
- [`04-ISSUES/005`](../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md) — coverage
|
||
assumed rather than checked, for two and a half months.
|
||
- The sibling HQ repository for the PAPA platform states the same boundary rule — the contract
|
||
is tested against the real system, mocking the client is forbidden, and a blocking gate is the
|
||
definition of done. This record adopts that posture and adds the decision pairing.
|