005 resolved: a suite that cannot run on every push says when it last ran

Retired in favour of the lab rather than repaired — that answers the
first open question. The second finding is the one that generalises:
"nothing runs it, and nothing reports that nothing runs it" is not a
fact about that harness, it is a fact about any suite too expensive to
run on every push. The replacement inherited the fault it was replacing.

Records the three rules that now hold, and what the fix taught twice:
the remedy rebuilt the symptom inside itself, and the code that counts
results passed every test while reading nothing.
This commit is contained in:
2026-08-31 15:02:26 +02:00
parent f3ffdae909
commit 345bbe0552
2 changed files with 86 additions and 6 deletions
+33 -2
View File
@@ -2,9 +2,8 @@
layer: to-be layer: to-be
status: in-progress status: in-progress
code: [mesh-lab] code: [mesh-lab]
updated: 2026-08-28 updated: 2026-08-31
decisions: decisions:
- 02-DECISIONS/0016-the-lab.md
- 02-DECISIONS/0016-the-lab.md - 02-DECISIONS/0016-the-lab.md
- 02-DECISIONS/0019-how-this-repository-works.md - 02-DECISIONS/0019-how-this-repository-works.md
--- ---
@@ -390,6 +389,38 @@ Everything a node itself does is real, because a node is a real machine.
--- ---
## A suite too expensive to run on every push says when it last ran
*Written 2026-08-31, from resolving [04-ISSUES/005](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md).*
This suite needs a machine with a hypervisor. It therefore cannot run on every push, and a suite
that does not run on every push runs **when somebody remembers**. Remembering is not a mechanism,
and the harness this one replaces proves it: it had not built for two and a half months, nothing
said so, and the coverage was assumed rather than checked.
**The danger is not that the suite breaks. It is that nobody notices it stopped running** — and
that danger belongs to *this* design, not to the harness it retired.
So three rules, each held by a test:
**A run leaves a receipt** — when, what passed, what it ran, and the commit each repository was
at. Kept **outside version control**: the question is *has this machine run it*, and a receipt in
git would be a claim about everybody's machine made by whoever committed last.
**A receipt says why it does not count.** Old, failed, taken against commits the repositories have
moved past, or a run that never raised a machine. Something can be asked, and answers non-zero.
**A receipt that says nothing about something is not a receipt that clears it** — including a
receipt written before it recorded a given fact, which claims nothing rather than everything.
**The run rebuilds what it tests.** The suite consumes artifacts from other repositories, and an
artifact rebuilt from memory is one rebuilt sometimes. A stale binary reporting success against
rules that have since changed is the same fault wearing different clothes.
**The general rule, which outlives this suite:** *silence and success must never look alike.*
It is the same rule the host follows about a service that does not exist
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) — absence must be distinguishable
from a failure to answer — applied to coverage instead of to a machine.
## Consequences ## Consequences
**Bringing a node into being is part of the framework.** A test creates its own nodes — one **Bringing a node into being is part of the framework.** A test creates its own nodes — one
@@ -1,9 +1,9 @@
--- ---
status: open status: resolved
opened: 2026-08-22 opened: 2026-08-22
located-in: [hal] located-in: [hal, mesh-lab]
fixed-by: fixed-by: mesh-lab — a run leaves a receipt, and the receipt says what it covered
amended-design: amended-design: 03-DESIGN/01-to-be/01-end-to-end-testing.md
--- ---
# 005 — The end-to-end pipeline harness has not built since the workspace was removed # 005 — The end-to-end pipeline harness has not built since the workspace was removed
@@ -45,3 +45,52 @@ unmentioned: until the lab exists, this is the coverage the pipeline is presumed
- Repair, or retire in favour of the lab? Leaving it in the repository unbuilt is the one - Repair, or retire in favour of the lab? Leaving it in the repository unbuilt is the one
option that keeps the false impression of coverage. option that keeps the false impression of coverage.
- Was anything relying on it, or had it already stopped running before the workspace removal? - Was anything relying on it, or had it already stopped running before the workspace removal?
## Resolution
*2026-08-31.* **Retired in favour of the lab, and the reason it went unnoticed was fixed
separately from the harness itself.**
The old harness is not repaired. What replaced it is the end-to-end suite on a lab mesh, which
raises real machines and proves the pipeline against them. That answers the first open question.
The second finding is the one worth keeping. *Nothing runs it, and nothing reports that nothing
runs it* is not a fact about that harness — it is a fact about **any** suite too expensive to run
on every push, and the lab suite is exactly that: it needs a machine with a hypervisor, so it runs
when somebody remembers. **Remembering is not a mechanism**, and the replacement inherited the
fault it was replacing.
So three things now hold, each checked by a test that was confirmed to fail without it:
- **A run leaves a receipt** — when it ran, what passed, and the commit each repository was at.
Kept outside version control, because the question is *has this machine run it*, and a receipt in
git would be a claim about everybody's machine made by whoever committed last.
- **The receipt can be judged, and says why it does not count.** Old, failed, taken against code
the repositories have since moved past, or a run that never raised a machine — each reads
differently, and only the last of those is new. **A receipt that says nothing about something is
not a receipt that clears it.**
- **The artifacts are rebuilt by the run, not beside it.** The suite consumes three artifacts from
two repositories. They were rebuilt by hand, from memory, and a rename that needed two of them
got one — leaving a binary eleven hours old refusing a field the mesh had just renamed, found by
a full run. That step now lives in the repository rather than in a terminal history.
### What this issue taught twice
**The fix reintroduced the fault, in miniature, and the second time was caught by running it.**
The suite takes paths, so it can be pointed at one quick unit file — and the receipt from that run
was, at first, indistinguishable from a receipt for the real thing. A green record standing for a
run that raised no machines is this issue's own symptom, rebuilt inside its remedy. The receipt now
records what it ran, and a run that did not include the end-to-end file is not coverage.
Separately, the code that decides *no receipt rather than a guessed one* — the rule that keeps the
record meaning something — was first written where no test could reach it. Writing "0 failed"
because nothing said otherwise is how a green record comes to mean nothing.
And the counting itself **passed every test while reading nothing**: the test runner colours its
summary even into a pipe, so the anchored pattern never matched, and the fixtures it was checked
against were output that had been imagined rather than captured. **A fixture that agrees with the
mistake proves the mistake.** It is now checked against the runner's real bytes.
Each of these was found by running the thing, not by reading it — which is the same argument this
issue makes about the pipeline.