jschoubben
345bbe0552
Retired in favour of the lab rather than repaired — that answers the first open question. The second finding is the one that generalises: "nothing runs it, and nothing reports that nothing runs it" is not a fact about that harness, it is a fact about any suite too expensive to run on every push. The replacement inherited the fault it was replacing. Records the three rules that now hold, and what the fix taught twice: the remedy rebuilt the symptom inside itself, and the code that counts results passed every test while reading nothing.
97 lines
5.1 KiB
Markdown
97 lines
5.1 KiB
Markdown
---
|
|
status: resolved
|
|
opened: 2026-08-22
|
|
located-in: [hal, mesh-lab]
|
|
fixed-by: mesh-lab — a run leaves a receipt, and the receipt says what it covered
|
|
amended-design: 03-DESIGN/01-to-be/01-end-to-end-testing.md
|
|
---
|
|
|
|
# 005 — The end-to-end pipeline harness has not built since the workspace was removed
|
|
|
|
## Symptom
|
|
|
|
The repository's end-to-end pipeline test harness depends on a workspace that no longer
|
|
exists. It has not been buildable since 2026-06-04. Nothing runs it, and nothing reports that
|
|
nothing runs it.
|
|
|
|
## Why this matters
|
|
|
|
The delivery pipeline is the mesh's most consequential machinery — every module reaches every
|
|
node through it — and its only end-to-end coverage has been silently dead for over two and a
|
|
half months.
|
|
|
|
That interval is not incidental. Several of the pipeline's most expensive defects landed
|
|
inside it: packaging that was never actually split from build, migrations and provisioning
|
|
scripts reading a source layout that no longer ships, selection files never packaged at all.
|
|
Whether this harness would have caught any of them is unknown — which is itself the point. The
|
|
coverage was assumed, not checked.
|
|
|
|
## Evidence
|
|
|
|
- The workspace was removed by pull request #240 on 2026-06-04
|
|
([ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md)).
|
|
- The harness has not built since that date.
|
|
- Recorded in the knowledge base as a standing entry, not as a fixed incident.
|
|
|
|
## Relationship to the lab
|
|
|
|
[`03-DESIGN/01-to-be/01-end-to-end-testing.md`](../../03-DESIGN/01-to-be/01-end-to-end-testing.md)
|
|
designs end-to-end testing on a lab mesh, which would replace this harness rather than repair
|
|
it. That is a reason to decide its fate deliberately, not a reason to leave it broken and
|
|
unmentioned: until the lab exists, this is the coverage the pipeline is presumed to have.
|
|
|
|
## Open questions
|
|
|
|
- Repair, or retire in favour of the lab? Leaving it in the repository unbuilt is the one
|
|
option that keeps the false impression of coverage.
|
|
- Was anything relying on it, or had it already stopped running before the workspace removal?
|
|
|
|
## Resolution
|
|
|
|
*2026-08-31.* **Retired in favour of the lab, and the reason it went unnoticed was fixed
|
|
separately from the harness itself.**
|
|
|
|
The old harness is not repaired. What replaced it is the end-to-end suite on a lab mesh, which
|
|
raises real machines and proves the pipeline against them. That answers the first open question.
|
|
|
|
The second finding is the one worth keeping. *Nothing runs it, and nothing reports that nothing
|
|
runs it* is not a fact about that harness — it is a fact about **any** suite too expensive to run
|
|
on every push, and the lab suite is exactly that: it needs a machine with a hypervisor, so it runs
|
|
when somebody remembers. **Remembering is not a mechanism**, and the replacement inherited the
|
|
fault it was replacing.
|
|
|
|
So three things now hold, each checked by a test that was confirmed to fail without it:
|
|
|
|
- **A run leaves a receipt** — when it ran, what passed, and the commit each repository was at.
|
|
Kept outside version control, because the question is *has this machine run it*, and a receipt in
|
|
git would be a claim about everybody's machine made by whoever committed last.
|
|
- **The receipt can be judged, and says why it does not count.** Old, failed, taken against code
|
|
the repositories have since moved past, or a run that never raised a machine — each reads
|
|
differently, and only the last of those is new. **A receipt that says nothing about something is
|
|
not a receipt that clears it.**
|
|
- **The artifacts are rebuilt by the run, not beside it.** The suite consumes three artifacts from
|
|
two repositories. They were rebuilt by hand, from memory, and a rename that needed two of them
|
|
got one — leaving a binary eleven hours old refusing a field the mesh had just renamed, found by
|
|
a full run. That step now lives in the repository rather than in a terminal history.
|
|
|
|
### What this issue taught twice
|
|
|
|
**The fix reintroduced the fault, in miniature, and the second time was caught by running it.**
|
|
|
|
The suite takes paths, so it can be pointed at one quick unit file — and the receipt from that run
|
|
was, at first, indistinguishable from a receipt for the real thing. A green record standing for a
|
|
run that raised no machines is this issue's own symptom, rebuilt inside its remedy. The receipt now
|
|
records what it ran, and a run that did not include the end-to-end file is not coverage.
|
|
|
|
Separately, the code that decides *no receipt rather than a guessed one* — the rule that keeps the
|
|
record meaning something — was first written where no test could reach it. Writing "0 failed"
|
|
because nothing said otherwise is how a green record comes to mean nothing.
|
|
|
|
And the counting itself **passed every test while reading nothing**: the test runner colours its
|
|
summary even into a pipe, so the anchored pattern never matched, and the fixtures it was checked
|
|
against were output that had been imagined rather than captured. **A fixture that agrees with the
|
|
mistake proves the mistake.** It is now checked against the runner's real bytes.
|
|
|
|
Each of these was found by running the thing, not by reading it — which is the same argument this
|
|
issue makes about the pipeline.
|