74 lines
6.0 KiB
Markdown
74 lines
6.0 KiB
Markdown
# The conformance kit's captured runner output
|
|
|
|
Every file here is the real output of a test runner, never written by hand (issue 1901: a fixture that agrees with the
|
|
mistake proves the mistake). The toolchains are the build nodes' own, as `mesh-build-go.build_go_toolchains` and
|
|
`mesh-build-node.build_node_toolchains` reported them on 2026-10-26: go1.27.3-X:nodwarf5 on every build node, node
|
|
v26.12.1 (npm 12.2.1) on shanks, novox and g14, and typescript 5.9.3 from mesh-build-node's toolchain/package.json.
|
|
To capture a case again, make its tiny project in a fresh directory as below and run the command shown; the kit's case
|
|
table (conformance.go) says what each failure must say.
|
|
|
|
## go/ — `go test -json`
|
|
|
|
- `passfirst.jsonl` and `subtests.jsonl` (2026-10-26, go1.27.3-X:nodwarf5, for issue 1901's thorough fix): in
|
|
`/tmp/1901k/go/<case>`, `GOTOOLCHAIN=local GOFLAGS=-mod=mod go test -json -count=1 ./... > ../<case>.jsonl 2>&1`.
|
|
- passfirst: module `example.com/passfirst`, `go 1.26`. Package `a` has `TestTheFirstPackagePasses` (empty). Package `b`
|
|
has `TestTheSecondPackagePassesFirst` (empty) and `TestTheSecondPackageFails`, which calls
|
|
`t.Logf("judged against the seat as it stands")` then `t.Fatalf("the seat has 2 holders, want 1")`: a passing package
|
|
and a passing test before the failing one.
|
|
- subtests: module `example.com/subtests`; `TestTheTable` runs `t.Run("passes", func(t *testing.T) {})` and
|
|
`t.Run("fails", func(t *testing.T) { t.Errorf("got 3 machines, want 4") })`: the parent fails only by its subtest.
|
|
- `subpanic.jsonl` (2026-10-26, go1.27.3-X:nodwarf5, review round 1 of issue 1901's fix), the same way: module
|
|
`example.com/subpanic`; `TestTable` runs `t.Run("passes", func(t *testing.T) {})` and `t.Run("panics", ...)`, which
|
|
writes to a nil map. go test -json files the subtest's panic under the parent `TestTable`.
|
|
- `logthenfatal.jsonl`, `panicking`, `timeout`, `deeptimeout`, `buildfail`, `exiting`, `killed`, `passthenfail`,
|
|
`mainexit`, `race`, `racemain` were captured for issue 1901's first and third meetings (2026-10-23 and 2026-10-25,
|
|
go1.27.3) in mesh-catalog's mesh-build-go (`testdata/captured/go-test-json-*.jsonl`, named there `go-test-json-pkg-<case>`
|
|
and `go-test-json-log-then-fatal`) and moved here unchanged. From that README: each is the standard output and error of
|
|
`go test -json -count=1 ./...` (the timeouts with `-timeout 2s`; race and racemain with `-race`, CGO_ENABLED=1) in a
|
|
throwaway one-file module that fails without a test's own assertion: panicking (a nil map write in TestPanics), timeout
|
|
(TestWaitsForever sleeps past the timeout), deeptimeout (two parallel tests and sixty blocked goroutines past the
|
|
timeout, so the goroutine dump pushes the panic line out of the last 4 KB), buildfail (a type error in a test file),
|
|
exiting (os.Exit(3) in TestExits), killed (TestKilled sends itself SIGKILL), passthenfail (TestMain panics in a
|
|
goroutine after m.Run passed), mainexit (TestMain exits 2 after m.Run passed: go test prints "PASS" and then
|
|
"FAIL <pkg>"), race (TestRacesOnTheSeat writes a package variable from itself and a goroutine), racemain (TestMain does
|
|
so after m.Run passed). logthenfatal is a test `TestThree` in package `x` that logs a line with t.Logf and then fails
|
|
with a multi-line t.Fatalf naming three subscriptions.
|
|
|
|
## node/ — `npm test` running `node --test` under the TAP reporter
|
|
|
|
Captured 2026-10-26 with node v26.12.1 and npm 12.2.1 (`/usr/bin/node`, `/usr/bin/npm`), in `/tmp/1901k/node/<case>`, the
|
|
way mesh-build-node runs a test script:
|
|
|
|
PATH=/usr/bin:$PATH NODE_OPTIONS="--test-reporter=tap --test-reporter-destination=/tmp/1901k/node/<case>.tap" \
|
|
npm test > ../<case>.stdout 2> ../<case>.stderr
|
|
|
|
`<case>.tap` is the report file, `<case>.console` the stdout followed by the stderr. Each project's package.json is
|
|
`{"name":"<case>","private":true,"type":"module","scripts":{"test":"node --test --experimental-strip-types 'test/*.test.ts'"}}`
|
|
(the script every TypeScript package of the catalogue and the SDK runs), with these test files:
|
|
|
|
- two-files: `test/a.test.ts` holds `test('passes in the first file', () => {})`; `test/b.test.ts` holds
|
|
`test('passes in the second file first', () => {})` and `test('fails in the second file', ...)`, which logs
|
|
`judged against the seat as it stands` and asserts `assert.strictEqual(1 + 1, 3, 'one plus one is not three')`.
|
|
- suite: `test/a.test.ts` has `describe('the seat', ...)` with `it('passes inside', () => {})` and `it('fails inside', ...)`
|
|
asserting `assert.deepStrictEqual({ holders: 2 }, { holders: 1 })`.
|
|
- crash: `test/a.test.ts` holds `test('passes before the crash', () => {})`; `test/b.test.ts` holds
|
|
`test('throws after it returned', ...)`, which throws `new Error('the bus closed under the test')` in a `setTimeout` of
|
|
1 ms, and `test('waits after the throw', ...)`, which waits 50 ms: node fails the file with an uncaught exception.
|
|
- logcrash (review round 1): `test/a.test.ts` holds `test('passes and logs', ...)`, which logs
|
|
`retrying after Error: the store was not ready yet`; `test/b.test.ts` is crash's. A passing test's console line
|
|
holding "Error" comes before the crash's own.
|
|
- timeout: `test/a.test.ts` holds `test('passes before the wait', () => {})` and
|
|
`test('waits for an answer that never comes', { timeout: 200 }, async () => { await new Promise(() => {}); })`.
|
|
- syntax: `test/a.test.ts` holds `test('passes in the first file', () => {})`; `test/b.test.ts` is
|
|
`import test from 'node:test';` and `test('never loads', () => {` (cut off): the file does not load.
|
|
|
|
## tsc/ — the type-check
|
|
|
|
`typeerror.console` (2026-10-26, typescript 5.9.3 under node v26.12.1): in `/tmp/1901k/tsc/typeerror`, `ok.ts` is
|
|
`export const holders: number = 1;` and `seat.ts` is `import { holders } from './ok.js'; export const name: string = holders;`;
|
|
the command is mesh-build-node's type-check line without a tsconfig.json:
|
|
|
|
node <typescript>/bin/tsc --noEmit --module NodeNext --moduleResolution NodeNext --target ES2022 --rootDir . ok.ts seat.ts
|
|
|
|
and the file is its stdout followed by its stderr.
|