Compare commits

...
Author SHA1 Message Date
mesh-tools b1e610b213 Merge pull request 'Build modules end the steps a previous process left running, and serve build_<lang>_step_end (issue 1890)' (#496) from fix/1890-left-running-steps into main 2026-10-26 12:57:34 +00:00
jschoubben 0134697f97 Build modules: the round ends asks past the step bound; pin the sdk head (issue 1890)
mesh/delivery merged: source.main moved to novox/mesh-catalog@b1e610b2135a
mesh/delivery-group group fix/1890-left-running-steps merged: every member is merged
mesh/check being carried to 0134697f9739 on b1e610b2135a | from a4f5df797b68
2026-10-26 13:52:53 +01:00
mesh-tools a4f5df797b Merge pull request 'gitea: the tools' forge account reads every repository (the operator, 2026-10-26)' (#495) from fix/forge-account-reads-all into main 2026-10-26 12:46:37 +00:00
jschoubben ac2c56f2a9 Build modules: steps end by compare-and-set; pin the sdk head with the review's fixes (issue 1890) 2026-10-26 13:44:00 +01:00
jschoubben b4a7bb9dea gitea: say that a repository made private later is granted only at the next start or by hand, and let one unreadable permission refuse its repository, not the whole pass
mesh/delivery merged: source.main moved to novox/mesh-catalog@a4f5df797b68
mesh/check being carried to b4a7bb9dea50 on a4f5df797b68 | from 879600202a09
2026-10-26 13:43:46 +01:00
jschoubben 5fe5f26e7e Build modules: a restart ends the steps it left running, a step shows what it is, build_*_step_end ends one (issue 1890) 2026-10-26 13:27:46 +01:00
jschoubben 0e7e220d49 gitea: give the tools' account read on every repository, at the start and on the forge's events, so a user's private repository no longer answers its tools 404 2026-10-26 13:22:08 +01:00
mesh-tools 879600202a Merge pull request 'Vault: adopt the hashes of bus accounts it holds no value for, handed sealed by the bus node (issue 1945, ADR 0550 clause 4)' (#494) from feat/1945-user-list-switch into main 2026-10-26 12:21:47 +00:00
mesh-tools 565d0a848a Merge pull request 'Gate: merge turns, a merge waits for a running check of its repository, merges on a carried verdict with no second ask (ADR 0548, issue 568)' (#493) from feat/548-one-check into main 2026-10-26 12:01:28 +00:00
mesh-tools 71e094b2de Merge pull request 'mesh-build-go: count skipped tests per package, so the checker keeps no pass that skipped (issue 890)' (#492) from fix/checker-skips-not-kept into main 2026-10-26 12:01:13 +00:00
mesh-tools f96704fa0f Merge pull request 'mesh-build-node tests: never install a toolchain from the registry in the integration level; skip as NOT RUN when none is installed (issue 890)' (#491) from fix/build-node-tests-offline into main 2026-10-26 11:34:17 +00:00
mesh-tools 86fdf74a0a Merge pull request 'Desk channel: keep what was shown in memory while the store does not answer, so a clearing replaces the urgent notification (issue 384)' (#490) from fix/desk-clear-lost into main 2026-10-26 11:33:47 +00:00
mesh-tools 843281c0e5 Merge pull request 'Knowledge base: withdraw a clause's 'changed by' mark once no accepted record changes it (issue 466)' (#489) from fix/kb-stale-marks into main 2026-10-26 11:33:34 +00:00
jschoubben e9279ce3c2 gate: find an approved merge by its ask, not by which member's run was seen first, so a group's turns are written before its first member is handed at every look (ADR 0548 §3; issue 568, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-catalog@565d0a848a85
mesh/delivery-group group feat/548-one-check merged: every member is merged
mesh/check checking e9279ce3c209 on 565d0a848a85 | novox/mesh-catalog#493 in group feat/548-one-check
2026-10-26 12:32:46 +01:00
jschoubben c9414b1c98 Vault: pin mesh-sdk at the switch group's head c2a410bb, one sdk for the adoption and the switch (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-catalog@879600202a09
mesh/delivery-group group feat/1945-user-list-switch merged: every member is merged
mesh/check checking c9414b1c9856 on 879600202a09 | novox/mesh-catalog#494 in group feat/1945-user-list-switch
2026-10-26 12:31:56 +01:00
jschoubben ea53b2f920 mesh-build-go: each package counts its skipped tests (issue 890)
mesh/delivery merged: source.main moved to novox/mesh-catalog@71e094b2de59
mesh/delivery-group group fix/checker-skips-not-kept merged: every member is merged
mesh/check failed: ace: the check of novox/mesh-catalog#492 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
2026-10-26 12:29:19 +01:00
jschoubben 99cc073e2a gate: pin mesh-sdk at the group's head 9c61918ac36f, where a turn never holds the check of heads waiting in it (ADR 0548 §3; issue 568, review round 1) 2026-10-26 12:15:20 +01:00
jschoubben b94150fcdb gitea: the merge tool reads a carried verdict's line as judged on the new trunk, so it merges a carried verdict (ADR 0548 §8; issue 568, review round 1) 2026-10-26 12:15:20 +01:00
jschoubben a01715ddf5 Skip the Node build module's toolchain tests by name where no Node toolchain is installed, instead of installing one from the registry in a level that has no network; merge three tests that share one fixture
mesh/delivery merged: source.main moved to novox/mesh-catalog@f96704fa0ff4
mesh/check checking a01715ddf508 on 565d0a848a85
On ace, which holds no Node build module, five integration tests each spent 70 s in npm's network retries and failed every mesh-catalog check run there (five verdicts since 00:07). The tests now copy the node's installed toolchain or are skipped as NOT RUN; the registry is never reached. Three tests sharing the fixture are one, so the toolchain is copied three times instead of five (10 integration tests in the module, now 8).
2026-10-26 12:05:09 +01:00
jschoubben e2136ec0c9 The vault takes a handing whole once, refusals and all, and names the refused users in its last adoption (review of the adoption, ADR 0547 clause 9) 2026-10-26 12:03:14 +01:00
jschoubben a30c168512 desk-channel: take the work when the store does not answer, and keep what it holds at the next work and the minute's standing
mesh/delivery merged: source.main moved to novox/mesh-catalog@86fdf74a0a83
mesh/check checking a30c16851246 on 565d0a848a85
Review round 1: offering the work again let a redelivered raise follow its own clearing and
replace it, and a redelivery is bounded where an outage is not. The work is now taken; what
was shown stays in memory, recalled first, and is kept at the start of the next work and at
the minute's standing, stopping at the first write the store does not answer. A desk that
restarts before then loses where those are shown, and its log said how many it held.
2026-10-26 11:46:21 +01:00
jschoubben f885250913 desk-channel: keep where a message is shown when the store does not answer, so its clearing still replaces it
On 2026-10-26 09:18 the bus reloading its accounts made the desk's state store answer
'no responders' just as the urgent 'A new build of mesh-controller cannot be delivered'
was shown. The desk logged that it could not keep the notification ids and acked the work;
the router's clearing at 09:34 then found no record, was shown as a new notification, and
the urgent one stayed on the operator's screen for hours.

Now what was shown and not kept is held in memory, recalled before the store by edits,
reminders, clicks and a notifier restart, kept at the next piece of work, and its work is
offered again until the store keeps it, without being shown twice. An edit of a message
nothing records is said in the journal.
2026-10-26 11:46:09 +01:00
jschoubben 7627503d9d gate: pin mesh-sdk at the group's head f5c233a9288c, which holds the merge turns and the carried verdict (ADR 0548 §10; issue 568) 2026-10-26 11:45:30 +01:00
jschoubben 351c7c4bf4 gate: deliveries and show say an approved merge waits for the check of the named delivery on the named repository, and the turns bucket is read by the checker alone, written by the gate alone (ADR 0548 §3, §9; issue 568 G6) 2026-10-26 11:44:29 +01:00
jschoubben 55a26b22cf gate: delivery_turns reads every merge turn with the check it waits for, and delivery_hand_now hands a held merge at once on the operator's word (ADR 0548 §4; issue 568 G5) 2026-10-26 11:43:40 +01:00
jschoubben 100e74f283 gate: an approved merge whose trunk moved waits for its carried verdict and merges on it with no second ask, never on the old one (ADR 0548 §8; issue 568 G4) 2026-10-26 11:43:18 +01:00
jschoubben c8e1e2b2fa gate: a merge held past 30 minutes by another delivery's check is a stalled line naming that check and since when, said as delivery.<id>.stalled (ADR 0548 §4; issue 568 G3) 2026-10-26 11:41:42 +01:00
jschoubben d0d00f922a gate: an approved merge writes its turn on every repository it goes into before it is handed, kept for a group until its last member merged, read by the checker (ADR 0548 §3; issue 568 G2) 2026-10-26 11:41:26 +01:00
jschoubben c2bd7760b2 gate: hand no approved merge into a repository a running check of other heads judges, and hand it on that check's end (ADR 0548 §2; issue 568 G1) 2026-10-26 11:40:35 +01:00
jschoubben f0fd53281b The vault says once why a look of the bus module's have adopted nothing, and again when it changes (ADR 0547 clause 9) 2026-10-26 11:37:14 +01:00
jschoubben fb039250f6 The vault adopts the hashes of the bus accounts it holds no value for from the bus node's handing (ADR 0547 clause 9, issue 1945)
take-over waits on a hash for every user of the loaded list, and the vault holds no value for the controller's own
account, the node accounts and the enrolled devices. The bus node's engine now hands every loaded user's hash in the
bus module's have, sealed to the vault's key (mesh-host and mesh-sdk on the same branch). The vault answers its key in
node-key, reads the haves (manifest reads node-engine.have, granted from the manifest, no controller change), watches
the bus module's have, opens the handing with its own key, and writes each user it holds no value for into
bus-accounts marked adopted, with the list and node it came from, saying account-hashed. It never touches a user whose
covered account it holds or whose key is its own hash, writes an adopted key again only for another hash, and refuses
a handing from any node the assignments do not name as holding the bus module.

Tools: account_adopt adopts now from the have as the watch does; secret_own, unnarrowed by module, lists the adopted
accounts and the last adoption.
2026-10-26 11:36:33 +01:00
jschoubben 447e6a13cd Knowledge base: withdraw a clause's changed mark when its changing record's accepted links drop it, on every judgement and by a tool (stale marks)
mesh/delivery merged: source.main moved to novox/mesh-catalog@843281c0e5f0
mesh/check checking 447e6a13cd68 on 565d0a848a85
2026-10-26 11:22:12 +01:00
mesh-tools c51a46b802 Merge pull request 'Build modules: run every package's tests of a level, with a fixture of two failing packages (issue 1921)' (#488) from fix/1921-every-failure-r into main 2026-10-26 10:18:08 +00:00
jschoubben 15a5fe09b6 mesh-build-go, mesh-build-node: run every package of a check's level after a failure (issue 1921)
mesh/delivery merged: source.main moved to novox/mesh-catalog@c51a46b802d3
mesh/delivery-group group fix/1921-every-failure-r merged: every member is merged
mesh/check checking 15a5fe09b62e on c51a46b802d3 | novox/mesh-catalog#488 in group fix/1921-every-failure-r
Pins mesh-sdk fix/1921-every-failure-r, whose check ask no longer stops a level at its first failing test step. A captured fixture with a failing test in each of two packages proves the build module's test step names both.
2026-10-26 10:58:07 +01:00
mesh-tools 3ded9e7800 Merge pull request 'Vault: name a hash record's secret in a field called of, so the runtime keeps it (issue 1945)' (#487) from fix/1945-vault-hash-field into main 2026-10-26 09:00:51 +00:00
jschoubben e7aaa8420e Vault: name a hash record's secret in a field called of, so the runtime accepts it
mesh/delivery merged: source.main moved to novox/mesh-catalog@3ded9e780043
mesh/check pass e7aaa8420eac on 3ded9e780043 | unit 2018 run, 2325 kept, 5 skipped | integration 2040 run, 2344 kept, 9 skipped | over bound: unit, in…
The runtime refuses a state field whose name ends in secret, so no bus-account hash could be written on novox. The test state now applies the same check.

Issue: 1945
2026-10-26 09:25:35 +01:00
mesh-tools 22381d7522 Merge pull request 'Vault: read own secrets' generations, and hash each bus account it is handed for the grants part (issue 1945, ADR 0547 clause 9)' (#486) from fix/1945-vault-own-read into main 2026-10-26 07:48:03 +00:00
jschoubben 1c55e5d44d Vault: take and hash a process's own bus account (broker-<process id>) as a covered account, so take-over is not held by every process user
mesh/delivery merged: source.main moved to novox/mesh-catalog@22381d75224f
mesh/check failed: integration: meshbuildnode/cmd/mesh-build-node.TestAFixtureBundleBuiltTwiceHasOneDigest: no toolchain of this lock is installed und…
The controller hands each process account of a core or moved module with origin issued (ADR 0436 §2); the vault took
only broker and provisioner-broker, refused these and never hashed them. Owed hashes are now said once per pass, and
account_hash says who asked when it refuses too (review round 1).

Issue: 1945
2026-10-26 02:35:57 +01:00
jschoubben 32b912cfe5 Vault: hash each bus account it is handed into bus-accounts.<user>, once per generation, so the grants part can compose the user list without the controller
The controller still mints each covered account and hands it to the vault (ADR 0374); ADR 0547 clause 9 has the vault
write the bcrypt hash of what it holds into its own state, readable by the grants part, and say account-hashed. It
hashes on keep, at start every held account with no key of its generation, and again by hand (account_hash); a refused
write is owed and written at the next pass. secret_own says per bus account whether this generation's hash is written.

Issue: 1945
2026-10-26 02:28:28 +01:00
jschoubben e572937d97 Vault: secret_own reads the module-own secrets it keeps, as fingerprints, so a generation can be told without the value
mesh/delivery closed: a newer head of the same pull request
mesh/check pass e572937d97fa on 567736f7795a | unit 2143 run, 2198 kept, 5 skipped | integration 2165 run, 2217 kept, 9 skipped | over bound: unit, in…
Issue: 1945
2026-10-26 02:09:03 +01:00
mesh-tools 567736f779 Merge pull request 'Build modules' step and toolchain lists read only the newest keys they answer, in parallel (issue 1827)' (#485) from fix/1827-bounded-reads into main 2026-10-25 23:56:12 +00:00
jschoubben e901769cb2 Read the newest steps first: sort the node's keys by id (start order) before the cut, and say older steps were not read
mesh/delivery merged: source.main moved to novox/mesh-catalog@567736f7795a
mesh/check failed: integration: meshbuildnode/cmd/mesh-build-node.TestAFixtureBundleBuiltTwiceHasOneDigest: no toolchain of this lock is installed und…
The review found a cut read took an arbitrary 1000 keys, so the newest steps could be missing. A step id is
base-36 milliseconds plus random digits, so its string order is its start order; the newest max(limit*4, 100)
keys are read now, 1000 at most. The toolchains read logs when it drops keys past its cap.
2026-10-26 00:43:53 +01:00
jschoubben 8f9c78bcba Read only the node's own steps, in parallel and bounded, so build_*_steps answers within the call's 30 s (issue 1827)
On a steps state of 1164 keys / 20 MB (mesh-build-node: 10034 keys) the read tools listed every key and
then read each value one round trip at a time, filtering by node only after the Get, so they timed out
at 30 s. Now the keys are filtered by the node's prefix before any Get, the values are read 16 at a
time, at most 1000 a call (the answer says it was cut), and one log line gives rows read and time.
build_*_toolchains reads its values the same parallel way. The kind: a read tool that reads a whole
state key by key; every read tool of the three modules is covered.
2026-10-26 00:39:33 +01:00
mesh-tools 82e7e3e326 Merge pull request 'mesh-provisioner-client: a module that leaves or is placed again asks the vault afresh (issue 1945)' (#484) from fix/1945-held-secret-on-leave into main 2026-10-25 23:21:44 +00:00
jschoubben 363a3d3f7e Provisioner client: drop a module's held vault secrets when it leaves or is placed again, so it asks afresh (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-catalog@82e7e3e32675
mesh/check pass 363a3d3f7e8f on 82e7e3e32675 | unit 2015 run, 2322 kept, 5 skipped | integration 2037 run, 2341 kept, 9 skipped | over bound: unit, in…
A re-assigned module was filled with the generation it held before the vault minted a new one; the bus refused the account.
2026-10-26 00:03:43 +01:00
jschoubben 609013a263 Merge pull request 'Gate: a merge learnt seconds late moves the trunk; and a setting check-gates-merge so the merge check can stop gating (operator 2026-10-25 22:56)' (#483) from fix/merge-without-check-gate into main
Reviewed-on: #483
2026-10-25 22:18:41 +00:00
jschoubben 32ad59fc64 mesh-delivery-gate and gitea: a setting check-gates-merge; false asks a reviewed pull request's Approve and merges it without waiting for mesh/check
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 32ad59fc6402 on 609013a2638f | unit 2010 run, 2322 kept, 5 skipped | integration 2032 run, 2341 kept, 9 skipped | over bound: unit, in…
The operator, 2026-10-25 22:56: "run the check locally only, but not as a gate, it loses us too much time". Tonight each
merge ask waited 3-7 minutes for mesh/check, and every merge into main refused the next approved merge for a check judged
on the trunk before (mesh-delivery#52). Both settings default to true, so nothing changes until the operator sets them.
False: the gate makes a reviewed pull request ready and asks its Approve with no verdict, a trunk's move puts nothing out
of date, a verdict is kept and shown, and a group still merges in one pass; the merge tool merges a head the Approve
names whatever its mesh/check says. delivery_status and show say the mode.
2026-10-25 23:07:50 +01:00
jschoubben 93f655dac9 mesh-delivery-gate: a merge learnt from the open list seconds after it landed moves its trunk and source.main; delivery_status shows source.main
mesh-delivery#51 merged 21:43:11 and the gate read the open list at 21:43:14, before the forge's merged event: taken as
learnt late, it moved no trunk and its group's finish left it out of source.main, so the checker kept judging every
mesh-delivery head on 73be3df. A late merge within replayLate of its landing is now taken as heard as it happened, and
one a gate before this left is written at the catch-up of the start (never backwards).
2026-10-25 23:01:37 +01:00
mesh-tools 573a59e3cc Merge pull request 'mesh-provisioner-client: retry a refused write on the node-engine's wake and ask the vault again at once (issue 924)' (#482) from fix/provisioner-retries-on-declaration into main 2026-10-25 21:08:48 +00:00
jschoubben d0566f9ade Fill a refused file again once the node-engine's declaration may let it through (ace, 2026-10-25 21:17)
mesh/delivery merged: source.main moved to novox/mesh-catalog@573a59e3cc07
mesh/delivery-group group fix/provisioner-retries-on-declaration merged: every member is merged
mesh/check pass d0566f9ade9a on 573a59e3cc07 | unit 2301 run, 3410 kept, 10 skipped | integration 2326 run, 3431 kept, 13 skipped | over bound: unit, …
The node-engine refused mesh-build-go's files as not named as filled 32 s before
its want arrived, and nothing filled them after the apply that named them: the
runtime crash-looped on its missing broker credential. The client now fills
every module again when the node-engine touches its write socket after such an
apply (mesh-host fix/provisioner-retries-on-declaration), says each retry and
each refusal it waits on, and asks the vault again at once for a refused secret
when that wake or the module's own assignment may have ended the refusal.
2026-10-25 21:42:34 +01:00
jschoubben 9d9dfda19b Merge pull request 'Gate: hand every member of an approved group in pin order, settle pull requests closed on the forge, and say a stopped pass as a condition (issue 1275)' (#480) from fix/1275-group-hands-every-member into main
Reviewed-on: #480
2026-10-25 19:28:33 +00:00
jschoubben cdbe2945cf Merge pull request 'Build modules' steps live two days and their pause has a state of its own, so the steps bucket no longer fills (2026-10-25)' (#481) from fix/steps-bucket-ages into main
Reviewed-on: #481
2026-10-25 18:06:39 +00:00
jschoubben 4e82a824be Announce the pauses state in the three build modules, and pin mesh-sdk 518dae5 (an unreadable pause pulls as not paused)
mesh/delivery-group group fix/steps-bucket-ages held: a member is held: novox/mesh-sdk@518dae5ec9c8 (merged without a passing check: only a person decides what…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 4e82a824be7a on cdbe2945cf6e | unit 1994 run, 2322 kept, 5 skipped | integration 2016 run, 2341 kept, 9 skipped | over bound: unit, in…
Without it in Kept, stdio.KeptState refuses the state before the bus (mesh-sdk go/keeps.go
refusedUnannounced), so the pause could be neither read nor written.
2026-10-25 18:57:08 +01:00
jschoubben 78d26867d2 Give the build modules' steps state an age of two days, and keep a node's pause in a pauses state of its own
mesh-build-go's steps bucket filled to the bus's 64 MiB cap (27,370 keys, none ever removed) and
refused the end of a passed check, so the check read failed; mesh-build-node's (17.8 MB) and
mesh-build-image's grow the same way. A step's key is per step: its reader reads its end within
the step's bound, and a check's verdict is kept by the checker, so two days is ample. The node's
pause key lived in the same bucket and would have ended by itself; it moves to a pauses state
without an age (mesh-sdk fix/steps-bucket-ages, Puller.PausePut). No node is paused today, so
nothing is carried over.
2026-10-25 18:53:04 +01:00
jschoubben 0c216d99d0 Gate takes a merge learnt after the fact as past: no trunk move, the forge's time, never source.main (issue 1275, review round 2)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 0c216d99d0c0 on 9d9dfda19b7e | unit 2181 run, 2147 kept, 5 skipped | integration 2203 run, 2166 kept, 9 skipped | over bound: unit, in…
A pull request found merged only by its absence from the open list went through PullMerged, which moved every
other open delivery of its repository back to an old trunk (ready ones to proposed, composed checks out of
date), dated the merge now, and so let a later catch-up write the old commit into source.main. pullMerged now
takes such a merge as late: its own delivery is merged and its group closed, no other trunk moves, its time is
the forge's or none, and LearntLate keeps it out of source.main in finish, settleForgeMerges and the catch-up.

The gitea tool gitea_commit_statuses now answers the pull request's merged_at, and the gate reads a gone pull
request by its number alone, so a deleted branch does not fail the read.
2026-10-25 18:37:59 +01:00
jschoubben 145c54d19c Gate settles pull requests closed with no event, looks again at a line's bound, and keeps a waiting head's condition over a restart (issue 1275, review round 1)
The conditions the gate now says would otherwise be wrong three ways:

- A pull request closed or merged on the forge with no event heard stayed ready here for days (de-spiegel #4,
  closed 2026-10-17) and would raise a condition. After a complete read of the open list (shorter than the
  forge tool's 1000), each delivery not final whose pull request is missing is read from the forge and taken as
  its close or merge; a merge learnt so does not move source.main, as it is undated and could move it back.
- A line that comes due on the clock (a state's bound, the author's bound, an answered refusal, an
  out-of-date refusal turning urgent after 10 minutes) waited for an unrelated event. The gate now asks a look
  at the earliest such time.
- After a restart the heads waiting on the forge start their bound again, so their conditions were cleared and
  raised again later. A condition is cleared only once the open list was read since start, and not while the
  forge still shows its head open and waiting.

Also: an empty path list ranks with a truncated one in the merge order, and a seat that keeps refusing
conditions is retried on the clock 10 looks in a row, then at events only, said once in the log.
2026-10-25 18:32:19 +01:00
jschoubben 67092161bd Gate hands every member of an approved group, in pin order, and says a stopped pass (issue 1275)
Met again 2026-10-25: group fix/puller-wakes-on-capacity was merged half. Three causes in the gate:

- The next member's turn compared the group's state with "delivering", a word GroupState no longer says
  since release 6 renamed it "merging" (ready.go groupTurnToMerge). Once the first member's merge was heard
  from the forge the group read merging, and the second member waited for ever. Group states are now named
  constants, the turn goes on in ready and merging (groupMayMerge), and a test holds every word to it.
- Members merged in id order, so mesh-catalog went before the mesh-sdk commit it pins. membersOf now orders
  by mergeRank: mesh-sdk first, then members that change no go.mod or go.sum, then those that do.
- No condition was raised: the controller's self-check that read `stalled` went with release 6
  (mesh-controller dc456d56). The gate now says its stalled lines as conditions itself on the
  operator-channel seat (ADR 0437 §9) after every look, clears those listed no more, and keeps which it
  raised in its new bucket conditions. A held group's line says "half merged" when a member merged.

A pass held for a required status no longer retries on a timer: the status turning success is the event
that asks the merge again.
2026-10-25 18:20:58 +01:00
mesh-tools 2df0503034 Merge pull request 'Let the build modules follow their capacity on the bus, so a node raised from capacity 0 starts pulling (issue 1887)' (#479) from fix/puller-wakes-on-capacity into main 2026-10-25 17:02:40 +00:00
jschoubben 23c827e791 Take the mesh-sdk head that wakes a puller on its capacity, within the test budget, in the three build modules (issue 1887)
mesh/delivery merged: source.main moved to novox/mesh-catalog@2df0503034dc
mesh/delivery-group group fix/puller-wakes-on-capacity merged: every member is merged
mesh/check pass 23c827e7910c on a85d90374195 | judged by the candidate's mesh-build-go at eab02c80fe86
2026-10-25 17:44:50 +01:00
jschoubben 3138570c6b Let the build modules follow their capacity on the bus, so a node raised from capacity 0 starts pulling (issue 1887)
g14's Go builder sat idle at capacity 2 because nothing woke its parked
puller. mesh-build-go, mesh-build-node and mesh-build-image now pin mesh-sdk
b9911dc1, whose puller watches the config value capacity as mesh-config
resolved it and wakes on each change, and hand it that watch.
2026-10-25 17:28:00 +01:00
jschoubben a85d903741 Merge pull request 'nextcloud: the module makes its own mesh-admin account right on a 401 and backs off on a refusal, so its shares poll stops being refused every minute (issue 1253)' (#478) from fix/nextcloud-admin-401 into main
Reviewed-on: #478
2026-10-25 16:01:37 +00:00
jschoubben 1cc54959af nextcloud: repair the account only on a 401, never reset the throttle (issue 1253, review 2)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 1cc54959afcf on a85d90374195 | unit 1994 run, 2322 kept, 5 skipped | integration 2016 run, 2341 kept, 9 skipped | over bound: unit, in…
Resetting Nextcloud's brute-force record for the docker gateway cleared it for
every client behind that address: someone guessing passwords could drive the
throttle to 429 and have the module's poll wipe it. A throttle is no evidence
the password is wrong, so a 429 is now backed off only, and the back-off
outlasts the throttle. The account's existence is read from user:info's exit
status, since user:list pages.
2026-10-25 16:57:46 +01:00
jschoubben 962bc359a7 nextcloud: repair only the module's own account, read its existence from the user list, no admin group (issue 1253, review 1)
occ's user:info says 'user not found' on stdout with an empty stderr, so the
error read as 'occ produced no output' and mesh-admin was never made on novox:
occ now says stderr and stdout both, and existence is read from user:list.
The repair could reset whatever account the user setting names, the
operator's included, and the 'admin' fallback is a hand-made install's human
administrator: only mesh-admin is repaired now, and it is the default. The
shares poll needs no admin rights, so the account gets no group. After a
repair the throttle of the caller's address is reset, and a failing repair
reaches the back-off's one loud line with occ's message.
2026-10-25 16:54:51 +01:00
jschoubben 67b13c42d3 nextcloud: repair the account on a throttle too, since a throttle follows refusals (issue 1253)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 67b13c42d32a on f27f4c8cd58c
Right after the delivery the first answer may be Nextcloud's 429, not a 401;
without a repair then the account would never be made and the back-off would
wait out a throttle that cannot end.
2026-10-25 16:40:54 +01:00
jschoubben 4021724ca4 nextcloud: make the mesh's own account match its secret when Nextcloud refuses it, and back off a refusal that stays (issue 1253)
The image creates NEXTCLOUD_ADMIN_USER only on a fresh install. novox's
instance was installed before the mesh adopted it, so mesh-admin was never
created there, and the shares poll asked Nextcloud every minute with a login
that cannot pass: 401, then Nextcloud's brute-force throttle (429), which
also held the backup before a delivery.

On a refused login the client now adds the account through occ (admin
group), or sets its password to the module's own secret, passing it by
OC_PASS so it is on no argv, and asks once more. A refusal that stays is
said once and the poll backs off from 5 minutes to an hour instead of nine
lines of XML every minute. An empty setting no longer becomes an empty name.
2026-10-25 16:40:01 +01:00
jschoubben f27f4c8cd5 Merge pull request 'mesh-build-go: a package that fails without an assertion carries its real cause, never "PASS" (issue 1901)' (#477) from fix/1901-package-cause into main
Reviewed-on: #477
2026-10-25 15:07:40 +00:00
jschoubben 155124b59a mesh-build-go: a cause is the runtime's line at column 0, and only go's exact summary lines are dropped (issue 1901, review round 1)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/delivery-group group fix/1901-package-cause held: a member is held: novox/mesh-catalog@155124b59af4 (merged without a passing check: only a person decides…
mesh/check pass 155124b59af4 on f27f4c8cd58c | unit 2015 run, 2883 kept, 5 skipped | integration 2037 run, 2978 kept, 9 skipped | over bound: unit, in…
An indented log line of another running test that held 'panic:' won over the
real timeout panic, since the lines were trimmed before matching. Go prints
the runtime's lines at column 0; a test's log is indented.
2026-10-25 15:58:07 +01:00
jschoubben 6ec6fdaad2 mesh-build-go: a race the race detector reports is a failure's cause, in a test or after PASS (issue 1901)
mesh-controller's try was worded 'PASS' again because the race detector
reported after the tests passed and the program exited 66. The cause is now
the race: its access, the frame that made it and where its goroutine was
created, instead of PASS or testing's generic 'race detected during execution
of test'. Tests from captured go test -race -json (1901/AC4).
2026-10-25 15:52:20 +01:00
jschoubben 663f3de975 mesh-build-go: a package that fails without an assertion carries its real cause, never "PASS" (issue 1901)
A try of mesh-controller kept 'PASS' as its only failure: the package's test
program stopped after printing PASS and the package-level failure took its
first line. Now a failure that is not one test's assertion names the tests that
were running and carries the panic line (a timeout's with its running tests),
a signal or fatal error line, a build's first error, or words saying the
program exited during which tests or after they passed; go test's summary lines
are never kept. Tests from captured go test -json of eight real failures
(1901/AC4).
2026-10-25 15:38:28 +01:00
jschoubben 68f7431d98 Merge pull request 'The provisioner client fills every module again when the socket returns and when its node's own composition moves (issue 1944)' (#476) from fix/r6-node-composition into main
Reviewed-on: #476
2026-10-25 00:18:25 +00:00
jschoubben 53749833bf Fill every module again when the node-engine's socket returns and when the node's own composition moves, so a file an apply left empty is filled (issue 1945)
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check checking 53749833bfbd on 2167b95a3699 | novox/mesh-catalog#476 in group fix/r6-node-composition
2026-10-25 02:02:02 +02:00
mesh-tools 2167b95a36 Merge pull request 'R6: the gate keeps only the merge's states; walk, plan and step code and their verbs go (issue 1946)' (#475) from feat/r6 into main 2026-10-24 23:00:25 +00:00
jschoubben d09cbf1523 Ignore the binaries a local go build leaves in three module roots (issue 1943)
mesh/delivery-group group feat/r6 merged: every member is merged
mesh/delivery delivering: 0 machine step(s) passed
mesh/check pass d09cbf152317 on 6fb0ceeef817 | unit 4378 run, 4719 kept, 15 skipped | integration 5060 run, 4917 kept, 23 skipped | over bound: unit, …
2026-10-25 00:02:45 +02:00
jschoubben ac6712cbb5 Pin mesh-sdk to feat/r6 748c564, which carries main's composer (issue 1943) 2026-10-25 00:02:45 +02:00
jschoubben 5b32342ac2 Merge main into feat/r6: keep R6's gate, which already carries #474's catch-up at start, and drop the seat and walk adoption R6 removed
The reads-outside markers, mesh-build-go's compose job and gitea's forge-account fix come in from main.
2026-10-24 23:03:42 +02:00
jschoubben b67b5af5e6 Pin the gate to mesh-sdk feat/r6 at d7a9b968 (issue 1946) 2026-10-24 21:18:26 +02:00
jschoubben d700ceca20 Pin the gate to mesh-sdk feat/r6 at 7f9cc0e5 (issue 1946) 2026-10-24 20:43:41 +02:00
jschoubben 3a1017f559 Pin the gate to mesh-sdk feat/r6 at e45bdc08 (issue 1946) 2026-10-24 20:22:07 +02:00
jschoubben a17d0c8e93 The messenger holds and tells what the delivery view says is in motion, not the controller's plans (issue 1946) 2026-10-24 18:25:38 +02:00
jschoubben 9fa6710296 Pin the gate to mesh-sdk feat/r6 at a523a8cb (issue 1946) 2026-10-24 18:18:06 +02:00
jschoubben 05f849979c Match a fix to a delivered revision by what the gate says of its merge; keep #474's source catch-up at start (issues 1946, 1726) 2026-10-24 18:13:40 +02:00
jschoubben 4dee77df23 Pin the gate to the head of mesh-sdk feat/r6 (issue 1946) 2026-10-24 17:54:30 +02:00
jschoubben 8a79877a15 A newer delivered revision covers the older merges of its repository, as the forge says ancestry (issue 1946) 2026-10-24 17:03:20 +02:00
jschoubben 2139e83a8a Remove a binary built by mistake 2026-10-24 16:58:33 +02:00
jschoubben e7ca8f7af1 The backup module says on the bus when a backup asked for now has ended, so no one polls backed-up (issue 1944, ADR 0539 §16) 2026-10-24 16:58:19 +02:00
jschoubben eb79996336 The tracker derives a fix's delivered from the delivery view's event, not the gate's gone states (issue 1946) 2026-10-24 16:50:50 +02:00
jschoubben 1a271a1fb1 The group's integration test hears the forge's merges once both members are handed (issue 1946, R6-C3) 2026-10-24 16:43:48 +02:00
jschoubben 30d04f6bef Pin the gate to mesh-sdk's release 6 verb tables and prove it holds the seat as they have it (issue 1946, R6-C3)
The integration tests hear the forge's own word that a pull request merged, as the real flow does, and fail on any call of the controller.
2026-10-24 16:35:15 +02:00
jschoubben 5ff337e0dc Gate keeps the merge's six states and asks the controller nothing (issue 1946, R6-C1, C2, C4)
Release 6 deletes the plans: the walk half of the holder, the controller port, the carry-over of the former module and the verbs times, what-if, release, stop and retire-history go; the table holds proposed, ready, rejected, merged, held and closed; the delivery's status and view name the source revision the merge moved.
2026-10-24 16:26:36 +02:00
208 changed files with 8125 additions and 11114 deletions
+5
View File
@@ -29,3 +29,8 @@ modules/mesh-build-go/cmd/mesh-build-go/mesh-build-go
modules/mesh-config/mesh-config modules/mesh-config/mesh-config
modules/mesh-build-image/cmd/mesh-build-image/mesh-build-image modules/mesh-build-image/cmd/mesh-build-image/mesh-build-image
*.test *.test
# binaries a local go build leaves in a module root
/checks/paths/paths
/modules/mesh-build-go/mesh-build-go
/modules/mesh-delivery-gate/mesh-delivery-gate
+107 -9
View File
@@ -15,6 +15,13 @@ package main
// the desk says in its standing when that happened (`answers-lost-at`), and the router, which alone knows // the desk says in its standing when that happened (`answers-lost-at`), and the router, which alone knows
// what is still open and unanswered, says that again under its handle — replacing the notification in place, // what is still open and unanswered, says that again under its handle — replacing the notification in place,
// so its buttons work again. Nothing resolved, silenced or cancelled comes back. // so its buttons work again. Nothing resolved, silenced or cancelled comes back.
// - **A store that does not answer does not lose where a message is shown, while the desk runs.** A message
// is shown first and its notification ids kept after; when the state store does not answer then (the bus
// reloading its accounts, 2026-10-26 09:18: the very outage that raised the condition), the work is taken
// all the same and the ids are held in memory, recalled before the store, and kept at the start of the next
// work and at the minute's standing. Otherwise the clearing of that message finds no notification to
// replace and the urgent one stays on screen for good. A desk that restarts while it still holds some loses
// where those are shown; the log says so when they are held.
// - **Standing**: it can send when machines are named in its settings; which of them answered last is in its // - **Standing**: it can send when machines are named in its settings; which of them answered last is in its
// status. A message no machine took is said to the router as failed, and tried again by it. // status. A message no machine took is said to the router as failed, and tried again by it.
@@ -93,6 +100,9 @@ type Desk struct {
pause map[string]pauseSeen pause map[string]pauseSeen
// lost is when each machine's notifier last started again: the answers its notifications waited for are lost. // lost is when each machine's notifier last started again: the answers its notifications waited for are lost.
lost map[string]time.Time lost map[string]time.Time
// unkept is what was shown and not yet kept in the store, because it did not answer: recalled from here
// before the store, and kept at the next piece of work.
unkept map[string]Shown
} }
func (d *Desk) init() { func (d *Desk) init() {
@@ -107,6 +117,9 @@ func (d *Desk) init() {
if d.lost == nil { if d.lost == nil {
d.lost = map[string]time.Time{} d.lost = map[string]time.Time{}
} }
if d.unkept == nil {
d.unkept = map[string]Shown{}
}
if d.Now == nil { if d.Now == nil {
d.Now = time.Now d.Now = time.Now
} }
@@ -138,6 +151,7 @@ func (d *Desk) Take(subject string, body json.RawMessage) error {
d.Logf("[desk-channel] work on %s could not be read; taken and dropped", subject) d.Logf("[desk-channel] work on %s could not be read; taken and dropped", subject)
return nil return nil
} }
d.KeepUnkept()
machines := d.Machines() machines := d.Machines()
if m.Secret { if m.Secret {
// Shown only where the notifier said lately it is not paused (the review of 2026-10-09): one held back // Shown only where the notifier said lately it is not paused (the review of 2026-10-09): one held back
@@ -167,12 +181,16 @@ func (d *Desk) Take(subject string, body json.RawMessage) error {
switch { switch {
case edit: case edit:
var err error var err error
if before, err = d.Store.Get(m.Handle); err != nil { if before, err = d.recall(m.Handle); err != nil {
return err return err
} }
if before == nil {
d.Logf("[desk-channel] an edit of %s, which nothing here records as shown: shown as a new message; "+
"a notification it was to replace, if any, stays", m.Handle)
}
case !m.Secret: case !m.Secret:
var err error var err error
if before, err = d.Store.Get(m.Handle); err != nil { if before, err = d.recall(m.Handle); err != nil {
d.Logf("[desk-channel] what is shown under this handle could not be read (%v): shown as a new message", err) d.Logf("[desk-channel] what is shown under this handle could not be read (%v): shown as a new message", err)
before = nil before = nil
} }
@@ -218,12 +236,91 @@ func (d *Desk) Take(subject string, body json.RawMessage) error {
// Shown once and kept nowhere of this module's (a link's code): no edit or re-showing reaches it. // Shown once and kept nowhere of this module's (a link's code): no edit or re-showing reaches it.
return nil return nil
} }
if err := d.Store.Put(m.Handle, s); err != nil { d.keep(m.Handle, s)
d.Logf("[desk-channel] shown, and what was shown could not be kept: an edit of it will not reach it: %v", err)
}
return nil return nil
} }
// keep keeps what a handle shows in the store. When the store does not answer it is held in memory, where an
// edit, a reminder and a click still find it, and kept later (KeepUnkept). The work is taken either way: offered
// again it would race the message's own later edits, and a redelivery is bounded where an outage is not.
func (d *Desk) keep(handle string, s Shown) {
err := d.Store.Put(handle, s)
d.mu.Lock()
defer d.mu.Unlock()
if err == nil {
delete(d.unkept, handle)
return
}
d.unkept[handle] = s
d.Logf("[desk-channel] shown, and where it is shown could not be kept yet (%v): held in memory, so its edit "+
"and clearing still reach it, and kept when the store answers; %d held, lost if the desk restarts first",
err, len(d.unkept))
}
// KeepUnkept keeps in the store what it did not answer for before: at the start of each piece of work and at
// the minute's standing. It stops at the first write the store does not answer, so one call waits on it once.
func (d *Desk) KeepUnkept() {
d.init()
d.mu.Lock()
held := make(map[string]Shown, len(d.unkept))
for h, s := range d.unkept {
held[h] = s
}
d.mu.Unlock()
for h, s := range held {
if err := d.Store.Put(h, s); err != nil {
return
}
d.mu.Lock()
if cur, ok := d.unkept[h]; ok && sameShown(cur, s) {
delete(d.unkept, h)
}
d.mu.Unlock()
d.Logf("[desk-channel] where %s is shown is kept now", h)
}
}
func (d *Desk) unkeptOf(handle string) (Shown, bool) {
d.mu.Lock()
defer d.mu.Unlock()
s, ok := d.unkept[handle]
return s, ok
}
// recall is what a handle shows: from memory when the store did not keep it yet, else from the store.
func (d *Desk) recall(handle string) (*Shown, error) {
if s, ok := d.unkeptOf(handle); ok {
return &s, nil
}
return d.Store.Get(handle)
}
// handles is every handle shown: the store's and those held in memory.
func (d *Desk) handles() ([]string, error) {
hs, err := d.Store.Handles()
d.mu.Lock()
defer d.mu.Unlock()
seen := map[string]bool{}
for _, h := range hs {
seen[h] = true
}
for h := range d.unkept {
if !seen[h] {
hs = append(hs, h)
}
}
if err != nil && len(d.unkept) == 0 {
return nil, err
}
return hs, nil
}
func sameShown(a, b Shown) bool {
x, err1 := json.Marshal(a)
y, err2 := json.Marshal(b)
return err1 == nil && err2 == nil && string(x) == string(y)
}
// show puts one message on one machine's notifier, in place of `replace` when it names one. // show puts one message on one machine's notifier, in place of `replace` when it names one.
func (d *Desk) show(machine string, m asks.Message, replace int) (int, error) { func (d *Desk) show(machine string, m asks.Message, replace int) (int, error) {
urgency := "normal" urgency := "normal"
@@ -308,12 +405,12 @@ func (d *Desk) Chosen(event, machine, ticket string) {
} }
func (d *Desk) find(ticket string) (string, *Shown) { func (d *Desk) find(ticket string) (string, *Shown) {
handles, err := d.Store.Handles() handles, err := d.handles()
if err != nil { if err != nil {
return "", nil return "", nil
} }
for _, h := range handles { for _, h := range handles {
s, err := d.Store.Get(h) s, err := d.recall(h)
if err != nil || s == nil { if err != nil || s == nil {
continue continue
} }
@@ -356,9 +453,9 @@ func (d *Desk) Started(event, machine string) int {
d.lost[machine] = d.Now() d.lost[machine] = d.Now()
d.mu.Unlock() d.mu.Unlock()
n := 0 n := 0
if handles, err := d.Store.Handles(); err == nil { if handles, err := d.handles(); err == nil {
for _, h := range handles { for _, h := range handles {
if s, err := d.Store.Get(h); err == nil && s != nil && len(s.Message.Buttons) > 0 { if s, err := d.recall(h); err == nil && s != nil && len(s.Message.Buttons) > 0 {
if _, here := s.On[machine]; here { if _, here := s.On[machine]; here {
n++ n++
} }
@@ -472,6 +569,7 @@ func (d *Desk) Status() map[string]any {
"it proves nothing about who clicked: any process of the operator's account can choose an action, so it answers acknowledgements only", "it proves nothing about who clicked: any process of the operator's account can choose an action, so it answers acknowledgements only",
"an answer is taken only on a machine its message was shown on; a message with buttons stays until dismissed", "an answer is taken only on a machine its message was shown on; a message with buttons stays until dismissed",
"what it remembers is never shown again: when a notifier started again, the answers its notifications waited for are lost; the desk says when in its standing (answers-lost-at), and the router says again what is still open and unanswered, replacing those notifications in place — nothing resolved, silenced or cancelled comes back (issue 384)", "what it remembers is never shown again: when a notifier started again, the answers its notifications waited for are lost; the desk says when in its standing (answers-lost-at), and the router says again what is still open and unanswered, replacing those notifications in place — nothing resolved, silenced or cancelled comes back (issue 384)",
"a store that does not answer does not lose where a message is shown while the desk runs: it is held in memory, its edit and clearing still replace it in place, and it is kept when the store answers again; a desk restarted before then loses where those are shown, and its log said how many it held",
"it can send while a named machine's notifier said lately that it is not paused; a link's code is shown only on such a machine, kept nowhere, and sent secret so the notifier keeps it out of its history", "it can send while a named machine's notifier said lately that it is not paused; a link's code is shown only on such a machine, kept nowhere, and sent secret so the notifier keeps it out of its history",
}} }}
if s.Why != "" { if s.Why != "" {
@@ -3,6 +3,7 @@ package main
import ( import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -363,9 +364,64 @@ func TestAMessageSaidAgainUnderItsHandleReplacesItsNotification(t *testing.T) {
} }
} }
// downStore is a store that does not answer while down: the bus reloading its accounts (2026-10-26 09:18).
type downStore struct {
memShown
down *bool
}
func (s downStore) Get(h string) (*Shown, error) {
if *s.down {
return nil, errors.New("nats: no responders available for request")
}
return s.memShown.Get(h)
}
func (s downStore) Put(h string, v Shown) error {
if *s.down {
return errors.New("nats: no responders available for request")
}
return s.memShown.Put(h, v)
}
// The review of PR 176: a replace that fails on a machine keeps that machine's notification on record, so a // The review of PR 176: a replace that fails on a machine keeps that machine's notification on record, so a
// later edit or clearing still reaches it; failing everywhere, it is said to the router as failed. // later edit or clearing still reaches it; failing everywhere, it is said to the router as failed. And
// (2026-10-26 09:18, an urgent "A new build of mesh-controller cannot be delivered" left on screen for hours
// after its clearing): a message shown while the store does not answer keeps where it is shown in memory, its
// work is taken, its clearing replaces it in place, and it is kept once the store answers.
func TestAFailedReplaceKeepsTheNotificationItWasToReplace(t *testing.T) { func TestAFailedReplaceKeepsTheNotificationItWasToReplace(t *testing.T) {
// The raise, then its clearing, taken while the store does not answer; the store then stays down or comes
// back. Each is taken (never offered again, where a redelivered raise would follow its clearing), the
// clearing replaces the urgent notification in place, and the screen and the store end on the clearing.
for _, back := range []bool{false, true} {
t.Run(fmt.Sprintf("the store does not answer, back %v", back), func(t *testing.T) {
d, shown, _, store := newDesk(t, "laptop")
down := true
d.Store = downStore{store, &down}
raised, _ := json.Marshal(asks.Message{Handle: "m9", Title: "A new build cannot be delivered", Urgent: true})
if err := d.Take(asks.ChannelSubject(asks.Send, Kind), raised); err != nil {
t.Fatalf("the raise was not taken: %v", err)
}
cleared, _ := json.Marshal(asks.Message{Handle: "m9", Title: "Cleared: a new build can be delivered"})
if err := d.Take(asks.ChannelSubject(asks.Edit, Kind), cleared); err != nil {
t.Fatalf("the clearing was not taken: %v", err)
}
d.KeepUnkept() // the minute's standing, store still down
down = !back
d.KeepUnkept()
take(t, d, asks.Send, asks.Message{Handle: "m10", Title: "another"})
if len(*shown) != 3 || (*shown)[1].args["replace_id"] != 1 || (*shown)[1].args["summary"] != "Cleared: a new build can be delivered" {
t.Fatalf("the clearing did not replace the urgent notification: %+v", *shown)
}
s, kept := store["m9"]
if back != kept || (kept && s.Message.Title != "Cleared: a new build can be delivered") {
t.Fatalf("store back %v, kept %v: %+v", back, kept, store)
}
if got, _ := d.recall("m9"); got == nil || got.Message.Title != "Cleared: a new build can be delivered" {
t.Fatalf("recalled %+v", got)
}
})
}
d, shown, said, store := newDesk(t, "laptop", "desktop") d, shown, said, store := newDesk(t, "laptop", "desktop")
take(t, d, asks.Send, asks.Message{Handle: "m1", Title: "2 failed services on shanks"}) take(t, d, asks.Send, asks.Message{Handle: "m1", Title: "2 failed services on shanks"})
first := store["m1"].On first := store["m1"].On
@@ -100,6 +100,7 @@ func run(d *Desk) {
}() }()
go func() { go func() {
for { for {
d.KeepUnkept()
d.SayStanding() d.SayStanding()
time.Sleep(time.Minute) time.Sleep(time.Minute)
} }
+23
View File
@@ -275,6 +275,29 @@ reported no key gets no entry, and its client says it waits. Keys titled `mesh:`
a key added by hand never is. Every consumer is made again every ten minutes and at once when a key changes. a key added by hand never is. Every consumer is made again every ten minutes and at once when a key changes.
`go test ./cmd/forge-account-provider ./internal/forgeaccount` runs it against a fake forge. `go test ./cmd/forge-account-provider ./internal/forgeaccount` runs it against a fake forge.
## The tools' account reads every repository (the operator, 2026-10-26)
The tools act as the plain account `mesh-tools` (ADR 0365 §2, issue 469), which writes to the organisation `novox`
through its team. Any other repository it saw only when public, so a private repository of a user answered every tool
404 and was missing from every list. `readall.ts` makes the account a collaborator that **reads only** on each
repository it cannot read: one repository at a time, no organisation membership, the least the forge offers.
- **When:** one pass over every repository of every user and organisation at the start (a forge not answering yet is
asked again a minute apart until one pass ran), and on the forge's system webhook: `forge.repository.created` and
`forge.fork` at once, and the first event of any repository this process has not seen readable (the forge may not
send a creation for every owner). Nothing polls for it.
- **Not caught:** a repository made private after it was seen readable. The forge sends no event for a change of
visibility, and the start's pass already saw it readable, so its events ask nothing. It is granted at the module's
next start, or now by `gitea_grant_read`; `gitea_unreadable_repos` shows it meanwhile. A repository whose permission
the forge will not answer is named as refused (or, in the listing, with permission `unknown`) and the pass goes on.
- **With what:** the site administrator's login (`MESH_GITEA_ADMIN_PASSWORD_FILE`), because adding a collaborator asks a
repository's administration, which on another user's repository only the site administrator holds. It is used for
four calls alone: list every repository, read one, read the account's permission on one, add the account as a read
collaborator. The tools' own calls keep their token.
- **Tools:** `gitea_unreadable_repos` (every repository the account cannot read, the permission the forge gives it
there and why) and `gitea_grant_read` (the same grant by hand, now, on every repository or on those named; each grant
read back from the forge, a refusal answered in the forge's own words).
## Files it does not know ## Files it does not know
Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416). Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).
+60
View File
@@ -853,6 +853,25 @@ interface AdminResponse {
readonly body: any; readonly body: any;
} }
/** A repository as the site administrator sees it: what saying why an account cannot read it needs (readall.ts). */
export interface ForgeRepository {
full_name: string;
owner: string;
name: string;
private: boolean;
archived: boolean;
/** The owner's own visibility: public, limited or private. A public repository of an owner not public is hidden
* from whoever is no member. */
owner_visibility: string;
}
function forgeRepository(r: any): ForgeRepository {
const owner = String(r?.owner?.login ?? r?.owner?.username ?? "");
const name = String(r?.name ?? "");
return { full_name: String(r?.full_name ?? `${owner}/${name}`), owner, name, private: r?.private === true,
archived: r?.archived === true, owner_visibility: String(r?.owner?.visibility ?? "public") };
}
/** /**
* The forge's admin client, over **basic auth** — gitea's own code, living in the module, used only * The forge's admin client, over **basic auth** — gitea's own code, living in the module, used only
* by the provisioner (novox/hq ADR 0048/0076). * by the provisioner (novox/hq ADR 0048/0076).
@@ -970,6 +989,47 @@ export class GiteaAdmin {
return repos; return repos;
} }
// ---- The tools' account reads every repository (readall.ts) ----
/** Every repository on the forge, of every user and organisation: the search as the site administrator, whom the
* forge answers every repository, private ones and those of private owners included. Only a read. */
async everyRepository(): Promise<ForgeRepository[]> {
const all: ForgeRepository[] = [];
for (let page = 1; page <= 200; page++) {
const res = await this.request(`/repos/search?limit=50&page=${page}`);
if (res.status !== 200) GiteaAdmin.fail("/repos/search", res);
const list: any[] = Array.isArray(res.body?.data) ? res.body.data : [];
all.push(...list.map(forgeRepository));
if (list.length < 50) break;
}
return all;
}
/** One repository as the site administrator sees it, or null when the forge holds none by that name. */
async repository(owner: string, repo: string): Promise<ForgeRepository | null> {
const res = await this.request(repoPath(owner, repo));
if (res.status === 404) return null;
if (res.status !== 200) GiteaAdmin.fail(repoPath(owner, repo), res);
return forgeRepository(res.body);
}
/** What the forge itself says an account may do on a repository: none, read, write, admin or owner. */
async repositoryPermission(owner: string, repo: string, user: string): Promise<string> {
const path = `${repoPath(owner, repo)}/collaborators/${encodeURIComponent(user)}/permission`;
const res = await this.request(path);
if (res.status !== 200) GiteaAdmin.fail(path, res);
return String(res.body?.permission ?? "none");
}
/** Make an account a collaborator of a repository that reads only. The forge asks a repository's administrator for
* it, which on a repository of another user only the site administrator is. Idempotent: the forge answers 204 again. */
async addReadCollaborator(owner: string, repo: string, user: string): Promise<void> {
const path = `${repoPath(owner, repo)}/collaborators/${encodeURIComponent(user)}`;
const res = await this.request(path, { method: "PUT", body: JSON.stringify({ permission: "read" }) });
if (res.status === 204 || res.status === 200) return;
GiteaAdmin.fail(path, res);
}
/** Ensure the npm-owner org exists. 201 created, 2xx/404-then-created, and 422/409 (a concurrent /** Ensure the npm-owner org exists. 201 created, 2xx/404-then-created, and 422/409 (a concurrent
* create won the race) are all success. */ * create won the race) are all success. */
async ensureOrg(name: string): Promise<void> { async ensureOrg(name: string): Promise<void> {
+27 -2
View File
@@ -60,6 +60,8 @@ import { HOOK_PATH, HOOK_PORT, Pace, claimHead, ensureSystemHook, eventsTool, ke
import { keepSiblings, noteComposed, noteVerdict, readSiblings, siblingsFile, siblingsOf, weighSiblings } from "./siblings.js"; import { keepSiblings, noteComposed, noteVerdict, readSiblings, siblingsFile, siblingsOf, weighSiblings } from "./siblings.js";
import { bindInBackground } from "./bind.js"; import { bindInBackground } from "./bind.js";
import { announcementOf, keepTook, lateVerdictComment, outOfDateWords, sameCommit, tookFile, weighTrunks } from "./trunk.js"; import { announcementOf, keepTook, lateVerdictComment, outOfDateWords, sameCommit, tookFile, weighTrunks } from "./trunk.js";
import { ReadAll, readAllTools } from "./readall.js";
import { TOOLS_ACCOUNT } from "./tools-account.js";
// Without a way to a token — configured, or mintable with the admin account (token.ts) — there is // Without a way to a token — configured, or mintable with the admin account (token.ts) — there is
// nothing to watch; log and stay quiet rather than crash the runtime. With one, the first poll mints // nothing to watch; log and stay quiet rather than crash the runtime. With one, the first poll mints
@@ -500,8 +502,14 @@ if (!gitea) pace.unheard("the forge is not watched: no way to a token", false);
// Said with `emit`, not the SDK's `keeps`, which the toolchain's SDK does not have yet: switch to `keeps` once that SDK is 0.1.15 or later (issue 1852). // Said with `emit`, not the SDK's `keeps`, which the toolchain's SDK does not have yet: switch to `keeps` once that SDK is 0.1.15 or later (issue 1852).
await emit("states", statesAnnouncement(process.env.MESH_MODULE || "gitea", process.env.MESH_NODE)); await emit("states", statesAnnouncement(process.env.MESH_MODULE || "gitea", process.env.MESH_NODE));
// **The tools' account reads every repository** (readall.ts; the operator, 2026-10-26): granted with the site
// administrator's login, which the module holds for setting the forge up and uses here for that grant alone. Made on
// first use, so a module whose admin login is missing still serves the rest and the tools say why.
let readAllOnce: ReadAll | null = null;
const readAll = (): ReadAll => (readAllOnce ??= new ReadAll(GiteaAdmin.fromEnv(), process.env.MESH_GITEA_TOOLS_USER || TOOLS_ACCOUNT));
// gitea_events is served whether or not the forge is watched: it says so (novox/hq issue 729). // gitea_events is served whether or not the forge is watched: it says so (novox/hq issue 729).
registerModuleTools("gitea", () => [eventsTool(pace), ...(gitea ? [commitStatusTool({ registerModuleTools("gitea", () => [eventsTool(pace), ...readAllTools(readAll), ...(gitea ? [commitStatusTool({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status), setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gates, gates,
}), mergeTool(gitea, { gates })] : [])]); }), mergeTool(gitea, { gates })] : [])]);
@@ -543,7 +551,12 @@ async function listenForForge(client: GiteaClient): Promise<boolean> {
const server = receiver({ secret, port, heard: (kind, delivery) => pace.heard(kind, delivery), const server = receiver({ secret, port, heard: (kind, delivery) => pace.heard(kind, delivery),
refused: (reason, from) => pace.refused(reason, from), refused: (reason, from) => pace.refused(reason, from),
publish: (event) => emit(event.name, event.body), publish: (event) => emit(event.name, event.body),
act: (event) => actOn(client, event) }); act: async (event) => {
// A repository made, or one not yet seen readable, is granted read before anything acts on it (readall.ts).
await Promise.resolve().then(() => readAll().onEvent(event)).catch((err) =>
console.error(`[gitea] whether ${TOOLS_ACCOUNT} reads the repository of ${event.name} could not be settled — ${err instanceof Error ? err.message : err}`));
await actOn(client, event);
} });
await new Promise<void>((resolve, reject) => { await new Promise<void>((resolve, reject) => {
server.once("error", reject); server.once("error", reject);
server.listen(port, host, () => resolve()); server.listen(port, host, () => resolve());
@@ -594,6 +607,18 @@ if (gitea) {
// of the node starts left gitea_commit_status, gitea_merge_pull_request and gitea_events served nowhere. Until it is // of the node starts left gitea_commit_status, gitea_merge_pull_request and gitea_events served nowhere. Until it is
// listening, Pace keeps the polls at 30 s, so nothing is missed meanwhile; listenForForge says its own failure. // listening, Pace keeps the polls at 30 s, so nothing is missed meanwhile; listenForForge says its own failure.
void listenForForge(client); void listenForForge(client);
// The start's one pass over every repository (readall.ts), not awaited for the same reason. A forge not answering yet
// is asked again, a minute apart, until one pass has run; after that only the forge's events grant.
const readAllAtStart = (): void => void Promise.resolve().then(() => readAll().atStart()).catch((err) => {
console.error(`[gitea] whether ${TOOLS_ACCOUNT} reads every repository could not be settled; asked again in a minute — ${err instanceof Error ? err.message : err}`);
setTimeout(readAllAtStart, 60_000);
});
try {
readAll();
readAllAtStart();
} catch (err) {
console.error(`[gitea] ${TOOLS_ACCOUNT} is not given read on every repository: no site administrator's login — ${err instanceof Error ? err.message : err}`);
}
// Main takes no push and only the tools' account merges, in every repository, kept and not only when a token is // Main takes no push and only the tools' account merges, in every repository, kept and not only when a token is
// minted: at the start and every ten minutes after, so a repository made later gets it too (issue 948, ADR 0398 §2). // minted: at the start and every ten minutes after, so a repository made later gets it too (issue 948, ADR 0398 §2).
// Only while the setting keep-main-rules is true (mainrules.ts); false, the default, leaves every repository's rule as set. // Only while the setting keep-main-rules is true (mainrules.ts); false, the default, leaves every repository's rule as set.
+13
View File
@@ -15,6 +15,19 @@ export function keepMainRules(env: NodeJS.ProcessEnv = process.env): boolean {
} catch { return false; } } catch { return false; }
} }
/** The setting check-gates-merge (the operator, 2026-10-25 22:56: "run the check locally only, but not as a gate, it loses
* us too much time"): whether the merge tool judges mesh/check before it merges. Absent, unreadable or anything but false
* is true; false lets a head the operator's Approve names merge with mesh/check pending, failed, missing or judged on an
* older trunk (mesh-delivery-gate's setting of the same name stops waiting for it before the Approve). */
export function checkGatesMerge(env: NodeJS.ProcessEnv = process.env): boolean {
const file = env.MESH_GITEA_CONFIG_FILE;
if (!file) return true;
try {
const v = (JSON.parse(readFileSync(file, "utf8")) as Record<string, unknown>)["check-gates-merge"];
return !(v === false || v === "false");
} catch { return true; }
}
/** The 10-minute step: puts the rules right only when kept; else says once that they are left and writes nothing. */ /** The 10-minute step: puts the rules right only when kept; else says once that they are left and writes nothing. */
export function mainRulesStep(keep: boolean, put: () => Promise<string[]>, say: (line: string) => void): () => Promise<void> { export function mainRulesStep(keep: boolean, put: () => Promise<string[]>, say: (line: string) => void): () => Promise<void> {
let said = false; let said = false;
+8 -2
View File
@@ -19,6 +19,7 @@ import { mergeableUnder, newestByContext } from "./checks.js";
import { MERGE_METHODS, Refused, checkName } from "./guard.js"; import { MERGE_METHODS, Refused, checkName } from "./guard.js";
import { CHECK_CONTEXT, CHECK_STATUS, REPO_CHECK_CONTEXT, maintained, requiredOf } from "./pulls.js"; import { CHECK_CONTEXT, CHECK_STATUS, REPO_CHECK_CONTEXT, maintained, requiredOf } from "./pulls.js";
import { approvedHeadsOf, groupRefusal, memberKey, namesSiblings, notApproved, readSiblings, siblingOutOfDate, siblingsFile } from "./siblings.js"; import { approvedHeadsOf, groupRefusal, memberKey, namesSiblings, notApproved, readSiblings, siblingOutOfDate, siblingsFile } from "./siblings.js";
import { checkGatesMerge } from "./mainrules.js";
import { announcementOf, askedWords, outOfDateWords, sameCommit, tookFile, tookWords, twelve, verdictAtTrunk, waitsForComposed } from "./trunk.js"; import { announcementOf, askedWords, outOfDateWords, sameCommit, tookFile, tookWords, twelve, verdictAtTrunk, waitsForComposed } from "./trunk.js";
/** A head named for a merge: at least 12 hex characters of its sha (novox/hq issue 381), up to a sha-256 commit's 64. */ /** A head named for a merge: at least 12 hex characters of its sha (novox/hq issue 381), up to a sha-256 commit's 64. */
@@ -31,11 +32,13 @@ export interface MergeDeps {
emit?: (type: string, body: unknown) => Promise<void>; emit?: (type: string, body: unknown) => Promise<void>;
stateDir?: string; stateDir?: string;
now?: () => Date; now?: () => Date;
/** Whether mesh/check gates the merge: the setting check-gates-merge, read at each merge (mainrules.ts). */
checkGates?: () => boolean;
} }
export function mergeTool(gitea: GiteaClient, given: MergeDeps): ToolDefinition { export function mergeTool(gitea: GiteaClient, given: MergeDeps): ToolDefinition {
const deps = { ...given, emit: given.emit ?? announce, stateDir: given.stateDir ?? process.env.MESH_GITEA_STATE_DIR, const deps = { ...given, emit: given.emit ?? announce, stateDir: given.stateDir ?? process.env.MESH_GITEA_STATE_DIR,
now: given.now ?? (() => new Date()) }; now: given.now ?? (() => new Date()), checkGates: given.checkGates ?? (() => checkGatesMerge()) };
return { return {
name: "gitea_merge_pull_request", name: "gitea_merge_pull_request",
description: "Merge a pull request at the head you name — the commit you reviewed — when that is still its head, " + description: "Merge a pull request at the head you name — the commit you reviewed — when that is still its head, " +
@@ -126,7 +129,10 @@ export function mergeTool(gitea: GiteaClient, given: MergeDeps): ToolDefinition
} }
const kept = maintained(required); const kept = maintained(required);
const byCheck = required.includes(CHECK_STATUS) || kept.length === 0; const byCheck = required.includes(CHECK_STATUS) || kept.length === 0;
if (byCheck) { // **check-gates-merge false** (the operator, 2026-10-25): mesh/check is information, not a gate. A head the operator's
// Approve names merges whatever its mesh/check says, pending, failed, missing or judged on an older trunk; without
// approved_heads the check still judges.
if (byCheck && (deps.checkGates() || !approved)) {
// **mesh/check is the checker's** (ADR 0447 §4): it is judged here against the trunk's head, and refused when it // **mesh/check is the checker's** (ADR 0447 §4): it is judged here against the trunk's head, and refused when it
// is not current; nothing is written or asked — the checker checks a head again when its trunk moves. // is not current; nothing is written or asked — the checker checks a head again when its trunk moves.
const check = checks[CHECK_STATUS]; const check = checks[CHECK_STATUS];
+8 -1
View File
@@ -181,6 +181,12 @@
"trusted": true, "trusted": true,
"default": {} "default": {}
}, },
"check-gates-merge": {
"type": "boolean",
"description": "Whether gitea_merge_pull_request judges mesh/check before it merges. True (the default): a head merges only on a passing mesh/check judged against the trunk's head (ADR 0447, ADR 0534 §4). False (the operator, 2026-10-25 22:56: \"run the check locally only, but not as a gate, it loses us too much time\"): a head the operator's Approve names in approved_heads merges whatever its mesh/check says (pending, failed, missing or judged on an older trunk); a head that moved, a branch no rule guards and a merge git refuses are still refused. Set together with mesh-delivery-gate's check-gates-merge.",
"trusted": true,
"default": true
},
"main-mergers": { "main-mergers": {
"type": "list", "type": "list",
"description": "The forge logins that may merge into main beside the tools' account, on every repository of the organisation (ADR 0534 §2): the operator's own, so the operator can always merge by hand at the forge. The tools' account is always on main's merge whitelist; no agent's account belongs here.", "description": "The forge logins that may merge into main beside the tools' account, on every repository of the organisation (ADR 0534 §2): the operator's own, so the operator can always merge by hand at the forge. The tools' account is always on main's merge whitelist; no agent's account belongs here.",
@@ -316,7 +322,8 @@
"takes-config": [ "takes-config": [
"url", "url",
"keep-main-rules", "keep-main-rules",
"main-mergers" "main-mergers",
"check-gates-merge"
] ]
}, },
{ {
+210
View File
@@ -0,0 +1,210 @@
// The tools' account reads every repository on the forge (the operator, 2026-10-26: "the mesh's forge account should
// get read access to all repositories").
//
// Since issue 469 the tools act as the plain account `mesh-tools`, which writes to the organisation novox's
// repositories through its team (tools-account.ts). Every other repository it saw only when public: a private
// repository of a user (jschoubben/car-hunter, jschoubben/snake) answered its tools 404 and was missing from every
// list, while that user's public ones were there. So the module makes its account a collaborator that reads only on
// each repository it cannot read, the least the forge offers: one repository, read, no organisation membership.
//
// - **at the start**, one pass over every repository of every user and organisation;
// - **on the forge's own event** (the system webhook, webhook.ts): `forge.repository.created` and `forge.fork` at
// once, and the first event of any repository this process has not yet seen readable, since the forge may not
// send a repository's creation for every owner. Nothing polls for it.
//
// **Not caught: a repository made private later.** The forge sends no event for a change of visibility, and a public
// repository is already seen readable by the start's pass, so its events ask nothing. Such a repository is granted at
// the module's next start, or now by `gitea_grant_read`; `gitea_unreadable_repos` shows it meanwhile.
//
// **The grant needs the site administrator's login.** Adding a collaborator asks the repository's administration,
// which on another user's repository only the site administrator holds; the module holds that login for setting the
// forge up (MESH_GITEA_ADMIN_PASSWORD_FILE). Here it is used through `ReadAllForge` alone: list the repositories, read
// the account's permission on one, add the account as a collaborator that reads. Nothing else, and never for the tools'
// own calls, which keep their token.
//
// What the forge refuses (a repository that went meanwhile, an archived one if the forge will not take a collaborator
// on it) is said per repository and not retried; the tool `gitea_unreadable_repos` shows what is left, and
// `gitea_grant_read` grants again now.
import type { ForgeRepository } from "./client.js";
/** What the module asks of the forge for this, with the site administrator's login (GiteaAdmin, client.ts). */
export interface ReadAllForge {
everyRepository(): Promise<ForgeRepository[]>;
repository(owner: string, repo: string): Promise<ForgeRepository | null>;
repositoryPermission(owner: string, repo: string, user: string): Promise<string>;
addReadCollaborator(owner: string, repo: string, user: string): Promise<void>;
}
/** A repository the account cannot read, and why. */
export interface Unreadable {
repository: string;
permission: string;
private: boolean;
archived: boolean;
why: string;
}
export interface Granted {
/** Repositories looked at. */
checked: number;
/** Made readable now, by a read collaborator. */
granted: string[];
/** The forge refused the grant: each repository and the forge's own words. */
refused: { repository: string; why: string }[];
}
const READS = new Set(["read", "write", "admin", "owner"]);
/** Why the account cannot read a repository, from what the forge says of it. */
export function whyUnreadable(repo: ForgeRepository, account: string): string {
const parts: string[] = [];
if (repo.private) parts.push(`a private repository of ${repo.owner}, on which ${account} is no collaborator and in no team that covers it`);
else if (repo.owner_visibility !== "public") {
parts.push(`a public repository of ${repo.owner}, whose own visibility is ${repo.owner_visibility}: the forge shows it to members only`);
} else parts.push(`the forge answers ${account} no read on it`);
if (repo.archived) parts.push("archived");
return parts.join("; ");
}
export class ReadAll {
/** Repositories this process has seen the account read, by full name: an event of one of them asks nothing more. */
readonly readable = new Set<string>();
constructor(private readonly forge: ReadAllForge, readonly account: string,
private readonly log: (line: string) => void = (l) => console.log(`[gitea] ${l}`)) {}
/** Every repository the account cannot read, and why, as the forge says it now. */
async unreadable(): Promise<{ checked: number; unreadable: Unreadable[] }> {
const repos = await this.forge.everyRepository();
const out: Unreadable[] = [];
for (const repo of repos) {
try {
const u = await this.judge(repo);
if (u) out.push(u);
} catch (err) {
// A permission the forge would not answer is said for that repository; the rest are still looked at.
out.push({ repository: repo.full_name, permission: "unknown", private: repo.private, archived: repo.archived,
why: `the forge did not answer ${this.account}'s permission on it — ${err instanceof Error ? err.message : String(err)}` });
}
}
return { checked: repos.length, unreadable: out };
}
/** Grant read on every repository the account cannot read (or on those named, `owner/name`). */
async grant(only?: string[]): Promise<Granted> {
let repos: ForgeRepository[];
if (only && only.length) {
repos = [];
for (const full of only) {
const [owner, name, ...rest] = full.split("/");
if (!owner || !name || rest.length) throw new Error(`"${full}" is not a repository: say owner/name`);
const repo = await this.forge.repository(owner, name);
if (!repo) throw new Error(`the forge holds no repository ${full}`);
repos.push(repo);
}
} else repos = await this.forge.everyRepository();
const result: Granted = { checked: repos.length, granted: [], refused: [] };
for (const repo of repos) {
let u: Unreadable | null;
try {
u = await this.judge(repo);
} catch (err) {
// One repository whose permission cannot be read is a refusal of its own, never the end of the pass.
result.refused.push({ repository: repo.full_name, why: `its permission could not be read — ${err instanceof Error ? err.message : String(err)}` });
continue;
}
if (!u) continue;
try {
await this.forge.addReadCollaborator(repo.owner, repo.name, this.account);
} catch (err) {
result.refused.push({ repository: repo.full_name, why: err instanceof Error ? err.message : String(err) });
continue;
}
// Read back: the forge's own word that the account reads it now, never the grant's answer alone.
let after: string;
try {
after = await this.forge.repositoryPermission(repo.owner, repo.name, this.account);
} catch (err) {
result.refused.push({ repository: repo.full_name, why: `granted, but its permission could not be read back — ${err instanceof Error ? err.message : String(err)}` });
continue;
}
if (READS.has(after)) {
this.readable.add(repo.full_name);
result.granted.push(repo.full_name);
} else {
result.refused.push({ repository: repo.full_name, why: `the grant was answered, but the forge still gives ${this.account} "${after}"` });
}
}
return result;
}
/** The start's one pass: grant what is missing and say it. Answers the result; a failure is the caller's to say. */
async atStart(): Promise<Granted> {
const r = await this.grant();
this.log(`${this.account} reads every repository: ${r.checked} looked at, ${r.granted.length} granted read now` +
`${r.granted.length ? ` (${r.granted.join(", ")})` : ""}${r.refused.length ? `, ${r.refused.length} refused: ` +
r.refused.map((x) => `${x.repository} — ${x.why}`).join("; ") : ""}`);
return r;
}
/** A forge event: a repository made (or forked), or the first event of one not seen readable, is granted read now. */
async onEvent(event: { name: string; body: unknown }): Promise<void> {
const b = (event.body ?? {}) as { owner?: string; repo?: string };
if (!b.owner || !b.repo) return;
const full = `${b.owner}/${b.repo}`;
if (event.name === "forge.repository.deleted") {
this.readable.delete(full);
return;
}
const made = event.name === "forge.repository.created" || event.name === "forge.fork";
// Seen readable once, its events ask nothing: so a repository made private after that is not caught here (the
// forge says no change of visibility), only at the next start or by gitea_grant_read.
if (!made && this.readable.has(full)) return;
const r = await this.grant([full]);
if (r.granted.length) this.log(`${this.account} was given read on ${full}, on the forge's ${event.name}`);
for (const x of r.refused) this.log(`${this.account} could not be given read on ${x.repository} — ${x.why}`);
}
private async judge(repo: ForgeRepository): Promise<Unreadable | null> {
const permission = await this.forge.repositoryPermission(repo.owner, repo.name, this.account);
if (READS.has(permission)) {
this.readable.add(repo.full_name);
return null;
}
this.readable.delete(repo.full_name);
return { repository: repo.full_name, permission, private: repo.private, archived: repo.archived, why: whyUnreadable(repo, this.account) };
}
}
type Tool = { name: string; description: string; input: Record<string, unknown>; run: (args: Readonly<Record<string, unknown>>) => Promise<unknown> };
/** The two tools: what the account cannot read and why, and the grant now. */
export function readAllTools(readAll: () => ReadAll): Tool[] {
return [
{
name: "gitea_unreadable_repos",
description: "Every repository on the forge the tools' own account (mesh-tools) cannot read, and why: the " +
"permission the forge gives it there, whether the repository is private or archived, and in words the reason " +
"(a private repository of a user with the account no collaborator, an owner not public). Empty when it reads " +
"every repository. Read with the site administrator's login, which alone sees every repository.",
input: {},
run: async () => readAll().unreadable(),
},
{
name: "gitea_grant_read",
description: "Give the tools' own account (mesh-tools) read on every repository it cannot read now, or on the " +
"repositories named: each becomes a collaborator that reads only, the least the forge offers. The module does " +
"this itself at its start and on the forge's event of a repository made; this is the same act by hand. Answers " +
"how many were looked at, which were granted (each read back from the forge) and which the forge refused, in its " +
"own words. Needs the site administrator's login, used for this grant alone.",
input: {
repositories: { type: "array", items: { type: "string" }, description: "owner/name of each repository; empty or absent, every repository" },
},
run: async (args) => {
const named = Array.isArray(args.repositories) ? args.repositories.map(String) : undefined;
return readAll().grant(named);
},
},
];
}
+12
View File
@@ -525,6 +525,18 @@ test("gitea_merge_pull_request judges mesh/check on a main requiring no status,
state.statuses = line(TRUNK); state.statuses = line(TRUNK);
assert.equal(((await merge()) as Record<string, unknown>).merged, true); assert.equal(((await merge()) as Record<string, unknown>).merged, true);
// check-gates-merge false (the operator, 2026-10-25): a head the Approve names merges whatever its mesh/check says;
// without approved_heads it is refused as before.
const ungated = mergeTool(client, { gates, stateDir, emit: async () => {}, checkGates: () => false });
const mergeUngated = (args: Record<string, unknown> = { approved_heads }) =>
ungated.run({ owner: "novox", repo: "mesh-catalog", number: 7, head: REVIEWED, ...args });
for (const statuses of [[], line(TRUNK, "pending", "checking"), line(TRUNK, "failure", "fail"), line(MOVED)]) {
state.merged = false;
state.statuses = statuses;
await assert.rejects(mergeUngated({}), /Approve/);
assert.equal(((await mergeUngated()) as Record<string, unknown>).merged, true, JSON.stringify(statuses));
}
}); });
test("a rule named for a branch is not made weaker than the glob rule that guards it now", async () => { test("a rule named for a branch is not made weaker than the glob rule that guards it now", async () => {
+144 -1
View File
@@ -26,6 +26,18 @@ interface Account { admin: boolean; password: string; active: boolean }
interface Token { id: number; name: string; sha1: string; user: string; scopes: string[] } interface Token { id: number; name: string; sha1: string; user: string; scopes: string[] }
interface Team { id: number; name: string; units: any; members: Set<string> } interface Team { id: number; name: string; units: any; members: Set<string> }
interface ForgeRepo { owner: string; name: string; full: string; private: boolean; archived?: boolean; ownerVisibility: string; collaborators: Map<string, string> }
const forgeRepos: ForgeRepo[] = [];
const repoFaults = { refuse: new Set<string>(), permission: new Set<string>() };
function addRepo(full: string, isPrivate: boolean, ownerVisibility = "public"): ForgeRepo {
const [owner, name] = full.split("/");
const r = { owner, name, full, private: isPrivate, ownerVisibility, collaborators: new Map<string, string>() };
forgeRepos.push(r);
return r;
}
const repoJSON = (r: ForgeRepo) => ({ full_name: r.full, name: r.name, owner: { login: r.owner, visibility: r.ownerVisibility },
private: r.private, archived: r.archived === true });
function fakeForge() { function fakeForge() {
const users = new Map<string, Account>([[ADMIN, { admin: true, password: PASSWORD, active: true }]]); const users = new Map<string, Account>([[ADMIN, { admin: true, password: PASSWORD, active: true }]]);
const tokens: Token[] = []; const tokens: Token[] = [];
@@ -61,6 +73,13 @@ function fakeForge() {
return null; return null;
}; };
const isAdmin = (u: string | null) => u !== null && users.get(u)?.admin === true; const isAdmin = (u: string | null) => u !== null && users.get(u)?.admin === true;
const permissionOf = (u: string | null, r: ForgeRepo): string => {
if (isAdmin(u)) return "admin";
if (u === r.owner) return "owner";
if (r.owner === "novox" && teams.some((t) => t.members.has(u ?? "") && t.units.units_map?.["repo.code"] === "write")) return "write";
if (u && r.collaborators.has(u)) return r.collaborators.get(u)!;
return !r.private && r.ownerVisibility === "public" ? "read" : "none";
};
const server = createServer(async (req, res) => { const server = createServer(async (req, res) => {
const url = new URL(req.url ?? "/", "http://fake"); const url = new URL(req.url ?? "/", "http://fake");
@@ -176,7 +195,32 @@ function fakeForge() {
if (!writes) return json(res, 403, { message: "write on code required" }); if (!writes) return json(res, 403, { message: "write on code required" });
return json(res, 201, { context: (await read(req)).context, creator: { login: me } }); return json(res, 201, { context: (await read(req)).context, creator: { login: me } });
} }
if (req.method === "GET" && p === "/repos/search") return json(res, 200, { data: [] }); // Every repository of every owner, and who reads it (readall.ts): the site administrator reads all; the tools' account
// a repository of novox through its team, a public one of a public owner, and one it is a collaborator of.
if (req.method === "GET" && p === "/repos/search") {
const page = Number(url.searchParams.get("page") ?? "1");
const seen = forgeRepos.filter((r) => permissionOf(me, r) !== "none");
return json(res, 200, { data: page === 1 ? seen.map(repoJSON) : [] });
}
m = p.match(/^\/repos\/([^/]+)\/([^/]+)(?:\/collaborators\/([^/]+)(\/permission)?)?$/);
if (m) {
const repo = forgeRepos.find((r) => r.owner === decodeURIComponent(m![1]) && r.name === decodeURIComponent(m![2]));
if (!repo || permissionOf(me, repo) === "none") return json(res, 404, { message: "repository does not exist" });
if (req.method === "GET" && !m[3]) return json(res, 200, repoJSON(repo));
const user = decodeURIComponent(m[3] ?? "");
if (req.method === "GET" && m[4]) {
if (repoFaults.permission.has(repo.full)) return json(res, 500, { message: "the forge is not well" });
if (!isAdmin(me) && me !== user) return json(res, 403, { message: "only the user or an administrator" });
return json(res, 200, { permission: permissionOf(user, repo) });
}
if (req.method === "PUT" && !m[4]) {
// gitea's reqAdmin: the repository's administration, which on a repository of another user the site administrator alone holds.
if (!isAdmin(me) && me !== repo.owner) return json(res, 403, { message: "user should be an owner or a collaborator with admin write of a repository" });
if (repoFaults.refuse.has(repo.full)) return json(res, 422, { message: "the forge would not have it" });
repo.collaborators.set(user, (await read(req))?.permission ?? "write");
return json(res, 204, null);
}
}
return json(res, 404, { message: "no such route in the fake" }); return json(res, 404, { message: "no such route in the fake" });
}); });
return new Promise<{ url: string; users: typeof users; tokens: typeof tokens; teams: typeof teams; calls: string[]; mainRules: typeof mainRules; return new Promise<{ url: string; users: typeof users; tokens: typeof tokens; teams: typeof teams; calls: string[]; mainRules: typeof mainRules;
@@ -402,3 +446,102 @@ test("a mint with keep-main-rules false (the default) writes no branch protectio
const writes = forge.calls.slice(from).filter((c: string) => /branch_protections/.test(c) && !/^GET /.test(c) && !/^(GET|HEAD) /.test(c)); const writes = forge.calls.slice(from).filter((c: string) => /branch_protections/.test(c) && !/^GET /.test(c) && !/^(GET|HEAD) /.test(c));
assert.deepEqual(writes, []); assert.deepEqual(writes, []);
}); });
// The tools' account reads every repository (readall.ts; the operator, 2026-10-26): jschoubben/car-hunter and
// jschoubben/snake, private repositories of a user, answered the tools 404 while that user's public ones were seen.
test("the start's pass gives the tools' account read, as a collaborator that reads only, on each repository it could not read, and on no other", async () => {
const env = await environment();
await fresh(env).listRepos(); // the account and its team exist, as after any mint
forgeRepos.splice(0);
addRepo("novox/hq", true);
addRepo("jschoubben/txt-game", false);
addRepo("jschoubben/car-hunter", true);
addRepo("jschoubben/snake", true);
addRepo("shy/open", false, "limited");
addRepo("elsewhere/secret", true);
const { GiteaAdmin } = await import("../dist/client.js");
const { ReadAll, readAllTools } = await import("../dist/readall.js");
const lines: string[] = [];
const readAll = new ReadAll(GiteaAdmin.fromEnv(env), TOOLS_ACCOUNT, (l: string) => lines.push(l));
const [unreadableTool, grantTool] = readAllTools(() => readAll);
// What the tool says before: the four the account cannot read, each with why.
const before = await unreadableTool.run({}) as any;
assert.equal(before.checked, 6);
assert.deepEqual(before.unreadable.map((u: any) => u.repository).sort(),
["elsewhere/secret", "jschoubben/car-hunter", "jschoubben/snake", "shy/open"]);
assert.match(before.unreadable.find((u: any) => u.repository === "jschoubben/car-hunter").why,
/a private repository of jschoubben, on which mesh-tools is no collaborator/);
assert.match(before.unreadable.find((u: any) => u.repository === "shy/open").why, /visibility is limited/);
assert.deepEqual((await fresh(env).listAllRepos()).map((r) => r.full_name).sort(), ["jschoubben/txt-game", "novox/hq"]);
// The account's own token cannot grant itself: the forge asks the repository's administration (the reason for the admin login).
const own = (await readFile(join(env.MESH_GITEA_STATE_DIR, "token"), "utf8")).trim();
const self = await fetch(`${forge.url}/api/v1/repos/jschoubben/car-hunter/collaborators/${TOOLS_ACCOUNT}`,
{ method: "PUT", headers: { Authorization: `token ${own}`, "Content-Type": "application/json" }, body: JSON.stringify({ permission: "read" }) });
assert.equal(self.status, 404, "a repository the account cannot read is not even there for it");
const from = forge.calls.length;
const r = await readAll.atStart();
assert.deepEqual(r.granted.sort(), ["elsewhere/secret", "jschoubben/car-hunter", "jschoubben/snake", "shy/open"]);
assert.deepEqual(r.refused, []);
const puts = forge.calls.slice(from).filter((c) => c.startsWith("PUT "));
assert.equal(puts.length, 4, "the repositories it read already were left alone");
assert.ok(puts.every((c) => c.endsWith(` as ${ADMIN}`)), "only the administrator's login grants");
for (const repo of forgeRepos.filter((x) => x.collaborators.size)) assert.equal(repo.collaborators.get(TOOLS_ACCOUNT), "read", `${repo.full} reads only`);
assert.match(lines.join("\n"), /6 looked at, 4 granted read now/);
// After: the account's own token sees every repository, and the tool names none.
assert.equal((await fresh(env).listAllRepos()).length, 6);
assert.deepEqual(((await unreadableTool.run({})) as any).unreadable, []);
// Granting again asks nothing of the forge but reads.
const again = forge.calls.length;
assert.deepEqual(((await grantTool.run({})) as any).granted, []);
assert.ok(forge.calls.slice(again).every((c) => c.startsWith("GET ")));
});
test("on the forge's events: a repository made is granted read at once, a known readable one asks nothing, a refusal is said, not thrown", async () => {
const env = await environment();
await fresh(env).listRepos();
forgeRepos.splice(0);
addRepo("jschoubben/txt-game", false);
const { GiteaAdmin } = await import("../dist/client.js");
const { ReadAll, readAllTools } = await import("../dist/readall.js");
const lines: string[] = [];
const readAll = new ReadAll(GiteaAdmin.fromEnv(env), TOOLS_ACCOUNT, (l: string) => lines.push(l));
await readAll.atStart();
const cases: { name: string; event: string; repo: string; make?: boolean; refuse?: boolean; unanswered?: boolean; asks: boolean; reads: string }[] = [
{ name: "a repository made", event: "forge.repository.created", repo: "jschoubben/new-private", make: true, asks: true, reads: "read" },
{ name: "a fork made", event: "forge.fork", repo: "someone/fork", make: true, asks: true, reads: "read" },
{ name: "a push to a repository seen readable", event: "forge.push", repo: "jschoubben/txt-game", asks: false, reads: "read" },
{ name: "a push to a private repository never seen", event: "forge.push", repo: "jschoubben/quiet", make: true, asks: true, reads: "read" },
{ name: "a repository the forge refuses the grant on", event: "forge.repository.created", repo: "jschoubben/stuck", make: true, refuse: true, asks: true, reads: "none" },
{ name: "a repository whose permission the forge will not answer", event: "forge.repository.created", repo: "jschoubben/mute", make: true, unanswered: true, asks: true, reads: "none" },
];
for (const c of cases) {
if (c.make) addRepo(c.repo, true);
if (c.refuse) repoFaults.refuse.add(c.repo);
if (c.unanswered) repoFaults.permission.add(c.repo);
const from = forge.calls.length;
const [owner, repo] = c.repo.split("/");
await readAll.onEvent({ name: c.event, body: { owner, repo } });
assert.equal(forge.calls.length > from, c.asks, `${c.name}: asks the forge`);
assert.equal(forgeRepos.find((x) => x.full === c.repo)!.collaborators.get(TOOLS_ACCOUNT) ?? (c.make ? "none" : "read"), c.reads, c.name);
}
assert.match(lines.join("\n"), /could not be given read on jschoubben\/stuck — .*422/);
const named = await (readAllTools(() => readAll)[1].run({ repositories: ["jschoubben/stuck"] })) as any;
assert.deepEqual(named.refused.map((x: any) => x.repository), ["jschoubben/stuck"]);
await assert.rejects(readAllTools(() => readAll)[1].run({ repositories: ["no-slash"] }), /not a repository/);
assert.match(lines.join("\n"), /could not be given read on jschoubben\/mute — its permission could not be read — .*500/);
// A whole pass goes past the repository it cannot judge: every other one is still granted, and the listing names it.
addRepo("jschoubben/after-mute", true);
const pass = await readAll.grant();
assert.ok(pass.granted.includes("jschoubben/after-mute"), "the pass did not stop at the repository it could not judge");
assert.ok(pass.refused.some((x: any) => x.repository === "jschoubben/mute" && /permission could not be read/.test(x.why)));
const listed = await readAllTools(() => readAll)[0].run({}) as any;
assert.equal(listed.unreadable.find((u: any) => u.repository === "jschoubben/mute").permission, "unknown");
repoFaults.refuse.clear();
repoFaults.permission.clear();
});
+3
View File
@@ -34,6 +34,9 @@ test("both statuses say the head and the trunk commit the verdict judged, first,
const [old] = statusesFor({ ...c, trunk: undefined, gate: { verdict: "pass", summary: "every machine composes" } }); const [old] = statusesFor({ ...c, trunk: undefined, gate: { verdict: "pass", summary: "every machine composes" } });
assert.equal(old.description, "pass: every machine composes"); assert.equal(old.description, "pass: every machine composes");
assert.equal(trunkNamed(old.description), undefined); assert.equal(trunkNamed(old.description), undefined);
// ADR 0548 §8: a carried verdict's line names the head on the new trunk first, so the merge tool merges it on that trunk.
assert.equal(trunkNamed(`pass ${HEAD.slice(0, 12)} on ${MOVED.slice(0, 12)} | carried from ${TRUNK.slice(0, 12)}, mesh-catalog 0123456789ab | unit 0 run, 4 kept`),
MOVED.slice(0, 12));
}); });
test("a trunk's move puts an open head's merge check out of date, naming both commits, unless a check is on its way", () => { test("a trunk's move puts an open head's merge check out of date, naming both commits, unless a check is on its way", () => {
+2 -2
View File
@@ -508,11 +508,11 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
if (!owner || !repo) throw new Error("name the repository: owner and repo"); if (!owner || !repo) throw new Error("name the repository: owner and repo");
if (!Number.isInteger(number) || number < 0) throw new Error(`${args.number} is not a pull request's number`); if (!Number.isInteger(number) || number < 0) throw new Error(`${args.number} is not a pull request's number`);
let pull: { number: number; title: string; state: string; merged: boolean; base?: string; head?: string; let pull: { number: number; title: string; state: string; merged: boolean; base?: string; head?: string;
head_sha?: string; merge_commit_sha?: string; html_url: string } | undefined; head_sha?: string; merge_commit_sha?: string; merged_at?: string; html_url: string } | undefined;
if (number > 0) { if (number > 0) {
const p = await gitea.getPullRequest(owner, repo, number); const p = await gitea.getPullRequest(owner, repo, number);
pull = { number: p.number, title: p.title, state: p.state, merged: p.merged, base: p.base, head: p.head, pull = { number: p.number, title: p.title, state: p.state, merged: p.merged, base: p.base, head: p.head,
head_sha: p.head_sha, merge_commit_sha: p.merge_commit_sha, html_url: p.html_url }; head_sha: p.head_sha, merge_commit_sha: p.merge_commit_sha, merged_at: p.merged_at, html_url: p.html_url };
if (!ref) ref = p.head_sha ?? ""; if (!ref) ref = p.head_sha ?? "";
if (!base) base = p.base ?? ""; if (!base) base = p.base ?? "";
if (!ref) throw new Error(`${owner}/${repo}#${number} names no head commit`); if (!ref) throw new Error(`${owner}/${repo}#${number} names no head commit`);
@@ -14,6 +14,7 @@ var Kept = []announce.State{
{Name: "toolchains", Readable: true}, {Name: "toolchains", Readable: true},
{Name: "keys", Readable: true}, {Name: "keys", Readable: true},
{Name: "steps", Readable: true}, {Name: "steps", Readable: true},
{Name: "pauses", Readable: true},
} }
// keepStates announces Kept as this module's (stdio.Keeps): main calls it before it serves. // keepStates announces Kept as this module's (stdio.Keeps): main calls it before it serves.
@@ -80,19 +80,23 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
source = buildstep.SeatAsks{Language: delivery.AskGo} source = buildstep.SeatAsks{Language: delivery.AskGo}
} }
capacity := m.Capacity capacity := m.Capacity
// The capacity's own change wakes the puller (issue 1887): the runtime's watch of the config value as mesh-config
// resolved it; the settings file is read only when that watch cannot start. A test's Capacity is its own.
var watch func(string, func(stdio.ProvisionChange) error) error
if capacity == nil { if capacity == nil {
capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say) capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say)
watch = buildstep.WatchConfigServed
} }
ask := m.askSteps(steps) ask := m.askSteps(steps)
var load func(string) ([]byte, error) var load func(string) ([]byte, error)
if m.ReadSteps.Get != nil { if m.ReadPauses.Get != nil {
load = func(key string) ([]byte, error) { load = func(key string) ([]byte, error) {
raw, err := m.ReadSteps.Get(key) raw, err := m.ReadPauses.Get(key)
return raw, err return raw, err
} }
} }
return &buildstep.Puller{Source: source, Language: delivery.AskGo, Node: m.Node, Put: m.Steps, Load: load, return &buildstep.Puller{Source: source, Language: delivery.AskGo, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load,
Capacity: capacity, Logf: m.Say, Capacity: capacity, WatchConfig: watch, Logf: m.Say,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { Run: func(ctx context.Context, a delivery.StepAsk) (any, error) {
if err := m.Check(ctx); err != nil { if err := m.Check(ctx); err != nil {
return nil, err return nil, err
@@ -111,3 +115,11 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
func (m *Module) SeatTools(p *buildstep.Puller) []stdio.Tool { func (m *Module) SeatTools(p *buildstep.Puller) []stdio.Tool {
return buildstep.SeatTools(Seat, "Go", p) return buildstep.SeatTools(Seat, "Go", p)
} }
// StepEndTool ends one step left running (issue 1890); the module's steps state is the one it reads and writes.
const StepEndTool = "build_go_step_end"
// stepScan is the module's steps state as buildstep reads it.
func (m *Module) stepScan() buildstep.Scan {
return buildstep.ScanOf(stdio.State(delivery.StepsState))
}
+202 -15
View File
@@ -10,6 +10,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"regexp" "regexp"
"slices"
"sort" "sort"
"strings" "strings"
"time" "time"
@@ -40,6 +41,8 @@ type PackageResult struct {
Tests int `json:"tests"` Tests int `json:"tests"`
// NoTests is a package with no test files (at this level: a file with the other level's build tag is none). // NoTests is a package with no test files (at this level: a file with the other level's build tag is none).
NoTests bool `json:"no_tests,omitempty"` NoTests bool `json:"no_tests,omitempty"`
// Skipped is how many of the package's tests skipped: a pass that skipped proved less than it says, so the checker keeps no verdict for it.
Skipped int `json:"skipped,omitempty"`
} }
// TestTime is one test and the time it took. // TestTime is one test and the time it took.
@@ -52,6 +55,10 @@ type TestTime struct {
// Failure is a test that failed, or a package that failed with no test to name (a build that failed, a TestMain that // Failure is a test that failed, or a package that failed with no test to name (a build that failed, a TestMain that
// panicked); Output is the last lines it wrote, up to outputBytes (issue 1901), and Cause the first line of what the // panicked); Output is the last lines it wrote, up to outputBytes (issue 1901), and Cause the first line of what the
// test reported as its error (t.Error, t.Fatal), without its file and line, when go test said which lines those were. // test reported as its error (t.Error, t.Fatal), without its file and line, when go test said which lines those were.
// **A failure that is not one test's assertion carries its real cause too** (issue 1901, its third meeting): the
// "panic:" line (a timeout's with the tests it found running), a "fatal error:" or "signal:" line, a build's first
// error, or in words that the test program exited, while which tests ran or after its tests passed. Test then names the
// tests that were running when the program stopped. A "PASS" or "ok" line is never a failure's words.
type Failure struct { type Failure struct {
Package string `json:"package"` Package string `json:"package"`
Test string `json:"test,omitempty"` Test string `json:"test,omitempty"`
@@ -126,11 +133,18 @@ type reader struct {
causes map[string]string // the first error line of a test, by the same key causes map[string]string // the first error line of a test, by the same key
times []TestTime times []TestTime
failedIn map[string]bool // packages with a failed test failedIn map[string]bool // packages with a failed test
// running is the tests that started and have not ended, by package, in the order they started: when a package
// fails with no failed test, these were running when its test program stopped (issue 1901).
running map[string][]string
// passed says a package's test program printed "PASS" (its tests passed) before it failed; built says its test
// program failed to build.
passed, built map[string]bool
} }
func newReader(res *TestResult, dir string) *reader { func newReader(res *TestResult, dir string) *reader {
return &reader{res: res, dir: dir, pkgs: map[string]*PackageResult{}, builds: map[string][]string{}, return &reader{res: res, dir: dir, pkgs: map[string]*PackageResult{}, builds: map[string][]string{},
pkgOut: map[string][]string{}, testOut: map[string][]string{}, causes: map[string]string{}, failedIn: map[string]bool{}} pkgOut: map[string][]string{}, testOut: map[string][]string{}, causes: map[string]string{}, failedIn: map[string]bool{},
running: map[string][]string{}, passed: map[string]bool{}, built: map[string]bool{}}
} }
// read takes go test's output. Lines that are not events are kept for a failure that has nothing else to say. // read takes go test's output. Lines that are not events are kept for a failure that has nothing else to say.
@@ -178,12 +192,21 @@ func (r *reader) event(ev event) {
p := r.pkg(ev.Package) p := r.pkg(ev.Package)
switch ev.Action { switch ev.Action {
case "output": case "output":
r.pkgOut[ev.Package] = keep(r.pkgOut[ev.Package], ev.Output) // go test's own summary of a package ("PASS", "FAIL", "ok <pkg>", "FAIL <pkg>") is never a failure's
// words: a "PASS" before a "FAIL <pkg>" says only that the tests passed and the program failed after them.
switch l := strings.TrimRight(ev.Output, "\r\n"); {
case l == "PASS":
r.passed[ev.Package] = true
case summaryLine(l):
default:
r.pkgOut[ev.Package] = keep(r.pkgOut[ev.Package], ev.Output)
}
case "pass": case "pass":
p.Passed, p.Elapsed = true, ev.Elapsed p.Passed, p.Elapsed = true, ev.Elapsed
case "fail": case "fail":
p.Passed, p.Elapsed = false, ev.Elapsed p.Passed, p.Elapsed = false, ev.Elapsed
if ev.FailedBuild != "" { if ev.FailedBuild != "" {
r.built[ev.Package] = true
r.pkgOut[ev.Package] = append(r.builds[ev.FailedBuild], r.pkgOut[ev.Package]...) r.pkgOut[ev.Package] = append(r.builds[ev.FailedBuild], r.pkgOut[ev.Package]...)
} }
case "skip": case "skip":
@@ -197,26 +220,37 @@ func (r *reader) event(ev event) {
if ev.OutputType == "error" && r.causes[key] == "" { if ev.OutputType == "error" && r.causes[key] == "" {
r.causes[key] = errorWords(ev.Output) r.causes[key] = errorWords(ev.Output)
} }
case "run":
r.running[ev.Package] = append(r.running[ev.Package], ev.Test)
case "pass", "fail", "skip": case "pass", "fail", "skip":
r.running[ev.Package] = slices.DeleteFunc(r.running[ev.Package], func(t string) bool { return t == ev.Test })
p := r.pkg(ev.Package) p := r.pkg(ev.Package)
p.Tests++ p.Tests++
r.res.Tests++ r.res.Tests++
if ev.Action == "skip" { if ev.Action == "skip" {
r.res.Skipped++ r.res.Skipped++
p.Skipped++
} }
r.times = append(r.times, TestTime{Package: ev.Package, Test: ev.Test, Elapsed: ev.Elapsed}) r.times = append(r.times, TestTime{Package: ev.Package, Test: ev.Test, Elapsed: ev.Elapsed})
if ev.Action == "fail" { if ev.Action == "fail" {
r.failedIn[ev.Package] = true r.failedIn[ev.Package] = true
r.res.Failed = append(r.res.Failed, Failure{Package: ev.Package, Test: ev.Test, Output: clip(r.testOut[key], r.causes[key]), Cause: r.causes[key]}) cause := r.causes[key]
if race := raceCause(r.testOut[key]); race != "" && (cause == "" || strings.HasPrefix(cause, "race detected")) {
cause = race // the race itself, not testing's "race detected during execution of test"
} else if cause == "" {
cause = causeOf(r.testOut[key])
}
r.res.Failed = append(r.res.Failed, Failure{Package: ev.Package, Test: ev.Test, Output: clip(r.testOut[key], cause), Cause: cause})
} }
} }
} }
} }
// clip joins the last lines of a failure's output that fit outputBytes, each cut to outputLineBytes, and says how many // clip joins the last lines of a failure's output that fit outputBytes, each cut to outputLineBytes, and says how many
// earlier lines it left out. **When it leaves lines out, its first line is still a real one** (issue 1901/AC3): the failure's // earlier lines it left out. **Its first line is the failure's cause** (issue 1901/AC3, AC4), put ahead when the kept
// cause, or failing that the first non-empty line of the output, then the marker. An older checker takes only the first // lines do not start with it, or, with no cause and lines left out, the first non-empty line of the output, then the
// non-empty line of the output, and a marker there would be less than the older pair kept. // marker. An older checker takes only the first non-empty line of the output, and a marker there would be less than the
// older pair kept.
func clip(lines []string, cause string) string { func clip(lines []string, cause string) string {
size, from := 0, len(lines) size, from := 0, len(lines)
for from > 0 { for from > 0 {
@@ -227,15 +261,15 @@ func clip(lines []string, cause string) string {
size += n size += n
from-- from--
} }
kept := make([]string, 0, len(lines)-from+1) kept := make([]string, 0, len(lines)-from+2)
head := strings.TrimSpace(cause)
for i := 0; head == "" && i < from; i++ {
head = strings.TrimSpace(lines[i])
}
if head != "" && (from > 0 || len(lines) == 0 || strings.TrimSpace(lines[from]) != head) {
kept = append(kept, cutLine(head))
}
if from > 0 { if from > 0 {
head := strings.TrimSpace(cause)
for i := 0; head == "" && i < from; i++ {
head = strings.TrimSpace(lines[i])
}
if head != "" {
kept = append(kept, cutLine(head))
}
kept = append(kept, fmt.Sprintf("... (%d earlier lines)", from)) kept = append(kept, fmt.Sprintf("... (%d earlier lines)", from))
} }
for _, l := range lines[from:] { for _, l := range lines[from:] {
@@ -244,6 +278,159 @@ func clip(lines []string, cause string) string {
return strings.Join(kept, "\n") return strings.Join(kept, "\n")
} }
// summaryLine says a line is go test's summary of a package, in go's exact forms only: "PASS" or "FAIL" alone,
// "ok \t<pkg>\t<time>" and "FAIL\t<pkg>\t<time>" (review round 1: a line of the package's own that starts "ok " is kept).
func summaryLine(l string) bool {
return l == "PASS" || l == "FAIL" || strings.HasPrefix(l, "ok \t") || strings.HasPrefix(l, "FAIL\t")
}
// causeOf is the cause in a failure's output that is not a test's assertion (issue 1901): a race the race detector
// reported (raceCause); else the first "panic:" line, a timeout's with the tests it names as running; else the first
// "fatal error:" (the runtime's) or "signal:" line (a test program killed); "" when there is none. It reads every line,
// so a long goroutine dump after the panic cannot hide it. **It matches only lines at column 0**, where go prints the
// runtime's: a test's indented log line holding the same words is not the cause (review round 1).
func causeOf(lines []string) string {
if race := raceCause(lines); race != "" {
return race
}
for _, prefix := range []string{"panic: ", "fatal error: ", "signal: "} {
for i, l := range lines {
l = strings.TrimRight(l, "\r\n") // at column 0: an indented line is a test's own log, not the runtime's
if !strings.HasPrefix(l, prefix) {
continue
}
if strings.HasPrefix(l, "panic: test timed out after") {
var names []string
for _, n := range lines[i+1:] {
if !strings.HasPrefix(n, "\t\t") {
if strings.TrimSpace(n) == "running tests:" {
continue
}
break
}
names = append(names, strings.TrimSpace(n))
}
if len(names) > 0 {
l += "; running tests: " + strings.Join(names, ", ")
}
}
return cutLine(l)
}
}
return ""
}
// raceCause is the first race the race detector reported in a failure's output, in one line (issue 1901: a race after
// the tests' "PASS" was worded "PASS"): its access and the frame that made it, and where the goroutine that made it was
// created, as "WARNING: DATA RACE: Write at 0x... by goroutine 10 in pkg.F.func1() (f_test.go:13); goroutine created at
// pkg.F() (f_test.go:13)". "" when the output holds no race.
func raceCause(lines []string) string {
at := -1
for i, l := range lines {
if strings.TrimRight(l, "\r\n") == "WARNING: DATA RACE" { // at column 0, as causeOf
at = i
break
}
}
if at < 0 {
return ""
}
// frame is the function and the file's base name and line that follow a heading line at i.
frame := func(i int) string {
var fn, file string
for _, l := range lines[i+1:] {
if l = strings.TrimSpace(l); l == "" {
break
}
if fn == "" {
fn = l
continue
}
file = l
if j := strings.LastIndexByte(file, '/'); j >= 0 {
file = file[j+1:]
}
if j := strings.Index(file, " +0x"); j >= 0 {
file = file[:j]
}
break
}
if file != "" {
return fn + " (" + file + ")"
}
return fn
}
words := "WARNING: DATA RACE"
for i := at + 1; i < len(lines); i++ {
l := strings.TrimSpace(lines[i])
if l == "" || strings.HasPrefix(l, "===") {
continue
}
words += ": " + strings.TrimSuffix(l, ":")
if f := frame(i); f != "" {
words += " in " + f
}
break
}
for i := at + 1; i < len(lines); i++ {
l := strings.TrimSpace(lines[i])
if strings.HasPrefix(l, "==================") {
break
}
if strings.HasPrefix(l, "Goroutine ") && strings.HasSuffix(l, " created at:") {
if f := frame(i); f != "" {
words += "; its goroutine created at " + f
}
break
}
}
return cutLine(words)
}
// buildCause is the first error line of a failed build: the first line that is not go's "# <package>" heading.
func buildCause(lines []string) string {
for _, l := range lines {
if l = strings.TrimSpace(l); l != "" && !strings.HasPrefix(l, "# ") {
return cutLine(l)
}
}
return ""
}
// testNames names the tests that were running, at most three, and how many more.
func testNames(tests []string) string {
if len(tests) <= 3 {
return strings.Join(tests, ", ")
}
return fmt.Sprintf("%s and %d more", strings.Join(tests[:3], ", "), len(tests)-3)
}
// packageFailure is a package that failed with no failed test to name it (issue 1901): its test program did not build,
// panicked, timed out, was killed or exited. It names the tests that were running when the program stopped, keeps
// their output with the package's, and says the cause.
func (r *reader) packageFailure(name string) Failure {
running := r.running[name]
var lines []string
for _, t := range running {
lines = append(lines, r.testOut[name+"\x00"+t]...)
}
lines = append(lines, r.pkgOut[name]...)
cause := causeOf(lines)
switch {
case cause != "":
case r.built[name]:
cause = buildCause(lines)
case len(running) > 0:
cause = "the test program exited while " + testNames(running) + " ran, with no panic: an os.Exit or a signal (go test -json reports no exit status)"
case r.passed[name]:
cause = "the tests passed, then the test program exited with a failure: an os.Exit or a panic in TestMain or after the tests (go test -json reports no exit status)"
}
if cause == "" {
cause = "the package failed with no test to name and no panic, signal or build error in its output"
}
return Failure{Package: name, Test: testNames(running), Output: clip(lines, cause), Cause: cause}
}
// cutLine cuts a line to outputLineBytes, on a character boundary. // cutLine cuts a line to outputLineBytes, on a character boundary.
func cutLine(l string) string { func cutLine(l string) string {
if len(l) <= outputLineBytes { if len(l) <= outputLineBytes {
@@ -265,7 +452,7 @@ func (r *reader) finish() {
for _, name := range r.order { for _, name := range r.order {
p := r.pkgs[name] p := r.pkgs[name]
if !p.Passed && !r.failedIn[name] { if !p.Passed && !r.failedIn[name] {
r.res.Failed = append(r.res.Failed, Failure{Package: name, Output: clip(r.pkgOut[name], "")}) r.res.Failed = append(r.res.Failed, r.packageFailure(name))
} }
if !p.NoTests && p.Elapsed > r.res.Elapsed { if !p.NoTests && p.Elapsed > r.res.Elapsed {
r.res.Elapsed = p.Elapsed r.res.Elapsed = p.Elapsed
@@ -8,9 +8,11 @@ package main
import ( import (
"context" "context"
"fmt" "fmt"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"os" "os"
"os/user" "os/user"
"strings" "strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -40,6 +42,8 @@ func main() {
Emit: func(t string, b any) error { return stdio.Emit(t, b) }, Emit: func(t string, b any) error { return stdio.Emit(t, b) },
Steps: stdio.State(delivery.StepsState).Put, Steps: stdio.State(delivery.StepsState).Put,
ReadSteps: readOf(stdio.State(delivery.StepsState)), ReadSteps: readOf(stdio.State(delivery.StepsState)),
Pauses: stdio.State(delivery.PausesState).Put,
ReadPauses: readOf(stdio.State(delivery.PausesState)),
ReadToolchainsState: readOf(stdio.State(delivery.ToolchainsState)), ReadToolchainsState: readOf(stdio.State(delivery.ToolchainsState)),
Ctx: ctx, Ctx: ctx,
Say: logf, Say: logf,
@@ -68,7 +72,13 @@ func main() {
os.Exit(1) os.Exit(1)
} }
steps := m.StepTools() steps := m.StepTools()
booted := time.Now()
puller := m.Puller(steps, nil) puller := m.Puller(steps, nil)
m.Held = puller
if m.Node != "" {
// A step a previous process left running is ended (issue 1890); this process's own steps start after booted.
go buildstep.EndStaleAtStart(ctx, m.stepScan(), m.Node, booted, logf)
}
if m.Node != "" { if m.Node != "" {
// The seat's asks, pulled once Serve has the runtime's channel; a pull before it is said and tried again. // The seat's asks, pulled once Serve has the runtime's channel; a pull before it is said and tried again.
go func() { _ = puller.Pull(ctx) }() go func() { _ = puller.Pull(ctx) }()
@@ -92,9 +92,13 @@ func TestManifestKeepsItsStatesClaimsItsBuildSeatAndDeclaresItsHealthTool(t *tes
states := []string{} states := []string{}
for _, s := range m.State { for _, s := range m.State {
states = append(states, s.Name) states = append(states, s.Name)
// A step's key is per step and dropped after its age, so the bucket cannot fill; a pause has no age.
if (s.Name == "steps") != (s.TTLSeconds > 0) {
t.Errorf("state %s has ttl-seconds %d: only steps has an age", s.Name, s.TTLSeconds)
}
} }
slices.Sort(states) slices.Sort(states)
if want := []string{"keys", "steps", "toolchains"}; !slices.Equal(states, want) { if want := []string{"keys", "pauses", "steps", "toolchains"}; !slices.Equal(states, want) {
t.Errorf("state = %v, want %v", states, want) t.Errorf("state = %v, want %v", states, want)
} }
if len(m.Claims) != 1 || m.Claims[0].Name != Seat { if len(m.Claims) != 1 || m.Claims[0].Name != Seat {
@@ -28,7 +28,12 @@ type Module struct {
// Toolchains and Steps are the Put of the states of those names. // Toolchains and Steps are the Put of the states of those names.
Toolchains, Steps buildstep.Put Toolchains, Steps buildstep.Put
// ReadSteps and ReadToolchainsState read those two states back for the read tools (issue 1562). // ReadSteps and ReadToolchainsState read those two states back for the read tools (issue 1562).
// Held is the puller whose running asks a step-end tool will not end (issue 1890); nil in a test.
Held *buildstep.Puller
ReadSteps, ReadToolchainsState StateRead ReadSteps, ReadToolchainsState StateRead
// Pauses is the Put of the pauses state, which keeps the node's pause key without an age; ReadPauses reads it back.
Pauses buildstep.Put
ReadPauses StateRead
// Emit tells the bus an event (stdio.Emit); nil emits nothing. The guard emits toolchain.reported after each write of // Emit tells the bus an event (stdio.Emit); nil emits nothing. The guard emits toolchain.reported after each write of
// the toolchains key, so the indexer, which holds no timer, learns that the bucket exists (issue 1444). // the toolchains key, so the indexer, which holds no timer, learns that the bucket exists (issue 1444).
Emit func(eventType string, body any) error Emit func(eventType string, body any) error
@@ -118,6 +123,7 @@ func (m *Module) Start(ctx context.Context) {
m.Say("the toolchain: %v", err) m.Say("the toolchain: %v", err)
} }
m.CleanWorkspaces(time.Now()) m.CleanWorkspaces(time.Now())
buildstep.EndStaleRound(m.stepScan(), m.Node, m.Say) // asks past the step bound, on the module's own round (issue 1890)
select { select {
case <-ctx.Done(): case <-ctx.Done():
return return
@@ -115,7 +115,7 @@ func TestAGoThatCannotBeRunRefusesEveryStepAndHealthIsUnhealthy(t *testing.T) {
{Name: "short_probe", Run: run}, {Name: "short_probe", Run: run},
}, m.StepTools()...) }, m.StepTools()...)
for _, tool := range m.Tools(steps) { for _, tool := range m.Tools(steps) {
if tool.Name == StatusTool || tool.Name == HealthTool || tool.Name == StepReadTool || tool.Name == StepsReadTool || tool.Name == ToolchainsReadTool || tool.Name == ReleaseTool { if tool.Name == StatusTool || tool.Name == HealthTool || tool.Name == StepReadTool || tool.Name == StepsReadTool || tool.Name == ToolchainsReadTool || tool.Name == ReleaseTool || tool.Name == StepEndTool {
continue continue
} }
var reason string var reason string
@@ -0,0 +1,125 @@
package main
import (
"os"
"strings"
"testing"
)
// A failure that is not one test's assertion carries its real cause (issue 1901, its third meeting: a try of
// mesh-controller kept only "PASS" for a package that stopped half-way). Each fixture is the real `go test -json` of a
// throwaway package (testdata/captured/README).
func TestAFailureThatIsNotAnAssertionCarriesItsRealCause(t *testing.T) { // 1901/AC4
cases := []struct {
fixture string
test string // the test the failure names; "" for none
cause []string // what the cause holds
}{
{"panicking", "TestPanics", []string{"panic: assignment to entry in nil map"}},
{"timeout", "TestWaitsForever", []string{"panic: test timed out after 2s", "TestWaitsForever"}},
{"deeptimeout", "TestHoldsTheSeat, TestWaitsForTheBus", []string{"panic: test timed out after 2s", "TestHoldsTheSeat", "TestWaitsForTheBus"}},
{"buildfail", "", []string{`cannot use "seat" (untyped string constant) as int value`}},
{"exiting", "TestExits", []string{"exited", "TestExits"}},
{"killed", "TestKilled", []string{"signal: killed"}},
{"passthenfail", "", []string{"panic: the bus closed after the tests: nats: connection closed"}},
{"mainexit", "", []string{"passed", "exited"}},
{"race", "TestRacesOnTheSeat", []string{"DATA RACE", "Write at", "example.com/race.TestRacesOnTheSeat.func1()", "p_test.go:13", "created at example.com/race.TestRacesOnTheSeat()"}},
{"racemain", "", []string{"DATA RACE", "Write at", "example.com/racemain.TestMain.func1()", "p_test.go:15", "created at example.com/racemain.TestMain()"}},
}
for _, c := range cases {
t.Run(c.fixture, func(t *testing.T) {
raw, err := os.ReadFile("testdata/captured/go-test-json-pkg-" + c.fixture + ".jsonl")
if err != nil {
t.Fatal(err)
}
res := &TestResult{}
rd := newReader(res, ".")
rd.read(string(raw))
rd.finish()
if len(res.Failed) != 1 {
t.Fatalf("failures = %+v, want one", res.Failed)
}
f := res.Failed[0]
if f.Test != c.test {
t.Errorf("1901/AC4: the failure names test %q, want %q", f.Test, c.test)
}
for _, want := range c.cause {
if !strings.Contains(f.Cause, want) {
t.Errorf("1901/AC4: cause = %q, want it to hold %q", f.Cause, want)
}
}
if len(f.Cause) > outputLineBytes+3 {
t.Errorf("1901/AC4: cause holds %d bytes, over the line bound %d", len(f.Cause), outputLineBytes)
}
// Never a "PASS" or "ok" line as a failure's words, and the first line of the output, all an older
// checker reads, is the cause.
for _, l := range strings.Split(f.Output, "\n") {
if l = strings.TrimSpace(l); l == "PASS" || strings.HasPrefix(l, "ok ") || strings.HasPrefix(l, "ok\t") {
t.Errorf("1901/AC4: the output keeps the line %q: %q", l, f.Output)
}
}
if first := strings.SplitN(f.Output, "\n", 2)[0]; strings.TrimSpace(first) != strings.TrimSpace(f.Cause) {
t.Errorf("1901/AC4: the output starts with %q, want the cause %q", first, f.Cause)
}
})
}
}
// A package that passes, and a test that fails on its own assertion, keep what they had: no cause is made up for them.
func TestAnAssertionKeepsItsOwnCause(t *testing.T) { // 1901/AC4
raw, err := os.ReadFile("testdata/captured/go-test-json-log-then-fatal.jsonl")
if err != nil {
t.Fatal(err)
}
res := &TestResult{}
rd := newReader(res, ".")
rd.read(string(raw))
rd.finish()
if len(res.Failed) != 1 || !strings.HasSuffix(res.Failed[0].Cause, "3 subscription(s) derive to a subject no emitter owns:") {
t.Errorf("1901/AC4: failures = %+v", res.Failed)
}
}
// go prints the runtime's lines ("panic:", "fatal error:", "signal:", "WARNING: DATA RACE") at column 0: a test's own
// log line that holds the same words is indented, and is not the cause (review round 1). The fixture is the deeptimeout
// capture with two log lines of the first running test added: a t.Logf and its indented continuation saying "panic:".
func TestALoggedPanicIsNotTheCause(t *testing.T) { // 1901/AC4
raw, err := os.ReadFile("testdata/captured/go-test-json-pkg-deeptimeout.jsonl")
if err != nil {
t.Fatal(err)
}
lines := strings.SplitAfter(string(raw), "\n")
added := `{"Action":"output","Package":"example.com/deeptimeout","Test":"TestHoldsTheSeat","Output":" p_test.go:9: the child said:\n"}` + "\n" +
`{"Action":"output","Package":"example.com/deeptimeout","Test":"TestHoldsTheSeat","Output":" panic: child retried and recovered\n"}` + "\n" +
`{"Action":"output","Package":"example.com/deeptimeout","Test":"TestHoldsTheSeat","Output":" WARNING: DATA RACE\n"}` + "\n"
out := strings.Join(lines[:10], "") + added + strings.Join(lines[10:], "")
res := &TestResult{}
rd := newReader(res, ".")
rd.read(out)
rd.finish()
if len(res.Failed) != 1 || !strings.HasPrefix(res.Failed[0].Cause, "panic: test timed out after 2s") {
t.Errorf("1901/AC4: failures = %+v, want the timeout's panic as the cause", res.Failed)
}
}
// A package counts its own skipped tests, so the checker withholds a kept verdict from that package alone (issue 890).
func TestEachPackageCountsItsOwnSkippedTests(t *testing.T) {
log := `{"Action":"run","Package":"m/a","Test":"TestA"}
{"Action":"skip","Package":"m/a","Test":"TestA","Elapsed":0}
{"Action":"run","Package":"m/b","Test":"TestB"}
{"Action":"pass","Package":"m/b","Test":"TestB","Elapsed":0}
{"Action":"pass","Package":"m/a","Elapsed":0}
{"Action":"pass","Package":"m/b","Elapsed":0}
`
res := &TestResult{}
rd := newReader(res, ".")
rd.read(log)
rd.finish()
got := map[string]int{}
for _, p := range res.Packages {
got[p.Package] = p.Skipped
}
if res.Skipped != 1 || got["m/a"] != 1 || got["m/b"] != 0 {
t.Errorf("skipped = %d, per package %v, want 1 with m/a 1 and m/b 0", res.Skipped, got)
}
}
+99 -23
View File
@@ -7,9 +7,12 @@ package main
import ( import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"log"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -39,6 +42,53 @@ func readOf(s stdio.KeptState) StateRead {
}} }}
} }
// MaxStateReads is the most values one read tool reads from a state in a call, and stateReadParallel how many of them
// are in flight at once. Every value is a round trip from the tool through the runtime to the bus, so a call that read
// every key of a large state one after the other outlasted the call's 30 s (issue 1827: 1164 keys, 20 MB).
const (
MaxStateReads = 1000
stateReadParallel = 16
)
// readValues reads the values of keys, stateReadParallel at a time, in the order of keys (nil for a key with none). It
// stops starting reads after the first error and returns it.
func readValues(keys []string, get func(string) (json.RawMessage, error)) ([]json.RawMessage, error) {
out := make([]json.RawMessage, len(keys))
var (
wg sync.WaitGroup
mu sync.Mutex
fail error
next = make(chan int)
)
for w := 0; w < stateReadParallel && w < len(keys); w++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := range next {
raw, err := get(keys[i])
mu.Lock()
if err != nil && fail == nil {
fail = err
}
mu.Unlock()
out[i] = raw
}
}()
}
for i := range keys {
mu.Lock()
stop := fail != nil
mu.Unlock()
if stop {
break
}
next <- i
}
close(next)
wg.Wait()
return out, fail
}
// stepKey is the key of a step of this node: the id alone, or the whole key <node>.<id> as a step tool answered it. // stepKey is the key of a step of this node: the id alone, or the whole key <node>.<id> as a step tool answered it.
func (m *Module) stepKey(id string) (string, error) { func (m *Module) stepKey(id string) (string, error) {
id = strings.TrimSpace(id) id = strings.TrimSpace(id)
@@ -78,29 +128,44 @@ type StepLine struct {
Started string `json:"started"` Started string `json:"started"`
Ended string `json:"ended,omitempty"` Ended string `json:"ended,omitempty"`
Reason string `json:"reason,omitempty"` Reason string `json:"reason,omitempty"`
What string `json:"what,omitempty"`
} }
// RecentSteps are this node's steps, the latest started first, at most limit (ten when limit is not above zero). // RecentSteps are this node's steps, the latest started first, at most limit (ten when limit is not above zero). Only
func (m *Module) RecentSteps(limit int) ([]StepLine, error) { // the node's own keys (<node>.) are listed and the newest max(limit*4, 100) of them read (at most MaxStateReads): a step's
// id is base-36 milliseconds and random digits, so its string order is its start order. cut says older steps were not
// read. The values still order the lines, since an ask step's id carries the ask time.
func (m *Module) RecentSteps(limit int) (lines []StepLine, cut bool, err error) {
if limit <= 0 { if limit <= 0 {
limit = 10 limit = 10
} }
if m.ReadSteps.Keys == nil || m.ReadSteps.Get == nil { if m.ReadSteps.Keys == nil || m.ReadSteps.Get == nil {
return nil, fmt.Errorf("the steps state cannot be read here") return nil, false, fmt.Errorf("the steps state cannot be read here")
} }
start := time.Now()
keys, err := m.ReadSteps.Keys() keys, err := m.ReadSteps.Keys()
if err != nil { if err != nil {
return nil, err return nil, false, err
}
own := keys[:0:0]
for _, k := range keys {
if strings.HasPrefix(k, m.Node+".") {
if node, _, err := delivery.ParseStepKey(k); err == nil && node == m.Node {
own = append(own, k)
}
}
}
sort.Sort(sort.Reverse(sort.StringSlice(own)))
if n := min(max(limit*4, 100), MaxStateReads); len(own) > n {
own, cut = own[:n], true
}
raws, err := readValues(own, m.ReadSteps.Get)
log.Printf("steps read: %d of %d keys listed read in %v (cut %v)", len(own), len(keys), time.Since(start).Round(time.Millisecond), cut)
if err != nil {
return nil, cut, err
} }
var steps []delivery.Step var steps []delivery.Step
for _, k := range keys { for _, raw := range raws {
if node, _, err := delivery.ParseStepKey(k); err != nil || node != m.Node {
continue
}
raw, err := m.ReadSteps.Get(k)
if err != nil {
return nil, err
}
if len(raw) == 0 { if len(raw) == 0 {
continue continue
} }
@@ -116,13 +181,13 @@ func (m *Module) RecentSteps(limit int) ([]StepLine, error) {
} }
out := make([]StepLine, 0, len(steps)) out := make([]StepLine, 0, len(steps))
for _, s := range steps { for _, s := range steps {
l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason} l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason, What: s.What}
if !s.Ended.IsZero() { if !s.Ended.IsZero() {
l.Ended = s.Ended.UTC().Format("2006-01-02T15:04:05Z") l.Ended = s.Ended.UTC().Format("2006-01-02T15:04:05Z")
} }
out = append(out, l) out = append(out, l)
} }
return out, nil return out, cut, nil
} }
// ReadToolchains is every key of the toolchains state (<node>.<language>) with its stored value. // ReadToolchains is every key of the toolchains state (<node>.<language>) with its stored value.
@@ -130,24 +195,30 @@ func (m *Module) ReadToolchains() (map[string]delivery.Toolchain, error) {
if m.ReadToolchainsState.Keys == nil || m.ReadToolchainsState.Get == nil { if m.ReadToolchainsState.Keys == nil || m.ReadToolchainsState.Get == nil {
return nil, fmt.Errorf("the toolchains state cannot be read here") return nil, fmt.Errorf("the toolchains state cannot be read here")
} }
start := time.Now()
keys, err := m.ReadToolchainsState.Keys() keys, err := m.ReadToolchainsState.Keys()
if err != nil { if err != nil {
return nil, err return nil, err
} }
cut := len(keys) > MaxStateReads
if cut {
keys = keys[:MaxStateReads] // a node has a key per language: far fewer in practice
}
raws, err := readValues(keys, m.ReadToolchainsState.Get)
log.Printf("toolchains read: %d keys read in %v (cut %v: keys past %d were not read)", len(keys), time.Since(start).Round(time.Millisecond), cut, MaxStateReads)
if err != nil {
return nil, err
}
out := map[string]delivery.Toolchain{} out := map[string]delivery.Toolchain{}
for _, k := range keys { for i, raw := range raws {
raw, err := m.ReadToolchainsState.Get(k)
if err != nil {
return nil, err
}
if len(raw) == 0 { if len(raw) == 0 {
continue continue
} }
t, err := delivery.DecodeToolchain(raw) t, err := delivery.DecodeToolchain(raw)
if err != nil { if err != nil {
return nil, fmt.Errorf("%s: %w", k, err) return nil, fmt.Errorf("%s: %w", keys[i], err)
} }
out[k] = t out[keys[i]] = t
} }
return out, nil return out, nil
} }
@@ -171,8 +242,13 @@ func (m *Module) ReadTools() []stdio.Tool {
Input: map[string]any{"type": "object", "properties": map[string]any{ Input: map[string]any{"type": "object", "properties": map[string]any{
"limit": map[string]any{"type": "integer", "description": "how many, at most; 10 when left out"}}}, "limit": map[string]any{"type": "integer", "description": "how many, at most; 10 when left out"}}},
Run: func(args map[string]any) (any, error) { Run: func(args map[string]any) (any, error) {
steps, err := m.RecentSteps(intArg(args["limit"])) steps, cut, err := m.RecentSteps(intArg(args["limit"]))
return map[string]any{"node": m.Node, "steps": steps}, err ans := map[string]any{"node": m.Node, "steps": steps}
if cut {
ans["cut"] = true
ans["note"] = fmt.Sprintf("this node has more steps stored than one call reads; older steps were not read (ask for a larger limit, at most %d reads)", MaxStateReads)
}
return ans, err
}}, }},
{Name: ToolchainsReadTool, {Name: ToolchainsReadTool,
Description: "What the module's toolchains state holds: each key <node>.<language> with the version installed, the " + Description: "What the module's toolchains state holds: each key <node>.<language> with the version installed, the " +
@@ -3,8 +3,10 @@ package main
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"slices" "slices"
"strings" "strings"
"sync/atomic"
"testing" "testing"
"time" "time"
@@ -94,7 +96,7 @@ func TestStepsToolListsRecentStepsOfThisNode(t *testing.T) {
t0 := time.Date(2026, 10, 21, 9, 0, 0, 0, time.UTC) t0 := time.Date(2026, 10, 21, 9, 0, 0, 0, time.UTC)
putStep(t, steps, delivery.Step{ID: "a", Tool: "build_go_fetch", Node: "ace", State: delivery.StepPassed, Started: t0, Ended: t0, Result: json.RawMessage(`1`)}) putStep(t, steps, delivery.Step{ID: "a", Tool: "build_go_fetch", Node: "ace", State: delivery.StepPassed, Started: t0, Ended: t0, Result: json.RawMessage(`1`)})
putStep(t, steps, delivery.Step{ID: "b", Tool: "build_go_test", Node: "ace", State: delivery.StepRunning, Started: t0.Add(time.Minute)}) putStep(t, steps, delivery.Step{ID: "b", Tool: "build_go_test", Node: "ace", State: delivery.StepRunning, Started: t0.Add(time.Minute)})
putStep(t, steps, delivery.Step{ID: "c", Tool: "build_go_build", Node: "ace", State: delivery.StepFailed, Started: t0.Add(2 * time.Minute), Reason: "boom"}) putStep(t, steps, delivery.Step{ID: "c", Tool: "build_go_build", Node: "ace", State: delivery.StepFailed, Started: t0.Add(2 * time.Minute), Reason: "boom", What: "check ask, novox/x@abc"})
putStep(t, steps, delivery.Step{ID: "d", Tool: "build_go_build", Node: "g14", State: delivery.StepPassed, Started: t0.Add(time.Hour)}) putStep(t, steps, delivery.Step{ID: "d", Tool: "build_go_build", Node: "g14", State: delivery.StepPassed, Started: t0.Add(time.Hour)})
m := readModule(steps, fakeState{}) m := readModule(steps, fakeState{})
out, err := tool(t, m, StepsReadTool)(map[string]any{}) out, err := tool(t, m, StepsReadTool)(map[string]any{})
@@ -109,7 +111,7 @@ func TestStepsToolListsRecentStepsOfThisNode(t *testing.T) {
if !slices.Equal(ids, []string{"c", "b", "a"}) { if !slices.Equal(ids, []string{"c", "b", "a"}) {
t.Errorf("ids = %v, want c b a", ids) t.Errorf("ids = %v, want c b a", ids)
} }
if lines[0].Reason != "boom" || lines[0].State != delivery.StepFailed { if lines[0].Reason != "boom" || lines[0].What != "check ask, novox/x@abc" || lines[0].State != delivery.StepFailed {
t.Errorf("first = %+v", lines[0]) t.Errorf("first = %+v", lines[0])
} }
out, _ = tool(t, m, StepsReadTool)(map[string]any{"limit": float64(1)}) out, _ = tool(t, m, StepsReadTool)(map[string]any{"limit": float64(1)})
@@ -145,3 +147,62 @@ func TestReadToolsAreInTheManifestAndNeedNoToolchain(t *testing.T) {
t.Errorf("the read refused: %v", err) t.Errorf("the read refused: %v", err)
} }
} }
// 1827: on the live shape (4 nodes, 600 steps each; a steps state of 1164 keys / 20 MB timed out at 30 s) the steps tool
// lists, then reads only this node's keys, in parallel, at most MaxStateReads of them, and says when it cut.
func TestRecentStepsReadsOnlyTheNodesKeysInParallelAndBounded(t *testing.T) {
t0 := time.Date(2026, 10, 25, 9, 0, 0, 0, time.UTC)
for _, row := range []struct {
perNode, limit int
reads int64
steps int
cut bool
}{
{20, 0, 20, 10, false},
{600, 0, 100, 10, true},
{1500, 300, MaxStateReads, 300, true},
} {
var reads, inFlight, maxFlight atomic.Int64
stored := map[string][]byte{}
for _, n := range []string{"ace", "g14", "novox", "shanks"} {
for i := 0; i < row.perNode; i++ {
b, _ := delivery.EncodeStep(delivery.Step{ID: fmt.Sprintf("s%04d", i), Tool: "t", Node: n, State: delivery.StepPassed, Started: t0.Add(time.Duration(i) * time.Second)})
stored[n+".s"+fmt.Sprintf("%04d", i)] = b
}
}
m := &Module{Node: "shanks"}
m.ReadSteps = StateRead{
Keys: func() ([]string, error) {
var k []string
for key := range stored {
k = append(k, key)
}
return k, nil
},
Get: func(key string) (json.RawMessage, error) {
reads.Add(1)
now := inFlight.Add(1)
for o := maxFlight.Load(); now > o && !maxFlight.CompareAndSwap(o, now); o = maxFlight.Load() {
}
time.Sleep(time.Millisecond)
inFlight.Add(-1)
if !strings.HasPrefix(key, "shanks.") {
t.Errorf("read %s, a key of another node", key)
}
return stored[key], nil
},
}
start := time.Now()
lines, cut, err := m.RecentSteps(row.limit)
if err != nil || len(lines) != row.steps || cut != row.cut || reads.Load() != row.reads {
t.Errorf("%d a node: %d steps, cut %v, %d reads, err %v; want %d, %v, %d", row.perNode, len(lines), cut, reads.Load(), err, row.steps, row.cut, row.reads)
}
if row.cut && len(lines) > 0 && lines[0].ID != fmt.Sprintf("s%04d", row.perNode-1) {
t.Errorf("%d a node: newest answered %s", row.perNode, lines[0].ID)
}
if maxFlight.Load() < 2 {
t.Errorf("%d a node: values read one at a time", row.perNode)
}
t.Logf("%d a node: %d reads in %v (one after the other: %v)", row.perNode, reads.Load(), time.Since(start), time.Duration(reads.Load())*time.Millisecond)
}
}
@@ -103,6 +103,33 @@ func TestUnitLevelNamesTheFailingTestAndFailsTheLevel(t *testing.T) {
} }
} }
// Issue 1921: a fixture with a failing test in each of two packages: one level's run names both, each with its own
// cause, and counts every test, the passing one beside a failure too. go test runs every package to its end; nothing
// stops at the first failure.
func TestUnitLevelNamesEveryFailingTestOfEveryPackage(t *testing.T) {
m, states, ws, calls := workspaceFor(t, "twofailing")
res, st := runUnit(t, m, states, ws)
if st.State != delivery.StepPassed || res.Passed {
t.Fatalf("step %s (%s), result %+v: want the step done and the level failed", st.State, st.Reason, res)
}
var got []string
for _, f := range res.Failed {
got = append(got, f.Package+"."+f.Test+": "+f.Cause)
}
want := []string{"fixture/twofailing/one.TestOneIsTwo: One() = 1, want 2", "fixture/twofailing/two.TestTwoIsThree: Two() = 2, want 3"}
if strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Errorf("failed\n%s\nwant\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
if res.Tests != 3 {
t.Errorf("%d tests counted, want 3", res.Tests)
}
for _, c := range *calls {
if strings.Contains(c.line, " test ") && strings.Contains(c.line, "-failfast") {
t.Errorf("go test ran with -failfast: %s", c.line)
}
}
}
// 1290/AC2: a fixture whose tests take more than the bound (lowered by the module's option, not by waiting five // 1290/AC2: a fixture whose tests take more than the bound (lowered by the module's option, not by waiting five
// seconds): the level fails naming the bound and the slowest tests; under the real bound the same run passes. // seconds): the level fails naming the bound and the slowest tests; under the real bound the same run passes.
func TestUnitLevelOverItsBoundFailsNamingTheBoundAndTheSlowestTests(t *testing.T) { func TestUnitLevelOverItsBoundFailsNamingTheBoundAndTheSlowestTests(t *testing.T) {
@@ -1,6 +1,7 @@
Each file is the standard output of `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go test -count=1 -json ./...` (go1.27.1) Each file is the standard output of `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go test -count=1 -json ./...` (go1.27.1)
run in the fixture module of the same name under ../repos, captured on 2026-10-20 for issue 1290. The unit run in the fixture module of the same name under ../repos, captured on 2026-10-20 for issue 1290. The unit
tests feed them to the parser through a fake runner, so no test here runs `go test`. tests feed them to the parser through a fake runner, so no test here runs `go test`. twofailing.json was captured the
same way on 2026-10-24 for issue 1921: two packages with a failing test each, and a passing test beside one of them.
golist-external.json is `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go list -json=ImportPath,Standard,DepOnly,Module golist-external.json is `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go list -json=ImportPath,Standard,DepOnly,Module
git.novox.be/novox/mesh-sdk/go/buildstep` run in this module (go1.27.1, 2026-10-20 for issue 1291): a package of a module git.novox.be/novox/mesh-sdk/go/buildstep` run in this module (go1.27.1, 2026-10-20 for issue 1291): a package of a module
@@ -24,3 +25,15 @@ The tar streams (the archives gunzipped) of the three archives above, captured t
gunzip and sha256) at origin/main 0205b8f7 on 2026-10-21: directory f8af1d94...d1b6, within 294e2bec...07fb, file gunzip and sha256) at origin/main 0205b8f7 on 2026-10-21: directory f8af1d94...d1b6, within 294e2bec...07fb, file
055cffc2...428e. The digests above hold the gzip bytes of go1.27.1. The first try of issue 1589 on the merge check's build node answered 055cffc2...428e. The digests above hold the gzip bytes of go1.27.1. The first try of issue 1589 on the merge check's build node answered
other digests; the tests therefore compare the stream always and the digest only under go1.27.1. other digests; the tests therefore compare the stream always and the digest only under go1.27.1.
go-test-json-pkg-*.jsonl are the standard output and error of `go test -json -count=1 ./...` (go1.27.3, the timeouts with
`-timeout 2s`), captured on 2026-10-25 for issue 1901 in throwaway one-file modules that each fail without a test's own
assertion: panicking (a nil map write in TestPanics), timeout (TestWaitsForever sleeps past the timeout), deeptimeout (two
parallel tests and sixty blocked goroutines past the timeout, so the goroutine dump pushes the panic line out of the last
4 KB), buildfail (a type error in a test file), exiting (os.Exit(3) in TestExits), killed (TestKilled sends itself SIGKILL),
passthenfail (TestMain panics in a goroutine after m.Run passed) and mainexit (TestMain exits 2 after m.Run passed: go test
prints "PASS" and then "FAIL <pkg>").
go-test-json-pkg-race.jsonl and go-test-json-pkg-racemain.jsonl are the same, with `go test -race -json -count=1 ./...`
(CGO_ENABLED=1, go1.27.3, 2026-10-25, issue 1901): race (TestRacesOnTheSeat writes a package variable from itself and a
goroutine) and racemain (TestMain does so after m.Run passed: go test prints "PASS", then the race report and "Found 1
data race(s)", then "FAIL <pkg>").
@@ -0,0 +1,6 @@
{"ImportPath":"example.com/buildfail [example.com/buildfail.test]","Action":"build-output","Output":"# example.com/buildfail [example.com/buildfail.test]\n"}
{"ImportPath":"example.com/buildfail [example.com/buildfail.test]","Action":"build-output","Output":"./p_test.go:5:40: cannot use \"seat\" (untyped string constant) as int value in variable declaration\n"}
{"ImportPath":"example.com/buildfail [example.com/buildfail.test]","Action":"build-fail"}
{"Time":"2026-10-25T15:34:33.261137737+01:00","Action":"start","Package":"example.com/buildfail"}
{"Time":"2026-10-25T15:34:33.261255019+01:00","Action":"output","Package":"example.com/buildfail","Output":"FAIL\texample.com/buildfail [build failed]\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.261268099+01:00","Action":"fail","Package":"example.com/buildfail","Elapsed":0,"FailedBuild":"example.com/buildfail [example.com/buildfail.test]"}
@@ -0,0 +1,417 @@
{"Time":"2026-10-25T15:35:18.991630993+01:00","Action":"start","Package":"example.com/deeptimeout"}
{"Time":"2026-10-25T15:35:18.993895188+01:00","Action":"run","Package":"example.com/deeptimeout","Test":"TestHoldsTheSeat"}
{"Time":"2026-10-25T15:35:18.993963909+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestHoldsTheSeat","Output":"=== RUN TestHoldsTheSeat\n","OutputType":"frame"}
{"Time":"2026-10-25T15:35:18.99399484+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestHoldsTheSeat","Output":"=== PAUSE TestHoldsTheSeat\n","OutputType":"frame"}
{"Time":"2026-10-25T15:35:18.9939982+01:00","Action":"pause","Package":"example.com/deeptimeout","Test":"TestHoldsTheSeat"}
{"Time":"2026-10-25T15:35:18.99400173+01:00","Action":"run","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus"}
{"Time":"2026-10-25T15:35:18.99400456+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"=== RUN TestWaitsForTheBus\n","OutputType":"frame"}
{"Time":"2026-10-25T15:35:18.99400838+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"=== PAUSE TestWaitsForTheBus\n","OutputType":"frame"}
{"Time":"2026-10-25T15:35:18.99401133+01:00","Action":"pause","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus"}
{"Time":"2026-10-25T15:35:18.99401441+01:00","Action":"cont","Package":"example.com/deeptimeout","Test":"TestHoldsTheSeat"}
{"Time":"2026-10-25T15:35:18.99401704+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestHoldsTheSeat","Output":"=== CONT TestHoldsTheSeat\n","OutputType":"frame"}
{"Time":"2026-10-25T15:35:18.994074141+01:00","Action":"cont","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus"}
{"Time":"2026-10-25T15:35:18.994102612+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"=== CONT TestWaitsForTheBus\n","OutputType":"frame"}
{"Time":"2026-10-25T15:35:21.00935195+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"panic: test timed out after 2s\n"}
{"Time":"2026-10-25T15:35:21.00938383+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\trunning tests:\n"}
{"Time":"2026-10-25T15:35:21.009394391+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t\tTestHoldsTheSeat (2s)\n"}
{"Time":"2026-10-25T15:35:21.009398021+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t\tTestWaitsForTheBus (2s)\n"}
{"Time":"2026-10-25T15:35:21.009401801+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009406481+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 81 [running]:\n"}
{"Time":"2026-10-25T15:35:21.009409911+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"testing.(*M).startAlarm.func1()\n"}
{"Time":"2026-10-25T15:35:21.009413361+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/testing/testing.go:2961 +0x34a\n"}
{"Time":"2026-10-25T15:35:21.009418341+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by time.goFunc\n"}
{"Time":"2026-10-25T15:35:21.009421781+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/time/sleep.go:182 +0x2d\n"}
{"Time":"2026-10-25T15:35:21.009424991+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009428341+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 1 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009431721+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"testing.tRunner.func1()\n"}
{"Time":"2026-10-25T15:35:21.009441811+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/testing/testing.go:2144 +0x425\n"}
{"Time":"2026-10-25T15:35:21.009445511+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"testing.tRunner(0x282e9702008, 0x282e9646bc8)\n"}
{"Time":"2026-10-25T15:35:21.009449641+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/testing/testing.go:2201 +0x123\n"}
{"Time":"2026-10-25T15:35:21.009453471+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"testing.runTests({0x55a45d, 0x17}, {0x55a45d, 0x17}, 0x282e95940f0, {0x6f19f0, 0x2, 0x2}, {0xc2afa2ca3b3c0c8d, 0x7737f168, ...})\n"}
{"Time":"2026-10-25T15:35:21.009462092+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/testing/testing.go:2742 +0x510\n"}
{"Time":"2026-10-25T15:35:21.009467222+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"testing.(*M).Run(0x282e96581e0)\n"}
{"Time":"2026-10-25T15:35:21.009470662+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/testing/testing.go:2602 +0x6af\n"}
{"Time":"2026-10-25T15:35:21.009473692+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"main.main()\n"}
{"Time":"2026-10-25T15:35:21.009476942+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t_testmain.go:48 +0x9b\n"}
{"Time":"2026-10-25T15:35:21.009480092+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009483242+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 7 [sleep]:\n"}
{"Time":"2026-10-25T15:35:21.009486502+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"time.Sleep(0xdf8475800)\n"}
{"Time":"2026-10-25T15:35:21.009490682+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/runtime/time.go:368 +0x165\n"}
{"Time":"2026-10-25T15:35:21.009493812+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat(0x282e9702248?)\n"}
{"Time":"2026-10-25T15:35:21.009498842+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:14 +0x9e\n"}
{"Time":"2026-10-25T15:35:21.009502242+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"testing.tRunner(0x282e9702248, 0x6d5e78)\n"}
{"Time":"2026-10-25T15:35:21.009505742+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/testing/testing.go:2195 +0xea\n"}
{"Time":"2026-10-25T15:35:21.009509292+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by testing.(*T).Run in goroutine 1\n"}
{"Time":"2026-10-25T15:35:21.009512752+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/testing/testing.go:2260 +0x4d4\n"}
{"Time":"2026-10-25T15:35:21.009515792+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009518623+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 8 [sleep]:\n"}
{"Time":"2026-10-25T15:35:21.009521303+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"time.Sleep(0xdf8475800)\n"}
{"Time":"2026-10-25T15:35:21.009524423+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/runtime/time.go:368 +0x165\n"}
{"Time":"2026-10-25T15:35:21.009531403+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestWaitsForTheBus(0x282e9702488?)\n"}
{"Time":"2026-10-25T15:35:21.009534943+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:19 +0x25\n"}
{"Time":"2026-10-25T15:35:21.009538403+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"testing.tRunner(0x282e9702488, 0x6d5e80)\n"}
{"Time":"2026-10-25T15:35:21.009541953+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/testing/testing.go:2195 +0xea\n"}
{"Time":"2026-10-25T15:35:21.009545233+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by testing.(*T).Run in goroutine 1\n"}
{"Time":"2026-10-25T15:35:21.009548323+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/usr/lib/go/src/testing/testing.go:2260 +0x4d4\n"}
{"Time":"2026-10-25T15:35:21.009551343+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009554263+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 9 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009557523+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009561063+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009564513+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009571113+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009574233+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009577433+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 10 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009580823+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009584174+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009587504+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009590974+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009594004+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009597184+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 11 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009600394+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009603874+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009610954+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009614714+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009618104+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009621364+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 12 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009624704+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009628324+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009631954+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009635404+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009638734+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009641904+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 13 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009645325+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009648645+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009652065+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009655625+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009658885+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009663385+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 14 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009666735+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009670235+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009676385+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009680205+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009683315+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009686475+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 15 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009689785+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009697235+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009700795+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009704985+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009708085+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009711346+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 16 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009714586+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009717946+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009721676+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009725626+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009728866+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009732106+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 17 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009735376+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009738956+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009742326+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009746436+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009750666+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009754126+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 18 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009758286+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009761846+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009765776+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009769816+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009773167+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009780507+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 19 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009784207+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009788007+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009791627+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009796347+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009799427+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009802937+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 20 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009806667+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009809987+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009813827+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009818087+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009821397+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009824227+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 21 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009827607+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009831307+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009835167+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009839528+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009842728+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009845888+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 22 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009849168+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009852548+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009855958+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009863648+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009866938+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009870088+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 23 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009879828+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009883728+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009887608+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009891548+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009895238+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009898748+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 24 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009902349+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009907109+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009911519+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009915609+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009918939+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009922329+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 25 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009928329+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009931909+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.009935489+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.009939429+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.009942699+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.009946019+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 26 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.009949559+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.009953489+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.00996244+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.00996647+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.00996987+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.00997314+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 27 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.00997664+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.00998035+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.00998418+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.00998793+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.00999131+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.00999534+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 28 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.00999894+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.01000268+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.0100087+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.01001258+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.01001605+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.01002007+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 29 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.01002393+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010027531+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010031351+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010035521+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010038781+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010042281+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 30 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010046091+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010054261+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010057881+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010061781+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010064931+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010068261+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 31 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010072151+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010075981+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010079841+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010098802+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010103202+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010106382+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 32 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010109852+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010113242+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010116952+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010120672+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010124112+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010127162+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 33 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010130662+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010134442+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010138142+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010141972+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010145362+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010152262+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 34 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010156353+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010159913+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010163533+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010167403+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010170533+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010173953+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 35 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010177453+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010181033+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010184693+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010188583+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010191753+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010195033+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 36 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010198603+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010202393+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010206123+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010210583+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010213503+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010216914+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 37 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010220134+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010223644+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010227154+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010235154+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010238354+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010241754+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 38 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010270284+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010275144+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010278714+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010282825+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010285795+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010288935+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 39 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010292405+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010295915+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010299545+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010311835+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010315565+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010318975+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 40 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010322655+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010326225+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010329815+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010333745+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010337095+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010340515+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 41 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010344066+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010347706+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010355596+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010359046+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010362036+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010365076+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 42 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010367956+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010370906+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010375366+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010378386+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010381006+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010383666+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 43 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010386676+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010389786+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010392636+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010395746+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010398466+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010401166+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 44 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010404226+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010407186+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010410117+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010413237+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010416157+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010418897+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 45 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010423157+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010430107+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010433197+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010451207+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010454217+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010456967+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 46 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010459877+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010463967+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010467067+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010470597+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010473418+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010476108+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 47 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010478998+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010481958+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010484978+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010488258+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010490888+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010493628+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 48 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010496588+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010499498+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010502498+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010505768+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010508558+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010514148+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 49 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010517178+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010520238+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010523478+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010526818+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010529718+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010532508+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 50 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010535639+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010538749+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010541939+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010546689+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010549869+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010552989+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 51 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010556299+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010560079+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010563479+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010567189+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010570299+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010573079+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 52 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010576319+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010579599+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010583099+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010590519+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010594029+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010596989+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 53 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.01059986+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.01060433+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.01060789+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.01061136+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.01061393+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.01061667+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 54 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.01061984+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.0106227+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.01062566+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.01062876+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.01063131+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.01063428+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 55 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.01063719+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.01063997+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.01064278+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.01064582+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.01064836+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.01065114+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 56 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.01065437+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.01065724+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010671481+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010675671+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010678431+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010691171+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 57 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010694531+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010697641+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010700781+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010704001+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010706611+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010709421+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 58 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010712331+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010715351+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010718431+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010721691+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010724371+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010727362+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 59 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010730592+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010733662+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010736712+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010740162+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010742922+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010745772+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 60 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010752222+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010755402+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010758542+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010761822+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010764592+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010767992+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 61 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010771062+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010774282+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010779382+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010782942+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010785722+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010788573+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 62 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010791893+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010794943+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010798083+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010801683+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010804293+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010807283+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 63 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010810443+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010813553+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010816793+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010820233+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010823063+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010829493+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 64 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010832683+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010835953+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010839223+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010842463+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010845333+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010848423+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 65 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010851663+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010854964+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010858364+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010861714+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010864914+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010867984+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 66 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010870944+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010874114+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010877464+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010880504+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010883144+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010886194+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 67 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010889154+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010892234+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010895524+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010902154+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.010904984+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\n"}
{"Time":"2026-10-25T15:35:21.010907984+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"goroutine 68 [chan receive]:\n"}
{"Time":"2026-10-25T15:35:21.010911064+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"example.com/deeptimeout.TestHoldsTheSeat.func1()\n"}
{"Time":"2026-10-25T15:35:21.010914074+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x19\n"}
{"Time":"2026-10-25T15:35:21.010917365+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"created by example.com/deeptimeout.TestHoldsTheSeat in goroutine 7\n"}
{"Time":"2026-10-25T15:35:21.010920505+01:00","Action":"output","Package":"example.com/deeptimeout","Test":"TestWaitsForTheBus","Output":"\t/tmp/fix-1901b/deeptimeout/p_test.go:12 +0x37\n"}
{"Time":"2026-10-25T15:35:21.013209581+01:00","Action":"output","Package":"example.com/deeptimeout","Output":"FAIL\texample.com/deeptimeout\t2.021s\n","OutputType":"frame"}
{"Time":"2026-10-25T15:35:21.013258851+01:00","Action":"fail","Package":"example.com/deeptimeout","Elapsed":2.022}
@@ -0,0 +1,10 @@
{"Time":"2026-10-25T15:34:33.768742837+01:00","Action":"start","Package":"example.com/exiting"}
{"Time":"2026-10-25T15:34:33.773822287+01:00","Action":"run","Package":"example.com/exiting","Test":"TestFine"}
{"Time":"2026-10-25T15:34:33.773896608+01:00","Action":"output","Package":"example.com/exiting","Test":"TestFine","Output":"=== RUN TestFine\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.773940319+01:00","Action":"output","Package":"example.com/exiting","Test":"TestFine","Output":"--- PASS: TestFine (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.773945359+01:00","Action":"pass","Package":"example.com/exiting","Test":"TestFine","Elapsed":0}
{"Time":"2026-10-25T15:34:33.773952529+01:00","Action":"run","Package":"example.com/exiting","Test":"TestExits"}
{"Time":"2026-10-25T15:34:33.773956439+01:00","Action":"output","Package":"example.com/exiting","Test":"TestExits","Output":"=== RUN TestExits\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.773959829+01:00","Action":"output","Package":"example.com/exiting","Test":"TestExits","Output":" p_test.go:11: closing the store\n"}
{"Time":"2026-10-25T15:34:33.775158538+01:00","Action":"output","Package":"example.com/exiting","Output":"FAIL\texample.com/exiting\t0.006s\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.775186509+01:00","Action":"fail","Package":"example.com/exiting","Elapsed":0.006}
@@ -0,0 +1,7 @@
{"Time":"2026-10-25T15:34:34.328258934+01:00","Action":"start","Package":"example.com/killed"}
{"Time":"2026-10-25T15:34:34.331465435+01:00","Action":"run","Package":"example.com/killed","Test":"TestKilled"}
{"Time":"2026-10-25T15:34:34.331699248+01:00","Action":"output","Package":"example.com/killed","Test":"TestKilled","Output":"=== RUN TestKilled\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:34.331853111+01:00","Action":"output","Package":"example.com/killed","Test":"TestKilled","Output":" p_test.go:10: the kernel is about to kill this\n"}
{"Time":"2026-10-25T15:34:34.334563213+01:00","Action":"output","Package":"example.com/killed","Test":"TestKilled","Output":"signal: killed\n"}
{"Time":"2026-10-25T15:34:34.334611114+01:00","Action":"output","Package":"example.com/killed","Output":"FAIL\texample.com/killed\t0.006s\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:34.334627134+01:00","Action":"fail","Package":"example.com/killed","Elapsed":0.006}
@@ -0,0 +1,8 @@
{"Time":"2026-10-25T15:34:35.274953775+01:00","Action":"start","Package":"example.com/mainexit"}
{"Time":"2026-10-25T15:34:35.278129565+01:00","Action":"run","Package":"example.com/mainexit","Test":"TestFine"}
{"Time":"2026-10-25T15:34:35.278191516+01:00","Action":"output","Package":"example.com/mainexit","Test":"TestFine","Output":"=== RUN TestFine\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:35.278341979+01:00","Action":"output","Package":"example.com/mainexit","Test":"TestFine","Output":"--- PASS: TestFine (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:35.278351739+01:00","Action":"pass","Package":"example.com/mainexit","Test":"TestFine","Elapsed":0}
{"Time":"2026-10-25T15:34:35.278359039+01:00","Action":"output","Package":"example.com/mainexit","Output":"PASS\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:35.279516487+01:00","Action":"output","Package":"example.com/mainexit","Output":"FAIL\texample.com/mainexit\t0.004s\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:35.279539298+01:00","Action":"fail","Package":"example.com/mainexit","Elapsed":0.005}
@@ -0,0 +1,27 @@
{"Time":"2026-10-25T15:34:33.031091436+01:00","Action":"start","Package":"example.com/panicking"}
{"Time":"2026-10-25T15:34:33.036353279+01:00","Action":"run","Package":"example.com/panicking","Test":"TestBefore"}
{"Time":"2026-10-25T15:34:33.036441781+01:00","Action":"output","Package":"example.com/panicking","Test":"TestBefore","Output":"=== RUN TestBefore\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.036568163+01:00","Action":"output","Package":"example.com/panicking","Test":"TestBefore","Output":"--- PASS: TestBefore (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.036609863+01:00","Action":"pass","Package":"example.com/panicking","Test":"TestBefore","Elapsed":0}
{"Time":"2026-10-25T15:34:33.036641304+01:00","Action":"run","Package":"example.com/panicking","Test":"TestPanics"}
{"Time":"2026-10-25T15:34:33.036645914+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"=== RUN TestPanics\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.036698995+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":" p_test.go:8: about to look up the seat\n"}
{"Time":"2026-10-25T15:34:33.036773996+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"--- FAIL: TestPanics (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.03897418+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"panic: assignment to entry in nil map [recovered, repanicked]\n"}
{"Time":"2026-10-25T15:34:33.040332172+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"\n"}
{"Time":"2026-10-25T15:34:33.040414493+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"goroutine 8 [running]:\n"}
{"Time":"2026-10-25T15:34:33.040521475+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"testing.tRunner.func1.2({0x6b8120, 0x6eff80})\n"}
{"Time":"2026-10-25T15:34:33.040601826+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"\t/usr/lib/go/src/testing/testing.go:2125 +0x232\n"}
{"Time":"2026-10-25T15:34:33.040641947+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"testing.tRunner.func1()\n"}
{"Time":"2026-10-25T15:34:33.040720648+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"\t/usr/lib/go/src/testing/testing.go:2128 +0x329\n"}
{"Time":"2026-10-25T15:34:33.04083286+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"panic({0x6b8120?, 0x6eff80?})\n"}
{"Time":"2026-10-25T15:34:33.040912921+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"\t/usr/lib/go/src/runtime/panic.go:859 +0x125\n"}
{"Time":"2026-10-25T15:34:33.040981652+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"example.com/panicking.TestPanics(0x228226cd0488?)\n"}
{"Time":"2026-10-25T15:34:33.041059903+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"\t/tmp/fix-1901b/panicking/p_test.go:10 +0x53\n"}
{"Time":"2026-10-25T15:34:33.041132704+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"testing.tRunner(0x228226cd0488, 0x6d5bc8)\n"}
{"Time":"2026-10-25T15:34:33.041215586+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"\t/usr/lib/go/src/testing/testing.go:2195 +0xea\n"}
{"Time":"2026-10-25T15:34:33.041221966+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"created by testing.(*T).Run in goroutine 1\n"}
{"Time":"2026-10-25T15:34:33.041225556+01:00","Action":"output","Package":"example.com/panicking","Test":"TestPanics","Output":"\t/usr/lib/go/src/testing/testing.go:2260 +0x4d4\n"}
{"Time":"2026-10-25T15:34:33.041984088+01:00","Action":"fail","Package":"example.com/panicking","Test":"TestPanics","Elapsed":0}
{"Time":"2026-10-25T15:34:33.041996638+01:00","Action":"output","Package":"example.com/panicking","Output":"FAIL\texample.com/panicking\t0.010s\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:33.042044079+01:00","Action":"fail","Package":"example.com/panicking","Elapsed":0.011}
@@ -0,0 +1,15 @@
{"Time":"2026-10-25T15:34:34.919624192+01:00","Action":"start","Package":"example.com/passthenfail"}
{"Time":"2026-10-25T15:34:34.922861224+01:00","Action":"run","Package":"example.com/passthenfail","Test":"TestFine"}
{"Time":"2026-10-25T15:34:34.922932965+01:00","Action":"output","Package":"example.com/passthenfail","Test":"TestFine","Output":"=== RUN TestFine\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:34.922969205+01:00","Action":"output","Package":"example.com/passthenfail","Test":"TestFine","Output":"--- PASS: TestFine (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:34.922979956+01:00","Action":"pass","Package":"example.com/passthenfail","Test":"TestFine","Elapsed":0}
{"Time":"2026-10-25T15:34:34.922989416+01:00","Action":"output","Package":"example.com/passthenfail","Output":"PASS\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:34.925022528+01:00","Action":"output","Package":"example.com/passthenfail","Output":"panic: the bus closed after the tests: nats: connection closed\n"}
{"Time":"2026-10-25T15:34:34.925043388+01:00","Action":"output","Package":"example.com/passthenfail","Output":"\n"}
{"Time":"2026-10-25T15:34:34.925046848+01:00","Action":"output","Package":"example.com/passthenfail","Output":"goroutine 20 [running]:\n"}
{"Time":"2026-10-25T15:34:34.925051338+01:00","Action":"output","Package":"example.com/passthenfail","Output":"example.com/passthenfail.TestMain.func1()\n"}
{"Time":"2026-10-25T15:34:34.925055878+01:00","Action":"output","Package":"example.com/passthenfail","Output":"\t/tmp/fix-1901b/passthenfail/p_test.go:13 +0x25\n"}
{"Time":"2026-10-25T15:34:34.925061428+01:00","Action":"output","Package":"example.com/passthenfail","Output":"created by example.com/passthenfail.TestMain in goroutine 1\n"}
{"Time":"2026-10-25T15:34:34.925065028+01:00","Action":"output","Package":"example.com/passthenfail","Output":"\t/tmp/fix-1901b/passthenfail/p_test.go:13 +0x25\n"}
{"Time":"2026-10-25T15:34:34.925390203+01:00","Action":"output","Package":"example.com/passthenfail","Output":"FAIL\texample.com/passthenfail\t0.005s\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:34.925402904+01:00","Action":"fail","Package":"example.com/passthenfail","Elapsed":0.006}
@@ -0,0 +1,53 @@
{"Time":"2026-10-25T15:50:42.908557309+01:00","Action":"start","Package":"example.com/race"}
{"Time":"2026-10-25T15:50:42.924298679+01:00","Action":"run","Package":"example.com/race","Test":"TestBefore"}
{"Time":"2026-10-25T15:50:42.92438195+01:00","Action":"output","Package":"example.com/race","Test":"TestBefore","Output":"=== RUN TestBefore\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:42.924648842+01:00","Action":"output","Package":"example.com/race","Test":"TestBefore","Output":"--- PASS: TestBefore (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:42.924685133+01:00","Action":"pass","Package":"example.com/race","Test":"TestBefore","Elapsed":0}
{"Time":"2026-10-25T15:50:42.924728573+01:00","Action":"run","Package":"example.com/race","Test":"TestRacesOnTheSeat"}
{"Time":"2026-10-25T15:50:42.924733573+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"=== RUN TestRacesOnTheSeat\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:42.92637298+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"==================\n"}
{"Time":"2026-10-25T15:50:42.92641567+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"WARNING: DATA RACE\n"}
{"Time":"2026-10-25T15:50:42.926444061+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"Write at 0x000000833468 by goroutine 10:\n"}
{"Time":"2026-10-25T15:50:42.926462621+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" example.com/race.TestRacesOnTheSeat.func1()\n"}
{"Time":"2026-10-25T15:50:42.926467121+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /tmp/fix-1901b/race/p_test.go:13 +0x24\n"}
{"Time":"2026-10-25T15:50:42.926483061+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"\n"}
{"Time":"2026-10-25T15:50:42.926509741+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"Previous write at 0x000000833468 by goroutine 9:\n"}
{"Time":"2026-10-25T15:50:42.926530142+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" example.com/race.TestRacesOnTheSeat()\n"}
{"Time":"2026-10-25T15:50:42.926534952+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /tmp/fix-1901b/race/p_test.go:14 +0x30\n"}
{"Time":"2026-10-25T15:50:42.926550462+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.tRunner()\n"}
{"Time":"2026-10-25T15:50:42.926554612+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /usr/lib/go/src/testing/testing.go:2195 +0x21c\n"}
{"Time":"2026-10-25T15:50:42.926571142+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.(*T).Run.gowrap1()\n"}
{"Time":"2026-10-25T15:50:42.926575812+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /usr/lib/go/src/testing/testing.go:2260 +0x38\n"}
{"Time":"2026-10-25T15:50:42.926592202+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"\n"}
{"Time":"2026-10-25T15:50:42.926607792+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"Goroutine 10 (running) created at:\n"}
{"Time":"2026-10-25T15:50:42.926624292+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" example.com/race.TestRacesOnTheSeat()\n"}
{"Time":"2026-10-25T15:50:42.926628923+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /tmp/fix-1901b/race/p_test.go:13 +0x24\n"}
{"Time":"2026-10-25T15:50:42.926643853+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.tRunner()\n"}
{"Time":"2026-10-25T15:50:42.926648103+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /usr/lib/go/src/testing/testing.go:2195 +0x21c\n"}
{"Time":"2026-10-25T15:50:42.926674843+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.(*T).Run.gowrap1()\n"}
{"Time":"2026-10-25T15:50:42.926680673+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /usr/lib/go/src/testing/testing.go:2260 +0x38\n"}
{"Time":"2026-10-25T15:50:42.926695433+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"\n"}
{"Time":"2026-10-25T15:50:42.926711063+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"Goroutine 9 (running) created at:\n"}
{"Time":"2026-10-25T15:50:42.926726614+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.(*T).Run()\n"}
{"Time":"2026-10-25T15:50:42.926730814+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /usr/lib/go/src/testing/testing.go:2260 +0xb12\n"}
{"Time":"2026-10-25T15:50:42.926746484+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.runTests.func1()\n"}
{"Time":"2026-10-25T15:50:42.926750764+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /usr/lib/go/src/testing/testing.go:2744 +0x84\n"}
{"Time":"2026-10-25T15:50:42.926765354+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.tRunner()\n"}
{"Time":"2026-10-25T15:50:42.926769844+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /usr/lib/go/src/testing/testing.go:2195 +0x21c\n"}
{"Time":"2026-10-25T15:50:42.926791794+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.runTests()\n"}
{"Time":"2026-10-25T15:50:42.926796704+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /usr/lib/go/src/testing/testing.go:2742 +0x9e9\n"}
{"Time":"2026-10-25T15:50:42.926811544+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.(*M).Run()\n"}
{"Time":"2026-10-25T15:50:42.926816694+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" /usr/lib/go/src/testing/testing.go:2602 +0xf44\n"}
{"Time":"2026-10-25T15:50:42.926831695+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" main.main()\n"}
{"Time":"2026-10-25T15:50:42.926835935+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" _testmain.go:50 +0x164\n"}
{"Time":"2026-10-25T15:50:42.926850905+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"==================\n"}
{"Time":"2026-10-25T15:50:42.935320241+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":" testing.go:1867: race detected during execution of test\n","OutputType":"error"}
{"Time":"2026-10-25T15:50:42.935520113+01:00","Action":"output","Package":"example.com/race","Test":"TestRacesOnTheSeat","Output":"--- FAIL: TestRacesOnTheSeat (0.01s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:42.935592973+01:00","Action":"fail","Package":"example.com/race","Test":"TestRacesOnTheSeat","Elapsed":0.01}
{"Time":"2026-10-25T15:50:42.935722225+01:00","Action":"run","Package":"example.com/race","Test":"TestAfter"}
{"Time":"2026-10-25T15:50:42.935728845+01:00","Action":"output","Package":"example.com/race","Test":"TestAfter","Output":"=== RUN TestAfter\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:42.935995928+01:00","Action":"output","Package":"example.com/race","Test":"TestAfter","Output":"--- PASS: TestAfter (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:42.936006328+01:00","Action":"pass","Package":"example.com/race","Test":"TestAfter","Elapsed":0}
{"Time":"2026-10-25T15:50:42.936009978+01:00","Action":"output","Package":"example.com/race","Output":"FAIL\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:42.937730745+01:00","Action":"output","Package":"example.com/race","Output":"FAIL\texample.com/race\t0.028s\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:42.937802066+01:00","Action":"fail","Package":"example.com/race","Elapsed":0.029}
@@ -0,0 +1,27 @@
{"Time":"2026-10-25T15:50:43.551701745+01:00","Action":"start","Package":"example.com/racemain"}
{"Time":"2026-10-25T15:50:43.570709708+01:00","Action":"run","Package":"example.com/racemain","Test":"TestFine"}
{"Time":"2026-10-25T15:50:43.570806059+01:00","Action":"output","Package":"example.com/racemain","Test":"TestFine","Output":"=== RUN TestFine\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:43.57085525+01:00","Action":"output","Package":"example.com/racemain","Test":"TestFine","Output":"--- PASS: TestFine (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:43.57086149+01:00","Action":"pass","Package":"example.com/racemain","Test":"TestFine","Elapsed":0}
{"Time":"2026-10-25T15:50:43.57086974+01:00","Action":"output","Package":"example.com/racemain","Output":"PASS\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:43.572155623+01:00","Action":"output","Package":"example.com/racemain","Output":"==================\n"}
{"Time":"2026-10-25T15:50:43.572168083+01:00","Action":"output","Package":"example.com/racemain","Output":"WARNING: DATA RACE\n"}
{"Time":"2026-10-25T15:50:43.572171933+01:00","Action":"output","Package":"example.com/racemain","Output":"Write at 0x000000836428 by goroutine 9:\n"}
{"Time":"2026-10-25T15:50:43.572176703+01:00","Action":"output","Package":"example.com/racemain","Output":" example.com/racemain.TestMain.func1()\n"}
{"Time":"2026-10-25T15:50:43.572180053+01:00","Action":"output","Package":"example.com/racemain","Output":" /tmp/fix-1901b/racemain/p_test.go:15 +0x24\n"}
{"Time":"2026-10-25T15:50:43.572184923+01:00","Action":"output","Package":"example.com/racemain","Output":"\n"}
{"Time":"2026-10-25T15:50:43.572189613+01:00","Action":"output","Package":"example.com/racemain","Output":"Previous write at 0x000000836428 by main goroutine:\n"}
{"Time":"2026-10-25T15:50:43.572192993+01:00","Action":"output","Package":"example.com/racemain","Output":" example.com/racemain.TestMain()\n"}
{"Time":"2026-10-25T15:50:43.572198573+01:00","Action":"output","Package":"example.com/racemain","Output":" /tmp/fix-1901b/racemain/p_test.go:16 +0x44\n"}
{"Time":"2026-10-25T15:50:43.572202433+01:00","Action":"output","Package":"example.com/racemain","Output":" main.main()\n"}
{"Time":"2026-10-25T15:50:43.572205623+01:00","Action":"output","Package":"example.com/racemain","Output":" _testmain.go:48 +0x171\n"}
{"Time":"2026-10-25T15:50:43.572208443+01:00","Action":"output","Package":"example.com/racemain","Output":"\n"}
{"Time":"2026-10-25T15:50:43.572211573+01:00","Action":"output","Package":"example.com/racemain","Output":"Goroutine 9 (running) created at:\n"}
{"Time":"2026-10-25T15:50:43.572214803+01:00","Action":"output","Package":"example.com/racemain","Output":" example.com/racemain.TestMain()\n"}
{"Time":"2026-10-25T15:50:43.572218043+01:00","Action":"output","Package":"example.com/racemain","Output":" /tmp/fix-1901b/racemain/p_test.go:15 +0x37\n"}
{"Time":"2026-10-25T15:50:43.572225414+01:00","Action":"output","Package":"example.com/racemain","Output":" main.main()\n"}
{"Time":"2026-10-25T15:50:43.572228654+01:00","Action":"output","Package":"example.com/racemain","Output":" _testmain.go:48 +0x171\n"}
{"Time":"2026-10-25T15:50:43.572231664+01:00","Action":"output","Package":"example.com/racemain","Output":"==================\n"}
{"Time":"2026-10-25T15:50:44.581262249+01:00","Action":"output","Package":"example.com/racemain","Output":"Found 1 data race(s)\n"}
{"Time":"2026-10-25T15:50:44.584693204+01:00","Action":"output","Package":"example.com/racemain","Output":"FAIL\texample.com/racemain\t1.031s\n","OutputType":"frame"}
{"Time":"2026-10-25T15:50:44.584737425+01:00","Action":"fail","Package":"example.com/racemain","Elapsed":1.033}
@@ -0,0 +1,43 @@
{"Time":"2026-10-25T15:34:36.049580618+01:00","Action":"start","Package":"example.com/timeout"}
{"Time":"2026-10-25T15:34:36.053250986+01:00","Action":"run","Package":"example.com/timeout","Test":"TestQuick"}
{"Time":"2026-10-25T15:34:36.053354018+01:00","Action":"output","Package":"example.com/timeout","Test":"TestQuick","Output":"=== RUN TestQuick\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:36.053459789+01:00","Action":"output","Package":"example.com/timeout","Test":"TestQuick","Output":"--- PASS: TestQuick (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:36.05348499+01:00","Action":"pass","Package":"example.com/timeout","Test":"TestQuick","Elapsed":0}
{"Time":"2026-10-25T15:34:36.0535114+01:00","Action":"run","Package":"example.com/timeout","Test":"TestWaitsForever"}
{"Time":"2026-10-25T15:34:36.05351549+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"=== RUN TestWaitsForever\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:36.053567451+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":" p_test.go:11: waiting for the bus\n"}
{"Time":"2026-10-25T15:34:38.064693307+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"panic: test timed out after 2s\n"}
{"Time":"2026-10-25T15:34:38.06487122+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\trunning tests:\n"}
{"Time":"2026-10-25T15:34:38.06491684+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t\tTestWaitsForever (2s)\n"}
{"Time":"2026-10-25T15:34:38.065034382+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\n"}
{"Time":"2026-10-25T15:34:38.065108573+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"goroutine 33 [running]:\n"}
{"Time":"2026-10-25T15:34:38.065154754+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"testing.(*M).startAlarm.func1()\n"}
{"Time":"2026-10-25T15:34:38.065386208+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/testing/testing.go:2961 +0x34a\n"}
{"Time":"2026-10-25T15:34:38.065431878+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"created by time.goFunc\n"}
{"Time":"2026-10-25T15:34:38.0655151+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/time/sleep.go:182 +0x2d\n"}
{"Time":"2026-10-25T15:34:38.06553471+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\n"}
{"Time":"2026-10-25T15:34:38.065847805+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"goroutine 1 [chan receive]:\n"}
{"Time":"2026-10-25T15:34:38.066038298+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"testing.(*T).Run(0x3a0209bc8008, {0x5580aa?, 0x3a0209a9caa0?}, 0x6d5b10)\n"}
{"Time":"2026-10-25T15:34:38.066047488+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/testing/testing.go:2268 +0x4f2\n"}
{"Time":"2026-10-25T15:34:38.066051368+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"testing.runTests.func1(0x3a0209bc8008)\n"}
{"Time":"2026-10-25T15:34:38.066056128+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/testing/testing.go:2744 +0x37\n"}
{"Time":"2026-10-25T15:34:38.066059478+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"testing.tRunner(0x3a0209bc8008, 0x3a0209a9cbc8)\n"}
{"Time":"2026-10-25T15:34:38.066064958+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/testing/testing.go:2195 +0xea\n"}
{"Time":"2026-10-25T15:34:38.066069599+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"testing.runTests({0x559017, 0x13}, {0x559017, 0x13}, 0x3a0209b8e0d8, {0x6f19f0, 0x2, 0x2}, {0xc2afa2bf832a3f56, 0x773908b5, ...})\n"}
{"Time":"2026-10-25T15:34:38.066083309+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/testing/testing.go:2742 +0x510\n"}
{"Time":"2026-10-25T15:34:38.066086879+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"testing.(*M).Run(0x3a0209b8c1e0)\n"}
{"Time":"2026-10-25T15:34:38.066090269+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/testing/testing.go:2602 +0x6af\n"}
{"Time":"2026-10-25T15:34:38.066093429+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"main.main()\n"}
{"Time":"2026-10-25T15:34:38.066097569+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t_testmain.go:48 +0x9b\n"}
{"Time":"2026-10-25T15:34:38.066100639+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\n"}
{"Time":"2026-10-25T15:34:38.066103769+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"goroutine 20 [sleep]:\n"}
{"Time":"2026-10-25T15:34:38.066106969+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"time.Sleep(0xdf8475800)\n"}
{"Time":"2026-10-25T15:34:38.066110279+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/runtime/time.go:368 +0x165\n"}
{"Time":"2026-10-25T15:34:38.066113639+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"example.com/timeout.TestWaitsForever(0x3a0209bc8488?)\n"}
{"Time":"2026-10-25T15:34:38.066117109+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/tmp/fix-1901b/timeout/p_test.go:12 +0x48\n"}
{"Time":"2026-10-25T15:34:38.066120239+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"testing.tRunner(0x3a0209bc8488, 0x6d5b10)\n"}
{"Time":"2026-10-25T15:34:38.066123509+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/testing/testing.go:2195 +0xea\n"}
{"Time":"2026-10-25T15:34:38.066126839+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"created by testing.(*T).Run in goroutine 1\n"}
{"Time":"2026-10-25T15:34:38.066129989+01:00","Action":"output","Package":"example.com/timeout","Test":"TestWaitsForever","Output":"\t/usr/lib/go/src/testing/testing.go:2260 +0x4d4\n"}
{"Time":"2026-10-25T15:34:38.066986453+01:00","Action":"output","Package":"example.com/timeout","Output":"FAIL\texample.com/timeout\t2.017s\n","OutputType":"frame"}
{"Time":"2026-10-25T15:34:38.067002113+01:00","Action":"fail","Package":"example.com/timeout","Elapsed":2.017}
@@ -0,0 +1,23 @@
{"Time":"2026-10-24T23:45:37.721718705+02:00","Action":"start","Package":"fixture/twofailing/one"}
{"Time":"2026-10-24T23:45:37.723685152+02:00","Action":"run","Package":"fixture/twofailing/one","Test":"TestOneIsTwo"}
{"Time":"2026-10-24T23:45:37.723726472+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":"=== RUN TestOneIsTwo\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723743682+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":" one_test.go:6: checking One\n"}
{"Time":"2026-10-24T23:45:37.723750622+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":" one_test.go:8: One() = 1, want 2\n","OutputType":"error"}
{"Time":"2026-10-24T23:45:37.723759302+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":"--- FAIL: TestOneIsTwo (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723889303+02:00","Action":"start","Package":"fixture/twofailing/two"}
{"Time":"2026-10-24T23:45:37.723762972+02:00","Action":"fail","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Elapsed":0}
{"Time":"2026-10-24T23:45:37.723952274+02:00","Action":"run","Package":"fixture/twofailing/one","Test":"TestOneIsOne"}
{"Time":"2026-10-24T23:45:37.723955484+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsOne","Output":"=== RUN TestOneIsOne\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723959304+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsOne","Output":"--- PASS: TestOneIsOne (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723964914+02:00","Action":"pass","Package":"fixture/twofailing/one","Test":"TestOneIsOne","Elapsed":0}
{"Time":"2026-10-24T23:45:37.723967704+02:00","Action":"output","Package":"fixture/twofailing/one","Output":"FAIL\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.724225126+02:00","Action":"output","Package":"fixture/twofailing/one","Output":"FAIL\tfixture/twofailing/one\t0.002s\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.724231936+02:00","Action":"fail","Package":"fixture/twofailing/one","Elapsed":0.003}
{"Time":"2026-10-24T23:45:37.725752949+02:00","Action":"run","Package":"fixture/twofailing/two","Test":"TestTwoIsThree"}
{"Time":"2026-10-24T23:45:37.725796309+02:00","Action":"output","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Output":"=== RUN TestTwoIsThree\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.725808239+02:00","Action":"output","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Output":" two_test.go:7: Two() = 2, want 3\n","OutputType":"error"}
{"Time":"2026-10-24T23:45:37.72581671+02:00","Action":"output","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Output":"--- FAIL: TestTwoIsThree (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.72582026+02:00","Action":"fail","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Elapsed":0}
{"Time":"2026-10-24T23:45:37.72582549+02:00","Action":"output","Package":"fixture/twofailing/two","Output":"FAIL\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.726319344+02:00","Action":"output","Package":"fixture/twofailing/two","Output":"FAIL\tfixture/twofailing/two\t0.002s\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.726362544+02:00","Action":"fail","Package":"fixture/twofailing/two","Elapsed":0.002}
@@ -0,0 +1,3 @@
module fixture/twofailing
go 1.25.0
@@ -0,0 +1,3 @@
package one
func One() int { return 1 }
@@ -0,0 +1,16 @@
package one
import "testing"
func TestOneIsTwo(t *testing.T) {
t.Log("checking One")
if One() != 2 {
t.Fatalf("One() = %d, want 2", One())
}
}
func TestOneIsOne(t *testing.T) {
if One() != 1 {
t.Fatal("one")
}
}
@@ -0,0 +1,3 @@
package two
func Two() int { return 2 }
@@ -0,0 +1,9 @@
package two
import "testing"
func TestTwoIsThree(t *testing.T) {
if Two() != 3 {
t.Errorf("Two() = %d, want 3", Two())
}
}
@@ -2,6 +2,7 @@ package main
import ( import (
"context" "context"
"git.novox.be/novox/mesh-sdk/go/buildstep"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
) )
@@ -40,6 +41,7 @@ func (m *Module) Tools(steps []StepTool) []stdio.Tool {
tools = append(tools, t) tools = append(tools, t)
} }
tools = append(tools, m.ReadTools()...) tools = append(tools, m.ReadTools()...)
tools = append(tools, buildstep.StepEndTool(StepEndTool, m.Node, m.stepScan(), m.Held))
tools = append(tools, m.releaseTool()) tools = append(tools, m.releaseTool())
return append(tools, return append(tools,
stdio.Tool{Name: StatusTool, stdio.Tool{Name: StatusTool,
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildgo
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024125710-bfc3883af980 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125154-6b8497314297
github.com/distribution/distribution/v3 v3.1.2 github.com/distribution/distribution/v3 v3.1.2
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024125710-bfc3883af980 h1:ZEEeYibEPD1rngOAqbODzRY5rUStGC46OpyQwtNCYbs= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125154-6b8497314297 h1:k31FtFp/DGXGatRjKV1S0z1gBAHfVgtvBX6xF0g1Vgw=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024125710-bfc3883af980/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125154-6b8497314297/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI= github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI=
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM= github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
+2 -1
View File
@@ -10,7 +10,7 @@
"state": [ "state": [
"toolchains", "toolchains",
"keys", "keys",
"steps" {"name": "steps", "ttl-seconds": 172800}, "pauses"
], ],
"claims": [ "claims": [
{ {
@@ -39,6 +39,7 @@
"build_go_release", "build_go_release",
"build_go_step", "build_go_step",
"build_go_steps", "build_go_steps",
"build_go_step_end",
"build_go_toolchains", "build_go_toolchains",
"build_go_status", "build_go_status",
"build_go_health" "build_go_health"
@@ -13,6 +13,7 @@ import (
var Kept = []announce.State{ var Kept = []announce.State{
{Name: "keys", Readable: true}, {Name: "keys", Readable: true},
{Name: "steps", Readable: true}, {Name: "steps", Readable: true},
{Name: "pauses", Readable: true},
} }
// keepStates announces Kept as this module's (stdio.Keeps): main calls it before it serves. // keepStates announces Kept as this module's (stdio.Keeps): main calls it before it serves.
@@ -37,16 +37,20 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
source = buildstep.SeatAsks{Language: delivery.AskImage} source = buildstep.SeatAsks{Language: delivery.AskImage}
} }
capacity := m.Capacity capacity := m.Capacity
// The capacity's own change wakes the puller (issue 1887): the runtime's watch of the config value as mesh-config
// resolved it; the settings file is read only when that watch cannot start. A test's Capacity is its own.
var watch func(string, func(stdio.ProvisionChange) error) error
if capacity == nil { if capacity == nil {
capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say) capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say)
watch = buildstep.WatchConfigServed
} }
ask := m.askSteps(steps) ask := m.askSteps(steps)
var load func(string) ([]byte, error) var load func(string) ([]byte, error)
if m.ReadSteps.Get != nil { if m.ReadPauses.Get != nil {
load = func(key string) ([]byte, error) { return m.ReadSteps.Get(key) } load = func(key string) ([]byte, error) { return m.ReadPauses.Get(key) }
} }
return &buildstep.Puller{Source: source, Language: delivery.AskImage, Node: m.Node, Put: m.Steps, Load: load, return &buildstep.Puller{Source: source, Language: delivery.AskImage, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load,
Capacity: capacity, Logf: m.Say, Capacity: capacity, WatchConfig: watch, Logf: m.Say,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { Run: func(ctx context.Context, a delivery.StepAsk) (any, error) {
if err := m.Check(ctx); err != nil { if err := m.Check(ctx); err != nil {
return nil, err return nil, err
@@ -62,3 +66,11 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
func (m *Module) SeatTools(p *buildstep.Puller) []stdio.Tool { func (m *Module) SeatTools(p *buildstep.Puller) []stdio.Tool {
return buildstep.SeatTools(Seat, "image", p) return buildstep.SeatTools(Seat, "image", p)
} }
// StepEndTool ends one step left running (issue 1890); the module's steps state is the one it reads and writes.
const StepEndTool = "build_image_step_end"
// stepScan is the module's steps state as buildstep reads it.
func (m *Module) stepScan() buildstep.Scan {
return buildstep.ScanOf(stdio.State(delivery.StepsState))
}
@@ -9,9 +9,11 @@ package main
import ( import (
"context" "context"
"fmt" "fmt"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"os" "os"
"os/user" "os/user"
"strings" "strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -29,13 +31,15 @@ func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-build-image]
func main() { func main() {
ctx := context.Background() ctx := context.Background()
m := &Module{ m := &Module{
Node: strings.TrimSpace(os.Getenv("MESH_NODE")), Node: strings.TrimSpace(os.Getenv("MESH_NODE")),
Steps: stdio.State(delivery.StepsState).Put, Steps: stdio.State(delivery.StepsState).Put,
ReadSteps: readOf(stdio.State(delivery.StepsState)), ReadSteps: readOf(stdio.State(delivery.StepsState)),
Ctx: ctx, Pauses: stdio.State(delivery.PausesState).Put,
Say: logf, ReadPauses: readOf(stdio.State(delivery.PausesState)),
Work: WorkDir, Ctx: ctx,
Store: StoreDir, Say: logf,
Work: WorkDir,
Store: StoreDir,
} }
m.Floor = floorOf(m.Node) m.Floor = floorOf(m.Node)
if u, err := user.Current(); err == nil { if u, err := user.Current(); err == nil {
@@ -55,7 +59,13 @@ func main() {
os.Exit(1) os.Exit(1)
} }
steps := m.StepTools() steps := m.StepTools()
booted := time.Now()
puller := m.Puller(steps, nil) puller := m.Puller(steps, nil)
m.Held = puller
if m.Node != "" {
// A step a previous process left running is ended (issue 1890); this process's own steps start after booted.
go buildstep.EndStaleAtStart(ctx, m.stepScan(), m.Node, booted, logf)
}
if m.Node != "" { if m.Node != "" {
// The seat's asks, pulled once Serve has the runtime's channel; a pull before it is said and tried again. // The seat's asks, pulled once Serve has the runtime's channel; a pull before it is said and tried again.
go func() { _ = puller.Pull(ctx) }() go func() { _ = puller.Pull(ctx) }()
@@ -97,9 +97,13 @@ func TestManifestKeepsItsStatesClaimsItsBuildSeatAndDeclaresItsHealthTool(t *tes
states := []string{} states := []string{}
for _, s := range m.State { for _, s := range m.State {
states = append(states, s.Name) states = append(states, s.Name)
// A step's key is per step and dropped after its age, so the bucket cannot fill; a pause has no age.
if (s.Name == "steps") != (s.TTLSeconds > 0) {
t.Errorf("state %s has ttl-seconds %d: only steps has an age", s.Name, s.TTLSeconds)
}
} }
slices.Sort(states) slices.Sort(states)
if want := []string{"keys", "steps"}; !slices.Equal(states, want) { if want := []string{"keys", "pauses", "steps"}; !slices.Equal(states, want) {
t.Errorf("state = %v, want %v", states, want) t.Errorf("state = %v, want %v", states, want)
} }
if len(m.Claims) != 1 || m.Claims[0].Name != Seat || !slices.Equal(m.Claims[0].Serves, []string{"current", "pause", "resume"}) { if len(m.Claims) != 1 || m.Claims[0].Name != Seat || !slices.Equal(m.Claims[0].Serves, []string{"current", "pause", "resume"}) {
@@ -22,8 +22,13 @@ type Module struct {
// Account and UID are the account the module runs as; Home is its home. // Account and UID are the account the module runs as; Home is its home.
Account, UID, Home string Account, UID, Home string
// Steps is the Put of the steps state; ReadSteps reads it back for the read tools. // Steps is the Put of the steps state; ReadSteps reads it back for the read tools.
Steps buildstep.Put Steps buildstep.Put
// Held is the puller whose running asks a step-end tool will not end (issue 1890); nil in a test.
Held *buildstep.Puller
ReadSteps StateRead ReadSteps StateRead
// Pauses is the Put of the pauses state, which keeps the node's pause key without an age; ReadPauses reads it back.
Pauses buildstep.Put
ReadPauses StateRead
// Ctx is the process's context, which a step runs under (the tool call's is cut at 30 s, the step's is not). // Ctx is the process's context, which a step runs under (the tool call's is cut at 30 s, the step's is not).
Ctx context.Context Ctx context.Context
Say func(format string, a ...any) Say func(format string, a ...any)
@@ -101,6 +106,7 @@ func (m *Module) Start(ctx context.Context) {
m.Say("this machine cannot build images: %v", err) m.Say("this machine cannot build images: %v", err)
} }
m.CleanWorkspaces(m.clock()) m.CleanWorkspaces(m.clock())
buildstep.EndStaleRound(m.stepScan(), m.Node, m.Say) // asks past the step bound, on the module's own round (issue 1890)
select { select {
case <-ctx.Done(): case <-ctx.Done():
return return
@@ -7,9 +7,12 @@ package main
import ( import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"log"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -38,6 +41,53 @@ func readOf(s stdio.KeptState) StateRead {
}} }}
} }
// MaxStateReads is the most values one read tool reads from a state in a call, and stateReadParallel how many of them
// are in flight at once. Every value is a round trip from the tool through the runtime to the bus, so a call that read
// every key of a large state one after the other outlasted the call's 30 s (issue 1827: 1164 keys, 20 MB).
const (
MaxStateReads = 1000
stateReadParallel = 16
)
// readValues reads the values of keys, stateReadParallel at a time, in the order of keys (nil for a key with none). It
// stops starting reads after the first error and returns it.
func readValues(keys []string, get func(string) (json.RawMessage, error)) ([]json.RawMessage, error) {
out := make([]json.RawMessage, len(keys))
var (
wg sync.WaitGroup
mu sync.Mutex
fail error
next = make(chan int)
)
for w := 0; w < stateReadParallel && w < len(keys); w++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := range next {
raw, err := get(keys[i])
mu.Lock()
if err != nil && fail == nil {
fail = err
}
mu.Unlock()
out[i] = raw
}
}()
}
for i := range keys {
mu.Lock()
stop := fail != nil
mu.Unlock()
if stop {
break
}
next <- i
}
close(next)
wg.Wait()
return out, fail
}
// stepKey is the key of a step of this node: the id alone, or the whole key <node>.<id> as a step tool answered it. // stepKey is the key of a step of this node: the id alone, or the whole key <node>.<id> as a step tool answered it.
func (m *Module) stepKey(id string) (string, error) { func (m *Module) stepKey(id string) (string, error) {
id = strings.TrimSpace(id) id = strings.TrimSpace(id)
@@ -77,29 +127,44 @@ type StepLine struct {
Started string `json:"started"` Started string `json:"started"`
Ended string `json:"ended,omitempty"` Ended string `json:"ended,omitempty"`
Reason string `json:"reason,omitempty"` Reason string `json:"reason,omitempty"`
What string `json:"what,omitempty"`
} }
// RecentSteps are this node's steps, the latest started first, at most limit (ten when limit is not above zero). // RecentSteps are this node's steps, the latest started first, at most limit (ten when limit is not above zero). Only
func (m *Module) RecentSteps(limit int) ([]StepLine, error) { // the node's own keys (<node>.) are listed and the newest max(limit*4, 100) of them read (at most MaxStateReads): a step's
// id is base-36 milliseconds and random digits, so its string order is its start order. cut says older steps were not
// read. The values still order the lines, since an ask step's id carries the ask time.
func (m *Module) RecentSteps(limit int) (lines []StepLine, cut bool, err error) {
if limit <= 0 { if limit <= 0 {
limit = 10 limit = 10
} }
if m.ReadSteps.Keys == nil || m.ReadSteps.Get == nil { if m.ReadSteps.Keys == nil || m.ReadSteps.Get == nil {
return nil, fmt.Errorf("the steps state cannot be read here") return nil, false, fmt.Errorf("the steps state cannot be read here")
} }
start := time.Now()
keys, err := m.ReadSteps.Keys() keys, err := m.ReadSteps.Keys()
if err != nil { if err != nil {
return nil, err return nil, false, err
}
own := keys[:0:0]
for _, k := range keys {
if strings.HasPrefix(k, m.Node+".") {
if node, _, err := delivery.ParseStepKey(k); err == nil && node == m.Node {
own = append(own, k)
}
}
}
sort.Sort(sort.Reverse(sort.StringSlice(own)))
if n := min(max(limit*4, 100), MaxStateReads); len(own) > n {
own, cut = own[:n], true
}
raws, err := readValues(own, m.ReadSteps.Get)
log.Printf("steps read: %d of %d keys listed read in %v (cut %v)", len(own), len(keys), time.Since(start).Round(time.Millisecond), cut)
if err != nil {
return nil, cut, err
} }
var steps []delivery.Step var steps []delivery.Step
for _, k := range keys { for _, raw := range raws {
if node, _, err := delivery.ParseStepKey(k); err != nil || node != m.Node {
continue
}
raw, err := m.ReadSteps.Get(k)
if err != nil {
return nil, err
}
if len(raw) == 0 { if len(raw) == 0 {
continue continue
} }
@@ -115,13 +180,13 @@ func (m *Module) RecentSteps(limit int) ([]StepLine, error) {
} }
out := make([]StepLine, 0, len(steps)) out := make([]StepLine, 0, len(steps))
for _, s := range steps { for _, s := range steps {
l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason} l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason, What: s.What}
if !s.Ended.IsZero() { if !s.Ended.IsZero() {
l.Ended = s.Ended.UTC().Format("2006-01-02T15:04:05Z") l.Ended = s.Ended.UTC().Format("2006-01-02T15:04:05Z")
} }
out = append(out, l) out = append(out, l)
} }
return out, nil return out, cut, nil
} }
// ReadTools are the module's read tools. All are read-only. // ReadTools are the module's read tools. All are read-only.
@@ -143,8 +208,13 @@ func (m *Module) ReadTools() []stdio.Tool {
Input: map[string]any{"type": "object", "properties": map[string]any{ Input: map[string]any{"type": "object", "properties": map[string]any{
"limit": map[string]any{"type": "integer", "description": "how many, at most; 10 when left out"}}}, "limit": map[string]any{"type": "integer", "description": "how many, at most; 10 when left out"}}},
Run: func(args map[string]any) (any, error) { Run: func(args map[string]any) (any, error) {
steps, err := m.RecentSteps(intArg(args["limit"])) steps, cut, err := m.RecentSteps(intArg(args["limit"]))
return map[string]any{"node": m.Node, "steps": steps}, err ans := map[string]any{"node": m.Node, "steps": steps}
if cut {
ans["cut"] = true
ans["note"] = fmt.Sprintf("this node has more steps stored than one call reads; older steps were not read (ask for a larger limit, at most %d reads)", MaxStateReads)
}
return ans, err
}}, }},
} }
} }
@@ -2,7 +2,11 @@ package main
import ( import (
"encoding/json" "encoding/json"
"fmt"
"strings"
"sync/atomic"
"testing" "testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
@@ -39,8 +43,67 @@ func TestTheReadToolsAnswerAStepsResult(t *testing.T) {
if err := json.Unmarshal(got.Result, &back); err != nil || back.Digest != res.Digest { if err := json.Unmarshal(got.Result, &back); err != nil || back.Digest != res.Digest {
t.Errorf("result = %s, %v", got.Result, err) t.Errorf("result = %s, %v", got.Result, err)
} }
lines, err := m.RecentSteps(0) lines, _, err := m.RecentSteps(0)
if err != nil || len(lines) != 1 || lines[0].Tool != BuildTool { if err != nil || len(lines) != 1 || lines[0].Tool != BuildTool {
t.Errorf("steps = %+v, %v", lines, err) t.Errorf("steps = %+v, %v", lines, err)
} }
} }
// 1827: on the live shape (4 nodes, 600 steps each; a steps state of 1164 keys / 20 MB timed out at 30 s) the steps tool
// lists, then reads only this node's keys, in parallel, at most MaxStateReads of them, and says when it cut.
func TestRecentStepsReadsOnlyTheNodesKeysInParallelAndBounded(t *testing.T) {
t0 := time.Date(2026, 10, 25, 9, 0, 0, 0, time.UTC)
for _, row := range []struct {
perNode, limit int
reads int64
steps int
cut bool
}{
{20, 0, 20, 10, false},
{600, 0, 100, 10, true},
{1500, 300, MaxStateReads, 300, true},
} {
var reads, inFlight, maxFlight atomic.Int64
stored := map[string][]byte{}
for _, n := range []string{"ace", "g14", "novox", "shanks"} {
for i := 0; i < row.perNode; i++ {
b, _ := delivery.EncodeStep(delivery.Step{ID: fmt.Sprintf("s%04d", i), Tool: "t", Node: n, State: delivery.StepPassed, Started: t0.Add(time.Duration(i) * time.Second)})
stored[n+".s"+fmt.Sprintf("%04d", i)] = b
}
}
m := &Module{Node: "shanks"}
m.ReadSteps = StateRead{
Keys: func() ([]string, error) {
var k []string
for key := range stored {
k = append(k, key)
}
return k, nil
},
Get: func(key string) (json.RawMessage, error) {
reads.Add(1)
now := inFlight.Add(1)
for o := maxFlight.Load(); now > o && !maxFlight.CompareAndSwap(o, now); o = maxFlight.Load() {
}
time.Sleep(time.Millisecond)
inFlight.Add(-1)
if !strings.HasPrefix(key, "shanks.") {
t.Errorf("read %s, a key of another node", key)
}
return stored[key], nil
},
}
start := time.Now()
lines, cut, err := m.RecentSteps(row.limit)
if err != nil || len(lines) != row.steps || cut != row.cut || reads.Load() != row.reads {
t.Errorf("%d a node: %d steps, cut %v, %d reads, err %v; want %d, %v, %d", row.perNode, len(lines), cut, reads.Load(), err, row.steps, row.cut, row.reads)
}
if row.cut && len(lines) > 0 && lines[0].ID != fmt.Sprintf("s%04d", row.perNode-1) {
t.Errorf("%d a node: newest answered %s", row.perNode, lines[0].ID)
}
if maxFlight.Load() < 2 {
t.Errorf("%d a node: values read one at a time", row.perNode)
}
t.Logf("%d a node: %d reads in %v (one after the other: %v)", row.perNode, reads.Load(), time.Since(start), time.Duration(reads.Load())*time.Millisecond)
}
}
@@ -2,6 +2,7 @@ package main
import ( import (
"context" "context"
"git.novox.be/novox/mesh-sdk/go/buildstep"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
) )
@@ -44,6 +45,7 @@ func (m *Module) Tools(steps []StepTool) []stdio.Tool {
tools = append(tools, t) tools = append(tools, t)
} }
tools = append(tools, m.ReadTools()...) tools = append(tools, m.ReadTools()...)
tools = append(tools, buildstep.StepEndTool(StepEndTool, m.Node, m.stepScan(), m.Held))
tools = append(tools, m.releaseTool()) tools = append(tools, m.releaseTool())
return append(tools, return append(tools,
stdio.Tool{Name: StatusTool, stdio.Tool{Name: StatusTool,
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildimage
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125154-6b8497314297
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93 h1:FAiFY2KBKEoZSH49OZTt7/zRBIQxyl5g/KhYrpggbN4= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125154-6b8497314297 h1:k31FtFp/DGXGatRjKV1S0z1gBAHfVgtvBX6xF0g1Vgw=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125154-6b8497314297/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+2 -1
View File
@@ -9,7 +9,7 @@
], ],
"state": [ "state": [
"keys", "keys",
"steps" {"name": "steps", "ttl-seconds": 172800}, "pauses"
], ],
"claims": [ "claims": [
{ {
@@ -33,6 +33,7 @@
"build_image_release", "build_image_release",
"build_image_step", "build_image_step",
"build_image_steps", "build_image_steps",
"build_image_step_end",
"build_image_status", "build_image_status",
"build_image_health" "build_image_health"
], ],
@@ -14,6 +14,7 @@ var Kept = []announce.State{
{Name: "toolchains", Readable: true}, {Name: "toolchains", Readable: true},
{Name: "keys", Readable: true}, {Name: "keys", Readable: true},
{Name: "steps", Readable: true}, {Name: "steps", Readable: true},
{Name: "pauses", Readable: true},
} }
// keepStates announces Kept as this module's (stdio.Keeps): main calls it before it serves. // keepStates announces Kept as this module's (stdio.Keeps): main calls it before it serves.
@@ -45,16 +45,20 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
source = buildstep.SeatAsks{Language: delivery.AskNode} source = buildstep.SeatAsks{Language: delivery.AskNode}
} }
capacity := m.Capacity capacity := m.Capacity
// The capacity's own change wakes the puller (issue 1887): the runtime's watch of the config value as mesh-config
// resolved it; the settings file is read only when that watch cannot start. A test's Capacity is its own.
var watch func(string, func(stdio.ProvisionChange) error) error
if capacity == nil { if capacity == nil {
capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say) capacity = buildstep.NodeCapacity(m.Settings, buildstep.DefaultCapacities, m.Node, m.Say)
watch = buildstep.WatchConfigServed
} }
ask := m.askSteps(steps) ask := m.askSteps(steps)
var load func(string) ([]byte, error) var load func(string) ([]byte, error)
if m.ReadSteps.Get != nil { if m.ReadPauses.Get != nil {
load = func(key string) ([]byte, error) { return m.ReadSteps.Get(key) } load = func(key string) ([]byte, error) { return m.ReadPauses.Get(key) }
} }
return &buildstep.Puller{Source: source, Language: delivery.AskNode, Node: m.Node, Put: m.Steps, Load: load, return &buildstep.Puller{Source: source, Language: delivery.AskNode, Node: m.Node, Put: m.Steps, PausePut: m.Pauses, Load: load,
Capacity: capacity, Logf: m.Say, Capacity: capacity, WatchConfig: watch, Logf: m.Say,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { Run: func(ctx context.Context, a delivery.StepAsk) (any, error) {
if err := m.Prepare(ctx); err != nil { if err := m.Prepare(ctx); err != nil {
return nil, err return nil, err
@@ -70,3 +74,11 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
func (m *Module) SeatTools(p *buildstep.Puller) []stdio.Tool { func (m *Module) SeatTools(p *buildstep.Puller) []stdio.Tool {
return buildstep.SeatTools(Seat, "Node", p) return buildstep.SeatTools(Seat, "Node", p)
} }
// StepEndTool ends one step left running (issue 1890); the module's steps state is the one it reads and writes.
const StepEndTool = "build_node_step_end"
// stepScan is the module's steps state as buildstep reads it.
func (m *Module) stepScan() buildstep.Scan {
return buildstep.ScanOf(stdio.State(delivery.StepsState))
}
@@ -17,11 +17,11 @@ import (
"git.novox.be/novox/mesh-sdk/go/forgekey" "git.novox.be/novox/mesh-sdk/go/forgekey"
) )
// realMachine gives the tests the real toolchain of the module's own lock and makes two workspaces holding one fixture // realMachine gives the tests the real toolchain of the module's own lock and makes three workspaces holding one fixture
// bundle. The toolchain is the one this node's installed Node build module keeps (ADR 0533 §1): the integration level // bundle. The toolchain is the one this node's installed Node build module keeps (ADR 0533 §1), copied into the test's
// is given its toolchains read-only and no network, so it is copied into the test's own cache from there. Only where // own cache: the integration level is given its toolchains read-only and no network, so it is never installed from the
// no toolchain is installed (a desk) is it installed with `npm ci --ignore-scripts` from the mesh's registry, as the // registry here. Where no Node build module has installed one (a node without it, a desk), the test is skipped by
// module does, which needs the network. // name: an install from the public registry inside a shut-off level only fails, after 70 s of npm's retries.
func realMachine(t *testing.T) (*Module, string) { func realMachine(t *testing.T) (*Module, string) {
t.Helper() t.Helper()
for _, p := range []string{"node", "npm", "bwrap"} { // bwrap: tsc, esbuild and the scripts run shut off (1309) for _, p := range []string{"node", "npm", "bwrap"} { // bwrap: tsc, esbuild and the scripts run shut off (1309)
@@ -44,16 +44,19 @@ func realMachine(t *testing.T) (*Module, string) {
run := forgekey.ExecRunnerFor(0) run := forgekey.ExecRunnerFor(0)
m := &Module{Node: "ace", Recipe: lock, Work: work, Run: run, Ctx: context.Background(), m := &Module{Node: "ace", Recipe: lock, Work: work, Run: run, Ctx: context.Background(),
Toolchain: &Toolchain{Paths: Paths{Home: home, Cache: cache}, Run: run, Registry: func() (Registry, error) { Toolchain: &Toolchain{Paths: Paths{Home: home, Cache: cache}, Run: run, Registry: func() (Registry, error) {
return Registry{URL: "https://git.novox.be/api/packages/novox/npm/"}, nil return Registry{}, fmt.Errorf("the integration test installs no toolchain from a registry: it has no network")
}}} }}}
withStore(t, m) withStore(t, m)
if from := installedToolchain(t, filepath.Join(CacheDir, "toolchain"), filepath.Join(home, "toolchain", "package-lock.json")); from != "" { from := installedToolchain(t, filepath.Join(CacheDir, "toolchain"), filepath.Join(home, "toolchain", "package-lock.json"))
copyToolchain(t, from, m.Toolchain.Paths.Dir(m.Recipe.Version())) if from == "" {
t.Skipf("NOT RUN: no Node toolchain is installed under %s for this account to read (no Node build module on this node); "+
"the integration level has no network to install one, so this runs on a node that holds the Node build module", filepath.Join(CacheDir, "toolchain"))
} }
if err := m.Toolchain.Ensure(context.Background(), m.Recipe.Version()); err != nil { copyToolchain(t, from, m.Toolchain.Paths.Dir(m.Recipe.Version()))
t.Fatalf("no toolchain of this lock is installed under %s, and installing it from the registry failed: %v", filepath.Join(CacheDir, "toolchain"), err) if err := m.Toolchain.Ensure(context.Background(), m.Recipe.Version()); err != nil { // present: opens it, installs nothing
t.Fatalf("the toolchain copied from %s cannot be used: %v", from, err)
} }
for _, id := range []string{"w1", "w2"} { for _, id := range []string{"w1", "w2", "w3"} {
for p, body := range map[string]string{ for p, body := range map[string]string{
"modules/m/module.json": `{"module":"m","build":{"artifacts":[{"name":"tools","kind":"bundle","language":"typescript","entrypoints":["tools/index.js"]}]}}`, "modules/m/module.json": `{"module":"m","build":{"artifacts":[{"name":"tools","kind":"bundle","language":"typescript","entrypoints":["tools/index.js"]}]}}`,
"modules/m/package.json": `{"name":"m","type":"module","scripts":{"test":"node -e \"process.exit(0)\""}}`, "modules/m/package.json": `{"name":"m","type":"module","scripts":{"test":"node -e \"process.exit(0)\""}}`,
@@ -65,10 +68,36 @@ func realMachine(t *testing.T) (*Module, string) {
return m, work return m, work
} }
// 1490/AC2: a fixture TypeScript bundle importing @novox/mesh-sdk, built twice by the real tsc and esbuild, has equal // 1490/AC2, 1491 and issue 1869, in one test so the toolchain is copied once: the real type-check and test script of the
// digests, and its archive holds the bundled .js, the .serve.mjs launcher and package.json. // fixture pass the unit level; a package whose test imports its own build output (as mesh-sdk's do) passes the unit level,
func TestAFixtureBundleBuiltTwiceHasOneDigest(t *testing.T) { // the test step emitting it with the real tsc, shut off, before the test script runs; and the fixture TypeScript bundle
// importing @novox/mesh-sdk, built twice by the real tsc and esbuild, has equal digests, its archive holding the bundled
// .js, the .serve.mjs launcher and package.json. The unit levels run in w3, so the two builds' workspaces stay alike.
func TestTheFixturePassesTheUnitLevelAndBuildsTwiceToOneDigest(t *testing.T) {
m, work := realMachine(t) m, work := realMachine(t)
out, err := m.test(context.Background(), map[string]any{"workspace": "w3", "module": "modules/m", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed || r.NoTests || len(r.Ran) != 2 {
t.Errorf("the fixture's unit level: result = %+v", r)
}
for p, body := range map[string]string{
"modules/p/package.json": `{"name":"p","type":"module","scripts":{"build":"tsc","test":"node --test test/a.test.mjs","prepare":"tsc"}}`,
"modules/p/tsconfig.json": `{"compilerOptions":{"module":"NodeNext","moduleResolution":"NodeNext","target":"ES2022","outDir":"dist","rootDir":"src"},"include":["src"]}`,
"modules/p/src/index.ts": "export const answer: number = 42;\n",
"modules/p/test/a.test.mjs": "import { test } from 'node:test';\nimport assert from 'node:assert';\nimport { answer } from '../dist/index.js';\n" +
"test('the build is there', () => assert.equal(answer, 42));\n",
} {
write(t, filepath.Join(work, "w3", "tree", filepath.FromSlash(p)), body, 0o644)
}
out, err = m.test(context.Background(), map[string]any{"workspace": "w3", "module": "modules/p", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed || !slices.Equal(r.Ran, []string{"build", "test"}) {
t.Errorf("the package importing its build: result = %+v", r)
}
var digests []string var digests []string
for _, id := range []string{"w1", "w2"} { for _, id := range []string{"w1", "w2"} {
out, err := m.build(context.Background(), map[string]any{"workspace": id, "module": "modules/m", "artifact": "tools"}) out, err := m.build(context.Background(), map[string]any{"workspace": id, "module": "modules/m", "artifact": "tools"})
@@ -89,43 +118,6 @@ func TestAFixtureBundleBuiltTwiceHasOneDigest(t *testing.T) {
} }
} }
// 1491: the real type-check and test script of the same fixture pass the unit level.
func TestTheFixturePassesTheUnitLevel(t *testing.T) {
m, _ := realMachine(t)
out, err := m.test(context.Background(), map[string]any{"workspace": "w1", "module": "modules/m", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed || r.NoTests || len(r.Ran) != 2 {
t.Errorf("result = %+v", r)
}
}
// Issue 1869: a package whose test imports its own build output (as mesh-sdk's do) passes the unit level: the test step
// emits it with the real tsc, shut off, before the test script runs.
func TestAPackageWhoseTestsImportItsBuildPassesTheUnitLevel(t *testing.T) {
if _, err := exec.LookPath("bwrap"); err != nil {
t.Skip("NOT RUN: no bubblewrap installed where this check runs; every build node has it (the module declares it)")
}
m, work := realMachine(t)
for p, body := range map[string]string{
"modules/p/package.json": `{"name":"p","type":"module","scripts":{"build":"tsc","test":"node --test test/a.test.mjs","prepare":"tsc"}}`,
"modules/p/tsconfig.json": `{"compilerOptions":{"module":"NodeNext","moduleResolution":"NodeNext","target":"ES2022","outDir":"dist","rootDir":"src"},"include":["src"]}`,
"modules/p/src/index.ts": "export const answer: number = 42;\n",
"modules/p/test/a.test.mjs": "import { test } from 'node:test';\nimport assert from 'node:assert';\nimport { answer } from '../dist/index.js';\n" +
"test('the build is there', () => assert.equal(answer, 42));\n",
} {
write(t, filepath.Join(work, "w1", "tree", filepath.FromSlash(p)), body, 0o644)
}
out, err := m.test(context.Background(), map[string]any{"workspace": "w1", "module": "modules/p", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed || !slices.Equal(r.Ran, []string{"build", "test"}) {
t.Errorf("result = %+v", r)
}
}
// installedToolchain is the toolchain directory under toolchains that this node's Node build module installed: the // installedToolchain is the toolchain directory under toolchains that this node's Node build module installed: the
// one installed from the lock at lockPath, byte for byte, or, when the change carries a lock no delivered release has // one installed from the lock at lockPath, byte for byte, or, when the change carries a lock no delivered release has
// installed yet, the one installed last, said in the test's log (ADR 0533 §3: a new lock is proven by the module that // installed yet, the one installed last, said in the test's log (ADR 0533 §3: a new lock is proven by the module that
@@ -9,10 +9,12 @@ package main
import ( import (
"context" "context"
"fmt" "fmt"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"os" "os"
"os/user" "os/user"
"path/filepath" "path/filepath"
"strings" "strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -46,6 +48,8 @@ func main() {
Emit: func(t string, b any) error { return stdio.Emit(t, b) }, Emit: func(t string, b any) error { return stdio.Emit(t, b) },
Steps: stdio.State(delivery.StepsState).Put, Steps: stdio.State(delivery.StepsState).Put,
ReadSteps: readOf(stdio.State(delivery.StepsState)), ReadSteps: readOf(stdio.State(delivery.StepsState)),
Pauses: stdio.State(delivery.PausesState).Put,
ReadPauses: readOf(stdio.State(delivery.PausesState)),
ReadToolchainsState: readOf(stdio.State(delivery.ToolchainsState)), ReadToolchainsState: readOf(stdio.State(delivery.ToolchainsState)),
Work: WorkDir, Work: WorkDir,
Run: forgekey.ExecRunnerClean(0), Run: forgekey.ExecRunnerClean(0),
@@ -67,7 +71,13 @@ func main() {
os.Exit(1) os.Exit(1)
} }
steps := m.StepTools() steps := m.StepTools()
booted := time.Now()
puller := m.Puller(steps, nil) puller := m.Puller(steps, nil)
m.Held = puller
if m.Node != "" {
// A step a previous process left running is ended (issue 1890); this process's own steps start after booted.
go buildstep.EndStaleAtStart(ctx, m.stepScan(), m.Node, booted, logf)
}
if m.Node != "" { if m.Node != "" {
// The seat's asks, pulled once Serve has the runtime's channel; a pull before it is said and tried again. // The seat's asks, pulled once Serve has the runtime's channel; a pull before it is said and tried again.
go func() { _ = puller.Pull(ctx) }() go func() { _ = puller.Pull(ctx) }()
@@ -94,9 +94,13 @@ func TestManifestKeepsItsStatesClaimsItsBuildSeatAndDeclaresItsHealthTool(t *tes
states := []string{} states := []string{}
for _, s := range m.State { for _, s := range m.State {
states = append(states, s.Name) states = append(states, s.Name)
// A step's key is per step and dropped after its age, so the bucket cannot fill; a pause has no age.
if (s.Name == "steps") != (s.TTLSeconds > 0) {
t.Errorf("state %s has ttl-seconds %d: only steps has an age", s.Name, s.TTLSeconds)
}
} }
slices.Sort(states) slices.Sort(states)
if want := []string{"keys", "steps", "toolchains"}; !slices.Equal(states, want) { if want := []string{"keys", "pauses", "steps", "toolchains"}; !slices.Equal(states, want) {
t.Errorf("state = %v, want %v", states, want) t.Errorf("state = %v, want %v", states, want)
} }
if len(m.Claims) != 1 || m.Claims[0].Name != Seat || !slices.Equal(m.Claims[0].Serves, []string{"current", "pause", "resume"}) { if len(m.Claims) != 1 || m.Claims[0].Name != Seat || !slices.Equal(m.Claims[0].Serves, []string{"current", "pause", "resume"}) {
@@ -42,7 +42,12 @@ type Module struct {
// Toolchains and Steps are the Put of the states of those names. // Toolchains and Steps are the Put of the states of those names.
Toolchains, Steps buildstep.Put Toolchains, Steps buildstep.Put
// ReadSteps and ReadToolchainsState read those two states back for the read tools (issue 1562). // ReadSteps and ReadToolchainsState read those two states back for the read tools (issue 1562).
// Held is the puller whose running asks a step-end tool will not end (issue 1890); nil in a test.
Held *buildstep.Puller
ReadSteps, ReadToolchainsState StateRead ReadSteps, ReadToolchainsState StateRead
// Pauses is the Put of the pauses state, which keeps the node's pause key without an age; ReadPauses reads it back.
Pauses buildstep.Put
ReadPauses StateRead
// Emit tells the bus an event (stdio.Emit); nil emits nothing. The guard emits toolchain.reported after each write of // Emit tells the bus an event (stdio.Emit); nil emits nothing. The guard emits toolchain.reported after each write of
// the toolchains key, so the indexer, which holds no timer, learns that the bucket exists (issue 1444). // the toolchains key, so the indexer, which holds no timer, learns that the bucket exists (issue 1444).
Emit func(eventType string, body any) error Emit func(eventType string, body any) error
@@ -134,6 +139,7 @@ func (m *Module) Start(ctx context.Context) {
m.Say("the toolchain: %v", err) m.Say("the toolchain: %v", err)
} }
m.CleanWorkspaces(m.now()) m.CleanWorkspaces(m.now())
buildstep.EndStaleRound(m.stepScan(), m.Node, m.Say) // asks past the step bound, on the module's own round (issue 1890)
select { select {
case <-ctx.Done(): case <-ctx.Done():
return return
@@ -117,7 +117,7 @@ func TestANodeThatCannotBeRunRefusesEveryStepAndHealthIsUnhealthy(t *testing.T)
{Name: "short_probe", Run: run}, {Name: "short_probe", Run: run},
}, (&Module{}).StepTools()...) }, (&Module{}).StepTools()...)
for _, tool := range m.Tools(steps) { for _, tool := range m.Tools(steps) {
if tool.Name == StatusTool || tool.Name == HealthTool || tool.Name == StepReadTool || tool.Name == StepsReadTool || tool.Name == ToolchainsReadTool || tool.Name == ReleaseTool { if tool.Name == StatusTool || tool.Name == HealthTool || tool.Name == StepReadTool || tool.Name == StepsReadTool || tool.Name == ToolchainsReadTool || tool.Name == ReleaseTool || tool.Name == StepEndTool {
continue continue
} }
var reason string var reason string
@@ -7,9 +7,12 @@ package main
import ( import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"log"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -39,6 +42,53 @@ func readOf(s stdio.KeptState) StateRead {
}} }}
} }
// MaxStateReads is the most values one read tool reads from a state in a call, and stateReadParallel how many of them
// are in flight at once. Every value is a round trip from the tool through the runtime to the bus, so a call that read
// every key of a large state one after the other outlasted the call's 30 s (issue 1827: 1164 keys, 20 MB).
const (
MaxStateReads = 1000
stateReadParallel = 16
)
// readValues reads the values of keys, stateReadParallel at a time, in the order of keys (nil for a key with none). It
// stops starting reads after the first error and returns it.
func readValues(keys []string, get func(string) (json.RawMessage, error)) ([]json.RawMessage, error) {
out := make([]json.RawMessage, len(keys))
var (
wg sync.WaitGroup
mu sync.Mutex
fail error
next = make(chan int)
)
for w := 0; w < stateReadParallel && w < len(keys); w++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := range next {
raw, err := get(keys[i])
mu.Lock()
if err != nil && fail == nil {
fail = err
}
mu.Unlock()
out[i] = raw
}
}()
}
for i := range keys {
mu.Lock()
stop := fail != nil
mu.Unlock()
if stop {
break
}
next <- i
}
close(next)
wg.Wait()
return out, fail
}
// stepKey is the key of a step of this node: the id alone, or the whole key <node>.<id> as a step tool answered it. // stepKey is the key of a step of this node: the id alone, or the whole key <node>.<id> as a step tool answered it.
func (m *Module) stepKey(id string) (string, error) { func (m *Module) stepKey(id string) (string, error) {
id = strings.TrimSpace(id) id = strings.TrimSpace(id)
@@ -78,29 +128,44 @@ type StepLine struct {
Started string `json:"started"` Started string `json:"started"`
Ended string `json:"ended,omitempty"` Ended string `json:"ended,omitempty"`
Reason string `json:"reason,omitempty"` Reason string `json:"reason,omitempty"`
What string `json:"what,omitempty"`
} }
// RecentSteps are this node's steps, the latest started first, at most limit (ten when limit is not above zero). // RecentSteps are this node's steps, the latest started first, at most limit (ten when limit is not above zero). Only
func (m *Module) RecentSteps(limit int) ([]StepLine, error) { // the node's own keys (<node>.) are listed and the newest max(limit*4, 100) of them read (at most MaxStateReads): a step's
// id is base-36 milliseconds and random digits, so its string order is its start order. cut says older steps were not
// read. The values still order the lines, since an ask step's id carries the ask time.
func (m *Module) RecentSteps(limit int) (lines []StepLine, cut bool, err error) {
if limit <= 0 { if limit <= 0 {
limit = 10 limit = 10
} }
if m.ReadSteps.Keys == nil || m.ReadSteps.Get == nil { if m.ReadSteps.Keys == nil || m.ReadSteps.Get == nil {
return nil, fmt.Errorf("the steps state cannot be read here") return nil, false, fmt.Errorf("the steps state cannot be read here")
} }
start := time.Now()
keys, err := m.ReadSteps.Keys() keys, err := m.ReadSteps.Keys()
if err != nil { if err != nil {
return nil, err return nil, false, err
}
own := keys[:0:0]
for _, k := range keys {
if strings.HasPrefix(k, m.Node+".") {
if node, _, err := delivery.ParseStepKey(k); err == nil && node == m.Node {
own = append(own, k)
}
}
}
sort.Sort(sort.Reverse(sort.StringSlice(own)))
if n := min(max(limit*4, 100), MaxStateReads); len(own) > n {
own, cut = own[:n], true
}
raws, err := readValues(own, m.ReadSteps.Get)
log.Printf("steps read: %d of %d keys listed read in %v (cut %v)", len(own), len(keys), time.Since(start).Round(time.Millisecond), cut)
if err != nil {
return nil, cut, err
} }
var steps []delivery.Step var steps []delivery.Step
for _, k := range keys { for _, raw := range raws {
if node, _, err := delivery.ParseStepKey(k); err != nil || node != m.Node {
continue
}
raw, err := m.ReadSteps.Get(k)
if err != nil {
return nil, err
}
if len(raw) == 0 { if len(raw) == 0 {
continue continue
} }
@@ -116,13 +181,13 @@ func (m *Module) RecentSteps(limit int) ([]StepLine, error) {
} }
out := make([]StepLine, 0, len(steps)) out := make([]StepLine, 0, len(steps))
for _, s := range steps { for _, s := range steps {
l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason} l := StepLine{ID: s.ID, Tool: s.Tool, State: s.State, Started: s.Started.UTC().Format("2006-01-02T15:04:05Z"), Reason: s.Reason, What: s.What}
if !s.Ended.IsZero() { if !s.Ended.IsZero() {
l.Ended = s.Ended.UTC().Format("2006-01-02T15:04:05Z") l.Ended = s.Ended.UTC().Format("2006-01-02T15:04:05Z")
} }
out = append(out, l) out = append(out, l)
} }
return out, nil return out, cut, nil
} }
// ReadToolchains is every key of the toolchains state (<node>.<language>) with its stored value. // ReadToolchains is every key of the toolchains state (<node>.<language>) with its stored value.
@@ -130,24 +195,30 @@ func (m *Module) ReadToolchains() (map[string]delivery.Toolchain, error) {
if m.ReadToolchainsState.Keys == nil || m.ReadToolchainsState.Get == nil { if m.ReadToolchainsState.Keys == nil || m.ReadToolchainsState.Get == nil {
return nil, fmt.Errorf("the toolchains state cannot be read here") return nil, fmt.Errorf("the toolchains state cannot be read here")
} }
start := time.Now()
keys, err := m.ReadToolchainsState.Keys() keys, err := m.ReadToolchainsState.Keys()
if err != nil { if err != nil {
return nil, err return nil, err
} }
cut := len(keys) > MaxStateReads
if cut {
keys = keys[:MaxStateReads] // a node has a key per language: far fewer in practice
}
raws, err := readValues(keys, m.ReadToolchainsState.Get)
log.Printf("toolchains read: %d keys read in %v (cut %v: keys past %d were not read)", len(keys), time.Since(start).Round(time.Millisecond), cut, MaxStateReads)
if err != nil {
return nil, err
}
out := map[string]delivery.Toolchain{} out := map[string]delivery.Toolchain{}
for _, k := range keys { for i, raw := range raws {
raw, err := m.ReadToolchainsState.Get(k)
if err != nil {
return nil, err
}
if len(raw) == 0 { if len(raw) == 0 {
continue continue
} }
t, err := delivery.DecodeToolchain(raw) t, err := delivery.DecodeToolchain(raw)
if err != nil { if err != nil {
return nil, fmt.Errorf("%s: %w", k, err) return nil, fmt.Errorf("%s: %w", keys[i], err)
} }
out[k] = t out[keys[i]] = t
} }
return out, nil return out, nil
} }
@@ -171,8 +242,13 @@ func (m *Module) ReadTools() []stdio.Tool {
Input: map[string]any{"type": "object", "properties": map[string]any{ Input: map[string]any{"type": "object", "properties": map[string]any{
"limit": map[string]any{"type": "integer", "description": "how many, at most; 10 when left out"}}}, "limit": map[string]any{"type": "integer", "description": "how many, at most; 10 when left out"}}},
Run: func(args map[string]any) (any, error) { Run: func(args map[string]any) (any, error) {
steps, err := m.RecentSteps(intArg(args["limit"])) steps, cut, err := m.RecentSteps(intArg(args["limit"]))
return map[string]any{"node": m.Node, "steps": steps}, err ans := map[string]any{"node": m.Node, "steps": steps}
if cut {
ans["cut"] = true
ans["note"] = fmt.Sprintf("this node has more steps stored than one call reads; older steps were not read (ask for a larger limit, at most %d reads)", MaxStateReads)
}
return ans, err
}}, }},
{Name: ToolchainsReadTool, {Name: ToolchainsReadTool,
Description: "What the module's toolchains state holds: each key <node>.<language> with the version installed, the " + Description: "What the module's toolchains state holds: each key <node>.<language> with the version installed, the " +
@@ -3,8 +3,10 @@ package main
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"slices" "slices"
"strings" "strings"
"sync/atomic"
"testing" "testing"
"time" "time"
@@ -94,7 +96,7 @@ func TestStepsToolListsRecentStepsOfThisNode(t *testing.T) {
t0 := time.Date(2026, 10, 21, 9, 0, 0, 0, time.UTC) t0 := time.Date(2026, 10, 21, 9, 0, 0, 0, time.UTC)
putStep(t, steps, delivery.Step{ID: "a", Tool: "build_node_fetch", Node: "ace", State: delivery.StepPassed, Started: t0, Ended: t0, Result: json.RawMessage(`1`)}) putStep(t, steps, delivery.Step{ID: "a", Tool: "build_node_fetch", Node: "ace", State: delivery.StepPassed, Started: t0, Ended: t0, Result: json.RawMessage(`1`)})
putStep(t, steps, delivery.Step{ID: "b", Tool: "build_node_test", Node: "ace", State: delivery.StepRunning, Started: t0.Add(time.Minute)}) putStep(t, steps, delivery.Step{ID: "b", Tool: "build_node_test", Node: "ace", State: delivery.StepRunning, Started: t0.Add(time.Minute)})
putStep(t, steps, delivery.Step{ID: "c", Tool: "build_node_build", Node: "ace", State: delivery.StepFailed, Started: t0.Add(2 * time.Minute), Reason: "boom"}) putStep(t, steps, delivery.Step{ID: "c", Tool: "build_node_build", Node: "ace", State: delivery.StepFailed, Started: t0.Add(2 * time.Minute), Reason: "boom", What: "check ask, novox/x@abc"})
putStep(t, steps, delivery.Step{ID: "d", Tool: "build_node_build", Node: "g14", State: delivery.StepPassed, Started: t0.Add(time.Hour)}) putStep(t, steps, delivery.Step{ID: "d", Tool: "build_node_build", Node: "g14", State: delivery.StepPassed, Started: t0.Add(time.Hour)})
m := readModule(steps, fakeState{}) m := readModule(steps, fakeState{})
out, err := tool(t, m, StepsReadTool)(map[string]any{}) out, err := tool(t, m, StepsReadTool)(map[string]any{})
@@ -109,7 +111,7 @@ func TestStepsToolListsRecentStepsOfThisNode(t *testing.T) {
if !slices.Equal(ids, []string{"c", "b", "a"}) { if !slices.Equal(ids, []string{"c", "b", "a"}) {
t.Errorf("ids = %v, want c b a", ids) t.Errorf("ids = %v, want c b a", ids)
} }
if lines[0].Reason != "boom" || lines[0].State != delivery.StepFailed { if lines[0].Reason != "boom" || lines[0].What != "check ask, novox/x@abc" || lines[0].State != delivery.StepFailed {
t.Errorf("first = %+v", lines[0]) t.Errorf("first = %+v", lines[0])
} }
out, _ = tool(t, m, StepsReadTool)(map[string]any{"limit": float64(1)}) out, _ = tool(t, m, StepsReadTool)(map[string]any{"limit": float64(1)})
@@ -145,3 +147,62 @@ func TestReadToolsAreInTheManifestAndNeedNoToolchain(t *testing.T) {
t.Errorf("the read refused: %v", err) t.Errorf("the read refused: %v", err)
} }
} }
// 1827: on the live shape (4 nodes, 600 steps each; a steps state of 1164 keys / 20 MB timed out at 30 s) the steps tool
// lists, then reads only this node's keys, in parallel, at most MaxStateReads of them, and says when it cut.
func TestRecentStepsReadsOnlyTheNodesKeysInParallelAndBounded(t *testing.T) {
t0 := time.Date(2026, 10, 25, 9, 0, 0, 0, time.UTC)
for _, row := range []struct {
perNode, limit int
reads int64
steps int
cut bool
}{
{20, 0, 20, 10, false},
{600, 0, 100, 10, true},
{1500, 300, MaxStateReads, 300, true},
} {
var reads, inFlight, maxFlight atomic.Int64
stored := map[string][]byte{}
for _, n := range []string{"ace", "g14", "novox", "shanks"} {
for i := 0; i < row.perNode; i++ {
b, _ := delivery.EncodeStep(delivery.Step{ID: fmt.Sprintf("s%04d", i), Tool: "t", Node: n, State: delivery.StepPassed, Started: t0.Add(time.Duration(i) * time.Second)})
stored[n+".s"+fmt.Sprintf("%04d", i)] = b
}
}
m := &Module{Node: "shanks"}
m.ReadSteps = StateRead{
Keys: func() ([]string, error) {
var k []string
for key := range stored {
k = append(k, key)
}
return k, nil
},
Get: func(key string) (json.RawMessage, error) {
reads.Add(1)
now := inFlight.Add(1)
for o := maxFlight.Load(); now > o && !maxFlight.CompareAndSwap(o, now); o = maxFlight.Load() {
}
time.Sleep(time.Millisecond)
inFlight.Add(-1)
if !strings.HasPrefix(key, "shanks.") {
t.Errorf("read %s, a key of another node", key)
}
return stored[key], nil
},
}
start := time.Now()
lines, cut, err := m.RecentSteps(row.limit)
if err != nil || len(lines) != row.steps || cut != row.cut || reads.Load() != row.reads {
t.Errorf("%d a node: %d steps, cut %v, %d reads, err %v; want %d, %v, %d", row.perNode, len(lines), cut, reads.Load(), err, row.steps, row.cut, row.reads)
}
if row.cut && len(lines) > 0 && lines[0].ID != fmt.Sprintf("s%04d", row.perNode-1) {
t.Errorf("%d a node: newest answered %s", row.perNode, lines[0].ID)
}
if maxFlight.Load() < 2 {
t.Errorf("%d a node: values read one at a time", row.perNode)
}
t.Logf("%d a node: %d reads in %v (one after the other: %v)", row.perNode, reads.Load(), time.Since(start), time.Duration(reads.Load())*time.Millisecond)
}
}
@@ -2,6 +2,7 @@ package main
import ( import (
"context" "context"
"git.novox.be/novox/mesh-sdk/go/buildstep"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
) )
@@ -40,6 +41,7 @@ func (m *Module) Tools(steps []StepTool) []stdio.Tool {
tools = append(tools, t) tools = append(tools, t)
} }
tools = append(tools, m.ReadTools()...) tools = append(tools, m.ReadTools()...)
tools = append(tools, buildstep.StepEndTool(StepEndTool, m.Node, m.stepScan(), m.Held))
tools = append(tools, m.releaseTool()) tools = append(tools, m.releaseTool())
return append(tools, return append(tools,
stdio.Tool{Name: StatusTool, stdio.Tool{Name: StatusTool,
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildnode
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125154-6b8497314297
github.com/distribution/distribution/v3 v3.1.2 github.com/distribution/distribution/v3 v3.1.2
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93 h1:FAiFY2KBKEoZSH49OZTt7/zRBIQxyl5g/KhYrpggbN4= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125154-6b8497314297 h1:k31FtFp/DGXGatRjKV1S0z1gBAHfVgtvBX6xF0g1Vgw=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125154-6b8497314297/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI= github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI=
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM= github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
+2 -1
View File
@@ -10,7 +10,7 @@
"state": [ "state": [
"toolchains", "toolchains",
"keys", "keys",
"steps" {"name": "steps", "ttl-seconds": 172800}, "pauses"
], ],
"claims": [ "claims": [
{ {
@@ -34,6 +34,7 @@
"build_node_release", "build_node_release",
"build_node_step", "build_node_step",
"build_node_steps", "build_node_steps",
"build_node_step_end",
"build_node_toolchains", "build_node_toolchains",
"build_node_status", "build_node_status",
"build_node_health" "build_node_health"
+16 -28
View File
@@ -10,19 +10,22 @@ Each directory or archive this module declares says how it treats a file it did
- `${dir:state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2). - `${dir:state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
## A backup that failed before the send ## What it keeps (ADR 0539 §17)
The controller backs up the data of the modules a delivery moves before it asks or sends anything (ADR 0404). When its The gate keeps the merge's states: proposed, ready, rejected, merged, held (a merge that reached the trunk without a passing
walk's record, in `plan-moved`, names a backup that failed or stayed silent and nothing of the plan was sent, the check) and closed. It asks the operator's Approve and merges once, moving `source.main`. It asks the controller nothing, follows
delivery goes from delivering to held (the observed row `backup failed`); the held reason names the machine, the no plan event and derives nothing after the merge: whether a merged change reached the machines is the view's, from
module and the backup's words, and the pull request's comment shows it. The hold does not lift by itself. `release` `source.main`, the wants and the haves. The seat `mesh-delivery` keeps the verbs `deliveries`, `show`, `groups`, `checks`,
of such a delivery asks the controller (`delivery-backups-again`) to take the backups again and the delivery goes on `table`, `stalled` (for a merge or a check past its bound), `recheck` and `close`; `times`, `what-if`, `release`, `stop` and
delivering; a record from before the controller cleared the failure does not hold it again. A held delivery already `retire-history` went with the plans, so the gate stays a bundle serving the seat. A delivery an earlier owner kept in a state that
ends as covered or delivered when a later merge covers it. is gone is read as the state it came to (merged, closed or proposed) when the gate starts.
## When a refused or rejected merge is said (ADR 0505 §4, §5) ## When a refused or rejected merge is said (ADR 0505 §4, §5)
`stalled` is what the controller raises as a condition for the operator. The gate's own list (`show`, `deliveries`) `stalled` is what the gate says to the operator as conditions, itself, on the operator-channel seat (ADR 0437 §9, issue
1275): after every look it raises each line, raises it again when its severity or need changes, and clears each that is
listed no more; its bucket `conditions` keeps which it raised. (The controller's self-check that raised them went in
release 6.) The gate's own list (`show`, `deliveries`)
names every refusal and rejection at once, and the pull request is told at once; only `stalled` waits: names every refusal and rejection at once, and the pull request is told at once; only `stalled` waits:
- a merge refused because its trunk moved, whose check the gate asked again, is stalled only once that check answered and - a merge refused because its trunk moved, whose check the gate asked again, is stalled only once that check answered and
@@ -33,23 +36,8 @@ names every refusal and rejection at once, and the pull request is told at once;
`needs: new-head` and names its author: the branch, and the session a `Pushed-by: <who@node>` line in the pull `needs: new-head` and names its author: the branch, and the session a `Pushed-by: <who@node>` line in the pull
request's description names. request's description names.
## Taking the seat over from today's module ## The seat's hand-over
The gate claims the seat `mesh-delivery` and took it over from the module `mesh-delivery`, which the catalogue no longer holds. The gate took the seat `mesh-delivery` over from the module `mesh-delivery` (ADR 0449 §4, ADR 0450 §6, ADR 0454) and carried its
The controller refuses a merge that removes a module that is assigned, so today's module stayed in the catalogue until it was deliveries, groups and pull keys once; the carry, and the reading of the controller's assignments and the node-engine's have keys
unassigned, and went by a handover of verbs after the merge, never by hand on a machine (novox/hq ADR 0449 §4, ADR 0450 §6, ADR 0454): that waited for it, went with release 6. `catalogue_test.go` keeps the catalogue to one claimant of the seat.
1. assign `mesh-delivery-gate` to the machine that holds the seat (`mesh-controller.assign`);
2. check that `mesh-controller.seats` still names `mesh-delivery`, held by the running module;
3. unassign `mesh-delivery` from that machine (`mesh-controller.unassign`) and push it (`mesh-controller.push`);
4. the seat passes: the gate reads the controller's assignments and the node-engine's have keys, finds the module on no machine,
carries its deliveries, groups and pull keys over once (the key `carried` in its `deliveries` says it did), reads the controller's
walks and the forge's open pull requests, and acts.
Until then the gate is silent: it takes no event, writes no key, asks nothing and answers no verb but `delivery_status`, which
says why (`holds-the-seat`, `seat-waits`).
The removal of `modules/mesh-delivery` followed the handover as a pull request of its own, merged by the gate (ADR 0454); the
catalogue has one claimant of the seat, and no manifest but the gate's and the issue tracker's names the old module but the names
the handover needed: the three buckets the gate carries over, and the event the issue tracker and the tool runner hear
(`catalogue_test.go`).
@@ -1,50 +1,32 @@
package main package main
import ( import (
"strings"
"testing" "testing"
"time" "time"
) )
// One build per change (novox/hq ADR 0350): a pull request closed unmerged has its check builds let go of, and a const (
// walk's record says per module whether its build reused a check's build or built, and why. cat = "novox/mesh-catalog"
headCC = "cc00000000000001"
headDu = "dd00000000000002"
)
// A pull request closed unmerged is stopped. Since issue 1318 (ADR 0503 §3) the controller is no longer asked to let go var atCC = time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
// of what its checks built (§5's forget-checks goes with the merge gate's checks); a controller that is down changes
// nothing. // A pull request closed unmerged is closed. Since issue 1318 (ADR 0503 §3) the controller is no longer asked to let go
func TestAPullRequestClosedUnmergedIsStoppedAndAsksTheControllerNothing(t *testing.T) { // of what its checks built, and since ADR 0539 §17 the gate asks it nothing at all.
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atCC) func TestAPullRequestClosedUnmergedIsClosedAndAsksNoOne(t *testing.T) {
w := newWorldOver(t, newMemStore(), newFakeForge(), atCC)
cc := pr(cat, 101, headCC, "feat/claude-code", "modules/claude-code/module.json") cc := pr(cat, 101, headCC, "feat/claude-code", "modules/claude-code/module.json")
w.h.PullUpdated(cc) w.h.PullUpdated(cc)
w.h.PullClosed(cc) w.h.PullClosed(cc)
if len(w.ctl.forgot) != 0 { if s := w.state(IDOf(cat, headCC)); s != Closed {
t.Fatalf("closed unmerged, the controller was asked to let go of %v", w.ctl.forgot)
}
if s := w.state(IDOf(cat, headCC)); s != Stopped {
t.Fatalf("closed unmerged it is %s", s) t.Fatalf("closed unmerged it is %s", s)
} }
w.ctl.down = true
du := pr(cat, 102, headDu, "feat/dunst", "modules/dunst/module.json") du := pr(cat, 102, headDu, "feat/dunst", "modules/dunst/module.json")
w.h.PullUpdated(du) w.h.PullUpdated(du)
w.h.PullClosed(du) w.h.PullClosed(du)
if s := w.state(IDOf(cat, headDu)); s != Stopped { if s := w.state(IDOf(cat, headDu)); s != Closed {
t.Fatalf("with the controller down, a closed delivery is %s", s) t.Fatalf("a closed delivery is %s", s)
}
}
// The walk's record says per module what its build did, and the delivery keeps it beside its walk (§4).
func TestAWalksRecordSaysWhetherEachModuleWasReusedOrBuilt(t *testing.T) {
walk := catWalk("plan-1", mergeCC, atCC.Add(time.Minute), "claude-code", "dunst")
walk.Modules["claude-code"].Made = "reused from the check of novox/mesh-catalog#101 at head cccccccccccc " +
"(merged into trunk 553b71910000): tools sha256:aa"
walk.Modules["dunst"].Made = "built: no check of novox/mesh-catalog#101 built it"
seen := walk.Seen()
if !strings.HasPrefix(seen.Built["claude-code"], "reused from the check of novox/mesh-catalog#101") ||
seen.Built["dunst"] != "built: no check of novox/mesh-catalog#101 built it" {
t.Fatalf("the delivery keeps %v", seen.Built)
}
if catWalk("plan-2", mergeCC, atCC, "dunst").Seen().Built != nil {
t.Fatal("a walk from a controller that says nothing of its builds reads as if it said something")
} }
} }
@@ -1,14 +1,11 @@
package main package main
import ( import (
"encoding/json"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
"time" "time"
stdio "git.novox.be/novox/mesh-sdk/go"
) )
// The tests of novox/hq ADR 0505 "How it is checked" for the gate: a refusal the gate is already answering is no // The tests of novox/hq ADR 0505 "How it is checked" for the gate: a refusal the gate is already answering is no
@@ -39,6 +36,15 @@ func TestTheBoundsAreSettingsWithTheirDefaults(t *testing.T) {
if h.refusalFloor() != 10*time.Minute || h.authorBound() != 2*time.Hour { if h.refusalFloor() != 10*time.Minute || h.authorBound() != 2*time.Hour {
t.Fatalf("empty file: floor %s, author %s", h.refusalFloor(), h.authorBound()) t.Fatalf("empty file: floor %s, author %s", h.refusalFloor(), h.authorBound())
} }
if !h.checkGates() {
t.Fatalf("check-gates-merge unset reads false; the default is true")
}
if s, err = read(`{"check-gates-merge": false}`); err != nil || s.CheckGatesMerge == nil || *s.CheckGatesMerge || h.checkGates() {
t.Fatalf("check-gates-merge false: %+v %v, gates %v", s, err, h.checkGates())
}
if _, err := read(`{"check-gates-merge": "no"}`); err == nil {
t.Fatalf("a check-gates-merge that is not true or false was taken")
}
if _, err := read(`{"author-new-head-minutes": 0}`); err == nil { if _, err := read(`{"author-new-head-minutes": 0}`); err == nil {
t.Fatalf("a bound of zero minutes was taken") t.Fatalf("a bound of zero minutes was taken")
} }
@@ -76,7 +82,7 @@ func refusedForAMovedTrunk(t *testing.T) (*world, string) {
t.Helper() t.Helper()
w := announced(t) w := announced(t)
id := IDOf("novox/app", readyHead) id := IDOf("novox/app", readyHead)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
w.forge.refuse = errAskedAgain w.forge.refuse = errAskedAgain
approve(w) approve(w)
if len(w.forge.merges) != 1 || w.delivery(id).Ready.RefusedSince == nil { if len(w.forge.merges) != 1 || w.delivery(id).Ready.RefusedSince == nil {
@@ -125,7 +131,7 @@ func TestATrunkMovedRefusalIsStalledAtTwiceTheLastCheckTimeAtLeastTheFloor(t *te
func TestAnyOtherRefusalIsStalledAtOnce(t *testing.T) { func TestAnyOtherRefusalIsStalledAtOnce(t *testing.T) {
w := announced(t) w := announced(t)
id := IDOf("novox/app", readyHead) id := IDOf("novox/app", readyHead)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
w.forge.refuse = refusedWords("novox/app#7") w.forge.refuse = refusedWords("novox/app#7")
approve(w) approve(w)
if got := linesFor(w, id); len(got) != 1 { if got := linesFor(w, id); len(got) != 1 {
@@ -146,7 +152,7 @@ func rejectedAfterItsApprove(t *testing.T, pushedBy string) (*world, string) {
w.h.PullUpdated(p) w.h.PullUpdated(p)
w.settleAll() w.settleAll()
id := IDOf("novox/app", readyHead) id := IDOf("novox/app", readyHead)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
// Judged again (the table's recheck, which the checker's verdict on a new trunk makes): proposed once more. // Judged again (the table's recheck, which the checker's verdict on a new trunk makes): proposed once more.
w.h.mu.Lock() w.h.mu.Lock()
d := w.h.deliveries[id] d := w.h.deliveries[id]
@@ -156,7 +162,7 @@ func rejectedAfterItsApprove(t *testing.T, pushedBy string) (*world, string) {
} }
d.Check = nil d.Check = nil
w.h.mu.Unlock() w.h.mu.Unlock()
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "fail", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "fail", "t1t1t1t1t1t1"))
approve(w) approve(w)
w.h.React() w.h.React()
if w.state(id) != Rejected { if w.state(id) != Rejected {
@@ -220,48 +226,3 @@ func TestANewHeadWithinTwoHoursKeepsTheRejectedHeadFromTheOperator(t *testing.T)
t.Fatalf("the superseded head reached the operator: %+v", got) t.Fatalf("the superseded head reached the operator: %+v", got)
} }
} }
// ADR 0505 §2: a walk's end said again by the controller with its declared machines (`declared_read`, a touch marked
// declared) acts once: no transition, comment or ask of the controller the first saying did not make.
func TestAWalkSaidAgainWithItsDeclaredMachinesActsOnceInTheGate(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
done := w.ctl.walk(plan)
sent := time.Date(2026, 10, 6, 12, 40, 0, 0, time.UTC)
done.Modules["app"] = &WalkModule{State: "built", First: []string{"anchor"}, FirstAt: &sent, SentAt: &sent, Build: "build-1",
Gate: &WalkGate{Machines: []string{"anchor"}, Since: &sent, Passes: 1, Verdict: "passed", JudgedAt: &sent}}
done.State = walkDone
done = w.ctl.put(done)
body := func(again bool) []byte {
raw, err := json.Marshal(done)
if err != nil {
t.Fatal(err)
}
var said map[string]any
_ = json.Unmarshal(raw, &said)
said["walk"], said["phase"], said["outcome"] = plan, "ended", "delivered"
said["touches"] = []map[string]any{{"machine": "anchor", "modules": []string{"app"}}}
if again {
said["touches"] = append(said["touches"].([]map[string]any),
map[string]any{"machine": "shanks", "modules": []string{}, "declared": true})
said["declared_read"] = true
}
out, _ := json.Marshal(said)
return out
}
if err := take(w.h, stdio.Envelope{Key: ControllerSeat + ".plan-moved", Body: body(false)}); err != nil {
t.Fatal(err)
}
w.settleAll()
if s := w.state(id); s != Delivered {
t.Fatalf("the walk's end left it %s", s)
}
transitions, comments, delivers := len(w.delivery(id).Transitions), len(w.forge.comments), len(w.ctl.delivers)
if err := take(w.h, stdio.Envelope{Key: ControllerSeat + ".plan-moved", Body: body(true)}); err != nil {
t.Fatal(err)
}
w.settleAll()
if len(w.delivery(id).Transitions) != transitions || len(w.forge.comments) != comments || len(w.ctl.delivers) != delivers {
t.Fatalf("said again it acted again: transitions %d→%d, comments %d→%d, asks of the controller %d→%d",
transitions, len(w.delivery(id).Transitions), comments, len(w.forge.comments), delivers, len(w.ctl.delivers))
}
}
@@ -0,0 +1,246 @@
package main
import (
"fmt"
"strings"
"sync"
"testing"
"time"
sdk "git.novox.be/novox/mesh-sdk/go/delivery"
)
// One check per delivery (novox/hq ADR 0548), the gate's part: nothing merges into a repository while a check of other heads
// judges it (§2); an approved merge holds a turn on its repositories (§3); a held merge past its bound is a stalled line, read
// and handed by two tools (§4); an approved delivery whose trunk moved merges on its carried verdict with no second ask (§8).
const otherHead = "c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3"
// approvedLone is a lone delivery into oneRepo whose check passed, approved by the operator (the world approves at once).
func approvedLone(w *world) string {
w.t.Helper()
p := pr(oneRepo, 1, oneHead, "feat/one")
requiresCheck(w, p)
w.h.PullUpdated(p)
putCheckerVerdict(w, "pass", headOf(oneRepo, 1, oneHead))
w.h.CommitStatus(checkStatus(oneRepo, oneHead, "success", oneTrunk))
return IDOf(oneRepo, oneHead)
}
// memTurns is the bucket turns in memory; it logs each write with the merges the fake bridge had made by then.
type memTurns struct {
mu sync.Mutex
m map[string]sdk.Turn
log []string
merges func() int
}
func newMemTurns(w *world) *memTurns {
return &memTurns{m: map[string]sdk.Turn{}, merges: func() int { return len(w.forge.merges) }}
}
func (f *memTurns) All() (map[string]sdk.Turn, error) {
f.mu.Lock()
defer f.mu.Unlock()
out := map[string]sdk.Turn{}
for k, v := range f.m {
out[k] = v
}
return out, nil
}
func (f *memTurns) Put(t sdk.Turn) error {
if _, err := sdk.EncodeTurn(t); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
f.m[t.Repository] = t
f.log = append(f.log, fmt.Sprintf("put %s after %d merges", t.Repository, f.merges()))
return nil
}
func (f *memTurns) Delete(repository string) error {
f.mu.Lock()
defer f.mu.Unlock()
delete(f.m, repository)
f.log = append(f.log, fmt.Sprintf("delete %s after %d merges", repository, f.merges()))
return nil
}
func (f *memTurns) said() string { f.mu.Lock(); defer f.mu.Unlock(); return strings.Join(f.log, "; ") }
func TestOneCheckPerDeliveryAtTheGate(t *testing.T) {
for name, run := range map[string]func(t *testing.T){
// §2: a merge into a repository a running verdict names is not written to merges, and is written on that verdict's pass.
"an approved merge waits for a running check on its repository and merges on its end": func(t *testing.T) {
w := newWorld(t)
putCheckerVerdict(w, "running", headOf(oneRepo, 9, otherHead))
id := approvedLone(w)
w.settleAll()
if w.op.count() != 1 || len(w.forge.merges) != 0 {
t.Fatalf("asked %d times, merges %v: want the Approve asked and the merge held", w.op.count(), w.forge.merges)
}
d := w.delivery(id)
if d.Ready == nil || !strings.Contains(d.Ready.Waits, "waits for the check of g") || !strings.Contains(d.Ready.Waits, oneRepo) ||
d.Ready.HeldSince == nil {
t.Fatalf("the merge does not say what it waits for: %+v", d.Ready)
}
// §9: deliveries says what it waits for.
for _, l := range w.h.Deliveries("", "", "", false) {
if l.ID == id && !strings.Contains(l.Waits, "approved; waits for the check of g") {
t.Errorf("deliveries says it waits for %q", l.Waits)
}
}
// §4: past 30 minutes the held merge is a stalled line naming the check it waits for and since when.
for _, l := range w.h.Stalled() {
if l.ID == id {
t.Fatalf("a stalled line within the bound: %+v", l)
}
}
w.later(turnBound + time.Minute)
w.settleAll()
var line *StalledLine
for _, l := range w.h.Stalled() {
if l.ID == id {
line = &l
}
}
if line == nil || line.Bound != "30m0s" || !strings.Contains(line.Says, "waits for the check of g") || !strings.Contains(line.Says, "held since") ||
!strings.Contains(line.Says, "checker_status") || !strings.Contains(line.H2, "delivery_hand_now") {
t.Fatalf("the stalled line of the held merge = %+v", line)
}
putCheckerVerdict(w, "pass", headOf(oneRepo, 9, otherHead))
w.settleAll()
if len(w.forge.merges) != 1 || w.op.count() != 1 {
t.Fatalf("after the check's end: merges %v, asked %d times; want merged with no second ask", w.forge.merges, w.op.count())
}
},
"a running check on another repository holds nothing": func(t *testing.T) {
w := newWorld(t)
putCheckerVerdict(w, "running", headOf(oneRepo2, 9, otherHead))
approvedLone(w)
w.settleAll()
if len(w.forge.merges) != 1 {
t.Fatalf("merges %v: a check of another repository held the merge (%s)", w.forge.merges, w.logs())
}
},
"the delivery's own running verdict does not hold it": func(t *testing.T) {
w := newWorld(t)
w.op.auto = false
id := approvedLone(w)
w.settleAll()
putCheckerVerdict(w, "running", headOf(oneRepo, 1, oneHead)) // its own heads judged again (a carry)
for _, k := range w.op.open() {
w.op.answer(k, OptionApprove)
}
w.settleAll()
if d := w.delivery(id); len(w.forge.merges) != 1 || d.Ready == nil || d.Ready.HeldSince != nil {
t.Fatalf("merges %v, run %+v: its own check held it", w.forge.merges, d.Ready)
}
},
// §3: the turn is written before the verdicts are read for the merge, and deleted after the last merge.
"a lone merge's turn is written before it is handed and deleted once it merged": func(t *testing.T) {
w := newWorld(t)
turns := newMemTurns(w)
w.h.Turns = turns
approvedLone(w)
w.settleAll()
if len(w.forge.merges) != 1 || turns.said() != "put "+oneRepo+" after 0 merges; delete "+oneRepo+" after 1 merges" {
t.Fatalf("merges %v; the turns were %q", w.forge.merges, turns.said())
}
},
"a group's turns stand on every member's repository until its last member merged": func(t *testing.T) {
w := newWorld(t)
turns := newMemTurns(w)
w.h.Turns = turns
a, b := pr(oneRepo, 1, oneHead, "feat/two"), pr(oneRepo2, 2, oneHead2, "feat/two")
requiresCheck(w, a, b)
w.h.PullUpdated(a)
w.h.PullUpdated(b)
w.h.wantForge()
putCheckerVerdict(w, "pass", headOf(oneRepo, 1, oneHead), headOf(oneRepo2, 2, oneHead2))
w.h.CommitStatus(checkStatus(oneRepo, oneHead, "success", oneTrunk))
w.h.CommitStatus(checkStatus(oneRepo2, oneHead2, "success", oneTrunk))
for range 3 {
w.settleAll()
}
want := "put " + oneRepo2 + " after 0 merges; put " + oneRepo + " after 0 merges; delete " + oneRepo2 + " after 2 merges; delete " + oneRepo + " after 2 merges"
if len(w.forge.merges) != 2 || turns.said() != want {
t.Fatalf("merges %v; the turns were %q, want %q", w.forge.merges, turns.said(), want)
}
},
// §8: an approved delivery whose trunk moved goes back to proposed, waits for its carried verdict rather than being
// handed on the old one, and merges on it with no second ask.
"an approved merge whose trunk moved merges on its carried verdict with no second ask": func(t *testing.T) {
w := newWorld(t)
putCheckerVerdict(w, "running", headOf(oneRepo, 9, otherHead)) // holds the approved merge (§2)
id := approvedLone(w)
w.settleAll()
if w.op.count() != 1 || len(w.forge.merges) != 0 {
t.Fatalf("asked %d times, merges %v; want approved and held", w.op.count(), w.forge.merges)
}
w.h.PullMerged(merged(oneRepo, 5, otherHead, oneTrunk2)) // a merge at the forge moves the trunk
putCheckerVerdict(w, "pass", headOf(oneRepo, 9, otherHead))
w.settleAll()
if len(w.forge.merges) != 0 {
t.Fatalf("merges %v: handed on a verdict judged on the old trunk", w.forge.merges)
}
d := w.delivery(id)
if d.Ready == nil || !strings.Contains(d.Ready.Waits, "being carried") {
t.Fatalf("after the trunk moved it waits for %q, want its verdict carried", d.Ready.Waits)
}
// The checker's carried line: the head on the new trunk, carried from the old one.
carried := checkStatus(oneRepo, oneHead, "success", oneTrunk2)
carried.Description = "pass " + oneHead[:12] + " on " + oneTrunk2[:12] + " | carried from " + oneTrunk[:12] + " | unit 0 run, 4 kept"
if head, trunk := checkLine(carried.Description); head != oneHead[:12] || trunk != oneTrunk2[:12] {
t.Fatalf("the carried line reads as %s on %s", head, trunk)
}
if head, trunk := checkLine("failed: a test"); head != "" || trunk != "" {
t.Fatalf("a line that names no commits gave %q %q", head, trunk)
}
putCheckerVerdict(w, "pass", headOf(oneRepo, 1, oneHead))
w.h.CommitStatus(carried)
w.settleAll()
if w.op.count() != 1 {
t.Errorf("the Approve was asked %d times, want once: it stands through the carry", w.op.count())
}
if d := w.delivery(id); d.Check == nil || d.Check.Trunk != oneTrunk2[:12] || len(w.forge.merges) != 1 {
t.Errorf("after the carry: verdict %+v, merges %v; want merged on the carried verdict", d.Check, w.forge.merges)
}
},
// §4: delivery_turns reads every turn with the check it waits for; delivery_hand_now hands a held merge at once on the
// operator's word, and refuses one no check holds.
"the turns are read, and a held merge is handed now on the operator's word": func(t *testing.T) {
w := newWorld(t)
w.h.Turns = newMemTurns(w)
putCheckerVerdict(w, "running", headOf(oneRepo, 9, otherHead))
id := approvedLone(w)
w.settleAll()
read := w.h.TurnsRead()
lines := read["turns"].([]TurnLine)
if len(lines) != 1 || lines[0].Repository != oneRepo || len(lines[0].Turn.Waiting) != 1 || lines[0].Turn.Waiting[0].WaitsFor == "" ||
!strings.Contains(lines[0].Waits["feat/one"].(string), "the check g.") || read["written"] != true {
t.Fatalf("delivery_turns = %+v", read)
}
if _, err := w.h.HandNow(id, ""); err == nil || !strings.Contains(err.Error(), "why") {
t.Errorf("a hand without the operator's why: %v", err)
}
if _, err := w.h.HandNow(IDOf(oneRepo2, oneHead2), "x"); err == nil {
t.Error("an unknown delivery was handed")
}
if _, err := w.h.HandNow(id, "the operator, on telegram: merge it now"); err != nil {
t.Fatal(err)
}
w.settleAll()
if len(w.forge.merges) != 1 || w.delivery(id).Ready.HandNow == "" {
t.Fatalf("merges %v after the hand", w.forge.merges)
}
if _, err := w.h.HandNow(id, "again"); err == nil {
t.Error("a merged delivery was handed again")
}
},
} {
t.Run(name, run)
}
}
@@ -1,672 +0,0 @@
package main
import (
"fmt"
"sort"
"strings"
"time"
)
// Which merges a walk answers (novox/hq issue 362, ADR 0276).
//
// On 2026-10-10 two merges of the catalogue 18 seconds apart were walked by one walk at the later merge's
// commit, which superseded the first. This owner found the walk a delivery follows by the walk's id or by the
// commit it was merged as, final or not: the surviving walk matched the dunst delivery, which the supersession
// had already ended, so nothing gave it its word, and the claude-code delivery, merged as another commit,
// matched no walk and was held for a person after ten minutes, for a reason that named neither.
//
// A walk answers every merge it carries: the merges it names (`delivery.merges`, said by a controller that
// batches merges), the merge of its own commit, a merge whose walk it took over, and a merge of its repository
// and branch the forge says its commit contains. A delivery follows the newest walk that answers it, and only a
// delivery not yet ended is ever tied to a walk by what it answers: an ended delivery is moved only by the
// record of its own walk, which no longer moves it.
// seenWalk is a walk's last record and when it was heard.
type seenWalk struct {
walk Walk
at time.Time
}
// keptWalkFor is how long a walk no longer open is kept for a merge heard after it ended; a batch is kept as
// long, past its window's latest close.
const keptWalkFor = 10 * time.Minute
// answer is a delivery a walk answers and the commit of its repository the walk carries for it.
type answer struct {
d *Delivery
carried string
}
// walkAnswers is whether a walk answers a delivery's merge, and the commit it carries for it.
func walkAnswers(w Walk, d *Delivery) (string, bool) {
if w.Release != nil || w.IsBatch() || d.MergedAs == "" {
return "", false
}
if c, ok := w.carriedFor(d.Repository, d.MergedAs); ok {
return c, true
}
if d.Walk != nil && d.Walk.TakenOverBy == w.ID && namesNoMerges(w) && strings.EqualFold(w.Repository, d.Repository) &&
sameBranch(w.Branch, d.Base) {
// The walk that took its walk over, from a controller that names no merges: it answers every merge the one
// it took over answered, by its commit of the same repository and branch. A walk that names its merges
// answers only those (carriedFor, above).
return w.Commit, true
}
return "", false
}
// containedAnswer is, for a controller that does not name the merges a walk answers, whether an open walk
// answers a ready merged delivery by containment: of its repository and branch, made for a merge after it, the
// forge says its commit contains this merge, and it moves every module this merge's plan moves. A later walk that
// moves only what its own merge moved does not deliver this one, however much of the trunk it contains.
func (h *Holder) containedAnswer(w Walk, d *Delivery) (string, bool) {
if !h.mayContain(w, d) || !d.ContainedIn[w.Commit] {
return "", false
}
return w.Commit, true
}
// mayContain is whether an open walk is one to ask the forge about for a ready merged delivery (containedAnswer).
func (h *Holder) mayContain(w Walk, d *Delivery) bool {
if w.Release != nil || w.IsBatch() || !w.Open() || !namesNoMerges(w) || d.State != Ready || d.MergedAs == "" || d.Walk != nil ||
d.Plan == nil || len(d.Plan.Moved) == 0 || !strings.EqualFold(w.Repository, d.Repository) ||
!sameBranch(w.Branch, d.Base) || sameCommit(w.Commit, d.MergedAs) {
return false
}
for _, m := range d.Plan.Moved {
if _, moves := w.Modules[m]; !moves {
return false
}
}
// Made for a merge after this one: the walk's own merge, where this owner holds it, or else the walk itself.
after := w.Created
for _, o := range h.deliveries {
if strings.EqualFold(o.Repository, w.Repository) && sameCommit(o.MergedAs, w.Commit) && !o.MergedAt.IsZero() {
after = o.MergedAt
}
}
return after.After(d.MergedAt)
}
// namesNoMerges is whether a walk is from a controller that does not name the merges it answers.
func namesNoMerges(w Walk) bool { return w.Delivery == nil || len(w.Delivery.Merges) == 0 }
func sameBranch(a, b string) bool { return a == "" || b == "" || strings.EqualFold(a, b) }
// follows is whether a delivery takes a walk that answers it as the one it follows: it has none, it is that walk,
// that walk took its own over, or that walk is newer than the one it follows. A walk record kept before its
// creation was read is followed past only once it is superseded or failed.
func follows(d *Delivery, w Walk) bool {
switch {
case d.Walk == nil || d.Walk.ID == w.ID || d.Walk.TakenOverBy == w.ID:
return true
case d.Walk.Created.IsZero():
return d.Walk.State == walkSuperseded || d.Walk.State == walkFailed
}
return w.Created.After(d.Walk.Created)
}
// deliveriesWalking are the deliveries a walk record moves: every delivery that already follows it — ended or
// not, its own record — and every delivery not ended that it answers and that follows it now. In id order.
func (h *Holder) deliveriesWalking(w Walk) []answer {
var out []answer
for _, d := range h.deliveries {
if d.Walk != nil && d.Walk.ID == w.ID {
carried := d.AnsweredBy
if c, ok := walkAnswers(w, d); ok {
carried = c
}
out = append(out, answer{d: d, carried: carried})
continue
}
if d.State.Final() || !follows(d, w) {
continue
}
if c, ok := walkAnswers(w, d); ok {
out = append(out, answer{d: d, carried: c})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].d.ID < out[j].d.ID })
return out
}
// accounted is whether any delivery, ended or not, follows the walk or is answered by it: a walk known for
// what it answers is never made a delivery of its own.
func (h *Holder) accounted(w Walk) bool {
for _, d := range h.deliveries {
if d.Walk != nil && d.Walk.ID == w.ID {
return true
}
if _, ok := walkAnswers(w, d); ok {
return true
}
}
return false
}
// held is whether a live delivery follows the walk, or a live delivery it answers follows a newer one. A walk whose
// word only stops that lapsed give (lapsedOn, novox/hq issue 753) is not held: it stays under the orphans' timer, so
// past the grace without its word it is listed by `stalled`, saying why it was not given.
func (h *Holder) heldWalk(w Walk) bool {
for _, d := range h.deliveries {
if d.State.Final() {
continue
}
if d.Walk != nil && d.Walk.ID == w.ID {
return true
}
if _, ok := walkAnswers(w, d); ok {
return true
}
}
return false
}
// pickWalk is the newest walk heard that answers a merged delivery with no walk yet; nil for none.
func (h *Holder) pickWalk(d *Delivery) (Walk, string, bool) {
var best Walk
var carried string
found := false
for _, s := range h.walks {
if !follows(d, s.walk) {
continue
}
c, ok := walkAnswers(s.walk, d)
if !ok {
c, ok = h.containedAnswer(s.walk, d)
}
if !ok {
continue
}
if !found || s.walk.Created.After(best.Created) || s.walk.Created.Equal(best.Created) && s.walk.ID > best.ID {
best, carried, found = s.walk, c, true
}
}
return best, carried, found
}
// linkAnswering ties a merged delivery with no walk to the newest walk heard that answers it; under the lock.
func (h *Holder) linkAnswering(d *Delivery) bool {
if d.State.Final() || d.MergedAs == "" || d.Walk != nil {
return false
}
w, carried, ok := h.pickWalk(d)
if !ok {
return false
}
h.linkWalk(d, w, carried)
delete(h.unmatched, w.ID)
return true
}
// containsAsks are the forge's asks whether an open walk of a merged delivery's repository and branch, at
// another commit, contains its merge: for a controller that does not name the merges a walk answers (issue 362),
// asked once per pair, the answer kept on the delivery.
func (h *Holder) containsAsks(d *Delivery, now time.Time) []func() {
var asks []func()
for _, s := range h.walks {
w := s.walk
if !h.mayContain(w, d) {
continue
}
if _, known := d.ContainedIn[w.Commit]; known {
continue
}
if a := h.askContains(d, w.Commit, now); a != nil {
asks = append(asks, a)
}
}
return asks
}
// batchOf is the batch a merged delivery's merge is in, when the controller said one; false for none.
func (h *Holder) batchOf(d *Delivery) (Walk, bool) {
if d.MergedAs == "" {
return Walk{}, false
}
for _, s := range h.walks {
if !s.walk.IsBatch() || s.walk.Delivery == nil {
continue
}
for _, m := range s.walk.Delivery.Merges {
if strings.EqualFold(m.Repository, d.Repository) && sameCommit(m.Commit, d.MergedAs) {
return s.walk, true
}
}
}
return Walk{}, false
}
// batchSaid is what a batch holding a delivery's merge waits for, in words.
func batchSaid(b Walk, now time.Time) string {
said := "its batch " + b.ID + ": " + b.State
bt := b.Delivery.Batch
switch {
case b.State == walkQueued && bt != nil && bt.Behind != "":
said += " behind " + bt.Behind
case b.State == walkAssembling && bt != nil && bt.ClosesAt != nil:
left := bt.ClosesAt.Sub(now).Round(time.Second)
if left < 0 {
left = 0
}
said += fmt.Sprintf(", %s left", left)
if bt.AtMost != nil {
said += " (at the latest " + bt.AtMost.Local().Format("15:04:05") + ")"
}
}
return said + "; plan not yet calculated"
}
// unwalkedWhy is why a merged delivery no walk answers is held: the open walks of its repository and that none
// answers its merge, or that the controller holds none — what happened, not what might have.
func (h *Holder) unwalkedWhy(d *Delivery) string {
var open []string
for _, s := range h.walks {
w := s.walk
if w.Release == nil && !w.IsBatch() && w.Open() && strings.EqualFold(w.Repository, d.Repository) {
open = append(open, fmt.Sprintf("%s at %s", w.ID, shortOf(w.Commit, 8)))
}
}
sort.Strings(open)
head := fmt.Sprintf("merged as %s, and no walk answers its merge within %s: ", shortOf(d.MergedAs, 8), unwalked)
if len(open) == 0 {
return head + "the controller holds no open walk of " + d.Repository + " — its merge was not heard, or " +
"nothing the mesh holds follows " + orNone(d.Base) + ": a person decides"
}
return head + "the open walk(s) of " + d.Repository + " — " + strings.Join(open, ", ") + " — name no merge of " +
"it and the forge did not say they contain it: a person decides"
}
func orNone(s string) string {
if s == "" {
return "its branch"
}
return s
}
// orphanLines are the waiting walks no live delivery holds, past the grace a merge is given to be heard, as
// stalled lines in the state Unheld: nothing would ever give such a walk its word (issue 362). Under the lock.
func (h *Holder) orphanLines(now time.Time) []StalledLine {
var out []StalledLine
for id, first := range h.orphans {
s, known := h.walks[id]
if !known || now.Sub(first) < matchGrace {
continue
}
w := s.walk
if len(h.lapsedOn(w)) > 0 {
// Owed its word by stops that lapsed, and not given it past the grace (review of issue 753): said, why.
says := fmt.Sprintf("the walk %s of %s at %s waits for this owner's word, which the stopped deliveries whose "+
"merge it answers owe it (novox/hq issue 753), and it has not started", w.ID, w.Repository, shortOf(w.Commit, 8))
if why := h.lapsedSaid(w); why != "" {
says += ": " + why
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: w.ID,
For: now.Sub(first).Round(time.Second).String(), Bound: matchGrace.String(),
H2: "none: its word is given at its turn — the operator's past the bound", Says: says})
continue
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Unheld, Walk: w.ID,
For: now.Sub(first).Round(time.Second).String(), Bound: matchGrace.String(),
H2: "none: no delivery holds the walk, so nothing gives it its word — the operator's",
Says: fmt.Sprintf("the walk %s of %s at %s waits for this owner's word, and no delivery that has not ended "+
"follows it or is answered by it: `mesh-delivery.show` for the deliveries of %s, and `plans go %s --why …` "+
"starts it by hand", w.ID, w.Repository, shortOf(w.Commit, 8), w.Repository, w.ID)})
out[len(out)-1].Says += h.endedSaid(w)
if hold := h.coverHeld(id); hold != "" {
out[len(out)-1].Says += "; it is held back meanwhile: " + hold
}
}
return out
}
// Unheld is a waiting walk no live delivery holds: not a state of the table — only what a stalled line says.
const Unheld State = "unheld"
// seeOrphans keeps when each waiting walk no live delivery holds was first seen so, and forgets the rest; a
// walk still unmatched waits for its merge or its adoption instead. Under the lock.
func (h *Holder) seeOrphans(now time.Time) {
for id, s := range h.walks {
w := s.walk
_, unmatched := h.unmatched[id]
if unmatched || w.Release != nil || w.IsBatch() || !w.Open() || !w.Seen().Waits || h.heldWalk(w) {
delete(h.orphans, id)
continue
}
if _, was := h.orphans[id]; !was {
h.orphans[id] = now
h.Logf("[mesh-delivery] the walk %s waits for this owner's word and no live delivery holds it: said once "+
"it is %s so", id, matchGrace)
}
}
for id := range h.orphans {
if _, known := h.walks[id]; !known {
delete(h.orphans, id)
}
}
}
// forgetWalks drops a walk record no longer open after keptWalkFor, and a batch heard of that long past its
// window's latest close. Under the lock.
func (h *Holder) forgetWalks(now time.Time) {
for id, s := range h.walks {
w := s.walk
switch {
case w.IsBatch():
end := s.at
if w.Delivery != nil && w.Delivery.Batch != nil && w.Delivery.Batch.AtMost != nil && w.Delivery.Batch.AtMost.After(end) {
end = *w.Delivery.Batch.AtMost
}
if now.Sub(end) > 3*time.Hour {
delete(h.walks, id)
}
case !w.Open() && now.Sub(s.at) > keptWalkFor:
delete(h.walks, id)
delete(h.deliverErr, id)
}
}
}
// --- the word, once per walk ------------------------------------------------------------------------------
// turnWant is a walk whose turn came for some of the deliveries it carries, and why.
type turnWant struct {
ids []string
whys []string
}
// endedSaid names the ended deliveries a waiting walk answers or that follow it, with their states, so a walk no live
// delivery holds says what it carries instead of only that nothing holds it (novox/hq issue 753); empty for none.
func (h *Holder) endedSaid(w Walk) string {
var ended []string
for _, d := range h.deliveries {
if !d.State.Final() {
continue
}
_, answers := walkAnswers(w, d)
if answers || d.Walk != nil && d.Walk.ID == w.ID {
ended = append(ended, fmt.Sprintf("%s (%s)", d.ID, d.State))
}
}
if len(ended) == 0 {
return ""
}
sort.Strings(ended)
return "; every delivery it carries ended: " + strings.Join(ended, ", ")
}
// lapsedOn are the deliveries, by id, whose word an open walk waiting for this owner's word takes from a stop that
// lapsed (novox/hq issue 753): each was stopped by its group for a member delivered since, so its own change waits for
// a later walk of its repository that contains its merge (issue 747), and this walk answers that merge — it names it,
// or is made at it — and has not left it out. Such a walk is the one that delivery waits for: it gives it its word, so
// a walk whose every other delivery ended is never left without one. A walk at the merge of a delivery stopped for
// itself, or by a person, gets nothing from it. Under the lock.
//
// No member is ordered before another (novox/hq ADR 0418 §2): each such delivery gives its word at once.
func (h *Holder) lapsedOn(w Walk) []string {
if w.Release != nil || w.IsBatch() || !w.Open() || !w.Seen().Waits {
return nil
}
var out []string
for _, d := range h.deliveries {
if h.lapsedAnswered(w, d) {
out = append(out, d.ID)
}
}
sort.Strings(out)
return out
}
// lapsedAnswered is whether a delivery's stop lapsed and the walk answers its merge, not left out.
func (h *Holder) lapsedAnswered(w Walk, d *Delivery) bool {
if d.MergedAs == "" || !h.stopLapsed(d) {
return false
}
if _, ok := walkAnswers(w, d); !ok {
return false
}
_, left := w.leftOut(d.Repository, d.MergedAs)
return !left
}
// lapsedSaid is why a walk whose word stops that lapsed give still waits: the members of their groups not yet
// delivered before them, what the last look held it for, or the controller's refusal of the word. Under the lock.
func (h *Holder) lapsedSaid(w Walk) string {
var parts []string
if why := h.waitsWhy[w.ID]; why != "" {
parts = append(parts, "not let go yet: "+why)
}
if err := h.deliverErr[w.ID]; err != "" {
parts = append(parts, "the controller refused its word: "+err)
}
for _, id := range h.lapsedOn(w) {
if d := h.deliveries[id]; d != nil && strings.Contains(d.LetGoWhy, lapsedWord(w.ID)) {
parts = append(parts, fmt.Sprintf("its word was given for %s at %s, and the controller has not started it",
id, d.LetGo.Local().Format("15:04:05")))
}
}
return strings.Join(parts, "; ")
}
// lapsedOwed is whether a walk's word is owed by stops that lapsed.
func (h *Holder) lapsedOwed(w Walk) bool {
return len(h.lapsedOn(w)) > 0
}
// lapsedTurns registers, for every open walk waiting for this owner's word, the turn of each delivery whose stop
// lapsed and whose merge it answers (lapsedOn, novox/hq issue 753). The word is then given as for any walk: once,
// when every live delivery it carries is at its turn too and it carries no stopped change still held back (words).
// Given once per walk: the walk's id is in the delivery's LetGoWhy once its word was given. Under the lock.
func (h *Holder) lapsedTurns() {
for id, s := range h.walks {
for _, did := range h.lapsedOn(s.walk) {
d := h.deliveries[did]
if strings.Contains(d.LetGoWhy, lapsedWord(id)) {
continue
}
if h.turning == nil {
h.turning = map[string]*turnWant{}
}
want := h.turning[id]
if want == nil {
want = &turnWant{}
h.turning[id] = want
}
if !containsString(want.ids, d.ID) {
want.ids = append(want.ids, d.ID)
want.whys = append(want.whys, fmt.Sprintf("%s: %s its merge %s, stopped by its group for %s, whose change "+
"was delivered since; it waits for a later walk of %s that contains its merge, and this is that "+
"walk (novox/hq issue 753)", d.ID, lapsedWord(id), shortOf(d.MergedAs, 8), d.stoppedFor(), d.Repository))
}
}
}
}
// lapsedWord is how the word a lapsed stop gives a walk names that walk.
func lapsedWord(walk string) string { return "the walk " + walk + " answers" }
// words are the asks of the word of every walk whose turn came for every live delivery it carries: one ask per
// walk, however many deliveries it carries, and none while one of them is not at its turn (ADR 0276 decision 5).
// Under the lock.
func (h *Holder) words() []func() {
walks := make([]string, 0, len(h.turning))
for walk := range h.turning {
walks = append(walks, walk)
}
sort.Strings(walks)
var asks []func()
h.waitsWhy = map[string]string{}
for _, walk := range walks {
want := h.turning[walk]
var waiting []string
for _, d := range h.deliveries {
if d.State.Final() || d.Walk == nil || d.Walk.ID != walk || containsString(want.ids, d.ID) {
continue
}
waiting = append(waiting, fmt.Sprintf("%s (%s)", d.ID, d.State))
}
if len(waiting) > 0 {
sort.Strings(waiting)
h.waitsWhy[walk] = strings.Join(waiting, ", ") + " carried by it not at its turn"
h.Logf("[mesh-delivery] the walk %s is not let go yet: %s", walk, h.waitsWhy[walk])
continue
}
ids, why := append([]string(nil), want.ids...), strings.Join(want.whys, "; ")
// **Never the word for a walk that carries a stopped change** (ADR 0299, review of #221): it is asked to
// leave the merge out first (leaveStopped).
if stopped, _ := h.carriesStopped(walk); len(stopped) > 0 {
h.waitsWhy[walk] = "it carries " + strings.Join(stopped, ", ") + ", whose delivery was stopped: it is " +
"asked to leave that merge out first"
if hold := h.coverHeld(walk); hold != "" {
h.waitsWhy[walk] = hold
}
h.Logf("[mesh-delivery] the walk %s is not let go: it carries %s, whose delivery was stopped", walk,
strings.Join(stopped, ", "))
continue
}
if h.asking[walk] {
continue // its word is being asked
}
h.asking[walk] = true
walk := walk
asks = append(asks, func() {
err := h.Controller.Deliver(walk, why)
h.mu.Lock()
defer h.mu.Unlock()
delete(h.asking, walk)
if err != nil {
if h.deliverErr == nil {
h.deliverErr = map[string]string{}
}
h.deliverErr[walk] = err.Error() // said by `stalled` past the grace (review of novox/hq issue 753)
h.Logf("[mesh-delivery] the walk %s could not be let go yet: %v", walk, err)
h.again("a walk's word the controller refused or did not answer")
return
}
delete(h.deliverErr, walk)
for _, id := range ids {
d := h.deliveries[id]
if d == nil {
continue
}
d.LetGo, d.LetGoWhy = h.Now(), why
if d.Walk != nil && d.Walk.ID == walk {
d.Walk.Waits, d.Walk.GoBy = false, byOwner
}
h.settleAndKeep(d)
}
})
}
h.turning = nil
return asks
}
// endedMovingNothing is whether a delivery ended delivered with nothing for a walk to move: merged, its plan
// moving no module the mesh holds, and no walk followed.
func endedMovingNothing(d *Delivery) bool {
return d.State == Delivered && d.MergedAs != "" && d.Walk == nil && d.MovesNothing()
}
// endedAsNothingToMove are the deliveries a walk answers that ended delivered as nothing for a walk to move (novox/hq
// issues 362, 529: the nzbgeek case); empty when it answers any other. Such a walk is held by nothing: said, never
// let go by this owner. Under the lock.
func (h *Holder) endedAsNothingToMove(w Walk) []string {
var out []string
for _, d := range h.deliveries {
_, answers := walkAnswers(w, d)
if !answers && (d.Walk == nil || d.Walk.ID != w.ID) {
continue
}
if !endedMovingNothing(d) {
return nil
}
out = append(out, d.ID)
}
sort.Strings(out)
return out
}
// WalksWaiting is `stalled` asked with `walks` (novox/hq issue 529): for each walk waiting for this owner's word,
// what holds it, in the shape of a stalled line naming the walk — each held delivery it carries (State held, its
// reason and the verb that releases it), else why its word was not given at the last look (State published), else
// that no live delivery holds it (State unheld). A walk whose word is being given is not listed. The controller's
// `plans` says these instead of pointing at a hand start; the self-check never raises a condition for them.
func (h *Holder) WalksWaiting() []StalledLine {
h.mu.Lock()
defer h.mu.Unlock()
now := h.Now()
var out []StalledLine
for id, s := range h.walks {
w := s.walk
if w.Release != nil || w.IsBatch() || !w.Open() || !w.Seen().Waits || h.asking[id] {
continue
}
waited := now.Sub(w.Created).Round(time.Second).String()
var held []StalledLine
live := false
for _, d := range h.deliveries {
if d.State.Final() || d.Walk == nil || d.Walk.ID != id {
continue
}
live = true
if d.State == Held {
held = append(held, StalledLine{ID: d.ID, Number: d.Number, State: Held, Walk: id, For: waited,
Bound: "0s", H2: "none: held for a person — the operator's", Says: HeldLine(d)})
}
}
switch {
case len(held) > 0:
out = append(out, held...)
case h.coverHeld(id) != "":
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: id, For: waited,
Bound: coverAskBound.String(), H2: "none: it is asked to leave the merge out past the bound",
Says: "held back: " + h.coverHeld(id)})
case h.waitsWhy[id] != "":
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: id, For: waited,
Bound: "0s", H2: "none: it goes at its turn", Says: "not let go yet: " + h.waitsWhy[id]})
case !live && h.lapsedOwed(w):
// Its word comes from the stopped deliveries whose merge it answers (novox/hq issue 753): said, never
// unheld, while the controller has not taken it, with what holds it.
lapsed := h.lapsedOn(w)
says := "its word is owed by the stopped deliveries whose merge it answers, each stopped for a member " +
"delivered since (novox/hq issue 753)"
if len(lapsed) > 0 {
says = "its word is given for " + strings.Join(lapsed, ", ") + ", stopped for a member delivered since, " +
"whose merge it answers (novox/hq issue 753)"
}
if why := h.lapsedSaid(w); why != "" {
says += "; " + why
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: id, For: waited,
Bound: "0s", H2: "none: it goes at its turn", Says: says})
case !live:
says := "no delivery that has not ended follows it or is answered by it, so nothing gives it its word: " +
"`mesh-delivery.show` for the deliveries of " + w.Repository
if ended := h.endedAsNothingToMove(w); len(ended) > 0 {
// The nzbgeek case (novox/hq issues 362, 529): its merges' deliveries ended as nothing for a walk to
// move, yet the walk moves modules — nothing judged those, so nothing here lets it go.
says = "held by nothing: its merges' deliveries ended as nothing-to-move (" + strings.Join(ended, ", ") +
"), yet the walk moves " + strings.Join(walkModules(w), ", ") + " — nothing gives it its word"
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Unheld, Walk: id, For: waited,
Bound: "0s", H2: "none: no delivery holds the walk — the operator's", Says: says + h.endedSaid(w)})
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Walk != out[j].Walk {
return out[i].Walk < out[j].Walk
}
return out[i].ID < out[j].ID
})
return out
}
// walkModules are the modules a walk's record names, sorted.
func walkModules(w Walk) []string {
out := make([]string, 0, len(w.Modules))
for m := range w.Modules {
out = append(out, m)
}
sort.Strings(out)
return out
}
@@ -140,6 +140,14 @@ func repoName(repository string) string {
func pullSaid(d *Delivery) string { return fmt.Sprintf("%s#%d", repoName(d.Repository), d.Number) } func pullSaid(d *Delivery) string { return fmt.Sprintf("%s#%d", repoName(d.Repository), d.Number) }
// touchesSaid is the modules the delivery's check says its change touches, for the ask; the gate plans no delivery.
func touchesSaid(d *Delivery) string {
if d.Check == nil || len(d.Check.Modules) == 0 {
return "no module the check named"
}
return strings.Join(d.Check.Modules, ", ")
}
// composeMergeAsk is the ask for a delivery, or for a group's members together. // composeMergeAsk is the ask for a delivery, or for a group's members together.
func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery, heads map[string]string, now time.Time) asks.Ask { func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery, heads map[string]string, now time.Time) asks.Ask {
var a asks.Ask var a asks.Ask
@@ -155,13 +163,10 @@ func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery,
unreviewed = pullSaid(m) + " is " + words unreviewed = pullSaid(m) + " is " + words
} }
reviewers = append(reviewers, fmt.Sprintf("%s: %s", pullSaid(m), words)) reviewers = append(reviewers, fmt.Sprintf("%s: %s", pullSaid(m), words))
plan := "no delivery plan yet" touches := touchesSaid(m)
if m.Plan != nil && m.Plan.Summary != "" { reaches = append(reaches, touches)
plan = m.Plan.Summary details = append(details, fmt.Sprintf("%s %q at %s: issues: %s; review: %s; touches: %s",
} pullSaid(m), m.Title, shortOf(m.Commit, 12), issuesSaid([]*Delivery{m}), words, touches))
reaches = append(reaches, plan)
details = append(details, fmt.Sprintf("%s %q at %s: issues: %s; review: %s; reaches: %s",
pullSaid(m), m.Title, shortOf(m.Commit, 12), issuesSaid([]*Delivery{m}), words, plan))
} }
if group == "" { if group == "" {
a.Headline = phone("Merge into main: "+pullSaid(d)+" "+d.Title, asks.HeadlineLength) a.Headline = phone("Merge into main: "+pullSaid(d)+" "+d.Title, asks.HeadlineLength)
@@ -174,7 +179,7 @@ func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery,
phone(summary, asks.SummaryLength), phone(summary, asks.SummaryLength),
phone("Issues: "+issuesSaid(members), asks.PointLength), phone("Issues: "+issuesSaid(members), asks.PointLength),
phone("Review: "+reviewers[0], asks.PointLength), phone("Review: "+reviewers[0], asks.PointLength),
phone("Reaches: "+reaches[0], asks.PointLength), phone("Touches: "+reaches[0], asks.PointLength),
}, "\n") }, "\n")
} else { } else {
var names []string var names []string
@@ -191,7 +196,7 @@ func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery,
phone(summary, asks.SummaryLength), phone(summary, asks.SummaryLength),
phone("Issues: "+issuesSaid(members), asks.PointLength), phone("Issues: "+issuesSaid(members), asks.PointLength),
phone("Review: "+strings.Join(reviewers, "; "), asks.PointLength), phone("Review: "+strings.Join(reviewers, "; "), asks.PointLength),
phone("Reaches: "+strings.Join(reaches, "; "), asks.PointLength), phone("Touches: "+strings.Join(reaches, "; "), asks.PointLength),
}, "\n") }, "\n")
} }
a.Details = phone(strings.Join(details, "\n"), 1500) a.Details = phone(strings.Join(details, "\n"), 1500)
@@ -43,7 +43,7 @@ func TestAMarkedGreenDeliveryIntoMainIsAskedAndNotMergedUntilApproved(t *testing
len(a.Options) != 2 || a.Options[0].ID != OptionApprove || a.Options[1].ID != OptionDecline { len(a.Options) != 2 || a.Options[0].ID != OptionApprove || a.Options[1].ID != OptionDecline {
t.Fatalf("the ask is not the merge ask: %+v", a) t.Fatalf("the ask is not the merge ask: %+v", a)
} }
for _, want := range []string{"issue 948", "issue 12", "reviewed by reviewer on shanks", "builds app"} { for _, want := range []string{"issue 948", "issue 12", "reviewed by reviewer on shanks", "Touches: "} {
if !strings.Contains(a.Explanation, want) { if !strings.Contains(a.Explanation, want) {
t.Fatalf("the ask does not say %q: %s", want, a.Explanation) t.Fatalf("the ask does not say %q: %s", want, a.Explanation)
} }
@@ -72,7 +72,6 @@ func TestAMarkedGreenDeliveryIntoMainIsAskedAndNotMergedUntilApproved(t *testing
t.Fatalf("the warrant is not kept: %+v", d.Ready) t.Fatalf("the warrant is not kept: %+v", d.Ready)
} }
w.h.PullMerged(merged("novox/app", 7, readyHead, "m7m7m7m7m7m7")) w.h.PullMerged(merged("novox/app", 7, readyHead, "m7m7m7m7m7m7"))
w.h.WalkMoved(w.ctl.put(aWalk("plan-7", "novox/app", "m7m7m7m7m7m7", true)))
w.settleAll() w.settleAll()
var last Transition var last Transition
for _, tr := range w.delivery(id).Transitions { for _, tr := range w.delivery(id).Transitions {
@@ -373,7 +372,7 @@ func TestIssue1025AnOwedMergedIsSaidAfterARestartAndOnce(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
w.h.mu.Unlock() w.h.mu.Unlock()
again := newWorldOver(t, w.store, w.ctl, w.forge, w.now.Add(time.Minute)) again := newWorldOver(t, w.store, w.forge, w.now.Add(time.Minute))
again.settleAll() again.settleAll()
again.later(time.Hour) again.later(time.Hour)
again.settleAll() again.settleAll()
@@ -1,154 +0,0 @@
package main
import (
"strings"
"testing"
)
// A failed backup before the send holds the delivery, and a release asks the backups again (novox/hq ADR 0404
// clause 4, issue 1068). The walk's backups field is the controller's contract.
func failedBackup() *WalkBackups {
return &WalkBackups{State: "failed", Asked: []WalkBackupAsked{
{Machine: "ace", Module: "mail", State: "done"},
{Machine: "ace", Module: "pg", State: "failed", Words: "no space left on device"}}}
}
// aDeliveringWalk is a delivery whose walk was let go and runs, nothing sent yet.
func aDeliveringWalk(t *testing.T) (*world, string, string) {
t.Helper()
w := newWorld(t)
id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
w.h.WalkMoved(w.ctl.put(aWalk("plan-1", "novox/app", merge, true)))
w.settleAll()
if s := w.state(id); s != Delivering {
t.Fatalf("setup: %s, want delivering", s)
}
return w, id, "plan-1"
}
func TestABackupFailedTakesADeliveringDeliveryToHeldOnlyWhileNothingWasSent(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
sent := w.ctl.walk(plan)
sent.Backups = failedBackup()
sent.Modules["app"].SentAt = &w.now
w.h.WalkMoved(w.ctl.put(sent))
w.settleAll()
if s := w.state(id); s == Held {
t.Fatal("held for a failed backup after a module of the plan was sent")
}
w, id, plan = aDeliveringWalk(t)
held := w.ctl.walk(plan)
held.Backups = failedBackup()
w.h.WalkMoved(w.ctl.put(held))
w.settleAll()
d := w.delivery(id)
if d.State != Held {
t.Fatalf("with a failed backup and nothing sent it is %s", d.State)
}
for _, want := range []string{"ace", "pg", "no space left on device"} {
if !strings.Contains(d.HeldWhy, want) {
t.Errorf("the held reason %q does not name %q", d.HeldWhy, want)
}
}
// It does not lift by itself: the same facts read again leave it held.
w.h.WalkMoved(w.ctl.put(w.ctl.walk(plan)))
w.settleAll()
if s := w.state(id); s != Held {
t.Fatalf("read again it is %s, want it still held", s)
}
}
func TestAReleaseOfADeliveryHeldForAFailedBackupAsksTheControllerAgainAndGoesToDelivering(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
held := w.ctl.walk(plan)
held.Backups = failedBackup()
w.h.WalkMoved(w.ctl.put(held))
w.settleAll()
if s := w.state(id); s != Held {
t.Fatalf("setup: %s, want held", s)
}
lettingGo := len(w.ctl.delivers)
if _, err := w.h.Release(id, "freed some space", "jochen"); err != nil {
t.Fatal(err)
}
if got := w.ctl.backupsAgain; len(got) != 1 || got[0] != plan {
t.Fatalf("the controller was asked to take the backups again %v, want once for %s", got, plan)
}
if len(w.ctl.delivers) != lettingGo {
t.Errorf("a started walk was let go again: %v", w.ctl.delivers)
}
if s := w.state(id); s != Delivering {
t.Fatalf("released, it is %s", s)
}
// The record the controller cleared is read: it stays delivering.
w.h.WalkMoved(w.ctl.walk(plan))
w.settleAll()
if s := w.state(id); s != Delivering {
t.Fatalf("with the backups cleared it is %s", s)
}
// Another failure, in a later record, holds it again.
again := w.ctl.walk(plan)
again.Backups = failedBackup()
w.h.WalkMoved(w.ctl.put(again))
w.settleAll()
if s := w.state(id); s != Held {
t.Fatalf("a failed backup in a later record left it %s", s)
}
}
func TestARecordThatStillHoldsTheFailureAfterAReleaseDoesNotHoldItAgain(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
held := w.ctl.walk(plan)
held.Backups = failedBackup()
stale := w.ctl.put(held)
w.h.WalkMoved(stale)
w.settleAll()
if _, err := w.h.Release(id, "freed some space", "jochen"); err != nil {
t.Fatal(err)
}
w.h.WalkMoved(stale) // the record as it was before the controller cleared it
w.settleAll()
if s := w.state(id); s != Delivering {
t.Fatalf("a record from before the release held it again: %s", s)
}
}
func TestADeliveryHeldForAFailedBackupIsNotLetGoByItsWalkAlone(t *testing.T) {
// The controller's own path starts a walk with nobody's word: the held-to-delivering row for that must not
// undo the backup hold.
d := &Delivery{ID: "novox/app@c0", State: Held, MergedAs: "aaaa", Walk: &WalkSeen{ID: "plan-1", State: walkBuilding, Revision: 3,
BackupsFailed: []WalkBackupFailure{{Machine: "ace", Module: "pg", Words: "no space left on device"}}}}
if _, err := Apply(d, EvGo, false, Facts{}); err == nil {
t.Fatal("a walk started on the controller's own path lifted the backup hold")
}
}
func TestAReleaseWhoseBackupsTheControllerRefusedToAskAgainLeavesTheDeliveryHeld(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
held := w.ctl.walk(plan)
held.Backups = failedBackup()
w.h.WalkMoved(w.ctl.put(held))
w.settleAll()
w.ctl.refuseBackupsAgain = true
_, err := w.h.Release(id, "freed some space", "jochen")
if err == nil || !strings.Contains(err.Error(), "held again") {
t.Fatalf("the release of a refused word answered %v, want an error saying it is held again", err)
}
w.settleAll()
d := w.delivery(id)
if d.State != Held || d.BackupsAgainAfter != 0 || !strings.Contains(d.HeldWhy, "no space left on device") {
t.Fatalf("after the refusal it is %s (again-after %d, why %q), want held for the backup", d.State, d.BackupsAgainAfter, d.HeldWhy)
}
// The controller takes the word the next time: the delivery goes on.
w.ctl.refuseBackupsAgain = false
if _, err := w.h.Release(id, "freed some space", "jochen"); err != nil {
t.Fatal(err)
}
if s := w.state(id); s != Delivering {
t.Fatalf("released again, it is %s", s)
}
}
@@ -11,8 +11,6 @@ import (
"io" "io"
"os" "os"
"os/exec" "os/exec"
"path/filepath"
"slices"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -85,8 +83,8 @@ func newTestBus(t *testing.T) *testBus {
} }
b.kvs[name] = kv b.kvs[name] = kv
} }
// The buckets of other modules the manifest reads (`<module>.<name>` is the bucket `<module>_<name>`): the gate reads // The buckets of other modules the manifest reads (`<module>.<name>` is the bucket `<module>_<name>`): the teller's answers
// who has the seat there, and today's state to carry (seat.go). // and the checker's verdicts.
for _, name := range m.Reads { for _, name := range m.Reads {
kv, err := js.CreateKeyValue(context.Background(), jetstream.KeyValueConfig{Bucket: strings.Replace(name, ".", "_", 1), History: 10}) kv, err := js.CreateKeyValue(context.Background(), jetstream.KeyValueConfig{Bucket: strings.Replace(name, ".", "_", 1), History: 10})
if err != nil { if err != nil {
@@ -94,13 +92,9 @@ func newTestBus(t *testing.T) *testBus {
} }
b.kvs[name] = kv b.kvs[name] = kv
} }
// The node-engine has said today's module removed on the one machine: the gate may hold the seat (ADR 0450 §6, issue 1369).
if _, err := b.kvs["node-engine.have"].Put(context.Background(), "test.mesh-delivery", []byte(`{"v":1,"phase":"removed","health":"unknown"}`)); err != nil {
t.Fatal(err)
}
// The streams the events are on (mesh.mod.<module>.event.<name>, mesh.seat.<seat>.event.<name>). // The streams the events are on (mesh.mod.<module>.event.<name>, mesh.seat.<seat>.event.<name>).
if _, err := js.CreateStream(context.Background(), jetstream.StreamConfig{Name: "EVENTS", if _, err := js.CreateStream(context.Background(), jetstream.StreamConfig{Name: "EVENTS",
Subjects: []string{"mesh.mod.gitea.event.>", "mesh.seat." + ControllerSeat + ".event.>", "mesh.seat." + asks.Seat + ".event.>", Subjects: []string{"mesh.mod.gitea.event.>", "mesh.seat." + asks.Seat + ".event.>",
"mesh.mod." + sdk.ForgeBridgeModule + ".event.>"}}); err != nil { "mesh.mod." + sdk.ForgeBridgeModule + ".event.>"}}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -208,15 +202,29 @@ func (r *runtime) read(out io.Reader) {
} }
} }
// untilHolding waits for the gate to hold the seat: it holds it when it reads the controller's walks once, the first thing it // untilListening waits for the gate to have read its kept state back and subscribed to every event it follows: it takes no
// does after (readOnHolding), and it takes no event before. // event before (loadKept), so a subscription is its word that it is ready.
func (r *runtime) untilHolding() { func (r *runtime) untilListening() {
r.t.Helper() r.t.Helper()
r.until("the gate to hold the seat", 20*time.Second, func() bool { r.until("the gate to listen to every event it follows", 20*time.Second, func() bool {
return len(r.asksOf("seat:"+ControllerSeat+".delivery-walks")) > 0 r.mu.Lock()
defer r.mu.Unlock()
return len(r.subs) >= len(followed)
}) })
} }
// noControllerAsk fails the test when the gate asked the controller anything: it asks it nothing (ADR 0539 §17, R6-C3).
func (r *runtime) noControllerAsk() {
r.t.Helper()
r.mu.Lock()
defer r.mu.Unlock()
for _, a := range r.asked {
if strings.HasPrefix(a.Key, "seat:mesh-controller.") {
r.t.Fatalf("the gate asked the controller %s: it asks it nothing", a.Key)
}
}
}
func (r *runtime) ping() { func (r *runtime) ping() {
select { select {
case r.notify <- struct{}{}: case r.notify <- struct{}{}:
@@ -334,11 +342,8 @@ func (r *runtime) answer(m rpcMessage) {
pulls := append([]OpenPull{}, r.openPulls...) pulls := append([]OpenPull{}, r.openPulls...)
r.mu.Unlock() r.mu.Unlock()
switch a.Key { switch a.Key {
case "seat:" + ControllerSeat + ".delivery-walks": case "":
return toolAnswer(map[string]any{"held": true, "walks": []any{}}), nil return nil, errors.New("an ask with no key")
case "seat:" + ControllerSeat + ".delivery-plan":
return toolAnswer(map[string]any{"plan": DeliveryPlan{Repository: "novox/app", Moved: []string{"app"},
Summary: "builds app, sent to the node"}}), nil
case "gitea.gitea_open_pulls": case "gitea.gitea_open_pulls":
return toolAnswer(map[string]any{"pulls": pulls}), nil return toolAnswer(map[string]any{"pulls": pulls}), nil
case "gitea.gitea_commit_statuses": case "gitea.gitea_commit_statuses":
@@ -355,6 +360,10 @@ func (r *runtime) answer(m rpcMessage) {
"gitea.gitea_add_comment": "gitea.gitea_add_comment":
return toolAnswer(map[string]any{}), nil return toolAnswer(map[string]any{}), nil
} }
if strings.HasPrefix(a.Key, "seat:mesh-controller.") {
// The fake controller of this test fails on any call: the gate has no business with it.
return nil, fmt.Errorf("the controller is not to be called by the gate: %s", a.Key)
}
return nil, fmt.Errorf("the runtime answers no ask %s in this test", a.Key) return nil, fmt.Errorf("the runtime answers no ask %s in this test", a.Key)
} }
kv := r.b.kvs[p.State] kv := r.b.kvs[p.State]
@@ -495,12 +504,7 @@ func TestAPassingVerdictAsksTheApproveAndItsApproveMergesWithinSecondsOverARealB
defer cancel() defer cancel()
rt.deliverEvents(evctx) rt.deliverEvents(evctx)
// The module has started when it hears what it follows. // The module has started when it hears what it follows.
rt.until("the module to subscribe to the events it follows", 20*time.Second, func() bool { rt.untilListening()
rt.mu.Lock()
defer rt.mu.Unlock()
return len(rt.subs) >= len(followed)
})
rt.untilHolding()
for _, k := range followed { for _, k := range followed {
if strings.Contains(k, "labeled") || strings.Contains(k, "commented") { if strings.Contains(k, "labeled") || strings.Contains(k, "commented") {
t.Fatalf("the module follows %s: a label or a comment starts nothing here (ADR 0421 §4)", k) t.Fatalf("the module follows %s: a label or a comment starts nothing here (ADR 0421 §4)", k)
@@ -580,6 +584,8 @@ func TestAPassingVerdictAsksTheApproveAndItsApproveMergesWithinSecondsOverARealB
// The bridge says it merged; the mesh's source then holds the merge, in one write. // The bridge says it merged; the mesh's source then holds the merge, in one write.
b.forgeSays(t, sdk.Forge{Kind: sdk.ForgeMerged, Repository: "novox/app", Number: 7, Head: head, Merge: strings.Repeat("b", 40), b.forgeSays(t, sdk.Forge{Kind: sdk.ForgeMerged, Repository: "novox/app", Number: 7, Head: head, Merge: strings.Repeat("b", 40),
Root: "r", Trees: map[string]string{"app": "t"}}) Root: "r", Trees: map[string]string{"app": "t"}})
// The forge's own word that the pull request merged follows the bridge's (its webhook): the delivery ends merged.
b.publish(t, "mesh.mod.gitea.event.pull.merged", merged("novox/app", 7, head, strings.Repeat("b", 40)))
rt.until("source.main to hold the merge", eventWithin, func() bool { rt.until("source.main to hold the merge", eventWithin, func() bool {
e, err := b.kvs["source"].Get(ctx, sdk.SourceMainKey) e, err := b.kvs["source"].Get(ctx, sdk.SourceMainKey)
if err != nil { if err != nil {
@@ -588,6 +594,13 @@ func TestAPassingVerdictAsksTheApproveAndItsApproveMergesWithinSecondsOverARealB
src, err := sdk.DecodeSource(e.Value()) src, err := sdk.DecodeSource(e.Value())
return err == nil && src.Group == "fix/x" && src.Heads["novox/app"] == strings.Repeat("b", 40) return err == nil && src.Group == "fix/x" && src.Heads["novox/app"] == strings.Repeat("b", 40)
}) })
// The delivery ends merged (ADR 0539 §17), and the gate asked the controller nothing (R6-C3).
rt.until("the delivery to be merged", eventWithin, func() bool {
e, err := b.kvs["deliveries"].Get(ctx, kvKey(IDOf("novox/app", head)))
var d Delivery
return err == nil && json.Unmarshal(e.Value(), &d) == nil && d.State == Merged
})
rt.noControllerAsk()
// Quiet follows: nothing asks the forge or the controller again with no event to say so. // Quiet follows: nothing asks the forge or the controller again with no event to say so.
quiet := len(rt.asked) quiet := len(rt.asked)
time.Sleep(1500 * time.Millisecond) time.Sleep(1500 * time.Millisecond)
@@ -690,12 +703,7 @@ func TestAGroupOfTwoWithGreenChecksGetsOneAskAndOnItsApproveBothMergeOneAfterThe
evctx, cancel := context.WithCancel(ctx) evctx, cancel := context.WithCancel(ctx)
defer cancel() defer cancel()
rt.deliverEvents(evctx) rt.deliverEvents(evctx)
rt.until("the module to subscribe to the events it follows", 20*time.Second, func() bool { rt.untilListening()
rt.mu.Lock()
defer rt.mu.Unlock()
return len(rt.subs) >= len(followed)
})
rt.untilHolding()
const ctlHead, catHead = "c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1", "d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2" const ctlHead, catHead = "c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1c1", "d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2"
for _, p := range []PullEvent{pr("novox/mesh-controller", 1, ctlHead, "feat/x", "internal/a.go"), for _, p := range []PullEvent{pr("novox/mesh-controller", 1, ctlHead, "feat/x", "internal/a.go"),
pr("novox/mesh-catalog", 2, catHead, "feat/x", "modules/app/module.json")} { pr("novox/mesh-catalog", 2, catHead, "feat/x", "modules/app/module.json")} {
@@ -706,9 +714,6 @@ func TestAGroupOfTwoWithGreenChecksGetsOneAskAndOnItsApproveBothMergeOneAfterThe
// Both members pass at the checker, whose one verdict covers both heads: the group is asked once, no composed check is asked // Both members pass at the checker, whose one verdict covers both heads: the group is asked once, no composed check is asked
// of the controller (issue 1318). // of the controller (issue 1318).
b.checkPasses(t, rt, "feat/x", "a1a1a1a1a1a1", judgedPull{"novox/mesh-controller", 1, ctlHead}, judgedPull{"novox/mesh-catalog", 2, catHead}) b.checkPasses(t, rt, "feat/x", "a1a1a1a1a1a1", judgedPull{"novox/mesh-controller", 1, ctlHead}, judgedPull{"novox/mesh-catalog", 2, catHead})
if n := len(rt.asksOf("seat:" + ControllerSeat + ".delivery-order")); n != 0 {
t.Fatalf("the controller's delivery-order was asked %d times: no order is asked for", n)
}
var key string var key string
rt.until("the operator to be asked once for the group", 10*time.Second, func() bool { key = onlyAsk(b); return key != "" }) rt.until("the operator to be asked once for the group", 10*time.Second, func() bool { key = onlyAsk(b); return key != "" })
time.Sleep(500 * time.Millisecond) time.Sleep(500 * time.Millisecond)
@@ -758,163 +763,25 @@ func TestAGroupOfTwoWithGreenChecksGetsOneAskAndOnItsApproveBothMergeOneAfterThe
if keys, _ := b.kvs["asks"].Keys(ctx); len(keys) != 1 { if keys, _ := b.kvs["asks"].Keys(ctx); len(keys) != 1 {
t.Fatalf("a second ask was made: %v", keys) t.Fatalf("a second ask was made: %v", keys)
} }
} // The forge's own word that each pull request merged follows the bridge's (its webhook), after the second was handed (the
// checker judges the members still open by the trunk the first merge moved, which this test does not play): both members
// **The acceptance of a failed backup before the send, on the mesh-delivery side** (novox/hq ADR 0404 clause 4, design 47 // end merged, the group with them, and the controller was never called (R6-C3).
// milestone 4, issue 1068): a merged delivery whose walk runs is held when the walk's record, as the controller's b.publish(t, "mesh.mod.gitea.event.pull.merged", merged(one.Repository, one.Number, one.Head.Head, strings.Repeat("a", 40)))
// `plan-moved` carries it, names a backup that failed and nothing was sent; the hold names the machine, the module and the b.publish(t, "mesh.mod.gitea.event.pull.merged", merged(two.Repository, two.Number, two.Head.Head, strings.Repeat("b", 40)))
// backup's words, and nothing asks the controller to let it go. One test function, one bus. The walk's backups field is the controller's for _, id := range []string{IDOf("novox/mesh-controller", ctlHead), IDOf("novox/mesh-catalog", catHead)} {
// contract (issue 1065). id := id
func TestAPlanMovedCarryingAFailedBackupHoldsTheDeliveryWithTheBackupsWordsOverARealBus(t *testing.T) { rt.until(id+" to be merged", eventWithin, func() bool {
b := newTestBus(t) e, err := b.kvs["deliveries"].Get(ctx, kvKey(id))
rt := b.start(t) var d Delivery
ctx, cancel := context.WithCancel(context.Background()) return err == nil && json.Unmarshal(e.Value(), &d) == nil && d.State == Merged
defer cancel() })
rt.deliverEvents(ctx) }
rt.until("the module to subscribe to the events it follows", 20*time.Second, func() bool { rt.until("the group to read merged", eventWithin, func() bool {
rt.mu.Lock() e, err := b.kvs["groups"].Get(ctx, kvKey("feat/x"))
defer rt.mu.Unlock() var g Group
return len(rt.subs) >= len(followed) return err == nil && json.Unmarshal(e.Value(), &g) == nil && g.Said == "merged"
}) })
rt.untilHolding() rt.noControllerAsk()
const head, merge = "3333333333333333333333333333333333333333", "4444444444444444444444444444444444444444"
announced := pr("novox/app", 9, head, "fix/y", "src/a.go")
announced.BaseSHA = "a1a1a1a1a1a1"
deliveryOf := func() Delivery {
e, err := b.kvs["deliveries"].Get(context.Background(), kvKey(IDOf("novox/app", head)))
var d Delivery
if err == nil {
_ = json.Unmarshal(e.Value(), &d)
}
return d
}
// Each event is published after the module has taken the one before it: they are on two subjects, and the runner
// may hand over a later one first, which the table would rightly judge on its own (merged before checked).
b.publish(t, "mesh.mod.gitea.event.pull.updated", announced)
rt.until("the pull request to be taken", 10*time.Second, func() bool { return deliveryOf().State != "" })
b.checkPasses(t, rt, "fix/y", "a1a1a1a1a1a1", judgedPull{"novox/app", 9, head})
rt.until("the verdict to be taken", 10*time.Second, func() bool { d := deliveryOf(); return d.State != "" && d.State != Proposed })
b.publish(t, "mesh.mod.gitea.event.pull.merged", merged("novox/app", 9, head, merge))
rt.until("the merge to be taken", 10*time.Second, func() bool { return deliveryOf().MergedAs != "" })
// Its walk runs, let go: the record names no failed backup yet.
now := time.Now()
walk := aWalk("plan-1", "novox/app", merge, true)
walk.Created, walk.State, walk.Revision = now, walkRolling, 1
walk.Delivery.Go, walk.Delivery.By = &now, "mesh-delivery"
b.publish(t, "mesh.seat."+ControllerSeat+".event.plan-moved", walk)
rt.until("the delivery to be delivering", 10*time.Second, func() bool { return deliveryOf().State == Delivering })
// A backup failed before the send: the delivery is held with the backup's words.
walk.Revision, walk.Backups = 2, &WalkBackups{State: "failed", Asked: []WalkBackupAsked{{Machine: "ace", Module: "pg", State: "failed", Words: "no space left on device"}}}
b.publish(t, "mesh.seat."+ControllerSeat+".event.plan-moved", walk)
rt.until("the delivery to be held for the failed backup", 10*time.Second, func() bool { return deliveryOf().State == Held })
d := deliveryOf()
for _, want := range []string{"ace", "pg", "no space left on device"} {
if !strings.Contains(d.HeldWhy, want) {
t.Fatalf("the held reason %q does not name %q", d.HeldWhy, want)
}
}
// It stays held, and the controller is asked nothing to let it go.
time.Sleep(500 * time.Millisecond)
if s := deliveryOf().State; s != Held {
t.Fatalf("the hold lifted by itself: %s", s)
}
if n := len(rt.asksOf("seat:"+ControllerSeat+".deliver")) + len(rt.asksOf("seat:"+ControllerSeat+".delivery-backups-again")); n != 0 {
t.Fatalf("the controller was asked %d times to let the held walk go or take its backups again", n)
}
}
// **AC3 of issue 1347 (ADR 0437 §7's check): the gate carries a live mesh's open deliveries and groups, once.** The buckets of
// today's module are filled from testdata/live-capture-2026-10-20.json (its source is said in the file), the gate starts with
// today's module on no machine, and every delivery and group arrives in its own buckets unchanged but for the ask an Approve
// waited on; a second start copies nothing.
func TestTheGateCarriesALiveCapturesDeliveriesAndGroupsOnceOverARealBus(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("testdata", "live-capture-2026-10-20.json"))
if err != nil {
t.Fatal(err)
}
var capture struct {
Deliveries []json.RawMessage `json:"deliveries"`
Groups []json.RawMessage `json:"groups"`
}
if err := json.Unmarshal(raw, &capture); err != nil {
t.Fatal(err)
}
b := newTestBus(t)
ctx := context.Background()
want := map[string]Delivery{}
for _, one := range capture.Deliveries {
var d Delivery
if err := json.Unmarshal(one, &d); err != nil {
t.Fatal(err)
}
if _, err := b.kvs["mesh-delivery.deliveries"].Put(ctx, kvKey(d.ID), one); err != nil {
t.Fatal(err)
}
if d.Ready != nil { // what the carry clears: the ask nobody answered
d.Ready.AskID, d.Ready.Ask, d.Ready.Heads = "", nil, nil
}
want[kvKey(d.ID)] = d
}
for _, one := range capture.Groups {
var g Group
if err := json.Unmarshal(one, &g); err != nil {
t.Fatal(err)
}
if _, err := b.kvs["mesh-delivery.groups"].Put(ctx, kvKey(g.ID), one); err != nil {
t.Fatal(err)
}
}
rt := b.start(t)
rt.untilHolding()
for key, d := range want {
e, err := b.kvs["deliveries"].Get(ctx, key)
if err != nil {
t.Fatalf("the delivery %s did not arrive: %v", key, err)
}
var got Delivery
if err := json.Unmarshal(e.Value(), &got); err != nil {
t.Fatal(err)
}
// What is compared is what the carry copies and the gate never rewrites: who the delivery is. By the time this reads, the
// gate holds the seat and acts on the delivery (it asks a check, owes a note), which changes its check ask, its owed
// effects and its transitions, so a whole-value comparison depends on timing. The carry's own exactness is shown without a
// bus by TestTheCarryCopiesWhatItDoesNotClearByteForByte and TestTheGateCarriesTodaysStateOnceWhenItFirstHoldsTheSeat.
if got.ID != d.ID || got.Repository != d.Repository || got.Commit != d.Commit || got.Number != d.Number || got.Base != d.Base ||
got.Group != d.Group || !got.Created.Equal(d.Created) {
t.Fatalf("the delivery %s changed in the carry:\n got %s\nwant %s", key, mustJSON(got), mustJSON(d))
}
}
for _, one := range capture.Groups {
var g Group
_ = json.Unmarshal(one, &g)
e, err := b.kvs["groups"].Get(ctx, kvKey(g.ID))
if err != nil {
t.Fatalf("the group %s did not arrive: %v", g.ID, err)
}
// The carry copies the group byte for byte (TestTheCarryCopiesWhatItDoesNotClearByteForByte); by the time this reads, the gate
// holds the seat and acts, and writes a group's said state as it does, so what is compared is what only the carry could change.
got := mustDecode[Group](t, e.Value())
if got.ID != g.ID || !slices.Equal(got.Members, g.Members) || !got.Created.Equal(g.Created) {
t.Fatalf("the group %s changed in the carry: %+v", g.ID, got)
}
}
if _, err := b.kvs["deliveries"].Get(ctx, carriedKey); err != nil {
t.Fatalf("the mark carried was not written: %v", err)
}
// A second start copies nothing: a delivery today's module gains meanwhile stays where it is.
_ = rt.in.Close()
_ = rt.cmd.Wait()
late := mustJSON(&Delivery{ID: "novox/app@999999999999", Repository: "novox/app", Commit: "999999999999", Number: 99, State: Proposed})
if _, err := b.kvs["mesh-delivery.deliveries"].Put(ctx, kvKey("novox/app@999999999999"), late); err != nil {
t.Fatal(err)
}
again := b.start(t)
again.untilHolding()
if _, err := b.kvs["deliveries"].Get(ctx, kvKey("novox/app@999999999999")); !errors.Is(err, jetstream.ErrKeyNotFound) {
t.Fatalf("a second start copied a key (%v)", err)
}
} }
func mustDecode[T any](t *testing.T, raw []byte) T { func mustDecode[T any](t *testing.T, raw []byte) T {
@@ -12,11 +12,10 @@ import (
// **The gate is the only claimant of the seat `mesh-delivery` in the catalogue** (novox/hq ADR 0437 §1, ADR 0449 §4, ADR 0454, // **The gate is the only claimant of the seat `mesh-delivery` in the catalogue** (novox/hq ADR 0437 §1, ADR 0449 §4, ADR 0454,
// issue 1351; how it is checked). Today's module `modules/mesh-delivery` was removed by a pull request of its own after the // issue 1351; how it is checked). Today's module `modules/mesh-delivery` was removed by a pull request of its own after the
// handover, so the claimants are exactly one. And no manifest but the gate's and mesh-issues' names the old module in what it // handover, so the claimants are exactly one. And no manifest but the gate's and mesh-issues' names the old module in what it
// reads or hears, but through the seat's name (`mesh-delivery` as a seat or a repository, `seat:mesh-delivery.<verb>`) and the // reads or hears, but through the seat's name (`mesh-delivery` as a seat or a repository, `seat:mesh-delivery.<verb>`). The
// names below: the three buckets the gate carries over (ADR 0450 §6). The event `mesh-delivery.transition` is no longer heard by anyone (issue 1406). // three buckets the gate once carried over (ADR 0450 §6) went with the carry (ADR 0539 §17), and the event
var oldModuleNames = map[string][]string{ // `mesh-delivery.transition` is heard by no one (issue 1406).
"mesh-delivery-gate": {"mesh-delivery.deliveries", "mesh-delivery.groups", "mesh-delivery.pulls"}, var oldModuleNames = map[string][]string{}
}
// seatClaimants are the modules that claim the seat: the gate alone. // seatClaimants are the modules that claim the seat: the gate alone.
var seatClaimants = []string{"mesh-delivery-gate"} var seatClaimants = []string{"mesh-delivery-gate"}
@@ -15,6 +15,9 @@ const (
ctxRepo = "mesh/repo-check" ctxRepo = "mesh/repo-check"
) )
// shimCheckedKey is the key a test hears a verdict under: said as the checker's (see Checked).
const shimCheckedKey = "mesh-controller.checked"
// CheckedEvent is a verdict as these tests state it. The gate follows no verdict of the controller any more (issue 1318, // CheckedEvent is a verdict as these tests state it. The gate follows no verdict of the controller any more (issue 1318,
// ADR 0524 §5): the shim below says it the way the mesh says it now, as the head's mesh/check and the checker's verdict at // ADR 0524 §5): the shim below says it the way the mesh says it now, as the head's mesh/check and the checker's verdict at
// the heads, so the table, the Approve and the merges are exercised on the one path they have. // the heads, so the table, the Approve and the merges are exercised on the one path they have.
@@ -38,7 +41,6 @@ type CheckedEvent struct {
Verdict string `json:"verdict"` Verdict string `json:"verdict"`
Summary string `json:"summary"` Summary string `json:"summary"`
} }
Plan *DeliveryPlan
Group string Group string
Members []struct { Members []struct {
Owner string `json:"owner"` Owner string `json:"owner"`
@@ -85,9 +87,6 @@ func (h *Holder) Checked(c CheckedEvent) {
if d == nil { if d == nil {
return return
} }
if c.Plan != nil {
d.Plan = c.Plan
}
words := map[string]string{"success": "pass", "warning": "pass", "failure": "failed: " + c.Summary, "error": "error"}[state] words := map[string]string{"success": "pass", "warning": "pass", "failure": "failed: " + c.Summary, "error": "error"}[state]
line := words + " " + shortOf(c.Commit, 12) line := words + " " + shortOf(c.Commit, 12)
if c.Trunk != "" { if c.Trunk != "" {
@@ -69,9 +69,11 @@ type ChecksOf struct {
Since string `json:"since"` Since string `json:"since"`
Waits string `json:"waits,omitempty"` Waits string `json:"waits,omitempty"`
Group string `json:"group,omitempty"` Group string `json:"group,omitempty"`
// Class, Opened and MergedAt are what the issue tracker measures a fix's time by (novox/hq ADR 0313): the // MergedAs is the commit its merge landed as on the trunk, once merged: what the delivery view's `delivered` event is
// class of what it moves, when its pull request was first heard, and when it merged. // matched to a fix by (ADR 0539 clause 17).
Class string `json:"class,omitempty"` MergedAs string `json:"merged_as,omitempty"`
// Opened and MergedAt are what the issue tracker measures a fix's time by (novox/hq ADR 0313): when its pull request
// was first heard, and when it merged.
Opened string `json:"opened,omitempty"` Opened string `json:"opened,omitempty"`
MergedAt string `json:"merged_at,omitempty"` MergedAt string `json:"merged_at,omitempty"`
} }
@@ -113,7 +115,7 @@ func (h *Holder) Checks(repository string, number int, commit string) (*ChecksAn
} }
if d != nil { if d != nil {
a.Delivery = &ChecksOf{ID: d.ID, State: d.State, Since: d.Since.UTC().Format(time.RFC3339), Waits: d.waits, a.Delivery = &ChecksOf{ID: d.ID, State: d.State, Since: d.Since.UTC().Format(time.RFC3339), Waits: d.waits,
Group: d.Group, Class: fixClass(&d.Delivery)} Group: d.Group, MergedAs: d.MergedAs}
if !d.opened.IsZero() { if !d.opened.IsZero() {
a.Delivery.Opened = d.opened.UTC().Format(time.RFC3339) a.Delivery.Opened = d.opened.UTC().Format(time.RFC3339)
} }
@@ -198,8 +200,8 @@ func (h *Holder) deliveryOf(repository string, number int, commit string) *check
// newerHead is whether d is a pull request's head to read before e: one not superseded first, then the newest. // newerHead is whether d is a pull request's head to read before e: one not superseded first, then the newest.
func newerHead(d, e *Delivery) bool { func newerHead(d, e *Delivery) bool {
if (d.State == Superseded) != (e.State == Superseded) { if (d.State == Closed) != (e.State == Closed) {
return e.State == Superseded return e.State == Closed
} }
return d.Created.After(e.Created) return d.Created.After(e.Created)
} }
@@ -0,0 +1,232 @@
package main
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/asks"
)
// The gate says its own stalled lines to the operator as conditions (novox/hq issue 1275, met again 2026-10-25; ADR 0437
// §9: any module says a condition on the operator-channel seat's one subject, and the router alone chooses the channels).
// Until release 6 the controller's self-check read the `stalled` verb and raised each line as `delivery.<id>.stalled`;
// release 6 deleted that self-check (mesh-controller dc456d56) and nothing read the verb since, so a half-merged group was
// said to no one. Now, after every look, the gate raises each line `stalled` lists, raises it again when its severity or
// its need changes, and clears each it raised that is listed no more. What it raised is kept in its bucket conditions, so a
// gate that restarts clears what ended while it was away.
// ConditionStore keeps the conditions the gate has raised: the condition's key to what it said (conditionSaid).
type ConditionStore interface {
Said() (map[string]string, error)
PutSaid(said map[string]string) error
}
// saidKey is the one key of the bucket conditions.
const saidKey = "said"
// kvConditions is the bucket conditions over the runtime.
type kvConditions struct{}
func (kvConditions) Said() (map[string]string, error) {
e, err := stdio.State("conditions").Get(saidKey)
if err != nil || e == nil {
return map[string]string{}, err
}
said := map[string]string{}
if err := json.Unmarshal(e.Value, &said); err != nil {
return nil, fmt.Errorf("the conditions the gate raised are not readable: %w", err)
}
return said, nil
}
func (kvConditions) PutSaid(said map[string]string) error {
_, err := stdio.State("conditions").Put(saidKey, said)
return err
}
// sayOnSeat says a condition on the operator-channel seat under the gate's own name.
func sayOnSeat(c asks.Condition) error {
subject, err := asks.ConditionSubject("mesh-delivery-gate")
if err != nil {
return err
}
raw, err := asks.EncodeCondition(c)
if err != nil {
return err
}
return stdio.SeatPublish(subject, json.RawMessage(raw), "")
}
// conditionKeyOf is the key of a stalled line's condition: its last part is the condition's kind.
func conditionKeyOf(id string) string { return "mesh-delivery-gate.delivery." + id + ".stalled" }
// conditionSaid is what a raised condition said that, when it changes, raises it again: its severity and what it needs.
// The words of a line hold durations, which change at every look and are no reason to tell the operator again.
func conditionSaid(l StalledLine) string { return severityOfLine(l) + "|" + l.H2 + "|" + l.Needs }
func severityOfLine(l StalledLine) string {
if l.Severity == asks.SeverityUrgent {
return asks.SeverityUrgent
}
return asks.SeverityWarning
}
// conditionOf is a stalled line as the condition the operator is told.
func conditionOf(l StalledLine) asks.Condition {
headline := "stalled: " + l.ID
if strings.HasPrefix(l.Says, "group ") {
group, _, _ := strings.Cut(strings.TrimPrefix(l.Says, "group "), " ")
headline = "group " + group + " is not merged whole"
}
if r := []rune(headline); len(r) > asks.HeadlineLength {
headline = string(r[:asks.HeadlineLength-1]) + "…"
}
needs := "read mesh-delivery.show " + l.ID + " and decide"
return asks.Condition{Key: conditionKeyOf(l.ID), Severity: severityOfLine(l), State: asks.StateRaised,
Subject: asks.Scope{Scope: "delivery", ID: l.ID}, Summary: l.Says, Headline: headline, Needs: needs,
Explanation: asks.NeedsYou + needs + ". " + l.Says + " What ends it: " + l.H2 + ".",
Resolved: "The gate lists " + l.ID + " as stalled no more."}
}
// sayConditions raises and clears the conditions of the stalled lines (see above). Outside the lock; React runs it after
// the look, so it follows every event. A condition that could not be said is said again at the next look, after
// askAgainAfter at the latest.
func (h *Holder) sayConditions() {
if h.Say == nil || h.Conditions == nil {
return
}
if h.said == nil {
said, err := h.Conditions.Said()
if err != nil {
h.Logf("[mesh-delivery] the conditions raised before cannot be read, so none is said yet: %v; asked again in %s", err, askAgainAfter)
h.again("the conditions the gate raised, read")
return
}
h.said = said
}
h.mu.Lock()
if at := h.nextStall(h.Now()); !at.IsZero() {
h.lookAt(at, "a delivery whose stalled line, or its severity, comes due with no event")
}
forgeRead := h.forgeRead
waitingOnForge := map[string]bool{}
for id := range h.unchecked {
waitingOnForge[id] = true
}
h.mu.Unlock()
lines := map[string]StalledLine{}
for _, l := range h.Stalled() {
// One condition for an id: the most severe of its lines.
if o, ok := lines[l.ID]; !ok || (severityOfLine(o) != asks.SeverityUrgent && severityOfLine(l) == asks.SeverityUrgent) {
lines[l.ID] = l
}
}
ids := make([]string, 0, len(lines))
for id := range lines {
ids = append(ids, id)
}
sort.Strings(ids)
changed, failed := false, false
for _, id := range ids {
l, key := lines[id], conditionKeyOf(id)
said := conditionSaid(l)
if h.said[key] == said {
continue
}
if err := h.Say(conditionOf(l)); err != nil {
h.Logf("[mesh-delivery] the condition of %s could not be said: %v", id, err)
failed = true
continue
}
h.Logf("[mesh-delivery] %s is said to the operator as %s: %s", id, severityOfLine(l), l.Says)
h.said[key], changed = said, true
}
// Cleared only on what was read: not before the forge's open pull requests were read once since the gate started (the
// heads waiting on the forge are kept in memory alone), and not while the forge still shows the head open and waiting:
// its line is gone only because its bound runs again from a later first sight, and it comes back with the same words.
var gone []string
for key := range h.said {
id := strings.TrimSuffix(strings.TrimPrefix(key, "mesh-delivery-gate.delivery."), ".stalled")
if _, ok := lines[id]; !ok && forgeRead && !waitingOnForge[id] {
gone = append(gone, key)
}
}
sort.Strings(gone)
for _, key := range gone {
id := strings.TrimSuffix(strings.TrimPrefix(key, "mesh-delivery-gate.delivery."), ".stalled")
c := asks.Condition{Key: key, Subject: asks.Scope{Scope: "delivery", ID: id}, State: asks.StateCleared,
Resolved: "The gate lists " + id + " as stalled no more."}
if err := h.Say(c); err != nil {
h.Logf("[mesh-delivery] the condition of %s could not be cleared: %v", id, err)
failed = true
continue
}
h.Logf("[mesh-delivery] the condition of %s is cleared: it is stalled no more", id)
delete(h.said, key)
changed = true
}
if changed {
if err := h.Conditions.PutSaid(h.said); err != nil {
h.Logf("[mesh-delivery] the conditions raised cannot be kept: %v", err)
}
}
if !failed {
h.sayFailures = 0
return
}
// A seat that keeps refusing is tried again after askAgainAfter, up to sayTries looks in a row; then only at the
// next event, and said once, loudly, in the log (the operator hears nothing while the seat refuses).
h.sayFailures++
switch {
case h.sayFailures < sayTries:
h.again("a condition that could not be said to the operator")
case h.sayFailures == sayTries:
h.Logf("[mesh-delivery] NO CONDITION REACHES THE OPERATOR: the operator-channel seat refused %d looks in a row; "+
"tried again at each event from now on", sayTries)
}
}
// sayTries is how many looks in a row a condition that could not be said is tried again on the clock.
const sayTries = 10
// nextStall is the earliest time after now when a delivery's stalled line comes due, or its severity changes, with nothing
// else happening: a state's bound, the author's bound of a rejected head, a head never announced, a refusal the gate
// answers, and an out-of-date refusal turning urgent (1275/AC6). Zero when none. Under the lock.
func (h *Holder) nextStall(now time.Time) time.Time {
var next time.Time
consider := func(at time.Time) {
if at.After(now) && (next.IsZero() || at.Before(next)) {
next = at
}
}
for _, d := range h.deliveries {
if d.State.Final() {
continue
}
if b, bounded := boundOf(d); bounded && b.For > 0 {
consider(boundSince(d).Add(b.For).Add(time.Second)) // pastBound is strictly after the bound
}
r := d.Ready
if r == nil || r.Merged != nil || d.MergedAs != "" {
continue
}
if d.Base == "" && d.Number != 0 {
consider(r.Seen.Add(headHeardBound))
}
if r.Failed != "" && d.State == Rejected {
consider(d.Since.Add(h.authorBound()))
}
if r.RefusedSince != nil {
consider(r.RefusedSince.Add(h.answeredRefusalBound(d.Repository)))
consider(r.RefusedSince.Add(outOfDateFor))
}
if r.HeldSince != nil {
consider(r.HeldSince.Add(turnBound))
}
}
return next
}
@@ -1,587 +0,0 @@
package main
import (
"fmt"
"strings"
"testing"
"time"
)
// novox/hq issues 319 and 316: a held delivery whose change already runs, or that builds nothing, is not a
// question for a person. A merge that moves nothing is delivered at once; a merge whose every module a later
// merge on the same trunk delivered is covered — never stopped, never an error on its pull request; and a
// group member held for a composed check that had not answered goes on once it passes.
var t0 = time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
// mergedAt is the forge's `pull.merged` at a moment.
func mergedAt(p PullEvent, merge string, at time.Time) PullEvent {
p.MergeCommit = merge
p.MergedAt = at.UTC().Format(time.RFC3339)
return p
}
// verdictMoving is a passing verdict whose plan moves these modules — or builds nothing, given none.
func verdictMoving(repo string, number int, sha string, modules ...string) CheckedEvent {
c := verdict(repo, number, sha, "pass")
c.Plan = planMoving(repo, modules...)
return c
}
func planMoving(repo string, modules ...string) *DeliveryPlan {
if len(modules) == 0 {
return &DeliveryPlan{Repository: repo, Base: "main", Summary: "builds nothing: the change touches no module of the mesh's graph"}
}
return &DeliveryPlan{Repository: repo, Base: "main", Moved: modules, Summary: "builds " + strings.Join(modules, ", ") + " → anchor"}
}
// deliverLater is a later merge into a trunk, walked: its walk opened for its merge commit, let go, and left
// in the state given, every module of it sent.
func deliverLater(w *world, repo string, number int, base, headSHA, merge string, at time.Time, state string,
modules ...string) string {
w.t.Helper()
p := pr(repo, number, headSHA, fmt.Sprintf("feat/later-%d", number))
p.Base = base
w.h.PullUpdated(p)
w.h.Checked(verdictMoving(repo, number, headSHA, modules...))
w.h.PullMerged(mergedAt(p, merge, at))
walk := aWalk("plan-"+merge[:6], repo, merge, true)
walk.Branch = base
walk.Modules = map[string]*WalkModule{}
for _, m := range modules {
walk.Modules[m] = &WalkModule{}
}
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
walk = w.ctl.walk(walk.ID)
sent := at.Add(5 * time.Minute)
for _, m := range modules {
walk.Modules[m] = &WalkModule{State: "built", SentAt: &sent}
}
walk.State = state
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
return IDOf(repo, headSHA)
}
func statusOn(w *world, repo, commit string) string {
w.forge.mu.Lock()
defer w.forge.mu.Unlock()
return w.forge.statuses[repo+"@"+commit+" mesh/delivery"]
}
// **A merge that moves nothing is delivered at once** (issue 319): there is nothing for a walk to move and
// nothing for a person to decide — whether its check passed, failed or never answered, and whether its plan
// builds nothing at all or only a new module the controller builds and sends nowhere (issue 300).
func TestAMergeThatMovesNothingIsDeliveredAtOnce(t *testing.T) {
for _, c := range []struct {
name string
verdict string // "" is a merge before any verdict
plan *DeliveryPlan
}{
{"builds nothing, merged before its check answered", "", planMoving("novox/hq")},
{"builds nothing, merged on a failed check", "fail", planMoving("novox/hq")},
{"builds nothing, merged on a passing check", "pass", planMoving("novox/hq")},
{"a new module, sent nowhere", "pass", &DeliveryPlan{Repository: "novox/hq", New: []string{"modules/zfs"},
Summary: "builds new: modules/zfs, sent nowhere; no bus step"}},
{"a new module, sent nowhere, merged before its check answered", "", &DeliveryPlan{Repository: "novox/hq",
New: []string{"modules/zfs"}, Summary: "builds new: modules/zfs, sent nowhere; no bus step"}},
} {
t.Run(c.name, func(t *testing.T) {
w := newWorld(t)
w.ctl.planned = c.plan
p := pr("novox/hq", 7, head, "docs/x", "02-DECISIONS/x.md")
w.h.PullUpdated(p)
if c.verdict != "" {
v := verdict("novox/hq", 7, head, c.verdict)
v.Plan = c.plan
w.h.Checked(v)
}
w.later(31 * time.Minute)
w.h.PullMerged(mergedAt(p, merge, t0.Add(30*time.Minute)))
w.settleAll()
// Past the bound a merge waits for its walk: nothing holds it then either.
w.later(15 * time.Minute)
if err := w.h.Reconcile(); err != nil {
t.Fatal(err)
}
w.settleAll()
d := w.delivery(IDOf("novox/hq", head))
if d.State != Delivered {
t.Fatalf("a merge whose plan is %q is %s (%s), not delivered", c.plan.Summary, d.State, d.HeldWhy)
}
if s := statusOn(w, "novox/hq", head); !strings.HasPrefix(s, "success") {
t.Fatalf("its pull request's mesh/delivery says %q", s)
}
})
}
}
// **A held delivery whose every module a later merge on its trunk delivered is covered** (issue 319): the
// later merge commit contains it, its walk is done, and it sent each module this one moves. Anything short
// of that — a module not carried, an earlier merge, another trunk, a walk that did not finish — leaves it held
// for a person, as before.
func TestAMergeALaterDeliveredWalkCarriedIsCovered(t *testing.T) {
const ctl = "novox/mesh-controller"
for _, c := range []struct {
name string
// The later merge: its trunk, minutes after t0 it merged (this one merged at 30), its walk's end, and
// what it sent.
base string
at int
state string
modules []string
// Whether this one was held for a person before the later merge was delivered.
heldFirst bool
// What the forge says of the later merge containing this one: yes, no, or away until asked again.
forge string
want State
}{
{"carried every module, delivered after this one was held", "main", 35, walkDone,
[]string{"build-agent", "mesh-controller"}, true, "yes", Covered},
{"carried every module, delivered before this one was due to be held", "main", 35, walkDone,
[]string{"build-agent", "mesh-controller", "route-proxy"}, false, "yes", Covered},
{"carried one of its two modules", "main", 35, walkDone, []string{"mesh-controller"}, true, "yes", Held},
{"merged before this one", "main", 25, walkDone, []string{"build-agent", "mesh-controller"}, true, "yes", Held},
{"merged into another branch", "release", 35, walkDone, []string{"build-agent", "mesh-controller"}, true, "yes", Held},
{"its walk failed", "main", 35, walkFailed, []string{"build-agent", "mesh-controller"}, true, "yes", Held},
{"the forge says the later merge does not contain it", "main", 35, walkDone,
[]string{"build-agent", "mesh-controller"}, true, "no", Held},
{"the forge could not say at first, then said it does", "main", 35, walkDone,
[]string{"build-agent", "mesh-controller"}, true, "away", Covered},
} {
t.Run(c.name, func(t *testing.T) {
w := newWorld(t)
pair := merge + " in nnnnnnnnnnnn8888"
switch c.forge {
case "yes", "away":
w.forge.contains[pair] = true
case "no":
w.forge.contains[pair] = false
}
w.forge.containsDown = c.forge == "away"
p := pr(ctl, 1, head, "feat/one", "cmd/a.go")
w.h.PullUpdated(p)
w.h.Checked(verdictMoving(ctl, 1, head, "mesh-controller", "build-agent"))
w.later(31 * time.Minute)
w.h.PullMerged(mergedAt(p, merge, t0.Add(30*time.Minute)))
w.settleAll()
if c.heldFirst {
w.later(15 * time.Minute)
if err := w.h.Reconcile(); err != nil {
t.Fatal(err)
}
w.settleAll()
if s := w.state(IDOf(ctl, head)); s != Held {
t.Fatalf("a merge no walk opened for is %s before the later one", s)
}
}
later := deliverLater(w, ctl, 2, c.base, "bbbbbbbbbbbb2222", "nnnnnnnnnnnn8888", t0.Add(time.Duration(c.at)*time.Minute),
c.state, c.modules...)
if c.state == walkDone && w.state(later) != Delivered {
t.Fatalf("the later merge is %s", w.state(later))
}
w.later(20 * time.Minute)
if err := w.h.Reconcile(); err != nil {
t.Fatal(err)
}
w.settleAll()
if c.forge == "away" {
// A forge that cannot say is "not yet covered": held, and asked again a minute on.
if s := w.state(IDOf(ctl, head)); s != Held {
t.Fatalf("with the forge away it is %s", s)
}
w.forge.mu.Lock()
w.forge.containsDown = false
w.forge.mu.Unlock()
w.later(2 * time.Minute)
w.settleAll()
}
d := w.delivery(IDOf(ctl, head))
if d.State != c.want {
t.Fatalf("it is %s (%s), want %s", d.State, d.HeldWhy, c.want)
}
s := statusOn(w, ctl, head)
if c.want == Covered {
if !strings.HasPrefix(s, "success covered: a later merge that contains it was delivered") ||
!strings.Contains(s, "nnnnnnnn") {
t.Fatalf("its pull request's mesh/delivery says %q", s)
}
last := d.Transitions[len(d.Transitions)-1]
if !strings.Contains(last.Why, IDOf(ctl, "bbbbbbbbbbbb2222")) {
t.Fatalf("its transition does not name what covered it: %q", last.Why)
}
}
if strings.HasPrefix(s, "error") {
t.Fatalf("its pull request's mesh/delivery is an error: %q", s)
}
// Asked once for the pair, however many ticks read it — once more only after the forge was away.
asked := 0
for _, a := range w.forge.containsAsks {
if a == pair {
asked++
}
}
candidate := c.base == "main" && c.at > 30 && c.state == walkDone && len(c.modules) >= 2
switch {
case !candidate && asked != 0:
t.Fatalf("the forge was asked whether %s for a merge order and walk that cannot cover it", pair)
case candidate && c.forge != "away" && asked != 1:
t.Fatalf("the forge was asked %d time(s) whether %s, want once", asked, pair)
case candidate && c.forge == "away" && asked < 2:
t.Fatalf("the forge, away, was asked %d time(s) whether %s: never again", asked, pair)
}
})
}
}
// A covered member of a group counts as delivered: the member after it is let go, not stopped.
func TestACoveredGroupMemberLetsTheNextOneGo(t *testing.T) {
w := newWorld(t)
const ctl, lab = "novox/mesh-controller", "novox/mesh-lab"
const ctlHead, labHead = "c1c1c1c1c1c1c1c1", "d2d2d2d2d2d2d2d2"
ctlID, labID := IDOf(ctl, ctlHead), IDOf(lab, labHead)
pc, pl := pr(ctl, 1, ctlHead, "feat/g", "cmd/a.go"), pr(lab, 2, labHead, "feat/g", "replays/a.go")
w.h.PullUpdated(pc)
w.h.PullUpdated(pl)
w.h.Checked(verdictMoving(ctl, 1, ctlHead, "mesh-controller"))
w.h.Checked(verdictMoving(lab, 2, labHead, "mesh-lab"))
w.settleAll()
w.h.Checked(CheckedEvent{Group: "feat/g", ID: "check-1", Verdict: "pass", Summary: "both compose"})
w.later(31 * time.Minute)
w.h.PullMerged(mergedAt(pc, "m1m1m1m1m1m1", t0.Add(30*time.Minute)))
w.h.PullMerged(mergedAt(pl, "m2m2m2m2m2m2", t0.Add(31*time.Minute)))
w.h.WalkMoved(w.ctl.put(aWalk("plan-lab", lab, "m2m2m2m2m2m2", true)))
w.settleAll()
w.forge.contains["m1m1m1m1m1m1 in m3m3m3m3m3m3"] = true
deliverLater(w, ctl, 3, "main", "e3e3e3e3e3e3e3e3", "m3m3m3m3m3m3", t0.Add(35*time.Minute), walkDone, "mesh-controller")
w.later(20 * time.Minute)
if err := w.h.Reconcile(); err != nil {
t.Fatal(err)
}
w.settleAll()
if w.state(ctlID) != Covered {
t.Fatalf("the controller's member is %s", w.state(ctlID))
}
if w.state(labID) != Delivering || !containsString(w.ctl.delivers, "plan-lab") {
t.Fatalf("the member after it is %s, let go %v", w.state(labID), w.ctl.delivers)
}
lw := w.ctl.walk("plan-lab")
lw.State = walkDone
w.h.WalkMoved(w.ctl.put(lw))
w.settleAll()
if state, why := GroupState(w.h.groups["feat/g"], w.h.membersOf(w.h.groups["feat/g"])); state != "delivered" {
t.Fatalf("the group is %s: %s", state, why)
}
}
// **On rollout, the deliveries already held close by themselves** (issue 319's held ones), and a restart
// after that moves nothing again: what was read back is where it stays.
func TestHeldDeliveriesReadBackCloseByThemselvesOnce(t *testing.T) {
const ctl, cat = "novox/mesh-controller", "novox/mesh-catalog"
merged1, merged2 := t0, t0.Add(2*time.Hour)
coverer := func(repo, module string) *Delivery {
return &Delivery{ID: IDOf(repo, "c0c0c0c0c0c0c0c0"), Repository: repo, Commit: "c0c0c0c0c0c0c0c0", Number: 90,
Base: "main", State: Delivered, Since: merged2.Add(time.Hour), Created: merged2.Add(-time.Hour),
MergedAs: "f0f0f0f0f0f0f0f0", MergedAt: merged2, Plan: planMoving(repo, module), Check: &Verdict{Gate: "pass"},
Walk: &WalkSeen{ID: "plan-90", Commit: "f0f0f0f0f0f0f0f0", State: walkDone},
Steps: []Step{{Machine: "anchor", Module: module, State: StepPassed}, {Machine: "the rest running it", Module: module, State: StepSent}}}
}
held := func(repo string, plan *DeliveryPlan, why string) *Delivery {
return &Delivery{ID: IDOf(repo, head), Repository: repo, Commit: head, Number: 7, Base: "main", State: Held,
Since: merged1.Add(10 * time.Minute), Created: merged1.Add(-time.Hour), MergedAs: merge, MergedAt: merged1,
HeldWhy: why, Plan: plan, Transitions: []Transition{{At: merged1, From: Ready, To: Held, Event: EvHold, Why: why}}}
}
const unwalkedWhy = "merged, and the controller opened no walk for it within 10m0s — nothing it holds follows that branch, " +
"or the merge was not heard: a person decides"
withCoverer := func(d *Delivery, change func(c *Delivery)) []*Delivery {
c := coverer(d.Repository, d.Plan.Moved[0])
change(c)
return []*Delivery{d, c}
}
const uncheckedWhy = "merged without a passing check: only a person decides that it goes on"
for _, c := range []struct {
name string
kept []*Delivery
group *Group
want State
}{
{"merged without a passing check, builds nothing",
[]*Delivery{held("novox/hq", planMoving("novox/hq"), "merged without a passing check: only a person decides that it goes on")},
nil, Delivered},
{"a new module sent nowhere, no walk opened",
[]*Delivery{held(cat, &DeliveryPlan{New: []string{"modules/zfs"}, Summary: "builds new: modules/zfs, sent nowhere"}, unwalkedWhy)},
nil, Delivered},
{"no walk opened, a later delivered walk carried its module",
[]*Delivery{held(ctl, planMoving(ctl, "mesh-controller"), unwalkedWhy), coverer(ctl, "mesh-controller")}, nil, Covered},
{"merged without a passing check, a later delivered walk whose own check passed carried its module",
withCoverer(held(ctl, planMoving(ctl, "mesh-controller"), uncheckedWhy), func(*Delivery) {}), nil, Covered},
{"merged without a passing check, the later delivery's own check did not pass",
withCoverer(held(ctl, planMoving(ctl, "mesh-controller"), uncheckedWhy), func(c *Delivery) {
c.Check = &Verdict{Gate: "fail"}
}), nil, Held},
{"no walk opened, the later delivery's walk was of another commit",
withCoverer(held(ctl, planMoving(ctl, "mesh-controller"), unwalkedWhy), func(c *Delivery) {
c.Walk.Commit = "e0e0e0e0e0e0e0e0"
}), nil, Held},
{"no walk opened, the later walk sent its module to its first machine only",
withCoverer(held(ctl, planMoving(ctl, "mesh-controller"), unwalkedWhy), func(c *Delivery) {
c.Steps = c.Steps[:1]
}), nil, Held},
{"no walk opened, the later walk's branch is another",
withCoverer(held(ctl, planMoving(ctl, "mesh-controller"), unwalkedWhy), func(c *Delivery) {
c.Walk.Branch = "release"
}), nil, Held},
{"no walk opened, a later delivered walk carried another module",
[]*Delivery{held(ctl, planMoving(ctl, "mesh-controller"), unwalkedWhy), coverer(ctl, "route-proxy")}, nil, Held},
{"held for its group, its own walk superseded, a later delivered walk carried its module",
func() []*Delivery {
d := held(cat, planMoving(cat, "systemd"), "its group feat/j's composed check did not pass for the heads that "+
"merged; a person decides that it goes on")
d.Group = "feat/j"
d.Walk = &WalkSeen{ID: "plan-7", Commit: merge, State: walkSuperseded, Waits: true, WaitedFor: "mesh-delivery"}
other := &Delivery{ID: IDOf("novox/mesh-host", "abababababababab"), Repository: "novox/mesh-host",
Commit: "abababababababab", Number: 3, Base: "main", Group: "feat/j", State: Delivered,
MergedAs: "acacacacacacacac", MergedAt: merged1}
return []*Delivery{d, other, coverer(cat, "systemd")}
}(),
&Group{ID: "feat/j", Members: []string{IDOf(cat, head), IDOf("novox/mesh-host", "abababababababab")}, Closed: true,
Check: &GroupCheck{Asked: "check-1", Verdict: "pass", Commits: map[string]string{IDOf(cat, head): head,
IDOf("novox/mesh-host", "abababababababab"): "abababababababab"}}},
Covered},
} {
t.Run(c.name, func(t *testing.T) {
store := newMemStore()
for _, d := range c.kept {
if err := store.PutDelivery(d); err != nil {
t.Fatal(err)
}
}
if c.group != nil {
if err := store.PutGroup(c.group); err != nil {
t.Fatal(err)
}
}
now := merged2.Add(30 * time.Hour)
forge := newFakeForge()
forge.contains[merge+" in f0f0f0f0f0f0f0f0"] = true
w := newWorldOver(t, store, newFakeController(), forge, now)
if err := w.h.Reconcile(); err != nil {
t.Fatal(err)
}
w.settleAll()
id := c.kept[0].ID
d := w.delivery(id)
if d.State != c.want {
t.Fatalf("read back held, it is %s, want %s", d.State, c.want)
}
if s := statusOn(w, d.Repository, d.Commit); c.want != Held && !strings.HasPrefix(s, "success") {
t.Fatalf("its pull request's mesh/delivery says %q", s)
}
// A restart over what it kept: nothing moves again.
again := newWorldOver(t, store, newFakeController(), forge, now.Add(time.Hour))
if err := again.h.Reconcile(); err != nil {
t.Fatal(err)
}
again.settleAll()
if d2 := again.delivery(id); d2.State != d.State || len(d2.Transitions) != len(d.Transitions) {
t.Fatalf("a restart moved it again: %s with %d transitions, was %s with %d", d2.State, len(d2.Transitions),
d.State, len(d.Transitions))
}
})
}
}
// **A module a release carried to every machine the held delivery sends it to covers it** (novox/hq issue
// 351). A build the controller held back for a person and released on their word reaches each machine by a
// backlog walk, which credits the delivery that published the build one step per machine — never the walk's
// own "every machine running it" or "the rest running it". The held delivery's plan says which machines
// receive the module: a passed step on each of them is the module sent everywhere. A machine still judging,
// one that failed, or a plan that names no machines leaves it held, as before.
func TestAModuleAReleaseCarriedToEveryMachineCovers(t *testing.T) {
const cat = "novox/mesh-catalog"
merged1, merged2 := t0, t0.Add(2*time.Hour)
laptop, desktop := "laptop", "desktop"
plan := func(machines ...string) *DeliveryPlan {
p := planMoving(cat, "openrazer")
for _, m := range machines {
p.Machines = append(p.Machines, MachinePlan{Machine: m, Receives: []string{"openrazer"}})
}
return p
}
held := func(p *DeliveryPlan) *Delivery {
why := "its group feat/j's composed check did not pass for the heads that merged; a person decides that it goes on"
return &Delivery{ID: IDOf(cat, head), Repository: cat, Commit: head, Number: 7, Base: "main", State: Held,
Since: merged1.Add(10 * time.Minute), Created: merged1.Add(-time.Hour), MergedAs: merge, MergedAt: merged1,
HeldWhy: why, Plan: p, Walk: &WalkSeen{ID: "plan-7", Commit: merge, State: walkSuperseded, Waits: true, WaitedFor: "mesh-delivery"},
Transitions: []Transition{{At: merged1, From: Published, To: Held, Event: EvHold, Why: why}}}
}
step := func(machine string, state StepState) Step {
return Step{Machine: machine, Module: "openrazer", Build: "build-1", State: state, Why: "carried by release-1"}
}
coverer := func(steps ...Step) *Delivery {
return &Delivery{ID: IDOf(cat, "c0c0c0c0c0c0c0c0"), Repository: cat, Commit: "c0c0c0c0c0c0c0c0", Number: 90,
Base: "main", State: Delivered, Since: merged2.Add(time.Hour), Created: merged2.Add(-time.Hour),
MergedAs: "f0f0f0f0f0f0f0f0", MergedAt: merged2, Plan: planMoving(cat, "power"), Check: &Verdict{Gate: "pass"},
Walk: &WalkSeen{ID: "plan-90", Commit: "f0f0f0f0f0f0f0f0", State: walkDone},
Steps: append([]Step{{Machine: "anchor", Module: "power", State: StepPassed},
{Machine: "the rest running it", Module: "power", State: StepSent}}, steps...)}
}
for _, c := range []struct {
name string
held *Delivery
by *Delivery
want State
}{
{"passed on every machine its plan sends it to", held(plan(laptop, desktop)),
coverer(step(laptop, StepPassed), step(desktop, StepPassed)), Covered},
{"still judging on one of them", held(plan(laptop, desktop)),
coverer(step(laptop, StepPassed), step(desktop, StepJudging)), Held},
{"failed on one of them", held(plan(laptop, desktop)),
coverer(step(laptop, StepPassed), step(desktop, StepFailed)), Held},
{"never sent to one of them", held(plan(laptop, desktop)), coverer(step(laptop, StepPassed)), Held},
{"its plan names no machines", held(plan()), coverer(step(laptop, StepPassed), step(desktop, StepPassed)), Held},
} {
t.Run(c.name, func(t *testing.T) {
store := newMemStore()
for _, d := range []*Delivery{c.held, c.by} {
if err := store.PutDelivery(d); err != nil {
t.Fatal(err)
}
}
forge := newFakeForge()
forge.contains[merge+" in f0f0f0f0f0f0f0f0"] = true
w := newWorldOver(t, store, newFakeController(), forge, merged2.Add(30*time.Hour))
if err := w.h.Reconcile(); err != nil {
t.Fatal(err)
}
w.settleAll()
d := w.delivery(c.held.ID)
if d.State != c.want {
t.Fatalf("it is %s (%s), want %s", d.State, d.HeldWhy, c.want)
}
if c.want == Covered {
if s := statusOn(w, cat, head); !strings.HasPrefix(s, "success covered: a later merge that contains it was delivered") {
t.Fatalf("its pull request's mesh/delivery says %q", s)
}
if last := d.Transitions[len(d.Transitions)-1]; !strings.Contains(last.Why, "openrazer") {
t.Fatalf("its transition does not name the module: %q", last.Why)
}
}
})
}
}
// **A finished release moves the steps it carried to passed** (novox/hq issue 351): a backlog walk's gate
// credits a step per machine while it judges, and its last record keeps no gate — only which machines it is
// done on. A step credited to it, on a machine it is done on, is passed; a step it failed on, one on a machine
// it is not done on, and a step another walk carried are left as they are.
func TestAFinishedReleaseMovesTheStepsItCarriedToPassed(t *testing.T) {
const cat = "novox/mesh-catalog"
carried := func(machine string, state StepState, by string) Step {
return Step{Machine: machine, Module: "openrazer", Build: "build-1", State: state, Why: "carried by " + by}
}
for _, c := range []struct {
name string
steps []Step
done []string
want map[string]StepState
}{
{"done on both", []Step{carried("laptop", StepJudging, "release-1"), carried("desktop", StepJudging, "release-1")},
[]string{"laptop", "desktop"}, map[string]StepState{"laptop": StepPassed, "desktop": StepPassed}},
{"done on one, the other still to come", []Step{carried("laptop", StepJudging, "release-1"), carried("desktop", StepJudging, "release-1")},
[]string{"laptop"}, map[string]StepState{"laptop": StepPassed, "desktop": StepJudging}},
{"a step it failed on stays failed", []Step{carried("laptop", StepFailed, "release-1")},
[]string{"laptop"}, map[string]StepState{"laptop": StepFailed}},
{"a step another walk carried is not its", []Step{carried("laptop", StepJudging, "release-2")},
[]string{"laptop"}, map[string]StepState{"laptop": StepJudging}},
} {
t.Run(c.name, func(t *testing.T) {
d := &Delivery{ID: IDOf(cat, "c0c0c0c0c0c0c0c0"), Repository: cat, Commit: "c0c0c0c0c0c0c0c0", Number: 90,
Base: "main", State: Delivered, Since: t0.Add(time.Hour), Created: t0, MergedAs: "f0f0f0f0f0f0f0f0", MergedAt: t0,
Plan: planMoving(cat, "power"), Check: &Verdict{Gate: "pass"},
Walk: &WalkSeen{ID: "plan-90", Commit: "f0f0f0f0f0f0f0f0", State: walkDone}, Steps: c.steps}
store := newMemStore()
if err := store.PutDelivery(d); err != nil {
t.Fatal(err)
}
w := newWorldOver(t, store, newFakeController(), newFakeForge(), t0.Add(2*time.Hour))
w.h.WalkMoved(Walk{ID: "release-1", Repository: "the builds waiting for a gate", State: walkDone,
Tiers: [][]string{{"openrazer"}}, Modules: map[string]*WalkModule{},
Release: &WalkRelease{Order: []string{"laptop", "desktop"}, Next: len(c.done), Done: c.done}})
got := map[string]StepState{}
for _, s := range w.delivery(d.ID).Steps {
got[s.Machine] = s.State
}
for machine, want := range c.want {
if got[machine] != want {
t.Fatalf("openrazer on %s is %s, want %s (steps %v)", machine, got[machine], want, got)
}
}
})
}
}
// The day's case whole (novox/hq issue 351): a member held for its group's check, its own walk taken over by
// later merges; the one later walk that carried its module failed its gate; a later merge's walk was delivered
// without the module; the controller then held the module's build back for a person, who released it, and the
// release reached both machines. The held delivery is covered by that later merge once the release is done —
// read by its id, since a walk that ended before the holder's restart may be older than the ended walks the
// controller lists back.
func TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone(t *testing.T) {
const cat = "novox/mesh-catalog"
merged1, merged2 := t0, t0.Add(2*time.Hour)
plan := planMoving(cat, "openrazer")
plan.Machines = []MachinePlan{{Machine: "laptop", Receives: []string{"openrazer"}}, {Machine: "desktop", Receives: []string{"openrazer"}}}
why := "its group feat/j's composed check did not pass for the heads that merged; a person decides that it goes on"
held := &Delivery{ID: IDOf(cat, head), Repository: cat, Commit: head, Number: 7, Base: "main", State: Held,
Since: merged1.Add(10 * time.Minute), Created: merged1.Add(-time.Hour), MergedAs: merge, MergedAt: merged1,
HeldWhy: why, Plan: plan, Walk: &WalkSeen{ID: "plan-7", Commit: merge, State: walkSuperseded, Waits: true, WaitedFor: "mesh-delivery"},
Transitions: []Transition{{At: merged1, From: Published, To: Held, Event: EvHold, Why: why}}}
later := &Delivery{ID: IDOf(cat, "c0c0c0c0c0c0c0c0"), Repository: cat, Commit: "c0c0c0c0c0c0c0c0", Number: 90,
Base: "main", State: Delivered, Since: merged2.Add(time.Hour), Created: merged2.Add(-time.Hour),
MergedAs: "f0f0f0f0f0f0f0f0", MergedAt: merged2, Plan: planMoving(cat, "power"), Check: &Verdict{Gate: "pass"},
Walk: &WalkSeen{ID: "plan-90", Commit: "f0f0f0f0f0f0f0f0", State: walkDone},
Steps: []Step{{Machine: "anchor", Module: "power", State: StepPassed}, {Machine: "the rest running it", Module: "power", State: StepSent}}}
store := newMemStore()
for _, d := range []*Delivery{held, later} {
if err := store.PutDelivery(d); err != nil {
t.Fatal(err)
}
}
forge := newFakeForge()
forge.contains[merge+" in f0f0f0f0f0f0f0f0"] = true
w := newWorldOver(t, store, newFakeController(), forge, merged2.Add(3*time.Hour))
if err := w.h.Reconcile(); err != nil {
t.Fatal(err)
}
w.settleAll()
if s := w.state(held.ID); s != Held {
t.Fatalf("before any release it is %s", s)
}
// The release judges the build on each machine in turn: its gate names what it carried.
release := Walk{ID: "release-1", Repository: "the builds waiting for a gate", State: walkRolling,
Tiers: [][]string{{"openrazer"}}, Modules: map[string]*WalkModule{},
Release: &WalkRelease{Order: []string{"laptop", "desktop"}, Gate: &WalkGate{Machines: []string{"laptop"},
Carried: []WalkCarry{{Module: "openrazer", Node: "laptop", To: "f0f0f0f0f0f0f0f0", Build: "build-1"},
{Module: "openrazer", Node: "desktop", To: "f0f0f0f0f0f0f0f0", Build: "build-1"}}}}}
w.h.WalkMoved(release)
w.settleAll()
if s := w.state(held.ID); s != Held {
t.Fatalf("while the release judges it is %s", s)
}
// Done on both: its last record keeps no gate. It ended while this holder was down, and is older than the
// ended walks the controller lists back: the holder asks for it by its id, since a step it carried still judges.
release.State, release.Release.Gate, release.Release.Done, release.Release.Next = walkDone, nil, []string{"laptop", "desktop"}, 2
w.ctl.put(release)
w.later(time.Hour) // a walk a step still judges is asked for once an hour; at a start, at once
w.settleAll()
d := w.delivery(held.ID)
if d.State != Covered {
t.Fatalf("once the release is done it is %s (%s), want covered", d.State, d.HeldWhy)
}
if last := d.Transitions[len(d.Transitions)-1]; !strings.Contains(last.Why, IDOf(cat, "c0c0c0c0c0c0c0c0")) ||
!strings.Contains(last.Why, "openrazer") {
t.Fatalf("its transition does not name what covered it: %q", last.Why)
}
}
@@ -31,15 +31,6 @@ func transitionEvent(d *Delivery, t Transition, opened time.Time) map[string]any
if d.MergedAs != "" { if d.MergedAs != "" {
e["merged_as"] = d.MergedAs e["merged_as"] = d.MergedAs
} }
if d.Walk != nil {
e["walk"] = d.Walk.ID
}
if d.Plan != nil {
e["plan"] = d.Plan.Summary
}
if c := fixClass(d); c != "" {
e["class"] = c
}
if !opened.IsZero() { if !opened.IsZero() {
e["opened"] = opened.UTC() e["opened"] = opened.UTC()
} }
@@ -49,24 +40,6 @@ func transitionEvent(d *Delivery, t Transition, opened time.Time) map[string]any
return e return e
} }
// fixClass is the class of what a delivery moves (novox/hq ADR 0313 §2): its plan's, as the planner said it;
// none for a plan that moves no module, from a planner that did not say a class; else its walk's class once its
// delivery time is known; empty when nothing says it.
func fixClass(d *Delivery) string {
switch {
case d.Plan != nil && d.Plan.Class != "":
return d.Plan.Class
case d.Plan != nil && len(d.Plan.Moved) == 0 && len(d.Plan.Dependents) == 0 && len(d.Plan.New) == 0:
return classNone
case d.Time != nil:
return d.Time.Class
}
return ""
}
// classNone is the class of a delivery that moves no module: nothing for a walk to move, and no fix budget.
const classNone = "none"
// openedOf is when a delivery's pull request was first heard: the earliest head of it this owner holds, never // openedOf is when a delivery's pull request was first heard: the earliest head of it this owner holds, never
// one made from the forge's word at its merge; zero when none is held or it has no pull request. Under the lock. // one made from the forge's word at its merge; zero when none is held or it has no pull request. Under the lock.
func (h *Holder) openedOf(d *Delivery) time.Time { func (h *Holder) openedOf(d *Delivery) time.Time {
@@ -96,88 +69,50 @@ func noteLine(d *Delivery, t Transition) string {
} }
line := fmt.Sprintf("%s delivery %s: %s -> %s (%s%s): %s", t.At.UTC().Format(time.RFC3339), d.ID, orNothing(t.From), line := fmt.Sprintf("%s delivery %s: %s -> %s (%s%s): %s", t.At.UTC().Format(time.RFC3339), d.ID, orNothing(t.From),
t.To, t.Event, by, t.Why) t.To, t.Event, by, t.Why)
if t.To == Proposed && d.Plan != nil { if d.Group != "" && (t.To == Proposed || t.To == Merged) {
line += "; predicted: " + d.Plan.Summary
}
if d.Group != "" && (t.To == Proposed || t.To == Published) {
line += "; group " + d.Group line += "; group " + d.Group
} }
return oneLine(line) return oneLine(line)
} }
// executedLine is what a delivery did, written on the commit it landed as when it ends.
func executedLine(d *Delivery) string {
var parts []string
for _, s := range d.Steps {
parts = append(parts, fmt.Sprintf("%s on %s %s", s.Module, s.Machine, s.State))
}
walk := ""
if d.Walk != nil {
walk = " (walk " + d.Walk.ID + ")"
}
return oneLine(fmt.Sprintf("%s delivery %s executed%s: %s", time.Now().UTC().Format(time.RFC3339), d.ID, walk,
strings.Join(parts, "; ")))
}
func oneLine(s string) string { return strings.Join(strings.Fields(s), " ") } func oneLine(s string) string { return strings.Join(strings.Fields(s), " ") }
// statusOf is the commit status `mesh/delivery`: where the delivery stands, in the forge's words. // statusOf is the commit status `mesh/delivery`: where the delivery stands, in the forge's words. It says the merge's
// states only (ADR 0539 §17): whether a merged change reached the machines is the view's.
func statusOf(d *Delivery) (string, string) { func statusOf(d *Delivery) (string, string) {
why := "" why := ""
if len(d.Transitions) > 0 { if len(d.Transitions) > 0 {
why = d.Transitions[len(d.Transitions)-1].Why why = d.Transitions[len(d.Transitions)-1].Why
} }
switch d.State { switch d.State {
case Proposed, Checked: case Proposed:
return "pending", "checking" return "pending", "checking"
case Ready: case Ready:
if d.holdsForReview() { if d.holdsForReview() {
return "pending", clip("ready, "+d.Ready.Waits, 140) // novox/hq ADR 0496: no Approve is asked before a review return "pending", clip("ready, "+d.Ready.Waits, 140) // novox/hq ADR 0496: no Approve is asked before a review
} }
return "success", "ready: it delivers once merged" return "success", "ready: it is merged on the Approve"
case Rejected: case Rejected:
return "failure", clip("rejected: "+why, 140) return "failure", clip("rejected: "+why, 140)
case Published: case Merged:
return "pending", "published: its walk waits for its turn" return "success", clip("merged: "+d.sourceSaid(), 140)
case Held: case Held:
return "pending", clip("held for a person: "+d.HeldWhy, 140) return "pending", clip("held for a person: "+d.HeldWhy, 140)
case Delivering: case Closed:
passed := 0 return "warning", clip("closed: "+why, 140)
for _, s := range d.Steps {
if s.State == StepPassed {
passed++
}
}
return "pending", fmt.Sprintf("delivering: %d machine step(s) passed", passed)
case Delivered:
return "success", "delivered"
case Failed:
if d.coveredStop() {
return "success", clip(d.coveredStopSaid(), 140) // its change reached the machines (novox/hq issue 747)
}
return "failure", clip("failed: "+why, 140)
case Superseded:
if d.coveredStop() {
return "success", clip(d.coveredStopSaid(), 140) // its walk's taker delivered it (issue 1786)
}
return "warning", clip("superseded: "+why, 140)
case Stopped:
if d.coveredStop() {
return "success", clip(d.coveredStopSaid(), 140) // its change reached the machines (novox/hq issue 480)
}
return "error", clip("stopped: "+why, 140)
case Retired:
return "success", "history: merged before this owner delivered anything, never walked"
case Covered:
by := d.CoveredBy
if by == "" {
by = d.Carrier
}
return "success", clip("covered: a later merge that contains it was delivered: "+by, 140)
} }
return "pending", string(d.State) return "pending", string(d.State)
} }
// sourceSaid names the source revision a merge moved: the trunk commit of its repository that source.main now holds
// (ADR 0539 §17: the gate's last word on a change; what the machines run is the view's).
func (d *Delivery) sourceSaid() string {
if d.MergedAs == "" {
return "not on the trunk"
}
return "source.main moved to " + d.Repository + "@" + shortOf(d.MergedAs, 12)
}
// viewMarker is how the forge's holder finds the one view comment of a pull request. // viewMarker is how the forge's holder finds the one view comment of a pull request.
const viewMarker = "<!-- mesh-delivery:view -->" const viewMarker = "<!-- mesh-delivery:view -->"
@@ -185,31 +120,12 @@ const viewMarker = "<!-- mesh-delivery:view -->"
func ViewBody(d *Delivery, g *Group) string { func ViewBody(d *Delivery, g *Group) string {
var b strings.Builder var b strings.Builder
fmt.Fprintf(&b, "%s\n**Delivery** `%s` — **%s** since %s\n\n", viewMarker, d.ID, d.State, d.Since.UTC().Format(time.RFC3339)) fmt.Fprintf(&b, "%s\n**Delivery** `%s` — **%s** since %s\n\n", viewMarker, d.ID, d.State, d.Since.UTC().Format(time.RFC3339))
if d.coveredStop() {
fmt.Fprintf(&b, "**Covered**: %s\n\n", d.coveredStopSaid())
}
if d.HeldWhy != "" && d.State == Held { if d.HeldWhy != "" && d.State == Held {
fmt.Fprintf(&b, "Held for a person: %s — `mesh-delivery.release` with why lets it go on.\n\n", d.HeldWhy) fmt.Fprintf(&b, "Held for a person: %s — the merge reached the trunk without a passing check.\n\n", d.HeldWhy)
} }
if d.Plan != nil { if d.MergedAs != "" {
fmt.Fprintf(&b, "**Delivery plan** — %s\n", d.Plan.Summary) fmt.Fprintf(&b, "**Source** — %s. Whether it reached the machines is `mesh-delivery-view`'s, derived from "+
for i, tier := range d.Plan.Tiers { "`source.main`, the wants and the haves.\n\n", d.sourceSaid())
fmt.Fprintf(&b, "- tier %d: %s\n", i, strings.Join(tier, ", "))
}
for _, m := range d.Plan.Machines {
var parts []string
if len(m.Receives) > 0 {
parts = append(parts, "receives "+strings.Join(m.Receives, ", "))
}
if len(m.Waits) > 0 {
parts = append(parts, "waits for a person: "+strings.Join(m.Waits, ", "))
}
fmt.Fprintf(&b, "- %s: %s\n", m.Machine, strings.Join(parts, "; "))
}
for _, s := range d.Plan.Steps {
fmt.Fprintf(&b, "- %s\n", s)
}
b.WriteString("\n")
} }
if g != nil { if g != nil {
fmt.Fprintf(&b, "**Group** `%s`: %s\n", g.ID, strings.Join(g.Members, ", ")) fmt.Fprintf(&b, "**Group** `%s`: %s\n", g.ID, strings.Join(g.Members, ", "))
@@ -218,17 +134,6 @@ func ViewBody(d *Delivery, g *Group) string {
} }
b.WriteString("\n") b.WriteString("\n")
} }
if len(d.Steps) > 0 {
b.WriteString("**Machines**\n")
for _, s := range d.Steps {
fmt.Fprintf(&b, "- %s on %s: %s", s.Module, s.Machine, s.State)
if s.Why != "" {
fmt.Fprintf(&b, " — %s", s.Why)
}
b.WriteString("\n")
}
b.WriteString("\n")
}
b.WriteString("**Transitions**\n") b.WriteString("**Transitions**\n")
from := 0 from := 0
if len(d.Transitions) > 12 { if len(d.Transitions) > 12 {
@@ -237,7 +142,7 @@ func ViewBody(d *Delivery, g *Group) string {
for _, t := range d.Transitions[from:] { for _, t := range d.Transitions[from:] {
fmt.Fprintf(&b, "- %s %s → %s (%s): %s\n", t.At.UTC().Format("2006-01-02 15:04"), orNothing(t.From), t.To, t.Event, t.Why) fmt.Fprintf(&b, "- %s %s → %s (%s): %s\n", t.At.UTC().Format("2006-01-02 15:04"), orNothing(t.From), t.To, t.Event, t.Why)
} }
b.WriteString("\nThe commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.\n") b.WriteString("\nThe commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.\n")
return b.String() return b.String()
} }
@@ -0,0 +1,62 @@
package main
import (
"strings"
"testing"
"time"
)
// R6-C4 (ADR 0539 §17): the delivery's status and view say the merge's six states and nothing after the merge: no delivery plan,
// no machine step, no covered or walk line; a merged delivery's view names the source revision its merge moved.
func TestStatusAndViewSayEachOfTheSixStatesWithNoPlanAndNoMachineStep(t *testing.T) {
at := time.Date(2026, 10, 24, 12, 0, 0, 0, time.UTC)
base := Delivery{ID: "novox/app@c0ffee000000", Repository: "novox/app", Commit: "c0ffee000000", Number: 7, Since: at,
Transitions: []Transition{{At: at, From: Proposed, To: Rejected, Event: EvRefused, Why: "the gate said fail"}}}
for _, c := range []struct {
state State
mutate func(d *Delivery)
status string
desc string
inView []string
keepOut []string
}{
{Proposed, nil, "pending", "checking", nil, nil},
{Ready, nil, "success", "ready: it is merged on the Approve", nil, nil},
{Rejected, nil, "failure", "rejected: the gate said fail", nil, nil},
{Merged, func(d *Delivery) { d.MergedAs = "bbbbbbbbbbbb1111" }, "success",
"merged: source.main moved to novox/app@bbbbbbbbbbbb", []string{"**Source**", "source.main moved to novox/app@bbbbbbbbbbbb", "mesh-delivery-view"}, nil},
{Held, func(d *Delivery) { d.MergedAs, d.HeldWhy = "bbbbbbbbbbbb1111", "merged without a passing check" }, "pending",
"held for a person: merged without a passing check", []string{"Held for a person: merged without a passing check"}, nil},
{Closed, nil, "warning", "closed: the gate said fail", nil, nil},
} {
d := base
d.State = c.state
if c.mutate != nil {
c.mutate(&d)
}
state, desc := statusOf(&d)
if state != c.status || desc != c.desc {
t.Errorf("%s: the status is %s %q, wanted %s %q", c.state, state, desc, c.status, c.desc)
}
view := ViewBody(&d, nil)
if !strings.Contains(view, "**"+string(c.state)+"**") {
t.Errorf("%s: the view does not say the state: %s", c.state, view)
}
for _, want := range c.inView {
if !strings.Contains(view, want) {
t.Errorf("%s: the view lacks %q: %s", c.state, want, view)
}
}
for _, never := range []string{"**Delivery plan**", "**Machines**", "tier ", "receives", "walk", "Covered", "plans go", "mesh-delivery.release"} {
if strings.Contains(view, never) {
t.Errorf("%s: the view still says %q: %s", c.state, never, view)
}
}
}
// A merge not on the trunk names no source revision.
d := base
d.State = Merged
if s := d.sourceSaid(); s != "not on the trunk" {
t.Errorf("a merged delivery with no merge commit says %q", s)
}
}
@@ -42,9 +42,7 @@ func (h *Holder) takeForgeWanted() bool {
// AnswersChanged takes a change of the teller's answers: the answers to the asks pending are read at the look it asks for. // AnswersChanged takes a change of the teller's answers: the answers to the asks pending are read at the look it asks for.
func (h *Holder) AnswersChanged() { func (h *Holder) AnswersChanged() {
if h.Holding() { h.Poke()
h.Poke()
}
} }
// CommitStatus takes a commit status set on the forge: one of the merge checks (mesh/check and, on a // CommitStatus takes a commit status set on the forge: one of the merge checks (mesh/check and, on a
@@ -69,7 +67,9 @@ func (h *Holder) CommitStatus(e ForgeStatusEvent) {
h.takeCheckStatus(e.Owner+"/"+e.Repo, e.SHA, e.State, e.Description) h.takeCheckStatus(e.Owner+"/"+e.Repo, e.SHA, e.State, e.Description)
} }
if e.State != "success" { if e.State != "success" {
h.withdrawForStatus(e) if h.checkGates() {
h.withdrawForStatus(e) // while the check gates no merge, its status is shown and takes back no ask
}
return return
} }
repository := strings.ToLower(e.Owner + "/" + e.Repo) repository := strings.ToLower(e.Owner + "/" + e.Repo)
@@ -19,7 +19,7 @@ func hear(t *testing.T, w *world, key string, body any) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if key == ControllerSeat+".checked" { if key == shimCheckedKey {
// The gate follows no verdict of the controller any more (issue 1318): a test says it as the checker's. // The gate follows no verdict of the controller any more (issue 1318): a test says it as the checker's.
var c CheckedEvent var c CheckedEvent
if err := json.Unmarshal(raw, &c); err != nil { if err := json.Unmarshal(raw, &c); err != nil {
@@ -47,7 +47,7 @@ func TestAPassingVerdictAsksTheOperatorAtOnceWithNoMark(t *testing.T) {
if w.op.count() != 0 { if w.op.count() != 0 {
t.Fatalf("asked before the checks passed: %d", w.op.count()) t.Fatalf("asked before the checks passed: %d", w.op.count())
} }
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
if w.op.count() != 1 { if w.op.count() != 1 {
t.Fatalf("a pull request whose checks passed asked the operator %d times, wanted once, with no mark", w.op.count()) t.Fatalf("a pull request whose checks passed asked the operator %d times, wanted once, with no mark", w.op.count())
} }
@@ -63,7 +63,7 @@ func TestAPassingVerdictAsksTheOperatorAtOnceWithNoMark(t *testing.T) {
func TestAFailedVerdictAsksNothing(t *testing.T) { func TestAFailedVerdictAsksNothing(t *testing.T) {
w := announced(t) w := announced(t)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "fail", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "fail", "t1t1t1t1t1t1"))
if w.op.count() != 0 { if w.op.count() != 0 {
t.Fatalf("a pull request whose checks failed asked the operator %d times", w.op.count()) t.Fatalf("a pull request whose checks failed asked the operator %d times", w.op.count())
} }
@@ -78,7 +78,7 @@ func approve(w *world) {
func TestAnApproveHeardAsAnEventMergesAtOnceAtTheApprovedHead(t *testing.T) { func TestAnApproveHeardAsAnEventMergesAtOnceAtTheApprovedHead(t *testing.T) {
w := announced(t) w := announced(t)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
if len(w.op.open()) != 1 || len(w.forge.merges) != 0 { if len(w.op.open()) != 1 || len(w.forge.merges) != 0 {
t.Fatalf("asks %v, merges %v", w.op.open(), w.forge.merges) t.Fatalf("asks %v, merges %v", w.op.open(), w.forge.merges)
} }
@@ -95,7 +95,7 @@ func TestAnApproveHeardAsAnEventMergesAtOnceAtTheApprovedHead(t *testing.T) {
func TestAPassingVerdictAsksTheMergeAtOnceEvenJustAfterARefusal(t *testing.T) { func TestAPassingVerdictAsksTheMergeAtOnceEvenJustAfterARefusal(t *testing.T) {
w := announced(t) w := announced(t)
id := IDOf("novox/app", readyHead) id := IDOf("novox/app", readyHead)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
w.forge.refuse = errAskedAgain w.forge.refuse = errAskedAgain
approve(w) approve(w)
if len(w.forge.merges) != 1 { if len(w.forge.merges) != 1 {
@@ -104,7 +104,7 @@ func TestAPassingVerdictAsksTheMergeAtOnceEvenJustAfterARefusal(t *testing.T) {
w.forge.refuse = nil w.forge.refuse = nil
w.h.PullMerged(merged("novox/app", 9, "9999999999999999", "t2t2t2t2t2t2")) w.h.PullMerged(merged("novox/app", 9, "9999999999999999", "t2t2t2t2t2t2"))
w.h.PullUpdated(pr("novox/app", 7, readyHead, "feat/x", "src/a.go")) w.h.PullUpdated(pr("novox/app", 7, readyHead, "feat/x", "src/a.go"))
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t2t2t2t2t2t2")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t2t2t2t2t2t2"))
if len(w.forge.merges) != 2 || w.delivery(id).Ready.Merged == nil { if len(w.forge.merges) != 2 || w.delivery(id).Ready.Merged == nil {
t.Fatalf("the passing verdict did not ask the merge at once: %v %+v", w.forge.merges, w.delivery(id).Ready) t.Fatalf("the passing verdict did not ask the merge at once: %v %+v", w.forge.merges, w.delivery(id).Ready)
} }
@@ -130,7 +130,7 @@ func TestAHeadNotReviewedAtThisHeadIsNotAsked(t *testing.T) {
p := pr("novox/app", 7, readyHead, "feat/x", "src/a.go") p := pr("novox/app", 7, readyHead, "feat/x", "src/a.go")
p.Body = c.body p.Body = c.body
w.h.PullUpdated(p) w.h.PullUpdated(p)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
if w.op.count() != 0 { if w.op.count() != 0 {
t.Fatalf("an unreviewed pull request was asked: %d", w.op.count()) t.Fatalf("an unreviewed pull request was asked: %d", w.op.count())
} }
@@ -144,7 +144,7 @@ func TestTheAskNamesTheReviewerOfTheHead(t *testing.T) {
p := pr("novox/app", 7, readyHead, "feat/x", "src/a.go") p := pr("novox/app", 7, readyHead, "feat/x", "src/a.go")
p.Body = "Fixes issue 7.\n\nReviewed-by: lead@shanks at " + readyHead + "\n" p.Body = "Fixes issue 7.\n\nReviewed-by: lead@shanks at " + readyHead + "\n"
w.h.PullUpdated(p) w.h.PullUpdated(p)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
a := w.op.ask(w.op.open()[0]) a := w.op.ask(w.op.open()[0])
if strings.Contains(a.Explanation, "not reviewed") || !strings.Contains(a.Explanation, "reviewed by lead") { if strings.Contains(a.Explanation, "not reviewed") || !strings.Contains(a.Explanation, "reviewed by lead") {
t.Fatalf("a reviewed head is not said reviewed:\n%s", a.Explanation) t.Fatalf("a reviewed head is not said reviewed:\n%s", a.Explanation)
@@ -167,7 +167,7 @@ func TestTheReviewLineCorrectedInTheDescriptionIsReadWithoutAPush(t *testing.T)
// Issue 1022 (ADR 0421 §6): a push ends the Approve and says so once on the pull request. // Issue 1022 (ADR 0421 §6): a push ends the Approve and says so once on the pull request.
func TestAPushAfterTheAskEndsItAndSaysSoOnceOnThePullRequest(t *testing.T) { func TestAPushAfterTheAskEndsItAndSaysSoOnceOnThePullRequest(t *testing.T) {
w := announced(t) w := announced(t)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
old := w.op.open()[0] old := w.op.open()[0]
w.h.PullUpdated(pr("novox/app", 7, pushed, "feat/x", "src/a.go")) w.h.PullUpdated(pr("novox/app", 7, pushed, "feat/x", "src/a.go"))
w.h.React() w.h.React()
@@ -180,7 +180,7 @@ func TestAPushAfterTheAskEndsItAndSaysSoOnceOnThePullRequest(t *testing.T) {
t.Fatalf("the end of the Approve is not said once on the pull request: %v", w.forge.comments) t.Fatalf("the end of the Approve is not said once on the pull request: %v", w.forge.comments)
} }
// The new head is asked on its own when its checks pass. // The new head is asked on its own when its checks pass.
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, pushed, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, pushed, "pass", "t1t1t1t1t1t1"))
if open := w.op.open(); len(open) != 1 || open[0] == old { if open := w.op.open(); len(open) != 1 || open[0] == old {
t.Fatalf("the new head was not asked afresh: %v", open) t.Fatalf("the new head was not asked afresh: %v", open)
} }
@@ -63,7 +63,7 @@ func (s *memStore) Deliveries() ([]*Delivery, error) {
defer s.mu.Unlock() defer s.mu.Unlock()
var out []*Delivery var out []*Delivery
for key, raw := range s.deliveries { for key, raw := range s.deliveries {
if key == carriedKey { // the gate's mark that today's state was carried (seat.go), as the real store skips it if key == carriedKey { // the mark an earlier owner's carry-over wrote, as the real store skips it
continue continue
} }
var d Delivery var d Delivery
@@ -88,299 +88,6 @@ func (s *memStore) Groups() ([]*Group, error) {
return out, nil return out, nil
} }
type fakeController struct {
mu sync.Mutex
walks map[string]Walk
held bool
checks []string
asked int
delivers []string
// backupsAgain are the walks whose failed backups were asked again.
backupsAgain []string
// refuseBackupsAgain refuses the word to take a walk's backups again.
refuseBackupsAgain bool
stops []string
// withouts are the stops that left merges out (novox/hq ADR 0299): "<walk> without <merge>,…".
withouts []string
// refuseWithout refuses every stop with merges to leave out, as a controller before ADR 0299 does; mergedAt are
// the merges' times, by commit, for what a later merge contains.
refuseWithout bool
mergedAt map[string]time.Time
// refuseStop refuses every whole stop too; moves are the modules each merge moves, by commit, for what a left-out
// merge shares (ADR 0299).
refuseStop bool
moves map[string][]string
plans int
down bool
// emptyCheck makes Check answer with no id and no error, as the port does when the answer holds none.
emptyCheck bool
// walkReads counts the reads of the controller's walks (Walks).
walkReads int
// checkReady is the ready flag of each member of each check asked.
checkReady []bool
// baseSHAs is the base head each member of each check asked was sent with.
baseSHAs []string
// refuseDeliver refuses every word, as a controller that will not take it does (review of novox/hq issue 753).
refuseDeliver bool
// onCheck runs as a check is asked, before it is answered: what the forge held at that moment.
onCheck func()
// planned is what Plan answers instead of "builds app", when set.
planned *DeliveryPlan
// forgot are the pull requests whose check builds were let go of (novox/hq ADR 0350 §5), "owner/repo#n".
forgot []string
}
// BackupsAgain records the word, and clears the walk's failed backups as the controller does.
func (c *fakeController) BackupsAgain(walk, why string) error {
c.mu.Lock()
defer c.mu.Unlock()
if c.down {
return errors.New("down")
}
if c.refuseBackupsAgain {
return errors.New("the controller refuses the backups")
}
w, known := c.walks[walk]
if !known || w.Backups == nil || w.Backups.State != "failed" {
return fmt.Errorf("%s holds no failed backup", walk)
}
w.Backups = nil
w.Revision++
c.walks[walk] = w
c.backupsAgain = append(c.backupsAgain, walk)
return nil
}
func (c *fakeController) ForgetChecks(owner, repo string, number int) error {
c.mu.Lock()
defer c.mu.Unlock()
if c.down {
return errors.New("down")
}
c.forgot = append(c.forgot, fmt.Sprintf("%s/%s#%d", owner, repo, number))
return nil
}
func newFakeController() *fakeController {
return &fakeController{walks: map[string]Walk{}, held: true, mergedAt: map[string]time.Time{},
moves: map[string][]string{}}
}
func (c *fakeController) Plan(repository, base, head string, paths, dirs, removed []string) (*DeliveryPlan, error) {
c.mu.Lock()
defer c.mu.Unlock()
c.plans++
if c.down {
return nil, errors.New("down")
}
if c.planned != nil {
p := *c.planned
return &p, nil
}
return &DeliveryPlan{Repository: repository, Base: base, Head: head, Moved: []string{"app"}, Summary: "builds app"}, nil
}
func (c *fakeController) Deliver(walk, why string) error {
c.mu.Lock()
defer c.mu.Unlock()
if c.down {
return errors.New("down")
}
if c.refuseDeliver {
return errors.New("the controller refuses the word")
}
w, known := c.walks[walk]
if !known || w.Delivery == nil || w.Delivery.Go != nil {
return fmt.Errorf("%s waits for nobody", walk)
}
now := time.Now()
w.Delivery.Go, w.Delivery.By, w.Delivery.Why = &now, byOwner, why
w.Revision++
c.walks[walk] = w
c.delivers = append(c.delivers, walk)
return nil
}
func (c *fakeController) Stop(walk, why, by string) error {
c.mu.Lock()
defer c.mu.Unlock()
if c.down {
return errors.New("down")
}
c.stops = append(c.stops, walk)
if c.refuseStop {
return errors.New("the controller refuses")
}
if w, known := c.walks[walk]; known && !w.Open() {
// As the controller: only a walk that is building or rolling is stopped (`delivery-stop`).
return fmt.Errorf("%s is already %s", walk, w.State)
} else if known {
w.State = walkFailed
if w.Delivery == nil {
w.Delivery = &WalkDelivery{}
}
w.Delivery.Stopped, w.Delivery.StoppedWhy = by, why
w.Revision++
c.walks[walk] = w
}
return nil
}
// StopWithout ends a walk without the merges named, as the controller does (novox/hq ADR 0299): each merge named,
// and each later merge of its repository by merge time (mergedAt), is left out; the others are put in a batch,
// "<walk>-again", assembling until a test cuts it. Given a batch, the merges left out go to a stopped record,
// "<walk>-left", and the batch goes on with the others. refuseWithout refuses every such stop, as an older
// controller does.
func (c *fakeController) StopWithout(walk, why, by string, without []string) (StopAnswer, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.down {
return StopAnswer{}, errors.New("down")
}
c.withouts = append(c.withouts, walk+" without "+strings.Join(without, ","))
if c.refuseWithout {
return StopAnswer{}, errors.New(`delivery-stop takes no argument "without"`)
}
w, known := c.walks[walk]
if !known || !(w.Open() || w.IsBatch()) {
return StopAnswer{}, fmt.Errorf("%s is no open walk or batch", walk)
}
wd := WalkDelivery{}
if w.Delivery != nil {
wd = *w.Delivery
}
named := func(m WalkMerge) bool { return containsString(without, m.Repository+"@"+m.Commit) }
contains := func(m WalkMerge) string {
for _, s := range wd.Merges {
if named(s) && s.Repository == m.Repository && c.mergedAt[m.Commit].After(c.mergedAt[s.Commit]) {
return s.Commit
}
}
return ""
}
shares := func(m WalkMerge) (string, []string) {
for _, s := range wd.Merges {
if named(s) {
var both []string
for _, x := range c.moves[m.Commit] {
if containsString(c.moves[s.Commit], x) {
both = append(both, x)
}
}
if len(both) > 0 {
return s.Commit, both
}
}
}
return "", nil
}
a := StopAnswer{Stopped: walk}
var again, left []WalkMerge
for _, m := range wd.Merges {
s, both := shares(m)
switch {
case named(m):
a.LeftOut = append(a.LeftOut, WalkLeftOut{Repository: m.Repository, Commit: m.Commit})
left = append(left, m)
case len(both) > 0:
a.LeftOut = append(a.LeftOut, WalkLeftOut{Repository: m.Repository, Commit: m.Commit, Contains: s, Shares: both})
left = append(left, m)
case contains(m) != "":
a.LeftOut = append(a.LeftOut, WalkLeftOut{Repository: m.Repository, Commit: m.Commit, Contains: contains(m)})
left = append(left, m)
default:
m.Carried = ""
again = append(again, m)
a.WalkedAgain = append(a.WalkedAgain, m.Repository+"@"+m.Commit)
}
}
if len(a.LeftOut) == 0 {
return StopAnswer{}, fmt.Errorf("%s answers no merge %s", walk, strings.Join(without, ", "))
}
if w.IsBatch() && len(again) == 0 {
// Every merge left out: the batch itself is stopped, as the controller does.
wd.Stopped, wd.StoppedWhy, wd.LeftOut, wd.Batch = by, why, a.LeftOut, nil
w.Delivery, w.State = &wd, walkFailed
w.Revision++
c.walks[walk] = w
return a, nil
}
if w.IsBatch() {
rid := walk + "-left"
c.walks[rid] = Walk{ID: rid, Repository: left[0].Repository, Branch: "main", Commit: left[0].Commit,
Created: w.Created, State: walkFailed, Revision: 1, Modules: map[string]*WalkModule{},
Delivery: &WalkDelivery{Stopped: by, StoppedWhy: why, Merges: left, LeftOut: a.LeftOut, WalkedAgainBy: walk}}
wd.Merges = again
w.Delivery = &wd
w.Repository, w.Commit = again[0].Repository, again[0].Commit
w.Revision++
c.walks[walk] = w
a.Stopped, a.WalkedAgainBy = rid, walk
return a, nil
}
if len(again) > 0 {
id := walk + "-again"
closes := w.Created.Add(time.Minute)
c.walks[id] = Walk{ID: id, Repository: again[0].Repository, Branch: "main", Commit: again[0].Commit,
Created: w.Created.Add(time.Minute), State: walkAssembling, Revision: 1,
Modules: map[string]*WalkModule{}, Delivery: &WalkDelivery{Merges: again, Batch: &WalkBatch{ClosesAt: &closes}}}
wd.WalkedAgainBy, a.WalkedAgainBy = id, id
}
wd.Stopped, wd.StoppedWhy, wd.LeftOut = by, why, a.LeftOut
w.Delivery = &wd
w.State = walkFailed
w.Note = "stopped by " + by + ": " + why
w.Revision++
c.walks[walk] = w
return a, nil
}
// cut makes a batch the walk under the same id, waiting for its word, as the controller cuts it.
func (c *fakeController) cut(id string) Walk {
c.mu.Lock()
w := c.walks[id]
c.mu.Unlock()
wd := *w.Delivery
wd.Batch, wd.Awaits = nil, "mesh-delivery"
w.Delivery = &wd
w.State, w.Tiers, w.Modules = walkBuilding, [][]string{{"app"}}, map[string]*WalkModule{"app": {}}
return c.put(w)
}
func (c *fakeController) Walks(walk string) (bool, []Walk, error) {
c.mu.Lock()
defer c.mu.Unlock()
c.walkReads++
if c.down {
return false, nil, errors.New("down")
}
var out []Walk
for id, w := range c.walks {
if walk == "" || walk == id {
out = append(out, w)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return c.held, out, nil
}
// put is the controller keeping a walk; it answers the walk as `plan-moved` would say it.
func (c *fakeController) put(w Walk) Walk {
c.mu.Lock()
defer c.mu.Unlock()
if old, known := c.walks[w.ID]; known {
w.Revision = old.Revision + 1
} else if w.Revision == 0 {
w.Revision = 1
}
c.walks[w.ID] = w
return w
}
func (c *fakeController) walk(id string) Walk {
c.mu.Lock()
defer c.mu.Unlock()
return c.walks[id]
}
type fakeForge struct { type fakeForge struct {
mu sync.Mutex mu sync.Mutex
notes map[string][]string notes map[string][]string
@@ -569,7 +276,6 @@ type world struct {
t *testing.T t *testing.T
h *Holder h *Holder
store *memStore store *memStore
ctl *fakeController
forge *fakeForge forge *fakeForge
op *fakeOperator op *fakeOperator
merges *fakeMerges merges *fakeMerges
@@ -584,12 +290,12 @@ type world struct {
} }
func newWorld(t *testing.T) *world { func newWorld(t *testing.T) *world {
return newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)) return newWorldOver(t, newMemStore(), newFakeForge(), time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC))
} }
func newWorldOver(t *testing.T, store *memStore, ctl *fakeController, forge *fakeForge, now time.Time) *world { func newWorldOver(t *testing.T, store *memStore, forge *fakeForge, now time.Time) *world {
w := &world{t: t, store: store, ctl: ctl, forge: forge, now: now} w := &world{t: t, store: store, forge: forge, now: now}
w.h = &Holder{Store: store, Controller: ctl, Forge: forge, w.h = &Holder{Store: store, Forge: forge,
Now: func() time.Time { w.mu.Lock(); defer w.mu.Unlock(); return w.now }, Now: func() time.Time { w.mu.Lock(); defer w.mu.Unlock(); return w.now },
Logf: func(format string, a ...any) { t.Logf(format, a...); fmt.Fprintf(&w.logbuf, format+"\n", a...) }, Logf: func(format string, a ...any) { t.Logf(format, a...); fmt.Fprintf(&w.logbuf, format+"\n", a...) },
Emit: func(event string, body any) error { Emit: func(event string, body any) error {
@@ -667,7 +373,7 @@ func pr(repo string, number int, sha, branch string, paths ...string) PullEvent
func verdict(repo string, number int, sha, gate string) CheckedEvent { func verdict(repo string, number int, sha, gate string) CheckedEvent {
owner, name, _ := strings.Cut(repo, "/") owner, name, _ := strings.Cut(repo, "/")
c := CheckedEvent{Owner: owner, Repo: name, Number: number, Commit: sha, Verdict: gate, Summary: "the gate said " + gate, c := CheckedEvent{Owner: owner, Repo: name, Number: number, Commit: sha, Verdict: gate, Summary: "the gate said " + gate,
ID: "check-of-" + sha, Plan: &DeliveryPlan{Repository: repo, Moved: []string{"app"}, Summary: "builds app → anchor"}} ID: "check-of-" + sha}
return c return c
} }
@@ -679,16 +385,6 @@ func merged(repo string, number int, head, merge string) PullEvent {
return p return p
} }
// aWalk is the controller's walk of a merge commit, waiting for its delivery's word.
func aWalk(id, repo, commit string, waits bool) Walk {
w := Walk{ID: id, Repository: repo, Branch: "main", Commit: commit, Created: time.Date(2026, 10, 6, 12, 31, 0, 0, time.UTC),
State: walkBuilding, Tiers: [][]string{{"app"}}, Modules: map[string]*WalkModule{"app": {}}}
if waits {
w.Delivery = &WalkDelivery{Awaits: "mesh-delivery"}
}
return w
}
// Comment records what was asked (novox/hq issue 722). // Comment records what was asked (novox/hq issue 722).
func (f *fakeForge) Comment(owner, repo string, number int, body string) error { func (f *fakeForge) Comment(owner, repo string, number int, body string) error {
f.mu.Lock() f.mu.Lock()
@@ -63,7 +63,8 @@ func (h *Holder) unmergedLive(g *Group) []*Delivery {
// groupMergeable reads every member's required statuses at the head the Approve covers, and says whether all are a success. // groupMergeable reads every member's required statuses at the head the Approve covers, and says whether all are a success.
// When one is not, or cannot be read, nothing is handed to the bridge: the group is held, said urgent (ADR 0441 §2, §4), and // When one is not, or cannot be read, nothing is handed to the bridge: the group is held, said urgent (ADR 0441 §2, §4), and
// the target's turn is given back so the next event that decides whether it may merge asks it again. Outside the lock. // the target's turn is given back so the next event that decides whether it may merge asks it again. A read that could not
// be made is a failed ask, tried again after askAgainAfter. Outside the lock.
func (h *Holder) groupMergeable(gid, target string, reads []memberRead) bool { func (h *Holder) groupMergeable(gid, target string, reads []memberRead) bool {
release := func() { release := func() {
if d := h.deliveries[target]; d != nil && d.Ready != nil { if d := h.deliveries[target]; d != nil && d.Ready != nil {
@@ -98,8 +99,9 @@ func (h *Holder) groupMergeable(gid, target string, reads []memberRead) bool {
g.Hold = &GroupHold{Kind: "blocked", Member: m.id, Says: says, Since: since} g.Hold = &GroupHold{Kind: "blocked", Member: m.id, Says: says, Since: since}
h.keepGroup(g) h.keepGroup(g)
} }
// No timer: the status turning success is an event on the forge (CommitStatus, mergeAgainNow), and it asks the
// merge again at once (issue 1275: a pass that stops part way resumes on an event).
release() release()
h.again("a group's merge held for a required status")
h.mu.Unlock() h.mu.Unlock()
return false return false
} }
@@ -112,7 +114,7 @@ func (h *Holder) groupMergeable(gid, target string, reads []memberRead) bool {
return true return true
} }
// groupLines are the groups that are held or half merged, as urgent stalled lines the controller raises as conditions: // groupLines are the groups that are held or half merged, as urgent stalled lines the gate says as conditions (conditions.go):
// one for each, on its first member not merged, until every member is merged or a person decides (issue 1275). A member // one for each, on its first member not merged, until every member is merged or a person decides (issue 1275). A member
// the merge tool refused is said by readyLines, with its group; this adds a group held for a required status, and a // the merge tool refused is said by readyLines, with its group; this adds a group held for a required status, and a
// group half merged with no refusal. Under the lock. // group half merged with no refusal. Under the lock.
@@ -141,12 +143,16 @@ func (h *Holder) groupLines(now time.Time) []StalledLine {
switch { switch {
// A held member the merge tool refused is said by readyLines already: one line for an id, not two. // A held member the merge tool refused is said by readyLines already: one line for an id, not two.
case g.Hold != nil && g.Hold.Kind == "blocked" && !running && !(heldOn != nil && heldOn.Ready != nil && heldOn.Ready.RefusedSince != nil): case g.Hold != nil && g.Hold.Kind == "blocked" && !running && !(heldOn != nil && heldOn.Ready != nil && heldOn.Ready.RefusedSince != nil):
stands := "is approved and nothing is merged"
if merged > 0 {
stands = "is half merged" // a member merged before the pass stopped (issue 1275)
}
out = append(out, StalledLine{ID: g.Hold.Member, Number: numberOf(h.deliveries[g.Hold.Member]), State: first.State, out = append(out, StalledLine{ID: g.Hold.Member, Number: numberOf(h.deliveries[g.Hold.Member]), State: first.State,
For: now.Sub(g.Hold.Since).Round(time.Second).String(), Bound: "0s", Severity: "urgent", For: now.Sub(g.Hold.Since).Round(time.Second).String(), Bound: "0s", Severity: "urgent",
H2: "none: the required status turns success, which asks the merge again — or the operator's decision", H2: "none: the required status turns success, which asks the merge again — or the operator's decision",
Says: fmt.Sprintf("group %s is approved and nothing is merged (merged %d of %d): %s waits on a required status "+ Says: fmt.Sprintf("group %s %s (merged %d of %d): %s waits on a required status "+
"at its head: %s. Needed: that status turns success (its check asked again), or the operator decides.", "at its head: %s. Needed: that status turns success (its check asked again), or the operator decides.",
g.ID, merged, len(h.membersOf(g)), g.Hold.Member, g.Hold.Says)}) g.ID, stands, merged, len(h.membersOf(g)), g.Hold.Member, g.Hold.Says)})
case merged > 0 && !refused && !running: case merged > 0 && !refused && !running:
out = append(out, StalledLine{ID: first.ID, Number: first.Number, State: first.State, out = append(out, StalledLine{ID: first.ID, Number: first.Number, State: first.State,
For: now.Sub(g.Updated).Round(time.Second).String(), Bound: "0s", Severity: "urgent", For: now.Sub(g.Updated).Round(time.Second).String(), Bound: "0s", Severity: "urgent",
@@ -21,25 +21,44 @@ const (
// approvedGroup is a group of two pull requests on one branch, checks green, composed green and the operator's Approve // approvedGroup is a group of two pull requests on one branch, checks green, composed green and the operator's Approve
// asked and not yet answered (w.op.auto is off). // asked and not yet answered (w.op.auto is off).
func approvedGroup(t *testing.T) *world { func approvedGroup(t *testing.T) *world {
return approvedGroupOf(t, member{gCtl, 1, gHc, "internal/a.go"}, member{gCat, 2, gHk, "modules/app/module.json"})
}
// member is a pull request of a group in feat/g: its repository, number, head and the paths it changes.
type member struct {
repo string
number int
head string
paths string
}
// approvedGroupOf is a group feat/g of these members, each checked, composed together and asked once.
func approvedGroupOf(t *testing.T, ms ...member) *world {
t.Helper()
w := newWorld(t) w := newWorld(t)
w.op.auto = false w.op.auto = false
w.h.PullUpdated(pr(gCtl, 1, gHc, "feat/g", "internal/a.go")) for _, m := range ms {
w.h.PullUpdated(pr(gCat, 2, gHk, "feat/g", "modules/app/module.json")) w.h.PullUpdated(pr(m.repo, m.number, m.head, "feat/g", strings.Split(m.paths, ",")...))
w.h.Checked(judged(gCtl, 1, gHc, "pass", gTrunk0)) }
w.h.Checked(judged(gCat, 2, gHk, "pass", gTrunk0)) for _, m := range ms {
w.h.Checked(judged(m.repo, m.number, m.head, "pass", gTrunk0))
}
w.settleAll() w.settleAll()
g := w.h.groups["feat/g"] g := w.h.groups["feat/g"]
if g == nil || g.Check == nil { if g == nil || g.Check == nil {
t.Fatalf("no composed check was asked: %+v", g) t.Fatalf("no composed check was asked: %+v", g)
} }
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed", c := CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed"}
Members: []struct { for _, m := range ms {
owner, repo, _ := strings.Cut(m.repo, "/")
c.Members = append(c.Members, struct {
Owner string `json:"owner"` Owner string `json:"owner"`
Repo string `json:"repo"` Repo string `json:"repo"`
Commit string `json:"commit"` Commit string `json:"commit"`
Trunk string `json:"trunk,omitempty"` Trunk string `json:"trunk,omitempty"`
}{{Owner: "novox", Repo: "mesh-controller", Commit: gHc, Trunk: gTrunk0[:12]}, }{Owner: owner, Repo: repo, Commit: m.head, Trunk: gTrunk0[:12]})
{Owner: "novox", Repo: "mesh-catalog", Commit: gHk, Trunk: gTrunk0[:12]}}}) }
w.h.Checked(c)
w.settleAll() w.settleAll()
if w.op.count() != 1 || len(w.op.open()) != 1 { if w.op.count() != 1 || len(w.op.open()) != 1 {
t.Fatalf("the green group is asked once: %d asks, open %v", w.op.count(), w.op.open()) t.Fatalf("the green group is asked once: %d asks, open %v", w.op.count(), w.op.open())
@@ -107,15 +126,3 @@ func TestARefusalLoopInAClosedGroupIsSaidAtOnce(t *testing.T) {
// check set its gate (issue 1049: the forge's "not a success: mesh/repo-check (pending)"). // check set its gate (issue 1049: the forge's "not a success: mesh/repo-check (pending)").
var ownPending = errors.New("the forge refused gitea_merge_pull_request: novox/mesh-controller#1 is not merged: main requires " + var ownPending = errors.New("the forge refused gitea_merge_pull_request: novox/mesh-controller#1 is not merged: main requires " +
"mesh/merge-gate, mesh/repo-check; not a success: mesh/repo-check (pending). A pull request merges once its checks succeed") "mesh/merge-gate, mesh/repo-check; not a success: mesh/repo-check (pending). A pull request merges once its checks succeed")
// ownChecks are the checks asked from index on that were a head's own (no group), not a group's composed check.
func ownChecks(w *world, from int) (own, composed []string) {
for _, c := range w.ctl.checks[from:] {
if strings.HasPrefix(c, ":") {
own = append(own, c)
} else {
composed = append(composed, c)
}
}
return
}
@@ -0,0 +1,29 @@
package main
import "time"
var t0 = time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
// mergedAt is the forge's `pull.merged` at a moment.
func mergedAt(p PullEvent, merge string, at time.Time) PullEvent {
p.MergeCommit = merge
p.MergedAt = at.UTC().Format(time.RFC3339)
return p
}
// verdictMoving is a passing verdict. The gate plans no delivery any more (ADR 0539 §17), so it names no module.
func verdictMoving(repo string, number int, sha string, _ ...string) CheckedEvent {
return verdict(repo, number, sha, "pass")
}
// ungated is the setting check-gates-merge false: mesh/check gates no merge (the operator, 2026-10-25 22:56).
func ungated(w *world) {
off := false
w.h.Settings = func() Settings { return Settings{CheckGatesMerge: &off} }
}
func statusOn(w *world, repo, commit string) string {
w.forge.mu.Lock()
defer w.forge.mu.Unlock()
return w.forge.statuses[repo+"@"+commit+" mesh/delivery"]
}
@@ -1,227 +0,0 @@
package main
import (
"errors"
sdk "git.novox.be/novox/mesh-sdk/go/delivery"
"strings"
"testing"
"time"
)
// History (novox/hq issue 313): merges long past, heard only from the forge's word when this owner first held
// the seat, reached the machines before it existed. They are retired — never held for a person, released or
// delivered — and what is owed for a pull request the forge no longer holds is given up, not tried for ever.
// aPastMerge is the forge's `pull.merged` replayed from the bus's history: merged days before it is heard.
func aPastMerge(repo string, number int, head, merge string, at time.Time) PullEvent {
p := merged(repo, number, head, merge)
p.MergedAt = at.UTC().Format(time.RFC3339)
return p
}
func TestAMergeLongPastHeardFromTheForgesWordIsRetiredAtAdoption(t *testing.T) {
w := newWorld(t)
id := IDOf("novox/app", head)
w.h.PullMerged(aPastMerge("novox/app", 274, head, merge, w.now.Add(-4*24*time.Hour)))
w.settleAll()
d := w.delivery(id)
if d.State != Retired || !strings.Contains(d.History, "history") {
t.Fatalf("a merge four days past heard from the forge's word is %s (%q)", d.State, d.History)
}
if got := statesOf([]State{d.Transitions[0].To, d.Transitions[1].To}); strings.Join(got, ",") != "proposed,retired" ||
d.Transitions[1].Event != EvHistory {
t.Fatalf("transitions %+v", d.Transitions)
}
// Nothing is said on its pull request — the forge may hold another under its number — and nothing stalls.
if w.forge.viewCalls != 0 || len(w.forge.statuses) != 0 {
t.Fatalf("history was shown on a pull request: %d view(s), statuses %v", w.forge.viewCalls, w.forge.statuses)
}
if len(d.Owed) != 0 {
t.Fatalf("still owed: %+v", d.Owed)
}
if s := w.h.Stalled(); len(s) != 0 {
t.Fatalf("history stalls: %+v", s)
}
// Its record is still kept, on the commit's note.
if len(w.forge.notesOn(head)) == 0 {
t.Fatal("no note of it on its commit")
}
// A merge heard from the forge's word within moments of happening is a live one: held for a person.
live := IDOf("novox/app", "bbbbbbbbbbbb2222")
w.h.PullMerged(aPastMerge("novox/app", 9, "bbbbbbbbbbbb2222", "cccccccccccc3333", w.now.Add(-time.Minute)))
if s := w.state(live); s != Held {
t.Fatalf("a live merge from the forge's word is %s", s)
}
// A merge long past whose walk the controller still holds open is delivered by it, never history.
walked := IDOf("novox/app", "dddddddddddd4444")
w.h.WalkMoved(aWalk("plan-9", "novox/app", "eeeeeeeeeeee5555", true))
w.h.PullMerged(aPastMerge("novox/app", 10, "dddddddddddd4444", "eeeeeeeeeeee5555", w.now.Add(-48*time.Hour)))
if s := w.state(walked); s == Retired {
t.Fatal("a merge with an open walk was retired as history")
}
// A merge the forge announces under a number an older delivery already merged with is not that delivery.
w.h.PullMerged(aPastMerge("novox/app", 274, "", "ffffffffffff6666", w.now.Add(-time.Minute)))
if d := w.delivery(id); d.MergedAs != merge {
t.Fatalf("a reused number merged the old delivery again as %s", d.MergedAs)
}
}
// oldHistory is a held delivery as the first adoption left it: made from the forge's word, held, owing a view
// on a pull request the forge no longer holds.
func oldHistory(id, repo string, number int, created, mergedAt time.Time) *Delivery {
commit := strings.SplitN(id, "@", 2)[1] + "0000"
return &Delivery{ID: id, Repository: repo, Commit: commit, Number: number, Base: "main", MergedAs: commit,
MergedAt: mergedAt, Created: created, Since: created, State: Held,
HeldWhy: "merged without a passing check: only a person decides that it goes on",
Transitions: []Transition{
{At: created, To: Proposed, Event: EvAnnounced, Why: repo + "#274 merged; its head was not heard by this " +
"owner, so it is made from the forge's word"},
{At: created, From: Proposed, To: Held, Event: EvMergedUnchecked, Why: "merged without a passing check"}},
Owed: []Effect{{Kind: EffectView, Since: created, Tries: 11051,
Last: "Gitea API /repos/" + repo + "/issues/274/comments: 500 issue does not exist"}}}
}
func TestRetireHistoryRetiresTheAdoptedHistoryInBulkAndLeavesEveryOtherHeldOne(t *testing.T) {
store := newMemStore()
start := time.Date(2026, 10, 6, 23, 5, 27, 0, time.UTC)
for _, d := range []*Delivery{
oldHistory("novox/hq@018ee359ae52", "novox/hq", 274, start, start.Add(-4*24*time.Hour)),
oldHistory("novox/lab@a4f6bb8ce054", "novox/lab", 53, start, start.Add(-2*time.Hour)),
// Made from the forge's word, but heard a minute after its merge: a live merge, a person's.
oldHistory("novox/hq@7777777777aa", "novox/hq", 3, start, start.Add(-time.Minute)),
} {
if err := store.PutDelivery(d); err != nil {
t.Fatal(err)
}
}
w := newWorldOver(t, store, newFakeController(), newFakeForge(), start.Add(36*time.Hour))
// And one heard live: rejected, then merged — a person's to decide.
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "fail"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
if len(w.h.Stalled()) != 3 { // the three adopted at the switch; the one held just now is within its bound
t.Fatalf("stalled before: %+v", w.h.Stalled())
}
dry, err := w.h.RetireHistory("", "", "history adopted at the switch", aPerson, true)
if err != nil || dry.Retired != 2 || dry.LeftHeld != 2 || len(dry.Left) != 2 {
t.Fatalf("a dry retire: %+v %v", dry, err)
}
if s := w.state("novox/hq@018ee359ae52"); s != Held {
t.Fatalf("a dry retire moved a delivery to %s", s)
}
if _, err := w.h.RetireHistory("", "", "", aPerson, false); err == nil {
t.Fatal("a retire with no why was taken")
}
got, err := w.h.RetireHistory("", "", "history adopted at the switch", aPerson, false)
if err != nil || got.Retired != 2 || got.ByRepository["novox/hq"] != 1 || got.ByRepository["novox/lab"] != 1 {
t.Fatalf("retire: %+v %v", got, err)
}
for _, id := range []string{"novox/hq@018ee359ae52", "novox/lab@a4f6bb8ce054"} {
d := w.delivery(id)
if d.State != Retired || d.Transitions[len(d.Transitions)-1].By != aPerson {
t.Fatalf("%s is %s", id, d.State)
}
for _, e := range d.Owed {
if e.Kind == EffectView || e.Kind == EffectStatus {
t.Fatalf("%s still owes its pull request: %+v", id, e)
}
}
}
for _, id := range []string{"novox/hq@7777777777aa", IDOf("novox/app", head)} {
if s := w.state(id); s != Held {
t.Fatalf("%s, not history, is %s", id, s)
}
}
// Nothing was released, delivered or let go.
if len(w.ctl.delivers) != 0 || len(w.ctl.stops) != 0 {
t.Fatalf("a retire asked the controller: deliver %v stop %v", w.ctl.delivers, w.ctl.stops)
}
if s := w.h.Stalled(); len(s) != 1 || s[0].ID != "novox/hq@7777777777aa" {
t.Fatalf("stalled after: %+v", s)
}
// Retired is final: a second retire takes nothing, and nothing leaves it.
again, _ := w.h.RetireHistory("", "", "again", aPerson, false)
if again.Retired != 0 {
t.Fatalf("retired twice: %+v", again)
}
if _, err := w.h.Release("novox/hq@018ee359ae52", "go on", aPerson); err == nil {
t.Fatal("a retired delivery was released")
}
}
func TestRetireHistoryIsRefusedForARealHeldDelivery(t *testing.T) {
w := newWorld(t)
id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "fail"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
w.later(48 * time.Hour)
_, err := w.h.RetireHistory(id, "", "looks old", aPerson, false)
var refused ErrRefused
if !errors.As(err, &refused) || !strings.Contains(err.Error(), "heard its head") {
t.Fatalf("a held delivery whose head was heard was retired: %v", err)
}
if s := w.state(id); s != Held {
t.Fatalf("a refused retire left it %s", s)
}
// A delivery that is not held is refused too, by the table.
other := IDOf("novox/app", "bbbbbbbbbbbb2222")
w.h.PullUpdated(pr("novox/app", 8, "bbbbbbbbbbbb2222", "feat/y", "src/b.go"))
if _, err := w.h.RetireHistory(other, "", "old", aPerson, false); err == nil || w.state(other) != Proposed {
t.Fatalf("a proposed delivery was retired: %v", err)
}
if _, err := w.h.RetireHistory("novox/app@nothing", "", "old", aPerson, false); err == nil {
t.Fatal("retired a delivery that does not exist")
}
}
// What the bus cannot take is owed and tried again later each time, and given up after retryLimit tries, saying so. What the
// forge refuses is the bridge's to say (issue 1349): the gate has no retry for it.
func TestOwedWorkTheBusCannotTakeIsTriedAgainLaterAndGivenUpAfterRetryLimit(t *testing.T) {
w := newWorld(t)
id := IDOf("novox/app", head)
w.owed.down = true
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
tries := func() int {
for _, e := range w.delivery(id).Owed {
if e.Kind == EffectView {
return e.Tries
}
}
return -1
}
w.h.Flush()
w.h.Flush()
w.h.Flush()
if tries() != 1 {
t.Fatalf("a view the bus could not take was tried %d time(s) at once", tries())
}
w.later(retryFirst + time.Second)
w.h.Flush()
if tries() != 2 {
t.Fatalf("a failed view was not tried again after its wait: %d", tries())
}
if retryAfter(1) != retryFirst || retryAfter(2) != 2*retryFirst || retryAfter(100) != retryCap {
t.Fatalf("the waits: %s %s %s", retryAfter(1), retryAfter(2), retryAfter(100))
}
for range retryLimit {
w.later(retryCap)
w.h.Flush()
}
if tries() != -1 {
t.Fatalf("a view failing %d times is still owed", tries())
}
if st := w.h.Status("listening"); !strings.Contains(strings.Join(st["refused"].([]string), "\n"), "given up") {
t.Fatalf("its drop was not said: %v", st["refused"])
}
// The bus back: what is owed anew is written.
w.owed.down = false
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.Flush()
if len(w.owed.acts(sdk.OwedView)) == 0 {
t.Fatal("nothing was written once the bus was back")
}
}
File diff suppressed because it is too large Load Diff
@@ -7,17 +7,16 @@ import (
"time" "time"
) )
// The holder, end to end over fakes (novox/hq ADR 0239 *How it is checked*): a pull request's head proposed, // The holder, end to end over fakes (novox/hq ADR 0239 *How it is checked*, cut by ADR 0539 §17): a pull request's head
// checked and ready; merged, published, let go, delivering and delivered; a gate that fails, put back and // proposed and ready; merged; a newer head closing the older; a restart resuming; the trunk rule; and every transition
// failed; a newer head superseding; a restart resuming; the trunk rule; and every transition kept before // kept before it is said, noted on the commit and shown on the pull request.
// it is said, noted on the commit and shown on the pull request.
const ( const (
head = "aaaaaaaaaaaa1111" head = "aaaaaaaaaaaa1111"
merge = "mmmmmmmmmmmm9999" merge = "mmmmmmmmmmmm9999"
) )
func TestAPullRequestsHeadIsProposedCheckedAndReady(t *testing.T) { func TestAPullRequestsHeadIsProposedAndReady(t *testing.T) {
w := newWorld(t) w := newWorld(t)
p := pr("novox/app", 7, head, "feat/x", "src/a.go") p := pr("novox/app", 7, head, "feat/x", "src/a.go")
p.Body = "" // an empty description is never read again, so the review set below stays p.Body = "" // an empty description is never read again, so the review set below stays
@@ -30,19 +29,19 @@ func TestAPullRequestsHeadIsProposedCheckedAndReady(t *testing.T) {
} }
w.h.Checked(verdict("novox/app", 7, head, "pass")) w.h.Checked(verdict("novox/app", 7, head, "pass"))
d := w.delivery(id) d := w.delivery(id)
if d.State != Ready || d.Plan == nil || d.Plan.Summary != "builds app → anchor" { if d.State != Ready {
t.Fatalf("a passing verdict left it %s with plan %+v", d.State, d.Plan) t.Fatalf("a passing verdict left it %s", d.State)
} }
var path []State var path []State
for _, tr := range d.Transitions { for _, tr := range d.Transitions {
path = append(path, tr.To) path = append(path, tr.To)
} }
if strings.Join(statesOf(path), ",") != "proposed,checked,ready" { if strings.Join(statesOf(path), ",") != "proposed,ready" {
t.Fatalf("its transitions are %v", path) t.Fatalf("its transitions are %v", path)
} }
w.settleAll() w.settleAll()
notes := w.forge.notesOn(head) notes := w.forge.notesOn(head)
if len(notes) != 3 || !strings.Contains(notes[2], "checked -> ready") { if len(notes) != 2 || !strings.Contains(notes[1], "proposed -> ready") {
t.Fatalf("the head's note is %q", notes) t.Fatalf("the head's note is %q", notes)
} }
if !strings.Contains(w.forge.statuses["novox/app@"+head+" mesh/delivery"], "success ready") { if !strings.Contains(w.forge.statuses["novox/app@"+head+" mesh/delivery"], "success ready") {
@@ -51,9 +50,9 @@ func TestAPullRequestsHeadIsProposedCheckedAndReady(t *testing.T) {
if v := w.forge.views["novox/app#7"]; !strings.Contains(v, viewMarker) || !strings.Contains(v, "**ready**") { if v := w.forge.views["novox/app#7"]; !strings.Contains(v, viewMarker) || !strings.Contains(v, "**ready**") {
t.Fatalf("the pull request's view is %q", v) t.Fatalf("the pull request's view is %q", v)
} }
// A failing verdict for another head rejects it, and a newer head supersedes the older. // A failing verdict for another head rejects it, and a newer head closes the older.
w.h.PullUpdated(pr("novox/app", 7, "bbbbbbbbbbbb2222", "feat/x")) w.h.PullUpdated(pr("novox/app", 7, "bbbbbbbbbbbb2222", "feat/x"))
if w.state(id) != Superseded { if w.state(id) != Closed {
t.Fatalf("a newer head left the older %s", w.state(id)) t.Fatalf("a newer head left the older %s", w.state(id))
} }
w.h.Checked(verdict("novox/app", 7, "bbbbbbbbbbbb2222", "fail")) w.h.Checked(verdict("novox/app", 7, "bbbbbbbbbbbb2222", "fail"))
@@ -70,58 +69,32 @@ func statesOf(ss []State) []string {
return out return out
} }
// A merge publishes; the walk waits for this owner's word; the word is given; the walk's record moves the // A merge puts the delivery on the trunk and ends it there: merged, with the merge commit's note and the pull request's
// delivery through its machines to delivered — and the note on the merge commit says what was executed. // status naming the source revision it moved. Nothing after the merge is derived (ADR 0539 §17).
func TestAMergeIsPublishedLetGoDeliveringAndDelivered(t *testing.T) { func TestAMergeIsMergedAndTheGateDerivesNothingAfterIt(t *testing.T) {
w := newWorld(t) w := newWorld(t)
id := IDOf("novox/app", head) id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go")) w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass")) w.h.Checked(verdict("novox/app", 7, head, "pass"))
// The walk is heard before the merge: it waits for its delivery.
walk := w.ctl.put(aWalk("plan-1", "novox/app", merge, true))
w.h.WalkMoved(walk)
if w.state(id) != Ready {
t.Fatalf("a walk for a commit no merge was heard of moved the delivery to %s", w.state(id))
}
w.h.PullMerged(merged("novox/app", 7, head, merge)) w.h.PullMerged(merged("novox/app", 7, head, merge))
if w.state(id) != Published {
t.Fatalf("a merge with its walk open left it %s", w.state(id))
}
w.settleAll()
if len(w.ctl.delivers) != 1 || w.ctl.delivers[0] != "plan-1" {
t.Fatalf("its walk was let go %v", w.ctl.delivers)
}
if w.state(id) != Delivering {
t.Fatalf("let go, it is %s", w.state(id))
}
// The first machine sent, judged, passed; the rest sent; the walk done.
sent := time.Date(2026, 10, 6, 12, 40, 0, 0, time.UTC)
walk = w.ctl.walk("plan-1")
walk.State = walkRolling
walk.Modules["app"] = &WalkModule{State: "built", First: []string{"anchor"}, FirstAt: &sent, Build: "build-1",
Gate: &WalkGate{Machines: []string{"anchor"}, Since: &sent, Passes: 1}}
w.h.WalkMoved(w.ctl.put(walk))
if d := w.delivery(id); len(d.Steps) != 1 || d.Steps[0].State != StepJudging {
t.Fatalf("judging on its first machine reads %+v", d.Steps)
}
judged := sent.Add(3 * time.Minute)
walk.Modules["app"].Gate.Verdict, walk.Modules["app"].Gate.JudgedAt = "passed", &judged
walk.Modules["app"].SentAt = &judged
walk.State = walkDone
w.h.WalkMoved(w.ctl.put(walk))
d := w.delivery(id) d := w.delivery(id)
if d.State != Delivered { if d.State != Merged || d.MergedAs != merge {
t.Fatalf("a walk done left it %s", d.State) t.Fatalf("a merge left it %s as %q", d.State, d.MergedAs)
}
if len(d.Steps) != 2 || d.Steps[0].State != StepPassed || d.Steps[1].Machine != "the rest running it" {
t.Fatalf("its machine steps are %+v", d.Steps)
} }
w.settleAll() w.settleAll()
onMerge := w.forge.notesOn(merge) w.later(72 * time.Hour)
if len(onMerge) == 0 || !strings.Contains(strings.Join(onMerge, "\n"), "executed (walk plan-1): app on anchor passed") { w.settleAll()
t.Fatalf("the merge commit's note is %q", onMerge) if s := w.state(id); s != Merged {
t.Fatalf("time moved a merged delivery to %s", s)
}
if st := w.forge.statuses["novox/app@"+head+" mesh/delivery"]; !strings.HasPrefix(st, "success merged: source.main moved to novox/app@"+merge[:12]) {
t.Fatalf("the pull request's mesh/delivery status is %q", st)
}
view := w.forge.views["novox/app#7"]
if !strings.Contains(view, "**Source**") || !strings.Contains(view, "source.main moved to novox/app@"+merge[:12]) ||
!strings.Contains(view, "mesh-delivery-view") || strings.Contains(view, "**Delivery plan**") || strings.Contains(view, "**Machines**") {
t.Fatalf("the pull request's view is %q", view)
} }
// Said on the bus, kept first: every transition emitted once.
n := 0 n := 0
for _, s := range w.said { for _, s := range w.said {
if s["id"] == id { if s["id"] == id {
@@ -133,110 +106,52 @@ func TestAMergeIsPublishedLetGoDeliveringAndDelivered(t *testing.T) {
} }
} }
// A first machine's gate fails: what it carried is put back (the controller's), the step is rolled back, // The trunk rule: a ready delivery whose commit never reached the trunk is never merged.
// and the delivery failed — never delivered. func TestACommitOffTheTrunkIsNeverMerged(t *testing.T) {
func TestAFailedGateIsRolledBackAndTheDeliveryFails(t *testing.T) {
w := newWorld(t) w := newWorld(t)
id := IDOf("novox/app", head) id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go")) w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass")) w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
w.h.WalkMoved(w.ctl.put(aWalk("plan-1", "novox/app", merge, true)))
w.settleAll()
sent := time.Date(2026, 10, 6, 12, 40, 0, 0, time.UTC)
walk := w.ctl.walk("plan-1")
walk.State = walkRolling
walk.Modules["app"] = &WalkModule{State: "built", First: []string{"anchor"}, FirstAt: &sent,
Gate: &WalkGate{Machines: []string{"anchor"}, Since: &sent}}
w.h.WalkMoved(w.ctl.put(walk))
at := sent.Add(10 * time.Minute)
walk.Modules["app"].Gate.Verdict, walk.Modules["app"].Gate.Why, walk.Modules["app"].Gate.JudgedAt =
"failed", "its tools were not served", &at
walk.Modules["app"].Gate.Rollback = "rolled-back"
walk.State = walkFailed
walk.Note = "app failed its gate on anchor"
w.h.WalkMoved(w.ctl.put(walk))
d := w.delivery(id)
if d.State != Failed || len(d.Steps) != 1 || d.Steps[0].State != StepRolledBack {
t.Fatalf("a failed gate left it %s, steps %+v", d.State, d.Steps)
}
// An older record arriving late changes nothing; a step the table does not hold is refused.
w.h.WalkMoved(aWalk("plan-1", "novox/app", merge, false))
if w.state(id) != Failed {
t.Fatal("an older record of the walk moved a final delivery")
}
}
// A walk superseded that names no walk taking it over (a record from before ADR 0276): superseded, from
// published or delivering. One that names its successor is followed (TestADeliveryFollowsTheWalkThatTookItsOver).
func TestANewerWalkSupersedesAnOlderDelivery(t *testing.T) {
w := newWorld(t)
id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
walk := aWalk("plan-1", "novox/app", merge, true)
w.h.WalkMoved(w.ctl.put(walk))
walk.State = walkSuperseded
walk.Note = "superseded"
w.ctl.down = true // its word could not be given meanwhile
w.settleAll()
w.h.WalkMoved(w.ctl.put(walk))
if s := w.state(id); s != Superseded {
t.Fatalf("its walk superseded left it %s", s)
}
}
// The trunk rule: a ready delivery whose commit never reached the trunk is never published, whatever walks
// are heard of; and a walk for a commit no delivery landed as is not taken as its.
func TestACommitOffTheTrunkIsNeverPublished(t *testing.T) {
w := newWorld(t)
id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.WalkMoved(w.ctl.put(aWalk("plan-9", "novox/app", head, true))) // a walk of the head itself: no merge
w.settleAll() w.settleAll()
if s := w.state(id); s != Ready { if s := w.state(id); s != Ready {
t.Fatalf("an off-trunk head is %s", s) t.Fatalf("an off-trunk head is %s", s)
} }
if len(w.ctl.delivers) != 0 {
t.Fatalf("a walk was let go for a commit off the trunk: %v", w.ctl.delivers)
}
} }
// A restart resumes: a new holder over the same state stands every delivery where it was, reads the walks it // A restart resumes: a new holder over the same state stands every delivery where it was.
// missed back from the controller, and goes on.
func TestARestartResumesFromTheState(t *testing.T) { func TestARestartResumesFromTheState(t *testing.T) {
w := newWorld(t) w := newWorld(t)
id := IDOf("novox/app", head) id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go")) w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass")) w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge)) w.h.PullMerged(merged("novox/app", 7, head, merge))
w.ctl.put(aWalk("plan-1", "novox/app", merge, true))
if err := w.h.Reconcile(); err != nil {
t.Fatal(err)
}
w.settleAll() w.settleAll()
if s := w.state(id); s != Delivering { if s := w.state(id); s != Merged {
t.Fatalf("before the restart it is %s", s) t.Fatalf("before the restart it is %s", s)
} }
// While it is down, the walk finishes and nobody says so. again := newWorldOver(t, w.store, w.forge, w.now.Add(time.Hour))
walk := w.ctl.walk("plan-1") if s := again.state(id); s != Merged {
walk.State = walkDone
w.ctl.put(walk)
again := newWorldOver(t, w.store, w.ctl, w.forge, w.now.Add(time.Hour))
if s := again.state(id); s != Delivering {
t.Fatalf("read back, it is %s", s) t.Fatalf("read back, it is %s", s)
} }
if err := again.h.Reconcile(); err != nil { }
t.Fatal(err)
// A delivery kept in a state an earlier owner had and this one does not is read as the state it came to and kept so.
func TestADeliveryKeptInAGoneStateIsReadAsTheMergeStateItCameTo(t *testing.T) {
w := newWorld(t)
id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
w.settleAll()
raw := w.store.deliveries[id]
w.store.deliveries[id] = []byte(strings.Replace(string(raw), `"state":"merged"`, `"state":"delivering"`, 1))
again := newWorldOver(t, w.store, w.forge, w.now.Add(time.Hour))
if s := again.state(id); s != Merged {
t.Fatalf("a delivering delivery was read as %s", s)
} }
if s := again.state(id); s != Delivered { again.settleAll()
t.Fatalf("the walk it missed was not read back: %s", s) if !strings.Contains(string(w.store.deliveries[id]), `"state":"merged"`) {
} t.Fatalf("the read state was not kept: %s", w.store.deliveries[id])
if len(w.ctl.delivers) != 1 {
t.Fatalf("a restarted holder let the walk go again: %v", w.ctl.delivers)
} }
} }
@@ -265,7 +180,7 @@ func TestATransitionIsKeptBeforeItIsSaid(t *testing.T) {
w.later(time.Hour) // each failure put its next try later: past every one of them w.later(time.Hour) // each failure put its next try later: past every one of them
w.settleAll() w.settleAll()
w.settleAll() w.settleAll()
if notes := w.forge.notesOn(head); len(notes) != 3 { if notes := w.forge.notesOn(head); len(notes) != 2 {
t.Fatalf("after the bus came back the note has %d line(s): %q", len(notes), notes) t.Fatalf("after the bus came back the note has %d line(s): %q", len(notes), notes)
} }
if d := w.delivery(IDOf("novox/app", head)); len(d.Owed) != 0 { if d := w.delivery(IDOf("novox/app", head)); len(d.Owed) != 0 {
@@ -273,66 +188,8 @@ func TestATransitionIsKeptBeforeItIsSaid(t *testing.T) {
} }
} }
// At the switch: a walk already running is adopted as a delivery that is delivering; a waiting walk whose pull
// request this owner never heard of waits for a person, and a person's release lets it go.
func TestTheSwitchAdoptsRunningWalksAndHoldsUnknownWaitingOnes(t *testing.T) {
w := newWorld(t)
running := aWalk("plan-old", "novox/mesh-catalog", "cccccccccccc3333", false)
running.Created = w.now.Add(-time.Hour)
running.State = walkRolling
w.ctl.put(running)
waiting := aWalk("plan-new", "novox/app", "dddddddddddd4444", true)
waiting.Created = w.now.Add(time.Minute)
w.ctl.put(waiting)
if err := w.h.Reconcile(); err != nil {
t.Fatal(err)
}
w.h.React()
if s := w.state(IDOf("novox/mesh-catalog", "cccccccccccc3333")); s != Delivering {
t.Fatalf("a running walk at the switch was adopted as %s", s)
}
if _, made := w.h.deliveries[IDOf("novox/app", "dddddddddddd4444")]; made {
t.Fatal("a new walk was made a delivery before its merge had time to be heard")
}
w.later(3 * time.Minute)
w.h.React()
held := IDOf("novox/app", "dddddddddddd4444")
if s := w.state(held); s != Held {
t.Fatalf("a waiting walk with no pull request is %s", s)
}
if _, err := w.h.Release(held, "", "jochen"); err == nil {
t.Fatal("released without why")
}
if _, err := w.h.Release(held, "checked by hand", "jochen"); err != nil {
t.Fatal(err)
}
if s := w.state(held); s != Delivering || len(w.ctl.delivers) != 1 {
t.Fatalf("released, it is %s and let go %v", s, w.ctl.delivers)
}
}
// mesh-delivery's controller down: nothing is lost — its word is asked again on the next tick.
func TestAControllerThatDoesNotAnswerIsAskedAgain(t *testing.T) {
w := newWorld(t)
id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
w.h.WalkMoved(w.ctl.put(aWalk("plan-1", "novox/app", merge, true)))
w.ctl.down = true
w.settleAll()
if s := w.state(id); s != Published {
t.Fatalf("with the controller down it is %s", s)
}
w.ctl.down = false
w.settleAll()
if s := w.state(id); s != Delivering {
t.Fatalf("with the controller back it is %s", s)
}
}
// A merge before the check passed is held for a person; a merge this owner never heard the head of is made // A merge before the check passed is held for a person; a merge this owner never heard the head of is made
// from the forge's word and published when the forge says its gate passed. // from the forge's word and merged when the forge says its gate passed.
func TestAMergeWithoutAPassingCheckIsHeld(t *testing.T) { func TestAMergeWithoutAPassingCheckIsHeld(t *testing.T) {
w := newWorld(t) w := newWorld(t)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go")) w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
@@ -348,50 +205,38 @@ func TestAMergeWithoutAPassingCheckIsHeld(t *testing.T) {
w.h.PullMerged(m) w.h.PullMerged(m)
w.settleAll() w.settleAll()
d := w.delivery(IDOf("novox/lab", "eeeeeeeeeeee5555")) d := w.delivery(IDOf("novox/lab", "eeeeeeeeeeee5555"))
// The fake planner says it moves app; with no walk yet it waits published. if d.State != Merged {
if d.State != Ready && d.State != Published {
t.Fatalf("a merge whose head passed by the forge's word is %s", d.State) t.Fatalf("a merge whose head passed by the forge's word is %s", d.State)
} }
} }
// A member that fails stops no other (ADR 0418 §5): the plan that fails fails as every plan does, and what was delivered // A group merged whole ends merged, member by member and as a group, and source.main holds its heads (ADR 0539 §17, R6-C2).
// stays. Nothing is stopped member by member. func TestAMergedGroupEndsMergedWithSourceMainMoved(t *testing.T) {
func TestAFailedMemberStopsNoOtherMember(t *testing.T) {
w := newWorld(t) w := newWorld(t)
heads := map[string]string{"novox/a": "a1a1a1a1a1a1a1", "novox/b": "b2b2b2b2b2b2b2", "novox/c": "c3c3c3c3c3c3c3"} heads := map[string]string{"novox/a": "a1a1a1a1a1a1a1", "novox/b": "b2b2b2b2b2b2b2"}
ids := map[string]string{} ids := map[string]string{}
for i, repo := range []string{"novox/a", "novox/b", "novox/c"} { for i, repo := range []string{"novox/a", "novox/b"} {
ids[repo] = IDOf(repo, heads[repo]) ids[repo] = IDOf(repo, heads[repo])
w.h.PullUpdated(pr(repo, i+1, heads[repo], "feat/y", "x")) w.h.PullUpdated(pr(repo, i+1, heads[repo], "feat/y", "x"))
w.h.Checked(verdict(repo, i+1, heads[repo], "pass")) w.h.Checked(verdict(repo, i+1, heads[repo], "pass"))
} }
w.settleAll() w.settleAll()
w.h.Checked(CheckedEvent{Group: "feat/y", ID: "check-1", Verdict: "pass", Summary: "compose"}) w.h.Checked(CheckedEvent{Group: "feat/y", ID: "check-1", Verdict: "pass", Summary: "compose"})
for i, repo := range []string{"novox/a", "novox/b", "novox/c"} { for i, repo := range []string{"novox/a", "novox/b"} {
mergeCommit := strings.Repeat(string(rune('p'+i)), 12) w.h.PullMerged(merged(repo, i+1, heads[repo], strings.Repeat(string(rune('p'+i)), 12)))
w.h.PullMerged(merged(repo, i+1, heads[repo], mergeCommit))
w.h.WalkMoved(w.ctl.put(aWalk("plan-"+repo[6:], repo, mergeCommit, true)))
} }
w.settleAll() w.settleAll()
if len(w.ctl.delivers) != 3 { for repo, id := range ids {
t.Fatalf("every member merged, yet the walks were let go %v", w.ctl.delivers) if s := w.state(id); s != Merged {
t.Errorf("%s is %s, wanted merged", repo, s)
}
} }
a := w.ctl.walk("plan-a") g := w.h.groups["feat/y"]
a.State = walkDone if state, why := GroupState(g, w.h.membersOf(g)); state != "merged" {
w.h.WalkMoved(w.ctl.put(a)) t.Fatalf("the group is %s (%s), wanted merged", state, why)
b := w.ctl.walk("plan-b")
b.State = walkFailed
b.Note = "b failed its gate on anchor"
w.h.WalkMoved(w.ctl.put(b))
w.settleAll()
if w.state(ids["novox/a"]) != Delivered || w.state(ids["novox/b"]) != Failed || w.state(ids["novox/c"]) == Stopped {
t.Fatalf("a %s, b %s, c %s", w.state(ids["novox/a"]), w.state(ids["novox/b"]), w.state(ids["novox/c"]))
} }
if len(w.ctl.stops) != 0 || len(w.ctl.withouts) != 0 { if w.source.main == nil || w.source.main.Heads["novox/a"] != strings.Repeat("p", 12) || w.source.main.Heads["novox/b"] != strings.Repeat("q", 12) {
t.Fatalf("a walk was stopped because another failed: %v %v", w.ctl.stops, w.ctl.withouts) t.Fatalf("source.main is %+v, wanted both merges", w.source.main)
}
if state, _ := GroupState(w.h.groups["feat/y"], w.h.membersOf(w.h.groups["feat/y"])); state != "failed" {
t.Fatalf("the group is %s", state)
} }
} }
@@ -434,44 +279,38 @@ func TestAnAfterLineIsNotReadAndSaidOnceAndNothingWaitsForAnOrder(t *testing.T)
} }
} }
// Stalled: a delivery past its state's bound is listed with what H2 may do; close takes only that. // Stalled: a delivery past its state's bound is listed with what H2 may do; close takes only that. Only a proposed
// delivery has a bound now: its check was asked and not answered.
func TestStalledAndCloseWorkFromTheTable(t *testing.T) { func TestStalledAndCloseWorkFromTheTable(t *testing.T) {
w := newWorld(t) w := newWorld(t)
id := IDOf("novox/app", head) id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go")) w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
walk := w.ctl.put(aWalk("plan-1", "novox/app", merge, false))
w.h.WalkMoved(walk)
if w.state(id) != Delivering {
t.Fatalf("it is %s", w.state(id))
}
if _, err := w.h.Close(id, "too early"); err == nil { if _, err := w.h.Close(id, "too early"); err == nil {
t.Fatal("closed within its bound") t.Fatal("closed within its bound")
} }
w.later(3 * time.Hour) w.later(3 * time.Hour)
stalled := w.h.Stalled() stalled := w.h.Stalled()
if len(stalled) != 1 || stalled[0].ID != id || !strings.Contains(stalled[0].H2, "close") { if len(stalled) != 1 || stalled[0].ID != id || stalled[0].State != Proposed {
t.Fatalf("stalled %+v", stalled) t.Fatalf("stalled %+v", stalled)
} }
if _, err := w.h.Close(id, "delivery stalled"); err == nil { // A merge is the gate's last word: it is never stalled, however old.
t.Fatal("closed with nothing in the walk's record saying it moved") w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
w.later(72 * time.Hour)
for _, s := range w.h.Stalled() {
if s.ID == id {
t.Fatalf("a merged delivery is stalled: %+v", s)
}
} }
walk.State = walkDone if _, err := w.h.Close(id, "x"); err == nil {
w.ctl.put(walk) // finished, and the event was missed t.Fatal("H2 closed a merged delivery")
said, err := w.h.Close(id, "delivery stalled")
if err != nil || w.state(id) != Delivered {
t.Fatalf("H2's close left it %s: %v", w.state(id), err)
} }
if !strings.Contains(said, "delivered") { // A merge without a passing check is held: a person's, with no transition for H2.
t.Fatalf("close says %q", said)
}
// Held is the operator's: H2 has no transition there.
w.h.PullUpdated(pr("novox/app", 8, "ffffffffffff0000", "feat/q")) w.h.PullUpdated(pr("novox/app", 8, "ffffffffffff0000", "feat/q"))
w.h.PullMerged(merged("novox/app", 8, "ffffffffffff0000", "eeeeeeeeeeee0000")) w.h.PullMerged(merged("novox/app", 8, "ffffffffffff0000", "eeeeeeeeeeee0000"))
w.later(48 * time.Hour) w.later(48 * time.Hour)
if _, err := w.h.Close(IDOf("novox/app", "ffffffffffff0000"), "x"); err == nil || !strings.Contains(err.Error(), "operator's") { if _, err := w.h.Close(IDOf("novox/app", "ffffffffffff0000"), "x"); err == nil {
t.Fatalf("H2 closed a held delivery: %v", err) t.Fatal("H2 closed a held delivery")
} }
} }
@@ -520,14 +359,3 @@ func TestGroupsNamesEachMembersPullRequestHeadBaseAndOpen(t *testing.T) {
} }
} }
} }
// composed are the asks of a group's composed check among the checks asked: a member's own check has no group before its colon.
func composed(checks []string) []string {
var out []string
for _, c := range checks {
if !strings.HasPrefix(c, ":") {
out = append(out, c)
}
}
return out
}
@@ -1,7 +1,6 @@
package main package main
import ( import (
"strings"
"testing" "testing"
) )
@@ -16,40 +15,7 @@ const (
branch1083 = "feat/backup-before-send" branch1083 = "feat/backup-before-send"
) )
func TestAPullRequestAtACommitWhoseDeliveryAPersonStoppedTakesItOver(t *testing.T) { // A closed record that was merged is not taken over, and the same pull request announced again stays as it was.
w := newWorld(t)
id := IDOf(catalog783, head1083)
w.h.PullUpdated(pr(catalog783, 375, head1083, branch1083, "src/a.go"))
w.h.Checked(verdict(catalog783, 375, head1083, "pass"))
if w.state(id) != Ready {
t.Fatalf("its passing verdict left it %s", w.state(id))
}
if _, err := w.h.Stop(id, "to break the loop", "jochen"); err != nil {
t.Fatal(err)
}
// The person stopped it first; the pull request closed afterwards: a final state is not moved by the close.
w.h.PullClosed(pr(catalog783, 375, head1083, branch1083))
if d := w.delivery(id); d.State != Stopped || d.ClosedUnmerged {
t.Fatalf("setup: %s, closed unmerged %v; wanted stopped by the person alone", d.State, d.ClosedUnmerged)
}
w.h.PullUpdated(pr(catalog783, 376, head1083, branch1083+"-2", "src/a.go"))
d := w.delivery(id)
if d.State != Proposed || d.Number != 376 {
t.Fatalf("#376 at the same commit left it %s, Number %d; wanted proposed under #376", d.State, d.Number)
}
why := d.Transitions[len(d.Transitions)-1].Why
if !strings.Contains(why, "#375 with the same head was stopped by jochen: to break the loop") {
t.Fatalf("the take-over is not said, with who stopped it and why: %q", why)
}
w.h.Checked(verdict(catalog783, 376, head1083, "pass"))
if w.state(id) != Ready {
t.Fatalf("a passing verdict for #376 left it %s", w.state(id))
}
}
// A stopped record that was merged is not taken over, and the same pull request announced again stays stopped.
func TestAStoppedRecordThatWasMergedIsNotTakenOverByAnotherPullRequest(t *testing.T) { func TestAStoppedRecordThatWasMergedIsNotTakenOverByAnotherPullRequest(t *testing.T) {
w := newWorld(t) w := newWorld(t)
id := IDOf(catalog783, head783c+"1") id := IDOf(catalog783, head783c+"1")
@@ -67,9 +33,6 @@ func TestATakenOverRecordLeavesTheGroupItWasStoppedIn(t *testing.T) {
w.h.PullUpdated(pr(gCtl, 1, gHc, "feat/g", "internal/a.go")) w.h.PullUpdated(pr(gCtl, 1, gHc, "feat/g", "internal/a.go"))
w.h.PullUpdated(pr(gCat, 2, gHk, "feat/g", "modules/app/module.json")) w.h.PullUpdated(pr(gCat, 2, gHk, "feat/g", "modules/app/module.json"))
id := IDOf(gCtl, gHc) id := IDOf(gCtl, gHc)
if _, err := w.h.Stop(id, "no", "jochen"); err != nil {
t.Fatal(err)
}
w.h.PullClosed(pr(gCtl, 1, gHc, "feat/g")) w.h.PullClosed(pr(gCtl, 1, gHc, "feat/g"))
w.h.PullUpdated(pr(gCtl, 9, gHc, "feat/other", "internal/a.go")) w.h.PullUpdated(pr(gCtl, 9, gHc, "feat/other", "internal/a.go"))
w.h.mu.Lock() w.h.mu.Lock()
@@ -2,9 +2,14 @@ package main
import ( import (
"errors" "errors"
"fmt"
"strings" "strings"
"sync"
"testing" "testing"
"time" "time"
"git.novox.be/novox/mesh-sdk/go/asks"
sdk "git.novox.be/novox/mesh-sdk/go/delivery"
) )
const time30s = 31 * time.Second const time30s = 31 * time.Second
@@ -75,24 +80,6 @@ func TestAnApprovedGroupWithAMemberWhoseRequiredStatusIsNotASuccessMergesNothing
} }
} }
// 1275/AC3: the merge tool refuses the first member in order; the second is not merged, and the group is said urgent,
// naming the group, the refused member and the forge's words.
func TestARefusedFirstMemberStopsTheGroupAndIsSaidUrgent(t *testing.T) {
w := approvedGroup(t)
allGreen(w)
w.forge.refuseRepo = map[string]error{gCat: refusedWords("novox/mesh-catalog#2")}
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
if len(w.forge.merges) != 1 || !strings.Contains(w.forge.merges[0], "mesh-catalog#2") {
t.Fatalf("wanted only the first member asked: %v", w.forge.merges)
}
lines := urgentLines(w)
if len(lines) != 1 || lines[0].ID != IDOf(gCat, gHk) || !strings.Contains(lines[0].Says, "feat/g") ||
!strings.Contains(lines[0].Says, "mesh/repo-check (missing)") || !strings.Contains(lines[0].Says, "Needed") {
t.Fatalf("not said as one urgent line: %+v", lines)
}
}
// 1275/AC4: a group half merged (the second member refused after the first merged) stays raised, pass after pass, until // 1275/AC4: a group half merged (the second member refused after the first merged) stays raised, pass after pass, until
// the second is merged; then it goes. // the second is merged; then it goes.
func TestAHalfMergedGroupStaysRaisedUntilEveryMemberIsMerged(t *testing.T) { func TestAHalfMergedGroupStaysRaisedUntilEveryMemberIsMerged(t *testing.T) {
@@ -344,3 +331,440 @@ func TestASeverityOnceUrgentNeverStepsDownToAWarning(t *testing.T) {
t.Fatalf("severity %v after an out of date refusal that followed an urgent one; wanted urgent", got) t.Fatalf("severity %v after an out of date refusal that followed an urgent one; wanted urgent", got)
} }
} }
// fakeConditions is the gate's bucket conditions and the operator-channel seat's subject for conditions: what it keeps,
// and every condition said, in order.
type fakeConditions struct {
mu sync.Mutex
kept map[string]string
told []asks.Condition
}
func (f *fakeConditions) Said() (map[string]string, error) {
f.mu.Lock()
defer f.mu.Unlock()
out := map[string]string{}
for k, v := range f.kept {
out[k] = v
}
return out, nil
}
func (f *fakeConditions) PutSaid(said map[string]string) error {
f.mu.Lock()
defer f.mu.Unlock()
f.kept = map[string]string{}
for k, v := range said {
f.kept[k] = v
}
return nil
}
func (f *fakeConditions) say(c asks.Condition) error {
if _, err := asks.EncodeCondition(c); err != nil {
return err // what the router would refuse
}
f.mu.Lock()
defer f.mu.Unlock()
f.told = append(f.told, c)
return nil
}
// last is the newest condition said for a delivery id, or nil.
func (f *fakeConditions) last(id string) *asks.Condition {
f.mu.Lock()
defer f.mu.Unlock()
for i := len(f.told) - 1; i >= 0; i-- {
if f.told[i].Subject.ID == id {
c := f.told[i]
return &c
}
}
return nil
}
// hearConditions has the gate say its conditions into a fake seat and bucket.
func hearConditions(w *world) *fakeConditions {
f := &fakeConditions{kept: map[string]string{}}
w.h.Say, w.h.Conditions = f.say, f
return f
}
// handedInOrder answers each member the gate hands the forge bridge as the forge does on 2026-10-25: the forge's word that
// it merged (gitea.pull.merged) first, the bridge's word after; or the bridge's first, as forgeFirst says. It returns the
// repositories in the order they were handed.
func handedInOrder(w *world, forgeFirst bool) []string {
var handed []string
numbers := map[string]int{}
for i := 0; i < 4; i++ {
var next *sdk.MergePull
for _, p := range w.merges.last("feat/g").Pulls {
if p.State == sdk.MergeMerging {
p := p
next = &p
}
}
if next == nil || numbers[next.Repository] > 0 {
break
}
handed = append(handed, next.Repository)
numbers[next.Repository] = next.Number
mergeCommit := strings.Repeat(string(rune('a'+i)), 40)
forge := func() { w.h.PullMerged(merged(next.Repository, next.Number, next.Head.Head, mergeCommit)) }
bridge := func() { bridgeMerged(w, next.Repository, next.Number, next.Head.Head, mergeCommit) }
if forgeFirst {
forge()
w.settleAll() // ten seconds pass between the two words on 2026-10-25: the gate looks in between
bridge()
} else {
bridge()
w.settleAll()
forge()
}
w.settleAll()
}
return handed
}
// Issue 1275, met again 2026-10-25 (group fix/puller-wakes-on-capacity): every member of an approved group is handed to
// the forge bridge, in the group's order, the next on the event that the one before merged; a group whose pass stops is
// said to the operator as a condition, which is cleared once it is whole; and every word a group's state is said in is
// one the merge decides on.
func TestAnApprovedGroupIsHandedWholeInItsOrderAndAStopIsSaid(t *testing.T) {
const (
sdkRepo, hS = "novox/mesh-sdk", "a8fbabbfc37369f9d1f6977c72e5523aff3d494c"
catRepo, hK = "novox/mesh-catalog", "23c827e7910c233608af4e4a8056cbc362dcb755"
ctlRepo, hC = "novox/mesh-controller", "c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0"
)
sdkMember := member{sdkRepo, 80, hS, "go/stdio.go,go/buildstep/pull.go"}
catMember := member{catRepo, 479, hK, "modules/mesh-build-go/asks.go,modules/mesh-build-go/go.mod,modules/mesh-build-go/go.sum"}
ctlMember := member{ctlRepo, 7, hC, "cmd/mesh-controller/main.go"}
inOrder := func(members []member, forgeFirst bool, want ...string) func(t *testing.T) {
return func(t *testing.T) {
w := approvedGroupOf(t, members...)
w.merges.manual = true
allGreenFor(w, members...)
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
if got := handedInOrder(w, forgeFirst); strings.Join(got, " ") != strings.Join(want, " ") {
t.Fatalf("handed %v, wanted %v, every member in the group's order", got, want)
}
if g := w.h.groups["feat/g"]; g == nil {
t.Fatal("the group is gone")
} else if state, why := GroupState(g, w.h.membersOf(g)); state != GroupMerged {
t.Fatalf("the group is %s (%s), not merged whole", state, why)
}
if w.source.main == nil || len(w.source.main.Heads) != len(want) {
t.Fatalf("source.main does not hold the group as one revision: %+v", w.source.main)
}
}
}
cases := []struct {
name string
run func(t *testing.T)
}{
// 1275/AC3 (2026-10-25): the case of the day. The catalogue member pins the sdk member's commit; the forge's word
// of a merge arrives before the bridge's. Before the fix the catalogue member went first, and the sdk member was
// never handed: its group read merging, a word the turn to merge did not know.
{"sdk first, then the member that pins it, the forge's word first", inOrder([]member{catMember, sdkMember}, true, sdkRepo, catRepo)},
{"sdk first, then the member that pins it, the bridge's word first", inOrder([]member{sdkMember, catMember}, false, sdkRepo, catRepo)},
{"a member that pins nothing before one that changes a go.mod", inOrder([]member{catMember, ctlMember}, true, ctlRepo, catRepo)},
{"three members", inOrder([]member{ctlMember, catMember, sdkMember}, true, sdkRepo, ctlRepo, catRepo)},
// 1275/AC3: the merge tool refuses the first member in order; the second is not merged, and the group is said
// urgent, naming the group, the refused member and the forge's words, as a stalled line and to the operator.
{"a refused first member stops the group and is said urgent", func(t *testing.T) {
w := approvedGroup(t)
told := hearConditions(w)
allGreen(w)
w.forge.refuseRepo = map[string]error{gCat: refusedWords("novox/mesh-catalog#2")}
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
if len(w.forge.merges) != 1 || !strings.Contains(w.forge.merges[0], "mesh-catalog#2") {
t.Fatalf("wanted only the first member asked: %v", w.forge.merges)
}
lines := urgentLines(w)
if len(lines) != 1 || lines[0].ID != IDOf(gCat, gHk) || !strings.Contains(lines[0].Says, "feat/g") ||
!strings.Contains(lines[0].Says, "mesh/repo-check (missing)") || !strings.Contains(lines[0].Says, "Needed") {
t.Fatalf("not said as one urgent line: %+v", lines)
}
if c := told.last(IDOf(gCat, gHk)); c == nil || c.State != asks.StateRaised || c.Severity != asks.SeverityUrgent ||
!strings.Contains(c.Summary, "feat/g") {
t.Fatalf("the operator was not told, urgent: %+v", c)
}
}},
// 1275/AC4: a group half merged (its pass stopped after the first member) raises one urgent condition, said once
// however many looks follow, and cleared once the group is whole.
{"a half-merged group is raised once and cleared once whole", func(t *testing.T) {
w := approvedGroup(t)
told := hearConditions(w)
allGreen(w)
w.merges.manual = true
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
w.h.PullMerged(merged(gCat, 2, gHk, gT1))
bridgeMerged(w, gCat, 2, gHk, gT1)
setChecks(w, gCtl, gHc, "success", "") // the second member's status went missing: the pass stops at it
w.settleAll()
for range 3 {
w.later(time30s)
w.settleAll()
}
id := IDOf(gCtl, gHc)
raised := 0
for _, c := range told.told {
if c.Subject.ID == id && c.State == asks.StateRaised {
raised++
}
}
if c := told.last(id); raised != 1 || c == nil || c.Severity != asks.SeverityUrgent || !strings.Contains(c.Summary, "feat/g") ||
!strings.Contains(c.Summary, "half merged (merged 1 of 2)") {
t.Fatalf("raised %d times, last %+v; wanted once, urgent, naming the group half merged", raised, c)
}
setChecks(w, gCtl, gHc, "success", "success")
w.h.CommitStatus(ForgeStatusEvent{Owner: "novox", Repo: "mesh-controller", SHA: gHc, Context: repoCheck, State: "success"})
w.settleAll() // the status is an event: the pass goes on with no timer
handed := false
for _, p := range w.merges.last("feat/g").Pulls {
handed = handed || (p.Repository == gCtl && p.State == sdk.MergeMerging)
}
if !handed {
t.Fatalf("the status turned success and the second member was not handed: %+v", w.merges.last("feat/g"))
}
w.h.PullMerged(merged(gCtl, 1, gHc, gT1))
bridgeMerged(w, gCtl, 1, gHc, gT1)
w.settleAll()
if c := told.last(id); c == nil || c.State != asks.StateCleared {
t.Fatalf("the group is whole and its condition is not cleared: %+v", c)
}
}},
// A condition the gate raised before it restarted, whose delivery stalls no more, is cleared at the first look.
{"a condition kept from before a restart is cleared", func(t *testing.T) {
w := newWorld(t)
told := hearConditions(w)
told.kept[conditionKeyOf("novox/app@0123456789ab")] = "urgent||"
w.settleAll()
if c := told.last("novox/app@0123456789ab"); c != nil {
t.Fatalf("cleared before the forge's open pull requests were read: %+v", c)
}
w.h.wantForge()
w.settleAll()
if c := told.last("novox/app@0123456789ab"); c == nil || c.State != asks.StateCleared || len(told.kept) != 0 {
t.Fatalf("not cleared: %+v, kept %v", c, told.kept)
}
}},
// Review round 1, blocker 1: a pull request closed or merged on the forge with no event heard (de-spiegel #4,
// closed 2026-10-17, ready here eight days later) settles at the next complete read of the open list, and no
// condition is raised for it; a merge learnt so does not move source.main.
{"a pull request gone from the open list settles and raises nothing", func(t *testing.T) {
for _, merged := range []bool{false, true} {
w := newWorld(t)
w.op.auto = false
const repo, head = "novox/de-spiegel", "fdcbaefb3123fdcbaefb3123fdcbaefb3123fdcb"
w.h.PullUpdated(pr(repo, 4, head, "feat/x", "src/a.go"))
w.h.Checked(judgedAt(repo, 4, head, "pass", "t1t1t1t1t1t1"))
w.settleAll()
w.forge.refuse = errors.New("novox/de-spiegel#4 merges into main, which no protection rule requiring the checks guards")
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
id := IDOf(repo, head)
if len(w.h.Stalled()) == 0 {
t.Fatalf("the refused head is not stalled to begin with")
}
p := ForgePull{Number: 4, State: "closed", HeadSHA: head, Base: "main"}
if merged {
p.Merged, p.MergeCommit = true, strings.Repeat("e", 40)
}
w.forge.pulls[repo+"#4"] = p
w.forge.open = []OpenPull{{Owner: "novox", Repo: "other", Number: 1, Base: "main", HeadSHA: "abab"}}
told := hearConditions(w)
moves := len(w.source.moves)
w.h.wantForge()
w.settleAll()
want := Closed
if merged {
want = Merged
}
if st := w.state(id); st != want {
t.Fatalf("merged %v: the delivery is %s, wanted %s", merged, st, want)
}
if c := told.last(id); c != nil {
t.Fatalf("merged %v: a condition was said for it: %+v", merged, c)
}
if len(w.source.moves) != moves {
t.Fatalf("merged %v: source.main moved for a merge learnt by its absence: %+v", merged, w.source.moves)
}
}
}},
// Review round 2: a merge learnt after the fact moves no other delivery's trunk (an approved group of the same
// repository keeps its trunk, its state and its composed check), takes the forge's time, and a catch-up after it
// does not move source.main.
{"a merge learnt late moves no trunk and no source", func(t *testing.T) {
w := approvedGroup(t)
allGreen(w)
w.merges.manual = true
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
const lateHead, lateMerge = "9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a", "8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b"
w.h.PullUpdated(pr(gCat, 9, lateHead, "fix/old", "modules/app/x.go"))
w.h.Checked(judged(gCat, 9, lateHead, "pass", gTrunk0))
w.settleAll()
late := IDOf(gCat, lateHead)
trunkOf := func() map[string]string {
out := map[string]string{}
for _, id := range []string{IDOf(gCat, gHk), IDOf(gCtl, gHc)} {
out[id] = w.delivery(id).TrunkNow + " " + string(w.delivery(id).State)
}
return out
}
before, moves := trunkOf(), len(w.source.moves)
w.forge.pulls[gCat+"#9"] = ForgePull{Number: 9, State: "closed", Merged: true, MergeCommit: lateMerge,
MergedAt: "2026-10-01T09:00:00Z", HeadSHA: lateHead, Base: "main"}
w.forge.open = []OpenPull{{Owner: "novox", Repo: "mesh-catalog", Number: 2, Base: "main", HeadSHA: gHk},
{Owner: "novox", Repo: "mesh-controller", Number: 1, Base: "main", HeadSHA: gHc}}
w.h.wantForge()
w.settleAll()
d := w.delivery(late)
if d.State != Merged || !d.MergedAt.Equal(time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)) || !d.LearntLate {
t.Fatalf("the late merge is %s at %v (learnt late %v)", d.State, d.MergedAt, d.LearntLate)
}
if after := trunkOf(); fmt.Sprint(after) != fmt.Sprint(before) {
t.Fatalf("the group's trunk or state moved: %v, was %v", after, before)
}
if g := w.h.groups["feat/g"]; g.Check == nil || g.Check.OutOfDate != "" {
t.Fatalf("the group's composed check went out of date: %+v", g.Check)
}
w.h.SourceCatchUp()
w.settleAll()
if len(w.source.moves) != moves {
t.Fatalf("source.main moved for a merge learnt late: %+v", w.source.moves[moves:])
}
}},
// mesh-delivery#51 (2026-10-25 21:43): the open list was read seconds after the bridge merged it and before the forge's
// merged event, so it was learnt late, moved no trunk, and its group's finish left it out of source.main. A merge
// learnt within replayLate of its landing is taken as heard as it happened; one a gate before the fix took late is
// written at the catch-up of the start.
{"a merge learnt from the open list seconds after it landed moves source.main", func(t *testing.T) {
const repo, head, merge = "novox/mesh-delivery", "4628ba26b41a4628ba26b41a4628ba26b41a4628", "db53ec975f01db53ec975f01db53ec975f01db53"
for _, oldGate := range []bool{false, true} {
w := newWorld(t)
w.merges.manual = true
w.h.PullUpdated(pr(repo, 51, head, "fix/1945", "modules/a/a.go"))
w.h.Checked(judgedAt(repo, 51, head, "pass", "t1t1t1t1t1t1"))
w.settleAll()
w.source.main = &sdk.Source{V: sdk.Version, Group: "before", Heads: map[string]string{repo: "73be3df28cd4"}}
w.forge.pulls[repo+"#51"] = ForgePull{Number: 51, State: "closed", Merged: true, MergeCommit: merge,
MergedAt: w.h.Now().Add(-3 * time.Second).UTC().Format(time.RFC3339), HeadSHA: head, Base: "main"}
w.forge.open = []OpenPull{{Owner: "novox", Repo: "other", Number: 1, Base: "main", HeadSHA: "abab"}}
w.h.wantForge()
w.settleAll()
if oldGate { // what the gate before the fix left
w.h.mu.Lock()
kept := w.h.deliveries[IDOf(repo, head)]
kept.LearntLate, kept.InSource, kept.Ready.SourceMoved = true, true, true
w.h.mu.Unlock()
w.source.main.Heads[repo] = "73be3df28cd4"
w.h.SourceCatchUp()
w.settleAll()
}
d := w.delivery(IDOf(repo, head))
if d.State != Merged || (!oldGate && d.LearntLate) || w.source.main.Heads[repo] != merge {
t.Fatalf("old gate %v: %s, learnt late %v, source.main %v", oldGate, d.State, d.LearntLate, w.source.main.Heads)
}
if heads, _ := w.h.SourceMain()["heads"].(map[string]string); heads[repo] != merge {
t.Fatalf("delivery_status's source-main says %v", w.h.SourceMain())
}
}
}},
// Review round 1, blocker 2 (1275/AC6): an out-of-date refusal at t0 with no other event is said urgent at t0+10
// minutes: the gate asks itself for a look then.
{"an out-of-date refusal turns urgent on the clock alone", func(t *testing.T) {
w := approvedGroup(t)
told := hearConditions(w)
allGreen(w)
w.forge.refuse = errAskedAgain
w.op.answer(w.op.open()[0], OptionApprove)
w.settleAll()
t0 := w.h.Now()
w.h.wanted.mu.Lock()
at := w.h.wanted.at
w.h.wanted.mu.Unlock()
if at.IsZero() || at.Sub(t0) > outOfDateFor {
t.Fatalf("no look is wanted by t0+%s: %v", outOfDateFor, at)
}
for i := 0; i < 5 && w.h.Now().Before(t0.Add(outOfDateFor)); i++ { // the timer fires a look, and nothing else happens
w.h.wanted.mu.Lock()
at = w.h.wanted.at
w.h.wanted.mu.Unlock()
if at.IsZero() {
break
}
w.later(at.Sub(w.h.Now()))
w.h.React()
}
c := told.last(IDOf(gCat, gHk))
if c == nil || c.State != asks.StateRaised || c.Severity != asks.SeverityUrgent {
t.Fatalf("at t0+%s with no event: %+v", w.h.Now().Sub(t0), c)
}
}},
// Review round 1, blocker 3: a gate that restarts while a head still waits on the forge does not clear its
// condition (the head's first sight is in memory alone, so its bound runs again), and does not say it again.
{"a restart keeps the condition of a head still waiting", func(t *testing.T) {
w := newWorld(t)
told := hearConditions(w)
const head = "bfe82315f42c8f4509d0598c32569a202d5eada6"
id := IDOf("novox/hq", head)
w.forge.open = []OpenPull{{Owner: "novox", Repo: "hq", Number: 243, Base: "main", HeadSHA: head, Required: []string{"mesh/repo-check"}}}
w.h.wantForge()
w.settleAll()
w.later(UncheckedAfter + time.Minute)
w.settleAll()
if c := told.last(id); c == nil || c.State != asks.StateRaised {
t.Fatalf("the unchecked head is not raised: %+v", c)
}
before := len(told.told)
again := newWorldOver(t, w.store, w.forge, w.h.Now()) // the gate restarts
again.h.Say, again.h.Conditions = told.say, told
again.h.wantForge()
again.settleAll()
again.later(time.Minute)
again.settleAll()
if len(told.told) != before {
t.Fatalf("the restart said something of a head still waiting: %+v", told.told[before:])
}
w.forge.open = nil // it closes
again.h.wantForge()
again.settleAll()
if c := told.last(id); c == nil || c.State != asks.StateCleared {
t.Fatalf("closed, and its condition is not cleared: %+v", c)
}
}},
// The kind behind the day's fault: a decision on a group's state named a word GroupState no longer says. Every word
// it says is one groupStatusState knows, and the merge goes on exactly in ready and merging.
{"every word of a group's state is decided on", func(t *testing.T) {
for _, word := range groupWords {
if groupStatusState(word) == "pending" && word != GroupChecking {
t.Errorf("groupStatusState does not know %q", word)
}
if may := groupMayMerge(word); may != (word == GroupReady || word == GroupMerging) {
t.Errorf("groupMayMerge(%q) = %v", word, may)
}
}
}},
}
for _, c := range cases {
t.Run(c.name, c.run)
}
}
// allGreenFor makes both merge checks a success at every member's head.
func allGreenFor(w *world, ms ...member) {
for _, m := range ms {
setChecks(w, m.repo, m.head, "success", "success")
}
}
// bridgeMerged is the forge bridge's word that it merged a member.
func bridgeMerged(w *world, repository string, number int, head, merge string) {
w.merges.say(sdk.Forge{Kind: sdk.ForgeMerged, Repository: repository, Number: number, Head: head, Merge: merge, Root: "root",
Trees: map[string]string{"app": "tree"}})
}
@@ -1,15 +1,11 @@
package main package main
import ( import (
"encoding/json"
"go/ast" "go/ast"
"go/parser" "go/parser"
"go/token" "go/token"
"path/filepath" "path/filepath"
"slices"
"strconv"
"strings" "strings"
"sync"
"testing" "testing"
sdk "git.novox.be/novox/mesh-sdk/go/delivery" sdk "git.novox.be/novox/mesh-sdk/go/delivery"
@@ -57,7 +53,6 @@ func TestANoteOnTheMergeCommitIsOwedWithNumberZero(t *testing.T) {
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go")) w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass")) w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge)) w.h.PullMerged(merged("novox/app", 7, head, merge))
w.h.WalkMoved(w.ctl.put(aWalk("plan-1", "novox/app", merge, true)))
w.settleAll() w.settleAll()
key, _ := sdk.OwedKey("novox/app", 0, merge, "note") key, _ := sdk.OwedKey("novox/app", 0, merge, "note")
if n, there := w.owed.keys[key]; !there || n.Act != sdk.OwedNote { if n, there := w.owed.keys[key]; !there || n.Act != sdk.OwedNote {
@@ -86,19 +81,15 @@ func TestTwoCommentsOnOneHeadAreTwoOwedKeys(t *testing.T) {
} }
} }
// AC2: the gate's code calls no gitea write tool and no `delivery-plan`, and no controller verb beyond the ones its manifest // AC2 (R6-C1): the gate's code calls no gitea write tool and no controller verb at all (ADR 0539 §17: the gate asks the
// names (ADR 0450 §7). Tests are left out: they name the words to prove them gone. // controller nothing, and holds no port to it). Tests are left out: they name the words to prove them gone.
func TestTheGatesCodeCallsNoForgeWriteToolAndNoDeliveryPlan(t *testing.T) { func TestTheGatesCodeCallsNoForgeWriteToolAndNoControllerVerb(t *testing.T) {
files, err := filepath.Glob("*.go") files, err := filepath.Glob("*.go")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
forbidden := []string{"delivery-plan", "gitea_commit_status\"", "gitea_delivery_view", "gitea_note_append", "gitea_add_comment", forbidden := []string{"delivery-plan", "gitea_commit_status\"", "gitea_delivery_view", "gitea_note_append", "gitea_add_comment",
"gitea_merge_pull_request\"", "gitea_open_pulls_write"} "gitea_merge_pull_request\"", "gitea_open_pulls_write", "seat:mesh-controller", "plan-moved"}
// The controller's verbs the gate calls: ADR 0450 §7, less the checks and the queue, which the controller no longer
// keeps (issue 1318, ADR 0524 §5).
allowed := []string{"deliver", "delivery-stop", "delivery-backups-again", "delivery-walks", "plans"}
called := map[string]bool{}
fset := token.NewFileSet() fset := token.NewFileSet()
for _, f := range files { for _, f := range files {
if strings.HasSuffix(f, "_test.go") { if strings.HasSuffix(f, "_test.go") {
@@ -118,79 +109,8 @@ func TestTheGatesCodeCallsNoForgeWriteToolAndNoDeliveryPlan(t *testing.T) {
} }
} }
} }
case *ast.CallExpr:
if sel, ok := x.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "verb" && len(x.Args) > 0 {
if lit, ok := x.Args[0].(*ast.BasicLit); ok {
verb, _ := strconv.Unquote(lit.Value)
called[verb] = true
if !slices.Contains(allowed, verb) {
t.Errorf("%s calls the controller's verb %q", f, verb)
}
}
}
} }
return true return true
}) })
} }
for _, verb := range []string{"plans"} {
if !called[verb] {
t.Errorf("the gate calls no %s: the test reads nothing", verb)
}
}
}
// stubController is a controller that answers every verb with one document, and keeps what it was asked.
type stubController struct {
mu sync.Mutex
keys []string
bodies []map[string]any
answer string
}
func (s *stubController) ask(key string, body any) (json.RawMessage, error) {
s.mu.Lock()
defer s.mu.Unlock()
s.keys = append(s.keys, key)
b, _ := body.(map[string]any)
s.bodies = append(s.bodies, b)
return json.RawMessage(s.answer), nil
}
func (s *stubController) called(key string) []map[string]any {
s.mu.Lock()
defer s.mu.Unlock()
var out []map[string]any
for i, k := range s.keys {
if k == key {
out = append(out, s.bodies[i])
}
}
return out
}
// AC3: a delivery with no plan gets it from `plans`, asked as data, with the change's diffset.
func TestADeliveryWithNoPlanGetsItFromPlansAsData(t *testing.T) {
w := newWorld(t)
stub := &stubController{answer: `{"ok":true,"output":"","answer":{"asked":"check-1","held":true,"walks":[],
"plan":{"repository":"novox/app","base":"main","head":"` + head + `","moved":["app"],"summary":"builds app, sent to the node"}}}`}
w.h.Controller = seatController{ask: stub.ask}
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
c := verdict("novox/app", 7, head, "pass")
c.Plan = nil
w.h.Checked(c)
w.settleAll()
asked := stub.called("seat:mesh-controller.plans")
if len(asked) == 0 {
t.Fatalf("`plans` was not asked; the controller was asked %v", stub.keys)
}
if len(stub.called("seat:mesh-controller.delivery-plan")) != 0 {
t.Fatal("delivery-plan was asked")
}
got := asked[0]
if got["repository"] != "novox/app" || got["paths"] != "src/a.go" || got["data"] != "true" || got["head"] != head || got["base"] != "main" {
t.Fatalf("plans was asked %v; wanted the repository, the paths, the head, the base and data", got)
}
if d := w.delivery(IDOf("novox/app", head)); d.Plan == nil || d.Plan.Summary != "builds app, sent to the node" {
t.Fatalf("the delivery's plan is %+v", d.Plan)
}
} }
@@ -1,72 +0,0 @@
package main
import (
"strings"
"testing"
"time"
)
// novox/hq issue 1581: a delivery stopped because its walk ended without building its modules (issue 474) waited for
// "a later delivered merge to carry" them, listed by `stalled` for good. When no later merge can: the pull request
// removed the module from the catalogue (mesh-catalog #384, scan), or the module is built for no machine (#391, "sent
// nowhere"), the stop reads ended, as covered reads covered (issue 480) — a reading of the stop, no transition — says
// why, and leaves `stalled`, so the condition that listed it ends by itself.
func strandedWith(t *testing.T, change func(d *Delivery)) (*world, string) {
w, id := strandedNight(t)
if len(w.h.Stalled()) != 1 {
t.Fatalf("before, stalled lists %+v", w.h.Stalled())
}
w.h.mu.Lock()
d := w.h.deliveries[id]
change(d)
if err := w.store.PutDelivery(d); err != nil {
t.Fatal(err)
}
w.h.mu.Unlock()
w.later(time.Minute)
w.settleAll()
return w, id
}
func TestAStrandedStopWhoseModuleIsSentNowhereEndsWithoutALaterMerge(t *testing.T) {
w, id := strandedWith(t, func(d *Delivery) {
d.Plan.Summary = "builds build-agent, sent nowhere; no bus step"
})
if s := w.h.Stalled(); len(s) != 0 {
t.Fatalf("sent nowhere, stalled still lists %+v", s)
}
shown, err := w.h.Show(id)
if err != nil {
t.Fatal(err)
}
reads, _ := shown.(map[string]any)["reads"].(string)
if !strings.HasPrefix(reads, "ended") || !strings.Contains(reads, "build-agent") {
t.Fatalf("show reads %q", reads)
}
if waits, _ := shown.(map[string]any)["waits"].(string); strings.Contains(waits, "later delivered merge") {
t.Fatalf("it still waits for %q", waits)
}
if d := w.delivery(id); d.State != Stopped {
t.Fatalf("a reading, not a transition: it is %s", d.State)
}
}
func TestAStrandedStopWhoseModuleThePullRequestRemovedEndsWithoutALaterMerge(t *testing.T) {
w, _ := strandedWith(t, func(d *Delivery) {
d.Removed = []string{"modules/build-agent/module.json", "modules/build-agent/cmd/main.go"}
})
if s := w.h.Stalled(); len(s) != 0 {
t.Fatalf("removed, stalled still lists %+v", s)
}
}
// A stop whose module still runs somewhere keeps waiting for a later merge, and stays listed.
func TestAStrandedStopWhoseModuleCanStillBeCarriedStaysListed(t *testing.T) {
w, _ := strandedWith(t, func(d *Delivery) {
d.Removed = []string{"modules/other/module.json"}
})
if s := w.h.Stalled(); len(s) != 1 {
t.Fatalf("stalled lists %+v", s)
}
}

Some files were not shown because too many files have changed in this diff Show More