139 pipelines started at once (2026-11-16 21:18, source revision 969); their composition steps (seat mesh-wants, ace) each read the whole catalog revision trunk.419 and timed out ('context deadline exceeded', 'nats: timeout'), ending pipelines whose build and tests passed as 'could not run'.
Kind: many composition steps read the same revision's entries through one uncached, unretried call. Now a revision's read is shared (single-flight cache keyed by branch and revision; entries are content-addressed), and a transport fault or timeout is retried with backoff (1 s doubling to 15 s, at most 3 min, each try in the journal) before 'could not run'.
Quick fix: a longer timeout.
Thorough fix (chosen): read once per revision and retry transient faults.
Test: TestTheCompositionStepReadsTheRevisionOnceAndAgainOnATimeout (4 rows). Recovery mode: merged directly; placed by hand on ace as pack28 (assigner-program).
139 pipelines started at once (2026-11-16 21:18, source revision 969); their composition steps (seat mesh-wants, ace) each read the whole catalog revision trunk.419 and timed out ('context deadline exceeded', 'nats: timeout'), ending pipelines whose build and tests passed as 'could not run'.
Kind: many composition steps read the same revision's entries through one uncached, unretried call. Now a revision's read is shared (single-flight cache keyed by branch and revision; entries are content-addressed), and a transport fault or timeout is retried with backoff (1 s doubling to 15 s, at most 3 min, each try in the journal) before 'could not run'.
- Quick fix: a longer timeout.
- Thorough fix (chosen): read once per revision and retry transient faults.
Test: TestTheCompositionStepReadsTheRevisionOnceAndAgainOnATimeout (4 rows). Recovery mode: merged directly; placed by hand on ace as pack28 (assigner-program).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
139 pipelines started at once (2026-11-16 21:18, source revision 969); their composition steps (seat mesh-wants, ace) each read the whole catalog revision trunk.419 and timed out ('context deadline exceeded', 'nats: timeout'), ending pipelines whose build and tests passed as 'could not run'.
Kind: many composition steps read the same revision's entries through one uncached, unretried call. Now a revision's read is shared (single-flight cache keyed by branch and revision; entries are content-addressed), and a transport fault or timeout is retried with backoff (1 s doubling to 15 s, at most 3 min, each try in the journal) before 'could not run'.
Test: TestTheCompositionStepReadsTheRevisionOnceAndAgainOnATimeout (4 rows). Recovery mode: merged directly; placed by hand on ace as pack28 (assigner-program).
{
"records": [
{
"instance": "record/main_970_7fb35e5d/12640b201d6c73bf",
"sequence": 9432,
"bytes": 2847,
"of-bytes": 262144,
"waits": "its difference is not known yet",
"record": {
"milestone": "main.970",
"heads": {
"novox/mesh-delivery@232": "38f1c2f10a476f0cb557f177c538e20c152de854"
},
"opened": "2026-11-16T20:44:19.514618252Z",
"form": 1,
"group": {
"group": "",
"members": [
"novox/mesh-catalog@694",
"novox/mesh-delivery@232",
"novox/mesh-host@231",
"novox/mesh-tools@73"
],
"pulls": [
{
"repository": "novox/mesh-delivery",
"number": 232,
"head": "38f1c2f10a476f0cb557f177c538e20c152de854",
"state": "merged",
"commit": "38f1c2f10a476f0cb557f177c538e20c152de854",
"base": "main"
}
]
},
"entries": {
"mesh-build-go|src2:9e7b7ccdb71ecd1150ecd44d80b3bcfa42be43b21b92bbd31523cb5933335043|": {
"key": "mesh-build-go|src2:9e7b7ccdb71ecd1150ecd44d80b3bcfa42be43b21b92bbd31523cb5933335043|",
"module": "mesh-build-go",
"fingerprint": "src2:9e7b7ccdb71ecd1150ecd44d80b3bcfa42be43b21b92bbd31523cb5933335043"
},
"mesh-delivery-assigner|src2:5b2ec30c52c6c8ab8aaee19e0d215b4c539cdd162822ffe11cd3819208e09561|": {
"key": "mesh-delivery-assigner|src2:5b2ec30c52c6c8ab8aaee19e0d215b4c539cdd162822ffe11cd3819208e09561|",
"module": "mesh-delivery-assigner",
"fingerprint": "src2:5b2ec30c52c6c8ab8aaee19e0d215b4c539cdd162822ffe11cd3819208e09561"
},
"mesh-delivery-pipeline|src2:baa4d89cf2851bf146e90272c245609e12e6d498f4d230936acb68b32d881ea5|": {
"key": "mesh-delivery-pipeline|src2:baa4d89cf2851bf146e90272c245609e12e6d498f4d230936acb68b32d881ea5|",
"module": "mesh-delivery-pipeline",
"fingerprint": "src2:baa4d89cf2851bf146e90272c245609e12e6d498f4d230936acb68b32d881ea5"
},
"mesh-delivery-teller|src2:73b0e0091096ce92b438fd0fd81c6db8ce62057f93b650af88cad4766d22338b|": {
"key": "mesh-delivery-teller|src2:73b0e0091096ce92b438fd0fd81c6db8ce62057f93b650af88cad4766d22338b|",
"module": "mesh-delivery-teller",
"fingerprint": "src2:73b0e0091096ce92b438fd0fd81c6db8ce62057f93b650af88cad4766d22338b"
},
"mesh-host|src2:86640fb59ef947e2ac9077693c6554ab02a5e7cadb086ceb28d45166116c1308|": {
"key": "mesh-host|src2:86640fb59ef947e2ac9077693c6554ab02a5e7cadb086ceb28d45166116c1308|",
"module": "mesh-host",
"fingerprint": "src2:86640fb59ef947e2ac9077693c6554ab02a5e7cadb086ceb28d45166116c1308"
},
"node-tools|src2:57c38bad9a3e9539e5142ae5cb68ea8219742b40c83baa1065d63f51585fb579|": {
"key": "node-tools|src2:57c38bad9a3e9539e5142ae5cb68ea8219742b40c83baa1065d63f51585fb579|",
"module": "node-tools",
"fingerprint": "src2:57c38bad9a3e9539e5142ae5cb68ea8219742b40c83baa1065d63f51585fb579"
}
},
"outcomes": {
"|||take-in": {
"key": "|||take-in",
"module": "",
"step": "take-in",
"outcome": "taken-in",
"why": "lead-47@shanks for the operator (G recovery, 'as long as G is not delivered, you continue'): 8 merges since 20:00 (teller #228/#229, indexer #230, pipeline runner #231, composition #232, build-go #694, mesh-host #231, node-tools probe #73) wait on a revision validation that restarts with each merge and queues behind 139 pipelines' steps; node-tools must get a pipeline to be released with mesh-sdk #197",
"at": "2026-11-16T20:44:19.506430649Z"
}
}
}
}
]
}