assigner: a composition step reads its catalog revision once and again on a timeout #232

Merged
mesh-tools merged 1 commits from fix/composition-reads-revision-once into main 2026-11-16 20:32:07 +00:00
Member

139 pipelines started at once (2026-11-16 21:18, source revision 969); their composition steps (seat mesh-wants, ace) each read the whole catalog revision trunk.419 and timed out ('context deadline exceeded', 'nats: timeout'), ending pipelines whose build and tests passed as 'could not run'.

Kind: many composition steps read the same revision's entries through one uncached, unretried call. Now a revision's read is shared (single-flight cache keyed by branch and revision; entries are content-addressed), and a transport fault or timeout is retried with backoff (1 s doubling to 15 s, at most 3 min, each try in the journal) before 'could not run'.

  • Quick fix: a longer timeout.
  • Thorough fix (chosen): read once per revision and retry transient faults.

Test: TestTheCompositionStepReadsTheRevisionOnceAndAgainOnATimeout (4 rows). Recovery mode: merged directly; placed by hand on ace as pack28 (assigner-program).

139 pipelines started at once (2026-11-16 21:18, source revision 969); their composition steps (seat mesh-wants, ace) each read the whole catalog revision trunk.419 and timed out ('context deadline exceeded', 'nats: timeout'), ending pipelines whose build and tests passed as 'could not run'. Kind: many composition steps read the same revision's entries through one uncached, unretried call. Now a revision's read is shared (single-flight cache keyed by branch and revision; entries are content-addressed), and a transport fault or timeout is retried with backoff (1 s doubling to 15 s, at most 3 min, each try in the journal) before 'could not run'. - Quick fix: a longer timeout. - Thorough fix (chosen): read once per revision and retry transient faults. Test: TestTheCompositionStepReadsTheRevisionOnceAndAgainOnATimeout (4 rows). Recovery mode: merged directly; placed by hand on ace as pack28 (assigner-program).
mesh-tools added 1 commit 2026-11-16 20:32:04 +00:00
mesh-tools merged commit 38f1c2f10a into main 2026-11-16 20:32:07 +00:00
mesh-tools deleted branch fix/composition-reads-revision-once 2026-11-16 20:32:08 +00:00
Author
Member

{
"records": [
{
"instance": "record/main_970_7fb35e5d/12640b201d6c73bf",
"sequence": 9432,
"bytes": 2847,
"of-bytes": 262144,
"waits": "its difference is not known yet",
"record": {
"milestone": "main.970",
"heads": {
"novox/mesh-delivery@232": "38f1c2f10a476f0cb557f177c538e20c152de854"
},
"opened": "2026-11-16T20:44:19.514618252Z",
"form": 1,
"group": {
"group": "",
"members": [
"novox/mesh-catalog@694",
"novox/mesh-delivery@232",
"novox/mesh-host@231",
"novox/mesh-tools@73"
],
"pulls": [
{
"repository": "novox/mesh-delivery",
"number": 232,
"head": "38f1c2f10a476f0cb557f177c538e20c152de854",
"state": "merged",
"commit": "38f1c2f10a476f0cb557f177c538e20c152de854",
"base": "main"
}
]
},
"entries": {
"mesh-build-go|src2:9e7b7ccdb71ecd1150ecd44d80b3bcfa42be43b21b92bbd31523cb5933335043|": {
"key": "mesh-build-go|src2:9e7b7ccdb71ecd1150ecd44d80b3bcfa42be43b21b92bbd31523cb5933335043|",
"module": "mesh-build-go",
"fingerprint": "src2:9e7b7ccdb71ecd1150ecd44d80b3bcfa42be43b21b92bbd31523cb5933335043"
},
"mesh-delivery-assigner|src2:5b2ec30c52c6c8ab8aaee19e0d215b4c539cdd162822ffe11cd3819208e09561|": {
"key": "mesh-delivery-assigner|src2:5b2ec30c52c6c8ab8aaee19e0d215b4c539cdd162822ffe11cd3819208e09561|",
"module": "mesh-delivery-assigner",
"fingerprint": "src2:5b2ec30c52c6c8ab8aaee19e0d215b4c539cdd162822ffe11cd3819208e09561"
},
"mesh-delivery-pipeline|src2:baa4d89cf2851bf146e90272c245609e12e6d498f4d230936acb68b32d881ea5|": {
"key": "mesh-delivery-pipeline|src2:baa4d89cf2851bf146e90272c245609e12e6d498f4d230936acb68b32d881ea5|",
"module": "mesh-delivery-pipeline",
"fingerprint": "src2:baa4d89cf2851bf146e90272c245609e12e6d498f4d230936acb68b32d881ea5"
},
"mesh-delivery-teller|src2:73b0e0091096ce92b438fd0fd81c6db8ce62057f93b650af88cad4766d22338b|": {
"key": "mesh-delivery-teller|src2:73b0e0091096ce92b438fd0fd81c6db8ce62057f93b650af88cad4766d22338b|",
"module": "mesh-delivery-teller",
"fingerprint": "src2:73b0e0091096ce92b438fd0fd81c6db8ce62057f93b650af88cad4766d22338b"
},
"mesh-host|src2:86640fb59ef947e2ac9077693c6554ab02a5e7cadb086ceb28d45166116c1308|": {
"key": "mesh-host|src2:86640fb59ef947e2ac9077693c6554ab02a5e7cadb086ceb28d45166116c1308|",
"module": "mesh-host",
"fingerprint": "src2:86640fb59ef947e2ac9077693c6554ab02a5e7cadb086ceb28d45166116c1308"
},
"node-tools|src2:57c38bad9a3e9539e5142ae5cb68ea8219742b40c83baa1065d63f51585fb579|": {
"key": "node-tools|src2:57c38bad9a3e9539e5142ae5cb68ea8219742b40c83baa1065d63f51585fb579|",
"module": "node-tools",
"fingerprint": "src2:57c38bad9a3e9539e5142ae5cb68ea8219742b40c83baa1065d63f51585fb579"
}
},
"outcomes": {
"|||take-in": {
"key": "|||take-in",
"module": "",
"step": "take-in",
"outcome": "taken-in",
"why": "lead-47@shanks for the operator (G recovery, 'as long as G is not delivered, you continue'): 8 merges since 20:00 (teller #228/#229, indexer #230, pipeline runner #231, composition #232, build-go #694, mesh-host #231, node-tools probe #73) wait on a revision validation that restarts with each merge and queues behind 139 pipelines' steps; node-tools must get a pipeline to be released with mesh-sdk #197",
"at": "2026-11-16T20:44:19.506430649Z"
}
}
}
}
]
}

<!-- mesh-delivery:view --> { "records": [ { "instance": "record/main_970_7fb35e5d/12640b201d6c73bf", "sequence": 9432, "bytes": 2847, "of-bytes": 262144, "waits": "its difference is not known yet", "record": { "milestone": "main.970", "heads": { "novox/mesh-delivery@232": "38f1c2f10a476f0cb557f177c538e20c152de854" }, "opened": "2026-11-16T20:44:19.514618252Z", "form": 1, "group": { "group": "", "members": [ "novox/mesh-catalog@694", "novox/mesh-delivery@232", "novox/mesh-host@231", "novox/mesh-tools@73" ], "pulls": [ { "repository": "novox/mesh-delivery", "number": 232, "head": "38f1c2f10a476f0cb557f177c538e20c152de854", "state": "merged", "commit": "38f1c2f10a476f0cb557f177c538e20c152de854", "base": "main" } ] }, "entries": { "mesh-build-go|src2:9e7b7ccdb71ecd1150ecd44d80b3bcfa42be43b21b92bbd31523cb5933335043|": { "key": "mesh-build-go|src2:9e7b7ccdb71ecd1150ecd44d80b3bcfa42be43b21b92bbd31523cb5933335043|", "module": "mesh-build-go", "fingerprint": "src2:9e7b7ccdb71ecd1150ecd44d80b3bcfa42be43b21b92bbd31523cb5933335043" }, "mesh-delivery-assigner|src2:5b2ec30c52c6c8ab8aaee19e0d215b4c539cdd162822ffe11cd3819208e09561|": { "key": "mesh-delivery-assigner|src2:5b2ec30c52c6c8ab8aaee19e0d215b4c539cdd162822ffe11cd3819208e09561|", "module": "mesh-delivery-assigner", "fingerprint": "src2:5b2ec30c52c6c8ab8aaee19e0d215b4c539cdd162822ffe11cd3819208e09561" }, "mesh-delivery-pipeline|src2:baa4d89cf2851bf146e90272c245609e12e6d498f4d230936acb68b32d881ea5|": { "key": "mesh-delivery-pipeline|src2:baa4d89cf2851bf146e90272c245609e12e6d498f4d230936acb68b32d881ea5|", "module": "mesh-delivery-pipeline", "fingerprint": "src2:baa4d89cf2851bf146e90272c245609e12e6d498f4d230936acb68b32d881ea5" }, "mesh-delivery-teller|src2:73b0e0091096ce92b438fd0fd81c6db8ce62057f93b650af88cad4766d22338b|": { "key": "mesh-delivery-teller|src2:73b0e0091096ce92b438fd0fd81c6db8ce62057f93b650af88cad4766d22338b|", "module": "mesh-delivery-teller", "fingerprint": "src2:73b0e0091096ce92b438fd0fd81c6db8ce62057f93b650af88cad4766d22338b" }, "mesh-host|src2:86640fb59ef947e2ac9077693c6554ab02a5e7cadb086ceb28d45166116c1308|": { "key": "mesh-host|src2:86640fb59ef947e2ac9077693c6554ab02a5e7cadb086ceb28d45166116c1308|", "module": "mesh-host", "fingerprint": "src2:86640fb59ef947e2ac9077693c6554ab02a5e7cadb086ceb28d45166116c1308" }, "node-tools|src2:57c38bad9a3e9539e5142ae5cb68ea8219742b40c83baa1065d63f51585fb579|": { "key": "node-tools|src2:57c38bad9a3e9539e5142ae5cb68ea8219742b40c83baa1065d63f51585fb579|", "module": "node-tools", "fingerprint": "src2:57c38bad9a3e9539e5142ae5cb68ea8219742b40c83baa1065d63f51585fb579" } }, "outcomes": { "|||take-in": { "key": "|||take-in", "module": "", "step": "take-in", "outcome": "taken-in", "why": "lead-47@shanks for the operator (G recovery, 'as long as G is not delivered, you continue'): 8 merges since 20:00 (teller #228/#229, indexer #230, pipeline runner #231, composition #232, build-go #694, mesh-host #231, node-tools probe #73) wait on a revision validation that restarts with each merge and queues behind 139 pipelines' steps; node-tools must get a pipeline to be released with mesh-sdk #197", "at": "2026-11-16T20:44:19.506430649Z" } } } } ] } <!-- mesh-act:view.record/main_970_7fb35e5d/12640b201d6c73bf.novox/mesh-delivery#232@1794861865382818267.2c293766 -->
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-delivery#232