nats: the tools bundle answers initialize again #696

Merged
mesh-tools merged 1 commits from fix/nats-starts-again into main 2026-11-16 21:29:13 +00:00
Member

Recovery under design 47 G (lead-47@shanks, 2026-11-16 22:35 CET).

Symptom: on novox at 22:21:57 and 22:26:45 node-tools wrote "nats's bundle did not answer initialize in 10s; its tools are not served here" after hand packs 29 and 30 placed a build of the nats tools from main.

Cause (goroutine dump): bundleKeepDays (A-112, 8d6b7fbb) reads the config value snapshot-keep-days inside allTools, before stdio.Serve; stdio.Config waits for the runtime's channel, which opens only once the bundle serves, so the bundle blocks for ever. Every build of the nats tools since A-112 has this; the bundle running before 22:21 was older.

Fix: the setting is read when nats_snapshots is called. Locally the bundle now answers initialize at once; go test ./cmd/nats-tools passes.

Quick vs thorough: quick is this one call site. Thorough is mesh-sdk: a config, state or provision read before Serve fails loudly at once instead of blocking (every bundle can meet it). Noted for the next sdk change; not done tonight.

Recovery under design 47 G (lead-47@shanks, 2026-11-16 22:35 CET). Symptom: on novox at 22:21:57 and 22:26:45 node-tools wrote "nats's bundle did not answer initialize in 10s; its tools are not served here" after hand packs 29 and 30 placed a build of the nats tools from main. Cause (goroutine dump): `bundleKeepDays` (A-112, 8d6b7fbb) reads the config value snapshot-keep-days inside `allTools`, before `stdio.Serve`; `stdio.Config` waits for the runtime's channel, which opens only once the bundle serves, so the bundle blocks for ever. Every build of the nats tools since A-112 has this; the bundle running before 22:21 was older. Fix: the setting is read when `nats_snapshots` is called. Locally the bundle now answers initialize at once; `go test ./cmd/nats-tools` passes. Quick vs thorough: quick is this one call site. Thorough is mesh-sdk: a config, state or provision read before Serve fails loudly at once instead of blocking (every bundle can meet it). Noted for the next sdk change; not done tonight.
mesh-tools added 1 commit 2026-11-16 21:29:06 +00:00
bundleKeepDays read the config value snapshot-keep-days while allTools
listed the tools, before stdio.Serve; a config read waits for the runtime's
channel, which opens only once the bundle serves, so the bundle never
answered initialize and node-tools served none of its tools (novox
2026-11-16 22:21 and 22:26, after the first build from main since A-112).
The setting is now read when nats_snapshots is called.
mesh-tools merged commit 3f3f6789dd into main 2026-11-16 21:29:13 +00:00
mesh-tools deleted branch fix/nats-starts-again 2026-11-16 21:29:13 +00:00
Author
Member

{
"records": [
{
"instance": "record/main_978_2dad35d4/e7d7ab585e6eb524",
"sequence": 9485,
"bytes": 1390,
"of-bytes": 262144,
"waits": "its difference is not known yet",
"record": {
"milestone": "main.978",
"heads": {
"novox/mesh-catalog@696": "3f3f6789dd07c7ff6c18b913f8781aa458f17f84"
},
"opened": "2026-11-16T22:01:27.393789538Z",
"form": 1,
"group": {
"group": "",
"members": [
"novox/mesh-catalog@696",
"novox/mesh-tools@74"
],
"pulls": [
{
"repository": "novox/mesh-catalog",
"number": 696,
"head": "3f3f6789dd07c7ff6c18b913f8781aa458f17f84",
"state": "merged",
"commit": "3f3f6789dd07c7ff6c18b913f8781aa458f17f84"
}
]
},
"entries": {
"nats|src2:6dd7434409832e1a3330a7a27d6fa526ab1043e6bf2c466314a0fff37fb7d321|": {
"key": "nats|src2:6dd7434409832e1a3330a7a27d6fa526ab1043e6bf2c466314a0fff37fb7d321|",
"module": "nats",
"fingerprint": "src2:6dd7434409832e1a3330a7a27d6fa526ab1043e6bf2c466314a0fff37fb7d321"
},
"node-tools|src2:b5c7a048a73a1f6aa03758d1708940f8a9e7edc33056a61d5594a8d4b3ca7369|": {
"key": "node-tools|src2:b5c7a048a73a1f6aa03758d1708940f8a9e7edc33056a61d5594a8d4b3ca7369|",
"module": "node-tools",
"fingerprint": "src2:b5c7a048a73a1f6aa03758d1708940f8a9e7edc33056a61d5594a8d4b3ca7369"
}
},
"outcomes": {
"|||take-in": {
"key": "|||take-in",
"module": "",
"step": "take-in",
"outcome": "taken-in",
"why": "lead-47@shanks: the wait holds mesh-catalog #695/#696 and mesh-tools #74 (node-tools probe) for an indexer build src2:4c7fa765 that is not coming tonight (no new indexer build before node-tools is released); taken in with the running indexer so node-tools gets its pipeline",
"at": "2026-11-16T22:01:27.390969845Z"
}
}
}
}
]
}

<!-- mesh-delivery:view --> { "records": [ { "instance": "record/main_978_2dad35d4/e7d7ab585e6eb524", "sequence": 9485, "bytes": 1390, "of-bytes": 262144, "waits": "its difference is not known yet", "record": { "milestone": "main.978", "heads": { "novox/mesh-catalog@696": "3f3f6789dd07c7ff6c18b913f8781aa458f17f84" }, "opened": "2026-11-16T22:01:27.393789538Z", "form": 1, "group": { "group": "", "members": [ "novox/mesh-catalog@696", "novox/mesh-tools@74" ], "pulls": [ { "repository": "novox/mesh-catalog", "number": 696, "head": "3f3f6789dd07c7ff6c18b913f8781aa458f17f84", "state": "merged", "commit": "3f3f6789dd07c7ff6c18b913f8781aa458f17f84" } ] }, "entries": { "nats|src2:6dd7434409832e1a3330a7a27d6fa526ab1043e6bf2c466314a0fff37fb7d321|": { "key": "nats|src2:6dd7434409832e1a3330a7a27d6fa526ab1043e6bf2c466314a0fff37fb7d321|", "module": "nats", "fingerprint": "src2:6dd7434409832e1a3330a7a27d6fa526ab1043e6bf2c466314a0fff37fb7d321" }, "node-tools|src2:b5c7a048a73a1f6aa03758d1708940f8a9e7edc33056a61d5594a8d4b3ca7369|": { "key": "node-tools|src2:b5c7a048a73a1f6aa03758d1708940f8a9e7edc33056a61d5594a8d4b3ca7369|", "module": "node-tools", "fingerprint": "src2:b5c7a048a73a1f6aa03758d1708940f8a9e7edc33056a61d5594a8d4b3ca7369" } }, "outcomes": { "|||take-in": { "key": "|||take-in", "module": "", "step": "take-in", "outcome": "taken-in", "why": "lead-47@shanks: the wait holds mesh-catalog #695/#696 and mesh-tools #74 (node-tools probe) for an indexer build src2:4c7fa765 that is not coming tonight (no new indexer build before node-tools is released); taken in with the running indexer so node-tools gets its pipeline", "at": "2026-11-16T22:01:27.390969845Z" } } } } ] } <!-- mesh-act:view.record/main_978_2dad35d4/e7d7ab585e6eb524.novox/mesh-catalog#696@1794866493238121572.aa9d67b8 -->
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-catalog#696