A kept call is called again through a transport fault (ADR 0668; issue 1966; group fix/1966-call-made-once, merge first) #185

2026-11-10T04:12:30Z

mesh-tools commented

2026-11-10 04:12:30 +00:00

Issue 1966 (notes 152-160), decision ADR 0668 (accepted): on 2026-11-09/10 four parts each made one call to another around a restart and never called again (the node-engine's seat list, the builder's ask put, the indexer's merge message, the assigner's manifest-reads), each recovered only by a restart on the operator's yes. Two causes: the process call retried no responders only during a bus blackout and never a deadline; a kept answer was not asked again when the callee said it changed.

Group fix/1966-call-made-once, one check, all members pushed together: mesh-sdk (merge first), mesh-contracts, mesh-host, mesh-delivery.

mesh-sdk: busretry.DoRepeatable (also a deadline), process CallKept/CallKeptOn (no responders without the blackout probe), nats.ErrNoStreamResponse counted as no responders, the seat-list reader called again on a deadline.
mesh-contracts: enginereads keeps the last answer at a fault while the kept call goes on, asks every node at the consumer's bind, and carries ReadsCalls (call-unanswered after five minutes while the machine is heard, none while away); meshconditionsclient's call-unanswered kind; the operator-channel seat declares its asks verb's input and answer; the mesh-watchdogs seat declares presence's answer.
mesh-host: the node-engine says its events once under a message id through a transport fault, and says an unsaid one at the next link.
mesh-delivery: the assigner reads operator-channel's asks at its consumer's bind and asks again an ask that ended, gains assigner_ask_again; the teller keeps a failed assigner-ask publish; the builder's ask put is a kept call, a key left building is call-unanswered and builder_build_again takes it; the indexer's merge message is a kept call (unread, not lost; call-unanswered while the forge is heard), drop_pending refuses a merge no later merge covers (changes ADR 0633 §5), a take-in waits for the verdicts state and keeps its new manifests unjudged until judged, a judge by digest of the trunk's manifest says its event; the indexer and builder ask mesh-watchdogs.presence (no read of the watchman's state, ADR 0596 §2); the watchman serves presence by machine.

Quick fix: none (each was a restart on the operator's yes; hand acts in the log, cause seat-list-unread / ask-once-transient). Thorough fix: this group, met four times in one night.

Tests: every change is a row or subtest of an existing test and fails without it; no test function added (mesh-delivery and mesh-host stay over budget, unchanged); go vet, vet -tags integration, go test and go test -tags integration pass in every touched module.

Manifest validator: re-created with the live seat list (60 seats) plus the head's watchdogs and operator-channel seats and the head's assigner, builder, indexer and watchman manifests: 167 of 167 accepted in both cases, no verdict differs. The live verdict comes from this check.

Not built, noted on 1966: the watchdogs seat's client has no Go package artifact, so the presence helper is copied in the builder and the indexer; the assigner's older read of the watchman's presence stays under ADR 0596 §3.

Reviewed-by: mesh-code-reviewer@shanks (3 rounds and a narrow check of round 3's fixes) at mesh-sdk 13998c2362, mesh-contracts bb16f99e2d5e854bb8f6552ce4b256e7c10e8ea0, mesh-host 9cd03a7430cc6b7dca3f5a97a08a5a799a84c673, mesh-delivery 262d21c431b3682f91190363689ab8f58cdb5032
Pushed-by: lead-47@shanks

https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ

Issue 1966 (notes 152-160), decision ADR 0668 (accepted): on 2026-11-09/10 four parts each made one call to another around a restart and never called again (the node-engine's seat list, the builder's ask put, the indexer's merge message, the assigner's manifest-reads), each recovered only by a restart on the operator's yes. Two causes: the process call retried no responders only during a bus blackout and never a deadline; a kept answer was not asked again when the callee said it changed. Group fix/1966-call-made-once, one check, all members pushed together: mesh-sdk (merge first), mesh-contracts, mesh-host, mesh-delivery. - mesh-sdk: busretry.DoRepeatable (also a deadline), process CallKept/CallKeptOn (no responders without the blackout probe), nats.ErrNoStreamResponse counted as no responders, the seat-list reader called again on a deadline. - mesh-contracts: enginereads keeps the last answer at a fault while the kept call goes on, asks every node at the consumer's bind, and carries ReadsCalls (call-unanswered after five minutes while the machine is heard, none while away); meshconditionsclient's call-unanswered kind; the operator-channel seat declares its asks verb's input and answer; the mesh-watchdogs seat declares presence's answer. - mesh-host: the node-engine says its events once under a message id through a transport fault, and says an unsaid one at the next link. - mesh-delivery: the assigner reads operator-channel's asks at its consumer's bind and asks again an ask that ended, gains assigner_ask_again; the teller keeps a failed assigner-ask publish; the builder's ask put is a kept call, a key left building is call-unanswered and builder_build_again takes it; the indexer's merge message is a kept call (unread, not lost; call-unanswered while the forge is heard), drop_pending refuses a merge no later merge covers (changes ADR 0633 §5), a take-in waits for the verdicts state and keeps its new manifests unjudged until judged, a judge by digest of the trunk's manifest says its event; the indexer and builder ask mesh-watchdogs.presence (no read of the watchman's state, ADR 0596 §2); the watchman serves presence by machine. Quick fix: none (each was a restart on the operator's yes; hand acts in the log, cause seat-list-unread / ask-once-transient). Thorough fix: this group, met four times in one night. Tests: every change is a row or subtest of an existing test and fails without it; no test function added (mesh-delivery and mesh-host stay over budget, unchanged); go vet, vet -tags integration, go test and go test -tags integration pass in every touched module. Manifest validator: re-created with the live seat list (60 seats) plus the head's watchdogs and operator-channel seats and the head's assigner, builder, indexer and watchman manifests: 167 of 167 accepted in both cases, no verdict differs. The live verdict comes from this check. Not built, noted on 1966: the watchdogs seat's client has no Go package artifact, so the presence helper is copied in the builder and the indexer; the assigner's older read of the watchman's presence stays under ADR 0596 §3. Reviewed-by: mesh-code-reviewer@shanks (3 rounds and a narrow check of round 3's fixes) at mesh-sdk 13998c2362d35f0d150e5cdc2d6234800fb5519e, mesh-contracts bb16f99e2d5e854bb8f6552ce4b256e7c10e8ea0, mesh-host 9cd03a7430cc6b7dca3f5a97a08a5a799a84c673, mesh-delivery 262d21c431b3682f91190363689ab8f58cdb5032 Pushed-by: lead-47@shanks https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ

mesh-tools added 5 commits 2026-11-10 04:12:31 +00:00

S1: busretry asks a call marked safe to repeat again on a timeout too (issue 1966, ADR 0668) 341bb67159

DoRepeatable waits out a timeout as well as no responders, with an optional blackout probe, so a
kept read is not lost when its callee restarts. Do is unchanged.

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ

S2: CallKept and CallKeptOn ask a kept call again through a callee's restart (issue 1966, ADR 0668) 23ecaddcf7

Call fails no responders at once on a bus that answers; a callee that re-serves answers no one for
seconds, and the caller kept nothing. CallKept asks again within busretry.Bound.

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ

S3: the seat-list reader asks again on a timeout (issue 1966, ADR 0668) 9f9e34301a

Closes the gap #184 left: the read is safe to repeat, so a timed-out ask is asked again.

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ

S4: say that a handler ending a kept call's wait also calls at the consumer's bind (issue 1966, ADR 0668) 6e66a6176c

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ

S5: the old JetStream client's no response from the stream is a request that reached no one (issue 1966, ADR 0668 §1; review round 1, finding 2)

mesh/check failed: mesh-delivery-checker: no published of the tagged version of go-meshconditions of mesh-health-and-re… | 13998c2362d3 on 5b63c67ce852

mesh/delivery-group group fix/1966-call-made-once rejected: composed together: the checker's verdict at the members' heads (c54b617d200d): mesh-delivery-checker

Details

mesh/delivery closed: a newer head of the same pull request

Details

13998c2362

The node-engine's natsLink publishes through the old client, which says no responders as nats.ErrNoStreamResponse: sayOnce gave up at once.

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ

mesh-tools commented

2026-11-10 04:12:33 +00:00

Delivery novox/mesh-sdk@32b14ce0ef4b — merged since 2026-11-10T07:20:27Z

Source — source.main moved to novox/mesh-sdk@317605963f. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Group fix/1966-call-made-once: novox/mesh-contracts@bb16f99e2d, novox/mesh-delivery@262d21c431, novox/mesh-host@d581acdeda, novox/mesh-sdk@32b14ce0ef

composed together: fail — the checker's verdict at the members' heads (6985553d412c): mesh-delivery-checker

Transitions

2026-11-10 05:03 (new) → proposed (announced): novox/mesh-sdk#185's head announced
2026-11-10 05:03 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-11-10 07:20 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-sdk@32b14ce0ef4b` — **merged** since 2026-11-10T07:20:27Z **Source** — source.main moved to novox/mesh-sdk@317605963ff1. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Group** `fix/1966-call-made-once`: novox/mesh-contracts@bb16f99e2d5e, novox/mesh-delivery@262d21c431b3, novox/mesh-host@d581acdedae3, novox/mesh-sdk@32b14ce0ef4b - composed together: fail — the checker's verdict at the members' heads (6985553d412c): mesh-delivery-checker **Transitions** - 2026-11-10 05:03 (new) → proposed (announced): novox/mesh-sdk#185's head announced - 2026-11-10 05:03 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-11-10 07:20 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools commented

2026-11-10 04:12:47 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: a head it covers moved on: the heads it covers are no longer the group's heads. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-tools commented

2026-11-10 04:12:54 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: a head it covers moved on: the heads it covers are no longer the group's heads. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-tools commented

2026-11-10 04:13:04 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: a head it covers moved on: the heads it covers are no longer the group's heads. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

jschoubben added 1 commit 2026-11-10 05:03:04 +00:00

S6: the bootstrapped mesh-watchdogs and operator-channel are the contracts' files at the group's head (issue 1966, ADR 0635 §3)

mesh/check checking 32b14ce0ef4b on 317605963ff1 | novox/mesh-sdk#185 in group fix/1966-call-made-once

mesh/delivery merged: source.main moved to novox/mesh-sdk@317605963ff1

Details

mesh/delivery-group group fix/1966-call-made-once merged: every member is merged

Details

32b14ce0ef

The group's contracts head changed both seats' verbs (presence's answer, asks' input and answer), so the
bootstrap copies, the tests' seat list and the captured copies follow mesh-contracts bb16f99e; the checker's
try of fix/1966-call-made-once found them apart.

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ

mesh-tools commented

2026-11-10 05:03:12 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: its head 9cd03a7430cc is no longer the pull request's head, or the pull request closed. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-tools commented

2026-11-10 05:03:14 +00:00

Heads replaced after the checker's try (try-1794284640769845068) failed on two things, both fixed and reviewed: mesh-sdk S6 32b14ce0 (the seatlist bootstrap and test copies of mesh-watchdogs and operator-channel are now mesh-contracts' files at bb16f99e, ADR 0635 §3) and mesh-host H4 d581acde (the toolchain resolution tests' stand-in machine gains seat-settle-delay, which mesh-catalog 6b8e7fc6 made mesh-build-go/-node require; these tests failed on main too, issue 1748 note 30). The group's earlier check (c54b617d200d) never reached the checker (issue 1966 note 161).

Reviewed-by: mesh-code-reviewer@shanks (narrow check of the two fixes, after 3 rounds) at mesh-sdk 32b14ce0ef, mesh-contracts bb16f99e2d5e854bb8f6552ce4b256e7c10e8ea0, mesh-host d581acdedae3b52afbb1e2a737b7f1282145bd21, mesh-delivery 262d21c431b3682f91190363689ab8f58cdb5032
Pushed-by: lead-47@shanks

Heads replaced after the checker's try (try-1794284640769845068) failed on two things, both fixed and reviewed: mesh-sdk S6 32b14ce0 (the seatlist bootstrap and test copies of mesh-watchdogs and operator-channel are now mesh-contracts' files at bb16f99e, ADR 0635 §3) and mesh-host H4 d581acde (the toolchain resolution tests' stand-in machine gains seat-settle-delay, which mesh-catalog 6b8e7fc6 made mesh-build-go/-node require; these tests failed on main too, issue 1748 note 30). The group's earlier check (c54b617d200d) never reached the checker (issue 1966 note 161). Reviewed-by: mesh-code-reviewer@shanks (narrow check of the two fixes, after 3 rounds) at mesh-sdk 32b14ce0ef4b2f77c3c55be4f838a4d2ae1622d2, mesh-contracts bb16f99e2d5e854bb8f6552ce4b256e7c10e8ea0, mesh-host d581acdedae3b52afbb1e2a737b7f1282145bd21, mesh-delivery 262d21c431b3682f91190363689ab8f58cdb5032 Pushed-by: lead-47@shanks

mesh-tools merged commit 317605963f into main

2026-11-10 07:20:23 +00:00

mesh-tools deleted branch fix/1966-call-made-once

2026-11-10 07:20:23 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-sdk#185