The node-engine retries an apply that met a passing bus error, so a reload does not leave a module failed (issue 1945) #144

2026-10-25T22:40:11Z

mesh-tools commented

2026-10-25 22:40:11 +00:00

Issue 1945 (notes 53, 55), a kind met twice on 2026-10-25: each bus user-list write the bridge (#143) makes triggers a ~3 s full reload (SIGHUP) in which JetStream answers 'no responders'. At 23:04:24 novox's node-engine failed to fetch mesh-vault's file by digest in that window, kept the attempt and retried only on a new want/composition revision, so mesh-vault's have stayed 'failed' and the assigner's rule 5 held every module needing the secret seat (gitea's new build).

internal/link/transient.go: transient = the bus client's no responders, timeout, reconnect, disconnect, JetStream not enabled / no response from stream (as errors or by their words; plain 'timeout' or 'context deadline exceeded' do not match). Backoff 2 s, 5 s, 15 s, 60 s, then every 5 min.
The agent keeps a transient failure as an attempt a timer clears (generation-checked: stale timers, removals and new revisions make it a no-op), so the module is applied again with no new revision, logged; the have says 'retrying in X (try N): cause', from the 5th try 'failed' while retries go on. A real failure keeps today's behaviour. The apply is a diff against what the machine already holds, so a repeat does not restart a converged container.
RetryTransient: 10 s after the engine writes the bus user list itself, every transiently failed module is settled again against its current want.
Have-bucket bind/get/set: a transient failure is written again on the same backoff, one pending at a time.
shorten keeps the head and the tail (the cause) within 300 bytes.

Not built (on issue 1945): a tool to apply one module again (needs an ask subject mesh.node..ask.apply-again and a verb to reach it).

Checked locally (operator 2026-10-25 22:56: checks local, not a gate): go vet with and without the integration tag; go test -race in every package; the integration level of internal/agent, internal/link and cmd/mesh-host. No new top-level tests (rows/subtests, each failing on the old code). This is the node-engine on every node.

Review notes kept for the issue: tries[module] is not cleared on removal or on a key change (a re-added module may say 'failed' early); a real failure whose detail also carries bus words is marked transient (about 80 s 'applying', then retried every 5 min).

Reviewed-by: review-retry@shanks at 99285cb080

Issue 1945 (notes 53, 55), a kind met twice on 2026-10-25: each bus user-list write the bridge (#143) makes triggers a ~3 s full reload (SIGHUP) in which JetStream answers 'no responders'. At 23:04:24 novox's node-engine failed to fetch mesh-vault's file by digest in that window, kept the attempt and retried only on a new want/composition revision, so mesh-vault's have stayed 'failed' and the assigner's rule 5 held every module needing the secret seat (gitea's new build). - internal/link/transient.go: transient = the bus client's no responders, timeout, reconnect, disconnect, JetStream not enabled / no response from stream (as errors or by their words; plain 'timeout' or 'context deadline exceeded' do not match). Backoff 2 s, 5 s, 15 s, 60 s, then every 5 min. - The agent keeps a transient failure as an attempt a timer clears (generation-checked: stale timers, removals and new revisions make it a no-op), so the module is applied again with no new revision, logged; the have says 'retrying in X (try N): cause', from the 5th try 'failed' while retries go on. A real failure keeps today's behaviour. The apply is a diff against what the machine already holds, so a repeat does not restart a converged container. - RetryTransient: 10 s after the engine writes the bus user list itself, every transiently failed module is settled again against its current want. - Have-bucket bind/get/set: a transient failure is written again on the same backoff, one pending at a time. - shorten keeps the head and the tail (the cause) within 300 bytes. Not built (on issue 1945): a tool to apply one module again (needs an ask subject mesh.node.<n>.ask.apply-again and a verb to reach it). Checked locally (operator 2026-10-25 22:56: checks local, not a gate): go vet with and without the integration tag; go test -race in every package; the integration level of internal/agent, internal/link and cmd/mesh-host. No new top-level tests (rows/subtests, each failing on the old code). This is the node-engine on every node. Review notes kept for the issue: tries[module] is not cleared on removal or on a key change (a re-added module may say 'failed' early); a real failure whose detail also carries bus words is marked transient (about 80 s 'applying', then retried every 5 min). Reviewed-by: review-retry@shanks at 99285cb080db65bfccfa1b223350de28d2171509

mesh-tools added 1 commit 2026-10-25 22:40:11 +00:00

Retry an apply, a fetch by digest and a have write that met the bus reloading, and keep the cause in the have

mesh/delivery merged: source.main moved to novox/mesh-host@1195cf45971e

Details

mesh/check pass 99285cb080db on 1195cf45971e | unit 303 run, 1080 kept, 5 skipped | integration 306 run, 1082 kept, 4 skipped | over bound: unit (slow…

99285cb080

The bus node's engine writes the bus user list (ADR 0547 bridge); each write makes the bus reload its accounts, and
for about three seconds every JetStream request meets no responders. An apply that failed then was kept failed until a
key changed (mesh-vault on novox, 2026-10-25 23:04:24), holding every module that needs the secret seat. A transient
bus failure is now retried with a backoff (2 s, 5 s, 15 s, 60 s, then every 5 min), the have saying retrying meanwhile;
after the engine writes the user list, the transiently failed modules are tried again once the reload is over. A have
write that met it is written again the same way. shorten keeps the head and the tail, the cause.

mesh-tools commented

2026-10-25 22:40:15 +00:00

Delivery novox/mesh-host@99285cb080db — merged since 2026-10-25T22:42:43Z

Source — source.main moved to novox/mesh-host@1195cf4597. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-25 22:40 (new) → proposed (announced): novox/mesh-host#144's head announced
2026-10-25 22:40 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-25 22:42 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@99285cb080db` — **merged** since 2026-10-25T22:42:43Z **Source** — source.main moved to novox/mesh-host@1195cf45971e. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-25 22:40 (new) → proposed (announced): novox/mesh-host#144's head announced - 2026-10-25 22:40 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-25 22:42 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools merged commit 1195cf4597 into main

2026-10-25 22:42:41 +00:00

mesh-tools deleted branch fix/apply-retries-transient

2026-10-25 22:42:41 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#144