Wait out a bus that answers nothing for a few seconds instead of losing the write (issue 774)
At each mesh-grants restart and user-list reload (16:37:24, 16:55:15) nats-server 2.11's reloadAuthorization left the mesh's account without its imports, $JS.API.> among them, for a few seconds: requests had no responders, JetStream calls said it was not enabled, and a build-test result on ace was lost. Quick fix, chosen as asked; the thorough fix is a reload that keeps the account's imports, in the bus's holder. The package busretry asks a request again, backing off within about 30 s, only while its error says it reached no one (no responders, JetStream not enabled, no response from the stream, or either in words): nothing was done, so even a create or an update is safe to ask again. A timeout or any answer is never asked again. After the bound it is said on standard error and returned. Used by process.State's requests, process.Conn's calls (asked again only while $JS.API.INFO says the bus itself answers nothing, so a callee that is not running still fails at once) and asks, and the bundle's state requests through the runtime.
This commit is contained in:
+24
-4
@@ -9,6 +9,8 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/busretry"
|
||||
)
|
||||
|
||||
// CallTimeout bounds a call to a tool or a seat verb (mesh-tools node-tools/internal/bus/bus.go:35,
|
||||
@@ -47,16 +49,34 @@ func (c *Conn) CallOn(node, key string, body any) (json.RawMessage, error) {
|
||||
return c.callOn(called, fallback, body)
|
||||
}
|
||||
|
||||
// callOn asks called, then fallback when nothing answers it; while the bus itself answers nothing (a user-list reload
|
||||
// drops the account's imports for a few seconds, busretry) both are asked again, within busretry.Bound. A call nobody
|
||||
// answers on a bus that answers is not asked again: its callee is not running. A call that may have been delivered (a
|
||||
// timeout, any answer) is never asked again.
|
||||
func (c *Conn) callOn(called, fallback string, body any) (json.RawMessage, error) {
|
||||
data, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result, err := c.request(called, data)
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
result, err = c.request(fallback, data)
|
||||
return busretry.Do("the call on "+fallback, func() (json.RawMessage, error) {
|
||||
result, err := c.request(called, data)
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
result, err = c.request(fallback, data)
|
||||
}
|
||||
return result, err
|
||||
}, c.blackedOut)
|
||||
}
|
||||
|
||||
// blackedOut says the bus answers this account nothing at all: its JetStream API, which every account imports, has no
|
||||
// responder or says it is not enabled.
|
||||
func (c *Conn) blackedOut() bool {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
msg, err := c.nc.RequestWithContext(ctx, "$JS.API.INFO", nil)
|
||||
if err != nil {
|
||||
return busretry.Undelivered(err)
|
||||
}
|
||||
return result, err
|
||||
return strings.Contains(string(msg.Data), `"err_code":10039`) || strings.Contains(string(msg.Data), "jetstream not enabled")
|
||||
}
|
||||
|
||||
// callerToken is this account's user name as the last token of a call it publishes, each dot written ~
|
||||
|
||||
Reference in New Issue
Block a user