hq/04-ISSUES/328-jetstream-api-errors-are-counted-and-attributed-to-nothing/00-report.md

opened

opened
2026-10-08

328. JetStream API errors are counted, and attributed to nothing

Symptom

Research 036 (§4.3, gap G7) read 4 688 JetStream API errors out of 6.08 million calls in the bus server's first 39 hours.

Measured again on 2026-10-08 with nats.nats_server:

• From 14:12 to 14:20 UTC the count rose from 4 794 to 4 810: 16 errors in 475 seconds, 2 a minute. Over the server's life the average is 120 an hour, out of 6.1 million calls (0.08 %).
• During the same minutes, three mesh-controller.conditions calls, three delivery-walks calls and a dozen reads of the nats module's tools added no error.

Which calls fail, and from whom, cannot be read. The nats module's tools read the server's totals (/varz, /jsz). The server publishes one audit advisory per failed API call on $JS.EVENT.ADVISORY.API, inside the mesh's own account, but nothing subscribes to it: the only advisory subscriptions on the server are the controller's two, for a consumer deleted and a message given up after its maximum deliveries. The server's log shows no API errors at its level.

Why it is an issue

An API error can be harmless: asking for the last message on a subject that has none is counted as one. It can also be a refusal that matters: a consumer that could not be made, a stream update the server refused, a request the account was denied. All of them are in one number that nothing watches, so a real refusal is hidden among the expected ones. Design 25 makes the bus the place the mesh happens, and this count has no reader.

The diagnosis so far is in 01-diagnosis.md.

Impact

None known. What is not known is whether any of the 120 an hour is a refusal that left a module without its consumer, its bucket or its grant.

How it is checked

• Attribution: a tool (in the nats module, which reads the server) answers the API errors of a window by API subject, error code and user, from the audit advisory.
• Resolution: every error code seen over a day is either fixed at its caller or named as expected (for example, "no message for this subject" on a pause subject). A condition is raised for any error code not on the expected list.

Core or not

Not known yet. It is core if diagnosis locates it in the controller, the node-engine, the tool runner, the SDK or the catalog's nats module, and then it resolves with a replay (replay:) or with replay-none: saying why none is possible.

Suggested fix

Subscribe to $JS.EVENT.ADVISORY.API in the nats module, keep counts per subject, code and user, and add the tool and the condition above. Then each caller found is fixed or named as expected.