jochen
c64a7b3d4a
The bus's own numbers were unexplained and raised by nothing. Measured again read-only: mesh-delivery rewrites unchanged deliveries (326), every controller verb dials its own connection (327), API errors have no reader (328), event consumers outlive their declaration (329), and design 25's dead-letter stream was never built (330). The two slow consumers join issue 270, whose missing system account they share.
2.8 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design |
|---|---|---|---|---|
| open | 2026-10-08 |
328. JetStream API errors are counted, and attributed to nothing
Symptom
Research 036 (§4.3, gap G7) read 4 688 JetStream API errors out of 6.08 million calls in the bus server's first 39 hours.
Measured again on 2026-10-08 with nats.nats_server:
- From 14:12 to 14:20 UTC the count rose from 4 794 to 4 810: 16 errors in 475 seconds, 2 a minute. Over the server's life the average is 120 an hour, out of 6.1 million calls (0.08 %).
- During the same minutes, three
mesh-controller.conditionscalls, threedelivery-walkscalls and a dozen reads of thenatsmodule's tools added no error.
Which calls fail, and from whom, cannot be read. The nats module's tools read the server's
totals (/varz, /jsz). The server publishes one audit advisory per failed API call on
$JS.EVENT.ADVISORY.API, inside the mesh's own account, but nothing subscribes to it: the only
advisory subscriptions on the server are the controller's two, for a consumer deleted and a message
given up after its maximum deliveries. The server's log shows no API errors at its level.
Why it is an issue
An API error can be harmless: asking for the last message on a subject that has none is counted as one. It can also be a refusal that matters: a consumer that could not be made, a stream update the server refused, a request the account was denied. All of them are in one number that nothing watches, so a real refusal is hidden among the expected ones. Design 25 makes the bus the place the mesh happens, and this count has no reader.
The diagnosis so far is in 01-diagnosis.md.
Impact
None known. What is not known is whether any of the 120 an hour is a refusal that left a module without its consumer, its bucket or its grant.
How it is checked
- Attribution: a tool (in the
natsmodule, which reads the server) answers the API errors of a window by API subject, error code and user, from the audit advisory. - Resolution: every error code seen over a day is either fixed at its caller or named as expected (for example, "no message for this subject" on a pause subject). A condition is raised for any error code not on the expected list.
Core or not
Not known yet. It is core if diagnosis locates it in the controller, the node-engine, the tool
runner, the SDK or the catalogue's nats module, and then it resolves with a replay (replay:) or with
replay-none: saying why none is possible.
Suggested fix
Subscribe to $JS.EVENT.ADVISORY.API in the nats module, keep counts per subject, code and user,
and add the tool and the condition above. Then each caller found is fixed or named as expected.