jschoubben
418261213c
ADR 0290 rule 9: once the import is checked record by record, hq keeps no copy. The records and efforts are removed; every link or path to them reads "ADR NNNN" or "research NNN"; a design's decisions: lists "ADR NNNN". 02-DECISIONS keeps a pointer README with the reading order's six topics, which the knowledge base reads through the records module; 01-RESEARCH keeps a pointer README. The checks drop the rules that judged records (the knowledge base holds each at propose; checks/README.md says where) and refuse a record written here again. Playbooks 01 and 02 and the skills write and read records with the knowledge base's verbs.
2.6 KiB
opened
| opened |
|---|
| 2026-10-08 |
328. JetStream API errors are counted, and attributed to nothing
Symptom
Research 036 (§4.3, gap G7) read 4 688 JetStream API errors out of 6.08 million calls in the bus server's first 39 hours.
Measured again on 2026-10-08 with nats.nats_server:
- From 14:12 to 14:20 UTC the count rose from 4 794 to 4 810: 16 errors in 475 seconds, 2 a minute. Over the server's life the average is 120 an hour, out of 6.1 million calls (0.08 %).
- During the same minutes, three
mesh-controller.conditionscalls, threedelivery-walkscalls and a dozen reads of thenatsmodule's tools added no error.
Which calls fail, and from whom, cannot be read. The nats module's tools read the server's
totals (/varz, /jsz). The server publishes one audit advisory per failed API call on
$JS.EVENT.ADVISORY.API, inside the mesh's own account, but nothing subscribes to it: the only
advisory subscriptions on the server are the controller's two, for a consumer deleted and a message
given up after its maximum deliveries. The server's log shows no API errors at its level.
Why it is an issue
An API error can be harmless: asking for the last message on a subject that has none is counted as one. It can also be a refusal that matters: a consumer that could not be made, a stream update the server refused, a request the account was denied. All of them are in one number that nothing watches, so a real refusal is hidden among the expected ones. Design 25 makes the bus the place the mesh happens, and this count has no reader.
The diagnosis so far is in 01-diagnosis.md.
Impact
None known. What is not known is whether any of the 120 an hour is a refusal that left a module without its consumer, its bucket or its grant.
How it is checked
- Attribution: a tool (in the
natsmodule, which reads the server) answers the API errors of a window by API subject, error code and user, from the audit advisory. - Resolution: every error code seen over a day is either fixed at its caller or named as expected (for example, "no message for this subject" on a pause subject). A condition is raised for any error code not on the expected list.
Core or not
Not known yet. It is core if diagnosis locates it in the controller, the node-engine, the tool
runner, the SDK or the catalogue's nats module, and then it resolves with a replay (replay:) or with
replay-none: saying why none is possible.
Suggested fix
Subscribe to $JS.EVENT.ADVISORY.API in the nats module, keep counts per subject, code and user,
and add the tool and the condition above. Then each caller found is fixed or named as expected.