jschoubben
418261213c
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery superseded: a newer head of the same pull request
ADR 0290 rule 9: once the import is checked record by record, hq keeps no copy. The records and efforts are removed; every link or path to them reads "ADR NNNN" or "research NNN"; a design's decisions: lists "ADR NNNN". 02-DECISIONS keeps a pointer README with the reading order's six topics, which the knowledge base reads through the records module; 01-RESEARCH keeps a pointer README. The checks drop the rules that judged records (the knowledge base holds each at propose; checks/README.md says where) and refuse a record written here again. Playbooks 01 and 02 and the skills write and read records with the knowledge base's verbs.
58 lines
2.6 KiB
Markdown
58 lines
2.6 KiB
Markdown
---
|
|
opened: 2026-10-08
|
|
---
|
|
|
|
# 328. JetStream API errors are counted, and attributed to nothing
|
|
|
|
## Symptom
|
|
|
|
Research 036 (§4.3, gap G7) read 4 688
|
|
JetStream API errors out of 6.08 million calls in the bus server's first 39 hours.
|
|
|
|
Measured again on 2026-10-08 with `nats.nats_server`:
|
|
|
|
- From 14:12 to 14:20 UTC the count rose from 4 794 to 4 810: **16 errors in 475 seconds, 2 a
|
|
minute.** Over the server's life the average is 120 an hour, out of 6.1 million calls (0.08 %).
|
|
- During the same minutes, three `mesh-controller.conditions` calls, three `delivery-walks` calls and a
|
|
dozen reads of the `nats` module's tools added no error.
|
|
|
|
**Which calls fail, and from whom, cannot be read.** The `nats` module's tools read the server's
|
|
totals (`/varz`, `/jsz`). The server publishes one audit advisory per failed API call on
|
|
`$JS.EVENT.ADVISORY.API`, inside the mesh's own account, but nothing subscribes to it: the only
|
|
advisory subscriptions on the server are the controller's two, for a consumer deleted and a message
|
|
given up after its maximum deliveries. The server's log shows no API errors at its level.
|
|
|
|
## Why it is an issue
|
|
|
|
An API error can be harmless: asking for the last message on a subject that has none is counted as
|
|
one. It can also be a refusal that matters: a consumer that could not be made, a stream update the
|
|
server refused, a request the account was denied. All of them are in one number that nothing watches,
|
|
so a real refusal is hidden among the expected ones.
|
|
Design 25 makes the bus the place the mesh happens, and this count has no reader.
|
|
|
|
The diagnosis so far is in [01-diagnosis.md](01-diagnosis.md).
|
|
|
|
## Impact
|
|
|
|
None known. What is not known is whether any of the 120 an hour is a refusal that left a module without
|
|
its consumer, its bucket or its grant.
|
|
|
|
## How it is checked
|
|
|
|
- **Attribution:** a tool (in the `nats` module, which reads the server) answers the API errors of a
|
|
window by API subject, error code and user, from the audit advisory.
|
|
- **Resolution:** every error code seen over a day is either fixed at its caller or named as expected
|
|
(for example, "no message for this subject" on a pause subject). A condition is raised for any error
|
|
code not on the expected list.
|
|
|
|
## Core or not
|
|
|
|
**Not known yet.** It is core if diagnosis locates it in the controller, the node-engine, the tool
|
|
runner, the SDK or the catalogue's `nats` module, and then it resolves with a replay (`replay:`) or with
|
|
`replay-none:` saying why none is possible.
|
|
|
|
## Suggested fix
|
|
|
|
Subscribe to `$JS.EVENT.ADVISORY.API` in the `nats` module, keep counts per subject, code and user,
|
|
and add the tool and the condition above. Then each caller found is fixed or named as expected.
|