jochen
c64a7b3d4a
The bus's own numbers were unexplained and raised by nothing. Measured again read-only: mesh-delivery rewrites unchanged deliveries (326), every controller verb dials its own connection (327), API errors have no reader (328), event consumers outlive their declaration (329), and design 25's dead-letter stream was never built (330). The two slow consumers join issue 270, whose missing system account they share.
62 lines
2.8 KiB
Markdown
62 lines
2.8 KiB
Markdown
---
|
|
status: open
|
|
opened: 2026-10-08
|
|
located-in: []
|
|
fixed-by:
|
|
amended-design:
|
|
---
|
|
|
|
# 328. JetStream API errors are counted, and attributed to nothing
|
|
|
|
## Symptom
|
|
|
|
[Research 036](../../01-RESEARCH/036-everything-over-nats/01-evidence.md) (§4.3, gap G7) read 4 688
|
|
JetStream API errors out of 6.08 million calls in the bus server's first 39 hours.
|
|
|
|
Measured again on 2026-10-08 with `nats.nats_server`:
|
|
|
|
- From 14:12 to 14:20 UTC the count rose from 4 794 to 4 810: **16 errors in 475 seconds, 2 a
|
|
minute.** Over the server's life the average is 120 an hour, out of 6.1 million calls (0.08 %).
|
|
- During the same minutes, three `mesh-controller.conditions` calls, three `delivery-walks` calls and a
|
|
dozen reads of the `nats` module's tools added no error.
|
|
|
|
**Which calls fail, and from whom, cannot be read.** The `nats` module's tools read the server's
|
|
totals (`/varz`, `/jsz`). The server publishes one audit advisory per failed API call on
|
|
`$JS.EVENT.ADVISORY.API`, inside the mesh's own account, but nothing subscribes to it: the only
|
|
advisory subscriptions on the server are the controller's two, for a consumer deleted and a message
|
|
given up after its maximum deliveries. The server's log shows no API errors at its level.
|
|
|
|
## Why it is an issue
|
|
|
|
An API error can be harmless: asking for the last message on a subject that has none is counted as
|
|
one. It can also be a refusal that matters: a consumer that could not be made, a stream update the
|
|
server refused, a request the account was denied. All of them are in one number that nothing watches,
|
|
so a real refusal is hidden among the expected ones.
|
|
Design 25 makes the bus the place the mesh happens, and this count has no reader.
|
|
|
|
The diagnosis so far is in [01-diagnosis.md](01-diagnosis.md).
|
|
|
|
## Impact
|
|
|
|
None known. What is not known is whether any of the 120 an hour is a refusal that left a module without
|
|
its consumer, its bucket or its grant.
|
|
|
|
## How it is checked
|
|
|
|
- **Attribution:** a tool (in the `nats` module, which reads the server) answers the API errors of a
|
|
window by API subject, error code and user, from the audit advisory.
|
|
- **Resolution:** every error code seen over a day is either fixed at its caller or named as expected
|
|
(for example, "no message for this subject" on a pause subject). A condition is raised for any error
|
|
code not on the expected list.
|
|
|
|
## Core or not
|
|
|
|
**Not known yet.** It is core if diagnosis locates it in the controller, the node-engine, the tool
|
|
runner, the SDK or the catalogue's `nats` module, and then it resolves with a replay (`replay:`) or with
|
|
`replay-none:` saying why none is possible.
|
|
|
|
## Suggested fix
|
|
|
|
Subscribe to `$JS.EVENT.ADVISORY.API` in the `nats` module, keep counts per subject, code and user,
|
|
and add the tool and the condition above. Then each caller found is fixed or named as expected.
|