ADR 0264 and issue 330: keep what a consumer gives up on until a person acts

Design 25 promised a dead-letter stream nothing built; a message a consumer gave up on was lost after a week, silently once its condition cleared.
2026-10-08 17:37:49 +02:00
parent 9fc09fe0d5
commit bcd8f41ab5
4 changed files with 179 additions and 5 deletions
@@ -0,0 +1,145 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-08
+deciders: jochen
+reconstructed: false
+---
+
+# 264. A message a consumer gives up on is kept until a person delivers it again or drops it
+
+## Context
+
+[Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) promises it three times: an event is
+"dead-lettered when it cannot be" delivered (§1); "after `max-deliver` attempts the advisory feeds
+`mesh.events.dead` (its own small stream)" (§3); and "an event whose consumer keeps failing dead-letters
+after `max-deliver`" is how it is checked (§10). [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md)
+decided the dead-letter destination on the broker the mesh had before, and
+[ADR 0106](0106-the-bus-is-nats.md) names dead-lettering in how the bus is checked.
+
+None of it exists ([issue 330](../04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md)).
+A durable consumer on the bus hands a message over at most five times. When it gives up, the server
+publishes a maximum-deliveries advisory naming the stream, the consumer and the message's sequence. The
+controller heard it and raised a `max-deliveries` condition, which cleared an hour after the advisories
+stopped. The message stayed only in its source stream: EVENTS drops an event after a week. On 2026-10-06 a
+media server's consumer gave up on several downloads that way; nothing records which, and they cannot be
+delivered again.
+
+Three things the mesh had decide the shape:
+
+- **The server says it, and says it once.** An advisory is a core message. A controller restarting, which
+  happens with every delivery of the controller, would miss one said meanwhile, and the message with it.
+- **A message on EVENTS has many consumers.** Publishing it again on its own subject hands it to every
+  module that consumes it, not only to the one that gave up. A merge would be built twice; a download taken
+  twice.
+- **A module's runtime reads an event's key from its subject**, from the token before `.event.` and what
+  follows it, in both runtimes (the TypeScript binding and the Go one). It does not read the subject's
+  first tokens.
+
+## Considered Options
+
+### Where the server's word is kept
+
+1. **The controller's subscription to the advisory, as it was.** Lost while no controller listens.
+   Rejected.
+2. **A stream that captures the advisory subject itself**, which the controller takes from. The server
+   writes it whether or not anyone listens. Chosen.
+
+### Where the message is kept
+
+1. **Leave it in its source stream and keep a pointer.** The source drops it after a week; a pointer to
+   nothing is the fault this issue describes. Rejected.
+2. **A copy in a stream of its own**, fetched by its sequence as soon as the notice is taken, while the
+   source still holds it. Chosen.
+
+### How it is delivered again to its consumer alone
+
+1. **Publish it again on its own subject.** Every consumer of the subject gets it again. Rejected.
+2. **Make the consumer start again before the message** (delete and re-create it). It would hand over
+   everything after the message as well. Rejected.
+3. **A subject only that consumer filters.** Every consumer on EVENTS also filters
+   `mesh.again.<consumer>.>`; a message delivered again is published on `mesh.again.<consumer>.` and its
+   own subject without `mesh.`, so the runtime reads the same key from it. Chosen. It needs no change to
+   any runtime, and the controller's own consumer reads it back to the subject it was published on.
+
+## Decision
+
+1. **Two streams beside EVENTS, declared by the controller like the others:**
+   - **DEAD_LETTER_NOTICES** captures `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>`, as a work queue kept
+     for a week (the longest a source keeps what a notice is about). The controller's consumer `controller`
+     on it has no maximum of deliveries: a notice it could not act on is offered again a minute later.
+   - **DEAD_LETTERS** holds the kept messages under `mesh.events.dead.<stream>.<consumer>`. It keeps them
+     with no age and up to 256 MiB, and when full it refuses the next copy rather than dropping one it
+     holds. Its de-duplication window is a day.
+2. **The serving controller takes each notice**, fetches the message it names from its source by sequence,
+   and keeps a copy in DEAD_LETTERS with the message's own headers and body, and with its stream, consumer,
+   subject, sequence, deliveries, the time it was given up and the time it was first published. The
+   server's own headers (`Nats-…`) are not copied, because they describe the first publish and would refuse
+   or de-duplicate the copy. The copy's id is `<stream>.<consumer>.<sequence>`, so one message is kept once
+   however often its notice is taken. A message its source no longer holds is kept as a record that says
+   so. A copy that cannot be made is said in the controller's log and as the consumer's condition, and the
+   notice is taken again.
+3. **Every consumer on EVENTS filters its own `mesh.again.<consumer>.>`**, and EVENTS takes `mesh.again.>`.
+   The controller's consumer reads a message there as the subject it was first published on.
+4. **The controller's verb `dead-letters`** lists what DEAD_LETTERS holds, newest first, or for one
+   consumer; shows one whole by its id; **delivers one again** to the consumer that gave it up; or **drops
+   one**. Delivering again and dropping each require `why`, are recorded in the hand-act log first, and are
+   a person's decision, not a repair (S15 does not count them). An event is delivered again on the
+   consumer's again subject; an ask to a seat back on its own subject, since a seat's queue has one worker.
+   A message from any other stream, or a record without its message, can only be dropped, and the verb says
+   why. Delivering again carries the message's id, so asking twice delivers once. It is served by the
+   serving controller on its own connection, not by a fresh process.
+5. **The `max-deliveries` condition is open while DEAD_LETTERS holds a message for that consumer**, one
+   condition per consumer, and clears when its messages are delivered again or dropped, never because the
+   server stopped saying it. Its words say who gave up on how many messages, that what they asked for was
+   not done, that the mesh keeps them, and what the operator does: "Needs you: deliver them again or drop
+   them, from the mesh MCP server." A copy that could not be made is its own condition (token `not-kept`),
+   kept for an hour after the last failure, as the advisories were.
+6. **The writers table gains a row** (to-be 45 §1): a message a consumer gave up on is written by the
+   controller alone, on `mesh.events.dead.>` and `mesh.again.>`, and read, delivered again or dropped
+   through its verb.
+
+## Consequences
+
+- Design 25 §1, §3 and §10 hold. The stream §3 names `mesh.events.dead` is DEAD_LETTERS; its subjects carry
+  the stream and consumer.
+- The controller may publish on `mesh.events.dead.>`, `mesh.again.>` and every seat's `accept` subjects
+  (to give an ask back to its seat), and acknowledge on DEAD_LETTER_NOTICES. It already reached the stream
+  API.
+- Every module's consumer changes once, when its filters gain its again subject. Its position is kept.
+- A message given up on is now visible and costs space until somebody acts. A consumer that gives up on
+  many fills DEAD_LETTERS; full, further copies are refused and said, which is the loud failure the mesh
+  prefers to dropping what it holds.
+- A module handler sees an event delivered again as the same key. It can tell it was delivered again from
+  the `Mesh-Delivered-Again` header, and need not.
+- What a desk click may do does not change ([ADR 0258](0258-a-desk-click-performs-only-an-acknowledgement-and-an-answer-is-a-one-time-ticket.md)):
+  delivering again and dropping take a person's why, so the notification offers no button for them.
+
+## How it is checked
+
+- mesh-controller `internal/link/replay330_test.go`, **R330** in mesh-lab's replays register: a module's
+  consumer as the controller makes it never acknowledges an event; once it gives up, the event is kept with
+  its consumer, stream, subject, deliveries and the time it was given up. It fails on the commit before the
+  fix and passes on it.
+- mesh-controller `internal/link/deadletters_test.go`: delivered again, an event reaches the consumer that
+  gave it up and not another module that handled it; it is no longer kept, and a second delivery finds
+  nothing; a dropped one is gone; a notice whose message is gone is kept as a record that cannot be
+  delivered again; a notice taken twice is kept once; the controller acts on its own event delivered again.
+- mesh-controller `cmd/mesh-controller/deadletters_test.go`: the verb lists, refuses an act without why or
+  with an argument it would pass over before anything is done, drops with the act recorded as a person's
+  decision; the condition is open per consumer while messages are held, in plain words that say where to
+  act, and is not raised when none are.
+- mesh-controller `internal/broker`: the streams' retention, the consumer's again filter, the writers table
+  against to-be 45 §1, and the composed grants against their golden file.
+- mesh-tools `node-tools/internal/bus/bus_test.go`: an event delivered again has the key it had.
+- **Live**, once delivered: the self-check (D7) finds both streams as declared; `nats_streams` lists them;
+  the next consumer that gives up shows in `mesh-controller.dead-letters`.
+
+## References
+
+- [Issue 330](../04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md)
+- [Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1, §3, §10;
+  [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §1 and S9
+- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md), [ADR 0106](0106-the-bus-is-nats.md),
+  [ADR 0253](0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md)
+- [Research 036](../01-RESEARCH/036-everything-over-nats/00-overview.md) (G7)