From d1e0e121d6eacebe50114c3efbff6b82a0831806 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 17:37:49 +0200 Subject: [PATCH 1/3] ADR 0264 and issue 330: keep what a consumer gives up on until a person acts Design 25 promised a dead-letter stream nothing built; a message a consumer gave up on was lost after a week, silently once its condition cleared. --- ...-a-person-delivers-it-again-or-drops-it.md | 145 ++++++++++++++++++ 03-DESIGN/01-to-be/25-the-bus-on-nats.md | 7 +- .../45-a-core-that-cannot-fail-silently.md | 4 +- .../00-report.md | 28 +++- 4 files changed, 179 insertions(+), 5 deletions(-) create mode 100644 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md diff --git a/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md new file mode 100644 index 00000000..eae7324a --- /dev/null +++ b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md @@ -0,0 +1,145 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-08 +deciders: jochen +reconstructed: false +--- + +# 264. A message a consumer gives up on is kept until a person delivers it again or drops it + +## Context + +[Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) promises it three times: an event is +"dead-lettered when it cannot be" delivered (§1); "after `max-deliver` attempts the advisory feeds +`mesh.events.dead` (its own small stream)" (§3); and "an event whose consumer keeps failing dead-letters +after `max-deliver`" is how it is checked (§10). [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) +decided the dead-letter destination on the broker the mesh had before, and +[ADR 0106](0106-the-bus-is-nats.md) names dead-lettering in how the bus is checked. + +None of it exists ([issue 330](../04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md)). +A durable consumer on the bus hands a message over at most five times. When it gives up, the server +publishes a maximum-deliveries advisory naming the stream, the consumer and the message's sequence. The +controller heard it and raised a `max-deliveries` condition, which cleared an hour after the advisories +stopped. The message stayed only in its source stream: EVENTS drops an event after a week. On 2026-10-06 a +media server's consumer gave up on several downloads that way; nothing records which, and they cannot be +delivered again. + +Three things the mesh had decide the shape: + +- **The server says it, and says it once.** An advisory is a core message. A controller restarting, which + happens with every delivery of the controller, would miss one said meanwhile, and the message with it. +- **A message on EVENTS has many consumers.** Publishing it again on its own subject hands it to every + module that consumes it, not only to the one that gave up. A merge would be built twice; a download taken + twice. +- **A module's runtime reads an event's key from its subject**, from the token before `.event.` and what + follows it, in both runtimes (the TypeScript binding and the Go one). It does not read the subject's + first tokens. + +## Considered Options + +### Where the server's word is kept + +1. **The controller's subscription to the advisory, as it was.** Lost while no controller listens. + Rejected. +2. **A stream that captures the advisory subject itself**, which the controller takes from. The server + writes it whether or not anyone listens. Chosen. + +### Where the message is kept + +1. **Leave it in its source stream and keep a pointer.** The source drops it after a week; a pointer to + nothing is the fault this issue describes. Rejected. +2. **A copy in a stream of its own**, fetched by its sequence as soon as the notice is taken, while the + source still holds it. Chosen. + +### How it is delivered again to its consumer alone + +1. **Publish it again on its own subject.** Every consumer of the subject gets it again. Rejected. +2. **Make the consumer start again before the message** (delete and re-create it). It would hand over + everything after the message as well. Rejected. +3. **A subject only that consumer filters.** Every consumer on EVENTS also filters + `mesh.again..>`; a message delivered again is published on `mesh.again..` and its + own subject without `mesh.`, so the runtime reads the same key from it. Chosen. It needs no change to + any runtime, and the controller's own consumer reads it back to the subject it was published on. + +## Decision + +1. **Two streams beside EVENTS, declared by the controller like the others:** + - **DEAD_LETTER_NOTICES** captures `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>`, as a work queue kept + for a week (the longest a source keeps what a notice is about). The controller's consumer `controller` + on it has no maximum of deliveries: a notice it could not act on is offered again a minute later. + - **DEAD_LETTERS** holds the kept messages under `mesh.events.dead..`. It keeps them + with no age and up to 256 MiB, and when full it refuses the next copy rather than dropping one it + holds. Its de-duplication window is a day. +2. **The serving controller takes each notice**, fetches the message it names from its source by sequence, + and keeps a copy in DEAD_LETTERS with the message's own headers and body, and with its stream, consumer, + subject, sequence, deliveries, the time it was given up and the time it was first published. The + server's own headers (`Nats-…`) are not copied, because they describe the first publish and would refuse + or de-duplicate the copy. The copy's id is `..`, so one message is kept once + however often its notice is taken. A message its source no longer holds is kept as a record that says + so. A copy that cannot be made is said in the controller's log and as the consumer's condition, and the + notice is taken again. +3. **Every consumer on EVENTS filters its own `mesh.again..>`**, and EVENTS takes `mesh.again.>`. + The controller's consumer reads a message there as the subject it was first published on. +4. **The controller's verb `dead-letters`** lists what DEAD_LETTERS holds, newest first, or for one + consumer; shows one whole by its id; **delivers one again** to the consumer that gave it up; or **drops + one**. Delivering again and dropping each require `why`, are recorded in the hand-act log first, and are + a person's decision, not a repair (S15 does not count them). An event is delivered again on the + consumer's again subject; an ask to a seat back on its own subject, since a seat's queue has one worker. + A message from any other stream, or a record without its message, can only be dropped, and the verb says + why. Delivering again carries the message's id, so asking twice delivers once. It is served by the + serving controller on its own connection, not by a fresh process. +5. **The `max-deliveries` condition is open while DEAD_LETTERS holds a message for that consumer**, one + condition per consumer, and clears when its messages are delivered again or dropped, never because the + server stopped saying it. Its words say who gave up on how many messages, that what they asked for was + not done, that the mesh keeps them, and what the operator does: "Needs you: deliver them again or drop + them, from the mesh MCP server." A copy that could not be made is its own condition (token `not-kept`), + kept for an hour after the last failure, as the advisories were. +6. **The writers table gains a row** (to-be 45 §1): a message a consumer gave up on is written by the + controller alone, on `mesh.events.dead.>` and `mesh.again.>`, and read, delivered again or dropped + through its verb. + +## Consequences + +- Design 25 §1, §3 and §10 hold. The stream §3 names `mesh.events.dead` is DEAD_LETTERS; its subjects carry + the stream and consumer. +- The controller may publish on `mesh.events.dead.>`, `mesh.again.>` and every seat's `accept` subjects + (to give an ask back to its seat), and acknowledge on DEAD_LETTER_NOTICES. It already reached the stream + API. +- Every module's consumer changes once, when its filters gain its again subject. Its position is kept. +- A message given up on is now visible and costs space until somebody acts. A consumer that gives up on + many fills DEAD_LETTERS; full, further copies are refused and said, which is the loud failure the mesh + prefers to dropping what it holds. +- A module handler sees an event delivered again as the same key. It can tell it was delivered again from + the `Mesh-Delivered-Again` header, and need not. +- What a desk click may do does not change ([ADR 0258](0258-a-desk-click-performs-only-an-acknowledgement-and-an-answer-is-a-one-time-ticket.md)): + delivering again and dropping take a person's why, so the notification offers no button for them. + +## How it is checked + +- mesh-controller `internal/link/replay330_test.go`, **R330** in mesh-lab's replays register: a module's + consumer as the controller makes it never acknowledges an event; once it gives up, the event is kept with + its consumer, stream, subject, deliveries and the time it was given up. It fails on the commit before the + fix and passes on it. +- mesh-controller `internal/link/deadletters_test.go`: delivered again, an event reaches the consumer that + gave it up and not another module that handled it; it is no longer kept, and a second delivery finds + nothing; a dropped one is gone; a notice whose message is gone is kept as a record that cannot be + delivered again; a notice taken twice is kept once; the controller acts on its own event delivered again. +- mesh-controller `cmd/mesh-controller/deadletters_test.go`: the verb lists, refuses an act without why or + with an argument it would pass over before anything is done, drops with the act recorded as a person's + decision; the condition is open per consumer while messages are held, in plain words that say where to + act, and is not raised when none are. +- mesh-controller `internal/broker`: the streams' retention, the consumer's again filter, the writers table + against to-be 45 §1, and the composed grants against their golden file. +- mesh-tools `node-tools/internal/bus/bus_test.go`: an event delivered again has the key it had. +- **Live**, once delivered: the self-check (D7) finds both streams as declared; `nats_streams` lists them; + the next consumer that gives up shows in `mesh-controller.dead-letters`. + +## References + +- [Issue 330](../04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md) +- [Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1, §3, §10; + [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §1 and S9 +- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md), [ADR 0106](0106-the-bus-is-nats.md), + [ADR 0253](0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md) +- [Research 036](../01-RESEARCH/036-everything-over-nats/00-overview.md) (G7) diff --git a/03-DESIGN/01-to-be/25-the-bus-on-nats.md b/03-DESIGN/01-to-be/25-the-bus-on-nats.md index 72d0f1f9..feb07657 100644 --- a/03-DESIGN/01-to-be/25-the-bus-on-nats.md +++ b/03-DESIGN/01-to-be/25-the-bus-on-nats.md @@ -8,8 +8,9 @@ code: - mesh-catalog modules/nats (to be written) - mesh-sdk src (the protocol's NATS binding, step 3) - mesh-tools node-tools/internal/bus (a module's state, ADR 0201) -updated: 2026-10-06 +updated: 2026-10-08 decisions: + - 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md - 02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md - 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md - 02-DECISIONS/0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md @@ -167,7 +168,9 @@ Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStrea |---|---|---|---| | CONTROL | `mesh.control.>` except `alive` (a build's outcome moved to its seat, ADR 0121) | work queue, one consumer (the controller), explicit ack | the store-window guarantee ([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)): the controller `nak`s with a delay while its store is away and the message is redelivered; nothing is dropped | | NODES | `mesh.node.>` | last per subject | one declaration per node, always the newest | -| EVENTS | `mesh.mod.*.event.>` and `mesh.seat.*.event.>` | limits (age, size), durable consumer per subscribing module | a subscriber that was down catches up; after `max-deliver` attempts the advisory feeds `mesh.events.dead` (its own small stream). *2026-10-01:* a build's whole log is here too, as the build seat's `log.` events ([ADR 0157](../../02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)) — one subject per build, a week of retention, read back by `builds --log ` with a consumer that is gone when the reading is done | +| EVENTS | `mesh.mod.*.event.>` and `mesh.seat.*.event.>`; and `mesh.again.>` (ADR 0264) | limits (age, size), durable consumer per subscribing module | a subscriber that was down catches up; after `max-deliver` attempts the advisory feeds `mesh.events.dead` (its own small stream). *2026-10-08, [ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md):* that stream is DEAD_LETTERS, below; each consumer also filters `mesh.again..>`, where a message it gave up on is delivered again to it alone. *2026-10-01:* a build's whole log is here too, as the build seat's `log.` events ([ADR 0157](../../02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)) — one subject per build, a week of retention, read back by `builds --log ` with a consumer that is gone when the reading is done | +| DEAD_LETTER_NOTICES | the server's maximum-deliveries advisory | work queue, a week; one consumer (the controller), no `max-deliver` | the server's word that a consumer gave up on a message, kept whether or not a controller is listening, until the controller has copied the message (ADR 0264) | +| DEAD_LETTERS | `mesh.events.dead..` | limits: no age, 256 MiB, refuses new when full; a day's de-duplication | every message a consumer gave up on, with its consumer, subject, deliveries and when, until a person delivers it again or drops it with why (ADR 0264); a `max-deliveries` condition is open per consumer while it holds any | | `KV__` | `$KV._.>` | the newest value per key — as many past values as the owner declared — no age unless the owner declared one; a value at most 256 KiB, a bucket at most 64 MiB | a module's state (ADR 0201): one per name in a manifest's `state`, created from the catalogue on every raise, so it exists before its owner runs anywhere; kept when the module is unassigned, because what it holds is data | Tool calls and heartbeats stay on core NATS: a lost heartbeat is the next heartbeat; a lost tool diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index c1396176..858f6fef 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -5,6 +5,7 @@ code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab] updated: 2026-10-08 decisions: - 02-DECISIONS/0265-a-verb-that-reads-is-answered-on-the-serving-controllers-own-connection-and-every-connection-says-what-it-is.md + - 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md - 02-DECISIONS/0261-an-assignment-made-while-its-modules-build-runs-is-kept-pending-and-the-controller-makes-it.md - 02-DECISIONS/0254-a-wait-for-a-persons-new-login-is-not-a-gates-failure-and-each-module-of-a-send-keeps-its-own-pass.md - 02-DECISIONS/0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md @@ -87,6 +88,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone | the operator-channel's open messages | the seat's holder | its own key-value state | — | | a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps | | the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads | +| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why | A bus subject two components may publish on is refused when the controller composes grants, unless this table marks it shared. Changing a writer is a change to this table, through a decision. @@ -223,7 +225,7 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age | S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — | | S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — | | S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — | -| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` | +| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` | | S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — | | S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — | | S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — | diff --git a/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md b/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md index 94ea33bb..50964ce2 100644 --- a/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md +++ b/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md @@ -2,8 +2,9 @@ status: located opened: 2026-10-08 located-in: [mesh-controller internal/broker (streams.go, the streams the controller declares; derived.go, each event consumer's maximum), mesh-controller internal/link (advisories.go, the maximum-deliveries advisory)] -fixed-by: -amended-design: +fixed-by: mesh-controller#159, mesh-tools#24, mesh-lab#70 +replay: R330 +amended-design: 03-DESIGN/01-to-be/25-the-bus-on-nats.md --- # 330. The dead-letter stream design 25 promises does not exist @@ -65,3 +66,26 @@ The controller declares a small dead-letter stream beside EVENTS. On each maximu copies the given-up message (the advisory names its stream and sequence) into that stream with the consumer's name and when. A verb lists what it holds, delivers one again or drops it, as a hand act. If the operator decides against a stream, design 25 §1, §3 and §10 are amended to say what happens instead. + +## Resolution + +Decided in [ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md), +which [design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §3 and +[to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §1 and S9 now name: + +- **DEAD_LETTER_NOTICES** captures the server's maximum-deliveries advisory, so a notice said while no + controller listens waits for the next one. +- **DEAD_LETTERS** keeps a copy of each message given up on, fetched by its sequence while the source still + holds it, with its consumer, stream, subject, deliveries and when it was given up. It keeps them with no + age until a person acts. +- **The controller's `dead-letters` verb** lists them, shows one whole, delivers one again to the consumer + that gave it up and to nobody else, or drops one. Delivering and dropping say why and are recorded in the + hand-act log. +- **The `max-deliveries` condition** is open per consumer while DEAD_LETTERS holds its messages, and says + where to act: "deliver them again or drop them, from the mesh MCP server." +- The comments that said a failing event dead-letters (the controller's streams and derived consumers, the + tool runner's NATS binding) say what happens now. + +**Replay R330** (mesh-controller `internal/link/replay330_test.go`, registered in mesh-lab) fails on the +commit before the fix and passes on it. + From b0986e2ae0bd2e5d0feb0cc3f754ae0019b2d931 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 17:56:37 +0200 Subject: [PATCH 2/3] Issue 330: name the installer's user list among its fixes --- .../00-report.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md b/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md index 50964ce2..f231b118 100644 --- a/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md +++ b/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md @@ -2,7 +2,7 @@ status: located opened: 2026-10-08 located-in: [mesh-controller internal/broker (streams.go, the streams the controller declares; derived.go, each event consumer's maximum), mesh-controller internal/link (advisories.go, the maximum-deliveries advisory)] -fixed-by: mesh-controller#159, mesh-tools#24, mesh-lab#70 +fixed-by: mesh-controller#159, mesh-tools#24, mesh-lab#70, mesh-host#60 replay: R330 amended-design: 03-DESIGN/01-to-be/25-the-bus-on-nats.md --- From 6fdaeb79ba1a6b92f3c2f5d8919a02e4d267eb32 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 18:38:54 +0200 Subject: [PATCH 3/3] Issue 334 and ADR 0264: an ask a seat gave up on can only be dropped for now The review refused a grant over every seat's queue, and a copy published again left the original stuck in it. --- ...-a-person-delivers-it-again-or-drops-it.md | 23 +++++---- .../00-report.md | 5 +- .../00-report.md | 49 +++++++++++++++++++ 3 files changed, 66 insertions(+), 11 deletions(-) create mode 100644 04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md diff --git a/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md index eae7324a..ce24b677 100644 --- a/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md +++ b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md @@ -84,11 +84,13 @@ Three things the mesh had decide the shape: 4. **The controller's verb `dead-letters`** lists what DEAD_LETTERS holds, newest first, or for one consumer; shows one whole by its id; **delivers one again** to the consumer that gave it up; or **drops one**. Delivering again and dropping each require `why`, are recorded in the hand-act log first, and are - a person's decision, not a repair (S15 does not count them). An event is delivered again on the - consumer's again subject; an ask to a seat back on its own subject, since a seat's queue has one worker. - A message from any other stream, or a record without its message, can only be dropped, and the verb says - why. Delivering again carries the message's id, so asking twice delivers once. It is served by the - serving controller on its own connection, not by a fresh process. + a person's decision, not a repair (S15 does not count them). Only an event is delivered again, on the + consumer's again subject, and only when that consumer is on the bus and filters its again subject: a + dead letter is never let go as delivered while nothing would receive it. An ask a seat's worker gave up + on, a message from any other stream, and a record without its message can only be dropped, and the verb + says why (asks: [issue 334](../04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md)). + Delivering again carries the message's id, so asking twice delivers once. It is served by the serving + controller on its own connection, not by a fresh process. 5. **The `max-deliveries` condition is open while DEAD_LETTERS holds a message for that consumer**, one condition per consumer, and clears when its messages are delivered again or dropped, never because the server stopped saying it. Its words say who gave up on how many messages, that what they asked for was @@ -103,9 +105,10 @@ Three things the mesh had decide the shape: - Design 25 §1, §3 and §10 hold. The stream §3 names `mesh.events.dead` is DEAD_LETTERS; its subjects carry the stream and consumer. -- The controller may publish on `mesh.events.dead.>`, `mesh.again.>` and every seat's `accept` subjects - (to give an ask back to its seat), and acknowledge on DEAD_LETTER_NOTICES. It already reached the stream - API. +- The controller may publish on `mesh.events.dead.>` and `mesh.again.>`, and acknowledge on + DEAD_LETTER_NOTICES. It already reached the stream API. It gets no grant over the seats' queues. +- Taking the notices never stops the controller serving: when they cannot be taken, that is said as a + condition (`not-kept`) and tried again every minute. - Every module's consumer changes once, when its filters gain its again subject. Its position is kept. - A message given up on is now visible and costs space until somebody acts. A consumer that gives up on many fills DEAD_LETTERS; full, further copies are refused and said, which is the loud failure the mesh @@ -122,7 +125,9 @@ Three things the mesh had decide the shape: its consumer, stream, subject, deliveries and the time it was given up. It fails on the commit before the fix and passes on it. - mesh-controller `internal/link/deadletters_test.go`: delivered again, an event reaches the consumer that - gave it up and not another module that handled it; it is no longer kept, and a second delivery finds + gave it up and not another module that handled it; one is refused, and kept, when its consumer is gone or + does not filter its again subject, and when it is not an event; the list's total is the held count however + few it lists; notices that cannot be taken are said while the controller serves on; it is no longer kept, and a second delivery finds nothing; a dropped one is gone; a notice whose message is gone is kept as a record that cannot be delivered again; a notice taken twice is kept once; the controller acts on its own event delivered again. - mesh-controller `cmd/mesh-controller/deadletters_test.go`: the verb lists, refuses an act without why or diff --git a/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md b/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md index f231b118..b644aa32 100644 --- a/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md +++ b/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md @@ -79,8 +79,9 @@ which [design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §3 and holds it, with its consumer, stream, subject, deliveries and when it was given up. It keeps them with no age until a person acts. - **The controller's `dead-letters` verb** lists them, shows one whole, delivers one again to the consumer - that gave it up and to nobody else, or drops one. Delivering and dropping say why and are recorded in the - hand-act log. + that gave it up and to nobody else, or drops one. An ask a seat's worker gave up on can only be dropped + for now ([issue 334](../334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md)). + Delivering and dropping say why and are recorded in the hand-act log. - **The `max-deliveries` condition** is open per consumer while DEAD_LETTERS holds its messages, and says where to act: "deliver them again or drop them, from the mesh MCP server." - The comments that said a failing event dead-letters (the controller's streams and derived consumers, the diff --git a/04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md b/04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md new file mode 100644 index 00000000..86d62ec5 --- /dev/null +++ b/04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md @@ -0,0 +1,49 @@ +--- +status: open +opened: 2026-10-08 +located-in: [mesh-controller internal/link (deadletters.go, AgainTo), mesh-controller internal/broker (nats.go, the controller's grants)] +fixed-by: +amended-design: +--- + +# 334. An ask a seat's worker gave up on can only be dropped + +## Symptom + +[ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md) +keeps every message a durable consumer gives up on in DEAD_LETTERS. The controller's `dead-letters` verb +delivers an **event** again to the one consumer that gave it up. An **ask** to a seat that its worker gave up +on (a seat's work queue, `SEAT_`, consumer `SEAT__worker`, five deliveries) is kept and said +the same way, but the verb refuses to deliver it again: it can only be dropped, and its sender asked to say +it again. + +Found in the review of the fix for issue 330 (2026-10-08). The first version published the ask again on its +own subject. The review refused it for two reasons: + +- **The grant.** It needed the controller to publish on every seat's `accept` subjects + (`mesh.seat.*.accept.>`), wider than the seats it asks today, and wider than least authority allows. +- **The original stays.** On a work-queue stream, a message its consumer gave up on is never acknowledged. + It stays in the queue until the stream's age drops it, so publishing a copy leaves two. + +## Why it is an issue + +An ask given up on is said and kept, but its answer is "drop it and have the sender ask again". The sender +may be a person or a module that does not know its ask was given up. The mesh holds the message and cannot +act on it. + +## How it is checked + +When fixed: a test in mesh-controller where a seat's worker gives up on an ask, the verb delivers it again to +that seat's worker, the original is gone from the work queue, and the controller's grant names only the +seats the mesh declares. + +## Core or not + +**Core.** It is located in the controller, so it resolves with a replay in mesh-lab's replays register or a +`replay-none:` saying why. + +## Suggested fix + +Grant the controller each declared seat's `accept` subjects from the catalogue (as its build seats are +granted now), deliver the ask again on its own subject, and delete the original from the work queue by its +`Mesh-Dead-Sequence` once the copy is published.