diff --git a/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md new file mode 100644 index 00000000..ce24b677 --- /dev/null +++ b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md @@ -0,0 +1,150 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-08 +deciders: jochen +reconstructed: false +--- + +# 264. A message a consumer gives up on is kept until a person delivers it again or drops it + +## Context + +[Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) promises it three times: an event is +"dead-lettered when it cannot be" delivered (§1); "after `max-deliver` attempts the advisory feeds +`mesh.events.dead` (its own small stream)" (§3); and "an event whose consumer keeps failing dead-letters +after `max-deliver`" is how it is checked (§10). [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) +decided the dead-letter destination on the broker the mesh had before, and +[ADR 0106](0106-the-bus-is-nats.md) names dead-lettering in how the bus is checked. + +None of it exists ([issue 330](../04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md)). +A durable consumer on the bus hands a message over at most five times. When it gives up, the server +publishes a maximum-deliveries advisory naming the stream, the consumer and the message's sequence. The +controller heard it and raised a `max-deliveries` condition, which cleared an hour after the advisories +stopped. The message stayed only in its source stream: EVENTS drops an event after a week. On 2026-10-06 a +media server's consumer gave up on several downloads that way; nothing records which, and they cannot be +delivered again. + +Three things the mesh had decide the shape: + +- **The server says it, and says it once.** An advisory is a core message. A controller restarting, which + happens with every delivery of the controller, would miss one said meanwhile, and the message with it. +- **A message on EVENTS has many consumers.** Publishing it again on its own subject hands it to every + module that consumes it, not only to the one that gave up. A merge would be built twice; a download taken + twice. +- **A module's runtime reads an event's key from its subject**, from the token before `.event.` and what + follows it, in both runtimes (the TypeScript binding and the Go one). It does not read the subject's + first tokens. + +## Considered Options + +### Where the server's word is kept + +1. **The controller's subscription to the advisory, as it was.** Lost while no controller listens. + Rejected. +2. **A stream that captures the advisory subject itself**, which the controller takes from. The server + writes it whether or not anyone listens. Chosen. + +### Where the message is kept + +1. **Leave it in its source stream and keep a pointer.** The source drops it after a week; a pointer to + nothing is the fault this issue describes. Rejected. +2. **A copy in a stream of its own**, fetched by its sequence as soon as the notice is taken, while the + source still holds it. Chosen. + +### How it is delivered again to its consumer alone + +1. **Publish it again on its own subject.** Every consumer of the subject gets it again. Rejected. +2. **Make the consumer start again before the message** (delete and re-create it). It would hand over + everything after the message as well. Rejected. +3. **A subject only that consumer filters.** Every consumer on EVENTS also filters + `mesh.again..>`; a message delivered again is published on `mesh.again..` and its + own subject without `mesh.`, so the runtime reads the same key from it. Chosen. It needs no change to + any runtime, and the controller's own consumer reads it back to the subject it was published on. + +## Decision + +1. **Two streams beside EVENTS, declared by the controller like the others:** + - **DEAD_LETTER_NOTICES** captures `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>`, as a work queue kept + for a week (the longest a source keeps what a notice is about). The controller's consumer `controller` + on it has no maximum of deliveries: a notice it could not act on is offered again a minute later. + - **DEAD_LETTERS** holds the kept messages under `mesh.events.dead..`. It keeps them + with no age and up to 256 MiB, and when full it refuses the next copy rather than dropping one it + holds. Its de-duplication window is a day. +2. **The serving controller takes each notice**, fetches the message it names from its source by sequence, + and keeps a copy in DEAD_LETTERS with the message's own headers and body, and with its stream, consumer, + subject, sequence, deliveries, the time it was given up and the time it was first published. The + server's own headers (`Nats-…`) are not copied, because they describe the first publish and would refuse + or de-duplicate the copy. The copy's id is `..`, so one message is kept once + however often its notice is taken. A message its source no longer holds is kept as a record that says + so. A copy that cannot be made is said in the controller's log and as the consumer's condition, and the + notice is taken again. +3. **Every consumer on EVENTS filters its own `mesh.again..>`**, and EVENTS takes `mesh.again.>`. + The controller's consumer reads a message there as the subject it was first published on. +4. **The controller's verb `dead-letters`** lists what DEAD_LETTERS holds, newest first, or for one + consumer; shows one whole by its id; **delivers one again** to the consumer that gave it up; or **drops + one**. Delivering again and dropping each require `why`, are recorded in the hand-act log first, and are + a person's decision, not a repair (S15 does not count them). Only an event is delivered again, on the + consumer's again subject, and only when that consumer is on the bus and filters its again subject: a + dead letter is never let go as delivered while nothing would receive it. An ask a seat's worker gave up + on, a message from any other stream, and a record without its message can only be dropped, and the verb + says why (asks: [issue 334](../04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md)). + Delivering again carries the message's id, so asking twice delivers once. It is served by the serving + controller on its own connection, not by a fresh process. +5. **The `max-deliveries` condition is open while DEAD_LETTERS holds a message for that consumer**, one + condition per consumer, and clears when its messages are delivered again or dropped, never because the + server stopped saying it. Its words say who gave up on how many messages, that what they asked for was + not done, that the mesh keeps them, and what the operator does: "Needs you: deliver them again or drop + them, from the mesh MCP server." A copy that could not be made is its own condition (token `not-kept`), + kept for an hour after the last failure, as the advisories were. +6. **The writers table gains a row** (to-be 45 §1): a message a consumer gave up on is written by the + controller alone, on `mesh.events.dead.>` and `mesh.again.>`, and read, delivered again or dropped + through its verb. + +## Consequences + +- Design 25 §1, §3 and §10 hold. The stream §3 names `mesh.events.dead` is DEAD_LETTERS; its subjects carry + the stream and consumer. +- The controller may publish on `mesh.events.dead.>` and `mesh.again.>`, and acknowledge on + DEAD_LETTER_NOTICES. It already reached the stream API. It gets no grant over the seats' queues. +- Taking the notices never stops the controller serving: when they cannot be taken, that is said as a + condition (`not-kept`) and tried again every minute. +- Every module's consumer changes once, when its filters gain its again subject. Its position is kept. +- A message given up on is now visible and costs space until somebody acts. A consumer that gives up on + many fills DEAD_LETTERS; full, further copies are refused and said, which is the loud failure the mesh + prefers to dropping what it holds. +- A module handler sees an event delivered again as the same key. It can tell it was delivered again from + the `Mesh-Delivered-Again` header, and need not. +- What a desk click may do does not change ([ADR 0258](0258-a-desk-click-performs-only-an-acknowledgement-and-an-answer-is-a-one-time-ticket.md)): + delivering again and dropping take a person's why, so the notification offers no button for them. + +## How it is checked + +- mesh-controller `internal/link/replay330_test.go`, **R330** in mesh-lab's replays register: a module's + consumer as the controller makes it never acknowledges an event; once it gives up, the event is kept with + its consumer, stream, subject, deliveries and the time it was given up. It fails on the commit before the + fix and passes on it. +- mesh-controller `internal/link/deadletters_test.go`: delivered again, an event reaches the consumer that + gave it up and not another module that handled it; one is refused, and kept, when its consumer is gone or + does not filter its again subject, and when it is not an event; the list's total is the held count however + few it lists; notices that cannot be taken are said while the controller serves on; it is no longer kept, and a second delivery finds + nothing; a dropped one is gone; a notice whose message is gone is kept as a record that cannot be + delivered again; a notice taken twice is kept once; the controller acts on its own event delivered again. +- mesh-controller `cmd/mesh-controller/deadletters_test.go`: the verb lists, refuses an act without why or + with an argument it would pass over before anything is done, drops with the act recorded as a person's + decision; the condition is open per consumer while messages are held, in plain words that say where to + act, and is not raised when none are. +- mesh-controller `internal/broker`: the streams' retention, the consumer's again filter, the writers table + against to-be 45 §1, and the composed grants against their golden file. +- mesh-tools `node-tools/internal/bus/bus_test.go`: an event delivered again has the key it had. +- **Live**, once delivered: the self-check (D7) finds both streams as declared; `nats_streams` lists them; + the next consumer that gives up shows in `mesh-controller.dead-letters`. + +## References + +- [Issue 330](../04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md) +- [Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1, §3, §10; + [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §1 and S9 +- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md), [ADR 0106](0106-the-bus-is-nats.md), + [ADR 0253](0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md) +- [Research 036](../01-RESEARCH/036-everything-over-nats/00-overview.md) (G7) diff --git a/03-DESIGN/01-to-be/25-the-bus-on-nats.md b/03-DESIGN/01-to-be/25-the-bus-on-nats.md index 72d0f1f9..feb07657 100644 --- a/03-DESIGN/01-to-be/25-the-bus-on-nats.md +++ b/03-DESIGN/01-to-be/25-the-bus-on-nats.md @@ -8,8 +8,9 @@ code: - mesh-catalog modules/nats (to be written) - mesh-sdk src (the protocol's NATS binding, step 3) - mesh-tools node-tools/internal/bus (a module's state, ADR 0201) -updated: 2026-10-06 +updated: 2026-10-08 decisions: + - 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md - 02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md - 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md - 02-DECISIONS/0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md @@ -167,7 +168,9 @@ Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStrea |---|---|---|---| | CONTROL | `mesh.control.>` except `alive` (a build's outcome moved to its seat, ADR 0121) | work queue, one consumer (the controller), explicit ack | the store-window guarantee ([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)): the controller `nak`s with a delay while its store is away and the message is redelivered; nothing is dropped | | NODES | `mesh.node.>` | last per subject | one declaration per node, always the newest | -| EVENTS | `mesh.mod.*.event.>` and `mesh.seat.*.event.>` | limits (age, size), durable consumer per subscribing module | a subscriber that was down catches up; after `max-deliver` attempts the advisory feeds `mesh.events.dead` (its own small stream). *2026-10-01:* a build's whole log is here too, as the build seat's `log.` events ([ADR 0157](../../02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)) — one subject per build, a week of retention, read back by `builds --log ` with a consumer that is gone when the reading is done | +| EVENTS | `mesh.mod.*.event.>` and `mesh.seat.*.event.>`; and `mesh.again.>` (ADR 0264) | limits (age, size), durable consumer per subscribing module | a subscriber that was down catches up; after `max-deliver` attempts the advisory feeds `mesh.events.dead` (its own small stream). *2026-10-08, [ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md):* that stream is DEAD_LETTERS, below; each consumer also filters `mesh.again..>`, where a message it gave up on is delivered again to it alone. *2026-10-01:* a build's whole log is here too, as the build seat's `log.` events ([ADR 0157](../../02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)) — one subject per build, a week of retention, read back by `builds --log ` with a consumer that is gone when the reading is done | +| DEAD_LETTER_NOTICES | the server's maximum-deliveries advisory | work queue, a week; one consumer (the controller), no `max-deliver` | the server's word that a consumer gave up on a message, kept whether or not a controller is listening, until the controller has copied the message (ADR 0264) | +| DEAD_LETTERS | `mesh.events.dead..` | limits: no age, 256 MiB, refuses new when full; a day's de-duplication | every message a consumer gave up on, with its consumer, subject, deliveries and when, until a person delivers it again or drops it with why (ADR 0264); a `max-deliveries` condition is open per consumer while it holds any | | `KV__` | `$KV._.>` | the newest value per key — as many past values as the owner declared — no age unless the owner declared one; a value at most 256 KiB, a bucket at most 64 MiB | a module's state (ADR 0201): one per name in a manifest's `state`, created from the catalogue on every raise, so it exists before its owner runs anywhere; kept when the module is unassigned, because what it holds is data | Tool calls and heartbeats stay on core NATS: a lost heartbeat is the next heartbeat; a lost tool diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index c1396176..858f6fef 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -5,6 +5,7 @@ code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab] updated: 2026-10-08 decisions: - 02-DECISIONS/0265-a-verb-that-reads-is-answered-on-the-serving-controllers-own-connection-and-every-connection-says-what-it-is.md + - 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md - 02-DECISIONS/0261-an-assignment-made-while-its-modules-build-runs-is-kept-pending-and-the-controller-makes-it.md - 02-DECISIONS/0254-a-wait-for-a-persons-new-login-is-not-a-gates-failure-and-each-module-of-a-send-keeps-its-own-pass.md - 02-DECISIONS/0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md @@ -87,6 +88,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone | the operator-channel's open messages | the seat's holder | its own key-value state | — | | a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps | | the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads | +| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why | A bus subject two components may publish on is refused when the controller composes grants, unless this table marks it shared. Changing a writer is a change to this table, through a decision. @@ -223,7 +225,7 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age | S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — | | S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — | | S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — | -| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` | +| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` | | S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — | | S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — | | S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — | diff --git a/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md b/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md index 94ea33bb..b644aa32 100644 --- a/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md +++ b/04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md @@ -2,8 +2,9 @@ status: located opened: 2026-10-08 located-in: [mesh-controller internal/broker (streams.go, the streams the controller declares; derived.go, each event consumer's maximum), mesh-controller internal/link (advisories.go, the maximum-deliveries advisory)] -fixed-by: -amended-design: +fixed-by: mesh-controller#159, mesh-tools#24, mesh-lab#70, mesh-host#60 +replay: R330 +amended-design: 03-DESIGN/01-to-be/25-the-bus-on-nats.md --- # 330. The dead-letter stream design 25 promises does not exist @@ -65,3 +66,27 @@ The controller declares a small dead-letter stream beside EVENTS. On each maximu copies the given-up message (the advisory names its stream and sequence) into that stream with the consumer's name and when. A verb lists what it holds, delivers one again or drops it, as a hand act. If the operator decides against a stream, design 25 §1, §3 and §10 are amended to say what happens instead. + +## Resolution + +Decided in [ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md), +which [design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §3 and +[to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §1 and S9 now name: + +- **DEAD_LETTER_NOTICES** captures the server's maximum-deliveries advisory, so a notice said while no + controller listens waits for the next one. +- **DEAD_LETTERS** keeps a copy of each message given up on, fetched by its sequence while the source still + holds it, with its consumer, stream, subject, deliveries and when it was given up. It keeps them with no + age until a person acts. +- **The controller's `dead-letters` verb** lists them, shows one whole, delivers one again to the consumer + that gave it up and to nobody else, or drops one. An ask a seat's worker gave up on can only be dropped + for now ([issue 334](../334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md)). + Delivering and dropping say why and are recorded in the hand-act log. +- **The `max-deliveries` condition** is open per consumer while DEAD_LETTERS holds its messages, and says + where to act: "deliver them again or drop them, from the mesh MCP server." +- The comments that said a failing event dead-letters (the controller's streams and derived consumers, the + tool runner's NATS binding) say what happens now. + +**Replay R330** (mesh-controller `internal/link/replay330_test.go`, registered in mesh-lab) fails on the +commit before the fix and passes on it. + diff --git a/04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md b/04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md new file mode 100644 index 00000000..86d62ec5 --- /dev/null +++ b/04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md @@ -0,0 +1,49 @@ +--- +status: open +opened: 2026-10-08 +located-in: [mesh-controller internal/link (deadletters.go, AgainTo), mesh-controller internal/broker (nats.go, the controller's grants)] +fixed-by: +amended-design: +--- + +# 334. An ask a seat's worker gave up on can only be dropped + +## Symptom + +[ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md) +keeps every message a durable consumer gives up on in DEAD_LETTERS. The controller's `dead-letters` verb +delivers an **event** again to the one consumer that gave it up. An **ask** to a seat that its worker gave up +on (a seat's work queue, `SEAT_`, consumer `SEAT__worker`, five deliveries) is kept and said +the same way, but the verb refuses to deliver it again: it can only be dropped, and its sender asked to say +it again. + +Found in the review of the fix for issue 330 (2026-10-08). The first version published the ask again on its +own subject. The review refused it for two reasons: + +- **The grant.** It needed the controller to publish on every seat's `accept` subjects + (`mesh.seat.*.accept.>`), wider than the seats it asks today, and wider than least authority allows. +- **The original stays.** On a work-queue stream, a message its consumer gave up on is never acknowledged. + It stays in the queue until the stream's age drops it, so publishing a copy leaves two. + +## Why it is an issue + +An ask given up on is said and kept, but its answer is "drop it and have the sender ask again". The sender +may be a person or a module that does not know its ask was given up. The mesh holds the message and cannot +act on it. + +## How it is checked + +When fixed: a test in mesh-controller where a seat's worker gives up on an ask, the verb delivers it again to +that seat's worker, the original is gone from the work queue, and the controller's grant names only the +seats the mesh declares. + +## Core or not + +**Core.** It is located in the controller, so it resolves with a replay in mesh-lab's replays register or a +`replay-none:` saying why. + +## Suggested fix + +Grant the controller each declared seat's `accept` subjects from the catalogue (as its build seats are +granted now), deliver the ask again on its own subject, and delete the original from the work queue by its +`Mesh-Dead-Sequence` once the copy is published.