From ab947a8c1caecf16618d20e1099cb3344a7758b1 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 11 Oct 2026 03:25:53 +0200 Subject: [PATCH 1/2] Propose ADR 0293: the controller delivers its own given-up asks again (issue 334) --- ...-a-person-delivers-it-again-or-drops-it.md | 5 + ...ker-and-every-other-ask-is-only-dropped.md | 111 ++++++++++++++++++ 2 files changed, 116 insertions(+) create mode 100644 02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md diff --git a/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md index ce24b677..d734d60b 100644 --- a/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md +++ b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md @@ -8,6 +8,11 @@ reconstructed: false # 264. A message a consumer gives up on is kept until a person delivers it again or drops it +> **Superseded in part — 2026-10-11, by [ADR 0293](0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md).** +> Decision 4's "an ask a seat's worker gave up on can only be dropped" no longer holds for an ask the controller +> made itself: it is delivered again to the seat's worker, after the original leaves the queue. Every other ask, +> and the rest of this record, stand. ADR 0293 is proposed: until it is accepted, decision 4 holds as written. + ## Context [Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) promises it three times: an event is diff --git a/02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md b/02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md new file mode 100644 index 00000000..4c0288b5 --- /dev/null +++ b/02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md @@ -0,0 +1,111 @@ +--- +topic: the mesh +status: proposed +date: 2026-10-11 +deciders: jochen (proposed by the agent session that fixed issue 334; the operator decides) +reconstructed: false +supersedes-in-part: + - 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md +--- + +# 293. An ask the controller made is delivered again to its seat's worker, and every other ask is only dropped + +## Context + +[ADR 0264](0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md) +keeps every message a durable consumer gives up on in DEAD_LETTERS, until a person delivers it again or drops +it. Its decision 4 delivers again only an event. An ask a seat's worker gave up on (a seat's work queue, its +worker handing it over five times) can only be dropped, and its sender asked to say it again. Issue 334 records +the gap: the mesh holds the ask and cannot act on it, and the sender may not know it was given up. + +ADR 0264 gave two reasons, from the review of the fix for issue 330: + +- **The grant.** Publishing an ask again on its own subject needed the controller to publish on every seat's + accept subjects, wider than the seats it asks. +- **The original stays.** On a work queue, a message its worker gave up on is never acknowledged. It stays in + the queue until the stream's age drops it, so publishing a copy leaves two. + +Neither reason holds for the asks the controller makes itself. Measured against the controller's composed +grants on 2026-10-11: + +- The controller already publishes on the accept subjects of the two seats it asks, the build seats (the node's + build agent and the build machine it replaces). It submits every build there. +- It already reaches the stream API, and [ADR 0219](0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md) + has it cancel a build ask by deleting it from that same work queue. Removing the original before its copy is + published is the same act. + +The reasons do hold for every other seat. A seat whose accept subject names its caller +([ADR 0259](0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md) §3, +`accept.ask.`) proves its asker because only that module may publish there. A controller granted every +declared seat's accept subjects could speak in any module's name. + +## Considered Options + +1. **Every ask stays drop-only** (ADR 0264 as decided). It needs nothing new. But the build asks the controller + made itself are dropped and asked for again by hand, though the controller holds everything delivering them + again needs. Rejected. +2. **The controller's own asks are delivered again; every other ask stays drop-only.** No grant changes, and no + asker is spoken for but the controller itself. Chosen. +3. **Grant the controller every declared seat's accept subjects, and deliver any ask again.** It closes issue + 334 whole. But it lets the controller publish in any module's name, against ADR 0259 §3, and gives it the + grant over the seats' queues that ADR 0264's consequences withhold. Rejected here. Asks to other seats wait + for a decision that answers who may speak for an asker. + +## Decision + +1. **An ask the controller made is delivered again.** A dead letter kept from the work queue of a seat the + controller asks, published on that seat's accept subject, is delivered again on its own subject. The seat's + queue has one worker per subject, so only the worker that gave it up takes it. The copy carries + `Mesh-Delivered-Again` with the dead letter's id, and a de-duplication id of its own. +2. **The original leaves the queue first, and only the same message.** Before the copy is published, the + message at the dead letter's sequence is read. It is deleted only when its subject and stored time are the + dead letter's. A queue deleted and made again numbers from one, and the build handover deletes one, so an old + sequence can name a live ask. Anything else is left as it is and the answer says so. A copy that cannot be + published leaves the dead letter kept, so delivering it again tries once more, and the queue never holds two. +3. **Only while the worker is on the bus.** An ask is not delivered again while the worker that gave it up is + not on the bus: the queue would hold it for a holder that may never come, and it would be let go as + delivered meanwhile. The verb refuses and the dead letter stays kept. +4. **The asker stays the controller.** The copy is published by the controller on the subject it asked on + first, so the asker the seat reads is the one that asked. ADR 0259 §3 is untouched: the build seats' accepts + do not name a caller. +5. **Every other ask is only dropped**, as ADR 0264 decision 4 says, and the verb says why. No grant of the + controller changes. + +## Consequences + +- ADR 0264 decision 4 no longer holds for the controller's own asks; it stands for every other ask, for a + message from any stream but EVENTS and a seat's queue, and for a record without its message. Its + consequence that the controller gets no grant over the seats' queues stands. +- **A retry can build twice.** The copy's de-duplication id is the dead letter's, so asking twice within the + queue's de-duplication window delivers once. The seat queues set no window, so it is the bus's default of + two minutes. A copy whose publish was taken by the server but whose answer was lost, retried after the + window, is delivered twice, and the build seat builds it twice. A build is repeatable, so the cost is the + time of one build. +- A build ask that ADR 0219 cancelled or cleared was written to the seat's cancelled set first. A holder checks + that set before it builds, so delivering such an ask again does not build it. +- The plan that waits for a build's outcome has no timeout (ADR 0219). An ask delivered again gives it an + outcome instead of a wait that only a person's drop ends. +- Issue 334 is not resolved by this record: asks to the other seats wait on option 3's question. + +## How it is checked + +- mesh-controller `internal/link/deadletters_ask_test.go`, on a real bus: + - a build ask its worker gave up on is delivered again to that worker on its own subject, the original is gone + from the queue, it is handed over once, and a second delivery finds nothing kept; + - a copy the server refuses leaves the dead letter kept and the original gone, the answer says both, and the + next delivery hands it over once; + - a queue made again, whose sequence now holds a live ask, keeps that ask untouched, and the answer says so; + - with no worker on the queue, the delivery is refused and the original and the dead letter both stay; + - an ask to a seat the controller does not ask, or a build subject on another seat's queue, is refused. +- mesh-controller `internal/broker`: the composed grants match their golden file, which this change leaves as + it was. + +## References + +- Issue 334 (an ask a seat's worker gave up on can only be dropped); mesh-controller pull request 219. +- [ADR 0264](0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md) + decision 4 and its consequences. +- [ADR 0219](0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md): the controller + deletes a build ask from its queue by sequence, and the cancelled set. +- [ADR 0259](0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md) §3: + a by-caller ask's asker is a fact the server enforces. From f9677dabead790632f527c6a4f2ca9b03fdec869 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 11 Oct 2026 04:21:12 +0200 Subject: [PATCH 2/2] Accept ADR 0293 on the operator's word, and give it its place in to-be 45 (issue 334) The operator chose 2026-10-11 to let the controller deliver its own given-up asks again; every other ask stays drop-only. --- ...n-is-kept-until-a-person-delivers-it-again-or-drops-it.md | 2 +- ...o-its-seats-worker-and-every-other-ask-is-only-dropped.md | 4 ++-- 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md | 5 +++-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md index d734d60b..7c8776a0 100644 --- a/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md +++ b/02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md @@ -11,7 +11,7 @@ reconstructed: false > **Superseded in part — 2026-10-11, by [ADR 0293](0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md).** > Decision 4's "an ask a seat's worker gave up on can only be dropped" no longer holds for an ask the controller > made itself: it is delivered again to the seat's worker, after the original leaves the queue. Every other ask, -> and the rest of this record, stand. ADR 0293 is proposed: until it is accepted, decision 4 holds as written. +> and the rest of this record, stand. ## Context diff --git a/02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md b/02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md index 4c0288b5..75bfd3e8 100644 --- a/02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md +++ b/02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md @@ -1,8 +1,8 @@ --- topic: the mesh -status: proposed +status: accepted date: 2026-10-11 -deciders: jochen (proposed by the agent session that fixed issue 334; the operator decides) +deciders: jochen (proposed by the agent session that fixed issue 334; accepted by the operator on 2026-10-11) reconstructed: false supersedes-in-part: - 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index e2fc12f8..9fb57a63 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -2,8 +2,9 @@ layer: to-be status: in-progress code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab] -updated: 2026-10-10 +updated: 2026-10-11 decisions: + - 02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md - 02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md - 02-DECISIONS/0285-an-issue-tracks-a-condition-by-its-key-and-every-message-about-the-condition-says-the-issue-and-where-it-stands.md - 02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md @@ -95,7 +96,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone | a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps | | the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads | | the record mirrors: nodes, assignments, walks, builds, settings ([ADR 0289](../../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md), *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) | -| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why | +| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why; an ask is delivered again only when the controller made it itself, every other ask is only dropped (ADR 0293) | A bus subject two components may publish on is refused when the controller composes grants, unless this table marks it shared. Changing a writer is a change to this table, through a decision.