Merge pull request 'Proposed: ADR 0293, an ask the controller made is delivered again to its seat's worker (issue 334)' (#310) from decisions/334-the-controllers-own-ask-is-delivered-again into main

2026-10-11 02:36:25 +00:00
parent d1edcf544b f9677dabea
commit 331e8d0bbe
3 changed files with 119 additions and 2 deletions
@@ -8,6 +8,11 @@ reconstructed: false

 # 264. A message a consumer gives up on is kept until a person delivers it again or drops it

+> **Superseded in part — 2026-10-11, by [ADR 0293](0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md).**
+> Decision 4's "an ask a seat's worker gave up on can only be dropped" no longer holds for an ask the controller
+> made itself: it is delivered again to the seat's worker, after the original leaves the queue. Every other ask,
+> and the rest of this record, stand.
+
 ## Context

 [Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) promises it three times: an event is
@@ -0,0 +1,111 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-11
+deciders: jochen (proposed by the agent session that fixed issue 334; accepted by the operator on 2026-10-11)
+reconstructed: false
+supersedes-in-part:
+  - 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md
+---
+
+# 293. An ask the controller made is delivered again to its seat's worker, and every other ask is only dropped
+
+## Context
+
+[ADR 0264](0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md)
+keeps every message a durable consumer gives up on in DEAD_LETTERS, until a person delivers it again or drops
+it. Its decision 4 delivers again only an event. An ask a seat's worker gave up on (a seat's work queue, its
+worker handing it over five times) can only be dropped, and its sender asked to say it again. Issue 334 records
+the gap: the mesh holds the ask and cannot act on it, and the sender may not know it was given up.
+
+ADR 0264 gave two reasons, from the review of the fix for issue 330:
+
+- **The grant.** Publishing an ask again on its own subject needed the controller to publish on every seat's
+  accept subjects, wider than the seats it asks.
+- **The original stays.** On a work queue, a message its worker gave up on is never acknowledged. It stays in
+  the queue until the stream's age drops it, so publishing a copy leaves two.
+
+Neither reason holds for the asks the controller makes itself. Measured against the controller's composed
+grants on 2026-10-11:
+
+- The controller already publishes on the accept subjects of the two seats it asks, the build seats (the node's
+  build agent and the build machine it replaces). It submits every build there.
+- It already reaches the stream API, and [ADR 0219](0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md)
+  has it cancel a build ask by deleting it from that same work queue. Removing the original before its copy is
+  published is the same act.
+
+The reasons do hold for every other seat. A seat whose accept subject names its caller
+([ADR 0259](0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md) §3,
+`accept.ask.<module>`) proves its asker because only that module may publish there. A controller granted every
+declared seat's accept subjects could speak in any module's name.
+
+## Considered Options
+
+1. **Every ask stays drop-only** (ADR 0264 as decided). It needs nothing new. But the build asks the controller
+   made itself are dropped and asked for again by hand, though the controller holds everything delivering them
+   again needs. Rejected.
+2. **The controller's own asks are delivered again; every other ask stays drop-only.** No grant changes, and no
+   asker is spoken for but the controller itself. Chosen.
+3. **Grant the controller every declared seat's accept subjects, and deliver any ask again.** It closes issue
+   334 whole. But it lets the controller publish in any module's name, against ADR 0259 §3, and gives it the
+   grant over the seats' queues that ADR 0264's consequences withhold. Rejected here. Asks to other seats wait
+   for a decision that answers who may speak for an asker.
+
+## Decision
+
+1. **An ask the controller made is delivered again.** A dead letter kept from the work queue of a seat the
+   controller asks, published on that seat's accept subject, is delivered again on its own subject. The seat's
+   queue has one worker per subject, so only the worker that gave it up takes it. The copy carries
+   `Mesh-Delivered-Again` with the dead letter's id, and a de-duplication id of its own.
+2. **The original leaves the queue first, and only the same message.** Before the copy is published, the
+   message at the dead letter's sequence is read. It is deleted only when its subject and stored time are the
+   dead letter's. A queue deleted and made again numbers from one, and the build handover deletes one, so an old
+   sequence can name a live ask. Anything else is left as it is and the answer says so. A copy that cannot be
+   published leaves the dead letter kept, so delivering it again tries once more, and the queue never holds two.
+3. **Only while the worker is on the bus.** An ask is not delivered again while the worker that gave it up is
+   not on the bus: the queue would hold it for a holder that may never come, and it would be let go as
+   delivered meanwhile. The verb refuses and the dead letter stays kept.
+4. **The asker stays the controller.** The copy is published by the controller on the subject it asked on
+   first, so the asker the seat reads is the one that asked. ADR 0259 §3 is untouched: the build seats' accepts
+   do not name a caller.
+5. **Every other ask is only dropped**, as ADR 0264 decision 4 says, and the verb says why. No grant of the
+   controller changes.
+
+## Consequences
+
+- ADR 0264 decision 4 no longer holds for the controller's own asks; it stands for every other ask, for a
+  message from any stream but EVENTS and a seat's queue, and for a record without its message. Its
+  consequence that the controller gets no grant over the seats' queues stands.
+- **A retry can build twice.** The copy's de-duplication id is the dead letter's, so asking twice within the
+  queue's de-duplication window delivers once. The seat queues set no window, so it is the bus's default of
+  two minutes. A copy whose publish was taken by the server but whose answer was lost, retried after the
+  window, is delivered twice, and the build seat builds it twice. A build is repeatable, so the cost is the
+  time of one build.
+- A build ask that ADR 0219 cancelled or cleared was written to the seat's cancelled set first. A holder checks
+  that set before it builds, so delivering such an ask again does not build it.
+- The plan that waits for a build's outcome has no timeout (ADR 0219). An ask delivered again gives it an
+  outcome instead of a wait that only a person's drop ends.
+- Issue 334 is not resolved by this record: asks to the other seats wait on option 3's question.
+
+## How it is checked
+
+- mesh-controller `internal/link/deadletters_ask_test.go`, on a real bus:
+  - a build ask its worker gave up on is delivered again to that worker on its own subject, the original is gone
+    from the queue, it is handed over once, and a second delivery finds nothing kept;
+  - a copy the server refuses leaves the dead letter kept and the original gone, the answer says both, and the
+    next delivery hands it over once;
+  - a queue made again, whose sequence now holds a live ask, keeps that ask untouched, and the answer says so;
+  - with no worker on the queue, the delivery is refused and the original and the dead letter both stay;
+  - an ask to a seat the controller does not ask, or a build subject on another seat's queue, is refused.
+- mesh-controller `internal/broker`: the composed grants match their golden file, which this change leaves as
+  it was.
+
+## References
+
+- Issue 334 (an ask a seat's worker gave up on can only be dropped); mesh-controller pull request 219.
+- [ADR 0264](0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md)
+  decision 4 and its consequences.
+- [ADR 0219](0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md): the controller
+  deletes a build ask from its queue by sequence, and the cancelled set.
+- [ADR 0259](0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md) §3:
+  a by-caller ask's asker is a fact the server enforces.
@@ -2,8 +2,9 @@
 layer: to-be
 status: in-progress
 code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
-updated: 2026-10-10
+updated: 2026-10-11
 decisions:
+  - 02-DECISIONS/0293-an-ask-the-controller-made-is-delivered-again-to-its-seats-worker-and-every-other-ask-is-only-dropped.md
  - 02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md
  - 02-DECISIONS/0285-an-issue-tracks-a-condition-by-its-key-and-every-message-about-the-condition-says-the-issue-and-where-it-stands.md
  - 02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md
@@ -95,7 +96,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
 | a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps |
 | the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads |
 | the record mirrors: nodes, assignments, walks, builds, settings ([ADR 0289](../../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md), *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
-| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why |
+| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why; an ask is delivered again only when the controller made it itself, every other ask is only dropped (ADR 0293) |

 A bus subject two components may publish on is refused when the controller composes grants, unless
 this table marks it shared. Changing a writer is a change to this table, through a decision.