diff --git a/02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md b/02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md new file mode 100644 index 00000000..6011f7d5 --- /dev/null +++ b/02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md @@ -0,0 +1,96 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-08 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md +--- + +# 250. A merge heard only from the bus's history is retired, and owed work the forge cannot do ends + +## Context + +[ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md) +gives the delivery's owner one table of states. Decision 10, *the switch*, adopts the walks open when the +owner is first assigned. It does not mention the merges the bus still holds from before. A new consumer of the +forge's `pull.merged` hears them all again. The owner took each as a fresh merge whose head it had never +heard, and its row *merged unchecked → held* held each one for a person. + +At the first switch that was 688 deliveries, merged from four days to two hours earlier, all on the machines +long before +([issue 313](../04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md)). +`held` could be left only by a release, which walks, or a stop, which says a person stopped a running +delivery. Neither is true of history. The self-check listed every one past its bound. + +Decision 5 says owed work is "retried until done". A view on a pull request the forge no longer holds can +never be done. It was tried every ten seconds, 11,000 times each, and nothing said so. + +## Considered Options + +1. **A final state `retired` for history, taken at adoption, and a person's act for what is already held.** + Chosen. +2. **Adopt history as `delivered`.** It did reach the machines, but this owner did not deliver it, and + `delivered` is the word for a walk this owner saw done. Rejected. +3. **Stop them.** `stopped` means a person ended a delivery, and its status is an error on the commit. + Rejected. +4. **Let healer H2's `close` take held history.** `held` is the operator's by design. A healer leaving it + unasked would hide the case where the rule is wrong. Rejected for the deliveries already held. New history + is not held at all, so no healer is needed for it. +5. **Skip merges from the bus's history altogether.** Then the commit's note would keep no record that this + owner heard them, and a merge heard late for another reason would go unrecorded. Rejected. + +## Decision + +1. **History is a delivery made from the forge's word at its merge, heard fifteen minutes or more after + that merge, and with no walk of it open.** The forge says a merge within seconds, so one heard that late was + replayed. A walk the controller keeps open or waiting for it makes it a delivery, never history. +2. **A new final state, `retired`.** It is reached in two ways: + - *history*, an observed row from `proposed`, `checked`, `ready` and `rejected`, taken before any other. + New history goes there at adoption and is never checked, held or walked; + - *retire*, a person's act from `held`, with why, whose guard is the rule of point 1. It is never a release: + nothing is walked, delivered or put back. +3. **The seat gains `retire-history`.** It takes one delivery by id and is refused when the rule does not + take it. Or it takes every held delivery, of one repository when named, and answers how many it retired and + which it left held and why. `dry` answers the same and changes nothing. The verb is promised by the + controller as optional before the holder serves it + ([ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md)). +4. **History is not shown on its pull request.** Its transitions are said on the bus and kept on the commit's + note. No view or `mesh/delivery` status is asked: the forge may no longer hold that pull request, or may + hold another under its number. +5. **Owed work ends.** This replaces "retried until done" in decision 5: + - the forge's own answer that what an effect is for does not exist (a 404, or "does not exist") is final. + The effect is given up with one line among the owner's refusals; + - any other failure is tried again later each time: ten seconds, doubling to an hour. After sixty tries, + about two days, it is given up the same way. + +## Consequences + +- The deliveries held at the first switch are retired by one call of `retire-history`, after a dry run says + how many. The self-check's findings for them end. +- A switch, a reassignment or a lost consumer position no longer turns the bus's history into work for a + person. +- A merge heard late that is not history, because its walk waits, is held as before. +- An owed effect can now be lost. When it is, the owner says so, and the commit's note keeps whatever was + already written. + +**How it is checked.** mesh-delivery's tests: `TestAMergeLongPastHeardFromTheForgesWordIsRetiredAtAdoption` +(history retired at adoption, with nothing asked of its pull request; a live merge from the forge's word and +a merge whose walk is open are not history; a reused number does not merge an old delivery again), +`TestRetireHistoryRetiresTheAdoptedHistoryInBulkAndLeavesEveryOtherHeldOne` (dry first, then the bulk retire; +nothing asked of the controller; a retired delivery cannot be released), +`TestRetireHistoryIsRefusedForARealHeldDelivery`, and `TestOwedWorkTheForgeSaysCanNeverBeDoneIsDropped` (a +final answer is given up after one try and said; other failures back off and end). The table's own tests +take and refuse both new rows. The controller's `TestTheDeliverySeatPromisesRetireHistoryOptionally` checks +the seat promises the verb as optional. + +## References + +- [Issue 313](../04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md), + its report and diagnosis. +- [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md), + decisions 2, 5 and 10. +- [ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md), a verb promised before it is + required. +- [Design 47](../03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md), *The state table*, *What every + transition does* and *The verbs*. diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index 9ab4c2e4..01708154 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -4,6 +4,7 @@ status: designed code: [] updated: 2026-10-08 decisions: + - 02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md - 02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md - 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md - 02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md @@ -73,12 +74,18 @@ allows. | published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a newer delivery took over its walk; `go` when its walk started | | held | it waits for a person | 1 day | nothing: it is the operator's | | delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says | -| delivered, failed, superseded, stopped | final | — | — | +| delivered, failed, superseded, stopped, retired | final | — | — | The transitions are those of ADR 0239 decision 2. A delivery that is merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A delivery merged with no walk opened for it within ten minutes is held too, saying so: nothing it moves follows that branch, or the merge -was not heard. Nothing waits silently. **Nothing of a published delivery is registered before its walk +was not heard. Nothing waits silently. + +**History is retired, not held** ([ADR 0250](../../02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md)). +A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no walk +of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes +to `retired` at adoption, before any check, hold or walk, and its pull request is not told. A held delivery that +fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its walk starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a build registered before its turn would be carried by somebody else's send. @@ -113,15 +120,16 @@ them. In this order, each by its owner: 1. **kept**: mesh-delivery puts the delivery in its state (`deliveries`, one key per delivery; `groups`, one - key per group). Nothing is said before it is kept. What it owes outside is kept with it and retried until - done; + key per group). Nothing is said before it is kept. What it owes outside is kept with it and retried, later + each time, from ten seconds doubling to an hour. The forge's answer that what it is for does not exist ends + it at once, and sixty failed tries end it too, each said among the owner's refusals (ADR 0250); 2. **said**: the event `mesh-delivery.transition` (and `mesh-delivery.group` when a group's derived state changes), with the delivery's id, from, to, event, why and plan summary, and no secret or address; 3. **noted**: mesh-delivery asks the forge's holder (`gitea_note_append`) to append a line to the commit's note: the head while the delivery is off the trunk, the commit it landed as once it is on it, and at its end a line of what was executed there. The forge's holder writes it in its own repository as the forge's own account, and a line already there is not appended again; -4. **reflected**: the delivery's view, one comment on the pull request with the state, the plan, the machine +4. **reflected** (not for history, whose pull request the forge may no longer hold): the delivery's view, one comment on the pull request with the state, the plan, the machine steps and the group, kept current (`gitea_delivery_view`); the status `mesh/delivery` on the head, and `mesh/delivery-group` on every member's head (`gitea_commit_status`). Every status links to the pull request, `mesh/merge-gate` among them. @@ -142,6 +150,7 @@ In this order, each by its owner: | `release` | a held delivery to delivering, with why: a person's word | | `stop` | any delivery not final to stopped, with why; its walk is ended through the controller | | `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise | +| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing | **The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`, `delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop`, diff --git a/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md b/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md new file mode 100644 index 00000000..9ca202b7 --- /dev/null +++ b/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md @@ -0,0 +1,73 @@ +--- +status: located +opened: 2026-10-08 +located-in: [mesh-catalog modules/mesh-delivery (holder.go, adoption of a merge from the forge's word; effects.go, the owed effects; table.go, the state table)] +fixed-by: +amended-design: 03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +--- + +# 313. History was held for a person, and its views were retried for ever + +## Symptom + +On 2026-10-08 the controller's self-check D14 failed with 689 findings of one kind: + +> the delivery novox/hq@018ee359ae52 has been held for 33h58m, past its bound of 24h (it waits for the +> operator) + +They spread over ten repositories: the hq repository 233, the catalogue 174, the controller 160, the +node-engine 58, the tool runner 37, the SDK 14, the media catalogue 7, two personal repositories 4 and the lab 2. +688 of them say the same thing when shown with `mesh-delivery.show`: + +- the pull request merged days before the delivery's owner existed — the one shown on 2026-10-02; +- the delivery was made at the owner's first start, 2026-10-06 23:05, within two seconds of all the + others, with the transition "merged; its head was not heard by this owner, so it is made from the + forge's word"; +- then "merged-unchecked → held: merged without a passing check: only a person decides that it goes on". + +Every one of those commits had reached the machines long before. Nothing was to be delivered, released or put +back for any of them, and yet each waited for a person, and the self-check said so 688 times. + +Beside them, 488 owed a **view** — the comment on the pull request — that failed every ten seconds: the one shown +11,051 times, each time with the forge's answer "issue does not exist". The forge's numbers began again around +2026-10-04, so the old pull requests are gone. Where a number had been reused, the view of an old delivery +was asked of a new, unrelated pull request. + +## Why it is a design issue + +[ADR 0239](../../02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md) +decision 10, *the switch*, adopts the walks open when mesh-delivery is assigned. It says nothing of the merges the +bus still holds from before: the forge's `pull.merged` is kept on the bus and heard again by a new consumer. +Each was taken as a merge just made whose head was never heard, so the table's row *merged unchecked → held* +held each for a person. The table had no state for "this already happened", and `held`'s only ways on were +a release, which would walk it, and a stop, which says a person stopped a delivery that was never running. +Healer H2's `close` has no transition from `held`, rightly: a held delivery is the operator's. + +Decision 5 says what a transition owes outside is "retried until done". An effect the forge answers can never +be done was retried without end, every ten seconds, and nothing said so. + +## Fix + +- **A new final state, `retired`: history.** A delivery made from the forge's word at its merge, whose merge + was heard fifteen minutes or more after it happened and whose walk, if the controller keeps one, is not + open, is history. At adoption it goes straight to `retired` by the observed row *history*, before any check, + hold or walk. Its pull request is not told: the forge may no longer hold it, or holds another under its + number. Its transitions are still said on the bus and kept on the commit's note. +- **A person's verb for the ones already held: `retire-history`**, with why. One delivery by id, refused when + the rule does not take it, or every held one (of one repository when named), answering how many it retired + and naming each one it left held. `dry` says what it would do and changes nothing. It is the act + *held → retired* of the table, never a release: nothing is walked, delivered or put back. +- **Owed work ends.** The forge's answer that what an effect is for does not exist (a 404, or "does not + exist") is final: the effect is given up with one line, kept among the owner's refusals. Any other failure is + tried again later each time, from ten seconds doubling to an hour, and given up after sixty tries, saying so. +- A merge announced under a number an older delivery already merged with is not merged into that delivery + again: a forge whose numbers began again reuses them. + +The seat's protocol gains `retire-history` as an optional verb +([ADR 0246](../../02-DECISIONS/0246-a-seats-new-verb-is-promised-before-it-is-required.md)): the controller +promises it first, then the holder serves it. The decision is +[ADR 0250](../../02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md). + +**Not core.** The defect and its fix are in the delivery's owner, a catalogue module that is not the bus, the +forge or the build agent. The controller's part is one optional verb in a seat's protocol. Per `cycle.py`'s +CORE rule, no replay is required. diff --git a/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/01-diagnosis.md b/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/01-diagnosis.md new file mode 100644 index 00000000..d8ec9d79 --- /dev/null +++ b/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/01-diagnosis.md @@ -0,0 +1,37 @@ +# 313 — diagnosis + +**2026-10-08.** Read from the delivery's owner, `mesh-delivery.show` and `deliveries`: + +- 699 deliveries were `held`. 688 say "merged without a passing check", all made between 23:05:27 and + 23:05:29 on 2026-10-06, the owner's first start. A sample across repositories shows merges from + 2026-10-02 to two hours before that start, each with the transition "its head was not heard by this owner, + so it is made from the forge's word", then *merged unchecked → held*. None has a walk. +- The other 11 are held for other reasons: a merge with no walk opened within ten minutes (10), and a group + whose composed check did not pass (1). They are not history, and this issue leaves them alone. One of them + was past its bound, which makes the 689th finding. + +**Where it comes from.** The holder subscribes to the forge's `pull.merged` on the bus after reading back its +state and the controller's walks. The bus hands a new consumer the merges it still holds. For each, the holder +finds no delivery for the head, so it makes one from the forge's word. The pull request's statuses at its +merge do not include a passing `mesh/merge-gate` (the check did not exist for most of them), so the table's +*merged unchecked* row holds it. + +**What would close them, and why not.** + +- `release` walks a held delivery: wrong, these reached the machines already. +- `stop` takes it to `stopped`, which says a person stopped a delivery and sets `mesh/delivery` to error on + the commit. Wrong in meaning, and it would ask the forge for a view on a pull request that is gone. +- Healer H2's `close` reads `held`'s bound, which names no H2 transition, and refuses: "the state is the + operator's". Checked in the code (`Close`, verbs.go) and in the table: it would not release them, and it would + not retire them either. + +**The view retried for ever.** `Flush` tries every owed effect on every tick, ten seconds apart, keeping only +a count and the last error. A view on a pull request the forge no longer holds fails with the forge's +"issue does not exist", and nothing in the holder treated any answer as final. Where a number was reused, +the old view was asked of the new pull request under that number. + +**Ruled out:** the forge's holder. It answers what the forge says. The forge's renumbering is outside the +mesh: the holder must tolerate it, not prevent it. + +**Located** in the delivery's owner: its adoption of a merge made from the forge's word (holder.go +`PullMerged`), its table, which had no state for history, and its owed effects (effects.go `Flush`).