diff --git a/00-META/glossary.md b/00-META/glossary.md index 8b749d22..9062b7fa 100644 --- a/00-META/glossary.md +++ b/00-META/glossary.md @@ -278,7 +278,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences` **Purpose.** To take one commit from its pull request to every node that should run it, and to put it back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the planner, the walk). **Downstream of** Module, Core, Health and repair and Data. -**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0267. +**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270. **Uses:** module, node, declaration and send (Placement), health (Health and repair), person. - **merge check** — the two statuses a pull request carries before it may merge: the **merge gate** @@ -313,12 +313,19 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data. modules live in. `mesh-catalog` and `catalog_modules` are identifiers and keep their spelling. *Not:* ~~catalog~~ (hq) - **delivery** — one commit in one repository on its way to the nodes, from its pull request's head - being announced to delivered, failed, superseded or stopped: one pull request, one status, one note + being announced to one of its final states: one pull request, one status, one note ([ADR 0239](../02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md)). Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`, - `held`, and the final four). Not *change* (a diff). *Pipeline* is the predecessor's build-and-deploy + `held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`; + [ADR 0250](../02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md), + [ADR 0270](../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md)). Not *change* (a diff). *Pipeline* is the predecessor's build-and-deploy mechanism, which the as-is designs describe; it is never a word for a delivery. *Not:* ~~deployment~~ (hq) +- **covered** — the end of a delivery that a later merge containing it delivered: every module it moves was + sent to every machine running it by the walk of a later merge into the same trunk, delivered on its own + passing check, whose merge commit the forge says contains this one. Not *superseded*, which is a delivery + whose walk a newer delivery took over before it ended; nor *delivered*, which its own walk or an empty plan + ends in ([ADR 0270](../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md)). - **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One level: a group holds deliveries, never groups. Its state is derived from its members, never set. diff --git a/02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md b/02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md new file mode 100644 index 00000000..857992ef --- /dev/null +++ b/02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md @@ -0,0 +1,203 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-09 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md +--- + +# 270. A held delivery that moves nothing, or that a later delivered merge contains, ends by itself, and a hold whose reason is gone lifts + +## Context + +[ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md) +decision 2 gives a delivery one table of states. A delivery is `held` when only a person can decide whether +it goes on. A person then has four ways out: `release` walks the delivery's own commit, `stop` ends it as +stopped, `retire-history` ends history +([ADR 0250](0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md)), +and the walk can start on another word (`go`). + +On 2026-10-09, fifteen deliveries were held, most for more than a day, each raising probe D14's +`delivery..stalled` (read with `mesh-delivery.deliveries` and `show`). None of them asked a question a +person could answer correctly: + +- **Four merges of hq build nothing.** Their plan says "builds nothing: the change touches no module of the + mesh's graph". They were merged before their check answered, and the row *merged unchecked → held* is tried + before anything reads the plan. +- **Four catalogue merges only add a module.** The controller builds and registers a new module and sends it + nowhere, so it opens no walk ([issue 300](../04-ISSUES/300-a-new-module-is-not-built-at-its-merge/00-report.md)). + mesh-delivery counted the new module as something a walk moves, so the no-walk rule held each one ten + minutes after its merge. +- **Five controller merges had no walk of their own.** A later merge into the same trunk was delivered, and + its walk sent every module these move. The diagnosis of [issue 319](../04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md) + checked each merge commit against the later delivered commit: each was contained in it, and the + controller's `status` listed none of their modules as behind. +- **Two catalogue merges were held for their group's composed check** + ([issue 316](../04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)). + One was published while the check was asked and not yet answered. The code read "not answered" as "did + not pass" and held it. Two minutes later the check passed for exactly the heads that merged, and nothing + read that answer for the held member. The own walk of each of the two was since taken over by a later walk + of the same trunk that sent its module again. + +`release` would walk an old commit over a newer one, or walk nothing. `retire-history` refuses each one, +because their heads were heard. `stop` is the only way out that works, and it is the wrong record: it sets +`mesh/delivery` to `error` on a pull request whose change a later delivered merge carried. Four were stopped that way on +2026-10-09 before the issue was found, and their pull requests now say "stopped: covered: …" as an error. A +condition that only a wrong act can clear teaches people to ignore the list it is in. + +## Considered Options + +1. **Stop them with why.** Rejected: it is what was done, and it records a delivered change as an error on + its pull request. Issue 313 ruled it out for the same reason. +2. **A person's act, `supersede`, that ends a held delivery as superseded.** Rejected as the main path. Every + fact the person would check can be read by the mesh: the plan, the walks' records and the forge's history. + A person asked to confirm a fact the mesh can read is a person asked for nothing. A person keeps `stop` and + `release` for what the facts do not settle. +3. **Trust merge order alone.** Rejected after review. On a trunk that takes no push, a commit the forge + merged later contains every commit merged before it, and mesh-delivery already holds the merge times. But + that rests on the trunk's protection, which is the forge's setting and which nothing in mesh-delivery + checks. A protection lifted once, or a merge time read wrongly, would end a delivery as covered when its + commit is not in the later one. Merge order stays, as the cheap first filter. +4. **Ask the forge whether the later merge commit contains this one** — chosen. The forge module gains a + read-only tool, `gitea_contains`, over the forge's compare of the two commits: none of this commit's history + missing from the later one means it is contained. mesh-delivery is not given the forge's general API tool, + which can write. The tool is TypeScript, as the rest of the forge module is; no Go route to the forge's API + exists for a module to call. Asking the controller what each machine runs was weighed and left: the + controller holds no clones, and what runs is already what the later walk's record says it sent. +5. **Read the facts, and end the delivery by observation**, with point 4 deciding containment — below. + +For issue 316 the alternative was to require `mesh/delivery-group` on the trunks, so that no member can merge +before its composed check passed. That stays the operator's setting, as ADR 0239 left it. It does not settle +a check that answers after the merge. + +## Decision + +1. **A merge that moves nothing is delivered at once.** A delivery whose plan moves no module the mesh holds + is delivered as soon as it is merged and has no walk. That covers a plan that builds nothing and a plan + that only adds a module the controller builds and sends nowhere. This holds whatever its check said: from + `proposed`, `checked` and `rejected`, the row is tried before *merged unchecked → held*, and from `held` + it is tried on every tick. `ready → published → delivered` already did this for a merge that passed. + **A delivery that only adds a module is reported delivered whatever its build's outcome.** The build is + the controller's, asked at the merge, registered when it lands and sent nowhere. A failed build of it + raises no condition and is not this delivery's: the controller prints a line in its log, and nothing + else says it until a person assigns the module, which is refused while no build of it is registered. + This is a known gap, not closed here. +2. **A new final state, `covered`: a later merge that contains it was delivered.** A merged delivery is + covered when, for every module its plan moves, another delivery: + - is of the same repository and was merged into the same trunk, after it by the forge's merge times (the + cheap filter); + - is `delivered`, and its own check passed; + - was delivered by its own walk: the walk's commit is its merge commit, on that trunk; + - sent the module to every machine running it, as its machine steps say (sent to all at once, or to the + rest after its first machines passed), with no step of it failed or put back; + - has a merge commit that contains this one's, as the forge says through `gitea_contains`. + + The forge is asked only once every module has a candidate, once per pair, outside the holder's lock, and its + answer is kept on the delivery. A forge that cannot answer means "not yet covered": the delivery stays where + it is and is asked about again a minute later. A delivery whose own walk is still open is never covered: its + walk delivers it. + + The row is observed. It is tried from `held`, and from `ready` once the ten minutes for a walk to open have + passed, before the no-walk hold. From `held` it is tried before `go`. If a walk a person started fails + between two readings of a covered delivery, the delivery ends `covered`, not `delivering` then `failed`: the + later merge that contains it was delivered whatever its own walk did. The transition's why names each + covering delivery, the commit it merged as and its walk. On the pull request, `mesh/delivery` says + `covered: a later merge that contains it was delivered: …` as a **success**, never an error. A group counts + a covered member as delivered, so the member after it gets its turn and is not stopped. + + `covered` says what was delivered, not what runs now: a revert merged after the covering merge is possible, + and is that revert's own delivery. +3. **A merge held without a passing check is covered without a person** when a later delivery whose own check + passed carries it, and only then. A later delivery's walk runs only on a passing check, so its check has + passed in every case this matters for. What a person would have been asked, "does this unchecked change go + on?", was answered by the later change, which was checked with this one's change in its history and + delivered. +4. **A group member is not held for a composed check that has not answered.** "Did not pass for the heads that + merged" means the check failed, ran for other heads, or was never asked. A member published while its + group's check is asked and not answered waits in `published`, within that state's bound. +5. **A hold whose reason is gone lifts.** A member held because its group's composed check had not passed is + taken back to `published` when that check passes or warns for the heads that merged, and its own walk is + still open. The event is *lifted*, observed, and the member then takes its turn in the group's order. Every + other hold stays a person's. +6. **Read again on every tick.** The holder reads the facts of every held delivery on each tick, not only when + an event arrives. A delivery that was held before these rows existed, or read back after a restart, ends by + itself. Once it is final, nothing moves it again. + +The table of ADR 0239 decision 2 gains these rows: + +| From | Event | To | Guard | +|---|---|---|---| +| proposed, checked, rejected, held | done | delivered | merged, no walk, and its plan moves no module the mesh holds | +| ready, held | covered | covered | a later merge that contains it was delivered: merged, no walk of its own open, and every module it moves sent to every machine running it by the walk of a later merge into the same trunk, delivered on its own passing check, whose merge commit the forge says contains this one | +| held | lifted | published | held only for its group's composed check, which has since passed or warned for the heads that merged, and its own walk still open | + +`delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered` are final. + +## Consequences + +- **The held deliveries end without a person.** When mesh-delivery with these rows is delivered, its first + ticks deliver the hq merges and the module-adding merges, and cover the controller and catalogue merges + that a later walk carried. Their D14 conditions clear at the self-check's next reading of `stalled`. +- **A pull request no longer says an error for a change a later merge delivered.** The four stopped on 2026-10-09 keep their + error: a final state is not undone. Their commit notes say why. +- **Containment is the forge's answer, not a premise.** The trunk's protection is no longer relied on: the + forge's compare decides. mesh-delivery gains one forge tool, which reads only. +- **The forge's tool must be delivered before `covered` can be reached.** Until `gitea_contains` answers, every + candidate is "not yet covered", and a held delivery stays held as before. Nothing ends wrongly while the forge + module is behind. +- **`covered` is only as complete as the walks' records.** A module the later walk held back for a person is + not sent, so it does not count. A delivery whose modules were only partly carried stays held for a person. +- **A module that runs on one machine only is never covered.** A walk records "the rest running it" only + after its first machines passed and the rest were sent. A module on one machine has no rest, so its steps + never say it reached every machine. mesh-delivery itself, on the control node alone, is such a module: a + held merge of it stays held for a person, as before this record. Reading "every machine running it" from + the first machines when they are all the machines would close it, and is left for a later change. +- **A group member waits longer, rather than being held too soon.** A composed check that never answers leaves + the member in `published`. Its bound of 30 minutes lists it as stalled. That is the right owner for a check + that does not answer. + +**How it is checked.** mesh-delivery's tests, in `covered_test.go`: + +- `TestAMergeThatMovesNothingIsDeliveredAtOnce`: builds nothing, or only a new module sent nowhere; merged + before its check answered, on a failed check, or on a passing one. Each is delivered, and its status is a + success. +- `TestAMergeALaterDeliveredWalkCarriedIsCovered`: covered whether it was held first or not. It stays held + when the later walk carried only one of its modules, merged before it, went into another branch, or failed, + and when the forge says the later merge does not contain it. With the forge away it stays held, and is + covered once the forge answers. The forge is asked once per pair, and never for a pair merge order rules + out. Its status is `success covered: a later merge that contains it was delivered` and names the commit + that covered it, never an error. +- `TestACoveredGroupMemberLetsTheNextOneGo`: the member after a covered one is let go, and the group ends + delivered. +- `TestHeldDeliveriesReadBackCloseByThemselvesOnce`: deliveries held before these rows existed are read back + from the store and end delivered or covered, or stay held when not covered. A merge held without a passing + check is covered by a later delivery whose own check passed, and stays held when that check did not pass. It + stays held when the later delivery's walk was of another commit or another branch, or sent its module to its + first machine only. A second restart moves nothing. +- The forge module's `test/checks.test.ts`: a compare with no commits ahead is contained, one with some is + not, an answer without a count is no answer, and only a sha reaches the compare's path. The forge module is + TypeScript. Its tests, and a type-check of every source, are run on the build seat by the catalogue's + TypeScript part of its check, `merge-check-typescript.sh` (mesh-catalog #151), in the TypeScript toolchain + with the SDK it holds. Before #151 no gate ran them: the catalogue's check ran only Go. +- `TestAGroupMemberWaitsForItsComposedCheckToAnswer` and `TestAMemberHeldForItsComposedCheckIsLiftedWhenItPasses`: + a pass or a warning lifts the member; a failure, or a pass for other heads, keeps it held. + +The table's own tests take and refuse every new row. They also check that the only way to `published` other +than a ready delivery's merge is *lifted* from `held`, and that nothing observed ends a held delivery as +stopped or failed. + +## References + +- [Issue 319](../04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md) and + [issue 316](../04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md), + their reports and diagnoses. +- [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md), + decisions 2, 4 and 9; [ADR 0250](0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md), + the last state added to the table. +- [Issue 300](../04-ISSUES/300-a-new-module-is-not-built-at-its-merge/00-report.md), why a new + module opens no walk. +- [Design 47](../03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md), *The state table* and *A + delivery group*, which carry these rows. +- mesh-catalog #150, which builds it, and the forge module's `gitea_contains` with it; mesh-catalog #151, + which has the catalogue's check run its TypeScript modules. diff --git a/02-DECISIONS/0271-a-send-held-for-the-buss-planned-step-is-told-to-the-operator-at-once-and-a-change-that-needs-it-says-so-before-it-merges.md b/02-DECISIONS/0271-a-send-held-for-the-buss-planned-step-is-told-to-the-operator-at-once-and-a-change-that-needs-it-says-so-before-it-merges.md new file mode 100644 index 00000000..f353c2ae --- /dev/null +++ b/02-DECISIONS/0271-a-send-held-for-the-buss-planned-step-is-told-to-the-operator-at-once-and-a-change-that-needs-it-says-so-before-it-merges.md @@ -0,0 +1,122 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-09 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md +--- + +# 271. A send held for the bus's planned step is told to the operator at once, and a change that needs it says so before it merges + +## Context + +A new build of the bus is a **planned step** +([ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md), +[to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §8). Only a person's +`mesh-controller.bus upgrade` moves it. Until then, every send to the bus's machine is refused, because such a +send would carry the new bus with it. The controller's `sendToEach` refuses the send whole, and a walk keeps +the refusal as its note and tries again on the next tick. + +On 2026-10-08 a catalogue merge built a new bus for the control-node +([issue 336](../04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md)). For 28 minutes every +send to that machine was held: the controller's own walk, the node-engine's, and builds kept back for a gate. +`mesh-controller.conditions` listed nothing about it. The operator found the hold only because a walk looked +stuck, and asked where the approve button was. + +Three things were missing, read from the controller's code +([diagnosis](../04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md)): + +- the refusal reaches only the walk's note, which nobody reads unless they already ask `plans`; +- the only watchdog that could fire, S3, fires no sooner than 30 minutes in. Its words say the delivery is late + and that nothing is lost. They name neither the bus nor the verb, and its resolver is not the operator; +- the merge gate and the delivery plan were green. The plan said "a bus step", but not that merging would + stop every later send to the bus's machine until a person acts. + +## Considered Options + +1. **Lower S3's bound for a walk refused for the bus.** Rejected: it would still say lateness, about one walk, + in words that name neither the cause nor the verb. A wait for a person is not a walk running late. +2. **Raise it from the self-check's probe DB, which already says a running bus step.** Rejected: the probe reads + only a step that has started, and it runs on the self-check's interval. The refusal happens in the walk, + which the watchdogs read every half minute. +3. **Let the walk send everything but the bus, and keep the bus at the build its machine runs.** Not decided + here. The diagnosis thinks it may now be possible, because every send keeps a recorded module at the build + its machine was last sent. That has to be checked against the bus's user list, which issue 249 sends first in + the same declaration. It would not remove the need to tell the operator: the step still waits for a person. +4. **Hold the merge for the operator's word when it builds a new bus.** Rejected for now. It would block a + change on a decision the operator can take after the merge as well. Asking the operator from the merge gate + belongs with the asks of ADR 0259 (an ask is a message on the bus that any channel answers), which is still + a pull request and not built. +5. **A condition of its own, raised at the first refusal, and a line in the delivery plan** — chosen. + +## Decision + +1. **Row S17 of the signals table: a send held for the bus's planned step is told to a person.** On the first + watchdog tick after a walk's send was refused for the bus, the controller raises + `bus..step-waiting`, kind `bus-step-waiting`. It is a warning, and its resolver is the operator. + Its summary names: + - the machines whose bus the new build would replace; + - each walk that waits, with its repository, commit and the modules of its tier; + - the bus build, from and to; + - since when: the first refusal this controller saw, or the save that kept the refusal after a restart; + - the call that ends it, as the operator makes it through the mesh MCP server: + `mesh_call mesh-controller.bus {"upgrade": "true", "why": "…", "reversible" or "irreversible": "true"}`. + + Its plain words say that sends to the machine wait for a bus upgrade, what waits, for how long, and that the + operator starts the upgrade from the mesh MCP server. It clears once the bus's machine has been sent the build the + mesh holds, that is, once the step was taken, whatever the walks' notes still say. From then on no send is + refused for the bus, so nothing waits for a person; whether the new bus came up healthy is the step's own + condition, `bus-maintenance` and then `bus-upgrade-failed` (probe DB), not this one. A walk whose note names + an older bus build than the one held now does not count. +2. **S3 leaves out a walk held only by the bus's step**, as it leaves out a walk under a paused build seat. The + wait is said once, as itself, never as lateness. +3. **The delivery plan says it before the merge.** When a change builds the module that provides the bus and + that module runs somewhere, its plan has the step: "merging this needs a person's bus upgrade: nats is never + sent by an ordinary send, and until the bus upgrade runs, nothing else is sent to either — + unless the new build turns out the same as the one running there (issue 280)". The plan cannot know that + before the build, so it says so rather than staying silent. Its summary says "merging this needs a person's bus + upgrade". The forge module writes that summary into the `mesh/merge-gate` status and the plan into the pull + request's comment. It is said, not enforced: the gate's verdict does not change. + +## Consequences + +- **The operator is told within half a minute of the first refusal**, not when a walk looks stuck. The + condition says what to run. +- **Merging a change that builds the bus is a known promise.** Its pull request says, before the merge, that a + person's step follows and what waits until then. +- **One more row in the signals table.** The generated test walks it like every other row. +- **The condition is about the bus, not about one walk.** Several walks waiting for the same step are one + condition, and the last one to be refused does not reopen it. +- Turning the condition into an ask, whose yes starts the step with the operator's word recorded, waits for + ADR 0259's asks to be built. + +**How it is checked.** In the controller: + +- the signals table's generated test for S17: with a new bus build waiting and no send refused, nothing is + raised; with a refused walk, the condition is raised; when the signal returns, it clears; +- `TestASendHeldForTheBusStepIsFoundFromTheWalksItHolds`: a walk refused for the bus is found, and since when. + A refusal for another reason, a refusal for an older bus build, a bus that already runs the build held, and a + walk waiting for its delivery's word are not; +- `TestASendHeldForTheBusStepIsSaidAtOnceAndNotAsLateness`: the condition's key, kind and operator resolver; + its summary names the machine, the module, the builds from and to, the verb and since when; its words are + plain. S3 says nothing of the walk past its bound, and still raises for a walk held for something else. Through + the keeper, the condition opens at the first tick and clears when the step was taken; +- the replay `TestReplay336`: a walk of the node-engine refused for a new bus build on the control-node raises + the condition on the next watchdog tick, through the store, the plans and the watchdogs' own gathering. It + clears once the machine is sent the new bus. It fails on the controller before the fix and passes on it; +- `TestAChangePlanSaysWhatEachMachineReceives`: a change that builds the bus says "merging this needs a + person's bus upgrade", the machine, and the verb. + +## References + +- [Issue 336](../04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md), its report and + diagnosis. +- [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md), + the bus as a planned step; [ADR 0227](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) + rule 4, no failure is silent; + [ADR 0254](0254-a-wait-for-a-persons-new-login-is-not-a-gates-failure-and-each-module-of-a-send-keeps-its-own-pass.md), + the precedent of a wait for a person said as its own condition. +- [To-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §3, the signals table, and §8, the + bus as a planned step, which carry this row. +- mesh-controller #173, which builds it. diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index 858f6fef..6a593c44 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -2,8 +2,9 @@ layer: to-be status: in-progress code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab] -updated: 2026-10-08 +updated: 2026-10-09 decisions: + - 02-DECISIONS/0271-a-send-held-for-the-buss-planned-step-is-told-to-the-operator-at-once-and-a-change-that-needs-it-says-so-before-it-merges.md - 02-DECISIONS/0265-a-verb-that-reads-is-answered-on-the-serving-controllers-own-connection-and-every-connection-says-what-it-is.md - 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md - 02-DECISIONS/0261-an-assignment-made-while-its-modules-build-runs-is-kept-pending-and-the-controller-makes-it.md @@ -233,6 +234,7 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age | S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | `facts-stale` | warning | — | | S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — | | S16 | a walk waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose walk waits | 30 min; urgent after 4 h | `waiting` (names the walk and `plans go`) | warning | — | +| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a walk out | warning | — | The bus advisories (S9) cost one read-only subscription: the server already publishes them. The controller translates each into a condition naming the thing in the mesh's words, as the refused reply @@ -562,7 +564,10 @@ bus machine's backup holder backing the bus module up now ([ADR 0235](../../02-D or one a person took and names with `--snapshot-taken`; the step's bound is fifteen minutes; **no send reaches the bus's machine while a new bus build waits for it** — a push, a plan's send for another module, a healer's — after a plan's send restarted the bus unasked on 2026-10-06; a change of the bus's -user list is reloaded in place, not a restart. *2026-10-06:* the snapshot +user list is reloaded in place, not a restart. **A send held for the step is told at once** ([ADR 0271](../../02-DECISIONS/0271-a-send-held-for-the-buss-planned-step-is-told-to-the-operator-at-once-and-a-change-that-needs-it-says-so-before-it-merges.md)): +S17 raises `bus..step-waiting` for the operator on the first refusal, and a change that builds the bus +says in its delivery plan, and so in `mesh/merge-gate`'s description, that merging it needs a person's bus +upgrade. *2026-10-06:* the snapshot and the way back from it exist — the bus image's own snapshot program, the same one the night's backup runs, and a restore that builds a new store beside the live one for a person to swap in ([ADR 0235](../../02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md), diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index 01708154..b69000e6 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -2,8 +2,9 @@ layer: to-be status: designed code: [] -updated: 2026-10-08 +updated: 2026-10-09 decisions: + - 02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md - 02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md - 02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md - 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md @@ -72,14 +73,26 @@ allows. | ready | it passed; it waits to be merged | none | — | | rejected | it failed its check | none | — | | published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a newer delivery took over its walk; `go` when its walk started | -| held | it waits for a person | 1 day | nothing: it is the operator's | +| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's | | delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says | -| delivered, failed, superseded, stopped, retired | final | — | — | +| delivered, failed, superseded, stopped, retired, covered | final | — | — | -The transitions are those of ADR 0239 decision 2. A delivery that is merged while not `ready` goes to -`held`. Its check did not pass, and only a person decides that it goes on. A delivery merged with no walk -opened for it within ten minutes is held too, saying so: nothing it moves follows that branch, or the merge -was not heard. Nothing waits silently. +The transitions are those of ADR 0239 decision 2 and [ADR 0270](../../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md). A delivery that is +merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A +delivery merged with no walk opened for it within ten minutes is held too, saying so: nothing it moves follows +that branch, or the merge was not heard. Nothing waits silently. + +**What needs no person ends by itself** (ADR 0270): + +- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and + sends nowhere) is `delivered` as soon as it is merged, whatever its check said; +- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine + running it by the walk of a later merge into the same trunk, delivered on its own passing check, and the + forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge + order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery` + says `covered: a later merge that contains it was delivered` as a success. A module that runs on one + machine only gets no "rest" step and is never covered; +- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes. **History is retired, not held** ([ADR 0250](../../02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md)). A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no walk @@ -108,7 +121,10 @@ controller's `plan-moved` event, never inferred from time. - **Check**: the members' heads composed together, one gate over every machine with all their definitions. It is asked by mesh-delivery through `delivery-check` when the group forms or a member's head moves, and posted on every member's head as `mesh/delivery-group`. -- **Delivering**: in order. A member waits in `published` until the one before is `delivered`. +- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`. + A member published while its group's composed check is asked and not answered waits there for the answer; + one held because that check had not passed is *lifted* back to `published` once it passes or warns for the + heads that merged and its own walk is still open (ADR 0270). - **A failed member**: the members after it are `stopped`, naming it. The members before it stay delivered. Its own builds were put back at its gate. diff --git a/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/01-diagnosis.md b/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/01-diagnosis.md index b0540d6d..1fe3bcc2 100644 --- a/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/01-diagnosis.md +++ b/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/01-diagnosis.md @@ -47,3 +47,21 @@ The group's order put it second. Open question 4 in the report asks whether that **Located** in mesh-delivery: `groupTurns` holds on a composed check that has not answered, and neither `Checked` nor the table brings a held member back when the check later passes. + +## 2026-10-09 — the fix, proposed + +[ADR 0270](../../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md) +(proposed) answers open questions 1 and 2, and mesh-catalog #150 builds it: + +- **Asked and not answered is a reason to wait, not to hold.** `groupTurns` leaves a member in `published` while + its group's composed check is asked for the current heads and has no verdict. +- **A hold whose reason is gone lifts, by observation.** A member held because its group's composed check had not + passed goes back to `published` (the event *lifted*) once the check passes or warns for the heads that merged, + while its own walk is still open. It then takes its turn in order. +- A held member whose own walk was since taken over by a later walk is not lifted, since its walk can no longer + deliver it. It is `covered` when that later walk was delivered and sent its module (issue 319). + +Open question 3, whether `mesh/delivery-group` is required on the trunks, stays the operator's setting. Open +question 4 is not touched. + +**Accepted 2026-10-09.** The operator accepted the record named above; it is no longer proposed. diff --git a/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md b/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md index 89a11b1e..ece2470f 100644 --- a/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md +++ b/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md @@ -107,3 +107,43 @@ remaining seven were left `held` for this issue's fix. The seven show the two ends this issue asks for: four hold nothing to deliver, and three are covered by a later walk. + +## 2026-10-09 — the fix, proposed + +[ADR 0270](../../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md) +(proposed) gives both ends this issue asks for, by observation, and mesh-catalog #150 builds them in +mesh-delivery: + +- **A merge that moves nothing is delivered at once.** `MovesNothing()` no longer counts `Plan.New`: a module + the controller builds and sends nowhere opens no walk, so there is nothing to wait for. A merge whose plan moves + nothing goes from `proposed`, `checked`, `rejected` or `held` to `delivered`, before the row *merged unchecked + → held*. +- **`covered`, a new final state: a later merge that contains it was delivered.** For every module its plan + moves, a delivered delivery of the same repository and trunk, merged after it, delivered on its own passing + check by the walk of its own merge commit, whose machine steps say the module reached every machine running + it. The forge then says, through a new read-only tool `gitea_contains` (its compare of the two commits), + whether the later merge commit contains this one; merge order is only the first filter. A forge that cannot + answer leaves it held. `mesh/delivery` says `covered: a later merge that contains it was delivered: …` as a + success. +- **Read on every tick**, so what is held now ends without an event. + +This answers open question 1: mesh-delivery reads it from its own records and asks the forge for ancestry; the +controller is not asked. Open question 3, why the controller's own merges opened no walk, is not answered by this fix; it no +longer leaves anything held. + +**How to verify after mesh-delivery is delivered with it.** Within a minute of its start: + +1. `mesh-delivery.deliveries` with `state: held` lists none of the four hq merges, none of the module-adding + merges (`builds new: …, sent nowhere`), and none of the five controller merges held for "no walk opened". + The hq and module-adding ones are `delivered`; the controller ones are `covered`. +2. `mesh-delivery.show` of one of them, for example `novox/mesh-controller@b98fd0f396c8`, ends with a + transition `held → covered (covered)` whose why names the later delivery and its walk. +3. `mesh-delivery.checks` for its pull request shows `mesh/delivery` as `success covered: …`. +4. The forge module is delivered in the same pull request. Until its `gitea_contains` answers, nothing is + covered, and the held controller merges stay held. +5. At the self-check's next reading, `mesh-controller.conditions` no longer lists their + `delivery..stalled`. +6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered walk sent + their module. One that stays held is a real question: its change may not run anywhere. + +**Accepted 2026-10-09.** The operator accepted the record named above; it is no longer proposed. diff --git a/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md index 70d78006..37c27199 100644 --- a/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md +++ b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md @@ -58,3 +58,26 @@ running build and carry the rest, so a pending bus step would stop holding every control-node. That needs checking against the bus's user list, which [issue 249](../249-a-modules-new-state-is-refused-until-a-push-the-merge-did-not-make/00-report.md) sends first in the same declaration, before it is proposed as a fix. It does not replace the condition: the step still waits for a person, and the person must be told. + +## 2026-10-09 — the fix, proposed + +[ADR 0271](../../02-DECISIONS/0271-a-send-held-for-the-buss-planned-step-is-told-to-the-operator-at-once-and-a-change-that-needs-it-says-so-before-it-merges.md) +(proposed) takes suggested fixes 1 and 3. mesh-controller #173 builds them: + +- **Row S17**, `bus..step-waiting`: raised on the first watchdog tick after a walk's send was refused + for the bus's planned step, while the new bus build would still replace the bus on its machine. It is a + warning, for the operator. It names the machines, each waiting walk and the modules of its tier, the bus build + from and to, since when, and the `mesh-controller.bus` call with `upgrade`. It clears once the bus's machine has + been sent the build the mesh holds. +- **S3 leaves out a walk held only by it.** +- **The delivery plan says it before the merge**: "merging this needs a person's bus upgrade: … nothing else is + sent to either", and its summary, which `mesh/merge-gate` carries, says the same. + +The replay `TestReplay336` in the controller fails on the commit before the fix ("the first tick after the refusal +told the operator nothing about the bus's step: 0 condition(s)") and passes on it. It still has to be registered +in mesh-lab's replays register as R336 before this issue resolves with `replay: R336`. + +Suggested fix 2 (an ask) waits for ADR 0259's asks. The guess about letting a send keep the bus at its running +build and carry the rest is not checked here. + +**Accepted 2026-10-09.** The operator accepted the record named above; it is no longer proposed.