novox/hq
6
0
Fork 0
Code Issues Pull Requests 6 Releases Wiki Activity

ADR 0270 and 0271 (proposed): a held delivery whose change runs ends by itself; a send held for the bus's step is told at once

Browse Source 
Issues 319 and 316: fifteen held deliveries asked a person a question with no right answer, and stop,
the only way out, records a delivered change as an error. ADR 0270 delivers a merge that moves nothing at
once, adds the final state covered for a merge a later delivered walk of the same trunk carried, and lifts
a group hold once its composed check passed.

Issue 336: a send held for the bus's planned step raised nothing for 28 minutes. ADR 0271 adds row S17,
said at the first refusal for the operator with the verb, keeps S3 from calling it lateness, and has the
delivery plan say before the merge that a person's bus upgrade follows.

Each issue's diagnosis says what the fix does and how to verify it once delivered.
This commit is contained in:
jochen
2026-10-09 02:34:18 +02:00
parent 9bd0dcc611
commit 6163a74bb2
5 changed files with 345 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02-DECISIONS/0270-a-held-delivery-whose-change-already-runs-or-has-nothing-to-run-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md
+157
View File
@@ -0,0 +1,157 @@
---
topic: the mesh
status: proposed
date: 2026-10-09
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md
---
# 270. A held delivery whose change already runs, or has nothing to run, ends by itself, and a hold whose reason is gone lifts
## Context
[ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md)
decision 2 gives a delivery one table of states. A delivery is `held` when only a person can decide whether
it goes on. A person then has four ways out: `release` walks the delivery's own commit, `stop` ends it as
stopped, `retire-history` ends history
([ADR 0250](0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md)),
and the walk can start on another word (`go`).
On 2026-10-09, fifteen deliveries were held, most for more than a day, each raising probe D14's
`delivery.<id>.stalled` (read with `mesh-delivery.deliveries` and `show`). None of them asked a question a
person could answer correctly:
- **Four merges of hq build nothing.** Their plan says "builds nothing: the change touches no module of the
mesh's graph". They were merged before their check answered, and the row *merged unchecked → held* is tried
before anything reads the plan.
- **Four catalogue merges only add a module.** The controller builds and registers a new module and sends it
nowhere, so it opens no walk ([issue 300](../04-ISSUES/300-a-new-module-is-not-built-at-its-merge/00-report.md)).
mesh-delivery counted the new module as something a walk moves, so the no-walk rule held each one ten
minutes after its merge.
- **Five controller merges had no walk of their own.** A later merge into the same trunk was delivered, and
its walk sent every module these move. The diagnosis of [issue 319](../04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md)
checked each merge commit against the later delivered commit: each was contained in it, and the
controller's `status` listed none of their modules as behind.
- **Two catalogue merges were held for their group's composed check**
([issue 316](../04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)).
One was published while the check was asked and not yet answered. The code read "not answered" as "did
not pass" and held it. Two minutes later the check passed for exactly the heads that merged, and nothing
read that answer for the held member. The own walk of each of the two was since taken over by a later walk
of the same trunk that sent its module again.
`release` would walk an old commit over a newer one, or walk nothing. `retire-history` refuses each one,
because their heads were heard. `stop` is the only way out that works, and it is the wrong record: it sets
`mesh/delivery` to `error` on a pull request whose change runs on every machine. Four were stopped that way on
2026-10-09 before the issue was found, and their pull requests now say "stopped: covered: …" as an error. A
condition that only a wrong act can clear teaches people to ignore the list it is in.
## Considered Options
1. **Stop them with why.** Rejected: it is what was done, and it records a delivered change as an error on
its pull request. Issue 313 ruled it out for the same reason.
2. **A person's act, `supersede`, that ends a held delivery as superseded.** Rejected as the main path. Every
fact the person would check is already in mesh-delivery's own records: the plan, the forge's merge times
and the walks' records. A person asked to confirm a fact the mesh can read is a person asked for nothing.
A person keeps `stop` and `release` for what the facts do not settle.
3. **Ask the forge, or the controller, whether each running build's commit contains the merge** (git
ancestry through a new forge tool, or through the controller's records of what each machine runs).
Rejected for now. The forge module has no such tool, and adding one means changing a TypeScript module.
The trunks are protected: they take merges only through pull requests, and no one can push to them
(`gitea.gitea_branch_protection_get` on each trunk says `push: false`). So a commit the forge merged later
into the same trunk contains every commit merged into it before. The merge times say that order, and
mesh-delivery already holds them. Ancestry would answer the same question with more parts.
4. **Read the facts, and end the delivery by observation** — chosen, below.
For issue 316 the alternative was to require `mesh/delivery-group` on the trunks, so that no member can merge
before its composed check passed. That stays the operator's setting, as ADR 0239 left it. It does not settle
a check that answers after the merge.
## Decision
1. **A merge that moves nothing is delivered at once.** A delivery whose plan moves no module the mesh holds
is delivered as soon as it is merged and has no walk. That covers a plan that builds nothing and a plan
that only adds a module the controller builds and sends nowhere. This holds whatever its check said: from
`proposed`, `checked` and `rejected`, the row is tried before *merged unchecked → held*, and from `held`
it is tried on every tick. `ready → published → delivered` already did this for a merge that passed.
2. **A new final state, `covered`.** A merged delivery is covered when every module its plan moves was sent
by a finished walk of another delivery that:
- is of the same repository and was merged into the same trunk;
- was merged after it, by the forge's merge times, so its merge commit contains this one;
- is `delivered`.
The modules a walk sent are read from the walk's own record, or from the machine steps kept for that
delivery. A delivery whose own walk is still open is never covered: its walk delivers it. The row is
observed. It is tried from `held`, and from `ready` once the ten minutes for a walk to open have passed,
before the no-walk hold. The transition's why names each covering delivery, the commit it merged as and its
walk. On the pull request, `mesh/delivery` says `covered: …` as a **success**, never an error. A group
counts a covered member as delivered, so the member after it gets its turn and is not stopped.
3. **A group member is not held for a composed check that has not answered.** "Did not pass for the heads that
merged" means the check failed, ran for other heads, or was never asked. A member published while its
group's check is asked and not answered waits in `published`, within that state's bound.
4. **A hold whose reason is gone lifts.** A member held because its group's composed check had not passed is
taken back to `published` when that check passes or warns for the heads that merged, and its own walk is
still open. The event is *lifted*, observed, and the member then takes its turn in the group's order. Every
other hold stays a person's.
5. **Read again on every tick.** The holder reads the facts of every held delivery on each tick, not only when
an event arrives. A delivery that was held before these rows existed, or read back after a restart, ends by
itself. Once it is final, nothing moves it again.
The table of ADR 0239 decision 2 gains these rows:
| From | Event | To | Guard |
|---|---|---|---|
| proposed, checked, rejected, held | done | delivered | merged, no walk, and its plan moves no module the mesh holds |
| ready, held | covered | covered | merged, no walk of its own open, and every module it moves sent by a finished walk of a later-merged, delivered delivery to the same trunk |
| held | lifted | published | held only for its group's composed check, which has since passed or warned for the heads that merged, and its own walk still open |
`delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered` are final.
## Consequences
- **The held deliveries end without a person.** When mesh-delivery with these rows is delivered, its first
ticks deliver the hq merges and the module-adding merges, and cover the controller and catalogue merges
that a later walk carried. Their D14 conditions clear at the self-check's next reading of `stalled`.
- **A pull request no longer says an error for a change that runs.** The four stopped on 2026-10-09 keep their
error: a final state is not undone. Their commit notes say why.
- **The trunk's protection is now a premise of a delivery's end.** If a trunk ever accepted a push, a later
merge might not contain an earlier one, and `covered` could be wrong. The protection is the forge's setting
on each trunk. mesh-delivery does not check it itself, and this is a known gap.
- **`covered` is only as complete as the walks' records.** A module the later walk held back for a person is
not sent, so it does not count. A delivery whose modules were only partly carried stays held for a person.
- **A group member waits longer, rather than being held too soon.** A composed check that never answers leaves
the member in `published`. Its bound of 30 minutes lists it as stalled. That is the right owner for a check
that does not answer.
**How it is checked.** mesh-delivery's tests, in `covered_test.go`:
- `TestAMergeThatMovesNothingIsDeliveredAtOnce`: builds nothing, or only a new module sent nowhere; merged
before its check answered, on a failed check, or on a passing one. Each is delivered, and its status is a
success.
- `TestAMergeALaterDeliveredWalkCarriedIsCovered`: covered whether it was held first or not. It stays held
when the later walk carried only one of its modules, merged before it, went into another branch, or failed.
Its status is `success covered` and names the commit that covered it, never an error.
- `TestACoveredGroupMemberLetsTheNextOneGo`: the member after a covered one is let go, and the group ends
delivered.
- `TestHeldDeliveriesReadBackCloseByThemselvesOnce`: deliveries held before these rows existed are read back
from the store and end delivered or covered, or stay held when not covered. A second restart moves nothing.
- `TestAGroupMemberWaitsForItsComposedCheckToAnswer` and `TestAMemberHeldForItsComposedCheckIsLiftedWhenItPasses`:
a pass or a warning lifts the member; a failure, or a pass for other heads, keeps it held.
The table's own tests take and refuse every new row. They also check that the only way to `published` other
than a ready delivery's merge is *lifted* from `held`, and that nothing observed ends a held delivery as
stopped or failed.
## References
- [Issue 319](../04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md) and
[issue 316](../04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md),
their reports and diagnoses.
- [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md),
decisions 2, 4 and 9; [ADR 0250](0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md),
the last state added to the table.
- [Issue 300](../04-ISSUES/300-a-new-module-is-not-built-at-its-merge/00-report.md), why a new
module opens no walk.
- [Design 47](../03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md), *The state table* and *A
delivery group*, which take these rows once this record is accepted.
- mesh-catalog, the branch `fix/319-316-covered-and-lifted`.
02-DECISIONS/0271-a-send-held-for-the-buss-planned-step-is-told-to-the-operator-at-once-and-a-change-that-needs-it-says-so-before-it-merges.md
+117
View File
@@ -0,0 +1,117 @@
---
topic: the mesh
status: proposed
date: 2026-10-09
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md
---
# 271. A send held for the bus's planned step is told to the operator at once, and a change that needs it says so before it merges
## Context
A new build of the bus is a **planned step**
([ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md),
[to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §8). Only a person's
`mesh-controller.bus upgrade` moves it. Until then, every send to the bus's machine is refused, because such a
send would carry the new bus with it. The controller's `sendToEach` refuses the send whole, and a walk keeps
the refusal as its note and tries again on the next tick.
On 2026-10-08 a catalogue merge built a new bus for the control-node
([issue 336](../04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md)). For 28 minutes every
send to that machine was held: the controller's own walk, the node-engine's, and builds kept back for a gate.
`mesh-controller.conditions` listed nothing about it. The operator found the hold only because a walk looked
stuck, and asked where the approve button was.
Three things were missing, read from the controller's code
([diagnosis](../04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md)):
- the refusal reaches only the walk's note, which nobody reads unless they already ask `plans`;
- the only watchdog that could fire, S3, fires no sooner than 30 minutes in. Its words say the delivery is late
and that nothing is lost. They name neither the bus nor the verb, and its resolver is not the operator;
- the merge gate and the delivery plan were green. The plan said "a bus step", but not that merging would
stop every later send to the bus's machine until a person acts.
## Considered Options
1. **Lower S3's bound for a walk refused for the bus.** Rejected: it would still say lateness, about one walk,
in words that name neither the cause nor the verb. A wait for a person is not a walk running late.
2. **Raise it from the self-check's probe DB, which already says a running bus step.** Rejected: the probe reads
only a step that has started, and it runs on the self-check's interval. The refusal happens in the walk,
which the watchdogs read every half minute.
3. **Let the walk send everything but the bus, and keep the bus at the build its machine runs.** Not decided
here. The diagnosis thinks it may now be possible, because every send keeps a recorded module at the build
its machine was last sent. That has to be checked against the bus's user list, which issue 249 sends first in
the same declaration. It would not remove the need to tell the operator: the step still waits for a person.
4. **Hold the merge for the operator's word when it builds a new bus.** Rejected for now. It would block a
change on a decision the operator can take after the merge as well. Asking the operator from the merge gate
belongs with the asks of ADR 0259 (an ask is a message on the bus that any channel answers), which is still
a pull request and not built.
5. **A condition of its own, raised at the first refusal, and a line in the delivery plan** — chosen.
## Decision
1. **Row S17 of the signals table: a send held for the bus's planned step is told to a person.** On the first
watchdog tick after a walk's send was refused for the bus, the controller raises
`bus.<bus module>.step-waiting`, kind `bus-step-waiting`. It is a warning, and its resolver is the operator.
Its summary names:
- the machines whose bus the new build would replace;
- each walk that waits, with its repository, commit and the modules of its tier;
- the bus build, from and to;
- since when: the first refusal this controller saw, or the save that kept the refusal after a restart;
- the verb `mesh-controller.bus upgrade --why … --reversible|--irreversible`.
Its plain words say that sends to the machine wait for a bus upgrade, what waits, for how long, and that the
operator starts the upgrade from the mesh MCP server. It clears once the bus's machine runs the build the
mesh holds, that is, once the step was taken, whatever the walks' notes still say. A walk whose note names
an older bus build than the one held now does not count.
2. **S3 leaves out a walk held only by the bus's step**, as it leaves out a walk under a paused build seat. The
wait is said once, as itself, never as lateness.
3. **The delivery plan says it before the merge.** When a change builds the module that provides the bus and
that module runs somewhere, its plan has the step: "merging this needs a person's bus upgrade: nats is never
sent by an ordinary send, and until `mesh-controller.bus upgrade --why … --reversible|--irreversible` runs,
nothing else is sent to <its machines> either". Its summary says "merging this needs a person's bus
upgrade". The forge module writes that summary into the `mesh/merge-gate` status and the plan into the pull
request's comment. It is said, not enforced: the gate's verdict does not change.
## Consequences
- **The operator is told within half a minute of the first refusal**, not when a walk looks stuck. The
condition says what to run.
- **Merging a change that builds the bus is a known promise.** Its pull request says, before the merge, that a
person's step follows and what waits until then.
- **One more row in the signals table.** The generated test walks it like every other row.
- **The condition is about the bus, not about one walk.** Several walks waiting for the same step are one
condition, and the last one to be refused does not reopen it.
- Turning the condition into an ask, whose yes starts the step with the operator's word recorded, waits for
ADR 0259's asks to be built.
**How it is checked.** In the controller:
- the signals table's generated test for S17: with a new bus build waiting and no send refused, nothing is
raised; with a refused walk, the condition is raised; when the signal returns, it clears;
- `TestASendHeldForTheBusStepIsFoundFromTheWalksItHolds`: a walk refused for the bus is found, and since when.
A refusal for another reason, a refusal for an older bus build, a bus that already runs the build held, and a
walk waiting for its delivery's word are not;
- `TestASendHeldForTheBusStepIsSaidAtOnceAndNotAsLateness`: the condition's key, kind and operator resolver;
its summary names the machine, the module, the builds from and to, the verb and since when; its words are
plain. S3 says nothing of the walk past its bound, and still raises for a walk held for something else. Through
the keeper, the condition opens at the first tick and clears when the step was taken;
- the replay `TestReplay336`: a walk of the node-engine refused for a new bus build on the control-node raises
the condition on the next watchdog tick, through the store, the plans and the watchdogs' own gathering. It
clears once the machine is sent the new bus. It fails on the controller before the fix and passes on it;
- `TestAChangePlanSaysWhatEachMachineReceives`: a change that builds the bus says "merging this needs a
person's bus upgrade", the machine, and the verb.
## References
- [Issue 336](../04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md), its report and
diagnosis.
- [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md),
the bus as a planned step; [ADR 0227](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md)
rule 4, no failure is silent;
[ADR 0254](0254-a-wait-for-a-persons-new-login-is-not-a-gates-failure-and-each-module-of-a-send-keeps-its-own-pass.md),
the precedent of a wait for a person said as its own condition.
- [To-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §3, the signals table, and §8, the
bus as a planned step, which take this row once this record is accepted.