Merge pull request 'Design 47 follows ADR 0633, 0637, 0638 and 0639: the take-in by the indexer it names, and the group try (issues 1966, 1921)' (#521) from docs/47-try-and-take-in into main

2026-11-05 20:06:31 +00:00
parent 674f88f9ab b926db338d
commit 1ca5a3a133
2 changed files with 239 additions and 0 deletions
@@ -543,6 +543,11 @@ controller (the planner, the rollout) until release 6 of ADR 0429; from each rel
  move changed, keeping the composition while no composed fingerprint changed; the verdict names the commits it
  was carried from (ADR 0548 §5, §6). A head git refuses onto the new commit is not carried: its next check is
  whole (ADR 0548 §7).
 - **try** — the checker's run of a delivery group's check on one or more members, each a repository at an exact
  commit, before or without a pull request: the same examiner as the group's check, stricter in that a missing
  candidate or a node's missing composition answer ends it as error, and owing the forge no status and no Approve.
  It keeps the verdicts of the packages it passed for the check to take (ADR 0637). Asked only when the operator
  asks for one (ADR 0637 §6).
 - **trunk commit** — the commit of the trunk a merge check's verdict was judged against: the trunk's head when
  the build module cloned it. The verdict holds for that commit only, goes out of date when the trunk moves, and
  is then carried to the new commit (ADR 0323, ADR 0548).
@@ -703,6 +708,10 @@ controller (the planner, the rollout) until release 6 of ADR 0429; from each rel
  merged head and the group that moved it. A group reaches the mesh as one source revision (ADR 0429 §11); a
  source revision is **delivered** when every want equals its module's fingerprint and every have equals its
  want and is healthy, derived from the keys and recorded by no part (§1).
 - **pending revision** — a source revision the running indexer has not taken in, because it names another build of
  the indexer or follows one that does: only the named indexer's own keys are written, the rest kept in the
  indexer's state `pending` until that indexer starts and takes it in (ADR 0633 §1 to §3). A group's members wait
  there too until the group's end, as members still gathered (§4).
 - **canary** — the first node of a module's nodes, ordered by ring then name: a new fingerprint reaches it
  first and the module's other nodes only once it runs healthy there (ADR 0429 §6). Replaces sending to every
  machine at once from release 6 (§15); a fingerprint that fails there is held and the canary set back, or for
@@ -4,10 +4,23 @@ status: in-progress
 code: [mesh-controller, mesh-catalog modules/mesh-delivery-gate, mesh-delivery, mesh-host, mesh-sdk, mesh-vault, hq 00-META/checks]
 updated: 2026-11-05
 decisions:
  - "ADR 0639"
  - "ADR 0638"
  - "ADR 0637"
  - "ADR 0636"
  - "ADR 0635"
  - "ADR 0634"
  - "ADR 0633"
  - "ADR 0632"
  - "ADR 0615"
  - "ADR 0614"
  - "ADR 0603"
  - "ADR 0601"
  - "ADR 0596"
  - "ADR 0595"
  - "ADR 0594"
  - "ADR 0593"
  - "ADR 0590"
  - "ADR 0572"
  - "ADR 0571"
  - "ADR 0570"
@@ -60,6 +73,7 @@ decisions:
  - "ADR 0446"
  - "ADR 0445"
  - "ADR 0444"
  - "ADR 0441"
  - "ADR 0437"
  - "ADR 0435"
  - "ADR 0434"
@@ -602,6 +616,104 @@ From its delivery, a check is this:
  The assigner writes the verdict under the checker's hash of the heads it judged, in its own bucket, and the checker reads it into the
  composition of its verdict, which says "not run" until the assigner's module exists (ADR 0528 §5).
 ### A try: a delivery group's check at exact commits
 *ADR 0637, ADR 0638; ADR 0593 §7, §9, ADR 0636 §4, ADR 0548 §5, ADR 0596 §1 to §3, ADR 0615, ADR 0639 §1 to
 §6, §10.*
 A try says beforehand whether a check will pass. It is the check's own code run on commits nobody pushed to a
 pull request, and it owes the forge nothing.
 - **A try is a delivery group at exact commits.** The checker's try takes one or more members, each a repository
  and the exact commit asked, never a branch's head, with an optional trunk commit. One repository at one commit
  is a try of one member, and the same members in any order are one try (ADR 0637 §1). Its steps are check asks
  on the build seats' streams, as above.
 - **A whole try runs the group's check.** It builds the case a group's pull-request check builds and runs the same
  examiner: each member merged onto its trunk, the unit and integration levels on the build modules, the manifest
  validator with the seat definitions of the members' commits (ADR 0636 §4), the module graph, and the
  composition on every placed node (ADR 0593 §7, §9). It names every failure. A try narrowed to some tests or one
  level runs only those, validates no manifest and no composition, and says so (ADR 0637 §2). A try of some tests
  keeps no verdict; a try of one level keeps only that level's (ADR 0639 §4, which changes ADR 0637 §5).
 - **A try is stricter than the check, on purpose.** When no candidate exists for its members, or a node gave no
  composition answer, the try ends as error, never as pass, naming what it could not validate, the node by name.
  A check passes such a node with a warning (ADR 0593 §9); a try cannot, because it would then not say whether the
  check will pass. A try that ended as error keeps no verdict (ADR 0637 §3).
 - **A whole try that waits for its candidate says why.** Its try read gives the reason it waits, and the wait is
  given within the checker's step bound (ADR 0639 §5).
 - **The indexer makes a try's candidate, on the checker's events.** For a whole try only, the checker emits
  `try.started`, carrying the try's id and its members at their commits, and `try.ended`, carrying its id and its
  state, on its stream `checks`; a narrowed try emits neither, because it makes no candidate. The events' contract
  is the package `tryevent` of mesh-delivery's sdk, read by the checker and the indexer, so no other module
  rebuilds for it (ADR 0638 §1). The indexer, the holder of the seat `mesh-catalog`, consumes them: on
  `try.started` it writes `candidates.<try id>` with one pull per member, its modules and fingerprints from the
  previews, validated with the seat definitions the members carry, and on `try.ended` it deletes it. It no longer
  reads the checker's state `tries` (ADR 0638 §2, which changes ADR 0637 §4; ADR 0596 §1, §2).
 - **Two reads across the seam stay until issue 1966 moves them.** The indexer reads the checker's previews, and the
  checker watches the indexer's `candidates.<try id>`. Both are readers of another module's bus state that ADR 0596
  §3, as ADR 0615 changes it, lets stay until their module moves; issue 1966 tracks the move to the owner's tools
  or events. The forge bridge's read of the gate's `merges` state (*Taken in by the indexer it names*, below) is
  the same kind.
 - **A try event is never lost.** The checker publishes each try event with retry until the stream `checks`
  acknowledges it, and `try.ended` carries the state on every end; whether it was sent is kept in the try's own
  state, so an end after a checker restart still sends it. Only the checker's account may publish `try.*` on its
  stream, a row of the checker's manifest test (ADR 0639 §2, which extends ADR 0638 §1).
 - **At its start the indexer recovers the open tries before it handles the events.** It creates its consumer on
  the stream `checks` first, then calls the checker's tool `checker_tries`, declared in its `invokes`, which
  answers every open try, with no bound on the newest, with its members and whether it is whole (ADR 0639 §3). It writes a candidate
  for each whole open try that has none, deletes every candidate whose try is not open, and only then handles the
  events it holds. A try that starts or ends between the call and the consumer is heard as an event, and each
  write is by try id, so a try seen both ways is written once and deleted once (ADR 0638 §3). The candidate is the
  try's alone: the check, the gate and the forge bridge leave it alone, and the forge is owed no status and no
  Approve (ADR 0637 §4).
 - **When `checker_tries` does not answer, nothing is deleted.** The indexer deletes no candidate, its take-in at
  start goes on, and it asks again on the checker's next try event and by its tool `indexer_recover_tries`; the
  failure is said in `indexer_try_candidates` (its state `try-recovery`) until a recovery succeeds. In the release that brings ADR 0638, the
  new indexer starts before the new checker, so the old checker does not serve `checker_tries` yet: an ordering
  edge of the module graph (a served verb, ADR 0614), and the reason the failure path is needed at once (ADR 0639
  §1, which changes ADR 0638 §3).
 - **How long it is kept.** The checker keeps an ended try's record for 7 days, readable by `checker_try_read`, and
  then removes it, so its state does not grow with every try (ADR 0639 §6). How long the stream `checks` keeps its events is not decided here (*What is not decided here*).
 - **A try and a check share kept verdicts.** The try computes each package's digest as the check does and keeps
  the verdict of each package it passed under that digest (ADR 0548 §5, ADR 0536 §1). The pull-request check of a
  tried head runs only the packages with no kept verdict for their digest, those a moved trunk or a new head
  changed, so a group tried before its push costs about one check's worth of build capacity (ADR 0637 §5, ADR 0482
  §1, ADR 0548 §1).
 - **No try in the normal flow.** A session asks no try; it asks a group try only when the operator asks for one,
  for instance to see a group's composition before its push. The review reports a try asked without the
  operator's ask (ADR 0637 §6).
 - **Asked through the seat verb from the registry milestone.** The seat verb `try` takes `members` once its seat
  definition in mesh-delivery-contracts gives them, at the registry milestone (issue 1966); until then a group try
  is asked through the checker's own tool `checker_try`. The names a try shares between the checker and the
  indexer, its group name and the two reasons a candidate waits, are defined once in mesh-delivery-sdk from that
  milestone, and no copy stays in either module (ADR 0637 §7, §8).
 - **The try's tools.** Beside the seat verb `try`, the checker's and the indexer's own tools, against the four
  lines a module's tools answer:
  | Line | The checker | The indexer |
  |---|---|---|
  | read everything it holds | `checker_try_read` (one try: its members, steps, composition per node, failures, why it waits), `checker_tries` (every open try), `checker_kept` (the kept verdicts by digest) | `indexer_try_candidates` (each try's candidate, and a failed recovery) |
  | do by hand what it does itself | `checker_try` (ask a try) | `indexer_recover_tries` (recover the open tries now) |
  | recover without help | `checker_try_end` (end a try, which emits `try.ended`) | `indexer_drop_try_candidate` (drop a try's candidate), `indexer_recover_tries` |
  | one seat, many tools | the seat verb `try` from the registry milestone | the seat `mesh-catalog`'s verbs, unchanged |
 - **How it is checked.** The checker's try test asks two members at exact commits and the same group in any
  order, refuses five malformed asks, ends as error without a candidate, keeps no verdict from an error, and fails
  naming the node on a refused composition; the examiner's table validates a try's manifests as the check does;
  the whole-module test starts a placed engine and a candidate and shows that the check of a tried head runs no
  package with a kept verdict (ADR 0637 §9). The indexer's take-in table writes a try's candidate on its start,
  deletes it on its end, and recovers at its start (an open whole try gets its candidate, a candidate of a try not
  open goes, a narrowed try gets none, a try started between the call and the consumer gets one), and the
  checker's try test shows both events for a whole try, neither for a narrowed one, and `checker_tries` answering
  the open tries; both manifests are accepted by the manifest validator (ADR 0638 §4). The indexer's
  take-in table also has a recovery that gets no answer (no candidate deleted, the failure in `try-recovery`, the
  retry on the next try event and by `indexer_recover_tries`); the checker's try test shows the retry until
  acknowledgement, `try.ended` after a restart, the whole list of open tries (more than 64), the verdicts kept by a
  try of some tests and of one level, the reason a waiting try gives, and an ended try's record kept 7 days and
  removed after; the checker's manifest test shows that only its account publishes `try.*` (ADR 0639 §10). The registry group's review finds no copy of the shared
  names. Live, the registry group is tried whole before its push, and the checker's try read names each member,
  the composition per node and every failure (ADR 0637 §9); during that try `indexer_try_candidates` reads its
  candidate, and after `try.ended` it is gone (ADR 0638).
 ## The Approve and the merge
 *ADR 0421, ADR 0398 §3, ADR 0406 §1.*
@@ -646,6 +758,8 @@ From its delivery, a check is this:
  revision of `source.main` in one write (ADR 0429 §11). There is no merge window, no batch and no plan: the
  indexer writes each module's fingerprint at the merge, the builder claims each new fingerprint once, and the
  assigner writes each want when its six rules hold. A merge is never refused, held or dropped once it is made.
  What of it is taken in waits only for the indexer its source revision names, or for the rest of its group
  (*Taken in by the indexer it names*, below; ADR 0633).
 - **Two merges close together are two revisions.** The second's fingerprints replace the first's in the
  indexer's modules; a node gets the newer fingerprint's want once its rules hold, and a fingerprint nobody wants
  any more is never applied. Nothing waits for a window to close.
@@ -665,6 +779,99 @@ From its delivery, a check is this:
  the move; its tools `assigner_wants`, `assigner_want`, `assigner_differences` and `node_haves` read the same
  keys, and its events `delivered`, `moving`, `settled` and `stuck` say them on the bus.
 ### Taken in by the indexer it names
 *ADR 0633 with its progressive insights of 2026-11-05, which changes ADR 0603 §1, §4, §6 and §9; ADR 0639 §7, §8,
 which change ADR 0633 §1, and ADR 0639 §9, §10; ADR 0636 §4, ADR 0614 §8, ADR 0601 §3, ADR 0632 §2, ADR 0441 §3, §4, ADR 0564 §2, ADR 0595
 §11, ADR 0596 §3, ADR 0615.*
 A change that brings a new manifest validator is judged by that validator: nothing else of it reaches the
 machines until the indexer it builds runs, and the merges behind it then move on the bus's own event, never on a
 timer. A group's merges arrive together.
 - **A source revision names its indexer.** The indexer is the module that claims the seat `mesh-catalog`; a source
  revision names the build of it whose fingerprint its module has once the revision is applied. The running
  indexer takes a revision in whole only when the revision names its own build (ADR 0633 §1).
 - **The indexer fixes its own build at its start.** It sets its own build once, at its start, from the build it
  was started with, and never learns it later from the node-engine seat's verb `composition`. A process the
  node-engine is replacing therefore keeps its old build to the end and cannot take the new revisions in with its
  old manifest validator; they wait for the new process. An indexer that learnt its build lazily and kept it
  could, while being replaced, learn the new fingerprint and take in with the old validator (ADR 0639 §7, which
  changes ADR 0633 §1).
 - **An indexer with no own build says so on the bus.** It leaves the merged message unacknowledged, so the bus
  delivers it again, says so in its journal, and sends `take-in.pending` with the reason `own build unknown` for
  the revision it holds, so the watchman's bound for `take-in-waits` covers that wait (ADR 0633 §1, ADR 0639 §8,
  which changes it).
 - **A revision that names another indexer moves only that indexer.** The running indexer writes only what
  building and moving the named indexer needs: the indexer's own module entry, its manifest record, its verdict
  (ADR 0614 §8), its key of the module graph and the packages the revision publishes. It keeps the rest in its own
  state `pending`, one key per merge carrying the merged message whole, written by compare-and-set, and acknowledges
  the merged message only once that key is written. `pending` is in the old form of bus state, the bucket the
  grants part creates, not a template in `requires`; it moves when the indexer next moves its bus state (ADR 0595
  §11 as ADR 0615 changes it). Nothing else of the revision reaches the module, verdict,
  graph, taken, predefined, candidate or config-candidate state, so no node-engine, assigner, grants part or builder
  reads it (ADR 0633 §1, which changes ADR 0603 §4).
 - **Validated with the seat definitions its merges carry.** The running indexer validates the revision's
  manifests, the named indexer's own among them, with the seat definitions read from the merges' trees, never a
  built-in seat list. When it refuses the named indexer's manifest, the revision stays pending, and the event
  `take-in.pending` and the tool `indexer_pending` say so in their field `refused` (ADR 0633 §1, its insight of
  2026-11-05, ADR 0636 §4).
 - **The named indexer takes it in at its start.** An indexer that starts as the build a pending revision names
  takes the pending revisions in, oldest first, as one take-in judged by its own manifest validator, before it
  starts its consumer on the forge bridge's stream; then it removes them from `pending`. It takes in only whole
  revisions, and its validation and holds act on them as on any merge (ADR 0633 §2, ADR 0601 §3, ADR 0632 §2).
 - **A merge behind a pending revision waits too, and the newest governs.** While a revision is pending, every later
  merge joins the pending revisions in merge order, unrelated modules included, because its trees build on theirs;
  which indexer they name is worked out over all of them. A later merge that brings the indexer's fingerprint back
  to the running build, as a revert does, lets the running indexer take them all in; a pending revision that names
  yet another build is taken in by the next indexer only as far as the first bullet allows (ADR 0633 §3, which
  changes ADR 0603 §1 and §9).
 - **A group's merges are taken in together.** The forge bridge sets the group on each member's merged message,
  from the gate's `merges` state, a read of another module's bus state that ADR 0596 §3 lets stay until issue 1966
  moves it, and once every approved member is merged it publishes `forge.group.merged` on
  its own stream `forge`, after the last member's merged message, naming the members and their merge commits. The
  indexer hears both through its one consumer, keeps each member's merged message in `pending` until the group's
  end, and then takes the members in as one source revision under the first bullet. A refused member is no
  group's end: the group stays gathered until it merges on, said by the wait below and the urgent condition of a
  held group (ADR 0441 §3, §4); the gate passes the group's end by without counting it a refusal. A later merge in
  a member's repository waits behind the group; a merge in another repository is taken in alone, unless a revision
  is pending for its indexer (ADR 0633 §4, which builds ADR 0429 §11 for take-in).
 - **A wait is bounded, said and recoverable.** The indexer's tool `indexer_pending` reads the pending revisions and
  the members still gathered, each with its merges, the indexer build it waits for, since when and `refused`; the
  view names a module waiting for its revision's indexer through that tool (ADR 0633 §5, which changes ADR 0603
  §6). The indexer emits `take-in.pending` when a revision becomes pending or a group's first member is held, and
  `take-in.taken` once it is taken in, both ordinary events of the indexer; each carries the wait's key, what
  waits (a revision with the indexer build it waits for, or a group with its members so far), since when, and
  `refused` (ADR 0633 §1, §5). The watchman writes the finding `take-in-waits` when no `take-in.taken` follows
  within 60 minutes for a revision or 30 minutes for a group, and the condition keeper raises the condition from
  it and clears it on `take-in.taken` (ADR 0564 §2 as ADR 0590 §2 changes it). At its start the watchman seeds its
  open waits from `indexer_pending`; the bounds only raise the condition, and nothing is taken in on a timer. The hand acts, each said in the journal and the audit, are the indexer's `indexer_take_in_pending`
  (take the pending revisions, or a group gathered so far, in with the running indexer, when the named indexer can
  never run or the group's end never comes) and `indexer_drop_pending` (drop a pending merge a later one replaced),
  and the forge bridge's `bridge_groups` (read the groups it holds open) and `bridge_group_end_again` (publish a
  group's end again from the gate's `merges` state) (ADR 0633 §5). Both of the indexer's hand acts emit
  `take-in.taken`: a hand take-in as any take-in does, and a hand drop naming the drop, for a merge whose wait was
  said pending, so the watchman's `take-in-waits` clears; a drop of a merge never said pending emits none (ADR 0633
  §5, ADR 0639 §9).
 - **The check refuses an indexer the running parts could not move.** For a change to the indexer's own module, the
  check validates the new indexer's manifest, found as the manifest that claims the seat `mesh-catalog`, with the
  trunk's manifest validator in its sandbox and the seat definitions of the members' heads, and refuses the change
  when that validator would refuse or hold it: such an indexer would never run, and every revision behind it would
  wait for ever (ADR 0633 §6, ADR 0636 §4).
 - **How it is checked.** One integration test starts the indexer whole against the bus: a group's two merged
  messages and its end are taken in as one revision, nothing written before the end; a revision changing the
  indexer writes only the indexer's own keys and shows in `indexer_pending`; restarted as the named build, the
  indexer takes every pending revision in, oldest first, with no timer, and a group still gathered stays pending.
  Unit tables cover a later merge joining, a revert, another named build, a refused member, a merge in a member's
  and in another repository, an unknown own build, a compare-and-set conflict, both events, the watchman's two
  bounds and its seeding, the view, the gate passing the end by, and each hand act; three cases cover the seat
  definitions of the revision (accepted without a built-in seat list, a refusal in `refused`, the check accepting
  the same candidate); and rows for an own build fixed at start (a composition answer that differs changes
  nothing), for `take-in.pending` with `own build unknown`, and for a hand drop (a merge said pending emits
  `take-in.taken` naming the drop, one never said pending emits none) (ADR 0639 §10). Live, the registry group's merge shows in `indexer_pending` waiting for the new indexer and
  is taken in when that indexer starts, and no node-engine reads distribution's new entry before the new indexer's
  hold: `node_manifest_reads` and `indexer_holds` on every node (ADR 0633, How it is checked).
 ## A backup before the apply
 *ADR 0214 §7, ADR 0233, ADR 0404, ADR 0429 §14, ADR 0539 §16.*
@@ -1320,7 +1527,30 @@ R4d, the three image reworks (route-proxy, nats, automx; n8n left by ADR 0506),
 ## What is not decided here
 - How long the checker's stream `checks` keeps its events (*A try*, above).
 - Who is granted `checker_try` and the take-in hand tools: the operator's grants, outside this design.
 - A web view of deliveries beyond the pull request's comment.
 - Whether `mesh/delivery-group` is required on the trunks: the operator's setting, through the forge module's
  protection tool.
 - How a builder's public SSH key reaches the forge: the forge's and the SSH client's mechanism.
 ## Reviewed by architectural domain
 The amendment of 2026-11-05 (ADR 0637, ADR 0638, and ADR 0633 with its insights of 2026-11-05) was reviewed by the
 four reviewers by architectural domain side by side (playbook 01), and the lead ruled on every finding in round 1.
 The rulings that no record decided became ADR 0639, which the text cites clause by clause. What was found, and what
 answers it:
 | Domain | Found | Answered by |
 |---|---|---|
 | seams | the indexer said to read none of the checker's state, while it reads the checker's previews and the checker watches its candidates; the forge bridge's read of the gate's `merges` unmarked | *A try* (two reads that stay under ADR 0596 §3 until issue 1966); *Taken in by the indexer it names* (the bridge's read marked the same) |
 | tools | the try's tools not listed against the four lines; the hand acts not said to clear the condition | *A try* (the table of the checker's and the indexer's tools, `indexer_recover_tries` new); *Taken in* (both hand acts emit `take-in.taken`, ADR 0639 §9) |
 | bus forms | the form of `pending` and the try events' contract unsaid | *Taken in* (`pending` in the old form the grants part creates); *A try* (the package `tryevent` of mesh-delivery's sdk) |
 | running qualities | `checker_tries` not answering; the new indexer before the new checker in this release; a lost try event; what a narrowed try keeps; a recovery cut at 64; how long tries are kept; a try waiting for its candidate | *A try* (nothing deleted, asked again, said in `indexer_try_candidates`, the ordering edge; publish with retry, `try.ended` on every end; the narrowed try's verdicts; every open try listed; 7 days; the reason in the try read, bounded by the step bound) |
 | testing | ADR 0633's live case on the node-engines' reads lost; no live case for ADR 0638 | *Taken in* (`node_manifest_reads` and `indexer_holds` on every node); *A try* (the candidate read during the registry group's try, gone after `try.ended`) |
 | health and events | an own build unknown left unbounded; who raises `take-in-waits`; what the two take-in events carry | *Taken in* (`take-in.pending` "own build unknown"; the watchman's finding and the condition keeper, ADR 0564 §2; the events' body) |
 | security | any account could publish a try event; a node's missing composition answer | *A try* (only the checker's account publishes `try.*`, a manifest-test row; the try ends as error naming the node, keeping no verdict) |
 | upgrade | the old indexer's take-in during its replacement: a build learnt lazily and kept let a process being replaced take in with its old validator | *Taken in* (the indexer fixes its own build at its start, never later from `composition`) |
 Left as notes, not in this amendment: the grants of `checker_try` and of the hand tools (the operator's), and the
 test budget of the checker and the indexer.