diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index 5cf49468..afbcf8e6 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -125,6 +125,7 @@ decisions: - "ADR 0250" - "ADR 0239" - "ADR 0238" + - "ADR 0235" - "ADR 0233" - "ADR 0214" - "ADR 0201" @@ -465,12 +466,12 @@ ADR 0656. research 085, not a measurement) the ledger grows about 4 MiB a day, about 360 MiB at the default retention; its cap is 768 MiB, twice that, so a busier month than estimated still fits. Removing a record at its retention removes its operations and its snapshot with it, so the ledger's size follows the retention, not the mesh's age. -- **The switch carries today's state in, and a rollback reads it back.** At the new gate's start, `merges`, `turns` - and every group in flight become records, and the new gate keeps `merges` and `turns` current beside the records, - so a gate rolled back reads current state and no approved group is asked again; milestone H (*Milestones*) stops - writing them. The switch is safe to run twice: a group in flight becomes a record keyed by its milestone and heads, - and a switch that finds that record already there leaves it as it is, so a roll forward after a rollback makes no - second record and asks nothing again. +- **The switch carries today's state in, once.** At the new gate's start, `merges`, `turns` and every group in + flight become records, and from then on the gate writes `merges` and `turns` no more. The switch is safe to run + twice: a group in flight becomes a record keyed by its milestone and heads, and a switch that finds that record + already there leaves it as it is, so delivering milestone G again after a revert makes no second record and asks + nothing again. A revert of the gate is the delivery system's restore of its kept data (*A module's kept data on the + bus, backed up and restored with its delivery*). - **The forge bridge asks the seat for a group.** It reads a group's members from the seat `mesh-delivery`'s verb `groups`, never the gate's `merges` state, both to set the group on each merged message and to publish a group's end again (ADR 0656, which changes ADR 0633 §4 and §5; ADR 0596). Its account holds no grant to read `merges`. @@ -486,7 +487,7 @@ ADR 0656. - **How it is checked.** In the gate's whole-module test: the gate stopped and restarted in the middle of a delivery leaves the record complete with no outcome counted twice; the events `pinned`, `freed`, `heads-replaced` and `record-ended` are said; a `freed` from an account other than the gate's is refused by the bus; a group approved - before the switch is not asked again after it, nor after a rollback; a record ends once every present node runs its + before the switch is not asked again after it, nor after milestone G is delivered again following a revert; a record ends once every present node runs its entries, naming the nodes away; a held and a failed entry are kept with their reason; a record open past 24 hours raises `record-open`, cleared by `record-ended`; a node away that wakes adds its outcomes and is no longer shown away; a group's heads approved and checked again ask the operator nothing again; a small fix's merge gets a record @@ -572,8 +573,8 @@ Findings 11). How each is kept, read and retained is this design's choice under on the bus module to start; the manifest validator refuses either manifest that declares them in the new form (ADR 0619 §3). An assignment graph revision is announced by an event of the seat `mesh-wants` and served by its verbs. This replaces ADR 0603 §2's one key per module with five past values as the graph's form (ADR 0656, which - changes ADR 0603 §2); the keys themselves stay written beside the revisions until milestone H, so that a rolled-back - indexer and assigner read current state (*Rolled back*, below). + changes ADR 0603 §2); the keys are written no more, and a revert of milestone G restores them from their backup + (*Reverted*, below). - **A revision is made by a merge, and also with no merge.** The indexer makes a catalog graph revision when it takes a source revision in. The assigner makes an assignment graph revision on each new catalog graph revision, on the seat `mesh-module-assigner`'s event `placement-changed` (ADR 0567 §5), and on the node-engine seat's event @@ -615,12 +616,13 @@ Findings 11). How each is kept, read and retained is this design's choice under - **The want stays the one record a node reads.** The assigner writes `want..` from a trunk revision's assignment graph only; a node-engine reads its want, never the assignment graph, and the have stays what it runs (ADR 0656, which changes ADR 0571 §8). -- **Rolled back.** A milestone that has to be reverted brings back the indexer and the assigner of before it. They - read ADR 0603 §2's per-module keys and write the wants, both of which the new indexer and assigner keep writing, - current, beside the revisions; the want's form is unchanged. So a reverted indexer and assigner start from current - state, and no want goes back to a fingerprint older than the one each node runs. Writing the old keys is chosen over - converting the revisions back at a rollback because a rollback is the moment nothing new should have to run. The - later change that removes them is milestone H, once milestone G has run for one record retention with no rollback. +- **Reverted.** Only the new form is written: the indexer, the assigner and the gate keep nothing of ADR 0603 §2's + keys or of `merges` and `turns` beside the new forms, since nothing reads them after milestone G. A revert of + milestone G is a change like any other, checked and delivered; before milestone G's delivery moves the indexer, the + assigner and the gate, their kept data on the bus is backed up, and the revert restores it before their reverted + builds start, after which each catches up forward (*A module's kept data on the bus, backed up and restored with + its delivery*, under *A backup before the apply*). That is the delivery system's way for every change of a kept + data's form, not a case of this milestone. ### A change's graph difference is its delivery @@ -743,13 +745,14 @@ ADR 0656 decides the direction and leaves every choice made in working it out to |---|---| | Both graphs kept as objects in an `objects` section in the old form, each with a cap of 512 MiB | both parts are the foundation, whose new kept data goes on the old sections so neither waits on the bus module to start (ADR 0619 §1, §2); the caps are many times the estimated 90 days | | An unchanged entry stored once, a revision being a list of its entries' digests | consecutive revisions differ by a few entries, so the stores grow by what changes; a larger cap over whole revisions would only delay the same fill | -| ADR 0603 §2's keys and the wants kept written beside the revisions until milestone H | a reverted indexer and assigner then read current state and send no node back to an older fingerprint, with nothing to convert at a rollback | +| No old form written beside the new; a revert restores the parts' kept data on the bus from a restore point taken before they moved, then they catch up forward | only the new form is looked after (the operator, 2026-11-06 21:3x), and a revert still works, as the delivery system's capability for every change of a kept data's form | +| A module's kept data on the bus backed up before a `form` change and restored beside the live data at a revert, a missing restore point holding the revert | ADR 0404 covers only the items on a module's node, so the bus's kept data had no restore point; a restore never goes over live data, and a revert with nothing to restore must not start on the newer form unasked | | A revision removed when no record pins it and it is not current, pins counted per revision | a trace must read every revision it moved between, and a revision neither pinned nor current is read by nobody; counting pins keeps a revision two records share | | The indexer and the assigner hear `pinned`, `freed` and `heads-replaced` through durable consumers | an event said while either was down is applied when it comes back, so no revision leaks and none is dropped early | | Records kept 90 days by default, the gate's setting `record-retention`, the ledger capped at 768 MiB, a record's operations removed with it | the traces asked so far went back weeks; about 840 outcome operations make a record about 200 KiB, so 90 days at an estimated 20 a day is about 360 MiB, and the cap is twice that | | `record-open` only when no other condition covers the record | the cause is already told once; a second condition for it is noise | | `full` cleared by `room` when the store falls back below its cap | the refused work is taken up again, and a condition that never clears would hide the next fill | -| The switch into records keyed by milestone and heads, safe to run twice | a roll forward after a rollback makes no second record and asks no group again | +| The switch into records keyed by milestone and heads, safe to run twice | delivering milestone G again after a revert makes no second record and asks no group again | | A store passing 80 % of its cap said as `filling` and raised; a full cap refused as `full`, naming the part | the operator hears of a store filling while there is time, and a full store stops work loudly instead of dropping what a trace needs | | A candidate's graphs kept apart and dropped on `heads-replaced` | a candidate must never write wants or be read by a running part, and its graphs mean nothing once its heads are gone | | The verbs `revision`, `entry` and `difference` on the two seats | each half of the view, and the difference, is served by its one owner through its seat, never read from its state | @@ -769,7 +772,7 @@ ADR 0656 decides the direction and leaves every choice made in working it out to | The running indexer makes a catalog graph revision with the named indexer's entry | the newer indexer can then be wanted and granted, so a revision waiting for it is never stuck behind it | | The acting rule's field `acts` with `step`, `on` and `bound`, the definition format staying 1 | the steps and the language match are what the indexer's map and the checker's cases held in code (research 085, Findings 2); the reader refuses an unknown field, so the runtime learns it in the one rebuild the milestone makes | | The fingerprint handed as `MESH_FINGERPRINT` at a process's start | one value every runtime reads before anything else, a service and a container alike | -| The delivery record's consumer, held and failed outcomes, end and `record-ended`, the finding `record-open`, node away and wake, switch with `merges` and `turns` kept current, and tools | a restart must lose and double nothing; a reader sees why an entry stands still; a node asleep must not hold a delivery open; a gate rolled back must find current state and ask no approved group again | +| The delivery record's consumer, held and failed outcomes, end and `record-ended`, the finding `record-open`, node away and wake, switch done once, and tools | a restart must lose and double nothing; a reader sees why an entry stands still; a node asleep must not hold a delivery open; a switch run twice must ask no approved group again | | A small fix's own record | a small fix is a milestone of one issue (ADR 0493 §3) | | An Approve given held while its heads stand (issue 1972) | the operator approved those heads; a check of them asked again changes nothing they approved (ADR 0441 §1, ADR 0548 §8) | | The milestone events on a subject of their own | a waiter on one member's pull request otherwise wakes on every member's | @@ -1535,6 +1538,56 @@ backup holder backs up nothing, and the have says so. - **What it costs.** About 85% of data modules back up in under 30 s; the slowest measured, postgres on the home server, about 165 s (research 055, finding 5). A module that keeps no backed-up item costs nothing more. +### A module's kept data on the bus, backed up and restored with its delivery + +*ADR 0404, ADR 0235, ADR 0233, ADR 0539 §16, ADR 0595, ADR 0619 §2; from research 085's milestone, this design's +choice under ADR 0656.* + +**ADR 0404's backup does not cover a module's kept data on the bus.** It backs up the items a moved module keeps on +its own node (ADR 0404 §2, ADR 0233), and the bus's streams are dumped whole by the bus module on its own rhythm +(ADR 0235), not per module and not before a delivery. A key-value state, a stream or an object store a module keeps on +the bus (ADR 0595, ADR 0619 §2) therefore has no restore point taken before its module moves. That is a gap of the +delivery system, and milestone G builds it, because a revert of milestone G is the first change that needs it. It is +built as the delivery system's capability for every later change of a kept data's form, not as a case of this +milestone. + +- **Its form is declared.** Each kept data a manifest declares names a `form`, a number its holder raises whenever it + changes how it writes that data. The declaration alone cannot tell a changed layout of the same bucket, so the + holder says it. +- **It is backed up before the module moves.** When a want moves a module whose new manifest declares a kept data + of another `form` than the running one, the node agent applying it first asks the seat `node-backup` of the node + holding `mesh-broker` for a restore point of that module's kept data on the bus, and waits for its event `ended`, + as for any backup before the apply. The bus module's account takes it, as it takes the bus's snapshots (ADR 0235 + §5), one restore point per module and fingerprint. A failed backup holds that module as any failed backup does. +- **A revert restores it before the reverted build starts.** A revert is delivered like any change. When a want takes + a module back across a change of a kept data's `form`, the node agent applying it asks that same seat's verb + `restore` for the restore point taken before the module moved forward, waits for the seat's event `restored`, and + only then starts the reverted build. As every restore is, it is brought back beside the live data and never over + it: the live buckets, streams and objects of the newer form are kept aside under a name of their own, and only the + operator removes them, so nothing written since is destroyed. +- **The restored part catches up forward.** The indexer takes in the source revisions merged since its restore + point; the assigner computes the wants again from the current catalog graph, placements and capabilities, and as + wants come only from the trunk's revision, no want goes to a fingerprint older than the trunk's; the gate reads + the open groups again from the forge and the operator's answers given since from the seat `operator-channel`'s + `asks`. +- **A missing restore point holds the revert, naming it.** When the restore point a revert needs does not exist, or + `restored` does not come, the node agent applies nothing of that module, keeps what runs, and writes the have phase + `held` with the reason `restore point missing` and the module and fingerprint it names; the assigner asks the + operator (hold, or start the reverted build on the live data as it is), raises the condition naming them, and on + the answer writes the node's want again naming it (ADR 0539 §16). +- **What the operator is told.** A revert across a change of a kept data's form puts that part's kept data back to + the moment before the change moved it: the deliveries made since stay delivered, but what the part wrote since + (for milestone G, its delivery records and its graph revisions) is set aside and not read again; an Approve given + since is read back from the operator's answers, and only an answer the mesh no longer holds is asked again. The + revert's Approve ask says this in those words, naming each part restored. +- **Its tools.** The seat `node-backup`'s `backed-up` lists each module's restore points of its bus data, with the + fingerprint and form each was taken before; its `restore` restores one. Both exist on the seat today and gain the + module's kept data on the bus as an item. +- **How it is checked.** Whole (node-engine, backup holder, bus module): a want moving a module across a `form` + change takes a restore point of its bus data first; a want taking it back restores it beside the live data before + the reverted build starts, and the live data of the newer form is kept aside; a revert whose restore point is + missing is held, naming it, and moves on only on the operator's answer. + ## Compositions, wants and the apply *ADR 0360 §8, §9, ADR 0377, ADR 0429 §4, §6, ADR 0539 §9, §14 to §16, ADR 0547 §2 to §9, ADR 0549, ADR 0555.* @@ -2104,12 +2157,12 @@ list: running build, also after six later merges; a group approved before the switch to the records is not asked again. 35. **The trace is links.** Research 085's milestone reads back from research 085 to its decisions, its milestone, its issues and its pull requests with no text match. -36. **The milestone can be undone, whole.** In a whole-module fixture of the indexer and the assigner, milestone G's - versions take in revisions and write wants, then the versions of before it are started in their place: they read - the per-module keys the new ones kept writing, compute the same wants, and write no want naming a fingerprint - older than the one each node runs; the gate rolled back the same way asks no approved group again, and rolled - forward again makes no second record. Whole (checker, gate, assigner): a revert of milestone G passes its check on - the checking program built from the revert's own heads, and is delivered. +36. **The milestone can be undone.** A revert of milestone G is checked on the checking program built from the + revert's own heads and delivered; before the reverted indexer, assigner and gate start, their kept data on the bus + is restored from the restore points taken before milestone G moved them, beside the live data; they catch up + forward, and no want goes to a fingerprint older than the trunk's; the operator is asked again for no Approve whose + answer the mesh still holds. A revert whose restore point is missing is held, naming it (whole, checker, gate, + assigner, node-engine, backup holder). 37. **Nothing is delivered ahead.** Research 085's milestone is delivered with no part delivered ahead of it, read from its delivery record (ADR 0656, How it is checked). 38. **A newer indexer is reached.** A merged revision naming a newer indexer gives, from the running indexer, a catalog @@ -2141,8 +2194,7 @@ Each change ADR 0656 makes in an earlier record, and its test: builder). 44. **The graph is kept per revision** (ADR 0603 §2). A source revision taken in gives one catalog graph revision and its assignment graph revision, kept as objects in an `objects` section in the old form, and a revision a delivery - record pins is not removed; every part reads the revisions, and the per-module keys are written only for a - rollback, read by no part of milestone G (whole, indexer and assigner). + record pins is not removed; no per-module key with five past values is written (whole, indexer and assigner). 45. **The tiers are a view of the difference** (ADR 0603 §6). The view's tiers of a group equal those read off its difference by `difference`, with no other source read (whole, view and assigner). 46. **A check fails by name** (ADR 0603 §10, ADR 0614 §7). A fixture change whose acting part does not build, one @@ -2209,9 +2261,7 @@ Each change ADR 0656 makes in an earlier record, and its test: changes them); the module graph's one key per module with five past values as what parts read (ADR 0603 §2, as ADR 0656 changes it); the tiers worked out from five live sources (ADR 0603 §6, as ADR 0656 changes it); a seat list compiled into a part, and every read of another module's bus state the revision view replaces (ADR 0656); and - the gate's `merges` and `turns` as its own state, carried into the delivery records. -- At milestone H's delivery: the writing of ADR 0603 §2's per-module keys beside the revisions, and of the gate's - `merges` and `turns` beside the records, kept until then only so that a rollback of milestone G reads current state. + the gate's `merges` and `turns`, carried into the delivery records and written no more. ## Milestones @@ -2262,8 +2312,7 @@ release reports the controller's code lines against research 063's baseline (acc | L6 | Provisioning leaves the controller (ADR 0570): every provision is one key of the seat `mesh-provisioner`, held by mesh-provisioner, holding a reference and never a value, and each placed module's provisioning plan a key of its state `plans`, both composed from placement, the wants and the manifests on change events; each provider acts once all a provision depends on is provided; the assigner moves a want only once its build's provisions are provided; a provision no want needs is retiring, and its account or credential is removed only on the operator's Approve, its data never; the vault mints each provision's secret sealed to the consumer's and the provider's node, and the consumer's node-engine writes it; a wait for a person is a provision's state, which the watchman reads; bus permissions stay with the grants part (ADR 0349, ADR 0230, ADR 0547 §11) | — | mesh-sdk, the node-engine (`mesh-host`), mesh-tools (the tool runner's provisioner client), mesh-catalog (mesh-provisioner, the vault, the provisioner client, postgres, keycloak, gitea, route-adapter), mesh-delivery (assigner, watchman) | 1954 | | L7 | History and records leave the controller (ADR 0571): the builder keeps its build log, one entry at a build's start and one at its end, from which build history and durations are read and the module mesh-catalog asks the builds it missed; the artifact sweep is the seat `mesh-artifact-sweep`, held by artifact-store-tools beside the artifact store, its keep set from the store's own listing, the wants and the build log, sweeping at each build's end from the lease write and recording each deletion in the hand-act log; disk readings are disk-load's own state on every node; who holds which seat is the state `seat-holders` of the seat `mesh-placement` (now `mesh-module-assigner`), and tools are found by the tool runners' announcements; what a node runs is its want and its have; the grants part writes `grants..` and keeps memberships in its own state; root-free is a probe of the self-check; no manifest but the controller's reads, invokes or consumes `mesh-controller`, but a named list kept to the lease write, so the lease write's check passes (ADR 0189, ADR 0547 §11, ADR 0549 §6) | — | mesh-sdk, mesh-controller (its pin only), mesh-delivery (builder, indexer, assigner, grants part, view), mesh-catalog (artifact-store-tools, disk-load, the backup seat's holder, mounts, nfs-server, minio, mesh-watcher, the messenger, the self-check, the module mesh-catalog, docker), mesh-tools (the tool runner) | 1955 | | B0 | The bootstrap delivery: a delivery group's check runs a checking program built from the group's own heads in the build seat's sandbox, and hands the manifest validator and the composer the heads' one seat list; it brings no seat definition, so the running checker checks it (ADR 0644) | 26 needs it | mesh-delivery (checker), mesh-catalog (mesh-build-go), mesh-delivery-contracts (the seat `node-build-go`'s ask `run`) | 1921 | -| G | Two graphs and one revision view: the catalog graph and the assignment graph kept per revision, a change delivered as its graph difference, the acting parts' rules on their seats as bounded waits that hold their modules when the new version fails, the check on the change's own acting parts, the revision view and each process's own build, the gate's delivery record, and the trace's links (ADR 0656); the registry milestone's reviewed heads are joined to its group as members | 26 to 49 | mesh-sdk (the runtime: the fingerprint, the view, the definition reader), mesh-contracts and the domains' contracts modules (`acts`, the new verbs and events), mesh-delivery (indexer, assigner, checker, builder, grants part, gate's view, forge bridge), mesh-catalog (the gate, the build modules, mesh-issues, mesh-knowledge), the node-engine (`mesh-host`), hq | 1966 (with the registry milestone's heads), 1961, 1660, 1962, 1972, 1721, 1783, 1785, 1794, 1791, 1373 | -| H | The old keys go: once milestone G has run for one record retention (90 days) with no rollback, the indexer stops writing ADR 0603 §2's per-module keys, the assigner reads none, and the gate stops writing `merges` and `turns` (*Two graphs, one source of truth*, *Rolled back*) | — | mesh-delivery (indexer, assigner), mesh-catalog (the gate) | 1921 | +| G | Two graphs and one revision view: the catalog graph and the assignment graph kept per revision, a change delivered as its graph difference, the acting parts' rules on their seats as bounded waits that hold their modules when the new version fails, the check on the change's own acting parts, the revision view and each process's own build, the gate's delivery record, and the trace's links (ADR 0656); the registry milestone's reviewed heads are joined to its group as members | 26 to 49 | mesh-sdk (the runtime: the fingerprint, the view, the definition reader), mesh-contracts and the domains' contracts modules (`acts`, the new verbs and events), mesh-delivery (indexer, assigner, checker, builder, grants part, gate's view, forge bridge), mesh-catalog (the gate, the build modules, mesh-issues, mesh-knowledge, the `node-backup` holder, nats: a module's kept data on the bus as a backed-up item), the node-engine (`mesh-host`), hq | 1966 (with the registry milestone's heads), 1961, 1660, 1962, 1972, 1721, 1783, 1785, 1794, 1791, 1373 | An issue is in one row only (ADR 0493 §2): where a row named an issue another row names too, the last row keeps it and the earlier row says so in words (the operator, 2026-10-26). Milestone M1 makes each row a record of the issue tracker