diff --git a/00-META/glossary.md b/00-META/glossary.md index b41fddcb..fc4eecb9 100644 --- a/00-META/glossary.md +++ b/00-META/glossary.md @@ -974,6 +974,12 @@ becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252, 0283, 0564, 0668. at one moment; **restore** — bringing it back beside the live data, never over it. - **stream snapshot** — the bus's own backup of each stream, taken by the module holding `mesh-broker` (ADR 0235). +- **recovery module** — the holder of the seat `mesh-recovery`, held once for the mesh: under an account of its own it + takes a restore point of a module's kept data on the live bus before the module moves across a `form` change, sets + the live data aside and restores the point at a revert, and lists and removes restore points; the bus module and + the snapshot account restore nothing into the live bus, and the backup holder only stores copies (design 47 G §554). + **aside** — live kept data a restore set aside, removed only by the operator's `discard-aside`. + *Not:* ~~restorer~~, ~~restore account~~ - **disk readings** — the size of each data item a node's modules declare, measured by its `measure` and kept with its largest size in seven days by disk-load on that node, in its own state, one key `..` written only when it changes; the holder of `node-backup`, mounts, nfs-server and the diff --git a/03-DESIGN/01-to-be/47-milestones/G-12-develop-answers-fourth.md b/03-DESIGN/01-to-be/47-milestones/G-12-develop-answers-fourth.md index 900f39d1..e597b32c 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-12-develop-answers-fourth.md +++ b/03-DESIGN/01-to-be/47-milestones/G-12-develop-answers-fourth.md @@ -10,6 +10,7 @@ decisions: - "ADR 0660" - "ADR 0656" - "ADR 0636" + - "ADR 0626" - "ADR 0619" - "ADR 0618" - "ADR 0614" @@ -17,6 +18,7 @@ decisions: - "ADR 0595" - "ADR 0564" - "ADR 0505" + - "ADR 0404" - "ADR 0363" - "ADR 0240" - "ADR 0235" @@ -32,63 +34,73 @@ answer what the four reviewers found in round 3 of [G-13](G-13-review-round-3.md of its own after it. Every other line answers a question develop met after [G-11](G-11-develop-answers-review.md), of internal form only (playbook 07, step 0b), each the design's choice with its reason. What a question of another kind leaves open waits for the operator and is said in the -line. Questions that an earlier line already answers have no line here: develop reads that line.* +line. The operator's answer of 12:55 CET (the recovery module, §554 and §555) and the +review of §516 to §553, round 1 (§556 to §558), are in [G-14](G-14-recovery-module-and-review.md); the findings of +that round are written into the lines they name, here. Questions that an earlier line already answers have no line here: develop reads that line.* ## Kept data on the bus: restore points and what is set aside -- §516 **A restore point's bytes, and the forms of `snapshot-kept` and `restore-kept`.** (§551) The seat `mesh-broker`'s two - verbs (§405) take and answer the forms its definition in mesh-contracts gives them, which this line makes the - design's: `snapshot-kept` takes `module`, `kept` (the kept data's local name), `form`, `fingerprint`, `why` and - `catch-up` as `{tool, arguments}` (§451), and answers `point` (the module, the kept data, its form, the - fingerprint, `snapshot`, `consumers` with each durable consumer's stream, name and acknowledged sequence, `size` in - bytes, `at` and `catch-up`) or `error` alone; `restore-kept` takes `module`, `kept`, `form`, `fingerprint`, `why` - and one of `snapshot` (the restore point's name), `aside-only` or `discard-aside` (an aside's `id`), and answers - `restored`, `aside` and `error`. No argument `local-name` or `from` is taken, and a catch-up is never text. The bus - module keeps a restore point in its own `snapshots` directory, which ADR 0235 §1 to §3 give it for the bus's dump: - the server's snapshot of each bus object of the kept data (its stream, or the stream under its bucket or object - store), taken one at a time with ADR 0235 §10's pacing and §11's bounds, and the point's record, in the directory - `kept///
-`, each token escaped as a key's name is (§73); that name is the point's - `snapshot`. The holder of `node-backup` backs that directory up as it backs up `snapshots` (ADR 0235 §3), so its - copy is the point's safeguard, never the point: when its `restore` finds the point's directory missing on the bus - module's node, it brings it back from its newest repository snapshot that holds it, under the same name, and only - then calls `restore-kept`; when none holds it, `restored` says `failed` with `restore point missing`, and §300 holds - the revert. `restore-kept` sets the live kept data aside by taking the server's snapshot of each of its bus objects - into `aside///` in the same directory (`at` the UTC time written `20261111T121200Z`), only then - removes the live objects, restores the point under their names and sets each consumer to its recorded sequence. The - aside's `id` is that directory's name, its `name` the bus name it was live under, its `size` its bytes on disk; - nothing of an aside stays on the bus, so it takes none of the bus's storage limit, and the definition's - description of `name` reads "the bus name it was live under". `discard-aside` removes that directory and nothing - else. A restore point is removed only as §551 says. This completes §405 and §406. Why: ADR 0235 has the bus - module's account write the bus's snapshots into its own directory and the backup holder back that directory up; - the server restores a stream only where no stream of that name exists, so the live data is set aside as a - snapshot before the point takes its name (develop questions BKP1, BKP3 and NATS4's argument names). How it is - checked: unit (mesh-broker's contract), each verb's input and answer read, an input naming `from` refused; whole - (bus module, backup holder), a point is taken, its directory removed, brought back from the repository and - restored, the live data's aside listed with its size and the consumer at its recorded sequence; a point missing - from both says `restore point missing`. -- §517 **`backed-up` reads the restore points and the data set aside from the backup holder's own state.** The holder of - `node-backup` on the bus module's node keeps what `snapshot-kept` and `restore-kept` answered it in its state - `bus-points`, asked in its `requires` by the template `state` (ADR 0595 §1; the holder is no foundation part, ADR - 0619 §1, ADR 0666 §4): a point under `point....` with the answer's `point` whole, - written when `snapshot-kept` answers done; an aside under `aside...` with the answer's `aside` - whole, written when `restore-kept` answers one, each token escaped as a key's name is (§73). The holder deletes an - aside's key when `discard-aside` answers done, and a point's key when it hears that point's `point-removed` (§551). `backed-up` lists both from - that state, each with its size (§302), and the holder's read tool of its state reads it. The seat `mesh-broker` - gains no read verb. The holder's old `state` entry `good` moves to `requires` in the same change, keeping its local - name and settings, as ADR 0595 §11 moves a module G touches. Why: the holder took each from its own call's answer, - so the list is its own fact (ADR 0596 §2), and a read verb on `mesh-broker` would answer what the caller already - holds (develop question BKP2). How it is checked: whole (backup holder), a restore's aside is listed by `backed-up` - with its size, is gone after `discard-aside`, and a restart of the holder lists the same. +- §516 **A restore point's bytes, and the forms of `snapshot-kept` and `restore-kept`.** (§551, §554) The seat + `mesh-recovery`'s two verbs (§554) take and answer the forms mesh-contracts' definition gave the seat `mesh-broker`'s, + which move to `mesh-recovery` and which this line makes the design's: `snapshot-kept` takes `module`, `kept` (the + kept data's local name), `form`, `fingerprint`, `why` and `catch-up` as `{tool, arguments}` (§451), and answers + `point` (the module, the kept data, its form, the fingerprint, `snapshot`, `consumers` with each durable consumer's + stream, name and acknowledged sequence, `size` in bytes, `at` and `catch-up`) or `error` alone; `restore-kept` takes + `module`, `kept`, `form`, `fingerprint`, `why`, `from-backup` (yes or no) and one of `snapshot` (the restore point's + name), `aside-only` or `discard-aside` (an aside's `id`), and answers `restored`, `aside`, `missing` (true when the + point is not in the recovery module's directory) and `error`. No argument `local-name` or `from` is taken, and a + catch-up is never text. + - **Where a point lives.** The recovery module takes a point through the bus's stream snapshot under its own account + (§554), one bus object of the kept data at a time (its stream, or the stream under its bucket or object store), + with ADR 0235 §10's pacing and §11's bounds, and writes it with the point's record in its own data directory, + `points///-`, each token escaped as a key's name is (§73) and a token that is + empty, `.` or `..` refused; that name is the point's `snapshot`. + - **Its copy.** That directory is an item of the recovery module's manifest `data`, backed up by the holder of + `node-backup` on its node as every item is (ADR 0404), which stores the copy and nothing else. When + `restore-kept` answers `missing`, the node agent calls it again with `from-backup: yes`: the recovery module asks + the seat `node-backup`'s `restore` of its item for that directory, which brings it back beside its path as that + verb always does, takes the point from there and restores it; when no repository snapshot holds it, `restored` + says `failed` with `restore point missing`, and §300 holds the revert. + - **The aside, and a restore that fails.** `restore-kept` sets the live kept data aside by taking a stream snapshot + of each of its bus objects into `aside///` in the same directory (`at` the UTC time written + `20261111T121200Z`, with `-2`, `-3` and on added when that name exists), only then removes the live objects, + restores the point under their names and sets each consumer to its recorded sequence. Each step is written first + in the aside's record (`set aside`, `removed`, `restored`): when a step after the removal fails, `restore-kept` + restores the aside under the live names and sets the consumers back before it answers `failed`, and at its start + the recovery module does the same for every aside whose record ends at `removed`, so a failed or stopped restore + never leaves the kept data gone. The aside's `id` is that directory's name, its `name` the bus name it was live + under, its `size` its bytes on disk; nothing of an aside stays on the bus. `discard-aside` removes that directory + and nothing else. A restore point is removed only as §551 says. + + This completes §405 and §406. Why: the operator's answer of 12:55 CET (§554) puts the taking and restoring of a + module's kept data on the live bus on a module of its own; the server restores a stream only where no stream of that + name exists, so the live data is set aside as a snapshot before the point takes its name (develop questions BKP1, + BKP3 and NATS4's argument names; review round 1, running and testing 1, seams and tools' notes). How it is checked: + unit (mesh-recovery's contract), each verb's input and answer read, an input naming `from` refused; whole (recovery + module, backup holder, bus), a point is taken, its directory removed, brought back with `from-backup` and restored, + the live data's aside listed with its size and the consumer at its recorded sequence; a point missing from both says + `restore point missing`; a restore made to fail after the removal, and one stopped there and started again, leave + the live data as before. +- §517 **The recovery module lists what it holds.** The seat `mesh-recovery`'s read verb `kept-points` (class `read`, + §391's default callers) lists every restore point and aside in its directory with its module, kept data, form, + fingerprint, size and `at`, and its tools are `recovery_kept_points`, which reads the same, and + `recovery_expire_points`, which runs §551's removal now (class `act`, §391's default callers, written in the + hand-act log). The directory is the one place they are kept, so no copy can list less: the backup holder keeps no + state of them (its `backed-up` lists the recovery module's item as any item, §302), and no `bus-points` state is + made. Why: a module's tools read everything it holds (*Tools are how a module is spoken to*), and the list is the + holder's own fact (ADR 0596 §2; develop question BKP2; review round 1, seams and tools 4). How it is checked: whole + (recovery module), `kept-points` lists a point and an aside with their sizes, an aside is gone after + `discard-aside`, and a restart lists the same. - §523 **A seat event whose subject has parts.** A seat definition's `subject` with `` tokens (§397), such as - `node-backup`'s `ended...` and `restored...` (§405), is said on + `mesh-recovery`'s `ended..` and `node-backup`'s `item-started..` (§554), is said on `mesh.seat..event.` followed by the subject, each part filled in the definition's order: the seat generator's subject function for the event takes each part as a string in that order and fills each as one token, - escaped as a key's name is (§73), refusing an empty part. A reader hears one module's events by the subject with + escaped as a key's name is (§73), refusing an empty part, `.` and `..`. A reader hears one module's events by the subject with that module's tokens and `>` after them, and every module's by `*` for each part. Such an event has no key of its own and is kept as its definition's `keep` says, or on EVENTS when it names no `keep` (§396). The holder's account publishes it as a seat's holder publishes its seat's events today; no grant changes. Why: §397 fixes the subject's words, not how a part is filled (develop question NATS4's wire). How it is checked: unit (seat generator), a subject - of three parts made in order and an empty part refused; whole (backup holder), `ended` heard by a reader of one + of three parts made in order and an empty part refused; whole (recovery module), `ended` heard by a reader of one module's subject. ## The tool runner and its bundles @@ -98,17 +110,23 @@ line. Questions that an earlier line already answers have no line here: develop §345's put session and §490's get session run on that channel, as every `mesh/` call of a bundle does (ADR 0595 §9); `` in §457, §489 and §490 names that call, and no bus subject. A value at or under the limit is put and got by one call, as today. Over it: + - **Whose object.** A session's `name` names an object of the bundle's own module alone: the tool runner resolves + it in that module's binding (its state's `-objects` or its sent store), and refuses a name of another + module's, naming it. A put's whole size is bounded by that store's cap, and a put whose chunks pass it is refused + at the chunk that does, naming the store. - **Put.** The bundle sends each chunk as the call `mesh/objects.put-chunk` with `name` (the object), `call` (the bundle's id for this put), `seq` (from 1, one more each chunk), `last` (true on the last chunk alone) and `data` (the chunk's bytes in base64, at most half the bus's maximum payload before encoding, §457). The tool runner answers each with `{seq}` or `{error}`, the bundle sends the next only after a `{seq}`, and the answer to the last is the put's answer; a `seq` that is not one more than the last taken is answered `{error}` naming both, and a put - not sealed within the call's bound is dropped (§345). The tool runner writes the whole as one object through the - object store's own chunking. + not sealed within the call's bound is dropped (§345). The tool runner writes each chunk into the store as it takes it, through the + object store's own chunking, holding no more than one chunk in memory, and seals the object on the last. - **Get.** The tool runner answers the get call `{chunked: , size, digest, call}` (§490), and the bundle asks each chunk in order by the call `mesh/objects.get-chunk` with `call` and `seq`, answered `{seq, last, data}` or `{error}`; the bundle checks the whole against `digest`, and a get not completed within the call's bound fails, - naming the object. + naming the object. The tool runner drops a get session it holds when the call's bound passes, or the bundle's + channel closes, whichever comes first. + - **The bound.** The call's bound counts once for the whole session, from its first chunk, never per chunk. The headers `Mesh-Put-Seq` and `Mesh-Put-Last` of §457 are these fields `seq` and `last`. Nothing of a session is on the bus, so §489's grants are not made and the grants part grants nothing for it. A module's process, which @@ -117,8 +135,9 @@ line. Questions that an earlier line already answers have no line here: develop calls. This amends §345, §457, §489 and §490; in the get the bundle asks each chunk, since on that channel every `mesh/` call is the bundle's. Why: the operator's answer to develop question D-13, chunks in order, each acknowledged before the next, is kept whole on the one channel a bundle has (develop questions T-5 and T-7). How it - is checked: whole (mesh-sdk, tool runner), a value of a hundred times the limit is put and got back whole by a - bundle, its digest checked, and a chunk out of order fails the put naming both sequences; the grants part's table + is checked: whole (mesh-sdk, tool runner), a value of a hundred times the limit, into a fixture state whose + `objects-max-mib` holds it, is put and got back whole by a bundle, its digest checked, a put naming another + module's object is refused, and a chunk out of order fails the put naming both sequences; the grants part's table test grants no session subject. - §519 **A bundle's fingerprint is in the environment the node-engine composes.** The node-engine writes `MESH_FINGERPRINT`, the fingerprint of the want it applies for the bundle's module, into each tools bundle's @@ -133,25 +152,31 @@ line. Questions that an earlier line already answers have no line here: develop - §520 **`running` says a build's health, and the assigner reads health from it alone.** The node-engine seat's `running..` gains `healthy` (true or false), the health of the build that runs as the node-engine's have holds it from the module's declared check (ADR 0240), and the node-engine says `running` again on the same - subject each time `healthy` changes, so the stream, kept last per subject, holds each build's newest health. - `stopped` (§365, §404) is said only when the have no longer holds the module, never for an unhealthy build. The + subject each time `healthy` changes, and for every module its have holds at its own start, so the stream, kept last per subject, holds each build's newest health. + `stopped` (§365, §404) is said only when the have no longer holds the module, never for an unhealthy build. A `running` with no + `healthy`, said by a node-engine of before G, reads as healthy while it stands, as `running` meant on main. The assigner reads a provider's health for rule 5, the canary's for rules 3 and 4, and a put's end from these events - alone, through its durable consumer that starts from the last message per subject (§199), and reads no have: a - provider of which nothing was heard since that consumer began counts as not healthy, so its consumers' new wants + alone, read as §556 has it, and reads no have: a + provider of which nothing was heard counts as not healthy, so its consumers' new wants are held as rule 5 holds any, and nothing falls back to the have. `SeatsToPassOn` reads no `healthy` (§459): an unhealthy holder still holds its seat. The assigner's `reads` of the node-engine's have go. This amends §309 and §404. Why: the assigner reads no other module's state (ADR 0596 §2), and a stopped build and an unhealthy one are told apart (develop questions AS15 and AS16). How it is checked: unit (the node-engine seat's fixtures), `running` with `healthy` read; whole (assigner, node-engine), a provider turning unhealthy holds a consumer's new want and its - recovery lets the want go, with the assigner's account holding no read of the have. + recovery lets the want go, with the assigner's account holding no read of the have; whole (assigner, a + node-engine of before G), its provider's `running` with no `healthy` holds no consumer's want; a restarted assigner + holds nothing it did not hold before. - §524 **The node-engine checks a want's parts by their digests.** A composition's archive, process and container - resources name a source and a digest, not an artifact, so the node-engine matches no resource to a `parts` key: it - applies a want only when the digest of every resource it fetched is among the values of the want's `parts`, and + resources name a source and a digest, not an artifact, so the node-engine matches no resource to a `parts` key. A want + with no `parts` (from an assigner of before G, or for a node whose record does not state `want-parts`, §525) is + applied as today, its resources fetched by their sources and checked by their own digests. A want with `parts` is + applied only when the digest of every resource it fetched is among the values of the want's `parts`, and every value of `parts` is the digest of a resource it fetched; otherwise it applies nothing of that module and its have says the phase `parts-differ`, naming the digest. `parts` stays keyed by `:` for people and the delivery record. This completes §379. Why: the digest is what is fetched and checked (develop question H-8). How it is checked: unit (node-engine), a want whose parts and fetched digests agree applies, and one with a fetched - digest outside `parts` or a part not fetched applies nothing, naming it. + digest outside `parts` or a part not fetched applies nothing, naming it; whole (node-engine of G), a want written by + an assigner of before G is applied. - §525 **A node-engine says it reads the want's new fields.** A node-engine of G states the capability `want-parts` in its node record's capabilities (ADR 0614 §4, ADR 0618 §2), and the assigner writes `version`, `parts` and `kept-on` (§526) into a node's want only while that node's record states it; that is §379's "once that node runs G's @@ -168,11 +193,12 @@ line. Questions that an earlier line already answers have no line here: develop the catalog half of their view: the module whose entry claims the seat `node-engine`, which §508 makes one per revision; the manifest validator, beside §508's refusal of a second claim, refuses a revision in which no module claims it, naming the seat. §364's, §365's and §508's "from the seat list" read so, the condition keeper's too. With - no entry claiming the seat, which only a revision from before G can show, the node-engine applies nothing of that - revision and its have says `entry-missing`, naming the seat. Why: §364 and §508 have the node-engine claim its seat + no entry claiming the seat, which only a revision from before G shows, as at a revert of G, the node-engine's own + module is the module of the catalog graph entry its own build's fingerprint names (§443, §205), so a revert of the + node-engine applies. Why: §364 and §508 have the node-engine claim its seat as one module, so the claim names it (develop question H-15). How it is checked: unit (manifest validator), a revision with no claim of `node-engine` refused; whole (node-engine), its guard protects the module its catalog - half names, and the code review finds no fixed name (§428). + half names, and with a revision of before G the module its own fingerprint's entry names, and the code review finds no fixed name (§428). ## Findings and the condition keeper @@ -186,8 +212,13 @@ line. Questions that an earlier line already answers have no line here: develop - **Every other part** asks both in its `requires`: the state `findings` by the template `state`, keeping its local name and carrying its settings as options (ADR 0595 §11), and the stream by the template `events` under the local name `finding-events`, with `max-age-days` 7 and `max-mib` 16, which the template drops oldest when full (ADR 0595 - §3), published on its binding's subject, `mesh.bus..finding-events..`. `mesh-healers`, whose - old `streams` declares `mesh-healers-findings` on main, moves it so in G's change. + §3), its entry listing the three finding events in its field `events`, which the catalog graph entry carries + (§365), published on its binding's subject, `mesh.bus..finding-events..`. + - **`mesh-healers`**, whose old `streams` declares `mesh-healers-findings` on main, moves it so in G's change, as + ADR 0595 §11 moves a module G touches. Its old stream stays on the bus as an orphan that `orphans` lists, removed + only by the operator's `drop` (ADR 0595 §4, §5); the grants part deletes the keeper's consumer on it once the + healers' entry no longer declares it (§386), and the healers say `findings-stated` at their start on the new + stream (§384), so the keeper loses no finding. The manifest validator applies ADR 0595 §11's count of old sections to no part of its foundation list (ADR 0619 §2), so the tool runner's and the checker's manifests are not refused for it, and accepts in a foundation part's @@ -196,8 +227,9 @@ line. Questions that an earlier line already answers have no line here: develop list, fixed in the validator (ADR 0619 §3). The validator refuses a manifest whose state `findings` has no finding stream in its part's form, or no `consumes` of `mesh-conditions.findings-asked` (§387). The writer (§461) is handed the subjects from the part's binding or its old stream, as it is handed every subject. The catalog graph entry - carries the finding stream in either form (§365); the grants part makes the keeper's consumer of §386 on either - stream and grants each part publish on its own finding stream's subjects of its own node only, in either form; the + carries the finding stream in either form (§365); the grants part makes the keeper's consumer of §386 on an + old stream at the revision that declares it, and on a template stream when it hears the bus module's `created` for + `finding-events` and grants each part publish on its own finding stream's subjects of its own node only, in either form; the keeper takes the observer and the node from either subject. This amends §383, §386 and §387, whose "into the manifest's `streams`" holds for a foundation part alone. Why: ADR 0666 §4 and ADR 0595 §11 put a part's kept data in the templates unless it is a foundation part, and ADR 0619 §2 keeps the foundation's old sections whole; the @@ -205,32 +237,42 @@ line. Questions that an earlier line already answers have no line here: develop name per module (develop questions on §387, both validator refusals). How it is checked: unit (manifest validator), a foundation part's old `findings` and finding stream accepted, a non-foundation part's old sections refused and its template form accepted; the catalog's validation at G's heads refuses none of §387's manifests; whole (keeper, a - template observer and the tool runner), both forms' events raise a condition. + template observer and the tool runner), both forms' events raise a condition; whole (keeper, healers), a healers' + finding raised before their move is still raised after it. - §540 **The keeper hands a failed head to the operator after two hours.** The condition keeper raises `pipeline-failed` (§33) with `owner` the commit's author, as the runner's finding names it, and two hours after the finding's `since` sets the condition's `owner` to `operator-channel`, the seat the operator is asked through (a seat's name, §451), and says that change as it says any change of a condition; the runner's finding keeps the - author. The keeper times the two hours on its own conditions from `since`, so after its restart it times them from + author. §451's `owner` and `waiting-on` gain a third form, `forge:`, the commit's author as the forge + names it. A `pipeline-failed` for a newer failing head is a finding of its own, its id naming that commit, so it is + its new author's with two hours of its own. The keeper's tool `conditions_owner` sets a condition's `owner` now, of + class `act` with §391's default callers, written in the hand-act log. The keeper times the two hours on its own conditions from `since`, so after its restart it times them from `since` again. Why: §33 has the keeper raise it as the author's, then the operator's (ADR 0505 §5, §6; develop question COND7). How it is checked: whole (keeper), a `pipeline-failed` is the operator's two hours after its - `since`, and so after a restart of the keeper. + `since`, and so after a restart of the keeper; unit (mesh-conditions' contract), `forge:` read. - §541 **The keeper hears what a node runs and the healers' attempts as events.** The keeper's `reads` of `node-engine.have` and `mesh-healers.attempts` go (§434). - **What a node runs.** The keeper consumes the node-engine seat's `composed`, `running` (with `healthy`, §520, and - `stopped`) and `wait-held` and `wait-ended` (§499), from the last message per subject (§199), and compares them - with the wants it hears from the assigner. The node-engine says `wait-held`, with `step: compose` and the reason, - for each hold its have writes (§300's `restore point missing`, §421's and §527's `entry-missing`, §524's - `parts-differ`, §499's kept build), and `wait-ended` when the hold ends. + `stopped`) and `wait-held` and `wait-ended` (§499), read as §556 has it, and compares them with the wants it hears + from the assigner, to hold a module's findings while it is delivered (ADR 0564 §13), as it did from the haves. + The node-engine says `wait-held` on `mesh.seat.node-engine.event.wait-held..`, with `step: compose` + and the reason, for each hold its have writes (§300's `restore point missing`, §421's `entry-missing`, §524's + `parts-differ`, §499's kept build), and `wait-ended` on `.wait-ended..` when the hold ends. The + keeper raises no condition of its own from them: the watchman writes `delivery-waits` for a `wait-held` of the + node-engine standing past composing's 30 minutes (§266), naming the node, the module and the reason, and it clears + on `wait-ended`; a `composed` whose fingerprint is not the want's is covered by that same wait or by the put's + wait (§498). - **The healers' attempts.** The seat `mesh-healers` says `attempt-changed` on `mesh.seat.mesh-healers.event.attempt-changed.`, the condition's key escaped as a key's name is (§73), carrying the attempt's value whole as its key in `attempts` holds it and `until`, the end of that key's lifetime, or `removed: true` when the healer ends the attempt; the seat's definition, in its contracts module, - keeps it last per subject, 7 days and 16 MiB (§396). The keeper's durable consumer starts from the last message per - subject, and the keeper drops an attempt at its `until` on its own clock. + keeps it last per subject, 7 days and 16 MiB (§396). The healers say it again for every open attempt at their start. The + keeper reads it as §556 has it, and drops an attempt at its `until` on its own clock. Why: the keeper reads no other module's state (ADR 0596 §2; develop question COND8). How it is checked: whole (keeper, healers, node-engine), an attempt said is shown on its condition and dropped at its `until` or its - removal, a node-engine's hold is heard from `wait-held`, and the keeper's manifest declares no `reads`. + removal, a node-engine's hold is heard from `wait-held` and past 30 minutes raises `delivery-waits`, a restarted keeper + holds the attempts it held, and the keeper's manifest declares no `reads`. ## Views of the catalog half, per watched branch @@ -263,14 +305,18 @@ line. Questions that an earlier line already answers have no line here: develop runner's own wiring of `catalog-revision.>` goes for it. - **The bus module across branches.** The bus module plans a module's kept data from the newest revision of each watched branch that lists the module: each local name once, from the first of the module's watched branches, in - its registration's order, whose revision lists it; another branch asking that local name with other settings is + its registration's order, whose revision lists it. A registration's watched branches are a list in that order: the + branch `register` names first (the repository's default branch, a placement's default, ADR 0666 §6), then each + branch in the order `watch` added it; `unwatch` removes one and keeps the others' order; another branch asking that local name with other settings is said as the bus module's `drifted`, naming the branch (ADR 0595 §6), and plans nothing. This amends §358, §365, §421, §433 and §507. Why: no part fixes a branch (§7, ADR 0666 §5), and a registration may add a watched branch while a part runs (develop questions NATS3, VW1 and COND6, and the client's lost per-branch events). How it is checked: unit (`go/views`), a view adopted after the start takes its first view and follows, and two branches' events reach their own views; whole (keeper, bus module), a branch registered while they run is - followed with no restart; the code review finds no branch's name in any part. + followed with no restart; whole (bus module), a feature branch asking a local name with other settings than the + default branch's is said `drifted` while the default branch's are planned; the code review finds no branch's name + in any part. ## The bus module's plans @@ -289,8 +335,10 @@ line. Questions that an earlier line already answers have no line here: develop a seat whose verbs come only from an included part gives its holder a sent store, and one including only bodies gives none. - §538 **The bus module plans again when the bus-state entries change.** The bus module compares, for each module, - the bus-state entries the catalog graph entry carries with those it last planned for that module, kept in its own - state, and plans again only when they differ; it never compares the manifest's digest (§506), which changes with + the bus-state entries the catalog graph entry carries with those it last planned for that module, kept in its old + `state` section `plans` (the bus module is a foundation part), one key `` holding the entries as planned + and their digest, written after each plan and deleted when the module is in no watched branch's newest revision, its + entries then orphans that `orphans` lists; its tool `nats_plans` reads it. It plans again only when they differ; it never compares the manifest's digest (§506), which changes with any field. Why: what it plans is those entries alone (develop question on §433 with §506). How it is checked: whole (bus module), a manifest changed outside `requires` plans nothing, and a changed option plans the difference. @@ -353,12 +401,17 @@ line. Questions that an earlier line already answers have no line here: develop lead-47's terminal: approval is a step whose status each module's registration sets, never a gate of its own, so an Approve given before G counts, once G is delivered, as that pull request's `approval` step and nothing else is special. At G's switch the gate says, for each `merges` key the old bridge had not performed, `approval-carried` on - `mesh.seat.mesh-delivery.event.approval-carried` (`repository`, `pull`, `commit` the head the Approve named, - `approver` the operator, `at`), declared in the seat's definition, and deletes the key once the bus acknowledges - it. The runner keeps each in its old `state` section `pipelines` as `carried-approval..`, - holding the commit, and takes it as the result `passed`, caller the operator, of the `approval` step of the first - pipeline or run at that commit, then deletes it; it deletes it unused when the seat `git` says the pull request - `pull-closed` or `pull-merged`. Where the module's policy makes `approval` `required`, a pull request whose head + `mesh.seat.mesh-delivery.event.approval-carried..` (`repository`, `pull`, `commit` the head the + Approve named, `approver` the operator, `at`), declared in the seat's definition with `keep` 90 days (§396). The gate + keeps its `merges` keys, read no more, until §97's removal, so a revert's gate of before G finds them. The runner + hears the event through a durable consumer made to start from the stream's first message, so a runner started + after the switch hears every one, and keeps each in its old `state` section `pipelines` as + `carried-approval..`, holding the commit. It takes it as the result `passed`, caller the + operator, only of the `approval` step of the delivering pipeline whose source revision takes in that pull + request's merge, as `take-in.taken`'s entries name it (each entry carrying the repository and the pull request of a + `merged` message, §493, §509), and only when the pull request's head merged is that commit; never in a run on + request and never in another pipeline at the same commit; then it deletes the key, and deletes it unused when the + seat `git` says the pull request `pull-closed`. Where the module's policy makes `approval` `required`, a pull request whose head moved after the Approve asks a new approval, as any step whose input changed does (an `approval` step is never carried, ADR 0666 §8); where it is `optional` or `bypass`, the carried Approve holds nothing. The merge itself is any merge (`merge-pull`, ADR 0666 §11): the bridge performs no carried act, the gate says no `wait-held` with @@ -367,8 +420,9 @@ line. Questions that an earlier line already answers have no line here: develop `cancelled`, `not asked`) or its pipeline is superseded or cancelled, so no approval wait stands without an end. This replaces §510 and §608, and `merge` leaves §498's list. Why: the operator's answer ("you're mixing in step and approval gates, this should be configurable"); the event, the key and the deletion are the design's (round 3, - health and events' R7). How it is checked: whole (gate, runner), a carried Approve of a head passes that head's - `approval` step with the operator as caller; with `approval` `required`, a head moved after the Approve asks a new + health and events' R7). How it is checked: whole (gate, runner), a carried Approve of a head passes the `approval` step + of the pipeline taking in that pull request's merge, with the operator as caller, and no run on request at that + commit; a runner started after the switch takes it all the same; with `approval` `required`, a head moved after the Approve asks a new approval; a pull request closed deletes its carried key; every `approval` wait-held in the whole test has its `wait-ended`; the code review finds no `merge-waits`. @@ -379,11 +433,13 @@ line. Questions that an earlier line already answers have no line here: develop ask under its name. The grants part grants each asker publish on `mesh.seat..accept..` (§600) for every queue of the seat on which a kind it is a caller of is taken, and the holder, which takes the asker from the subject (§481), refuses an ask whose body names a kind that asker is not a caller of, answering `error` naming - the kind and the asker, and ends it with `step-ended` `failed`. This amends §338 and §600. Why: §600 puts the queue + the kind and the asker, and ends it with `step-ended` `failed`. On main's subject `accept.`, which names no asker and is granted to + the checker and the builder alone (§600), the holder takes only the kinds `build` and `run` and refuses any other, + `step` among them, as above. This amends §338 and §600. Why: §600 puts the queue in the subject and the kind in the body, so a subject's grant alone cannot hold a caller to its kinds (round 3, security and upgrade, seams and tools: R1). How it is checked: unit (Go build module), a `step` published from the indexer's account is answered `error` naming `step` and `indexer`; unit (grants part), each asker's grants are the - queues of its kinds. + queues of its kinds; unit (Go, Node and image build modules), a `step` on main's subject is answered `error`. - §544 **Named asks have a worker of their own.** For each queue the grants part makes, beside main's worker `SEAT___worker` filtered to main's `mesh.seat..accept.` alone, a second worker `SEAT___named_worker` filtered to `mesh.seat..accept..*`; the two filters do not overlap, @@ -392,16 +448,23 @@ line. Questions that an earlier line already answers have no line here: develop part runs takes no named ask. This replaces §601's "until every node's holder of a build seat is of G the grants part's worker keeps main's one filter", and with it the edge from the seats' holders to the grants part, which no manifest fact gives: the order is the asker's `invokes` of `.accept.` (kind 6, from the seat's holders - to the asker, §601), and the grants, written for a catalog revision before any want of it (*A credential and a - composition before the want*), exist before the asker runs. A named ask waits on its worker while no holder of G + to the asker, §601), and the grants part comes before every asker as the acting part of the step `grant` on + `every` (§203): each entry of a revision waits for the grants part's new version to act for it (§266), so no asker + of G runs before a grants part of G has made its named worker and its grant. A holder of G pulls main's worker + alone until its membership names the named worker, which the grants part writes as it binds every consumer + (§386), and says meanwhile, in its journal and its read tool, `named worker not bound`. A named ask waits on its worker while no holder of G runs, said by `build-capacity-waits` after 60 minutes, which corrects §483's "waits on the work queue, whose - consumer of before G takes only main's subject". At a revert, which reverts the askers before the grants part (ADR - 0656 §4), the grants part of G, when it no longer hears any asker of G, purges the subjects - `accept..*` of the stream and deletes the named workers, saying each in its journal; an ask is transport, + consumer of before G takes only main's subject". At a revert, the grants part of G takes its view of the revert's catalog revision before + it is itself reverted, since it acts for every entry of it (§203); at the first revision in which no entry `invokes` + any `.accept.`, it purges the subjects `accept..*` of the stream and deletes the named workers, + within its granting bound of 10 minutes (§266), saying each in its journal. Where that did not run (the grants part + stopped first), its tool `grants_named_purge`, of class `act` with §391's default callers and written in the + hand-act log, does it by hand; an ask is transport, not kept data (as §59 has a sent object). Why: one consumer per filter, and a filter that never moves cannot be pulled by a holder that does not read it (round 3, R2, the security reviewer's form). How it is checked: whole (grants part, Go build module), a holder of before G started after the grants part of G takes main's asks and no - named one; a revert leaves no named worker and no named ask. + named one; a revert leaves no named worker and no named ask; a holder of G before its binding pulls main's worker + and says `named worker not bound`. - §545 **The tool runner's artifact declares the words it states.** The artifact entry of the tool runner's manifest takes `states`, the capability words (§488) it states; the manifest validator refuses a package artifact's `calls` word (§605) that no artifact of the revision `states`, naming the word, and the delivery plan's edge of §605 runs to @@ -410,32 +473,39 @@ line. Questions that an earlier line already answers have no line here: develop reads the stated words at its own start; no event is heard and nothing polls. This amends §605's "each time the tool runner of its node starts". Why: the words are stated only at run time, and the plan needs them in a manifest (round 3, seams and tools: R3). How it is checked: unit (manifest validator), a `calls` word nothing `states` - refused; unit (delivery plan), the edge to the module stating the word. + refused; unit (tool runner), the words it states are exactly its manifest's `states`; unit (delivery plan), the edge to the module stating the word. - §546 **When the builder forgets a node's toolchain.** The builder deletes its `toolchains-heard` key `.` (§604) when the node-engine seat's `running..` says `stopped` (§365) for the module whose catalog graph entry claims the build seat of that language (its `acts` rule `build` with `on` naming - it, §535), heard through the durable consumer it already has of that seat's `running`; never on a placement, which + it, §535), heard through its consumer of that seat's `running`, read as §556 has it; never on a placement, which is the seat `mesh-module-assigner`'s state. The state is an old `state` section of the builder, a foundation part, one key per node and language, about 12 keys of about 0.3 KiB each, capped at 1 MiB, so it adds no objects store to §607's count (§58 plans one only for a state of the template); `builder_toolchains` reads it and - `builder_toolchains_drop` deletes one key, with why, written in the hand-act log of the seat `mesh-hand-acts`; a + `builder_toolchains_drop`, of class `act` with §391's default callers (`operator`, `sessions`), deletes one key, + with why, written in the hand-act log of the seat `mesh-hand-acts`; a holder says the key again at its start and on each change (§492). This amends §604's "when the node leaves the catalog's placements". Why: the builder reads no other module's state (ADR 0596 §2; round 3, seams and tools: R4). How it is checked: whole (builder), a build module stopped on a node deletes that node's key, and the drop tool - deletes one key and logs it. -- §547 **The sum of the bus's caps is read against its limit.** The bus module reads the bus's storage limit from the - server's account information at its start and at each plan, sums the cap of every entry it plans (states, objects - stores, sent stores, streams, logs) and of every old section the catalog graph's entries declare, and shows both in - its `list` and its read tool. A plan whose sum passes the limit is refused, naming the sum and the limit, said as - its `refused` (ADR 0595 §6), whether or not the server reserves a cap at creation. When G's check measures that the - server reserves a cap when a store is made (§607), each `-objects` is planned with 16 MiB instead of 64 MiB - (about 2.4 GiB for 150 states), a state needing more naming `objects-max-mib`. This amends §607. Why: the limit is a + deletes one key and logs it; a drop by another caller is refused by the bus. +- §547 **The sum of the bus's caps is read against its limit.** Measured on 2026-11-11 against an embedded bus + (nats-server 2.11.17): an account's storage limit reserves each stream's maximum size when the stream is made (a + second stream of 64 MiB under an account limit of 100 MiB is refused, "insufficient storage resources"), and a + server's own limit does not. So each `-objects` is planned with 16 MiB (about 2.4 GiB for 150 states), a + state needing more naming `objects-max-mib`, and no later measure is waited for. The bus module reads the account's + and the server's storage limits at its start and at each plan, sums the cap of every entry it plans (states, objects + stores, sent stores, streams, logs) and of every old section the catalog graph's entries declare, and shows the sum and + both limits in its `list` and its read tool. A plan whose sum would pass the account's limit is refused, naming the + sum and the limit, said as its `refused` (ADR 0595 §6): the plan refused is the one that adds or raises a cap, and + the caps already made stay as they are. With no account limit (unlimited) nothing is refused and the sum is shown + against the server's limit. This amends §607's 64 MiB and its measure in G's check. Why: the limit is a fact of the server, read rather than written in the design, and a sum over it fails loudly (round 3, running and - testing: R5). How it is checked: whole (bus module), a fixture server with a small limit refuses a plan past it, - naming the sum; the sum and limit are in `list`. -- §548 **The forge module writes `seen` only from the runner's steps.** §487's `seen..` is written - only from a `step-ended` whose asker is `runner`; the `step-ended` of any other ask (§602) carries `asker`, `ask`, - `node`, `outcome` and `at` and nothing of a pipeline, and the forge module writes nothing from it. This amends §487. + testing: R5). How it is checked: whole (bus module), a fixture bus with a small account limit refuses the plan + that would pass it, naming the sum, and keeps the caps made; with no account limit nothing is refused; the sum and + the limits are in `list`. +- §548 **The forge module writes `seen` only from a pipeline's steps.** §487's `seen..` is written only + from a `step-ended` that carries a pipeline's `commit` and `fingerprint`, which a pipeline's step carries; the + `step-ended` of any other ask (§602) carries `asker`, `ask`, `node`, `outcome` and `at` and neither, so the forge + module writes nothing from it and names no asker. This amends §487. Why: since §602 every ask says `step-ended`, and a builder's build is no run of a commit (round 3, health and events: R6). How it is checked: unit (forge module), a builder's failed build writes no `seen`; whole (watchman), a `stuck` raised for an ask clears on its late `step-ended`. @@ -454,16 +524,17 @@ line. Questions that an earlier line already answers have no line here: develop ## The operator's answers of 12:41 CET - §551 **A restore point is removed after 90 days, never the newest.** The operator, 2026-11-11 12:41 CET at lead-47's - terminal. The bus module removes a restore point's directory (§516) 90 days after its `at`, as long as delivery + terminal. The recovery module removes a restore point's directory (§516) 90 days after its `at`, as long as delivery records are kept, unless it is the newest point of its module and kept data, which it never removes; it checks at - its start, once a day and at each `snapshot-kept`, on its own clock and its own directory. Each removal is said as - the seat `mesh-broker`'s event `point-removed` (`module`, `kept`, `form`, `fingerprint`, `snapshot`, `at`), declared - in its definition; the backup holder consumes it and deletes that point's key in `bus-points` (§517), and its - repository's copy goes by the repository's own rotation (ADR 0235 §4). Data set aside is removed only by - `discard-aside` (§516). This completes §516 and answers issue 1921's note on the retention of restore points. Why: - the operator's answer; the event, the checks and the holder's deletion are the design's. How it is checked: whole - (bus module, backup holder), a point older than 90 days is removed and said and its key deleted, and the newest - point of a kept data, older than 90 days, stays. + its start, once a day and at each `snapshot-kept`, on its own clock and its own directory, as one of its acts, never + beside a `restore-kept` (§554). Each removal is written in the hand-act log of the seat `mesh-hand-acts` and said as + `mesh-recovery`'s event `point-removed` on `mesh.seat.mesh-recovery.event.point-removed..` (`module`, + `kept`, `form`, `fingerprint`, `snapshot`, `at`), kept 90 days (§396). The backup holder's copies go by its + repository's own rotation (ADR 0235 §4). Data set aside is removed only by `discard-aside` (§516). This completes + §516 and answers issue 1921's note on the retention of restore points. Why: the operator's answer; the event, the + checks and their order are the design's. How it is checked: whole (recovery module), a point older than 90 days is + removed and said, the newest point of a kept data older than 90 days stays, and a removal due during a restore runs + after it. - §552 **The checker may drop its own asks until its removal.** The operator, 2026-11-11 12:41 CET at lead-47's terminal: the build seats' verb `drop` (§338) gains the checker as a caller until its entry is removed with its buckets and bus account (ADR 0667 §12), as `run` has it, so the validator accepts the checker's manifest's diff --git a/03-DESIGN/01-to-be/47-milestones/G-14-recovery-module-and-review.md b/03-DESIGN/01-to-be/47-milestones/G-14-recovery-module-and-review.md new file mode 100644 index 00000000..c57a01fc --- /dev/null +++ b/03-DESIGN/01-to-be/47-milestones/G-14-recovery-module-and-review.md @@ -0,0 +1,128 @@ +--- +layer: to-be +status: in-progress +design: 47 +milestone: G +updated: 2026-11-11 +decisions: + - "ADR 0667" + - "ADR 0666" + - "ADR 0660" + - "ADR 0656" + - "ADR 0636" + - "ADR 0626" + - "ADR 0619" + - "ADR 0618" + - "ADR 0614" + - "ADR 0596" + - "ADR 0595" + - "ADR 0564" + - "ADR 0505" + - "ADR 0404" + - "ADR 0363" + - "ADR 0240" + - "ADR 0235" +--- + +# 47, milestone G: the recovery module, and the review of §516 to §553 + +One file of milestone G's part of [design 47](../47-delivery-from-commit-to-delivered.md); the part opens at +[G.md](G.md). Its design lines are numbered across every file of the part and cited as "design 47 G §n". + +*§554 and §555 are the operator's answer of 2026-11-11 at 12:55 CET at lead-47's terminal. §556 to §558 answer the +four reviewers' round 1 of [G-12](G-12-develop-answers-fourth.md) (§516 to §553); the rest of that round's findings +are written into the lines of G-12 they name.* + +## The recovery module + +- §554 **The recovery module takes, sets aside and restores kept data on the live bus.** The operator, 2026-11-11 + 12:55 CET at lead-47's terminal: a module of its own, on a seat of its own, with an account of its own, owns + snapshotting, setting aside and restoring a module's kept data on the live bus, each act in the hand-act log; the + bus module keeps no restore permission (ADR 0626 §1 stays as it is), the snapshot account stays limited to the + nightly dump (ADR 0626 §2), and the backup holder only stores copies. The record of the recovery module, which + lead-47 proposes to the knowledge base for the operator's Approve, decides the account and its grants; this line + is written to it. + - **Its seat and module.** The seat `mesh-recovery`, held once for the mesh, is defined in mesh-contracts beside + `mesh-broker`, and the module `mesh-recovery`, in mesh-catalog, claims it; its glossary word is the *recovery + module*. It keeps no kept data on the bus, its one store being its data directory (§516), so it is no foundation + part (ADR 0619 §1) and waits on no bus module. + - **Its verbs and callers.** `snapshot-kept` and `restore-kept` (with `aside-only`), class `act`, are called by the + node agent applying a want (`seat:node-engine`, §296, §297) alone; `restore-kept` with `discard-aside` by + `operator` alone; `kept-points` (§517), class `read`, takes §391's default. Each act is written in the hand-act + log of the seat `mesh-hand-acts` first, with the caller the bus proved and `why`; an act whose log write fails + does nothing and answers `error`, naming the log (as §480). + - **Its account.** Composed by the grants part from the seat's definition as the record decides: the bus's stream + snapshot and restore, a stream's deletion and a consumer's creation, only for the bus objects of a module's kept + data (the names a binding or an old section of a catalog graph entry gives), never for the mesh's own streams nor + for any other object; it publishes only on its own seat's subjects and reads no module's data otherwise. + - **Its events.** On `mesh.seat.mesh-recovery.event.`: `ended..` (a point taken, with `outcome` `done` + or `failed` and its `error`), `restored..` (likewise), `set-aside..` (with the aside's + `id` and `size`) and `point-removed..` (§551), each filled as §523 has it and kept 90 days (§396). + The node agent's wait for `ended` or `restored` is bounded at 90 minutes (30 for a nightly dump it waits behind, + 30 for a point brought back from the backup, 30 for the restore), past which its have says the wait, naming the + seat, and the watchman's `delivery-waits` raises it. This amends §405's 30 minutes. + - **One act at a time, and the nightly dump.** The recovery module runs its acts (`snapshot-kept`, `restore-kept` + and a removal of §551) one at a time, the next waiting within its own bound. The holder of `node-backup` says + `item-started..` and `item-ended..` for every item it backs up, declared in its seat's + definition; the recovery module consumes those of the bus module's item and starts no `restore-kept` while the + nightly dump of the bus runs, so the dump never meets a stream a restore removed (ADR 0235 §13). A `snapshot-kept` + may run beside the dump, both paced as ADR 0235 §10 has it, so the server has at most two snapshots at once. + - **Its order and its tools.** The node-engine's `invokes` of `mesh-recovery`'s verbs gives the served-verb edge + (kind 6) from the recovery module to the node-engine, so §298's "the new versions of the `node-backup` holder and + of the bus module run before any module whose kept data's form changes is moved" reads "the new version of the + recovery module". Its tools are named `recovery_`, and its declared health check calls `kept-points`. + + Why: the operator's answer; the seat's placement in mesh-contracts, its callers, its events, its bound and its + order are the design's, from §296 to §303 and §405 as they were. How it is checked: test 36 with the recovery + module; unit (grants part), the recovery module's account holds the grants the record names and no other, and the + bus module's and the snapshot account's hold no restore; whole (recovery module), a `restore-kept` asked while the + bus's item is backed up starts after `item-ended`; an act whose hand-act log write fails does nothing. +- §555 **What moves to the recovery module.** §296 asks `mesh-recovery`'s `snapshot-kept` and waits for its `ended`, + never the seat `node-backup`; §297 asks its `restore-kept` and waits for its `restored`; §302's `backed-up` lists + the recovery module's data item as any item, and the restore points and asides are listed by `kept-points` (§517), + `discard-aside` being `restore-kept`'s; §303 and test 36 name the recovery module where they named the backup + holder and the bus module; §337's and §405's callers `holder:node-backup@same-node` go with the two verbs of + `mesh-broker`, which gains none; the seat `node-backup` gains nothing for kept data on the bus: its `aside` and + `discard-aside`, the arguments `kept`, `form` and `fingerprint` of its `now` and `restore`, and its events + `ended...` and `restored...` go from its definition in mesh-contracts, and + it gains only `item-started` and `item-ended` (§554); §450's `aside` answer is `restore-kept`'s with `aside-only`; + §406's catch-up is recorded in the point as before. Why: one owner of the acts (§554). How it is checked: unit + (definitions), `node-backup` and `mesh-broker` hold none of the moved verbs; test 36. + +## The review of §516 to §553, round 1 + +- §556 **A part that follows a stream kept last per subject reads it whole at each start.** A durable consumer goes on + from its acknowledged position after a restart, so a part that acts on what it heard from a stream kept last per + subject, and keeps no state of it, reads at each of its starts every subject's last message through an ordered + consumer it makes for that one read, then follows its durable consumer from its acknowledged position, an event + older by `at` than what that read gave changing nothing. Until the read ends the part acts on none of those events + and its `_first_views` names the stream `waiting`. So read: the node-engine seat's `running`, `composed`, + `wait-held` and `wait-ended` (the last two kept on the stream `running` is kept on, last per subject, a node's + module's newer one by `at` standing), by the assigner (§520), the condition keeper (§541), the builder (§546), the + node-engine and the grants part (§199); and `mesh-healers`' `attempt-changed` by the keeper (§541). This amends + §199's "each part's durable consumer starts from the last message per subject". Why: what a part heard must not + depend on when it last restarted (review round 1, health and events 2, with running and testing 3). How it is + checked: whole (assigner, keeper), a restarted assigner holds no want it did not hold before, and a restarted + keeper holds the healers' attempts it held. +- §557 **A whole test's bounds are settings.** Every bound a whole test waits on (§535's five minutes, §534's 30 + minutes, §544's 60 minutes, §540's two hours, §551's 90 days, §554's 90 minutes) is a setting of the part that + times it, read at its start, its default the design's value, so a whole test sets it in seconds and never waits on + real time. The bus module's and the recovery module's whole cases in these lines are cases of each module's one + whole test against the real bus, within the test budget. Why: a whole test that waits hours runs never (review + round 1, running and testing's notes). How it is checked: the code review; each whole test of these lines runs in + under a minute. +- §558 **A field of a served verb's answer is an edge of kind 6.** Kind 6 (§374) orders a module after a version that + serves a verb or a word it uses, and a verb's presence alone says nothing of a field its answer gained, so a + consumer relying on a new field of a served answer could run before the provider's new version and fall back in + silence (the grants part's first view relies on `binding` in `mesh-broker`'s `list`). An `invokes` entry therefore + takes a long form, `{verb, fields}`: `verb` as today (`seat:.`), `fields` the names of its answer's + fields the consumer relies on. The manifest validator refuses a field the seat's definition does not give that + verb's answer, naming it. The delivery plan derives the edge of kind 6 from the holders of that seat to the + consumer, held until the holder runs a build whose catalog graph entry's seat definition (§194) gives the verb's + answer every named field; §374's entry of kind 6 records the verb and its fields, and `outdated` (§375) says it + behind when the delivered version no longer gives them. The grants part's manifest names `mesh-broker`'s `list` + with `fields: [binding]`, and every part of G relying on a field G adds to a served answer names it so in G's + change. Why: the module graph orders a consumer after what it relies on, and a field is part of the contract + (lead-47's question from the operator's "the module graph is for this", 2026-11-11). How it is checked: unit + (manifest validator), a field the definition lacks refused; unit (delivery plan), the grants part ordered after + a bus module whose definition gives `binding`, and not before. diff --git a/03-DESIGN/01-to-be/47-milestones/G-4-revision-view-and-steps.md b/03-DESIGN/01-to-be/47-milestones/G-4-revision-view-and-steps.md index 6ce9f057..f69a6cd9 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-4-revision-view-and-steps.md +++ b/03-DESIGN/01-to-be/47-milestones/G-4-revision-view-and-steps.md @@ -105,7 +105,7 @@ One file of milestone G's part of [design 47](../47-delivery-from-commit-to-deli asked it (1921's note of 2026-11-07: "the indexer reads its seat definitions at start"). How it is checked: in the indexer's whole-module test, started with a definition on a contracts trunk that no event announced, the first manifest claiming that seat is judged with it. -- §199 **Who holds a seat is applied by one rule, from what each node runs.** The node-engine and the grants part decide a +- §199 **Who holds a seat is applied by one rule, from what each node runs.** (§556) The node-engine and the grants part decide a seat's pass by mesh-sdk's `SeatsToPassOn` (ADR 0660 §3), reading the catalog graph's entry of the build each assignment runs, never a manifest (ADR 0660 §2; §442). What each node runs is heard, never asked: each part keeps it from the node-engine seat's events `running..` (*What each step says*), through a durable consumer of its own declared in its `consumes`, diff --git a/03-DESIGN/01-to-be/47-milestones/G-5-acting-parts-check-and-kept-data.md b/03-DESIGN/01-to-be/47-milestones/G-5-acting-parts-check-and-kept-data.md index 88526d31..f5ded827 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-5-acting-parts-check-and-kept-data.md +++ b/03-DESIGN/01-to-be/47-milestones/G-5-acting-parts-check-and-kept-data.md @@ -329,7 +329,7 @@ later change of a kept data's form, not as a case of this milestone. changes how it writes that data, or stops declaring it; a declaration naming none reads as form 0. The declaration alone cannot tell a changed layout of the same bucket, so the holder says it. Only kept data whose `form` changes is backed up and restored, so a part's other kept data, and every decision it holds, is never set back. -- §296 **It is backed up before the module moves.** When a want moves a module whose new manifest declares a kept data at +- §296 **It is backed up before the module moves.** (§555) When a want moves a module whose new manifest declares a kept data at another `form` than the running one, or no longer declares it, the node agent applying it first asks the seat `node-backup` of the node holding `mesh-broker` for a restore point of that kept data, and waits for its event `ended`, as for any backup before the apply. The bus module's account takes it, as it takes the bus's snapshots @@ -337,14 +337,14 @@ later change of a kept data's form, not as a case of this milestone. before, and it holds the kept data whole, the position of each durable consumer the module reads with, and the catch-up tool the newer manifest declares beside that kept data. A failed backup holds that module as any failed backup does. -- §297 **A revert restores it before the reverted build starts.** When a want takes a module back across a change of a +- §297 **A revert restores it before the reverted build starts.** (§555) When a want takes a module back across a change of a kept data's `form`, the node agent applying it stops the newer build, then asks that seat's verb `restore` (§405) for the restore point of that module, kept data and form, waits for the seat's event `restored`, and only then starts the reverted build. The node agent that restores is the one running when the revert starts. As every restore is, it is brought back beside the live data and never over it: the live kept data of the newer form is set aside under a name of its own, and only the operator removes it, so nothing written since is destroyed. The consumer's position is set back with it, so the reverted build hears everything since as after any restart. -- §298 **The order of a delivery that crosses a form change** is the module graph's (*Order without a person*), with three +- §298 **The order of a delivery that crosses a form change** (§554) is the module graph's (*Order without a person*), with three edges the assigner, which writes every want, keeps: - the new versions of the `node-backup` holder and of the bus module run before any module whose kept data's form changes is moved, since the restore point needs both; @@ -391,12 +391,12 @@ later change of a kept data's form, not as a case of this milestone. (for milestone G, its delivery records and its graph revisions) is set aside and not read again; an Approve asked and answered since is asked again, and one the gate asked before still reads its answer. The revert's Approve ask says this in those words, naming each kept data restored. -- §302 **Its tools.** (§517) The seat `node-backup`'s `backed-up` lists each module's restore points of its bus data, with the +- §302 **Its tools.** (§517, §555) The seat `node-backup`'s `backed-up` lists each module's restore points of its bus data, with the kept data, the form and the fingerprint each was taken before, and the data set aside at each restore, with its size; `restore` restores one; `discard-aside` removes data set aside by a restore, only on the operator's call, and writes the act, who and why in the hand-act log of the seat `mesh-hand-acts`. `backed-up` and `restore` exist on the seat today and gain the module's kept data on the bus as an item; `discard-aside` is new. -- §303 **How it is checked.** Whole (node-engine, backup holder, bus module): a want moving a module across a `form` +- §303 **How it is checked.** (§555) Whole (node-engine, backup holder, bus module): a want moving a module across a `form` change, and one moving a module whose kept data declares no `form` to one that does, each take a restore point first, keyed by module, kept data, form and fingerprint; a want taking it back restores it beside the live data and the consumer's position before the reverted build starts, and the data set aside is listed with its size and diff --git a/03-DESIGN/01-to-be/47-milestones/G-7-answers.md b/03-DESIGN/01-to-be/47-milestones/G-7-answers.md index 0ebd9f32..6a3342b9 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-7-answers.md +++ b/03-DESIGN/01-to-be/47-milestones/G-7-answers.md @@ -161,7 +161,7 @@ record it follows from, and replaces the builder's own choice where one was made definitions in `mesh-connectivity-contracts` names `invokers`, `persons` and `sessions` in this milestone, since sessions and persons call them today through the mesh MCP server. How it is checked: unit (definition reader), each form accepted and any other entry refused; the grants part's table test grants each form. -- §337 **The bus module's restore points.** The verbs the holder of `node-backup` on the bus module's node calls to take +- §337 **The bus module's restore points.** (§555) The verbs the holder of `node-backup` on the bus module's node calls to take and restore a module's kept data on the bus name `holder:node-backup@same-node`. Why: only that holder takes them (*A backup before the apply: kept data on the bus*). How it is checked: a call from another node's `node-backup` holder is refused by the bus. @@ -406,7 +406,7 @@ record it follows from, and replaces the builder's own choice where one was made at G's delivery (design 51, *Tagged versions*). Why: the operator, 2026-11-10 about 21:40 CET, no custom versions for now, a tagged version being one; a run delivers nothing (ADR 0666 §18), so it needs no version; the digest's source and the wait are the design's choice, round 2 of the review by architectural domain, 2026-11-10. How it is checked: test 103. -- §374 **A release records, per edge, the part it was built against or delivered after.** One version per module +- §374 **A release records, per edge, the part it was built against or delivered after.** (§558) One version per module stays (§52), and an edge is judged by the digest of the part it points to. The catalog records on each released version (§349), in its key of `versions`, one entry per catalog-graph edge of the module's artifacts (the six kinds of *Dependencies between artifacts*), a lockfile per released version. An entry names the kind, whether the edge diff --git a/03-DESIGN/01-to-be/47-milestones/G-9-prepare-answers.md b/03-DESIGN/01-to-be/47-milestones/G-9-prepare-answers.md index b24b1345..8f006242 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-9-prepare-answers.md +++ b/03-DESIGN/01-to-be/47-milestones/G-9-prepare-answers.md @@ -116,7 +116,7 @@ line says so.* ### Kept data on the bus, restore points and the reader release -- §405 **The bus module's restore points and node-backup's verbs.** (§516, §517) The seat `mesh-broker` gains `snapshot-kept` and +- §405 **The bus module's restore points and node-backup's verbs.** (§516, §517, §554, §555) The seat `mesh-broker` gains `snapshot-kept` and `restore-kept`, of class `act`, their callers `holder:node-backup@same-node` alone (§337): the bus module's account is the one that moves bus data, so `restore-kept` sets the live kept data aside under a name of its own and then restores the restore point beside it, with the argument `aside-only` it sets the live data aside and restores @@ -132,7 +132,7 @@ line says so.* choice, one answer to three preparers (Q8, S9, Q-H6, C19; lead-47's ruling, 2026-11-11). How it is checked: test 36; a call of `snapshot-kept` from another node's `node-backup` holder, and a `discard-aside` from any caller but the operator, are refused by the bus; a failed restore says `restored` with `failed` and its error. -- §406 **A catch-up names its tool and arguments.** (§516) A restore point records the catch-up tool and its arguments whole +- §406 **A catch-up names its tool and arguments.** (§516, §555) A restore point records the catch-up tool and its arguments whole (§299), and the fingerprint of the newer build it was taken before; the grants part derives the grant of that tool to `holder:node-engine@same-node` from the catalog graph's entry of that recorded fingerprint (§443). Why: the design's choice (Q-H7). How it is checked: whole (node-engine), a restore runs its recorded catch-up, granted from the @@ -380,10 +380,10 @@ line says so.* - §449 **The grants part's fill events say what `mesh-wants`' say.** The seat `mesh-grants`' events `filling` and `room` carry the same body as `mesh-wants`' fill events: `store`, `bytes`, `cap` and `at` (§181, §331). Why: develop question D-4, 2026-11-11. How it is checked: the seat's fixtures, against `mesh-wants`'. -- §450 **`ask-again` and `aside` answer.** `operator-channel`'s `ask-again` (§394) takes `key`, the ask's, and answers +- §450 **`ask-again` and `aside` answer.** (§555) `operator-channel`'s `ask-again` (§394) takes `key`, the ask's, and answers `state`, the ask's state after it, or `error`; `node-backup`'s `aside` (§405) answers `aside`, the name the data was set aside under, or `error`. Why: develop question D-5, 2026-11-11. How it is checked: the seats' fixtures. -- §451 **The forms of `owner`, `waiting-on` and a catch-up.** A finding's `owner` and `waiting-on` (§434, §439) are each +- §451 **The forms of `owner`, `waiting-on` and a catch-up.** (§540) A finding's `owner` and `waiting-on` (§434, §439) are each a string: a person or a session as who@node, or a seat's name. A restore point's catch-up (§406) is `{tool, arguments}`: `tool` an address as the mesh MCP server names one, `arguments` an object. Why: develop question D-6, 2026-11-11. How it is checked: unit (mesh-conditions' contract, node-backup's contract), each form read and another diff --git a/03-DESIGN/01-to-be/47-milestones/G.md b/03-DESIGN/01-to-be/47-milestones/G.md index 3e8bd514..f96ae217 100644 --- a/03-DESIGN/01-to-be/47-milestones/G.md +++ b/03-DESIGN/01-to-be/47-milestones/G.md @@ -23,6 +23,7 @@ decisions: - "ADR 0641" - "ADR 0639" - "ADR 0636" + - "ADR 0626" - "ADR 0635" - "ADR 0633" - "ADR 0625" @@ -76,7 +77,7 @@ file on 2026-11-10, and is cited as "design 47 G §n". A sentence that holds a d stayed in the main file, and G builds what it says from G's delivery. A section named in italics is in this part, or, when it is not here, in the main file. -The part is thirteen files, each read whole, its design lines numbered across them: +The part is fourteen files, each read whole, its design lines numbered across them: - [G-1-pipelines-registration-and-steps](G-1-pipelines-registration-and-steps.md): pipelines per module: registration, watched branches, steps and policy, what holds a delivery, approval, groups, dependencies, a publish. - [G-2-pipelines-sizes-checks-and-tools](G-2-pipelines-sizes-checks-and-tools.md): pipelines per module: any size, the pipeline makes the check, who may say what, health, recovery, tools, kept data, how G itself is delivered, the choices under ADR 0666 and ADR 0667. @@ -91,3 +92,4 @@ The part is thirteen files, each read whole, its design lines numbered across th - [G-11-develop-answers-review](G-11-develop-answers-review.md): the operator's answers of 2026-11-11 noon, the review of §456 to §479, and develop's next questions (§480 to §515). - [G-12-develop-answers-fourth](G-12-develop-answers-fourth.md): the operator's answers of 2026-11-11 noon, round 3's leftovers and develop's fourth batch of questions (§516 to §553). - [G-13-review-round-3](G-13-review-round-3.md): the review of §480 to §515, rounds 2 and 3 (§600 on). +- [G-14-recovery-module-and-review](G-14-recovery-module-and-review.md): the recovery module, the operator's answer of 2026-11-11 12:55 CET, and the review of §516 to §553 (§554 to §558).