Put restoring kept data on a recovery module, and answer round 1 of the review of batch 4

The operator chose a module of its own for snapshotting and restoring a module's
kept data on the live bus, so the bus module and the snapshot account keep their
narrow grants; the reviewers' musts are written into the lines they name, and the
new lines go to G-14 so G-12 stays within a part's size.

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ
This commit is contained in:
2026-11-11 12:57:01 +01:00
parent e504dac534
commit 6d0506402d
8 changed files with 342 additions and 135 deletions
+6
View File
@@ -974,6 +974,12 @@ becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252, 0283, 0564, 0668.
at one moment; **restore** — bringing it back beside the live data, never over it.
- **stream snapshot** — the bus's own backup of each stream, taken by the module holding `mesh-broker`
(ADR 0235).
- **recovery module** — the holder of the seat `mesh-recovery`, held once for the mesh: under an account of its own it
takes a restore point of a module's kept data on the live bus before the module moves across a `form` change, sets
the live data aside and restores the point at a revert, and lists and removes restore points; the bus module and
the snapshot account restore nothing into the live bus, and the backup holder only stores copies (design 47 G §554).
**aside** — live kept data a restore set aside, removed only by the operator's `discard-aside`.
*Not:* ~~restorer~~, ~~restore account~~
- **disk readings** — the size of each data item a node's modules declare, measured by its `measure` and kept
with its largest size in seven days by disk-load on that node, in its own state, one key
`<node>.<module>.<item>` written only when it changes; the holder of `node-backup`, mounts, nfs-server and the
@@ -10,6 +10,7 @@ decisions:
- "ADR 0660"
- "ADR 0656"
- "ADR 0636"
- "ADR 0626"
- "ADR 0619"
- "ADR 0618"
- "ADR 0614"
@@ -17,6 +18,7 @@ decisions:
- "ADR 0595"
- "ADR 0564"
- "ADR 0505"
- "ADR 0404"
- "ADR 0363"
- "ADR 0240"
- "ADR 0235"
@@ -32,63 +34,73 @@ answer what the four reviewers found in round 3 of [G-13](G-13-review-round-3.md
of its own after it. Every other line answers a question
develop met after [G-11](G-11-develop-answers-review.md), of internal form only (playbook 07, step 0b), each the
design's choice with its reason. What a question of another kind leaves open waits for the operator and is said in the
line. Questions that an earlier line already answers have no line here: develop reads that line.*
line. The operator's answer of 12:55 CET (the recovery module, §554 and §555) and the
review of §516 to §553, round 1 (§556 to §558), are in [G-14](G-14-recovery-module-and-review.md); the findings of
that round are written into the lines they name, here. Questions that an earlier line already answers have no line here: develop reads that line.*
## Kept data on the bus: restore points and what is set aside
- §516 **A restore point's bytes, and the forms of `snapshot-kept` and `restore-kept`.** (§551) The seat `mesh-broker`'s two
verbs (§405) take and answer the forms its definition in mesh-contracts gives them, which this line makes the
design's: `snapshot-kept` takes `module`, `kept` (the kept data's local name), `form`, `fingerprint`, `why` and
`catch-up` as `{tool, arguments}` (§451), and answers `point` (the module, the kept data, its form, the
fingerprint, `snapshot`, `consumers` with each durable consumer's stream, name and acknowledged sequence, `size` in
bytes, `at` and `catch-up`) or `error` alone; `restore-kept` takes `module`, `kept`, `form`, `fingerprint`, `why`
and one of `snapshot` (the restore point's name), `aside-only` or `discard-aside` (an aside's `id`), and answers
`restored`, `aside` and `error`. No argument `local-name` or `from` is taken, and a catch-up is never text. The bus
module keeps a restore point in its own `snapshots` directory, which ADR 0235 §1 to §3 give it for the bus's dump:
the server's snapshot of each bus object of the kept data (its stream, or the stream under its bucket or object
store), taken one at a time with ADR 0235 §10's pacing and §11's bounds, and the point's record, in the directory
`kept/<module>/<kept>/<form>-<fingerprint>`, each token escaped as a key's name is (§73); that name is the point's
`snapshot`. The holder of `node-backup` backs that directory up as it backs up `snapshots` (ADR 0235 §3), so its
copy is the point's safeguard, never the point: when its `restore` finds the point's directory missing on the bus
module's node, it brings it back from its newest repository snapshot that holds it, under the same name, and only
then calls `restore-kept`; when none holds it, `restored` says `failed` with `restore point missing`, and §300 holds
the revert. `restore-kept` sets the live kept data aside by taking the server's snapshot of each of its bus objects
into `aside/<module>/<kept>/<at>` in the same directory (`at` the UTC time written `20261111T121200Z`), only then
removes the live objects, restores the point under their names and sets each consumer to its recorded sequence. The
aside's `id` is that directory's name, its `name` the bus name it was live under, its `size` its bytes on disk;
nothing of an aside stays on the bus, so it takes none of the bus's storage limit, and the definition's
description of `name` reads "the bus name it was live under". `discard-aside` removes that directory and nothing
else. A restore point is removed only as §551 says. This completes §405 and §406. Why: ADR 0235 has the bus
module's account write the bus's snapshots into its own directory and the backup holder back that directory up;
the server restores a stream only where no stream of that name exists, so the live data is set aside as a
snapshot before the point takes its name (develop questions BKP1, BKP3 and NATS4's argument names). How it is
checked: unit (mesh-broker's contract), each verb's input and answer read, an input naming `from` refused; whole
(bus module, backup holder), a point is taken, its directory removed, brought back from the repository and
restored, the live data's aside listed with its size and the consumer at its recorded sequence; a point missing
from both says `restore point missing`.
- §517 **`backed-up` reads the restore points and the data set aside from the backup holder's own state.** The holder of
`node-backup` on the bus module's node keeps what `snapshot-kept` and `restore-kept` answered it in its state
`bus-points`, asked in its `requires` by the template `state` (ADR 0595 §1; the holder is no foundation part, ADR
0619 §1, ADR 0666 §4): a point under `point.<module>.<kept>.<form>.<fingerprint>` with the answer's `point` whole,
written when `snapshot-kept` answers done; an aside under `aside.<module>.<kept>.<at>` with the answer's `aside`
whole, written when `restore-kept` answers one, each token escaped as a key's name is (§73). The holder deletes an
aside's key when `discard-aside` answers done, and a point's key when it hears that point's `point-removed` (§551). `backed-up` lists both from
that state, each with its size (§302), and the holder's read tool of its state reads it. The seat `mesh-broker`
gains no read verb. The holder's old `state` entry `good` moves to `requires` in the same change, keeping its local
name and settings, as ADR 0595 §11 moves a module G touches. Why: the holder took each from its own call's answer,
so the list is its own fact (ADR 0596 §2), and a read verb on `mesh-broker` would answer what the caller already
holds (develop question BKP2). How it is checked: whole (backup holder), a restore's aside is listed by `backed-up`
with its size, is gone after `discard-aside`, and a restart of the holder lists the same.
- §516 **A restore point's bytes, and the forms of `snapshot-kept` and `restore-kept`.** (§551, §554) The seat
`mesh-recovery`'s two verbs (§554) take and answer the forms mesh-contracts' definition gave the seat `mesh-broker`'s,
which move to `mesh-recovery` and which this line makes the design's: `snapshot-kept` takes `module`, `kept` (the
kept data's local name), `form`, `fingerprint`, `why` and `catch-up` as `{tool, arguments}` (§451), and answers
`point` (the module, the kept data, its form, the fingerprint, `snapshot`, `consumers` with each durable consumer's
stream, name and acknowledged sequence, `size` in bytes, `at` and `catch-up`) or `error` alone; `restore-kept` takes
`module`, `kept`, `form`, `fingerprint`, `why`, `from-backup` (yes or no) and one of `snapshot` (the restore point's
name), `aside-only` or `discard-aside` (an aside's `id`), and answers `restored`, `aside`, `missing` (true when the
point is not in the recovery module's directory) and `error`. No argument `local-name` or `from` is taken, and a
catch-up is never text.
- **Where a point lives.** The recovery module takes a point through the bus's stream snapshot under its own account
(§554), one bus object of the kept data at a time (its stream, or the stream under its bucket or object store),
with ADR 0235 §10's pacing and §11's bounds, and writes it with the point's record in its own data directory,
`points/<module>/<kept>/<form>-<fingerprint>`, each token escaped as a key's name is (§73) and a token that is
empty, `.` or `..` refused; that name is the point's `snapshot`.
- **Its copy.** That directory is an item of the recovery module's manifest `data`, backed up by the holder of
`node-backup` on its node as every item is (ADR 0404), which stores the copy and nothing else. When
`restore-kept` answers `missing`, the node agent calls it again with `from-backup: yes`: the recovery module asks
the seat `node-backup`'s `restore` of its item for that directory, which brings it back beside its path as that
verb always does, takes the point from there and restores it; when no repository snapshot holds it, `restored`
says `failed` with `restore point missing`, and §300 holds the revert.
- **The aside, and a restore that fails.** `restore-kept` sets the live kept data aside by taking a stream snapshot
of each of its bus objects into `aside/<module>/<kept>/<at>` in the same directory (`at` the UTC time written
`20261111T121200Z`, with `-2`, `-3` and on added when that name exists), only then removes the live objects,
restores the point under their names and sets each consumer to its recorded sequence. Each step is written first
in the aside's record (`set aside`, `removed`, `restored`): when a step after the removal fails, `restore-kept`
restores the aside under the live names and sets the consumers back before it answers `failed`, and at its start
the recovery module does the same for every aside whose record ends at `removed`, so a failed or stopped restore
never leaves the kept data gone. The aside's `id` is that directory's name, its `name` the bus name it was live
under, its `size` its bytes on disk; nothing of an aside stays on the bus. `discard-aside` removes that directory
and nothing else. A restore point is removed only as §551 says.
This completes §405 and §406. Why: the operator's answer of 12:55 CET (§554) puts the taking and restoring of a
module's kept data on the live bus on a module of its own; the server restores a stream only where no stream of that
name exists, so the live data is set aside as a snapshot before the point takes its name (develop questions BKP1,
BKP3 and NATS4's argument names; review round 1, running and testing 1, seams and tools' notes). How it is checked:
unit (mesh-recovery's contract), each verb's input and answer read, an input naming `from` refused; whole (recovery
module, backup holder, bus), a point is taken, its directory removed, brought back with `from-backup` and restored,
the live data's aside listed with its size and the consumer at its recorded sequence; a point missing from both says
`restore point missing`; a restore made to fail after the removal, and one stopped there and started again, leave
the live data as before.
- §517 **The recovery module lists what it holds.** The seat `mesh-recovery`'s read verb `kept-points` (class `read`,
§391's default callers) lists every restore point and aside in its directory with its module, kept data, form,
fingerprint, size and `at`, and its tools are `recovery_kept_points`, which reads the same, and
`recovery_expire_points`, which runs §551's removal now (class `act`, §391's default callers, written in the
hand-act log). The directory is the one place they are kept, so no copy can list less: the backup holder keeps no
state of them (its `backed-up` lists the recovery module's item as any item, §302), and no `bus-points` state is
made. Why: a module's tools read everything it holds (*Tools are how a module is spoken to*), and the list is the
holder's own fact (ADR 0596 §2; develop question BKP2; review round 1, seams and tools 4). How it is checked: whole
(recovery module), `kept-points` lists a point and an aside with their sizes, an aside is gone after
`discard-aside`, and a restart lists the same.
- §523 **A seat event whose subject has parts.** A seat definition's `subject` with `<part>` tokens (§397), such as
`node-backup`'s `ended.<node>.<module>.<kept>` and `restored.<node>.<module>.<kept>` (§405), is said on
`mesh-recovery`'s `ended.<module>.<kept>` and `node-backup`'s `item-started.<node>.<module>` (§554), is said on
`mesh.seat.<seat>.event.` followed by the subject, each part filled in the definition's order: the seat
generator's subject function for the event takes each part as a string in that order and fills each as one token,
escaped as a key's name is (§73), refusing an empty part. A reader hears one module's events by the subject with
escaped as a key's name is (§73), refusing an empty part, `.` and `..`. A reader hears one module's events by the subject with
that module's tokens and `>` after them, and every module's by `*` for each part. Such an event has no key of its
own and is kept as its definition's `keep` says, or on EVENTS when it names no `keep` (§396). The holder's account
publishes it as a seat's holder publishes its seat's events today; no grant changes. Why: §397 fixes the subject's
words, not how a part is filled (develop question NATS4's wire). How it is checked: unit (seat generator), a subject
of three parts made in order and an empty part refused; whole (backup holder), `ended` heard by a reader of one
of three parts made in order and an empty part refused; whole (recovery module), `ended` heard by a reader of one
module's subject.
## The tool runner and its bundles
@@ -98,17 +110,23 @@ line. Questions that an earlier line already answers have no line here: develop
§345's put session and §490's get session run on that channel, as every `mesh/` call of a bundle does (ADR 0595
§9); `<bundle call subject>` in §457, §489 and §490 names that call, and no bus subject. A value at or under the
limit is put and got by one call, as today. Over it:
- **Whose object.** A session's `name` names an object of the bundle's own module alone: the tool runner resolves
it in that module's binding (its state's `<state>-objects` or its sent store), and refuses a name of another
module's, naming it. A put's whole size is bounded by that store's cap, and a put whose chunks pass it is refused
at the chunk that does, naming the store.
- **Put.** The bundle sends each chunk as the call `mesh/objects.put-chunk` with `name` (the object), `call` (the
bundle's id for this put), `seq` (from 1, one more each chunk), `last` (true on the last chunk alone) and `data`
(the chunk's bytes in base64, at most half the bus's maximum payload before encoding, §457). The tool runner
answers each with `{seq}` or `{error}`, the bundle sends the next only after a `{seq}`, and the answer to the last
is the put's answer; a `seq` that is not one more than the last taken is answered `{error}` naming both, and a put
not sealed within the call's bound is dropped (§345). The tool runner writes the whole as one object through the
object store's own chunking.
not sealed within the call's bound is dropped (§345). The tool runner writes each chunk into the store as it takes it, through the
object store's own chunking, holding no more than one chunk in memory, and seals the object on the last.
- **Get.** The tool runner answers the get call `{chunked: <count>, size, digest, call}` (§490), and the bundle asks
each chunk in order by the call `mesh/objects.get-chunk` with `call` and `seq`, answered `{seq, last, data}` or
`{error}`; the bundle checks the whole against `digest`, and a get not completed within the call's bound fails,
naming the object.
naming the object. The tool runner drops a get session it holds when the call's bound passes, or the bundle's
channel closes, whichever comes first.
- **The bound.** The call's bound counts once for the whole session, from its first chunk, never per chunk.
The headers `Mesh-Put-Seq` and `Mesh-Put-Last` of §457 are these fields `seq` and `last`. Nothing of a session is
on the bus, so §489's grants are not made and the grants part grants nothing for it. A module's process, which
@@ -117,8 +135,9 @@ line. Questions that an earlier line already answers have no line here: develop
calls. This amends §345, §457, §489 and §490; in the get the bundle asks each chunk, since on that channel every
`mesh/` call is the bundle's. Why: the operator's answer to develop question D-13, chunks in order, each
acknowledged before the next, is kept whole on the one channel a bundle has (develop questions T-5 and T-7). How it
is checked: whole (mesh-sdk, tool runner), a value of a hundred times the limit is put and got back whole by a
bundle, its digest checked, and a chunk out of order fails the put naming both sequences; the grants part's table
is checked: whole (mesh-sdk, tool runner), a value of a hundred times the limit, into a fixture state whose
`objects-max-mib` holds it, is put and got back whole by a bundle, its digest checked, a put naming another
module's object is refused, and a chunk out of order fails the put naming both sequences; the grants part's table
test grants no session subject.
- §519 **A bundle's fingerprint is in the environment the node-engine composes.** The node-engine writes
`MESH_FINGERPRINT`, the fingerprint of the want it applies for the bundle's module, into each tools bundle's
@@ -133,25 +152,31 @@ line. Questions that an earlier line already answers have no line here: develop
- §520 **`running` says a build's health, and the assigner reads health from it alone.** The node-engine seat's
`running.<node>.<module>` gains `healthy` (true or false), the health of the build that runs as the node-engine's
have holds it from the module's declared check (ADR 0240), and the node-engine says `running` again on the same
subject each time `healthy` changes, so the stream, kept last per subject, holds each build's newest health.
`stopped` (§365, §404) is said only when the have no longer holds the module, never for an unhealthy build. The
subject each time `healthy` changes, and for every module its have holds at its own start, so the stream, kept last per subject, holds each build's newest health.
`stopped` (§365, §404) is said only when the have no longer holds the module, never for an unhealthy build. A `running` with no
`healthy`, said by a node-engine of before G, reads as healthy while it stands, as `running` meant on main. The
assigner reads a provider's health for rule 5, the canary's for rules 3 and 4, and a put's end from these events
alone, through its durable consumer that starts from the last message per subject (§199), and reads no have: a
provider of which nothing was heard since that consumer began counts as not healthy, so its consumers' new wants
alone, read as §556 has it, and reads no have: a
provider of which nothing was heard counts as not healthy, so its consumers' new wants
are held as rule 5 holds any, and nothing falls back to the have. `SeatsToPassOn` reads no `healthy` (§459): an
unhealthy holder still holds its seat. The assigner's `reads` of the node-engine's have go. This amends §309 and
§404. Why: the assigner reads no other module's state (ADR 0596 §2), and a stopped build and an unhealthy one are
told apart (develop questions AS15 and AS16). How it is checked: unit (the node-engine seat's fixtures), `running`
with `healthy` read; whole (assigner, node-engine), a provider turning unhealthy holds a consumer's new want and its
recovery lets the want go, with the assigner's account holding no read of the have.
recovery lets the want go, with the assigner's account holding no read of the have; whole (assigner, a
node-engine of before G), its provider's `running` with no `healthy` holds no consumer's want; a restarted assigner
holds nothing it did not hold before.
- §524 **The node-engine checks a want's parts by their digests.** A composition's archive, process and container
resources name a source and a digest, not an artifact, so the node-engine matches no resource to a `parts` key: it
applies a want only when the digest of every resource it fetched is among the values of the want's `parts`, and
resources name a source and a digest, not an artifact, so the node-engine matches no resource to a `parts` key. A want
with no `parts` (from an assigner of before G, or for a node whose record does not state `want-parts`, §525) is
applied as today, its resources fetched by their sources and checked by their own digests. A want with `parts` is
applied only when the digest of every resource it fetched is among the values of the want's `parts`, and
every value of `parts` is the digest of a resource it fetched; otherwise it applies nothing of that module and its
have says the phase `parts-differ`, naming the digest. `parts` stays keyed by `<module>:<artifact>` for people and
the delivery record. This completes §379. Why: the digest is what is fetched and checked (develop question H-8). How
it is checked: unit (node-engine), a want whose parts and fetched digests agree applies, and one with a fetched
digest outside `parts` or a part not fetched applies nothing, naming it.
digest outside `parts` or a part not fetched applies nothing, naming it; whole (node-engine of G), a want written by
an assigner of before G is applied.
- §525 **A node-engine says it reads the want's new fields.** A node-engine of G states the capability `want-parts` in
its node record's capabilities (ADR 0614 §4, ADR 0618 §2), and the assigner writes `version`, `parts` and `kept-on`
(§526) into a node's want only while that node's record states it; that is §379's "once that node runs G's
@@ -168,11 +193,12 @@ line. Questions that an earlier line already answers have no line here: develop
the catalog half of their view: the module whose entry claims the seat `node-engine`, which §508 makes one per
revision; the manifest validator, beside §508's refusal of a second claim, refuses a revision in which no module
claims it, naming the seat. §364's, §365's and §508's "from the seat list" read so, the condition keeper's too. With
no entry claiming the seat, which only a revision from before G can show, the node-engine applies nothing of that
revision and its have says `entry-missing`, naming the seat. Why: §364 and §508 have the node-engine claim its seat
no entry claiming the seat, which only a revision from before G shows, as at a revert of G, the node-engine's own
module is the module of the catalog graph entry its own build's fingerprint names (§443, §205), so a revert of the
node-engine applies. Why: §364 and §508 have the node-engine claim its seat
as one module, so the claim names it (develop question H-15). How it is checked: unit (manifest validator), a
revision with no claim of `node-engine` refused; whole (node-engine), its guard protects the module its catalog
half names, and the code review finds no fixed name (§428).
half names, and with a revision of before G the module its own fingerprint's entry names, and the code review finds no fixed name (§428).
## Findings and the condition keeper
@@ -186,8 +212,13 @@ line. Questions that an earlier line already answers have no line here: develop
- **Every other part** asks both in its `requires`: the state `findings` by the template `state`, keeping its local
name and carrying its settings as options (ADR 0595 §11), and the stream by the template `events` under the local
name `finding-events`, with `max-age-days` 7 and `max-mib` 16, which the template drops oldest when full (ADR 0595
§3), published on its binding's subject, `mesh.bus.<module>.finding-events.<event>.<node>`. `mesh-healers`, whose
old `streams` declares `mesh-healers-findings` on main, moves it so in G's change.
§3), its entry listing the three finding events in its field `events`, which the catalog graph entry carries
(§365), published on its binding's subject, `mesh.bus.<module>.finding-events.<event>.<node>`.
- **`mesh-healers`**, whose old `streams` declares `mesh-healers-findings` on main, moves it so in G's change, as
ADR 0595 §11 moves a module G touches. Its old stream stays on the bus as an orphan that `orphans` lists, removed
only by the operator's `drop` (ADR 0595 §4, §5); the grants part deletes the keeper's consumer on it once the
healers' entry no longer declares it (§386), and the healers say `findings-stated` at their start on the new
stream (§384), so the keeper loses no finding.
The manifest validator applies ADR 0595 §11's count of old sections to no part of its foundation list (ADR 0619
§2), so the tool runner's and the checker's manifests are not refused for it, and accepts in a foundation part's
@@ -196,8 +227,9 @@ line. Questions that an earlier line already answers have no line here: develop
list, fixed in the validator (ADR 0619 §3). The validator refuses a manifest whose state `findings` has no finding
stream in its part's form, or no `consumes` of `mesh-conditions.findings-asked` (§387). The writer (§461) is handed
the subjects from the part's binding or its old stream, as it is handed every subject. The catalog graph entry
carries the finding stream in either form (§365); the grants part makes the keeper's consumer of §386 on either
stream and grants each part publish on its own finding stream's subjects of its own node only, in either form; the
carries the finding stream in either form (§365); the grants part makes the keeper's consumer of §386 on an
old stream at the revision that declares it, and on a template stream when it hears the bus module's `created` for
`finding-events` and grants each part publish on its own finding stream's subjects of its own node only, in either form; the
keeper takes the observer and the node from either subject. This amends §383, §386 and §387, whose "into the
manifest's `streams`" holds for a foundation part alone. Why: ADR 0666 §4 and ADR 0595 §11 put a part's kept data
in the templates unless it is a foundation part, and ADR 0619 §2 keeps the foundation's old sections whole; the
@@ -205,32 +237,42 @@ line. Questions that an earlier line already answers have no line here: develop
name per module (develop questions on §387, both validator refusals). How it is checked: unit (manifest validator),
a foundation part's old `findings` and finding stream accepted, a non-foundation part's old sections refused and its
template form accepted; the catalog's validation at G's heads refuses none of §387's manifests; whole (keeper, a
template observer and the tool runner), both forms' events raise a condition.
template observer and the tool runner), both forms' events raise a condition; whole (keeper, healers), a healers'
finding raised before their move is still raised after it.
- §540 **The keeper hands a failed head to the operator after two hours.** The condition keeper raises
`pipeline-failed` (§33) with `owner` the commit's author, as the runner's finding names it, and two hours after the
finding's `since` sets the condition's `owner` to `operator-channel`, the seat the operator is asked through (a
seat's name, §451), and says that change as it says any change of a condition; the runner's finding keeps the
author. The keeper times the two hours on its own conditions from `since`, so after its restart it times them from
author. §451's `owner` and `waiting-on` gain a third form, `forge:<account>`, the commit's author as the forge
names it. A `pipeline-failed` for a newer failing head is a finding of its own, its id naming that commit, so it is
its new author's with two hours of its own. The keeper's tool `conditions_owner` sets a condition's `owner` now, of
class `act` with §391's default callers, written in the hand-act log. The keeper times the two hours on its own conditions from `since`, so after its restart it times them from
`since` again. Why: §33 has the keeper raise it as the author's, then the operator's (ADR 0505 §5, §6; develop
question COND7). How it is checked: whole (keeper), a `pipeline-failed` is the operator's two hours after its
`since`, and so after a restart of the keeper.
`since`, and so after a restart of the keeper; unit (mesh-conditions' contract), `forge:<account>` read.
- §541 **The keeper hears what a node runs and the healers' attempts as events.** The keeper's `reads` of
`node-engine.have` and `mesh-healers.attempts` go (§434).
- **What a node runs.** The keeper consumes the node-engine seat's `composed`, `running` (with `healthy`, §520, and
`stopped`) and `wait-held` and `wait-ended` (§499), from the last message per subject (§199), and compares them
with the wants it hears from the assigner. The node-engine says `wait-held`, with `step: compose` and the reason,
for each hold its have writes (§300's `restore point missing`, §421's and §527's `entry-missing`, §524's
`parts-differ`, §499's kept build), and `wait-ended` when the hold ends.
`stopped`) and `wait-held` and `wait-ended` (§499), read as §556 has it, and compares them with the wants it hears
from the assigner, to hold a module's findings while it is delivered (ADR 0564 §13), as it did from the haves.
The node-engine says `wait-held` on `mesh.seat.node-engine.event.wait-held.<node>.<module>`, with `step: compose`
and the reason, for each hold its have writes (§300's `restore point missing`, §421's `entry-missing`, §524's
`parts-differ`, §499's kept build), and `wait-ended` on `.wait-ended.<node>.<module>` when the hold ends. The
keeper raises no condition of its own from them: the watchman writes `delivery-waits` for a `wait-held` of the
node-engine standing past composing's 30 minutes (§266), naming the node, the module and the reason, and it clears
on `wait-ended`; a `composed` whose fingerprint is not the want's is covered by that same wait or by the put's
wait (§498).
- **The healers' attempts.** The seat `mesh-healers` says `attempt-changed` on
`mesh.seat.mesh-healers.event.attempt-changed.<condition>`, the condition's key escaped as a key's name is (§73),
carrying the attempt's value whole as its key in `attempts` holds it and `until`, the end of that key's
lifetime, or `removed: true` when the healer ends the attempt; the seat's definition, in its contracts module,
keeps it last per subject, 7 days and 16 MiB (§396). The keeper's durable consumer starts from the last message per
subject, and the keeper drops an attempt at its `until` on its own clock.
keeps it last per subject, 7 days and 16 MiB (§396). The healers say it again for every open attempt at their start. The
keeper reads it as §556 has it, and drops an attempt at its `until` on its own clock.
Why: the keeper reads no other module's state (ADR 0596 §2; develop question COND8). How it is checked: whole
(keeper, healers, node-engine), an attempt said is shown on its condition and dropped at its `until` or its
removal, a node-engine's hold is heard from `wait-held`, and the keeper's manifest declares no `reads`.
removal, a node-engine's hold is heard from `wait-held` and past 30 minutes raises `delivery-waits`, a restarted keeper
holds the attempts it held, and the keeper's manifest declares no `reads`.
## Views of the catalog half, per watched branch
@@ -263,14 +305,18 @@ line. Questions that an earlier line already answers have no line here: develop
runner's own wiring of `catalog-revision.>` goes for it.
- **The bus module across branches.** The bus module plans a module's kept data from the newest revision of each
watched branch that lists the module: each local name once, from the first of the module's watched branches, in
its registration's order, whose revision lists it; another branch asking that local name with other settings is
its registration's order, whose revision lists it. A registration's watched branches are a list in that order: the
branch `register` names first (the repository's default branch, a placement's default, ADR 0666 §6), then each
branch in the order `watch` added it; `unwatch` removes one and keeps the others' order; another branch asking that local name with other settings is
said as the bus module's `drifted`, naming the branch (ADR 0595 §6), and plans nothing.
This amends §358, §365, §421, §433 and §507. Why: no part fixes a branch (§7, ADR 0666 §5), and a registration may
add a watched branch while a part runs (develop questions NATS3, VW1 and COND6, and the client's lost per-branch
events). How it is checked: unit (`go/views`), a view adopted after the start takes its first view and follows,
and two branches' events reach their own views; whole (keeper, bus module), a branch registered while they run is
followed with no restart; the code review finds no branch's name in any part.
followed with no restart; whole (bus module), a feature branch asking a local name with other settings than the
default branch's is said `drifted` while the default branch's are planned; the code review finds no branch's name
in any part.
## The bus module's plans
@@ -289,8 +335,10 @@ line. Questions that an earlier line already answers have no line here: develop
a seat whose verbs come only from an included part gives its holder a sent store, and one including only bodies
gives none.
- §538 **The bus module plans again when the bus-state entries change.** The bus module compares, for each module,
the bus-state entries the catalog graph entry carries with those it last planned for that module, kept in its own
state, and plans again only when they differ; it never compares the manifest's digest (§506), which changes with
the bus-state entries the catalog graph entry carries with those it last planned for that module, kept in its old
`state` section `plans` (the bus module is a foundation part), one key `<module>` holding the entries as planned
and their digest, written after each plan and deleted when the module is in no watched branch's newest revision, its
entries then orphans that `orphans` lists; its tool `nats_plans` reads it. It plans again only when they differ; it never compares the manifest's digest (§506), which changes with
any field. Why: what it plans is those entries alone (develop question on §433 with §506). How it is checked: whole
(bus module), a manifest changed outside `requires` plans nothing, and a changed option plans the difference.
@@ -353,12 +401,17 @@ line. Questions that an earlier line already answers have no line here: develop
lead-47's terminal: approval is a step whose status each module's registration sets, never a gate of its own, so
an Approve given before G counts, once G is delivered, as that pull request's `approval` step and nothing else is
special. At G's switch the gate says, for each `merges` key the old bridge had not performed, `approval-carried` on
`mesh.seat.mesh-delivery.event.approval-carried` (`repository`, `pull`, `commit` the head the Approve named,
`approver` the operator, `at`), declared in the seat's definition, and deletes the key once the bus acknowledges
it. The runner keeps each in its old `state` section `pipelines` as `carried-approval.<repository>.<pull>`,
holding the commit, and takes it as the result `passed`, caller the operator, of the `approval` step of the first
pipeline or run at that commit, then deletes it; it deletes it unused when the seat `git` says the pull request
`pull-closed` or `pull-merged`. Where the module's policy makes `approval` `required`, a pull request whose head
`mesh.seat.mesh-delivery.event.approval-carried.<repository>.<pull>` (`repository`, `pull`, `commit` the head the
Approve named, `approver` the operator, `at`), declared in the seat's definition with `keep` 90 days (§396). The gate
keeps its `merges` keys, read no more, until §97's removal, so a revert's gate of before G finds them. The runner
hears the event through a durable consumer made to start from the stream's first message, so a runner started
after the switch hears every one, and keeps each in its old `state` section `pipelines` as
`carried-approval.<repository>.<pull>`, holding the commit. It takes it as the result `passed`, caller the
operator, only of the `approval` step of the delivering pipeline whose source revision takes in that pull
request's merge, as `take-in.taken`'s entries name it (each entry carrying the repository and the pull request of a
`merged` message, §493, §509), and only when the pull request's head merged is that commit; never in a run on
request and never in another pipeline at the same commit; then it deletes the key, and deletes it unused when the
seat `git` says the pull request `pull-closed`. Where the module's policy makes `approval` `required`, a pull request whose head
moved after the Approve asks a new approval, as any step whose input changed does (an `approval` step is never
carried, ADR 0666 §8); where it is `optional` or `bypass`, the carried Approve holds nothing. The merge itself is
any merge (`merge-pull`, ADR 0666 §11): the bridge performs no carried act, the gate says no `wait-held` with
@@ -367,8 +420,9 @@ line. Questions that an earlier line already answers have no line here: develop
`cancelled`, `not asked`) or its pipeline is superseded or cancelled, so no approval wait stands without an end.
This replaces §510 and §608, and `merge` leaves §498's list. Why: the operator's answer ("you're mixing in step and
approval gates, this should be configurable"); the event, the key and the deletion are the design's (round 3,
health and events' R7). How it is checked: whole (gate, runner), a carried Approve of a head passes that head's
`approval` step with the operator as caller; with `approval` `required`, a head moved after the Approve asks a new
health and events' R7). How it is checked: whole (gate, runner), a carried Approve of a head passes the `approval` step
of the pipeline taking in that pull request's merge, with the operator as caller, and no run on request at that
commit; a runner started after the switch takes it all the same; with `approval` `required`, a head moved after the Approve asks a new
approval; a pull request closed deletes its carried key; every `approval` wait-held in the whole test has its
`wait-ended`; the code review finds no `merge-waits`.
@@ -379,11 +433,13 @@ line. Questions that an earlier line already answers have no line here: develop
ask under its name. The grants part grants each asker publish on `mesh.seat.<seat>.accept.<queue>.<asker>` (§600)
for every queue of the seat on which a kind it is a caller of is taken, and the holder, which takes the asker from
the subject (§481), refuses an ask whose body names a kind that asker is not a caller of, answering `error` naming
the kind and the asker, and ends it with `step-ended` `failed`. This amends §338 and §600. Why: §600 puts the queue
the kind and the asker, and ends it with `step-ended` `failed`. On main's subject `accept.<queue>`, which names no asker and is granted to
the checker and the builder alone (§600), the holder takes only the kinds `build` and `run` and refuses any other,
`step` among them, as above. This amends §338 and §600. Why: §600 puts the queue
in the subject and the kind in the body, so a subject's grant alone cannot hold a caller to its kinds (round 3,
security and upgrade, seams and tools: R1). How it is checked: unit (Go build module), a `step` published from the
indexer's account is answered `error` naming `step` and `indexer`; unit (grants part), each asker's grants are the
queues of its kinds.
queues of its kinds; unit (Go, Node and image build modules), a `step` on main's subject is answered `error`.
- §544 **Named asks have a worker of their own.** For each queue the grants part makes, beside main's worker
`SEAT_<SEAT>_<QUEUE>_worker` filtered to main's `mesh.seat.<seat>.accept.<queue>` alone, a second worker
`SEAT_<SEAT>_<QUEUE>_named_worker` filtered to `mesh.seat.<seat>.accept.<queue>.*`; the two filters do not overlap,
@@ -392,16 +448,23 @@ line. Questions that an earlier line already answers have no line here: develop
part runs takes no named ask. This replaces §601's "until every node's holder of a build seat is of G the grants
part's worker keeps main's one filter", and with it the edge from the seats' holders to the grants part, which no
manifest fact gives: the order is the asker's `invokes` of `<seat>.accept.<queue>` (kind 6, from the seat's holders
to the asker, §601), and the grants, written for a catalog revision before any want of it (*A credential and a
composition before the want*), exist before the asker runs. A named ask waits on its worker while no holder of G
to the asker, §601), and the grants part comes before every asker as the acting part of the step `grant` on
`every` (§203): each entry of a revision waits for the grants part's new version to act for it (§266), so no asker
of G runs before a grants part of G has made its named worker and its grant. A holder of G pulls main's worker
alone until its membership names the named worker, which the grants part writes as it binds every consumer
(§386), and says meanwhile, in its journal and its read tool, `named worker not bound`. A named ask waits on its worker while no holder of G
runs, said by `build-capacity-waits` after 60 minutes, which corrects §483's "waits on the work queue, whose
consumer of before G takes only main's subject". At a revert, which reverts the askers before the grants part (ADR
0656 §4), the grants part of G, when it no longer hears any asker of G, purges the subjects
`accept.<queue>.*` of the stream and deletes the named workers, saying each in its journal; an ask is transport,
consumer of before G takes only main's subject". At a revert, the grants part of G takes its view of the revert's catalog revision before
it is itself reverted, since it acts for every entry of it (§203); at the first revision in which no entry `invokes`
any `<seat>.accept.<queue>`, it purges the subjects `accept.<queue>.*` of the stream and deletes the named workers,
within its granting bound of 10 minutes (§266), saying each in its journal. Where that did not run (the grants part
stopped first), its tool `grants_named_purge`, of class `act` with §391's default callers and written in the
hand-act log, does it by hand; an ask is transport,
not kept data (as §59 has a sent object). Why: one consumer per filter, and a filter that never moves cannot be
pulled by a holder that does not read it (round 3, R2, the security reviewer's form). How it is checked: whole
(grants part, Go build module), a holder of before G started after the grants part of G takes main's asks and no
named one; a revert leaves no named worker and no named ask.
named one; a revert leaves no named worker and no named ask; a holder of G before its binding pulls main's worker
and says `named worker not bound`.
- §545 **The tool runner's artifact declares the words it states.** The artifact entry of the tool runner's manifest
takes `states`, the capability words (§488) it states; the manifest validator refuses a package artifact's `calls`
word (§605) that no artifact of the revision `states`, naming the word, and the delivery plan's edge of §605 runs to
@@ -410,32 +473,39 @@ line. Questions that an earlier line already answers have no line here: develop
reads the stated words at its own start; no event is heard and nothing polls. This amends §605's "each time the
tool runner of its node starts". Why: the words are stated only at run time, and the plan needs them in a manifest
(round 3, seams and tools: R3). How it is checked: unit (manifest validator), a `calls` word nothing `states`
refused; unit (delivery plan), the edge to the module stating the word.
refused; unit (tool runner), the words it states are exactly its manifest's `states`; unit (delivery plan), the edge to the module stating the word.
- §546 **When the builder forgets a node's toolchain.** The builder deletes its `toolchains-heard` key
`<node>.<language>` (§604) when the node-engine seat's `running.<node>.<module>` says `stopped` (§365) for the
module whose catalog graph entry claims the build seat of that language (its `acts` rule `build` with `on` naming
it, §535), heard through the durable consumer it already has of that seat's `running`; never on a placement, which
it, §535), heard through its consumer of that seat's `running`, read as §556 has it; never on a placement, which
is the seat `mesh-module-assigner`'s state. The state is an old `state` section of the builder, a foundation part,
one key per node and language, about 12 keys of about 0.3 KiB each, capped at 1 MiB, so it adds no objects store
to §607's count (§58 plans one only for a state of the template); `builder_toolchains` reads it and
`builder_toolchains_drop` deletes one key, with why, written in the hand-act log of the seat `mesh-hand-acts`; a
`builder_toolchains_drop`, of class `act` with §391's default callers (`operator`, `sessions`), deletes one key,
with why, written in the hand-act log of the seat `mesh-hand-acts`; a
holder says the key again at its start and on each change (§492). This amends §604's "when the node leaves the
catalog's placements". Why: the builder reads no other module's state (ADR 0596 §2; round 3, seams and tools: R4).
How it is checked: whole (builder), a build module stopped on a node deletes that node's key, and the drop tool
deletes one key and logs it.
- §547 **The sum of the bus's caps is read against its limit.** The bus module reads the bus's storage limit from the
server's account information at its start and at each plan, sums the cap of every entry it plans (states, objects
stores, sent stores, streams, logs) and of every old section the catalog graph's entries declare, and shows both in
its `list` and its read tool. A plan whose sum passes the limit is refused, naming the sum and the limit, said as
its `refused` (ADR 0595 §6), whether or not the server reserves a cap at creation. When G's check measures that the
server reserves a cap when a store is made (§607), each `<state>-objects` is planned with 16 MiB instead of 64 MiB
(about 2.4 GiB for 150 states), a state needing more naming `objects-max-mib`. This amends §607. Why: the limit is a
deletes one key and logs it; a drop by another caller is refused by the bus.
- §547 **The sum of the bus's caps is read against its limit.** Measured on 2026-11-11 against an embedded bus
(nats-server 2.11.17): an account's storage limit reserves each stream's maximum size when the stream is made (a
second stream of 64 MiB under an account limit of 100 MiB is refused, "insufficient storage resources"), and a
server's own limit does not. So each `<state>-objects` is planned with 16 MiB (about 2.4 GiB for 150 states), a
state needing more naming `objects-max-mib`, and no later measure is waited for. The bus module reads the account's
and the server's storage limits at its start and at each plan, sums the cap of every entry it plans (states, objects
stores, sent stores, streams, logs) and of every old section the catalog graph's entries declare, and shows the sum and
both limits in its `list` and its read tool. A plan whose sum would pass the account's limit is refused, naming the
sum and the limit, said as its `refused` (ADR 0595 §6): the plan refused is the one that adds or raises a cap, and
the caps already made stay as they are. With no account limit (unlimited) nothing is refused and the sum is shown
against the server's limit. This amends §607's 64 MiB and its measure in G's check. Why: the limit is a
fact of the server, read rather than written in the design, and a sum over it fails loudly (round 3, running and
testing: R5). How it is checked: whole (bus module), a fixture server with a small limit refuses a plan past it,
naming the sum; the sum and limit are in `list`.
- §548 **The forge module writes `seen` only from the runner's steps.** §487's `seen.<commit>.<module>` is written
only from a `step-ended` whose asker is `runner`; the `step-ended` of any other ask (§602) carries `asker`, `ask`,
`node`, `outcome` and `at` and nothing of a pipeline, and the forge module writes nothing from it. This amends §487.
testing: R5). How it is checked: whole (bus module), a fixture bus with a small account limit refuses the plan
that would pass it, naming the sum, and keeps the caps made; with no account limit nothing is refused; the sum and
the limits are in `list`.
- §548 **The forge module writes `seen` only from a pipeline's steps.** §487's `seen.<commit>.<module>` is written only
from a `step-ended` that carries a pipeline's `commit` and `fingerprint`, which a pipeline's step carries; the
`step-ended` of any other ask (§602) carries `asker`, `ask`, `node`, `outcome` and `at` and neither, so the forge
module writes nothing from it and names no asker. This amends §487.
Why: since §602 every ask says `step-ended`, and a builder's build is no run of a commit (round 3, health and
events: R6). How it is checked: unit (forge module), a builder's failed build writes no `seen`; whole (watchman), a
`stuck` raised for an ask clears on its late `step-ended`.
@@ -454,16 +524,17 @@ line. Questions that an earlier line already answers have no line here: develop
## The operator's answers of 12:41 CET
- §551 **A restore point is removed after 90 days, never the newest.** The operator, 2026-11-11 12:41 CET at lead-47's
terminal. The bus module removes a restore point's directory (§516) 90 days after its `at`, as long as delivery
terminal. The recovery module removes a restore point's directory (§516) 90 days after its `at`, as long as delivery
records are kept, unless it is the newest point of its module and kept data, which it never removes; it checks at
its start, once a day and at each `snapshot-kept`, on its own clock and its own directory. Each removal is said as
the seat `mesh-broker`'s event `point-removed` (`module`, `kept`, `form`, `fingerprint`, `snapshot`, `at`), declared
in its definition; the backup holder consumes it and deletes that point's key in `bus-points` (§517), and its
repository's copy goes by the repository's own rotation (ADR 0235 §4). Data set aside is removed only by
`discard-aside` (§516). This completes §516 and answers issue 1921's note on the retention of restore points. Why:
the operator's answer; the event, the checks and the holder's deletion are the design's. How it is checked: whole
(bus module, backup holder), a point older than 90 days is removed and said and its key deleted, and the newest
point of a kept data, older than 90 days, stays.
its start, once a day and at each `snapshot-kept`, on its own clock and its own directory, as one of its acts, never
beside a `restore-kept` (§554). Each removal is written in the hand-act log of the seat `mesh-hand-acts` and said as
`mesh-recovery`'s event `point-removed` on `mesh.seat.mesh-recovery.event.point-removed.<module>.<kept>` (`module`,
`kept`, `form`, `fingerprint`, `snapshot`, `at`), kept 90 days (§396). The backup holder's copies go by its
repository's own rotation (ADR 0235 §4). Data set aside is removed only by `discard-aside` (§516). This completes
§516 and answers issue 1921's note on the retention of restore points. Why: the operator's answer; the event, the
checks and their order are the design's. How it is checked: whole (recovery module), a point older than 90 days is
removed and said, the newest point of a kept data older than 90 days stays, and a removal due during a restore runs
after it.
- §552 **The checker may drop its own asks until its removal.** The operator, 2026-11-11 12:41 CET at lead-47's
terminal: the build seats' verb `drop` (§338) gains the checker as a caller until its entry is removed with its
buckets and bus account (ADR 0667 §12), as `run` has it, so the validator accepts the checker's manifest's
@@ -0,0 +1,128 @@
---
layer: to-be
status: in-progress
design: 47
milestone: G
updated: 2026-11-11
decisions:
- "ADR 0667"
- "ADR 0666"
- "ADR 0660"
- "ADR 0656"
- "ADR 0636"
- "ADR 0626"
- "ADR 0619"
- "ADR 0618"
- "ADR 0614"
- "ADR 0596"
- "ADR 0595"
- "ADR 0564"
- "ADR 0505"
- "ADR 0404"
- "ADR 0363"
- "ADR 0240"
- "ADR 0235"
---
# 47, milestone G: the recovery module, and the review of §516 to §553
One file of milestone G's part of [design 47](../47-delivery-from-commit-to-delivered.md); the part opens at
[G.md](G.md). Its design lines are numbered across every file of the part and cited as "design 47 G §n".
*§554 and §555 are the operator's answer of 2026-11-11 at 12:55 CET at lead-47's terminal. §556 to §558 answer the
four reviewers' round 1 of [G-12](G-12-develop-answers-fourth.md) (§516 to §553); the rest of that round's findings
are written into the lines of G-12 they name.*
## The recovery module
- §554 **The recovery module takes, sets aside and restores kept data on the live bus.** The operator, 2026-11-11
12:55 CET at lead-47's terminal: a module of its own, on a seat of its own, with an account of its own, owns
snapshotting, setting aside and restoring a module's kept data on the live bus, each act in the hand-act log; the
bus module keeps no restore permission (ADR 0626 §1 stays as it is), the snapshot account stays limited to the
nightly dump (ADR 0626 §2), and the backup holder only stores copies. The record of the recovery module, which
lead-47 proposes to the knowledge base for the operator's Approve, decides the account and its grants; this line
is written to it.
- **Its seat and module.** The seat `mesh-recovery`, held once for the mesh, is defined in mesh-contracts beside
`mesh-broker`, and the module `mesh-recovery`, in mesh-catalog, claims it; its glossary word is the *recovery
module*. It keeps no kept data on the bus, its one store being its data directory (§516), so it is no foundation
part (ADR 0619 §1) and waits on no bus module.
- **Its verbs and callers.** `snapshot-kept` and `restore-kept` (with `aside-only`), class `act`, are called by the
node agent applying a want (`seat:node-engine`, §296, §297) alone; `restore-kept` with `discard-aside` by
`operator` alone; `kept-points` (§517), class `read`, takes §391's default. Each act is written in the hand-act
log of the seat `mesh-hand-acts` first, with the caller the bus proved and `why`; an act whose log write fails
does nothing and answers `error`, naming the log (as §480).
- **Its account.** Composed by the grants part from the seat's definition as the record decides: the bus's stream
snapshot and restore, a stream's deletion and a consumer's creation, only for the bus objects of a module's kept
data (the names a binding or an old section of a catalog graph entry gives), never for the mesh's own streams nor
for any other object; it publishes only on its own seat's subjects and reads no module's data otherwise.
- **Its events.** On `mesh.seat.mesh-recovery.event.`: `ended.<module>.<kept>` (a point taken, with `outcome` `done`
or `failed` and its `error`), `restored.<module>.<kept>` (likewise), `set-aside.<module>.<kept>` (with the aside's
`id` and `size`) and `point-removed.<module>.<kept>` (§551), each filled as §523 has it and kept 90 days (§396).
The node agent's wait for `ended` or `restored` is bounded at 90 minutes (30 for a nightly dump it waits behind,
30 for a point brought back from the backup, 30 for the restore), past which its have says the wait, naming the
seat, and the watchman's `delivery-waits` raises it. This amends §405's 30 minutes.
- **One act at a time, and the nightly dump.** The recovery module runs its acts (`snapshot-kept`, `restore-kept`
and a removal of §551) one at a time, the next waiting within its own bound. The holder of `node-backup` says
`item-started.<node>.<module>` and `item-ended.<node>.<module>` for every item it backs up, declared in its seat's
definition; the recovery module consumes those of the bus module's item and starts no `restore-kept` while the
nightly dump of the bus runs, so the dump never meets a stream a restore removed (ADR 0235 §13). A `snapshot-kept`
may run beside the dump, both paced as ADR 0235 §10 has it, so the server has at most two snapshots at once.
- **Its order and its tools.** The node-engine's `invokes` of `mesh-recovery`'s verbs gives the served-verb edge
(kind 6) from the recovery module to the node-engine, so §298's "the new versions of the `node-backup` holder and
of the bus module run before any module whose kept data's form changes is moved" reads "the new version of the
recovery module". Its tools are named `recovery_`, and its declared health check calls `kept-points`.
Why: the operator's answer; the seat's placement in mesh-contracts, its callers, its events, its bound and its
order are the design's, from §296 to §303 and §405 as they were. How it is checked: test 36 with the recovery
module; unit (grants part), the recovery module's account holds the grants the record names and no other, and the
bus module's and the snapshot account's hold no restore; whole (recovery module), a `restore-kept` asked while the
bus's item is backed up starts after `item-ended`; an act whose hand-act log write fails does nothing.
- §555 **What moves to the recovery module.** §296 asks `mesh-recovery`'s `snapshot-kept` and waits for its `ended`,
never the seat `node-backup`; §297 asks its `restore-kept` and waits for its `restored`; §302's `backed-up` lists
the recovery module's data item as any item, and the restore points and asides are listed by `kept-points` (§517),
`discard-aside` being `restore-kept`'s; §303 and test 36 name the recovery module where they named the backup
holder and the bus module; §337's and §405's callers `holder:node-backup@same-node` go with the two verbs of
`mesh-broker`, which gains none; the seat `node-backup` gains nothing for kept data on the bus: its `aside` and
`discard-aside`, the arguments `kept`, `form` and `fingerprint` of its `now` and `restore`, and its events
`ended.<node>.<module>.<kept>` and `restored.<node>.<module>.<kept>` go from its definition in mesh-contracts, and
it gains only `item-started` and `item-ended` (§554); §450's `aside` answer is `restore-kept`'s with `aside-only`;
§406's catch-up is recorded in the point as before. Why: one owner of the acts (§554). How it is checked: unit
(definitions), `node-backup` and `mesh-broker` hold none of the moved verbs; test 36.
## The review of §516 to §553, round 1
- §556 **A part that follows a stream kept last per subject reads it whole at each start.** A durable consumer goes on
from its acknowledged position after a restart, so a part that acts on what it heard from a stream kept last per
subject, and keeps no state of it, reads at each of its starts every subject's last message through an ordered
consumer it makes for that one read, then follows its durable consumer from its acknowledged position, an event
older by `at` than what that read gave changing nothing. Until the read ends the part acts on none of those events
and its `<module>_first_views` names the stream `waiting`. So read: the node-engine seat's `running`, `composed`,
`wait-held` and `wait-ended` (the last two kept on the stream `running` is kept on, last per subject, a node's
module's newer one by `at` standing), by the assigner (§520), the condition keeper (§541), the builder (§546), the
node-engine and the grants part (§199); and `mesh-healers`' `attempt-changed` by the keeper (§541). This amends
§199's "each part's durable consumer starts from the last message per subject". Why: what a part heard must not
depend on when it last restarted (review round 1, health and events 2, with running and testing 3). How it is
checked: whole (assigner, keeper), a restarted assigner holds no want it did not hold before, and a restarted
keeper holds the healers' attempts it held.
- §557 **A whole test's bounds are settings.** Every bound a whole test waits on (§535's five minutes, §534's 30
minutes, §544's 60 minutes, §540's two hours, §551's 90 days, §554's 90 minutes) is a setting of the part that
times it, read at its start, its default the design's value, so a whole test sets it in seconds and never waits on
real time. The bus module's and the recovery module's whole cases in these lines are cases of each module's one
whole test against the real bus, within the test budget. Why: a whole test that waits hours runs never (review
round 1, running and testing's notes). How it is checked: the code review; each whole test of these lines runs in
under a minute.
- §558 **A field of a served verb's answer is an edge of kind 6.** Kind 6 (§374) orders a module after a version that
serves a verb or a word it uses, and a verb's presence alone says nothing of a field its answer gained, so a
consumer relying on a new field of a served answer could run before the provider's new version and fall back in
silence (the grants part's first view relies on `binding` in `mesh-broker`'s `list`). An `invokes` entry therefore
takes a long form, `{verb, fields}`: `verb` as today (`seat:<seat>.<verb>`), `fields` the names of its answer's
fields the consumer relies on. The manifest validator refuses a field the seat's definition does not give that
verb's answer, naming it. The delivery plan derives the edge of kind 6 from the holders of that seat to the
consumer, held until the holder runs a build whose catalog graph entry's seat definition (§194) gives the verb's
answer every named field; §374's entry of kind 6 records the verb and its fields, and `outdated` (§375) says it
behind when the delivered version no longer gives them. The grants part's manifest names `mesh-broker`'s `list`
with `fields: [binding]`, and every part of G relying on a field G adds to a served answer names it so in G's
change. Why: the module graph orders a consumer after what it relies on, and a field is part of the contract
(lead-47's question from the operator's "the module graph is for this", 2026-11-11). How it is checked: unit
(manifest validator), a field the definition lacks refused; unit (delivery plan), the grants part ordered after
a bus module whose definition gives `binding`, and not before.
@@ -105,7 +105,7 @@ One file of milestone G's part of [design 47](../47-delivery-from-commit-to-deli
asked it (1921's note of 2026-11-07: "the indexer reads its seat definitions at start"). How it is checked: in the
indexer's whole-module test, started with a definition on a contracts trunk that no event announced, the first
manifest claiming that seat is judged with it.
- §199 **Who holds a seat is applied by one rule, from what each node runs.** The node-engine and the grants part decide a
- §199 **Who holds a seat is applied by one rule, from what each node runs.** (§556) The node-engine and the grants part decide a
seat's pass by mesh-sdk's `SeatsToPassOn` (ADR 0660 §3), reading the catalog graph's entry of the build each assignment
runs, never a manifest (ADR 0660 §2; §442). What each node runs is heard, never asked: each part keeps it from the node-engine seat's events
`running.<node>.<module>` (*What each step says*), through a durable consumer of its own declared in its `consumes`,
@@ -329,7 +329,7 @@ later change of a kept data's form, not as a case of this milestone.
changes how it writes that data, or stops declaring it; a declaration naming none reads as form 0. The declaration
alone cannot tell a changed layout of the same bucket, so the holder says it. Only kept data whose `form` changes is
backed up and restored, so a part's other kept data, and every decision it holds, is never set back.
- §296 **It is backed up before the module moves.** When a want moves a module whose new manifest declares a kept data at
- §296 **It is backed up before the module moves.** (§555) When a want moves a module whose new manifest declares a kept data at
another `form` than the running one, or no longer declares it, the node agent applying it first asks the seat
`node-backup` of the node holding `mesh-broker` for a restore point of that kept data, and waits for its event
`ended`, as for any backup before the apply. The bus module's account takes it, as it takes the bus's snapshots
@@ -337,14 +337,14 @@ later change of a kept data's form, not as a case of this milestone.
before, and it holds the kept data whole, the position of each durable consumer the module reads with, and the
catch-up tool the newer manifest declares beside that kept data. A failed
backup holds that module as any failed backup does.
- §297 **A revert restores it before the reverted build starts.** When a want takes a module back across a change of a
- §297 **A revert restores it before the reverted build starts.** (§555) When a want takes a module back across a change of a
kept data's `form`, the node agent applying it stops the newer build, then asks that seat's verb `restore` (§405) for the restore point of that module,
kept data and form, waits for the seat's event `restored`, and only then starts the reverted build. The node agent
that restores is the one running when the revert starts. As every restore is, it is brought back beside the live data
and never over it: the live kept data of the newer form is set aside under a name of its own, and only the operator
removes it, so nothing written since is destroyed. The consumer's position is set back with it, so the reverted
build hears everything since as after any restart.
- §298 **The order of a delivery that crosses a form change** is the module graph's (*Order without a person*), with three
- §298 **The order of a delivery that crosses a form change** (§554) is the module graph's (*Order without a person*), with three
edges the assigner, which writes every want, keeps:
- the new versions of the `node-backup` holder and of the bus module run before any module whose kept data's form
changes is moved, since the restore point needs both;
@@ -391,12 +391,12 @@ later change of a kept data's form, not as a case of this milestone.
(for milestone G, its delivery records and its graph revisions) is set aside and not read again; an Approve asked
and answered since is asked again, and one the gate asked before still reads its answer. The revert's Approve ask
says this in those words, naming each kept data restored.
- §302 **Its tools.** (§517) The seat `node-backup`'s `backed-up` lists each module's restore points of its bus data, with the
- §302 **Its tools.** (§517, §555) The seat `node-backup`'s `backed-up` lists each module's restore points of its bus data, with the
kept data, the form and the fingerprint each was taken before, and the data set aside at each restore, with its
size; `restore` restores one; `discard-aside` removes data set aside by a restore, only on the operator's call, and
writes the act, who and why in the hand-act log of the seat `mesh-hand-acts`. `backed-up` and `restore` exist on
the seat today and gain the module's kept data on the bus as an item; `discard-aside` is new.
- §303 **How it is checked.** Whole (node-engine, backup holder, bus module): a want moving a module across a `form`
- §303 **How it is checked.** (§555) Whole (node-engine, backup holder, bus module): a want moving a module across a `form`
change, and one moving a module whose kept data declares no `form` to one that does, each take a restore point
first, keyed by module, kept data, form and fingerprint; a want taking it back restores it beside the live data and
the consumer's position before the reverted build starts, and the data set aside is listed with its size and
@@ -161,7 +161,7 @@ record it follows from, and replaces the builder's own choice where one was made
definitions in `mesh-connectivity-contracts` names `invokers`, `persons` and `sessions` in this milestone, since
sessions and persons call them today through the mesh MCP server. How it is checked: unit (definition reader), each
form accepted and any other entry refused; the grants part's table test grants each form.
- §337 **The bus module's restore points.** The verbs the holder of `node-backup` on the bus module's node calls to take
- §337 **The bus module's restore points.** (§555) The verbs the holder of `node-backup` on the bus module's node calls to take
and restore a module's kept data on the bus name `holder:node-backup@same-node`. Why: only that holder takes them
(*A backup before the apply: kept data on the bus*). How it is checked: a call from another node's `node-backup`
holder is refused by the bus.
@@ -406,7 +406,7 @@ record it follows from, and replaces the builder's own choice where one was made
at G's delivery (design 51, *Tagged versions*). Why: the operator, 2026-11-10 about 21:40 CET, no custom versions for
now, a tagged version being one; a run delivers nothing (ADR 0666 §18), so it needs no version; the digest's source
and the wait are the design's choice, round 2 of the review by architectural domain, 2026-11-10. How it is checked: test 103.
- §374 **A release records, per edge, the part it was built against or delivered after.** One version per module
- §374 **A release records, per edge, the part it was built against or delivered after.** (§558) One version per module
stays (§52), and an edge is judged by the digest of the part it points to. The catalog records on each released
version (§349), in its key of `versions`, one entry per catalog-graph edge of the module's artifacts (the six kinds
of *Dependencies between artifacts*), a lockfile per released version. An entry names the kind, whether the edge
@@ -116,7 +116,7 @@ line says so.*
### Kept data on the bus, restore points and the reader release
- §405 **The bus module's restore points and node-backup's verbs.** (§516, §517) The seat `mesh-broker` gains `snapshot-kept` and
- §405 **The bus module's restore points and node-backup's verbs.** (§516, §517, §554, §555) The seat `mesh-broker` gains `snapshot-kept` and
`restore-kept`, of class `act`, their callers `holder:node-backup@same-node` alone (§337): the bus module's account is
the one that moves bus data, so `restore-kept` sets the live kept data aside under a name of its own and then
restores the restore point beside it, with the argument `aside-only` it sets the live data aside and restores
@@ -132,7 +132,7 @@ line says so.*
choice, one answer to three preparers (Q8, S9, Q-H6, C19; lead-47's ruling, 2026-11-11). How it is checked: test 36;
a call of `snapshot-kept` from another node's `node-backup` holder, and a `discard-aside` from any caller but the
operator, are refused by the bus; a failed restore says `restored` with `failed` and its error.
- §406 **A catch-up names its tool and arguments.** (§516) A restore point records the catch-up tool and its arguments whole
- §406 **A catch-up names its tool and arguments.** (§516, §555) A restore point records the catch-up tool and its arguments whole
(§299), and the fingerprint of the newer build it was taken before; the grants part derives the grant of that tool to
`holder:node-engine@same-node` from the catalog graph's entry of that recorded fingerprint (§443). Why: the design's
choice (Q-H7). How it is checked: whole (node-engine), a restore runs its recorded catch-up, granted from the
@@ -380,10 +380,10 @@ line says so.*
- §449 **The grants part's fill events say what `mesh-wants`' say.** The seat `mesh-grants`' events `filling` and `room`
carry the same body as `mesh-wants`' fill events: `store`, `bytes`, `cap` and `at` (§181, §331). Why: develop
question D-4, 2026-11-11. How it is checked: the seat's fixtures, against `mesh-wants`'.
- §450 **`ask-again` and `aside` answer.** `operator-channel`'s `ask-again` (§394) takes `key`, the ask's, and answers
- §450 **`ask-again` and `aside` answer.** (§555) `operator-channel`'s `ask-again` (§394) takes `key`, the ask's, and answers
`state`, the ask's state after it, or `error`; `node-backup`'s `aside` (§405) answers `aside`, the name the data was
set aside under, or `error`. Why: develop question D-5, 2026-11-11. How it is checked: the seats' fixtures.
- §451 **The forms of `owner`, `waiting-on` and a catch-up.** A finding's `owner` and `waiting-on` (§434, §439) are each
- §451 **The forms of `owner`, `waiting-on` and a catch-up.** (§540) A finding's `owner` and `waiting-on` (§434, §439) are each
a string: a person or a session as who@node, or a seat's name. A restore point's catch-up (§406) is `{tool,
arguments}`: `tool` an address as the mesh MCP server names one, `arguments` an object. Why: develop question D-6,
2026-11-11. How it is checked: unit (mesh-conditions' contract, node-backup's contract), each form read and another
+3 -1
View File
@@ -23,6 +23,7 @@ decisions:
- "ADR 0641"
- "ADR 0639"
- "ADR 0636"
- "ADR 0626"
- "ADR 0635"
- "ADR 0633"
- "ADR 0625"
@@ -76,7 +77,7 @@ file on 2026-11-10, and is cited as "design 47 G §n". A sentence that holds a d
stayed in the main file, and G builds what it says from G's delivery. A section named in italics is in this part, or,
when it is not here, in the main file.
The part is thirteen files, each read whole, its design lines numbered across them:
The part is fourteen files, each read whole, its design lines numbered across them:
- [G-1-pipelines-registration-and-steps](G-1-pipelines-registration-and-steps.md): pipelines per module: registration, watched branches, steps and policy, what holds a delivery, approval, groups, dependencies, a publish.
- [G-2-pipelines-sizes-checks-and-tools](G-2-pipelines-sizes-checks-and-tools.md): pipelines per module: any size, the pipeline makes the check, who may say what, health, recovery, tools, kept data, how G itself is delivered, the choices under ADR 0666 and ADR 0667.
@@ -91,3 +92,4 @@ The part is thirteen files, each read whole, its design lines numbered across th
- [G-11-develop-answers-review](G-11-develop-answers-review.md): the operator's answers of 2026-11-11 noon, the review of §456 to §479, and develop's next questions (§480 to §515).
- [G-12-develop-answers-fourth](G-12-develop-answers-fourth.md): the operator's answers of 2026-11-11 noon, round 3's leftovers and develop's fourth batch of questions (§516 to §553).
- [G-13-review-round-3](G-13-review-round-3.md): the review of §480 to §515, rounds 2 and 3 (§600 on).
- [G-14-recovery-module-and-review](G-14-recovery-module-and-review.md): the recovery module, the operator's answer of 2026-11-11 12:55 CET, and the review of §516 to §553 (§554 to §558).