From 437523554efb0af27046fde19ee35fdac29f24ea Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 15 Oct 2026 19:55:11 +0200 Subject: [PATCH] Say rollout, not walk, for the sending of a batch across nodes (issue 646) --- 00-META/glossary.md | 32 +-- 03-DESIGN/01-to-be/00-work-breakdown.md | 2 +- 03-DESIGN/01-to-be/09-the-node-lifecycle.md | 6 +- 03-DESIGN/01-to-be/10-delivery.md | 10 +- 03-DESIGN/01-to-be/20-writing-a-module.md | 2 +- 03-DESIGN/01-to-be/24-the-secrets-vault.md | 2 +- 03-DESIGN/01-to-be/25-the-bus-on-nats.md | 2 +- 03-DESIGN/01-to-be/28-building-the-bus.md | 2 +- .../30-the-mesh-updates-itself-on-a-push.md | 4 +- .../01-to-be/32-what-a-module-declares.md | 6 +- .../01-to-be/43-backups-against-mistakes.md | 4 +- .../45-a-core-that-cannot-fail-silently.md | 20 +- .../46-the-conversation-with-the-operator.md | 38 +-- .../47-delivery-from-commit-to-delivered.md | 244 +++++++++--------- .../48-a-module-says-how-it-is-healthy.md | 2 +- 03-DESIGN/01-to-be/49-the-mesh-in-domains.md | 2 +- .../54-the-controllers-records-on-the-bus.md | 22 +- 03-DESIGN/01-to-be/55-the-knowledge-base.md | 2 +- .../57-building-config-and-provisioning.md | 14 +- 03-DESIGN/01-to-be/README.md | 4 +- .../00-report.md | 2 +- .../00-report.md | 2 +- .../00-report.md | 2 +- .../00-report.md | 2 +- .../01-diagnosis.md | 2 +- .../00-report.md | 2 +- .../00-report.md | 2 +- .../00-report.md | 2 +- .../00-report.md | 2 +- .../00-report.md | 2 +- .../01-diagnosis.md | 2 +- .../00-report.md | 2 +- .../00-report.md | 2 +- .../00-report.md | 6 +- .../01-diagnosis.md | 10 +- .../00-report.md | 2 +- .../01-diagnosis.md | 2 +- .../00-report.md | 18 +- .../01-diagnosis.md | 10 +- .../01-diagnosis.md | 2 +- .../01-diagnosis.md | 2 +- .../00-report.md | 10 +- .../01-diagnosis.md | 10 +- .../00-report.md | 4 +- .../01-diagnosis.md | 12 +- .../00-report.md | 10 +- .../01-diagnosis.md | 18 +- .../00-report.md | 16 +- .../01-diagnosis.md | 22 +- .../00-report.md | 14 +- .../01-diagnosis.md | 60 ++--- .../00-report.md | 4 +- .../00-report.md | 2 +- .../00-report.md | 6 +- .../01-diagnosis.md | 16 +- .../00-report.md | 2 +- .../01-diagnosis.md | 4 +- .../01-diagnosis.md | 2 +- .../00-report.md | 14 +- .../01-diagnosis.md | 16 +- .../00-report.md | 20 +- .../01-diagnosis.md | 16 +- .../00-report.md | 6 +- .../01-diagnosis.md | 22 +- .../00-report.md | 4 +- .../01-diagnosis.md | 2 +- .../00-report.md | 8 +- .../01-diagnosis.md | 12 +- .../00-report.md | 6 +- .../00-report.md | 28 +- .../01-diagnosis.md | 42 +-- .../00-report.md | 6 +- .../01-diagnosis.md | 14 +- .../00-report.md | 2 +- .../01-diagnosis.md | 6 +- .../00-report.md | 8 +- .../00-report.md | 2 +- .../00-report.md | 8 +- .../01-diagnosis.md | 74 +++--- .../00-report.md | 48 ++-- .../01-diagnosis.md | 46 ++-- README.md | 2 +- 82 files changed, 561 insertions(+), 561 deletions(-) diff --git a/00-META/glossary.md b/00-META/glossary.md index 4490cf86..c64dad16 100644 --- a/00-META/glossary.md +++ b/00-META/glossary.md @@ -178,7 +178,7 @@ every other domain has something to run on. **Recorded by** the controller and t (ADR 0229). - **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its - records (nodes, assignments, walks, builds, settings), after each change commits in its store and only + records (nodes, assignments, rollouts, builds, settings), after each change commits in its store and only when the value differs, with one event per write. The store stays the record; the self-check compares the two and raises `mirror-drifted` on any difference (ADR 0289). Shortened to *mirror* where the @@ -306,7 +306,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences` **Purpose.** To take one commit from its pull request to every node that should run it, and to put it back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the -planner, the walk). **Downstream of** Module, Core, Health and repair and Data. +planner, the rollout). **Downstream of** Module, Core, Health and repair and Data. **Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323. **Uses:** module, node, declaration and send (Placement), health (Health and repair), person. @@ -348,7 +348,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data. - **delivery** — one commit in one repository on its way to the nodes, from its pull request's head being announced to one of its final states: one pull request, one status, one note (ADR 0239). - From its merge on it is delivered by the walk of the batch that answers it, and ends with that walk + From its merge on it is delivered by the rollout of the batch that answers it, and ends with that rollout (ADR 0276). Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`, `held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`; @@ -357,9 +357,9 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data. mechanism, which the as-is designs describe; it is never a word for a delivery. *Not:* ~~deployment~~ (hq) - **covered** — the end of a delivery that a later merge containing it delivered: every module it moves was - sent to every machine running it by the walk of a later merge into the same trunk, delivered on its own + sent to every machine running it by the rollout of a later merge into the same trunk, delivered on its own passing check, whose merge commit the forge says contains this one. Not *superseded*, which is a delivery - whose walk a newer delivery took over before it ended; nor *delivered*, which its own walk or an empty plan + whose rollout a newer delivery took over before it ended; nor *delivered*, which its own rollout or an empty plan ends in (ADR 0270). - **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One @@ -372,18 +372,18 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data. - **merge window** — the rolling wait in which the controller collects merges before it plans them: each merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest 10 minutes after its first merge (ADR 0276). -- **batch** — the merges one merge window collected, planned once as one walk. For a repository merged more - than once in it, the walk takes the latest commit, and the earlier merges are answered by it. Not *group*, +- **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more + than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*, which is a delivery group. - **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat of the same name. It records and decides; the controller sends, judges and rolls back when it is asked. -- **walk** — the controller's sending of one batch's builds across nodes, at most one trunk commit per +- **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236, - ADR 0276). One walk is open at a time; it names every merge it answers. A primitive the delivering stage - asks for, not an object anyone manages. A **tier** is one step of a walk: a module, then the modules - built against it. The controller's verb `plans` lists the walks. Walking is not a noun of its own: - a module *rolls out* by its `roll` policy. - *Not:* ~~rollout~~ (hq) + ADR 0276). One rollout is open at a time; it names every merge it answers. A primitive the delivering stage + asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules + built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll` + policy. + *Not:* ~~walk~~ (hq) - **first-node gate** — the judgement on a delivery's first node: healthy three times, no witness put it back, no condition raised since the send (ADR 0236 §2). Not *the gate* bare (see [Homonyms](#homonyms)). - **core module** and **leaf module** — a core module is one on the controller's own path (the controller, @@ -394,7 +394,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data. the core modules add mesh-delivery and the resolver. - **delivery time** — from a delivery's merge to every node of its deploy plan that is heard from running its modules' builds and reporting them healthy. Its **delivery budget** is what it is held to: three minutes for - a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the walk's time, which ends at the send. A + a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the rollout's time, which ends at the send. A measure inside the fix budget, not the operator's target (ADR 0313). - **fix time** — from an issue's first move to `fixing` to the moment the last of its fixes is delivered: the operator's measure of a fix, the agent's work, the review and the checks included, the delivery time its last @@ -672,7 +672,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical. |---|---|---|---| | plan | what the controller would send one node | that node's **declaration** (verb `plan`) | Placement | | | what a delivery would do to the mesh | **delivery plan** | Change and delivery | -| | the sending of one batch of commits across nodes | **walk** (verb `plans`) | Change and delivery | +| | the sending of one batch of commits across nodes | **rollout** (verb `plans`) | Change and delivery | | | a step a person starts, like the bus's | **planned step** | Core | | membership | the subjects a bus account is issued | **membership** | Provisioning | | | an account in a group of the machine | **account group** | Placement | @@ -686,7 +686,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical. | | the pull request status | **merge gate** | Change and delivery | | | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement | | tier | a level of the mesh | **layer** | Core | -| | a step of a walk | **tier** | Change and delivery | +| | a step of a rollout | **tier** | Change and delivery | | | how much proof an ask needs | **assurance level** | Identity and access | | ask | a request for the operator's input | **ask** | Operator and conversation | | | an entry in the build queue | **build request** | Change and delivery | diff --git a/03-DESIGN/01-to-be/00-work-breakdown.md b/03-DESIGN/01-to-be/00-work-breakdown.md index f3f5c6f0..606b9604 100644 --- a/03-DESIGN/01-to-be/00-work-breakdown.md +++ b/03-DESIGN/01-to-be/00-work-breakdown.md @@ -340,7 +340,7 @@ once, and maintain for a year. The modules are the input; a person reads what on writes what it declares tomorrow. **It also changes what "safe" means for the system being retired.** A fix to it has to be safe on -its own, because there is no careful walk to sequence it into: the thing is being switched off +its own, because there is no careful rollout to sequence it into: the thing is being switched off by hand, not managed into retirement. A change needing three steps in the right order is a change that will be half-applied. diff --git a/03-DESIGN/01-to-be/09-the-node-lifecycle.md b/03-DESIGN/01-to-be/09-the-node-lifecycle.md index 78e65c65..d34fefc2 100644 --- a/03-DESIGN/01-to-be/09-the-node-lifecycle.md +++ b/03-DESIGN/01-to-be/09-the-node-lifecycle.md @@ -54,7 +54,7 @@ rather than two kinds of thing node** — it is a machine with a program on it that has not been told which mesh it belongs to. There is no state for *the first node*. That is the point of -ADR 0004: the first node walks the +ADR 0004: the first node rollouts the same path, in an unusual order. --- @@ -239,7 +239,7 @@ carries, to the machine it is already on. **After step 3 the first node is not special in any way**, which is the property `adopt.sh` and the bootstrap script never had. Its specialness lasted two commands. -**And enrolment is exercised on node one.** The path every other node depends on is walked +**And enrolment is tried on node one.** The path every other node depends on is exercised immediately, against a controller on the same machine, rather than being written and first used months later on node two. @@ -524,7 +524,7 @@ remains locally authoritative for *operating*; the copy exists only for this. The node-engine is delivered like anything else (ADR 0010), and this is worth -walking through because tier 0 looks like it should be special and is not. +going through step by step because tier 0 looks like it should be special and is not. ``` push to mesh-host diff --git a/03-DESIGN/01-to-be/10-delivery.md b/03-DESIGN/01-to-be/10-delivery.md index 2430902d..19681764 100644 --- a/03-DESIGN/01-to-be/10-delivery.md +++ b/03-DESIGN/01-to-be/10-delivery.md @@ -277,15 +277,15 @@ A transition the table does not hold is refused. Two or more deliveries that sha together as one future state of the mesh. **This answers the first open question above.** A fit artifact does not declare itself. Off the trunk it is -only checked. On the trunk it is published, and its walk across the machines starts when its delivery says +only checked. On the trunk it is published, and its rollout across the machines starts when its delivery says so: by itself for the core, which cannot wait for anything to be repaired, and by `mesh-delivery` for everything else while that module is held. A person can always deliver by hand. The controller keeps the comparison, the planner, the gate, sending and rollback. `mesh-delivery` keeps the record and the order, and asks. -**Merges are walked in batches** (2026-10-10, by ADR 0276). The controller collects +**Merges are rolled out in batches** (2026-10-10, by ADR 0276). The controller collects merges in a rolling merge window, 90 seconds by default and at most 10 minutes, and plans each batch once, as -one walk with at most one commit per repository: the latest of its branch, which contains the earlier ones. -One walk is open at a time, and a walk that started is never superseded. `plans` shows a batch while it -assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's walk. +one rollout with at most one commit per repository: the latest of its branch, which contains the earlier ones. +One rollout is open at a time, and a rollout that started is never superseded. `plans` shows a batch while it +assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's rollout. diff --git a/03-DESIGN/01-to-be/20-writing-a-module.md b/03-DESIGN/01-to-be/20-writing-a-module.md index 8fff0307..c79d04b6 100644 --- a/03-DESIGN/01-to-be/20-writing-a-module.md +++ b/03-DESIGN/01-to-be/20-writing-a-module.md @@ -215,7 +215,7 @@ and recreate a container naming a step after the step ran. *Written 2026-10-02, from ADR 0189 and [issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).* -Some work cannot be done underneath a running service: an artifact store's collector walks the +Some work cannot be done underneath a running service: an artifact store's collector rollouts the storage and requires every writer stopped. A `run-once` step runs *beside* containers and a scheduled one is the same container again, so until this a module had no way to say it — and the mesh inherited a store that has never collected anything, because the predecessor said it with a diff --git a/03-DESIGN/01-to-be/24-the-secrets-vault.md b/03-DESIGN/01-to-be/24-the-secrets-vault.md index 9475e36e..5bfadea9 100644 --- a/03-DESIGN/01-to-be/24-the-secrets-vault.md +++ b/03-DESIGN/01-to-be/24-the-secrets-vault.md @@ -163,7 +163,7 @@ again restarts nothing. The provider asks when it starts, when the software's va consumer that binds in between holds a value the mesh made, which the software refuses, for at most the hour until the next ask. A change is recorded in the hand-act log as a decision, not a repair, and `rotate` of such a provision is refused. *How it is checked:* the tests ADR 0305 §13 names, in both repositories; the live row is `hand-acts` showing `secret issued` for -the media server's provision once both walks are done. +the media server's provision once both rollouts are done. ## Beyond generate and hold diff --git a/03-DESIGN/01-to-be/25-the-bus-on-nats.md b/03-DESIGN/01-to-be/25-the-bus-on-nats.md index 4ca3cd83..f6896cee 100644 --- a/03-DESIGN/01-to-be/25-the-bus-on-nats.md +++ b/03-DESIGN/01-to-be/25-the-bus-on-nats.md @@ -75,7 +75,7 @@ and uses, and the subjects, streams, consumers and permissions all follow from t (ADR 0039). **Added 2026-10-10** (ADR 0289): the controller's own records join -the bus as **record mirrors** — nodes, assignments, walks, builds and settings, each a bucket written from +the bus as **record mirrors** — nodes, assignments, rollouts, builds and settings, each a bucket written from the controller's store only when the record changed, with one event per write, and audited against the store on every self-check run. Designed in [to-be 54](54-the-controllers-records-on-the-bus.md). The rule it states for every bucket, that a write is made only when what the bucket describes changed, holds diff --git a/03-DESIGN/01-to-be/28-building-the-bus.md b/03-DESIGN/01-to-be/28-building-the-bus.md index 1d1e5871..590d4382 100644 --- a/03-DESIGN/01-to-be/28-building-the-bus.md +++ b/03-DESIGN/01-to-be/28-building-the-bus.md @@ -709,7 +709,7 @@ healthy while reacting to nothing. Found on the way: **no build had ever recorded what it stood on.** A recipe reads its base from a build argument, so the digest was never in the file the builder derived edges from, and every order that says *bases first* — `build --on`, `build --behind`, the merge follow-up of - [issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — walked a + [issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — traversed a graph with no edges. The builder now reports the bases it was handed, the controller records them by artifact path, and the graph is read from the newest build of each module — a recorded manifest carries no `build.on`, so the edge is derived from the build or it does not exist. diff --git a/03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md b/03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md index 77409778..80f05753 100644 --- a/03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md +++ b/03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md @@ -141,7 +141,7 @@ Revision, ADR 0218. Three rules on how a plan delivers what it built. A membership that could not be issued fails the send. - **One machine first.** Unless a module's upgrade policy says *together*, a plan sends it to one machine, the first by name, and to the rest only once that machine reports the new declaration applied and - current. A first machine that fails stops the module's walk there, with the reason in the plan. + current. A first machine that fails stops the module's rollout there, with the reason in the plan. - **A newer merge takes over.** A merge's plan supersedes every older open plan for the same repository and branch, and takes in the modules they had not yet built. A plan that waits on nothing can be closed by hand, by its id. @@ -182,7 +182,7 @@ policy or a plan is holding back builds are not known is left too. The push names the machine, the module, the builds and the reason, and says `push ` sends it. - The machine a push names, a push of every machine, and `push --behind` send held builds as before. - Walking a change through the mesh under `record` is one `push ` per machine. + Rolling out a change through the mesh under `record` is one `push ` per machine. ## Why now, and why not yet diff --git a/03-DESIGN/01-to-be/32-what-a-module-declares.md b/03-DESIGN/01-to-be/32-what-a-module-declares.md index 691f146e..50e256dd 100644 --- a/03-DESIGN/01-to-be/32-what-a-module-declares.md +++ b/03-DESIGN/01-to-be/32-what-a-module-declares.md @@ -253,7 +253,7 @@ the controller creates from the catalogue, mesh-wide, existing from registration watch before the owner runs anywhere ([design 25](25-the-bus-on-nats.md) §3). Every instance of the owner writes; a reader reads and watches. A key may name a machine by the module's own convention; the mesh keeps one bucket per name, not one per machine, because "every server, for every machine" -is then one list rather than a walk. +is then one list rather than a rollout. What a module sees is what it named. Its assignment's membership lists its buckets by those names, with whether it may write, and the runtime answers `get`, `put`, `delete`, `keys` and `watch` for @@ -338,7 +338,7 @@ module, version, digest. An event again: a fact about the builder. **The build cascade is that event fanning out through a graph the mesh already has.** A module whose image is built *on* another's artifact declares that in `build.on`. So `built` reaches the -controller, which walks the declared graph and submits rebuild jobs for everything downstream. A +controller, which traverses the declared graph and submits rebuild jobs for everything downstream. A dependency cascade is not special machinery — it is one event, one derived graph, and the same job queue. @@ -351,7 +351,7 @@ entrypoint that brings its state to the shape that version needs — the same vo tools and its provisioner, and nothing about how a machine runs it. The mesh runs that entrypoint as it runs the module's own code, to completion, in the module's own context, and a version whose preparation did not succeed does not run: the step gates that module and nothing else on the machine -(ADR 0136), and the walk stops +(ADR 0136), and the rollout stops at the first machine that did not take it (ADR 0135, superseding ADR 0133). diff --git a/03-DESIGN/01-to-be/43-backups-against-mistakes.md b/03-DESIGN/01-to-be/43-backups-against-mistakes.md index f02c96f2..cae17eb4 100644 --- a/03-DESIGN/01-to-be/43-backups-against-mistakes.md +++ b/03-DESIGN/01-to-be/43-backups-against-mistakes.md @@ -53,9 +53,9 @@ bucket, runs a bad migration — and not for the day a disk dies (ADR 0214). *Added 2026-10-06, ADR 0233.* The mesh composes a second file for the holder: every data item every module on the machine declares, with its class, its path, the path that covers it and its protection. Once an hour and after every -night the holder measures each item that is not a cache, by the item's `measure`: a bounded walk at +night the holder measures each item that is not a cache, by the item's `measure`: a bounded rollout at most daily for a small item (stopped after ten minutes or two million files and said as partial), a ZFS -dataset's own counters hourly, or the top-level entries only — never an hourly walk of anything large — +dataset's own counters hourly, or the top-level entries only — never an hourly rollout of anything large — and reads the redundant storage it is on (a ZFS pool's health, an md array's members, a btrfs filesystem's error counters). `backed-up` says each item with that measurement and its newest good backup. The holder judges nothing: the controller's self-check (to-be 45, D13) keeps the diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index 2aee0d1b..5244a1a0 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -101,7 +101,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone | the operator-channel's open messages | the seat's holder | its own key-value state | — | | a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps | | the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads | -| the record mirrors: nodes, assignments, walks, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) | +| the record mirrors: nodes, assignments, rollouts, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) | | a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why; an ask is delivered again only when the controller made it itself, every other ask is only dropped (ADR 0293) | A bus subject two components may publish on is refused when the controller composes grants, unless @@ -176,7 +176,7 @@ process of its own reads them back from the store. *Added 2026-10-08, ADR 0253:* **a condition says itself to the operator in plain words, beside its summary, and says whether the operator is needed.** The headline, needs, explanation, resolved line and actions are made where the condition is -made: by the producer when it knows more than the subject (a walk's modules, a failed unit's name, a +made: by the producer when it knows more than the subject (a rollout's modules, a failed unit's name, a hand-act's cause), otherwise by the wording registered for the kind, otherwise from the scope, which is reported. The explanation opens with its verdict. All of it is held to the plain rule when the controller takes an observation: no hash, numbered id, dotted name, duration in code, clock time, flag, markup or line @@ -267,10 +267,10 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age | S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — | | S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | `facts-stale` | warning | — | | S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — | -| S16 | a walk waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose walk waits | 30 min; urgent after 4 h | `waiting` (names the walk and `plans go`) | warning | — | -| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a walk out | warning | — | -| S18 | a batch of merges is cut into its walk (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no walk is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — | -| S19 | a batch waiting behind an open walk is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a walk is open | that walk's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the walk) | warning | — | +| S16 | a rollout waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose rollout waits | 30 min; urgent after 4 h | `waiting` (names the rollout and `plans go`) | warning | — | +| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a rollout out | warning | — | +| S18 | a batch of merges is cut into its rollout (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no rollout is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — | +| S19 | a batch waiting behind an open rollout is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a rollout is open | that rollout's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the rollout) | warning | — | The bus advisories (S9) cost one read-only subscription: the server already publishes them. The controller translates each into a condition naming the thing in the mesh's words, as the refused reply @@ -540,7 +540,7 @@ except an act recorded by a verb that is a person's decision by design, which is counts, whatever cause it gives. The controller keeps one table of the verbs that write the log, and each says whether it records a repair or a decision, and why: approving or rejecting a retirement and deleting what was retired (ADR 0230); `bus upgrade`, because the bus is never rolled by the mesh, and -`upgrade release-backlog`, because after a failed walk the next opens only on a person's word +`upgrade release-backlog`, because after a failed rollout the next opens only on a person's word (ADR 0236); and `secret rotate` when its cause is a leak (`leaked-in-logs`) — a person judges what was disclosed, one leak rotates several values, and a leak that recurs is a defect of the module that prints them, an issue against it, not a healer that rotates. A rotation for any other cause counts. A @@ -619,7 +619,7 @@ the other machines follow. "Reported applied" is not enough. **As built** (ADR 0236): -- **Every walk is gated, not only the core's.** A module is judged by its own health: it reported +- **Every rollout is gated, not only the core's.** A module is judged by its own health: it reported applied, no witness put it back, no condition was raised since its send about the machine or about the module there, and its tools are served on that machine where it has tools. Three healthy judgings at least forty seconds apart and two minutes after the send, within ten minutes of it. A policy of @@ -642,7 +642,7 @@ the other machines follow. "Reported applied" is not enough. - **No build reaches a machine without a gate**: a gated send carries and judges everything waiting on its machine; every other send — a plan's rest, a cascade, a healer's, a whole-mesh push — is refused or leaves the machine while a build no gate has seen waits there; a rebuild with the same artifacts and - manifest is no move. What waits is moved by a **walk**, one machine at a time, the control + manifest is no move. What waits is moved by a **rollout**, one machine at a time, the control node last, each judged; one that fails holds the next until `upgrade release-backlog --why`. - **A wait for a person passes, and each module of a send keeps its own pass** (ADR 0254): a module whose only fault is a new login the operator owes @@ -938,7 +938,7 @@ and "the commit is the build at hand"), on branches, not yet merged: | hq | its own `merge-check.sh`, running `records.py`, `index.py` and `cycle.py` | **Not yet:** the delivery plan kept (id, inputs, result) and the release comparing its plan with it; the -state machine replacing the walks' states, with its table, its test, its conditions and healer H2; +state machine replacing the rollouts' states, with its table, its test, its conditions and healer H2; the status's link to the kept plan; the notes under `refs/notes/mesh-plan`; check builds in a scratch namespace (the check builds no module today, so there is none to keep apart yet); the statuses made required — prepared as calls of the forge module's tool, applied by the operator. diff --git a/03-DESIGN/01-to-be/46-the-conversation-with-the-operator.md b/03-DESIGN/01-to-be/46-the-conversation-with-the-operator.md index d9b060f1..0fa388a9 100644 --- a/03-DESIGN/01-to-be/46-the-conversation-with-the-operator.md +++ b/03-DESIGN/01-to-be/46-the-conversation-with-the-operator.md @@ -327,36 +327,36 @@ Presence facts are events on the bus. The router keeps them as **current state o machine and per intake kind, overwritten — never as a history, and never in a message's words. A consumer that wants a timeline of the operator's day is a decision of its own. -### While a delivery walks +### While a delivery rollouts *2026-10-12, ADR 0308; the build is tracked by issue 532.* A delivery restarts what it moves, and those restarts raise conditions that clear once it is done; a router that restarted used to say again what it had been told to keep quiet (issue 531). So the router holds what a -walk touches and tells the walk's result once. +rollout touches and tells the rollout's result once. -- **A walk names what it touches.** The controller's walk events say, when a walk starts and again when it +- **A rollout names what it touches.** The controller's rollout events say, when a rollout starts and again when it ends with its outcome (delivered, failed or stopped), the modules it moves and the machines it sends them to. What it touches is those modules on those machines, and every module a moved tool runner or node-engine carries there. -- **The router holds what an open walk touches.** A message whose subject is a touched module or machine is +- **The router holds what an open rollout touches.** A message whose subject is a touched module or machine is held instead of sent, kept with when it was held, and `history` says it was held. A message several open - walks touch is held until the last of them ends or its bound passes. + rollouts touch is held until the last of them ends or its bound passes. - **Some messages are never held:** an ask (§6), a secret given or replaced, a message about anything no open - walk touches, and a walk's own failure. -- **A walk's end is told once:** one message with its outcome, the modules and machines it reached and how + rollout touches, and a rollout's own failure. +- **A rollout's end is told once:** one message with its outcome, the modules and machines it reached and how long it took, then each held message whose condition is still open, one line each. One whose condition - cleared during the walk is only counted. A walk delivered with nothing held and nothing open is one line, or + cleared during the rollout is only counted. A rollout delivered with nothing held and nothing open is one line, or nothing on a channel set to say only what needs the operator. -- **A hold has a bound:** twice the largest delivery budget in force among the walk's deliveries +- **A hold has a bound:** twice the largest delivery budget in force among the rollout's deliveries ([to-be 47](47-delivery-from-commit-to-delivered.md)): 20 minutes for a core module, 10 for leaf modules at - a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a walk none of whose - deliveries has a budget. Past it, every held message still open is sent with a line saying the walk still - runs and since when, and the cleared ones are counted. A walk that hangs never silences the mesh. + a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a rollout none of whose + deliveries has a budget. Past it, every held message still open is sent with a line saying the rollout still + runs and since when, and the cleared ones are counted. A rollout that hangs never silences the mesh. - **A restart says nothing it has said:** a router that restarts says again only what is open and was never answered, never a silenced message or one withheld for its content. -*How it is checked:* the controller's tests of the walk events, and the router's tests of holding, never +*How it is checked:* the controller's tests of the rollout events, and the router's tests of holding, never holding, the end message, the bound and the restart, as ADR 0308 lists them; live, the next delivery of the tool runner gives the operator one message at its end, and the channel's `history` shows what was held and released. @@ -467,13 +467,13 @@ the factors are the router's state. ### An approved settings change takes effect (ADR 0344) *2026-10-13, issue 641.* Approving a settings proposal is the act that makes it take effect. The -serving controller sets the layer and, in the same act, opens a **settings walk**: every machine +serving controller sets the layer and, in the same act, opens a **settings rollout**: every machine that runs the module and that the layer reaches, the control node last, each sent and its module -judged at the first-node gate before the next. A settings walk sends the layer and no build: every +judged at the first-node gate before the next. A settings rollout sends the layer and no build: every module on the machine stays at the build it was last sent, so a build held for a person, or waiting for a first-node gate, stays where it is. A send that would move a running module's data stops the -walk and names the `push --move` that sends it. The walk says its start and end like every walk, so -its end is told as every walk's is. A failure stops the walk and leaves the layer as approved; the +rollout and names the `push --move` that sends it. The rollout says its start and end like every rollout, so +its end is told as every rollout's is. A failure stops the rollout and leaves the layer as approved; the previous layer is in the settings history. The ask says all this before the Approve, and no longer names a push. A layer set by hand still waits for a push. @@ -481,9 +481,9 @@ A delivered file is not yet an applied one: a module whose program reads its set starts applies them itself (a service's `restart-on`, or a tool of its own). Betterbird is such a module; issue 643 makes it restart Betterbird at a quiet moment. -How it is checked: the controller's tests open the walk on the warrant, send one machine at a time +How it is checked: the controller's tests open the rollout on the warrant, send one machine at a time with every build kept and the module judged, stop before a data move and on a failure without -holding other walks back, and read the ask's words. +holding other rollouts back, and read the ask's words. ### Proofs and tiers diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index 7ba21d09..94b59e00 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -30,9 +30,9 @@ delivery group is two or more deliveries that share a head branch name, delivere are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.** (ADR 0239.) -**From its merge on, a delivery is delivered by the walk of its batch.** The controller collects merges in a -rolling merge window and plans each batch once, as one walk with at most one commit per repository, one walk -at a time. Every delivery the walk answers ends with it +**From its merge on, a delivery is delivered by the rollout of its batch.** The controller collects merges in a +rolling merge window and plans each batch once, as one rollout with at most one commit per repository, one rollout +at a time. Every delivery the rollout answers ends with it (ADR 0276). This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison @@ -53,7 +53,7 @@ commit's journey and in what order the steps are allowed. - **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants, registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk - rule, a gated send, the rollback and the walk of one trunk commit's tiers across machines. It still answers + rule, a gated send, the rollback and the rollout of one trunk commit's tiers across machines. It still answers every pull request's head with its check itself, so a check never depends on mesh-delivery. - **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the group, the state table and their state, and nothing else. @@ -61,9 +61,9 @@ commit's journey and in what order the steps are allowed. request, and appends to the commit's note under `refs/notes/mesh-plan`. - **The build seat** builds and checks, asked by the controller as before. **A change is built once** (ADR 0350): a check that passes also builds what its merge would move into the artifact store's scratch namespace, under a - reuse key (the source fingerprint and the builder's version), and the walk of the merge commit copies a build - of the same key, which the check's verdict names, by digest instead of building. Otherwise the walk builds and - says why, per module, in its record. Check builds go when the walk answering the merge ends, when the pull + reuse key (the source fingerprint and the builder's version), and the rollout of the merge commit copies a build + of the same key, which the check's verdict names, by digest instead of building. Otherwise the rollout builds and + says why, per module, in its record. Check builds go when the rollout answering the merge ends, when the pull request closes unmerged, and after seven days. ## A delivery @@ -76,7 +76,7 @@ Its id is `/@`. It holds: its diffset (what ADR 0238 called the "change plan", under the glossary's name); - its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its **machine steps** while delivering; -- once merged, **the commit it landed on the trunk as**, and the walk that delivers it: the walk of the batch +- once merged, **the commit it landed on the trunk as**, and the rollout that delivers it: the rollout of the batch that answers its merge, which may carry a later commit of its repository that contains it (see *The merge window and the batch*). @@ -93,9 +93,9 @@ allows. | checked | a verdict arrived | 1 minute | nothing: it is decided at once | | ready | it passed; it waits to be merged | none | — | | rejected | it failed its check | none | — | -| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a walk that does not answer its merge took over its walk; `go` when its walk started | +| published | merged on the trunk its modules follow, its rollout opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a rollout that does not answer its merge took over its rollout; `go` when its rollout started | | held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's | -| delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says | +| delivering | its rollout runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the rollout's record says | | delivered, failed, superseded, stopped, retired, covered | final | — | — | **A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head @@ -105,7 +105,7 @@ commits in the transition's reason; a `rejected` one stays `rejected`, its statu of date. Nothing is asked for it: a recheck, a new head or the merge tool asks the check again. The row from `proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by -mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own walk. +mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own rollout. What this leaves as it is, said so nobody reads more into it: @@ -125,7 +125,7 @@ What this leaves as it is, said so nobody reads more into it: The transitions are those of ADR 0239 decision 2, ADR 0270 and ADR 0323. A delivery that is merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A -delivery merged with no walk answering it within ten minutes is held too, with the true reason, and listed by +delivery merged with no rollout answering it within ten minutes is held too, with the true reason, and listed by `stalled` at once. Nothing waits silently. **What needs no person ends by itself** (ADR 0270): @@ -133,37 +133,37 @@ delivery merged with no walk answering it within ten minutes is held too, with t - *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and sends nowhere) is `delivered` as soon as it is merged, whatever its check said; - *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine - running it by the walk of a later merge into the same trunk, delivered on its own passing check, and the + running it by the rollout of a later merge into the same trunk, delivered on its own passing check, and the forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery` says `covered: a later merge that contains it was delivered` as a success. A module that runs on one - machine only is covered too: its walk records the empty rest as sent once its first machine passes + machine only is covered too: its rollout records the empty rest as sent once its first machine passes (ADR 0270, as corrected 2026-10-12); -- *a merge whose walk said its modules unchanged* moves nothing too, and is `delivered` when that walk ends, - whichever commit the walk carried (issue 611). A delivery plan made while another walk built a module reads - that module whole and names it; the walk, cut later, reads the build source that build recorded (ADR 0267) and - may find none of the changed files in it. The walk's record then says so in `delivery.unchanged`: each such +- *a merge whose rollout said its modules unchanged* moves nothing too, and is `delivered` when that rollout ends, + whichever commit the rollout carried (issue 611). A delivery plan made while another rollout built a module reads + that module whole and names it; the rollout, cut later, reads the build source that build recorded (ADR 0267) and + may find none of the changed files in it. The rollout's record then says so in `delivery.unchanged`: each such module with why, "its build source holds none of the changed files", and not in its modules, which keep - meaning what the walk builds. A delivery whose every module the walk built or said unchanged is delivered, - saying which matched what runs. One with a module the walk neither built nor said unchanged is read as below. + meaning what the rollout builds. A delivery whose every module the rollout built or said unchanged is delivered, + saying which matched what runs. One with a module the rollout neither built nor said unchanged is read as below. Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's `TestAMergeWhoseModulesMatchWhatRunsIsDelivered`; -- *a stop whose walk ended without building its modules* (issue 474) stays `stopped`, a final state, and +- *a stop whose rollout ended without building its modules* (issue 474) stays `stopped`, a final state, and is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after its stop` with the covering merge, once, as a success. No state or transition is added (issue 480); - the covered row is tried before `go`, from `held` and from `ready` past the ten minutes. **History is retired, not held** (ADR 0250). -A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no walk +A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no rollout of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes -to `retired` at adoption, before any check, hold or walk, and its pull request is not told. A held delivery that -fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its walk +to `retired` at adoption, before any check, hold or rollout, and its pull request is not told. A held delivery that +fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its rollout starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a build registered before its turn would be carried by somebody else's send. -**The machine steps.** While a delivery is `delivering`, each machine its walk reaches has a row with the +**The machine steps.** While a delivery is `delivering`, each machine its rollout reaches has a row with the module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy -readings so far), `passed`, `failed`, `rolled-back`. They are read from the walk's own record in the +readings so far), `passed`, `failed`, `rolled-back`. They are read from the rollout's own record in the controller's `plan-moved` event, never inferred from time. ## The merge window and the batch @@ -174,95 +174,95 @@ controller's `plan-moved` event, never inferred from time. Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its merges and its times are kept in the controller's store and survive a restart. -- **The cut.** When the window closes and no walk is open, the batch becomes one walk: one delivery plan for +- **The cut.** When the window closes and no rollout is open, the batch becomes one rollout: one delivery plan for every commit in it, tiered along the module graph across repositories, with at most one commit per - repository, the latest of its branch. The walk names every merge it answers: its own commits, and the + repository, the latest of its branch. The rollout names every merge it answers: its own commits, and the earlier merges of a repository that its later commit contains. -- **One walk at a time.** Merges that arrive while a walk is open join the next batch. A walk that started - is never superseded. A walk still waiting for its word when the next batch is cut is folded into the new - walk, which answers every merge the folded one answered. -- **The controller's own path.** A merge that moves a module whose walk waits for nobody's word (the controller, +- **One rollout at a time.** Merges that arrive while a rollout is open join the next batch. A rollout that started + is never superseded. A rollout still waiting for its word when the next batch is cut is folded into the new + rollout, which answers every merge the folded one answered. +- **The controller's own path.** A merge that moves a module whose rollout waits for nobody's word (the controller, the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its - walk like any other. An own-path walk folds no waiting catalogue walk; that walk's merges are batched again + rollout like any other. An own-path rollout folds no waiting catalogue rollout; that rollout's merges are batched again behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10). -- **A late merge.** A merge heard after a later merge of its repository and branch was put in a walk is - answered by that walk: the assembling batch, the open walk, or, when that walk was delivered, it ends +- **A late merge.** A merge heard after a later merge of its repository and branch was put in a rollout is + answered by that rollout: the assembling batch, the open rollout, or, when that rollout was delivered, it ends `covered` at once. -- **The ends.** A walk done ends every delivery it answers: `delivered` for its own commits, `covered` for - an earlier merge of a repository it carried later, naming that merge. A walk that failed fails its own - commits' deliveries and marks none delivered; each earlier merge it answered is then walked on its own - commit, the newest first, one walk each, until one is delivered. A walk stopped stops every delivery whose - merge it walks again nowhere (below). -- **A group inside a walk.** The pairs of a delivery group's order (ADR 0249) are edges of the walk's - tiers beside the module graph. mesh-delivery gives a walk its word only when every delivery it carries is +- **The ends.** A rollout done ends every delivery it answers: `delivered` for its own commits, `covered` for + an earlier merge of a repository it carried later, naming that merge. A rollout that failed fails its own + commits' deliveries and marks none delivered; each earlier merge it answered is then rolled out on its own + commit, the newest first, one rollout each, until one is delivered. A rollout stopped stops every delivery whose + merge it rollouts again nowhere (below). +- **A group inside a rollout.** The pairs of a delivery group's order (ADR 0249) are edges of the rollout's + tiers beside the module graph. mesh-delivery gives a rollout its word only when every delivery it carries is at its turn. -- **The bookkeeping.** mesh-delivery ties a walk to every live delivery whose merge it answers, follows a - folded walk to the walk that took it over, and never ties a walk to a delivery that has ended. A waiting - walk no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once. +- **The bookkeeping.** mesh-delivery ties a rollout to every live delivery whose merge it answers, follows a + folded rollout to the rollout that took it over, and never ties a rollout to a delivery that has ended. A waiting + rollout no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once. - **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped: novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated", - and a batch waiting behind an open walk names that walk. The controller says the batch on the bus as + and a batch waiting behind an open rollout names that rollout. The controller says the batch on the bus as `plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged delivery it holds. -- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no walk open, and a batch - waiting behind an open walk past that walk's bound, each raise a condition. A merged delivery that no walk - answers within ten minutes is held with the true reason (the open walks of its repository and that none +- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no rollout open, and a batch + waiting behind an open rollout past that rollout's bound, each raise a condition. A merged delivery that no rollout + answers within ten minutes is held with the true reason (the open rollouts of its repository and that none names its merge, or that the controller holds none) and listed by `stalled` at once. -## A stop on a shared walk +## A stop on a shared rollout *2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.* -One walk carries every merge of its batch, so ending a walk for one delivery ends it for every other +One rollout carries every merge of its batch, so ending a rollout for one delivery ends it for every other delivery on it. A stop therefore ends only the change it is for: -- **A group's stop.** When a member after a failed one is stopped and its walk is open, mesh-delivery asks - `delivery-stop` for that walk with the stopped members' merges to leave out, every member the same pass - stops on that walk in one ask. The controller ends the walk, said as stopped, and walks every other merge +- **A group's stop.** When a member after a failed one is stopped and its rollout is open, mesh-delivery asks + `delivery-stop` for that rollout with the stopped members' merges to leave out, every member the same pass + stops on that rollout in one ask. The controller ends the rollout, said as stopped, and rolls out every other merge it answered again. - **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too. Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is - walked again. -- **Where the others go.** They join the open batch of their kind, where merges heard while the walk ran - already wait, so an older commit never walks after a newer one; only with none open do they form a new + rolled out again. +- **Where the others go.** They join the open batch of their kind, where merges heard while the rollout ran + already wait, so an older commit never rollouts after a newer one; only with none open do they form a new batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop - before it is cut, are cut without the rest of their window. Its walk withholds the left-out merges' modules + before it is cut, are cut without the rest of their window. Its rollout withholds the left-out merges' modules where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every merge moved are written in one transaction. -- **The walk's record.** The stopped walk names the batch that walks its merges again, every merge it left - out, and each module it already sent that nothing walks again, with the machines it reached. A delivery - whose merge is walked again goes from `delivering` back to `published` (the row `rewalk`), or stays - `published`, and follows the fresh walk once it is cut. A delivery whose merge was left out, and every - delivery on a walk stopped whole, ends `stopped`, naming the walk and the stop. -- **A stopped merge never ships by a walk.** mesh-delivery asks every open walk or batch that names a stopped - delivery's merge to leave it out, and never gives such a walk its word. When the controller refuses, it - stops the whole walk. -- **A person's stop.** `stop` of a delivery whose open walk carries other deliveries does nothing at first: it +- **The rollout's record.** The stopped rollout names the batch that rolls out its merges again, every merge it left + out, and each module it already sent that nothing rollouts again, with the machines it reached. A delivery + whose merge is rolled out again goes from `delivering` back to `published` (the row `rewalk`), or stays + `published`, and follows the fresh rollout once it is cut. A delivery whose merge was left out, and every + delivery on a rollout stopped whole, ends `stopped`, naming the rollout and the stop. +- **A stopped merge never ships by a rollout.** mesh-delivery asks every open rollout or batch that names a stopped + delivery's merge to leave it out, and never gives such a rollout its word. When the controller refuses, it + stops the whole rollout. +- **A person's stop.** `stop` of a delivery whose open rollout carries other deliveries does nothing at first: it answers with each of them, which of them contain this merge and end either way, and two ways given with - `way`: `again` (the walk ended and the others walked again without this merge) or `walk` (the whole walk - stopped, every delivery on it with it). A delivery alone on its walk is stopped at once. A delivery is - marked stopped only once its walk is; a stop the controller refuses leaves it as it was. -- **What it does not do.** What the stopped walk already sent stays on the machines it reached, and its + `way`: `again` (the rollout ended and the others rolled out again without this merge) or `walk` (the whole rollout + stopped, every delivery on it with it). A delivery alone on its rollout is stopped at once. A delivery is + marked stopped only once its rollout is; a stop the controller refuses leaves it as it was. +- **What it does not do.** What the stopped rollout already sent stays on the machines it reached, and its asked builds still register, so a later send may carry them further; the stop names the machines, and putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the - stopped merge's modules, contains the stopped change and delivers it when walked; the stop names each it + stopped merge's modules, contains the stopped change and delivers it when rolled out; the stop names each it can see, and holding a merged change back for good is a revert's job. ## The delivery budget *Decided 2026-10-10 by ADR 0282. -Where it differs, this section replaces the window for every merge and the one walk at a time above, slice by +Where it differs, this section replaces the window for every merge and the one rollout at a time above, slice by slice as each slice below is built; until a slice is built, the text above is what runs.* **The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes; a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is -shown and held to no budget. Measured on 51 walks on 2026-10-09 and 10, a leaf walk took a median of 6.0 minutes -(7 of 20 within five), a core walk of one tier 5.6 (worst 8.8), and a controller walk dragging two catalogue tiers +shown and held to no budget. Measured on 51 rollouts on 2026-10-09 and 10, a leaf rollout took a median of 6.0 minutes +(7 of 20 within five), a core rollout of one tier 5.6 (worst 8.8), and a controller rollout dragging two catalogue tiers 14.2 (issue 338). *A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget @@ -273,56 +273,56 @@ phases — implementing, review, checks, and merge to running (the delivery time budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget. The delivery budget stays as written: a measure inside the fix budget, not replaced by it. -**What changes in the walk.** +**What changes in the rollout.** - **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged. -- **Independent walks run at once**: no shared repository, module or module-graph edge with any open walk. One - node judges one walk at a time; a send to a node under another walk's judgement waits for its verdict; no leaf +- **Independent rollouts run at once**: no shared repository, module or module-graph edge with any open rollout. One + node judges one rollout at a time; a send to a node under another rollout's judgement waits for its verdict; no leaf build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged. - **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf module with a long-running resource; one for a leaf module whose every resource is a tools bundle. - **A tier's first nodes are judged at once.** -- **The bus's grants arrive before the gate** (ADR 0298): when a walk's builds change the bus's user list (a new +- **The bus's grants arrive before the gate** (ADR 0298): when a rollout's builds change the bus's user list (a new tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before anything is judged; a step that would carry anything else is refused unsent and said on the gate line. - **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy report; the plan shows an expected landing time from the last seven days' medians. -*Walks beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what -a change shares).* Building the walks at once found what those bullets left open, and ADR 0306 decides it: +*Rollouts beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what +a change shares).* Building the rollouts at once found what those bullets left open, and ADR 0306 decides it: -- **A walk's builds are its own, never another's backlog.** A send composes a module whose build another open - walk owns at the build its node was last sent, and neither carries nor judges it: that build is moved and - judged only by its own walk, so a failed walk puts back only its own builds. What waits on a node with no open - walk to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17 - for builds an open walk owns). +- **A rollout's builds are its own, never another's backlog.** A send composes a module whose build another open + rollout owns at the build its node was last sent, and neither carries nor judges it: that build is moved and + judged only by its own rollout, so a failed rollout puts back only its own builds. What waits on a node with no open + rollout to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17 + for builds an open rollout owns). - **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the - merge window whether or not a batch is assembling, so a group's members walk in one batch, in their order. -- **Related means everything the walks would move.** A module in common is any module a walk plans to build or + merge window whether or not a batch is assembling, so a group's members roll out in one batch, in their order. +- **Related means everything the rollouts would move.** A module in common is any module a rollout plans to build or send, the dependents of what its merges moved included; a change whose modules are not known yet is related to - every change. Two related walks are neither cut beside each other nor judged beside each other on one node. + every change. Two related rollouts are neither cut beside each other nor judged beside each other on one node. - **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a - warning naming the walk and the core walks that set it aside. + warning naming the rollout and the core rollouts that set it aside. - **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an - unhealthy network or engine) fails every judgement open on it, leaf and core, each walk putting back only its + unhealthy network or engine) fails every judgement open on it, leaf and core, each rollout putting back only its own builds. -- **A rollback beside another walk's judgement** behaves as it does today; whether it should wait is tracked by +- **A rollback beside another rollout's judgement** behaves as it does today; whether it should wait is tracked by issue 528. -*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of walk A left out -of walk B's send and not taken up as backlog while A is open; a group's second member joining its first's batch; -overlapping dependents keeping two walks apart; the warning on the third set-aside; a failed apply failing every +*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of rollout A left out +of rollout B's send and not taken up as backlog while A is open; a group's second member joining its first's batch; +overlapping dependents keeping two rollouts apart; the warning on the third set-aside; a failed apply failing every open judgement on its node. *A busy container on the first node (2026-10-13, ADR 0335, citing ADR 0282 §1).* A container may declare `busy`, a command that says whether it holds work a replacement would lose; the build agent declares it, so a merge -into the controller's repository no longer throws away the checks in flight. When a walk's send would replace +into the controller's repository no longer throws away the checks in flight. When a rollout's send would replace such a container while it says busy, the node-engine leaves it running, applies the rest, and replaces it once -it says idle, or after 15 minutes at the latest. Only that node's walk waits: the walk says it waits for a busy +it says idle, or after 15 minutes at the latest. Only that node's rollout waits: the rollout says it waits for a busy container there, its first-node gate starts at the report that says the container replaced, and the wait is a phase of the delivery time of its own, never exempted from the delivery budget. A delivery it puts over budget is said over budget, with that wait as its longest phase. *How it is checked:* the controller's test that a -walk whose module's container waits as busy on its first node says so and starts that node's gate only once the +rollout whose module's container waits as busy on its first node says so and starts that node's gate only once the node reports it replaced, and the walk-phase test that the wait is a measured phase between the send and the judgement. @@ -330,11 +330,11 @@ judgement. | Part | Owner | What it adds | |---|---|---| -| the walk's phases and readings | mesh-controller | each phase's start and end and each reading in the walk and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration | +| the rollout's phases and readings | mesh-controller | each phase's start and end and each reading in the rollout and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration | | the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` | | the condition | mesh-controller | a self-check probe reading `times`, raising `delivery..over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` | | the lone merge | mesh-controller | the cut at once when nothing waits to join | -| walks at once | mesh-controller | the independence test, one judgement per node, a core walk's ground, a repository's next batch | +| rollouts at once | mesh-controller | the independence test, one judgement per node, a core rollout's ground, a repository's next batch | | judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once | | the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` | @@ -342,11 +342,11 @@ judgement. | Slice | Delivers | Done when | |---|---|---| -| 1 — the measure | the walk's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase | -| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its walk's phases | -| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its walk | -| 4 — walks at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other | -| 5 — the landing time | the expected landing time | `plans` shows it for an open walk, and the week's deliveries land within a minute of it at the median | +| 1 — the measure | the rollout's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase | +| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its rollout's phases | +| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its rollout | +| 4 — rollouts at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other | +| 5 — the landing time | the expected landing time | `plans` shows it for an open rollout, and the week's deliveries land within a minute of it at the median | Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges only once it is built. @@ -374,9 +374,9 @@ only once it is built. - **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`. A member published while its group's composed check is asked and not answered waits there for the answer; one held because that check had not passed is *lifted* back to `published` once it passes or warns for the - heads that merged and its own walk is still open (ADR 0270). + heads that merged and its own rollout is still open (ADR 0270). - **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its - walk while the walk's other merges are walked again (ADR 0299). The members before it stay delivered. + rollout while the rollout's other merges are rolled out again (ADR 0299). The members before it stay delivered. Its own builds were put back at its gate. A group's state is derived from its members and its composed check, and is shown, never stored apart from @@ -408,22 +408,22 @@ In this order, each by its owner: | Verb | Answers or does | |---|---| | `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group | -| `show` | one delivery or group whole: plan, transitions, machine steps, order, the walk | +| `show` | one delivery or group whole: plan, transitions, machine steps, order, the rollout | | `groups` | every group with its members in order and its derived state | | `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere | | `table` | the state table and the machine-step table, with bounds and H2's repairs | -| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting walk no live delivery holds | +| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting rollout no live delivery holds | | `recheck` | a rejected or ready delivery back to proposed, with why | | `release` | a held delivery to delivering, with why: a person's word | -| `stop` | any delivery not final to stopped, with why; its walk is ended through the controller. On a walk that carries other deliveries it first answers what ending the walk would take down, and acts on `way`: `again` or `walk` (ADR 0299) | +| `stop` | any delivery not final to stopped, with why; its rollout is ended through the controller. On a rollout that carries other deliveries it first answers what ending the rollout would take down, and acts on `way`: `again` or `walk` (ADR 0299) | | `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise | | `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing | **The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`, `delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which -takes the merges to leave out and walks the walk's other merges again without them, ADR 0299), +takes the merges to leave out and rolls out the rollout's other merges again without them, ADR 0299), `delivery-walks`. A person has `plans go --why`, which does what `deliver` does under their name and -is recorded as a hand act. The controller says every walk it keeps as `plan-moved`, from a verb's command +is recorded as a hand act. The controller says every rollout it keeps as `plan-moved`, from a verb's command too. **The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It @@ -438,12 +438,12 @@ up. ## The bootstrap -- A walk that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the +- A rollout that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the controller's own, started by the merge and witnessed on the machine. mesh-delivery records it. -- Any other walk waits for `deliver` only while the seat has a holder on record. With none on record, the +- Any other rollout waits for `deliver` only while the seat has a holder on record. With none on record, the controller starts it as it always did. -- A waiting walk is said by the controller whatever mesh-delivery says of itself: S16 raises - `plan..waiting` after 30 minutes, urgent after 4 hours, naming the merges the walk answers and +- A waiting rollout is said by the controller whatever mesh-delivery says of itself: S16 raises + `plan..waiting` after 30 minutes, urgent after 4 hours, naming the merges the rollout answers and `plans go --why` as the way on. A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone. @@ -453,22 +453,22 @@ up. merged and rolls out as the core does. With no holder on record it behaves exactly as before. 2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built and published. -3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Walks that start - from then on wait for its word. Walks already open finish on their own and are adopted as deliveries in +3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Rollouts that start + from then on wait for its word. Rollouts already open finish on their own and are adopted as deliveries in `delivering`. -4. To step back, unassign it. Waiting walks are then started by the controller at its next pass. +4. To step back, unassign it. Waiting rollouts are then started by the controller at its next pass. ## Phases | Phase | Delivers | Done when | |---|---|---| | A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` | -| B — the conditions | S16, the controller's own `plan..waiting`; probe D14 reads `stalled` and raises `delivery..stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a walk never let go is raised by S16 | -| C — the walk itself | the per-tier walk moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while | +| B — the conditions | S16, the controller's own `plan..waiting`; probe D14 reads `stalled` and raises `delivery..stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a rollout never let go is raised by S16 | +| C — the rollout itself | the per-tier rollout moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while | ## What is not decided here - A web view of deliveries beyond the pull request's comment. - Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge module's protection tool. -- Moving the per-tier walk out of the controller (Phase C). +- Moving the per-tier rollout out of the controller (Phase C). diff --git a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md index 16eaa8ca..1c5b7cd9 100644 --- a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md +++ b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md @@ -201,7 +201,7 @@ build alike, so any build of the module is excused while the wait checks out. **Each module of a send keeps its own pass** (ADR 0254). A send is still judged as one, with one verdict. The gate also counts, per module, the judgings in a row that found it healthy or waiting for a person; when the send fails, a module whose count reached the passes asked keeps a pass, said "on its own", and is -neither put back nor left as a move no gate has seen for every other walk to wait on. A module found broken +neither put back nor left as a move no gate has seen for every other rollout to wait on. A module found broken is marked failed and put back at once, and the send's verdict waits until the modules beside it have their own, or the bound, so every module of a failed send leaves it with a verdict: its own pass, or put back. diff --git a/03-DESIGN/01-to-be/49-the-mesh-in-domains.md b/03-DESIGN/01-to-be/49-the-mesh-in-domains.md index 01a93f91..6fe02208 100644 --- a/03-DESIGN/01-to-be/49-the-mesh-in-domains.md +++ b/03-DESIGN/01-to-be/49-the-mesh-in-domains.md @@ -31,7 +31,7 @@ shape — what each is for, where its records live, and how they depend on one a | **Placement** | to decide, send and apply what each node runs, and say why | the controller's `inventory` database | | **Provisioning** | to resolve which module serves a provision for which consumer, and wire the two | the controller | | **Identity and access** | to issue, hold, rotate and check credentials and grants | the controller's `identity` and `licences` databases, and the vault | -| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and walk | +| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and rollout | | **Health and repair** | to notice what is wrong, say it once, repair what may be repaired unattended, and record what was done by hand | the controller's condition store | | **Data** | to know every item of data a module holds, how precious it is, and keep it recoverable | the manifests' data declarations, and every node's `node-backup` seat | | **Connectivity** | to make every node and module reachable by name where it should be, and unreachable where it should not | the controller | diff --git a/03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md b/03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md index 14ec0d00..fa9f3d43 100644 --- a/03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md +++ b/03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md @@ -12,7 +12,7 @@ decisions: # 54 — The controller's records on the bus -**What each node runs, how far each walk has got, what each build is doing and what each setting holds +**What each node runs, how far each rollout has got, what each build is doing and what each setting holds can be watched on the bus, and what is watched never says something the controller's store does not** (ADR 0289, from gap G2 of research 036). @@ -51,7 +51,7 @@ do not have: a check that the bucket says what the store says. Each value is built from a named list of fields. A field not on the list is never in the value. A value holds only what is built from its record: no epoch, no time of writing and no time of the last look, because any lease holder must build the same bytes from the same record, and the audit must too. Times -that belong to the record itself, such as when a walk was opened or a resource's state began, are fields +that belong to the record itself, such as when a rollout was opened or a resource's state began, are fields like any other. The epoch and the time of the write travel in the event. ### nodes — `mesh-controller_nodes`, key: the node's name @@ -74,15 +74,15 @@ like any other. The epoch and the time of the write travel in the event. | the build the node was last sent for it and the build its node reports running | — | | the seats it holds there, and the provisions it is bound to | the bindings' credentials, which are secrets | -### walks — `mesh-controller_walks`, key: the walk's id +### rollouts — `mesh-controller_walks`, key: the rollout's id | Holds | Leaves out, and why | |---|---| -| the repository and commit, or the batch of merges it walks (ADR 0276) | — | +| the repository and commit, or the batch of merges it rollouts (ADR 0276) | — | | its state (building, rolling, done, failed), its current tier, and each tier's modules and nodes with each step's state | the build and apply logs, which the `plans` and `builds` verbs answer | | when it was opened, when it last moved, when it ended, and its note | — | -A walk is removed from the mirror 14 days after it ended. +A rollout is removed from the mirror 14 days after it ended. ### builds — `mesh-controller_builds`, key: the build's id @@ -137,8 +137,8 @@ called). caller today): a numbered migration of `inventory` and the consolidated schema, in the same change. - **A removal is a purge** of the key, said as a `removed` event. A purge leaves one marker for the key. Each purge gives its marker a time to live of one day (a per-message time to live, which the bus server - has offered since release 2.11 and the bucket enables), so the markers of removed walks and builds do - not pile up in a bucket that has no age. The controller's one-minute tick removes walks and builds 14 + has offered since release 2.11 and the bucket enables), so the markers of removed rollouts and builds do + not pile up in a bucket that has no age. The controller's one-minute tick removes rollouts and builds 14 days after they ended. - **A write that fails** sends no event, is logged, and is left to the audit. The writer does not retry in a loop: a retry loop against a refusing bus is the reconnect loop of issue 327 in another place. @@ -169,7 +169,7 @@ counts in §6 shows the gap. **D17, the mirrors match their store**, is a probe in the self-check's registry (to-be 45 §4). Every run does this for each mirror: -1. It reads every record the mirror should hold from the store (for walks and builds, those open or ended +1. It reads every record the mirror should hold from the store (for rollouts and builds, those open or ended within 14 days) and builds each value as the writer would. 2. It lists the bucket's keys and reads their values. 3. It compares. A key missing, a value that differs, and a key with no record are each a candidate. A @@ -225,10 +225,10 @@ remain open there. | The bucket matches the store | D17 on every self-check run; its own test, which changes a key behind the writer's back and expects `mirror-drifted`, then H6's write, then the condition cleared on the next run; and a test that a change of lease alone raises nothing | | Quiet means no writes | live: over ten minutes in which `plans`, `nodes`, `builds` and `settings` show no change, no mirror's write count rises | | A watcher sees a change | live: a setting changed through `settings` is seen by a watch on `mesh-controller_settings` and as one `setting-changed` within a second | -| A value is under its cap, or cut and marked | the controller's test, which builds the largest walk the mesh could make (every module on every node) | +| A value is under its cap, or cut and marked | the controller's test, which builds the largest rollout the mesh could make (every module on every node) | | Only the audit reads a mirror | a test over the controller's packages, which fails when any package but the audit's reads a mirror bucket | | A write follows the commit | the controller's test that a rolled-back transaction makes no write and sends no event | -| Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals | +| Ended rollouts and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the rollouts bucket's keys including deleted ones at most its live keys plus a day's removals | | A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run | | A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set | | A lost acknowledgement is no conflict | the controller's test in which the bus stores a write and drops its acknowledgement: the next write to that key succeeds and no condition is raised | @@ -236,7 +236,7 @@ remain open there. ## What it leaves out -- **History.** A mirror is current state. What happened beyond the 14 days a walk or build is kept, and +- **History.** A mirror is current state. What happened beyond the 14 days a rollout or build is kept, and beyond EVENTS' week, is gap G8 of research 036, which remains open there. - **People reaching the mirrors**, through a WebSocket and an auth callout, is gap G1, open in the same effort. diff --git a/03-DESIGN/01-to-be/55-the-knowledge-base.md b/03-DESIGN/01-to-be/55-the-knowledge-base.md index 924dbcb0..af6d6061 100644 --- a/03-DESIGN/01-to-be/55-the-knowledge-base.md +++ b/03-DESIGN/01-to-be/55-the-knowledge-base.md @@ -130,7 +130,7 @@ rules that judge the whole record set run (an accepted decision cited by nothing undo the acceptance. It is a fault of the records, which their authors repair, not of the module: it is told to the operator once, as a warning naming the rule and the records, and cleared once the rule holds again. The module's health names it and stays healthy, because health says only whether the module works, and a finding never holds a -walk at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never +rollout at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never reviewed). The rule `cited` is also judged ahead, in the review, before the judgement: a decision that names no home and that no diff --git a/03-DESIGN/01-to-be/57-building-config-and-provisioning.md b/03-DESIGN/01-to-be/57-building-config-and-provisioning.md index 92365906..f9a8a7c9 100644 --- a/03-DESIGN/01-to-be/57-building-config-and-provisioning.md +++ b/03-DESIGN/01-to-be/57-building-config-and-provisioning.md @@ -32,10 +32,10 @@ reviewed independently at its head and merged only with its checks green. **It ships as one release.** The operator's decision of 2026-10-14: one delivery, not one per package. Each of mesh-controller, `mesh-host`, mesh-sdk, mesh-catalog and hq has the branch `feat/57-config-and-provisioning`, and every package's pull request merges into that branch, never into -main. When the last package is merged, the whole is rehearsed in the lab: the walk, the moves of data in +main. When the last package is merged, the whole is rehearsed in the lab: the rollout, the moves of data in their order, and every proof that is run live. Then each repository's branch reaches main in one pull request, and the five, sharing their branch's name, are one delivery group (ADR 0239 §2), judged as one -future state. The five are merged within one merge window, so they are one batch and one walk (ADR 0276 +future state. The five are merged within one merge window, so they are one batch and one rollout (ADR 0276 §1, §2), in an order declared with `after:` lines in each pull request (ADR 0239 §11): mesh-sdk, then `mesh-host`, then mesh-controller, then mesh-catalog, then hq. The release holds every package but T6, R1 and R2: the media catalogue and the applications live in repositories of their own and move after it, and only @@ -52,9 +52,9 @@ database consumer connecting) is first seen in the lab rehearsal, and a fault fo release, not one package. A merge's builds go to every machine at once (ADR 0360), so the release has no first node to hold it: after the pull requests' checks, the lab rehearsal is the only live check before it reaches every node. A machine that fails to apply it, or whose provisioner reports a module's files failed -(ADR 0362), is named in the walk's end and raises apply-failing, and is fixed forward with a new pull +(ADR 0362), is named in the rollout's end and raises apply-failing, and is fixed forward with a new pull request into main; one that says nothing within fifteen minutes is named silent and gets its builds from -the backlog walk once it is heard from (ADR 0360 §10). +the backlog rollout once it is heard from (ADR 0360 §10). **Nothing breaks on the way.** ADR 0349 clause 17: each file is filled the old way or the new way, never both, by the form of its manifest, and the old forms stay accepted until the last module has moved. @@ -247,11 +247,11 @@ back on drift, removes a module's files and needs when it leaves, fills from cur comes back. Its first owners are placement (from the declaration) and config (from `mesh-config_resolved`). **Proof:** the client's test list in ADR 0349, against a real bus. -**P6 (issue 672) — The controller hands a module to the provisioner, and the walk waits for it.** *mesh-controller. +**P6 (issue 672) — The controller hands a module to the provisioner, and the rollout waits for it.** *mesh-controller. Two days.* For a module marked moved, the declaration carries placement, the templates' digest and the -filled-files list, and the controller runs no other filling pass for it; a walk counts the module applied +filled-files list, and the controller runs no other filling pass for it; a rollout counts the module applied on a machine only after the node-engine's report and that node's `prov.applied` for the new build, and a -failed `prov.applied` is a failed apply (ADR 0362). dunst moves first. **Proof:** composition and walk +failed `prov.applied` is a failed apply (ADR 0362). dunst moves first. **Proof:** composition and rollout tests; dunst on the first workstation filled by the client, live. **P7 (issue 673) — The node-engine stops writing placed files from the declaration.** *`mesh-host`. A day.* For a moved diff --git a/03-DESIGN/01-to-be/README.md b/03-DESIGN/01-to-be/README.md index f3999865..95b3e4da 100644 --- a/03-DESIGN/01-to-be/README.md +++ b/03-DESIGN/01-to-be/README.md @@ -47,13 +47,13 @@ document is written and this one's status becomes `implemented`. | [`42-the-machines-modules-in-order.md`](42-the-machines-modules-in-order.md) | **In progress.** The order the machines' modules of research 026 and 027 are built and rolled out: every machine's first (sudo, localization, time sync, pacman, logrotate, avahi, systemd, docker, `~/.ssh`, scripts, kernel), then both workstations', then one machine model's; each proven on one workstation before the rest | ADR 0173, ADR 0182, ADR 0205 | | [`45-a-core-that-cannot-fail-silently.md`](45-a-core-that-cannot-fail-silently.md) | **Designed.** The core says when it is wrong, refuses what is stale or unreadable, heals what it knows, upgrades one machine at a time with a witness that rolls it back, and is checked against the real mesh before merge: the writers and signals tables, the condition store, `doctor`, the minimal output channel, healers, the lease and report order, staged upgrades, the facts snapshot and replays, in six phases | ADR 0227, ADR 0224, ADR 0218 | | [`46-the-conversation-with-the-operator.md`](46-the-conversation-with-the-operator.md) | **Designed.** The mesh tells and asks its operator over channels that are holders of two kinded benches, `channel` and `intake`, declaring capabilities from a fixed vocabulary; the router orders them by work context and never lowers the bar; an answer that performs an action is checked and performed by the controller, on a TOTP code or a verified Telegram sender, never on a desk click alone; operator messages addressed or answered by the mesh's responder, untrusted input kept as data, references for what words may not carry, and a recoverable factor; Telegram first, in six phases | ADR 0234, ADR 0227 | -| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the walk, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 | +| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the rollout, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 | | [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of eleven domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | ADR 0244, ADR 0006, ADR 0008, ADR 0302 | | [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | ADR 0247, ADR 0223 | | [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | ADR 0251, ADR 0189 | | [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **In progress.** A node shares one folder at a time through `nfs-server`, which provides it as a provision to the private network only, mapping every client to the folder's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | ADR 0263, ADR 0051, ADR 0092 | | [`53-the-issue-tracker.md`](53-the-issue-tracker.md) | **Designed.** The mesh keeps its issues in one module, `issues`, on the control-node: every number from one counter advanced by compare-and-set, the cycle's issue rules refused at the write, an event per change; an issue lives whole in the tracker, hq keeping no issue content (ADR 0291) | ADR 0275, ADR 0291 | -| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, walks, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 | +| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, rollouts, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 | | [`55-the-knowledge-base.md`](55-the-knowledge-base.md) | **Designed.** The mesh keeps its decision records and research efforts whole in one module, `mesh-knowledge`, on the control-node: number, status, topic, typed links both ways, each decision's clauses with their own status, and the reasoning, with every version; a record is proposed, checked at the write, reviewed by the knowledge base's own judgement and accepted there, with the operator's Approve when it needs it; agents ask what holds in one bounded call | ADR 0290 | | [`56-the-mail-client-set-up-by-the-mesh.md`](56-the-mail-client-set-up-by-the-mesh.md) | **Designed.** On every node with a display server, the `betterbird` module installs Betterbird from a pinned upstream release mirrored into the artifact store, ships the Owl add-on pinned and installed by Betterbird's enterprise policy, and declares the operator's mail accounts in a trusted setting without credentials; each account signs in once at a person's first use, and one the profile already holds is adopted by its address | ADR 0316, ADR 0317 | | [`57-building-config-and-provisioning.md`](57-building-config-and-provisioning.md) | **Designed.** The work of ADR 0348 and ADR 0349 as 37 packages, each an issue with its proof: foundations, template files, config in its own module, the provisioner and its client, secrets in the vault, bindings from providers, collections, a module's runtime readers, and removing the old forms; at most five worked at once | ADR 0348, ADR 0349 | diff --git a/04-ISSUES/021-a-consumer-on-the-providers-machine-is-given-no-credential/00-report.md b/04-ISSUES/021-a-consumer-on-the-providers-machine-is-given-no-credential/00-report.md index ec842b5d..30f41351 100644 --- a/04-ISSUES/021-a-consumer-on-the-providers-machine-is-given-no-credential/00-report.md +++ b/04-ISSUES/021-a-consumer-on-the-providers-machine-is-given-no-credential/00-report.md @@ -28,7 +28,7 @@ for _, n := range nodes { ``` So a provider on the same machine is never a `Provider` in `world.Offered`, never becomes a -`Needed`, and the credential loop — which walks `resolved.Needs` — has nothing to walk. Every step +`Needed`, and the credential loop — which traverses `resolved.Needs` — has nothing to traverse. Every step is individually reasonable and the sum is a silent gap. ## Why it was not noticed diff --git a/04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md b/04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md index b0ed4615..04cb66c1 100644 --- a/04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md +++ b/04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md @@ -92,7 +92,7 @@ node that looks like a real one. ## Fixed -Needs fan out per consuming module in one place, after the resolution walk. The credential's key +Needs fan out per consuming module in one place, after the resolution rollout. The credential's key gains the consuming module, the grant file is named after both halves of the consumer, needs are matched by provision *and* module, and the provisioners name the role and the access key after the module rather than the machine. The refusal is gone because there is nothing left to refuse. diff --git a/04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md b/04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md index 7e71c8a9..1561efe6 100644 --- a/04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md +++ b/04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md @@ -87,7 +87,7 @@ too new to have anybody watching it. ## Fixed **The registry module names its image and is added, not built.** The lab's artifact-store test now -walks the only path open to a real first mesh — the manifest goes in directly, no builder and no +rollouts the only path open to a real first mesh — the manifest goes in directly, no builder and no store involved — and passes. **And the cycle is refused where it is written.** A manifest that provides `artifact-store` and diff --git a/04-ISSUES/107-a-declaration-carries-no-order/00-report.md b/04-ISSUES/107-a-declaration-carries-no-order/00-report.md index c7d8f257..12c82c87 100644 --- a/04-ISSUES/107-a-declaration-carries-no-order/00-report.md +++ b/04-ISSUES/107-a-declaration-carries-no-order/00-report.md @@ -45,7 +45,7 @@ field it does not know — whole. That was mesh now records which host each machine reports and `status` names every machine running an older one than another does. -So the flag day is visible before it is walked into, which it was not when this was filed. It does not +So the flag day is visible before the mesh runs into it, which it was not when this was filed. It does not make the field free: **the oldest host in the mesh is still what the mesh may send**, and one machine of four is behind today. A sequence that an old host refuses takes that machine out of the mesh's reach entirely — worse than the replay it prevents, which has never been observed. diff --git a/04-ISSUES/107-a-declaration-carries-no-order/01-diagnosis.md b/04-ISSUES/107-a-declaration-carries-no-order/01-diagnosis.md index bf3b80be..f5a0a091 100644 --- a/04-ISSUES/107-a-declaration-carries-no-order/01-diagnosis.md +++ b/04-ISSUES/107-a-declaration-carries-no-order/01-diagnosis.md @@ -18,7 +18,7 @@ half-understood declaration off a machine, and exactly what makes this expensive [Issue 087](../087-the-controller-cannot-tell-a-host-is-too-old/00-report.md) is resolved: the mesh now records the host version each machine reports and `status` names every machine running an older host -than another does. The flag day is visible before it is walked into, which it was not on 2026-09-23. +than another does. The flag day is visible before the mesh runs into it, which it was not on 2026-09-23. That makes the cost measurable rather than hypothetical, and the measurement is the reason this is not being built today. diff --git a/04-ISSUES/117-a-modules-own-code-is-a-container-and-a-process/00-report.md b/04-ISSUES/117-a-modules-own-code-is-a-container-and-a-process/00-report.md index 79d6cbef..59612ce8 100644 --- a/04-ISSUES/117-a-modules-own-code-is-a-container-and-a-process/00-report.md +++ b/04-ISSUES/117-a-modules-own-code-is-a-container-and-a-process/00-report.md @@ -72,7 +72,7 @@ what the sidecar is has nowhere in the design layer to look, which is how this w resource type no decision records and contradicting an accepted one. The rule is "no design without a decision"; the check is "no design without *a* decision." An unenforced rule is indistinguishable from a wrong one, and these two documents are what that gap looks like when - something walks through it. + something rollouts through it. ## Open questions diff --git a/04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md b/04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md index d225b36a..f9b042d8 100644 --- a/04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md +++ b/04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md @@ -92,7 +92,7 @@ built from a repository elsewhere, or a mesh whose forge module is down — and event performs, so the two cannot disagree about what "moved" means. **Bases first needed edges, and there were none.** The order this report asked for was written and -walked a graph that no build had ever recorded: a recipe reads its base from a build argument, so the +traversed a graph that no build had ever recorded: a recipe reads its base from a build argument, so the digest was never in the file the builder read edges from. A build now reports what it was handed, the control plane records it by artifact path, and the order is read from each module's newest build. diff --git a/04-ISSUES/178-a-routed-name-resolves-to-a-provider-merely-told-it/00-report.md b/04-ISSUES/178-a-routed-name-resolves-to-a-provider-merely-told-it/00-report.md index f0072d4d..ec9202b4 100644 --- a/04-ISSUES/178-a-routed-name-resolves-to-a-provider-merely-told-it/00-report.md +++ b/04-ISSUES/178-a-routed-name-resolves-to-a-provider-merely-told-it/00-report.md @@ -38,7 +38,7 @@ Every node's resolution is read first, and the names are attributed across them terminus serves the name**: among the providers a name reaches, the one that is not itself published under a labelled name through another provider. An identity provider is routed through the proxy and so is a consumer of names, not their end; the proxy contributes no label to anyone and is. The rule -knows nothing of what "route" means — it reads the graph the modules declared — and it walks nodes, +knows nothing of what "route" means — it reads the graph the modules declared — and it rollouts nodes, requirements and contributions in order, so one mesh yields one region. Every shape of contribution is counted, so a module routed under several names has every one of them resolved. diff --git a/04-ISSUES/186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md b/04-ISSUES/186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md index 2f7a839f..b4984f59 100644 --- a/04-ISSUES/186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md +++ b/04-ISSUES/186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md @@ -73,5 +73,5 @@ give. All three faults are closed on the controller's main: one ask in flight (8d4e940), a merge rebuilds what stands on it (7e701c0), and a merge produces a tiered plan the mesh keeps, read with `plans` (d2ed6d5, -76f2756, ADR 0162). Plans were since carried into walks and deliveries (ADR 0236, ADR 0239); the order a +76f2756, ADR 0162). Plans were since carried into rollouts and deliveries (ADR 0236, ADR 0239); the order a person kept is the order the tiers give. diff --git a/04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/00-report.md b/04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/00-report.md index f386aff6..9c5139b2 100644 --- a/04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/00-report.md +++ b/04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/00-report.md @@ -86,6 +86,6 @@ machine. That is the path the console exists to replace, so it is a workaround a Both pull requests named above are merged and live. Read through the console on 2026-10-08: `mesh_describe mesh-controller.plan` declares `node` (required) and `files`; `mesh-controller.push` declares `node`, `behind`, `cause` and `why`, and its description says that a push naming no machine -is a push of the whole mesh. The controller's `seatverbs_schema_test.go` walks every verb it serves: +is a push of the whole mesh. The controller's `seatverbs_schema_test.go` exercises every verb it serves: an argument a verb does not declare is refused, and every flag of a verb's command is in its schema or accounted for. diff --git a/04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/01-diagnosis.md b/04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/01-diagnosis.md index d0dceda8..b21d831d 100644 --- a/04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/01-diagnosis.md +++ b/04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/01-diagnosis.md @@ -46,7 +46,7 @@ Pull requests: novox/mesh-tools#14 and novox/mesh-controller#71 — open, not me first line of its answer, that it is a push of the whole mesh and which machines it is sending, and its last line names the machines told. `plan` gains `files`, `push` gains `behind` (the whole mesh said outright, refused beside `node`), and the listing verbs a `limit`. -- **Checked by tests that walk every verb the controller announces:** for every combination of a +- **Checked by tests that exercise every verb the controller announces:** for every combination of a verb's declared arguments, the verb either refuses or every argument given changes the command it runs; every verb refuses an argument it does not declare, and `node` where it takes none; and every flag of the command a verb runs — read from the command's own source — is an argument of the verb's diff --git a/04-ISSUES/259-a-named-push-sent-every-machine/00-report.md b/04-ISSUES/259-a-named-push-sent-every-machine/00-report.md index 391a34ba..41df68d5 100644 --- a/04-ISSUES/259-a-named-push-sent-every-machine/00-report.md +++ b/04-ISSUES/259-a-named-push-sent-every-machine/00-report.md @@ -26,7 +26,7 @@ list alone. The flush has no such narrowing. ## What it costs -`record` is the policy for a change that must be walked through the mesh by hand. A named push cannot +`record` is the policy for a change that must be rolled out through the mesh by hand. A named push cannot do that, so the policy does not hold the change back from anything but the merge. The command's name says one machine, and the command's output is the only place that says otherwise. diff --git a/04-ISSUES/290-a-delivery-adopted-while-it-waited-never-got-its-check/00-report.md b/04-ISSUES/290-a-delivery-adopted-while-it-waited-never-got-its-check/00-report.md index ef0f8b1d..6aa1a85e 100644 --- a/04-ISSUES/290-a-delivery-adopted-while-it-waited-never-got-its-check/00-report.md +++ b/04-ISSUES/290-a-delivery-adopted-while-it-waited-never-got-its-check/00-report.md @@ -61,7 +61,7 @@ The mesh-delivery tests `checkask_test.go` cover: - An announcement that carries a decided gate status uses it. - H2 re-asks once, a second re-ask is refused by the table, and a late verdict still settles the delivery. -The table tests walk the new row, and the pairs the table does not hold are refused by name. Live: once +The table tests exercise the new row, and the pairs the table does not hold are refused by name. Live: once the fix is rolled out, the stalled delivery has its check asked at the owner's first tick. The controller journal then shows its verdict, and `show` moves from proposed to checked to ready without a person acting. diff --git a/04-ISSUES/295-a-recorded-build-was-carried-by-a-plans-send/00-report.md b/04-ISSUES/295-a-recorded-build-was-carried-by-a-plans-send/00-report.md index 69e93b76..eb5d1860 100644 --- a/04-ISSUES/295-a-recorded-build-was-carried-by-a-plans-send/00-report.md +++ b/04-ISSUES/295-a-recorded-build-was-carried-by-a-plans-send/00-report.md @@ -39,7 +39,7 @@ The mesh's own record shows it: - **What each machine reports.** `postgres.server`, `mongodb.server` and nine `mailu.*` containers have run only since 18:57 on the control node, and `postgres.server` since 18:56 on the anchor. -No walk held back for later was involved. The sends were the plan's own sends to its first machines. +No rollout held back for later was involved. The sends were the plan's own sends to its first machines. ## Why it happened @@ -53,7 +53,7 @@ No walk held back for later was involved. The sends were the plan's own sends to push". So the gate did not judge the recorded builds, the refusal of ungated sends did not see them, and the gated send carried them. `record` held only against the end of a person's push to another machine (issue 259's `heldBack`) and against the bus (the bus step). It did not hold - against a plan's walk, a rollback, a healer or a rotation. + against a plan's rollout, a rollback, a healer or a rotation. 3. **Nothing tells a person, before or during a send, that it will recreate every container of a module at once.** A change that only alters how containers are declared reads as harmless. A health check, an environment variable or a label each recreates the container all the same, @@ -64,7 +64,7 @@ No walk held back for later was involved. The sends were the plan's own sends to ## What is at stake `record` is how the mesh keeps the providers, the network path and the irreplaceable data out of -unattended walks. Tonight showed that it holds only until some other module on the same machine +unattended rollouts. Tonight showed that it holds only until some other module on the same machine rolls out. This was still true after the incident: plex (`record`, irreplaceable data) was rebuilt by a media-catalogue merge at 19:01 and sent nowhere. The next gated send to the anchor, for any module, would recreate it. diff --git a/04-ISSUES/296-a-plans-clock-restarted-at-every-save/01-diagnosis.md b/04-ISSUES/296-a-plans-clock-restarted-at-every-save/01-diagnosis.md index a70a6dc6..0cdf6911 100644 --- a/04-ISSUES/296-a-plans-clock-restarted-at-every-save/01-diagnosis.md +++ b/04-ISSUES/296-a-plans-clock-restarted-at-every-save/01-diagnosis.md @@ -13,8 +13,8 @@ the tier durations measured for the repository: three times their ninetieth perc than 30 minutes. So the watchdog and the line a person reads used different start times and could use different bounds. The line was the one that was wrong. -The line for a walk waiting on its delivery's word had the same flaw. It said "for" the time since the -last save. The watchdog for that wait (S16) counts from when the walk was opened. +The line for a rollout waiting on its delivery's word had the same flaw. It said "for" the time since the +last save. The watchdog for that wait (S16) counts from when the rollout was opened. Ruled out: a clock difference between the machine running the verb and the store. The count was small and kept growing again from zero, which only a column being rewritten explains. @@ -44,8 +44,8 @@ nothing. ## Fix (mesh-controller) - The line, the `since` and `late` fields of `status --json`, and the count of late plans all start from - when the plan entered its tier. If the delivery let the walk go after that, they start from the go. A - walk still waiting for its delivery's word counts from when it was opened, as S16 does, and is never + when the plan entered its tier. If the delivery let the rollout go after that, they start from the go. A + rollout still waiting for its delivery's word counts from when it was opened, as S16 does, and is never called late. A plan read without the tier stamp falls back to its last save. - LATE uses the bound the watchdog uses: the measured bound for the plan's repository, or the 30-minute floor when nothing has been measured. The watchdog's start time and bound now come from the same two @@ -53,4 +53,4 @@ nothing. - The loop compares the plan before and after each step and saves only when they differ. The fields the store stamps itself are left out of that comparison. - A refused step is logged and saved only when its note is new. The refusal's wording called the gated - walk a "release plan", a name the glossary retires; it now says "walk". + rollout a "release plan", a name the glossary retires; it now says "walk" (since renamed rollout, issue 646). diff --git a/04-ISSUES/300-a-new-module-is-not-built-at-its-merge/00-report.md b/04-ISSUES/300-a-new-module-is-not-built-at-its-merge/00-report.md index 493b21ea..dab19592 100644 --- a/04-ISSUES/300-a-new-module-is-not-built-at-its-merge/00-report.md +++ b/04-ISSUES/300-a-new-module-is-not-built-at-its-merge/00-report.md @@ -23,7 +23,7 @@ another, with nothing on the mesh noting the difference. ADR 0238 makes the delivery plan the planner's one answer for a pull request's check, the merge gate and the merge. The plan counts a new module's directory as something the merge builds. The merge handler -read the same answer (`touchedBy`'s `added`) and dropped it: it only walks modules the catalogue already +read the same answer (`touchedBy`'s `added`) and dropped it: it only rollouts modules the catalogue already holds, and a new one is not held until it is built. The code said so outright: "a new module, which a check judges before it merges and a merge does not build". The take-in refuses a build off the trunk with "merge it, and the merge builds it". So one part of the controller promised the build and another diff --git a/04-ISSUES/300-a-new-module-is-not-built-at-its-merge/01-diagnosis.md b/04-ISSUES/300-a-new-module-is-not-built-at-its-merge/01-diagnosis.md index 65dab261..59b18771 100644 --- a/04-ISSUES/300-a-new-module-is-not-built-at-its-merge/01-diagnosis.md +++ b/04-ISSUES/300-a-new-module-is-not-built-at-its-merge/01-diagnosis.md @@ -29,7 +29,7 @@ already built from it are, and without waiting (`askNewModules`). The build's ta module, as it does for a hand `build`. Nothing is sent: no machine is assigned a new module, and assigning one stays a person's act. -The new module is outside the merge's plan. A plan walks modules the catalogue holds, tier by tier, and +The new module is outside the merge's plan. A plan rollouts modules the catalogue holds, tier by tier, and sends each built one to the machines running it. A new module has no machine and nothing standing on it, and the plan's ask for a module the catalogue does not hold fails the plan. diff --git a/04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/00-report.md b/04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/00-report.md index 748e5cd3..101b47ed 100644 --- a/04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/00-report.md +++ b/04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/00-report.md @@ -7,16 +7,16 @@ opened: 2026-10-08 ## Symptom On 2026-10-08 an agent proving the tunnel join (novox/mesh-lab PR #61) on the laptop could not run the -two-node walk, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was +two-node rollout, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was reported by any check before someone tried to run a bed: -1. **The walk's scenario stocks a packet-filter runtime image that cannot be built.** +1. **The rollout's scenario stocks a packet-filter runtime image that cannot be built.** `scripts/build-module-runtime.sh` copied the tool runner's compiled TypeScript and its `node_modules` out of mesh-tools. mesh-tools has had no package at its root since the runtime became `node-tools`, a Go binary that launches every bundle (ADR 0175, ADR 0193), so the script failed at its first step. -2. **The walk's builder dialled the old broker.** It was started with the old bus's guest account on +2. **The rollout's builder dialled the old broker.** It was started with the old bus's guest account on loopback. The mesh moved its bus to NATS (ADR 0106, ADR 0127, ADR 0131), and the builder @@ -27,8 +27,8 @@ reported by any check before someone tried to run a bed: 4. **The replays prover could not run a test in mesh-tools**, because that repository's Go module lives in a subdirectory (`node-tools/`), and the prover ran `go test` only at a repository's root. -The same day, while trying to run the walk again, two more gaps surfaced: the object store's -provisioner image, which the walk also stocks, cannot be built, because the client image it copies +The same day, while trying to run the rollout again, two more gaps surfaced: the object store's +provisioner image, which the rollout also stocks, cannot be built, because the client image it copies from is gone upstream; and the laptop could not reach the public container registry, which the lab's egress check needs (an outside condition, not the lab's fault). @@ -56,9 +56,9 @@ In novox/mesh-lab PR #63, stacked on #61: builds `node-tools` from mesh-tools' Go module, and runs it as the image's entrypoint. The image serves on the node's runtime credential: the Go runtime refuses to serve a module on that module's own credential. -- **The walk no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers +- **The rollout no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers names an image (its verbs are a tools bundle the node's runtime launches), so the image was never run. -- **The walk's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with +- **The rollout's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with an own `broker` secret, as the catalogue's build agent does. Assigning it to the anchor issues the account ([issue 203](../203-a-fresh-assignment-is-pushed-before-its-credential-exists/00-report.md)). The bus's user list is placed, the push writes the sealed credential, and the builder starts on it. @@ -71,12 +71,12 @@ In novox/mesh-lab PR #63, stacked on #61: Still open: - `builds.test.ts` still dials the old broker. Its bed enrols no machine, so the mesh has nowhere to - deliver a builder's credential. It needs the walk's enrolment, or it should be retired into the walk. + deliver a builder's credential. It needs the rollout's enrolment, or it should be retired into the rollout. - Beds that run a per-module runtime container on the module's own credential (the tools beds for the wiki and the code forge, and the audit bed) run into the runtime's refusal. They need the node's runtime credential instead. - The object store provisioner's image needs a client source that still exists. -- The walk has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the +- The rollout has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the public container registry. ## How it is checked diff --git a/04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/01-diagnosis.md b/04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/01-diagnosis.md index 5c8b0195..f47d783a 100644 --- a/04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/01-diagnosis.md +++ b/04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/01-diagnosis.md @@ -14,25 +14,25 @@ script now does those steps by hand. Verified: the image for the packet filter b answers `initialize` and `tools/list`; `node-tools` in it, started against a real bus with a plain URL, launches the bundle and serves nothing until the mesh issues a membership, which is correct. -**Why the walk stocked that image at all.** The tunnel branch passed the held images to +**Why the rollout stocked that image at all.** The tunnel branch passed the held images to `deriveTheFilterOn`, which turns the catalogue's manifest into the lab's form: build section gone, each container's artifact replaced by a stocked image. The catalogue's nftables manifest has no container, so no image is referenced. Removing the build section is what the bed needed; the stocked image was not. **The builder.** mesh-controller's `cmd/mesh-builder` reads `MESH_BROKER_FILE` and nothing else for the -bus. Without it, it says it has no credential. The walk set the old broker's variable, which the builder +bus. Without it, it says it has no credential. The rollout set the old broker's variable, which the builder no longer reads. A builder's account is a module's account (`builder issue` and `module issue` both mint through the inventory), composed into the bus's user list and sealed to the machine as the module's `broker` own secret. Assigning a module that declares that secret now issues it in the same act (issue 203). The catalogue's build agent is that module, but it runs the builder as a container from an -artifact the mesh builds, and it needs the artifact store and package registry bound. The walk has +artifact the mesh builds, and it needs the artifact store and package registry bound. The rollout has neither before it can build, so it declares a lab module with the same claim and the same secret. In -this bed the bus comes from the bundle, so its user list is placed by hand, the step the walk already +this bed the bus comes from the bundle, so its user list is placed by hand, the step the rollout already takes at enrolment. **The uplink and the prover** were each fixed by the agent that met them (mesh-lab #61 and #62), and are listed here only so the lab's state on that day is in one place. -**Ruled out.** That the walk failed because of the tunnel changes. The builder and runtime faults are in +**Ruled out.** That the rollout failed because of the tunnel changes. The builder and runtime faults are in code that #61 does not touch, and they reproduce on main. diff --git a/04-ISSUES/309-two-order-rules-ordered-one-pair-both-ways/01-diagnosis.md b/04-ISSUES/309-two-order-rules-ordered-one-pair-both-ways/01-diagnosis.md index eb64ccb5..af3bc702 100644 --- a/04-ISSUES/309-two-order-rules-ordered-one-pair-both-ways/01-diagnosis.md +++ b/04-ISSUES/309-two-order-rules-ordered-one-pair-both-ways/01-diagnosis.md @@ -21,7 +21,7 @@ the planner's sort stalled. Every two-way pair was a cycle, whatever the rules. - The person's order, the controller first, was right for this change. The new node-engine calls a verb the new controller adds, which is ADR 0246's case: a verb is promised before it is required. - *Engine before controller* comes from ADR 0236's order for its walks, where the node-engine had to + *Engine before controller* comes from ADR 0236's order for its rollouts, where the node-engine had to accept something new first. The rule reads only module names, so it cannot tell the two cases apart. *Built by* is read from the graph. diff --git a/04-ISSUES/310-a-pull-request-ran-its-own-copy-of-its-check/01-diagnosis.md b/04-ISSUES/310-a-pull-request-ran-its-own-copy-of-its-check/01-diagnosis.md index 0f2effb5..4250caaa 100644 --- a/04-ISSUES/310-a-pull-request-ran-its-own-copy-of-its-check/01-diagnosis.md +++ b/04-ISSUES/310-a-pull-request-ran-its-own-copy-of-its-check/01-diagnosis.md @@ -59,7 +59,7 @@ and fails only when it cannot read. **Merged.** mesh-controller#135 (the build agent judges a pull request by the base branch's `merge-check.sh`) merged on 2026-10-08 at 09:10 UTC, and mesh-lab#65 (R310) at 09:10 UTC. Every later -controller delivery carries it; the last, of mesh-controller#170, finished its walk on 2026-10-09 at +controller delivery carries it; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at 00:11 UTC, and `mesh-controller.status` named no machine behind. **The open half is closed.** `gitea_branch_protection_get` for the media catalogue's main, read on diff --git a/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md b/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md index 65a35db0..874192b6 100644 --- a/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md +++ b/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md @@ -32,11 +32,11 @@ was asked of a new, unrelated pull request. ## Why it is a design issue ADR 0239 -decision 10, *the switch*, adopts the walks open when mesh-delivery is assigned. It says nothing of the merges the +decision 10, *the switch*, adopts the rollouts open when mesh-delivery is assigned. It says nothing of the merges the bus still holds from before: the forge's `pull.merged` is kept on the bus and heard again by a new consumer. Each was taken as a merge just made whose head was never heard, so the table's row *merged unchecked → held* held each for a person. The table had no state for "this already happened", and `held`'s only ways on were -a release, which would walk it, and a stop, which says a person stopped a delivery that was never running. +a release, which would roll it out, and a stop, which says a person stopped a delivery that was never running. Healer H2's `close` has no transition from `held`, rightly: a held delivery is the operator's. Decision 5 says what a transition owes outside is "retried until done". An effect the forge answers can never @@ -45,14 +45,14 @@ be done was retried without end, every ten seconds, and nothing said so. ## Fix - **A new final state, `retired`: history.** A delivery made from the forge's word at its merge, whose merge - was heard fifteen minutes or more after it happened and whose walk, if the controller keeps one, is not + was heard fifteen minutes or more after it happened and whose rollout, if the controller keeps one, is not open, is history. At adoption it goes straight to `retired` by the observed row *history*, before any check, - hold or walk. Its pull request is not told: the forge may no longer hold it, or holds another under its + hold or rollout. Its pull request is not told: the forge may no longer hold it, or holds another under its number. Its transitions are still said on the bus and kept on the commit's note. - **A person's verb for the ones already held: `retire-history`**, with why. One delivery by id, refused when the rule does not take it, or every held one (of one repository when named), answering how many it retired and naming each one it left held. `dry` says what it would do and changes nothing. It is the act - *held → retired* of the table, never a release: nothing is walked, delivered or put back. + *held → retired* of the table, never a release: nothing is rolled out, delivered or put back. - **Owed work ends.** The forge's answer that what an effect is for does not exist (a 404, or "does not exist") is final: the effect is given up with one line, kept among the owner's refusals. Any other failure is tried again later each time, from ten seconds doubling to an hour, and given up after sixty tries, saying so. diff --git a/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/01-diagnosis.md b/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/01-diagnosis.md index 7331b822..e84696a9 100644 --- a/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/01-diagnosis.md +++ b/04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/01-diagnosis.md @@ -5,20 +5,20 @@ - 699 deliveries were `held`. 688 say "merged without a passing check", all made between 23:05:27 and 23:05:29 on 2026-10-06, the owner's first start. A sample across repositories shows merges from 2026-10-02 to two hours before that start, each with the transition "its head was not heard by this owner, - so it is made from the forge's word", then *merged unchecked → held*. None has a walk. -- The other 11 are held for other reasons: a merge with no walk opened within ten minutes (10), and a group + so it is made from the forge's word", then *merged unchecked → held*. None has a rollout. +- The other 11 are held for other reasons: a merge with no rollout opened within ten minutes (10), and a group whose composed check did not pass (1). They are not history, and this issue leaves them alone. One of them was past its bound, which makes the 689th finding. **Where it comes from.** The holder subscribes to the forge's `pull.merged` on the bus after reading back its -state and the controller's walks. The bus hands a new consumer the merges it still holds. For each, the holder +state and the controller's rollouts. The bus hands a new consumer the merges it still holds. For each, the holder finds no delivery for the head, so it makes one from the forge's word. The pull request's statuses at its merge do not include a passing `mesh/merge-gate` (the check did not exist for most of them), so the table's *merged unchecked* row holds it. **What would close them, and why not.** -- `release` walks a held delivery: wrong, these reached the machines already. +- `release` rollouts a held delivery: wrong, these reached the machines already. - `stop` takes it to `stopped`, which says a person stopped a delivery and sets `mesh/delivery` to error on the commit. Wrong in meaning, and it would ask the forge for a view on a pull request that is gone. - Healer H2's `close` reads `held`'s bound, which names no H2 transition, and refuses: "the state is the @@ -43,6 +43,6 @@ mesh-catalog#121 (the delivery's owner retires history and ends owed work the fo under ADR 0250. **Seen live.** `mesh-delivery.deliveries` on 2026-10-09 lists the held history as `retired`, all at -2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The walk of +2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The rollout of mesh-catalog#121's own delivery reads `stopped`; the holder that answers today serves the fix all the same, and `mesh-controller.status` names no machine behind. diff --git a/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md b/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md index 9b7435f8..8b1ab01e 100644 --- a/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md +++ b/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md @@ -53,6 +53,6 @@ lift. every other state in the table moves on its facts? 3. Should `mesh/delivery-group` be required on the trunks, so a group cannot merge before its composed check passed? To-be 47 leaves this to the operator; this issue is the case that setting decides. -4. The group's order put the catalogue's member first, and the node-engine's member walked at once on the - controller's own path. Is a group's order meant to hold for a member whose walk never waits for +4. The group's order put the catalogue's member first, and the node-engine's member rolled out at once on the + controller's own path. Is a group's order meant to hold for a member whose rollout never waits for mesh-delivery? diff --git a/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/01-diagnosis.md b/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/01-diagnosis.md index 7416ff15..c02459f5 100644 --- a/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/01-diagnosis.md +++ b/04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/01-diagnosis.md @@ -3,7 +3,7 @@ **2026-10-08.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`), from `mesh-delivery.show` of both members, and from the holder's journal on the control node. -**Where the hold is set.** `groupTurns` (holder.go) walks a closed group's members in order. For a member +**Where the hold is set.** `groupTurns` (holder.go) rolls out a closed group's members in order. For a member in `published` it computes `composed`: the group's check exists, its verdict is `pass` or `warning`, and it names the members' current heads. When `composed` is false it writes the member's `HeldWhy` ("its group's composed check did not pass for the heads that merged") and settles it, and the table's `hold` row takes it @@ -20,7 +20,7 @@ to `held`. `composed` is false in three different cases, and the code does not t look at the members. - `groupTurns` skips a member in `held` (`default: return nil`), so the tick after the verdict sees the passing check and does nothing with it. -- The table has two rows out of `held`: `go` (the walk started on a word that was not this owner's) and +- The table has two rows out of `held`: `go` (the rollout started on a word that was not this owner's) and `release` (a person's act). There is no observed row that takes a held delivery back when the reason it was held is gone, and `HeldWhy` is never cleared once set. @@ -41,7 +41,7 @@ check was asked. To-be 47 lists whether to require it as not decided. - A lost verdict. The holder logged it at 10:10:21 and the group's record holds it. - The order. The group's order was asked and kept before the check; it plays no part in the hold. -**Not this issue, noted.** The node-engine's member walked at once (`published` → `delivering` at the +**Not this issue, noted.** The node-engine's member rolled out at once (`published` → `delivering` at the same instant), because the controller's own path does not wait for mesh-delivery (ADR 0239 decision 8). The group's order put it second. Open question 4 in the report asks whether that is intended. @@ -57,9 +57,9 @@ ADR 0270 its group's composed check is asked for the current heads and has no verdict. - **A hold whose reason is gone lifts, by observation.** A member held because its group's composed check had not passed goes back to `published` (the event *lifted*) once the check passes or warns for the heads that merged, - while its own walk is still open. It then takes its turn in order. -- A held member whose own walk was since taken over by a later walk is not lifted, since its walk can no longer - deliver it. It is `covered` when that later walk was delivered and sent its module (issue 319). + while its own rollout is still open. It then takes its turn in order. +- A held member whose own rollout was since taken over by a later rollout is not lifted, since its rollout can no longer + deliver it. It is `covered` when that later rollout was delivered and sent its module (issue 319). Open question 3, whether `mesh/delivery-group` is required on the trunks, stays the operator's setting. Open question 4 is not touched. diff --git a/04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md b/04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md index 0c59f88e..f9f03ed8 100644 --- a/04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md +++ b/04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md @@ -2,7 +2,7 @@ opened: 2026-10-08 --- -# 317. A held delivery's module was walked by a later merge's plan +# 317. A held delivery's module was rolled out by a later merge's plan ## Symptom @@ -22,7 +22,7 @@ took `openrazer` along. `mesh-controller.plans`: | c78b5fc9 (#124's merge) | 11:07:02 | superseded at tier 0 by the plan at 86077780; docker, gitea, openrazer planned there again | | 86077780 (#123's merge) | 11:07:03 | tier 0: distribution, docker, gitea and openrazer built from c78b5fc9; openrazer's gate on the laptop judging | -The controller's walk of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 → +The controller's rollout of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 → c78b5fc9, beside `docker` and `node-tools`. From 11:10:58 UTC the laptop's condition for openrazer said its account was *relogin needed*. That verdict exists only in the build #125 added, so the held change was running on the laptop while its delivery said it waited for a person. @@ -41,10 +41,10 @@ gone out. A person who releases it later releases nothing; a person who stops it ## Open questions -1. When a newer merge supersedes an older plan, does a module the older plan's walk was holding for its +1. When a newer merge supersedes an older plan, does a module the older plan's rollout was holding for its delivery's word come along, or stay held with that delivery? 2. If it comes along, what does the held delivery become: superseded by the newer delivery, with the hold - carried over to it, or something else? Today the table has no row from `held` for a superseded walk. + carried over to it, or something else? Today the table has no row from `held` for a superseded rollout. 3. A trunk commit holds every earlier commit, so a later build of a module carries the held change in - any case. Is a hold on one delivery meaningful once a later merge of the same repository is walked, + any case. Is a hold on one delivery meaningful once a later merge of the same repository is rolled out, or does a hold have to hold the module on that trunk until it is released? diff --git a/04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/01-diagnosis.md b/04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/01-diagnosis.md index d50ed1ed..9af5bb03 100644 --- a/04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/01-diagnosis.md +++ b/04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/01-diagnosis.md @@ -3,7 +3,7 @@ **2026-10-08.** Read from `mesh-controller.plans` (the list, and each plan by id), `mesh-delivery.show` and `deliveries`, and the code on the main branches of the controller and the catalogue. -**The held delivery's walk.** #125's merge made the plan at a6385479. Its walk waited for mesh-delivery's +**The held delivery's rollout.** #125's merge made the plan at a6385479. Its rollout waited for mesh-delivery's word (`waited_for: mesh-delivery`, `waits: true`) and was never let go, because its delivery was held. Its one module, `openrazer`, was "not yet asked": nothing was built. @@ -11,32 +11,32 @@ one module, `openrazer`, was "not yet asked": nothing was built. plan. For every open plan of the same repository and branch made before it, it takes each module the older plan had not built (not asked, asked and not answered, or built and not yet sent), adds it to the newer plan, and closes the older plan as superseded with the note "… planned there again". It does not -ask why the older plan had not built them. A walk waiting for a held delivery's word is open, so it is +ask why the older plan had not built them. A rollout waiting for a held delivery's word is open, so it is superseded like any slow plan. This happened three times over (3f552274, c78b5fc9, 86077780), each newer plan folding in what the one before it held. -The newest plan's own delivery (#123) was not held, so mesh-delivery let its walk go, and `openrazer` was -built from c78b5fc9. The controller's walk of the builds waiting for a gate then carried it to the laptop, +The newest plan's own delivery (#123) was not held, so mesh-delivery let its rollout go, and `openrazer` was +built from c78b5fc9. The controller's rollout of the builds waiting for a gate then carried it to the laptop, since a send carries every move waiting on its machine (ADR 0236). **Why the held delivery did not notice.** mesh-delivery's table has `superseded` rows only from `published` and `delivering` ("a newer delivery to the same trunk took over its walk"). From `held` there -are only `go` and `release`. When the walk record said `superseded`, no row applied, so the delivery stayed -`held` with `held_why` unchanged, and its walk shows "superseded … openrazer planned there again". Two of +are only `go` and `release`. When the rollout record said `superseded`, no row applied, so the delivery stayed +`held` with `held_why` unchanged, and its rollout shows "superseded … openrazer planned there again". Two of the three later deliveries (#122, #124) went to `superseded` correctly, through the `published` row. **What the design says.** ADR 0218 decision 3 makes a newer merge take over an older plan's unfinished work, and was written before deliveries could be held. ADR 0239 makes a held delivery wait for a person, -and to-be 47 says that nothing of a published delivery is registered before its walk starts, so that a +and to-be 47 says that nothing of a published delivery is registered before its rollout starts, so that a send cannot carry it early. Neither says what a supersession does to a module a hold is keeping back. **Ruled out.** -- A person's act. #125's transitions end at `hold`, with no `release`, and its own walk was superseded +- A person's act. #125's transitions end at `hold`, with no `release`, and its own rollout was superseded while still waiting (`let_go` empty), so no `plans go` started it. - Issue 295's path, a recorded build carried by a send: here the build was asked by a plan, not left recorded. -**Located** in the controller's `supersededBy`, which folds a held walk's modules into a newer plan, and in +**Located** in the controller's `supersededBy`, which folds a held rollout's modules into a newer plan, and in mesh-delivery's table, which has no way out of `held` when that happens, so the delivery's state stops being true. diff --git a/04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md b/04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md index a984695a..c8eab31c 100644 --- a/04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md +++ b/04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md @@ -2,11 +2,11 @@ opened: 2026-10-08 --- -# 318. One module's wait for a new login held every other module's walk +# 318. One module's wait for a new login held every other module's rollout ## Symptom -On 2026-10-08 at 11:08 UTC the controller opened its walk of the builds waiting for a gate: the laptop +On 2026-10-08 at 11:08 UTC the controller opened its rollout of the builds waiting for a gate: the laptop first, then the workstation, then the control node. On the laptop it sent three moves in one send and judged them together: @@ -23,14 +23,14 @@ account was *relogin needed* from 11:10:58 UTC, which ADR 0252 expects until the operator logs out and in again. -Meanwhile three other walks could not send to the rest of the mesh. `mesh-controller.plans` said of the +Meanwhile three other rollouts could not send to the rest of the mesh. `mesh-controller.plans` said of the tool runner's (`novox/mesh-tools` d4845f7c, `node-tools`), the catalogue's (86077780, `docker`) and the controller's (950562af, `build-agent`): "a build waits there for its gate: … the workstation (…, openrazer -88135ad0 → c78b5fc9) — a walk sends them, one machine at a time, each judged". +88135ad0 → c78b5fc9) — a rollout sends them, one machine at a time, each judged". At 11:21:06 UTC the laptop's gate failed: "not healthy within 10m0s of its apply: its unit openrazer.daemon … is unhealthy", and `openrazer` was put back to 88135ad0 on the laptop -(`build.openrazer..rolled-back`). The walk ended `failed`. After it, all three walks still said +(`build.openrazer..rolled-back`). The rollout ended `failed`. After it, all three rollouts still said the same: `openrazer` c78b5fc9 waits on the workstation for a gate, so nothing is sent there or to the machines after it. @@ -40,14 +40,14 @@ The wait was known, and it was a person's: ADR 0252 says a new login holds a del that machine. It held every module on every machine after it instead, for changes that had nothing to do with it: the tool runner, the container engine and the build agent. The end was worse than the wait. The gate's bound turned "waiting for a person" into "failed", the build that added the account group was put -back, and the walk that carries the other modules failed with it. A failed walk of the waiting builds +back, and the rollout that carries the other modules failed with it. A failed rollout of the waiting builds does not open again on its own: a person releases it. Three things meet here, each sound alone: - a send carries every move waiting on its machine, and the gate judges them all at once (ADR 0236); -- no other send may go where a move waits for a gate, so every walk waits for the one that judges it; +- no other send may go where a move waits for a gate, so every rollout waits for the one that judges it; - a module's health that waits for a person reads as *not yet healthy*, which fails at the gate's bound. ## Open questions @@ -61,5 +61,5 @@ Three things meet here, each sound alone: back the account group it added, so the login the gate waited for can no longer help. 4. The daemon's unit had failed since 10:13 UTC, almost an hour before the new build reached the laptop. Should a gate fail a build for a failure that was there before it was sent? -5. While a move waits for a gate on one machine, should walks of unrelated modules still be refused there, +5. While a move waits for a gate on one machine, should rollouts of unrelated modules still be refused there, or only those whose send would carry it? diff --git a/04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/01-diagnosis.md b/04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/01-diagnosis.md index 092b0f7c..dea661a8 100644 --- a/04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/01-diagnosis.md +++ b/04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/01-diagnosis.md @@ -1,6 +1,6 @@ # 318 — diagnosis -**2026-10-08.** Read from `mesh-controller.plans` (the list, and the walk of the waiting builds by id) between +**2026-10-08.** Read from `mesh-controller.plans` (the list, and the rollout of the waiting builds by id) between 11:17 and 11:22 UTC, `mesh-controller.conditions` and `status`, and the controller's code on its main branch (`cmd/mesh-controller`). @@ -9,15 +9,15 @@ gate all of them as carried moves. `judgeMoves` (gate.go) reads each module on e worst reading. Any module *not yet healthy* resets the passes to zero for the whole send: `docker` and `node-tools` were healthy, and the gate still said "0 of 3 passes" because `openrazer` was not. Past `gateBound` (ten minutes after the apply) a *not yet healthy* gate fails. `g.Failing` names only the -modules found wanting, so only `openrazer` was put back, but the walk itself ended `failed`. +modules found wanting, so only `openrazer` was put back, but the rollout itself ended `failed`. **Everything else waits for that verdict.** `ungatedIn` (release.go) refuses any send that is not a person's and would carry a move no gate has seen, outside the send's own scope. A plan's sends to the rest of its machines carry the machine's whole declaration, so while `openrazer` c78b5fc9 waited for a gate on the -workstation, the tool runner's, the catalogue's and the controller's walks were each refused there, and -they say so on every retry. They do not fail; they wait for the walk of the waiting builds, which goes one +workstation, the tool runner's, the catalogue's and the controller's rollouts were each refused there, and +they say so on every retry. They do not fail; they wait for the rollout of the waiting builds, which goes one machine at a time and had stopped on the laptop. After it failed, the release code's rule applies: a -failed walk of the waiting builds stops the next one opening on its own until a person releases it. +failed rollout of the waiting builds stops the next one opening on its own until a person releases it. **How *relogin needed* reached the gate.** ADR 0252 decision 5 says the node-engine states the account's health as the module's resource of kind `account`, unhealthy with *relogin needed*, and that "the @@ -33,8 +33,8 @@ running session lacks the group the daemon checks for. That is a guess, not chec readings existed before or because of the person's step, and neither is a fault the build brought. **What the design says.** ADR 0252's consequences say "a new login holds a delivery of that module on that -machine" and accept that a walk that starts on a workstation waits for the person. They do not consider the -other modules carried in the same send, the walks refused behind it, or the gate's ten-minute bound turning +machine" and accept that a rollout that starts on a workstation waits for the person. They do not consider the +other modules carried in the same send, the rollouts refused behind it, or the gate's ten-minute bound turning the wait into a rollback. ADR 0236 makes one send judge everything waiting on a machine, to stop unjudged builds from moving; it does not distinguish a module waiting for a person from one that is broken. @@ -50,15 +50,15 @@ Whether the account actually left the group was not checked. - A machine-wide fault on the laptop: its node-engine reported `applied` and current, and the gate's wanting named one module's unit. -**Located** in the controller's walk and first-node gate: one verdict per send (`judgeMoves`), sends refused +**Located** in the controller's rollout and first-node gate: one verdict per send (`judgeMoves`), sends refused on any machine where a move waits (`ungatedIn`), and no reading for a health that waits for a person, so a -known wait fails at the bound and stops every walk behind it. +known wait fails at the bound and stops every rollout behind it. **Decided and fixed, 2026-10-08**, by ADR 0254. Open questions 1 to 3: *relogin needed*, and a unit failing in that same account's own manager, read *waits for a person*, which passes with the wait carried and is said as the module's condition `relogin-needed`; nothing is put back for it. Question 2's other half: each module of a send counts its own passes, and, after -the settle time, in the walk path, keeps a pass when another fails beside it. Question 5: other walks are +the settle time, in the rollout path, keeps a pass when another fails beside it. Question 5: other rollouts are still refused while a move waits for a gate (ADR 0236 §4a stands). With the wait a pass, 318's case leaves nothing waiting for them. A fault decided before the settle time, and a plan's own tier path, still leave the healthy modules of a failed send without a verdict (ADR 0254, consequences). Question 4, a unit that @@ -71,4 +71,4 @@ between 12:41 and 13:03 UTC; the node-engine's and the controller's deliveries r **Seen live.** On 2026-10-09 `mesh-controller.conditions` holds `module.openrazer..relogin-needed` for the laptop and the workstation, raised at 21:02 UTC: "Its update is in place and nothing was undone", -and no walk waits behind it (`mesh-controller.status`: nothing waiting, no machine behind). +and no rollout waits behind it (`mesh-controller.status`: nothing waiting, no machine behind). diff --git a/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md b/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md index ec5a7a5e..5b7acb0a 100644 --- a/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md +++ b/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md @@ -15,8 +15,8 @@ What they are, read with `mesh-delivery.deliveries` and `show`: | how many | repository | why held | what it builds | |---|---|---|---| | 4 | hq | merged without a passing check | nothing: "the change touches no module of the mesh's graph" | -| 2 | catalogue | no walk opened within ten minutes of the merge | a new module, sent nowhere | -| 8 | controller | no walk opened within ten minutes of the merge | the controller, the build agent, the route proxy | +| 2 | catalogue | no rollout opened within ten minutes of the merge | a new module, sent nowhere | +| 8 | controller | no rollout opened within ten minutes of the merge | the controller, the build agent, the route proxy | | 2 | catalogue | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | one module each | Two examples: @@ -26,7 +26,7 @@ Two examples: `proposed` to `held`: "merged without a passing check". Its plan builds nothing. - `novox/mesh-controller@ec3769a8a669`, controller #105. Ready at 2026-10-07 00:26 UTC, merged at 00:27, held at 00:37: "merged, and the controller opened no walk for it within 10m0s". The controller's modules - have been walked since from later commits of the same main, for example the group + have been rolled out since from later commits of the same main, for example the group `feat/delivery-checks-verb`, whose controller member is `delivered`. Five of them (the four hq merges and controller #105) are past the held bound of one day. The self-check's @@ -39,7 +39,7 @@ A held delivery asks a person "does this go on?". For these sixteen the question a later commit of the same trunk has already delivered what they held, or they hold nothing to deliver. The state table offers a person four ways out of `held`, and none fits: -- `release` starts a walk of the old commit: wrong for a commit a later one superseded, and nothing for one +- `release` starts a rollout of the old commit: wrong for a commit a later one superseded, and nothing for one that builds nothing. - `retire-history` is for history made from the forge's word (ADR 0250), and refuses every one of these because their heads were heard. @@ -57,7 +57,7 @@ The issue states the need; the decision is for graduation. - **An end for a held delivery that was superseded**, as `published` and `delivering` already have (`superseded`, "a newer delivery to the same trunk took over its walk"): either because a later commit's - delivered walk carried the same modules on the same trunk, or because it builds nothing at all. + delivered rollout carried the same modules on the same trunk, or because it builds nothing at all. - **Automatic where nothing is lost**: a held delivery whose plan builds nothing has nothing for a person to decide. - **A person's act, with why, otherwise**: a held delivery whose modules a later commit delivered is ended @@ -70,5 +70,5 @@ The issue states the need; the decision is for graduation. 2. The four hq merges were heard at the owner's first start from the bus's history, as announcements rather than merges. Are they history in the sense of ADR 0250, missed by its rule because their head was replayed too? -3. Why did the controller open no walk for eight of its own merges within ten minutes? The controller's own - path never waits, so its walk may have been missed rather than absent. Not diagnosed here. +3. Why did the controller open no rollout for eight of its own merges within ten minutes? The controller's own + path never waits, so its rollout may have been missed rather than absent. Not diagnosed here. diff --git a/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md b/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md index 83668241..5e5e338d 100644 --- a/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md +++ b/04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md @@ -7,14 +7,14 @@ | event | kind | needs | |---|---|---| -| `go` | observed | its walk started on a word not this owner's (the controller's own path, or a person's `plans go`) | -| `release` | a person's act | why and who; it walks the delivery's own commit | -| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no walk open | +| `go` | observed | its rollout started on a word not this owner's (the controller's own path, or a person's `plans go`) | +| `release` | a person's act | why and who; it rolls out the delivery's own commit | +| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no rollout open | | `stop` | a person's act | why and who; from any state not final | -There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the walk's -record saying `superseded`. A held delivery whose walk never opened has no walk record to read, and one -whose walk was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)) +There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the rollout's +record saying `superseded`. A held delivery whose rollout never opened has no rollout record to read, and one +whose rollout was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)) does not match it either. **Why `retire-history` refuses them.** `isHistory` requires `FromForgesWord()`: the delivery was made from @@ -41,7 +41,7 @@ Neither considers a held delivery whose content a later commit delivered. **Ruled out.** -- A missing release. Releasing any of these would walk an old commit over a newer one, or walk nothing. +- A missing release. Releasing any of these would roll out an old commit over a newer one, or roll out nothing. - D14 itself. It reports the state truthfully; the state has no way to change. **Located** in mesh-delivery's state table (no end for a superseded or empty held delivery, and history @@ -52,14 +52,14 @@ read by how a delivery was made) and in to-be 47, whose state table carries the **Eleven held deliveries, read again.** On 2026-10-09 eleven deliveries were `held`, each raising D14's `delivery..stalled` and held for 24 to 48 hours. For each, `mesh-delivery.show` gave its state, why it was held, its merge commit and its plan. A fresh clone of the repository then showed whether the merge commit is an -ancestor of the commit a finished walk delivered. `mesh-controller.status` listed none of the modules involved +ancestor of the commit a finished rollout delivered. `mesh-controller.status` listed none of the modules involved as behind. All eleven were covered: none held a change missing where it should run. -**Why the controller opened no walk (open question 3).** This part comes from reading the code, and part of it +**Why the controller opened no rollout (open question 3).** This part comes from reading the code, and part of it is still a guess. - *A new module.* When a merge only adds a module, the controller's `SourceMoved` - (`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no walk ("built, + (`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no rollout ("built, registered when the build lands, and sent nowhere", issue 300). mesh-delivery's `MovesNothing()` (`model.go`) counts `Plan.New` as something to move, though. So a delivery whose plan is "builds new: …, sent nowhere" is not let through to `published` → `delivered`. Instead the unwalked rule in `holder.go` holds it @@ -67,25 +67,25 @@ is still a guess. read from the code, and it matches `novox/mesh-catalog@60641176abaf`. - *The controller's own merges (a guess, not proved).* `SourceMoved` also returns without a plan in two cases: when the modules the merge touches are "already built from it" (`already > 0`), and when the merge is read as - history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no walk - of its own when another walk, or a person's push, had already built from a later commit. Which of the two hit + history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no rollout + of its own when another rollout, or a person's push, had already built from a later commit. Which of the two hit these merges is not yet known. The controller's journal for the minutes after each merge would say. -- *The link from a walk to a delivery.* `deliveryWalking()` (`holder.go`) links a walk to the delivery whose - `merged_as` equals the walk's commit, and to no other delivery. A later walk that carried an earlier merge's - change therefore never reaches that earlier delivery, and nothing reads "this walk's commit contains your +- *The link from a rollout to a delivery.* `deliveryWalking()` (`holder.go`) links a rollout to the delivery whose + `merged_as` equals the rollout's commit, and to no other delivery. A later rollout that carried an earlier merge's + change therefore never reaches that earlier delivery, and nothing reads "this rollout's commit contains your merge". That is the missing input the first open question asks about. mesh-delivery could answer it from the - forge (is the merge commit an ancestor of the delivered walk's commit) together with the walk's moved modules, + forge (is the merge commit an ancestor of the delivered rollout's commit) together with the rollout's moved modules, without asking the controller what each machine runs. **Four were closed with `stop`, before this issue was found.** On 2026-10-09 the operator agreed to close the held deliveries. Four were stopped with why "covered: its merge … is contained in … which is delivered (walk …)": -| delivery | merge | contained in | walk | +| delivery | merge | contained in | rollout | |---|---|---|---| -| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier walk of the controller | -| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same walk | -| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same walk | -| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue walk | +| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier rollout of the controller | +| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same rollout | +| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same rollout | +| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue rollout | Their stalled conditions cleared at the next D14 observation. Open D14 conditions went from 11 to 7. As this issue already said, `stop` is the wrong record for them: their merged pull requests now carry `mesh/delivery` @@ -101,12 +101,12 @@ remaining seven were left `held` for this issue's fix. | `novox/hq@2221be11a6ef` | merged without a passing check | merge on hq's main; builds nothing | the same | | `novox/hq@897bcce50252` | merged without a passing check | merge on hq's main; builds nothing | the same | | `novox/hq@ef6ab814b053` | merged without a passing check | merge on hq's main; builds nothing | the same | -| `novox/mesh-controller@b98fd0f396c8` | no walk opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too | -| `novox/mesh-controller@ea92af2a7d77` | no walk opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release walk that is done; the step looks stale but is not final | -| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished walk of the same module | its group is still `delivering`, and its own walk is `superseded` | +| `novox/mesh-controller@b98fd0f396c8` | no rollout opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too | +| `novox/mesh-controller@ea92af2a7d77` | no rollout opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release rollout that is done; the step looks stale but is not final | +| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished rollout of the same module | its group is still `delivering`, and its own rollout is `superseded` | The seven show the two ends this issue asks for: four hold nothing to deliver, and three are covered by a -later walk. +later rollout. ## 2026-10-09 — the fix, proposed @@ -115,12 +115,12 @@ ADR 0270 mesh-delivery: - **A merge that moves nothing is delivered at once.** `MovesNothing()` no longer counts `Plan.New`: a module - the controller builds and sends nowhere opens no walk, so there is nothing to wait for. A merge whose plan moves + the controller builds and sends nowhere opens no rollout, so there is nothing to wait for. A merge whose plan moves nothing goes from `proposed`, `checked`, `rejected` or `held` to `delivered`, before the row *merged unchecked → held*. - **`covered`, a new final state: a later merge that contains it was delivered.** For every module its plan moves, a delivered delivery of the same repository and trunk, merged after it, delivered on its own passing - check by the walk of its own merge commit, whose machine steps say the module reached every machine running + check by the rollout of its own merge commit, whose machine steps say the module reached every machine running it. The forge then says, through a new read-only tool `gitea_contains` (its compare of the two commits), whether the later merge commit contains this one; merge order is only the first filter. A forge that cannot answer leaves it held. `mesh/delivery` says `covered: a later merge that contains it was delivered: …` as a @@ -128,7 +128,7 @@ mesh-delivery: - **Read on every tick**, so what is held now ends without an event. This answers open question 1: mesh-delivery reads it from its own records and asks the forge for ancestry; the -controller is not asked. Open question 3, why the controller's own merges opened no walk, is not answered by this fix; it no +controller is not asked. Open question 3, why the controller's own merges opened no rollout, is not answered by this fix; it no longer leaves anything held. **How to verify after mesh-delivery is delivered with it.** Within a minute of its start: @@ -137,13 +137,13 @@ longer leaves anything held. merges (`builds new: …, sent nowhere`), and none of the five controller merges held for "no walk opened". The hq and module-adding ones are `delivered`; the controller ones are `covered`. 2. `mesh-delivery.show` of one of them, for example `novox/mesh-controller@b98fd0f396c8`, ends with a - transition `held → covered (covered)` whose why names the later delivery and its walk. + transition `held → covered (covered)` whose why names the later delivery and its rollout. 3. `mesh-delivery.checks` for its pull request shows `mesh/delivery` as `success covered: …`. 4. The forge module is delivered in the same pull request. Until its `gitea_contains` answers, nothing is covered, and the held controller merges stay held. 5. At the self-check's next reading, `mesh-controller.conditions` no longer lists their `delivery..stalled`. -6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered walk sent +6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered rollout sent their module. One that stays held is a real question: its change may not run anywhere. **Accepted 2026-10-09.** The operator accepted the record named above; it is no longer proposed. diff --git a/04-ISSUES/320-the-operator-could-not-read-the-meshs-notifications/00-report.md b/04-ISSUES/320-the-operator-could-not-read-the-meshs-notifications/00-report.md index 24564bec..3c7d9646 100644 --- a/04-ISSUES/320-the-operator-could-not-read-the-meshs-notifications/00-report.md +++ b/04-ISSUES/320-the-operator-could-not-read-the-meshs-notifications/00-report.md @@ -52,7 +52,7 @@ not addresses, paths or secrets — so they reached the operator untouched. **Nothing said whether the operator was needed, or let them answer.** No field of a condition said whether the operator was needed, so every warning was shown, and an explanation could imply an instruction. The -verbs that answer the common cases existed (release or stop a delivery, start a waiting walk, restart a +verbs that answer the common cases existed (release or stop a delivery, start a waiting rollout, restart a service, silence a condition), but the desk could not take an answer: `node-notifier.send` had no actions, and the conversation design that adds them (to-be 46) was not built. The one way to act was a session. @@ -70,7 +70,7 @@ nothing is not sent. The desk offers the actions, and the operator channel's hol through its seat verb. Every channel says the time in the operator's zone; a clearance is one line. - novox/mesh-controller: the condition's three new fields, the plain rule, a wording for every kind the - controller raises, the richer words at the producers that know more (a walk's modules, a unit's name, a + controller raises, the richer words at the producers that know more (a rollout's modules, a unit's name, a hand-act's cause, a delivery's repository), and a lint over every condition the suite raises. - novox/mesh-catalog: the messenger shows the words, keeps quiet what needs nothing, offers and performs the answers, says since when in the operator's zone (the new setting `time-zone`), says a clearance in diff --git a/04-ISSUES/325-assign-said-no-such-module-while-its-build-ran/00-report.md b/04-ISSUES/325-assign-said-no-such-module-while-its-build-ran/00-report.md index 97309326..94fc0c93 100644 --- a/04-ISSUES/325-assign-said-no-such-module-while-its-build-ran/00-report.md +++ b/04-ISSUES/325-assign-said-no-such-module-while-its-build-ran/00-report.md @@ -88,5 +88,5 @@ To-be 45 §2 is amended: what `status` lists, and the new condition. ## Resolved, 2026-10-09 mesh-controller#150 and mesh-lab#68 (R325) merged on 2026-10-08 at 14:58 UTC. Every later controller -delivery carries #150; the last, of mesh-controller#170, finished its walk on 2026-10-09 at 00:11 UTC, and +delivery carries #150; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at 00:11 UTC, and `mesh-controller.status` named no machine behind. diff --git a/04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/00-report.md b/04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/00-report.md index 4f0f6999..2b8c8eca 100644 --- a/04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/00-report.md +++ b/04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/00-report.md @@ -20,7 +20,7 @@ Measured again on 2026-10-08 with the `nats` module's tools: 1.4 a second, or about 43 every 30 seconds. - Over the server's whole uptime (39 hours 49 minutes at 14:13 UTC) the average is 39.7 writes a second. -In that window one walk was in progress, so a few of those writes were real changes. The diagnosis +In that window one rollout was in progress, so a few of those writes were real changes. The diagnosis shows why most were not. ## Why it is an issue @@ -40,7 +40,7 @@ watcher, and the write rate cannot be used to see that deliveries are moving. ## How it is checked -- **In mesh-delivery's tests:** a reconcile pass over walks that did not change performs no write. The +- **In mesh-delivery's tests:** a reconcile pass over rollouts that did not change performs no write. The test's fake store counts writes, and the test fails on any write. - **Live:** over ten minutes in which no delivery changes state, the write count on `$KV.mesh-delivery_deliveries.>` (`nats.nats_subscriptions`) rises by no more than the number of @@ -56,6 +56,6 @@ build agent. So it resolves without a replay. The test above is still the proof. - `keep` writes only when the delivery differs from the value it last wrote. Either compare the encoded value with the last one written, or set a changed flag wherever the delivery is changed. -- `Reconcile` skips a walk whose revision the delivery already holds. +- `Reconcile` skips a rollout whose revision the delivery already holds. - A retry of an owed effect that fails again changes only its try count and next time. Whether that needs a write at every try is a choice for the fix: the count may stay in memory between writes. diff --git a/04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/01-diagnosis.md b/04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/01-diagnosis.md index 726655f4..97ec1439 100644 --- a/04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/01-diagnosis.md +++ b/04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/01-diagnosis.md @@ -12,16 +12,16 @@ runtime. `keep` does not compare the delivery with what it wrote before. Every c of transitions is empty. - `settleAndKeep` calls `settle` and passes the result to `moved`. A delivery with nothing to move is still written. -- `walkMoved` takes one walk from the controller, records it on its delivery (`linkWalk`) and calls - `settleAndKeep`. A walk that has not changed since the last pass leaves the delivery as it was, and +- `walkMoved` takes one rollout from the controller, records it on its delivery (`linkWalk`) and calls + `settleAndKeep`. A rollout that has not changed since the last pass leaves the delivery as it was, and the delivery is written anyway. -- `Reconcile` asks the controller's `delivery-walks` for the walks it keeps and calls `walkMoved` for each. - The answer holds every open walk and the last 50 that ended. The ticker in main.go runs `Reconcile` +- `Reconcile` asks the controller's `delivery-walks` for the rollouts it keeps and calls `walkMoved` for each. + The answer holds every open rollout and the last 50 that ended. The ticker in main.go runs `Reconcile` every 30 seconds. -So each pass writes about one delivery per walk the controller returns. On 2026-10-08 that was 50 walks -(32 done, 14 superseded, 3 failed, 1 walking), against about 43 writes per pass measured. *I think* the -difference is walks that match no delivery: `walkMoved` keeps those aside in memory and writes nothing. +So each pass writes about one delivery per rollout the controller returns. On 2026-10-08 that was 50 rollouts +(32 done, 14 superseded, 3 failed, 1 `walking`), against about 43 writes per pass measured. *I think* the +difference is rollouts that match no delivery: `walkMoved` keeps those aside in memory and writes nothing. **What made it 40 a second before today.** Before mesh-catalog #121 (the fix of [issue 313](../313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md), merged @@ -39,5 +39,5 @@ doubling to an hour) and ends them after 60 tries. A retry still writes its deli - *The groups bucket.* `mesh-delivery_groups` has 26 keys, and its subscription counted 719 writes in the same 39 hours. -**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every walk the +**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every rollout the controller keeps through `settleAndKeep` every 30 seconds. diff --git a/04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/00-report.md b/04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/00-report.md index 09544a14..00b0e0f3 100644 --- a/04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/00-report.md +++ b/04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/00-report.md @@ -24,7 +24,7 @@ Measured again on 2026-10-08 with the `nats` module's tools: The long-lived connections were young too. At 14:15 UTC the controller's two were 4 minutes old, the build agents' 2 to 19 minutes, the route proxies' 13 to 17 minutes and the tool runners' 10 minutes to 3 -hours. Two walks were sending new builds of the controller, the build agent and catalogue modules +hours. Two rollouts were sending new builds of the controller, the build agent and catalogue modules between 14:00 and 14:20 UTC, and the build agent's container on the control node started at 14:19 UTC. *I think* those young connections are restarts by deliveries; they are tens, not thousands. diff --git a/04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/01-diagnosis.md b/04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/01-diagnosis.md index b8abe10e..8aae7b50 100644 --- a/04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/01-diagnosis.md +++ b/04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/01-diagnosis.md @@ -17,7 +17,7 @@ controller's, so each place that needs the bus opens one, uses it and closes it: command. - `sayingOnTheBus` (delivery.go): a delivery's `check`, `go` and `stop`, which mesh-delivery asks for. -The serving controller dials too. Its watchdogs run every 30 seconds, and while a walk waits for a +The serving controller dials too. Its watchdogs run every 30 seconds, and while a rollout waits for a build, `buildSeatPause` (plan_retry.go) opens a fresh connection to read whether the build seat's holders paused, and closes it. `status` does the same. @@ -41,7 +41,7 @@ its long-lived connections are named `mesh-controller` too and are told apart on minute. The research's guess came from their age alone. **Located** in the controller: one connection per command process that touches the bus, and one per -watchdog tick while a walk waits for a build. +watchdog tick while a rollout waits for a build. ## 2026-10-09, resolved diff --git a/04-ISSUES/328-jetstream-api-errors-are-counted-and-attributed-to-nothing/01-diagnosis.md b/04-ISSUES/328-jetstream-api-errors-are-counted-and-attributed-to-nothing/01-diagnosis.md index 8a276330..e3a296e4 100644 --- a/04-ISSUES/328-jetstream-api-errors-are-counted-and-attributed-to-nothing/01-diagnosis.md +++ b/04-ISSUES/328-jetstream-api-errors-are-counted-and-attributed-to-nothing/01-diagnosis.md @@ -17,7 +17,7 @@ the fix is the attribution tool; with it, this diagnosis continues. - **A holder that never said it paused.** `PausedSaid` (controller, internal/link/queue.go) asks for the last message on each build holder's pause subject. A holder that never said one gets a not-found answer, which the server counts as an API error. The serving controller's watchdog reads it every 30 - seconds while a walk waits for a build (issue 327). + seconds while a rollout waits for a build (issue 327). - **Short-lived consumers.** The controller heard 495 "consumer deleted" advisories in the first 8 minutes of its current connection, about one a second. They are short-lived consumers: the controller's advisory reader (internal/link/advisories.go) ignores them and says why, "every watch of a bucket and diff --git a/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/00-report.md b/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/00-report.md index e80dfe07..1276a224 100644 --- a/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/00-report.md +++ b/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/00-report.md @@ -7,12 +7,12 @@ opened: 2026-10-08 ## What was observed On 2026-10-08 a person released a delivery of the catalogue that the group check (issue 316) had held. Its -walk had one tier with one module, `dunst`, which runs on the laptop and the workstation. The walk sent the +rollout had one tier with one module, `dunst`, which runs on the laptop and the workstation. The rollout sent the new build to the laptop first, and the gate passed it there: > dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent -Twenty-eight minutes later the same walk said the opposite, about the same gate: +Twenty-eight minutes later the same rollout said the opposite, about the same gate: > dunst failed its gate on the laptop in tier 0: healthy 3 times over 2m5s; put back dunst to ea462666 on > the laptop, in one send; the workstation left as it was @@ -31,22 +31,22 @@ build that worked, recorded it as failed, and so blocked its delivery until a ne ## Fix -The controller judged the first machine's report again on every step of the walk, also after the gate had +The controller judged the first machine's report again on every step of the rollout, also after the gate had passed (see `01-diagnosis.md`). The fix merged in mesh-controller pull request 165 (8170fc5), with its replay in mesh-lab pull request 72. - Once a module's gate has passed, the first machine's later reports no longer decide anything about it: - the walk sends the rest. + the rollout sends the rest. - A module carried by another module's send takes over that pass before its step is read. -- A guard, not reached with the two changes above: should a later path still stop a walk at a module whose - gate passed, that module is not put back and its build is not marked failed. The walk stops with the +- A guard, not reached with the two changes above: should a later path still stop a rollout at a module whose + gate passed, that module is not put back and its build is not marked failed. The rollout stops with the reason, and the module is left sent to its first node only, which `plans retry` takes up again. ## How it is checked `TestReplay335` in the controller replays the live case with only what the controller had before the fix: a passed gate on the laptop, the laptop's later report not about this build (or a failure of another -send), and the walk 30 minutes and 9 seconds after the first send. A second test drives the walk over a +send), and the rollout 30 minutes and 9 seconds after the first send. A second test drives the rollout over a tier of two modules, the second carried by the first's send, and fails if the carried module's step is read before it takes over the pass. On the commit before the fix it fails with "laptop did not report it applied within 30m0s", and the passed build reaches the put-back. It is R335 diff --git a/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/01-diagnosis.md b/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/01-diagnosis.md index 860a2457..9ef67586 100644 --- a/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/01-diagnosis.md +++ b/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/01-diagnosis.md @@ -6,15 +6,15 @@ found ADR 0236 (what a rollback is), issue 318 (a wait for a login failing a gate) and issue 279, none of them about a gate that passed and was then failed. -**The walk's own lines.** The controller's journal on the control-node said, for this walk (times UTC): +**The rollout's own lines.** The controller's journal on the control-node said, for this rollout (times UTC): | time | what the controller said | |---|---| | 16:44:45 | tier 0 built; sent dunst to the laptop first, the rest once its gate passes | | 16:44:55 | the laptop applied what it was sent | | 16:46:52 | dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent | -| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another walk sends | -| 16:49:12 | the other walk (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation | +| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another rollout sends | +| 16:49:12 | the other rollout (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation | | 17:14:54 | dunst failed its gate on the laptop: healthy 3 times over 2m5s; put back on the laptop; the workstation left as it was | The laptop's node-engine logged `applied` at 16:44:55 and nothing more until 18:01:58. It did not report @@ -25,14 +25,14 @@ on the 16:49:12 send for over an hour. Why it did not is not yet known; the lapt waits 30 minutes (`planWaitBound`) for a first machine to report what it was sent. The failure came at that bound and not at the gate's. -**The code path.** In `advanceOnce` (`release_plan.go`), every step of a walk reads +**The code path.** In `advanceOnce` (`release_plan.go`), every step of a rollout reads `nextRollout(state, …)` before anything else, for every module in the tier not yet sent to the rest. For a module sent first, `nextRollout` looks at the first machine's last report. If that report is not about the declaration the machine was last sent (`Current` false), it waits; past the bound it answers `failed: did not report it applied within 30m0s`. Nothing in it looked at the gate's verdict. -After the pass at 16:46:52, the send to the rest was refused (the error returns, and the walk tries again on -the next step). At 16:49:12 the other walk's send made the laptop's last report no longer current. From +After the pass at 16:46:52, the send to the rest was refused (the error returns, and the rollout tries again on +the next step). At 16:49:12 the other rollout's send made the laptop's last report no longer current. From then on `nextRollout` waited for a report it would never get about this build. At the bound it failed, and `advanceOnce` called `failFirstSend`. @@ -47,7 +47,7 @@ laptop. That is the line seen at 17:14:54. - The person-wait of issue 318 (ADR 0254), `putBackBroken` and the carried-module paths: the plan had one module, nothing was carried, and nothing was found broken. - Sleep: the laptop did not sleep until 18:01, and the failure was at 17:14:54. -- The release itself (`mesh-delivery.release`): it let the walk go at 16:44, and the walk ran as any other. +- The release itself (`mesh-delivery.release`): it let the rollout go at 16:44, and the rollout ran as any other. **Located.** mesh-controller, `cmd/mesh-controller/release_plan.go`: `nextRollout` judged a passed gate's first machine again, and `failFirstSend` put back a module whose gate had passed. The path has been there @@ -57,4 +57,4 @@ step as the pass and another send reaches the first machine before the 30-minute **What it needs to deliver dunst.** The build is recorded as failed at its gate, and the controller does not send a build marked that way again on its own. The registered build is back at the one before. Once the fix is delivered, a new build of dunst (`rebuild dunst`, or a newer merge of it) is judged at the gate -again and walked to both machines. +again and rolled out to both machines. diff --git a/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md index 00616942..c7260d3e 100644 --- a/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md +++ b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md @@ -20,10 +20,10 @@ From the merge on, every send to the control-node was held. `mesh-controller.pla > planned step: sending control-node would replace the bus (nats 88135ad0 → 32307bd1), which is a planned > step ... — tried again for 27m45s -The same refusal held the controller's own walk and the release of builds kept back for a gate. +The same refusal held the controller's own rollout and the release of builds kept back for a gate. For those 28 minutes `mesh-controller.conditions` listed **no condition** naming the hold or saying it -waited for a person. The operator was not told. The hold was found only because a walk looked stuck, and +waited for a person. The operator was not told. The hold was found only because a rollout looked stuck, and the operator asked: "where would you be waiting then? Where is my approve button?" The operator then gave the word, the bus upgrade started at 18:52 and the bus answered healthy at 18:54. @@ -33,7 +33,7 @@ The held sends then went through. The design makes the bus wait for a person, and that part worked: nothing replaced the bus unasked. But a wait that only a person can end is said nowhere a person looks. The refusal is kept as the note on each -held walk, which is read only by somebody already asking `plans`. Rule 4 of +held rollout, which is read only by somebody already asking `plans`. Rule 4 of ADR 0227 (no failure is silent) is met in the letter and missed in effect: the mesh held every send to its most important machine and said nothing until asked. @@ -47,10 +47,10 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md). ## Impact -- Every send to the control-node stops: the controller's own walk, the node-engine's, every module's walk +- Every send to the control-node stops: the controller's own rollout, the node-engine's, every module's rollout that includes the control-node, and gated builds waiting for release. Other machines in the same tier wait with it. -- The operator learns of it only by noticing a stuck walk, which can take hours when nobody is watching. +- The operator learns of it only by noticing a stuck rollout, which can take hours when nobody is watching. - The only condition that could have fired, a tier past its bound (watchdog S3, "stalled"), fires no sooner than 30 minutes in, and its words ("stuck halfway ... longer than usual. Nothing is lost") name neither the bus nor the verb a person runs. It never fired in this case, because the step was taken @@ -62,9 +62,9 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md). | Rule | Check | |---|---| -| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a walk wants to send raises a condition on the next watchdog tick, before any tier bound | -| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held walks or modules, the bus build from and to, and `mesh-controller.bus upgrade` | -| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a walk whose only refusal is the bus's planned step | +| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a rollout wants to send raises a condition on the next watchdog tick, before any tier bound | +| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held rollouts or modules, the bus build from and to, and `mesh-controller.bus upgrade` | +| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a rollout whose only refusal is the bus's planned step | | it clears when the step starts | the test ends with a bus step started: the condition clears, `bus-maintenance` opens | | the merge says it before it happens | the merge gate's test: a change that moves the bus's build reports that a bus upgrade by a person is needed and which machine waits until then | @@ -81,8 +81,8 @@ why none is possible. with the operator as its resolver). Its headline says, in plain words, what waits ("sends to the control-node: the controller, the node-engine, …"), behind what (the bus build, from and to), since when, and the verb that ends it (`mesh-controller.bus upgrade --why … --reversible|--irreversible`). It - is raised on the first refusal, not after a bound. A walk held only by it is excluded from S3, as a - walk held by a paused build seat is. + is raised on the first refusal, not after a bound. A rollout held only by it is excluded from S3, as a + rollout held by a paused build seat is. 2. **An ask, once asks exist.** When the decision flow of ADR 0259 (an ask is a message on the bus, any channel answers it; on an open pull request, not yet merged) is built, the condition becomes an ask to the operator: "start the bus upgrade now?", whose yes runs the step with the operator's word diff --git a/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md index b54a2a46..6775c0e4 100644 --- a/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md +++ b/04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md @@ -13,13 +13,13 @@ The send is refused whole. `busHeld` (`bus_step.go`) gives the words "sending X (nats from → to), which is a planned step: `bus upgrade --why …` …". So the remedy is in the words, but only in the error. -**Where the error goes.** In `release_plan.go`, the walk's step loop catches the error, keeps it as the +**Where the error goes.** In `release_plan.go`, the rollout's step loop catches the error, keeps it as the plan's note ("tier N: … — tried again"), saves the plan when the note changed, prints it to the controller's log once, and tries again on the next tick. Nothing raises a condition from it. That is why `plans` showed it and `conditions` did not. **Which watchdog could have said it.** Only S3 (`watchPlans`, `signals.go`), "plan tier progress", raises -`plan..stalled` when a walk has been in one tier past its bound. The bound is +`plan..stalled` when a rollout has been in one tier past its bound. The bound is `max(30 min, 3 × the p90 of that repository's measured tiers)` (`tierAtLeast = planWaitBound = 30 min`, `watchdogs.go` `boundsOf`). So: @@ -29,14 +29,14 @@ controller's log once, and tries again on the next tick. Nothing raises a condit mesh keeps it where it is."). Its summary carries the plan's note, so the bus's reason would have appeared there, but the headline a person reads names neither the bus nor the verb, and its resolver is not the operator; -- a walk held by a paused build seat is excluded from S3 (`pausedWaiting`), and a walk waiting for its - delivery's word has its own watchdog (S16). A walk held for the bus step has neither. +- a rollout held by a paused build seat is excluded from S3 (`pausedWaiting`), and a rollout waiting for its + delivery's word has its own watchdog (S16). A rollout held for the bus step has neither. **What the bus step does raise.** Probe DB (`probeBusStep`, `bus_step.go`) raises `bus-maintenance` only while a step runs, and `bus-upgrade-failed` when one ended unhealthy. It reads only the latest bus step. Before a step is started there is nothing for it to read, so a pending bus build raises nothing. -**Before the merge.** `delivery.go` lists the bus among the modules whose walk starts at the merge +**Before the merge.** `delivery.go` lists the bus among the modules whose rollout starts at the merge (`onTheControllersPath`), and the merge gate (`merge_gate.go`) composes the machines with the change. Neither reads `pendingBus` or says that the change will hold the bus's machine until a person runs the step. So a merge that moves the bus's build passes green and blocks every later send to the control-node. @@ -64,12 +64,12 @@ person, and the person must be told. ADR 0271 (proposed) takes suggested fixes 1 and 3. mesh-controller #173 builds them: -- **Row S17**, `bus..step-waiting`: raised on the first watchdog tick after a walk's send was refused +- **Row S17**, `bus..step-waiting`: raised on the first watchdog tick after a rollout's send was refused for the bus's planned step, while the new bus build would still replace the bus on its machine. It is a - warning, for the operator. It names the machines, each waiting walk and the modules of its tier, the bus build + warning, for the operator. It names the machines, each waiting rollout and the modules of its tier, the bus build from and to, since when, and the `mesh-controller.bus` call with `upgrade`. It clears once the bus's machine has been sent the build the mesh holds. -- **S3 leaves out a walk held only by it.** +- **S3 leaves out a rollout held only by it.** - **The delivery plan says it before the merge**: "merging this needs a person's bus upgrade: … nothing else is sent to either", and its summary, which `mesh/merge-gate` carries, says the same. diff --git a/04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/00-report.md b/04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/00-report.md index 77726660..a4b0b6f8 100644 --- a/04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/00-report.md +++ b/04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/00-report.md @@ -20,7 +20,7 @@ The plan rebuilt and moved them whatever the merge changed. Two examples: - Mesh-controller PR #159, merged as `ca09a07f`, changed files under `cmd/mesh-controller`, `internal/broker`, `internal/link`, `internal/catalogue/verbs.go` and the root `module.json`. Nothing - under `examples/route-proxy` changed. Its walk said, on the home server, "recreates 1 of route-proxy's + under `examples/route-proxy` changed. Its rollout said, on the home server, "recreates 1 of route-proxy's 3 container(s) (with a new image: server)" and "recreates 1 of build-agent's 1 container(s)". - Mesh-controller PR #163, merged as `9b028b42`, did the same. The route proxy's program at that commit is byte for byte the program at the commit before it (measured in the diagnosis), and it was still @@ -43,8 +43,8 @@ The operator stated the rule the delivery plan should follow: ## Why it is an issue -- **Deliveries are slow.** A controller merge walked for about fifteen minutes instead of about five, - and the walks of one repository must not overlap (issue 335), so every merge after it waits too. The +- **Deliveries are slow.** A controller merge rolled out for about fifteen minutes instead of about five, + and the rollouts of one repository must not overlap (issue 335), so every merge after it waits too. The operator considers this a major bug and a main reason deliveries are slow. - **A public service restarts for nothing.** The route proxy on the home server was recreated at merges that did not change its program. diff --git a/04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/01-diagnosis.md b/04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/01-diagnosis.md index 492720d3..58694fee 100644 --- a/04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/01-diagnosis.md +++ b/04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/01-diagnosis.md @@ -31,7 +31,7 @@ Read in mesh-controller at `8170fc58`. 4. `mesh-controller` itself is built from the repository's root (its manifest's bundle compiles `cmd/mesh-controller`; its source path is empty), so every changed file in the repository touches it as well. -5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the walk serial here: +5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the rollout serial here: - a `packages` edge, from a module whose build read a repository to the module whose source that repository is. So `route-proxy` and `build-agent` both depend on `mesh-controller`, because they share its repository, not because either is built from the controller's artifacts; @@ -108,9 +108,9 @@ change. The build source of the route proxy is `examples/route-proxy`, `internal ## Measured: what the extra tiers cost From `mesh-controller.plans` (read only), the first-node gate of each tier of tonight's controller -walks: +rollouts: -| walk | mesh-controller | build-agent | route-proxy | three gates | +| rollout | mesh-controller | build-agent | route-proxy | three gates | |---|---|---|---|---| | `3f68a495` | 3m40s | 3m43s | 3m46s | 11m09s | | `0cf5a3a5` | 3m39s | 3m26s | 3m34s | 10m39s | @@ -120,12 +120,12 @@ walks: | `ca09a07f` | 3m43s | 3m46s | 3m40s | 11m09s | | `efcdd5dd` | 3m38s | 3m40s | 3m37s | 10m55s | -Two more walks (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still walking at -21:50. The gates alone are 10 to 11 minutes per walk, of which 7 minutes are the two packaging tiers. +Two more rollouts (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still rolling out at +21:50. The gates alone are 10 to 11 minutes per rollout, of which 7 minutes are the two packaging tiers. Each tier also builds before it sends: the build seat's image took 53 seconds at 21:36. Estimated from -the plan's creation time and the build records, `ca09a07f` walked from 21:07 to about 21:23 and +the plan's creation time and the build records, `ca09a07f` rolled out from 21:07 to about 21:23 and `efcdd5dd` from 21:32 to about 21:47, about 15 minutes each. The controller's own tier, alone, is about -4 to 5 minutes. Of the seven completed walks, five recreated `route-proxy` on the home server +4 to 5 minutes. Of the seven completed rollouts, five recreated `route-proxy` on the home server (`3f68a495`, `e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`), and its program had changed in one of them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`). @@ -133,7 +133,7 @@ them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07 The deploy plan names a build of a packaging module by the module's own source commit, which is the catalogue's. `moveFacts.identical` (cmd/mesh-controller/release.go) calls two builds identical when -their commits are equal. In `efcdd5dd`'s walk both modules went `7b307a06 → 7b307a06`, nothing was +their commits are equal. In `efcdd5dd`'s rollout both modules went `7b307a06 → 7b307a06`, nothing was said to be recreated, and the gate still took about 3.5 minutes for each. Yet the route proxy's program did change at `efcdd5dd` (table above). **I do not know whether that program reached the home server.** Either it was sent and the plan said nothing, or it was not sent and the route proxy runs the @@ -146,7 +146,7 @@ commit would also make issue 280's "repeat" rule register a different program as - **The changed paths not reaching the planner.** PR #159's delivery record carries its paths and `module_dirs_said: true`. They are read, and `touchedBy` uses them for the controller's own module. They are never consulted for a packaging module. -- **The catalogue moving at the same time.** In some walks the catalogue's commit also changed +- **The catalogue moving at the same time.** In some rollouts the catalogue's commit also changed (`32307bd1 → 7b307a06`), but the README-only what-if shows the controller's repository alone is enough to plan both modules. @@ -181,7 +181,7 @@ the commit into the program. with the build; that is exact for Go and needs a reader per language. 2. **Reproducible builds**: `-buildvcs=false` (or a `.dockerignore` that leaves `.git` out of the context) in both recipes, so an unchanged source gives an identical program. This alone does not - stop the walk: the image digest still differs (issue 280), and the planner would still plan and gate + stop the rollout: the image digest still differs (issue 280), and the planner would still plan and gate three tiers. It is cheap and worth doing beside 1, so the program's identity no longer depends on the commit. 3. **Split the route proxy and the build seat into repositories of their own.** Then the existing @@ -191,5 +191,5 @@ the commit into the program. and a change to that library then moves all three again, through a real build edge. Recommendation: 1 with its check, and 2 beside it. A further question for the fix: whether `built-by` -should order a walk at all. It is a running-order edge for the build seat's worker (issue 206); the +should order a rollout at all. It is a running-order edge for the build seat's worker (issue 206); the operator's rule 2 counts only modules built *from* a changed module. diff --git a/04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/00-report.md b/04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/00-report.md index 9801921b..0aca1df6 100644 --- a/04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/00-report.md +++ b/04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/00-report.md @@ -23,7 +23,7 @@ What happened, in order (UTC): **under the same JSON key `declared`**. The two edits touched different lines, so git merged them without a conflict. The result does not compile: `internal/store/store.go:145:2: Declared redeclared`. -4. **00:55.** The walk the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build +4. **00:55.** The rollout the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build in tier 0". `mesh/delivery` on the head reads `failure`. Nothing was sent to a node, and every node kept `d28ca95`. 5. **01:09.** Node-engine #67 repaired the trunk. It renamed #66's field to `DeclaredUnit`, key @@ -33,7 +33,7 @@ The facts behind the "different lines" claim, read from the node-engine's reposi contain `d28ca95`. Their common ancestor is `a2c2c35` (#63's merge). `dfb1415` holds one `Declared` (store.go line 107), `d28ca95` holds the other (line 139), and `690966d` holds both (lines 107 and 145). -No node was harmed, because a walk builds before it sends. But the merge check said "this may merge" of +No node was harmed, because a rollout builds before it sends. But the merge check said "this may merge" of a merge that did not build, and both required statuses were green when it merged. ## The cost before this: checks that went stale diff --git a/04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/01-diagnosis.md b/04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/01-diagnosis.md index 5f0e0ded..dbfefe11 100644 --- a/04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/01-diagnosis.md +++ b/04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/01-diagnosis.md @@ -33,7 +33,7 @@ merging made, with two, was never built before the merge. or ready delivery back to proposed. Its comment says "a person asked for it again, or its group changed". - `holder.go`, `PullMerged` (line 347): a merge records the merge commit on that pull request's own - delivery and links its walk. It does not touch the repository's other open deliveries into the same + delivery and links its rollout. It does not touch the repository's other open deliveries into the same trunk. - The forge marks a head pending only when it announces a new head (issue 305). A merge into the trunk is not a new head of any other pull request. diff --git a/04-ISSUES/344-known-gaps-in-the-agents-boundary-accepted-for-now/00-report.md b/04-ISSUES/344-known-gaps-in-the-agents-boundary-accepted-for-now/00-report.md index 6f6769c6..f4612aa9 100644 --- a/04-ISSUES/344-known-gaps-in-the-agents-boundary-accepted-for-now/00-report.md +++ b/04-ISSUES/344-known-gaps-in-the-agents-boundary-accepted-for-now/00-report.md @@ -60,14 +60,14 @@ note. Each gap says what is open, why it matters, and what would close it. 9. **On a large data pool the root search may never finish.** The node-engine's search for programs that run as root now has one bound of fifteen minutes (`mesh-host` #61; the inner two-minute limit is gone). On a machine with - a very large data pool the walk can still run past it, and the verdict then stays "not judged". That is loud, + a very large data pool the traversal can still run past it, and the verdict then stays "not judged". That is loud, not silent. - *Closes with:* setuid turned off on the data datasets, so the walk skips them. + *Closes with:* setuid turned off on the data datasets, so the traversal skips them. 10. **An agent can merge its own pull request.** No mesh repository's main requires an approval on the forge (`required_approvals: 0`), and the forge tool merges as the forge's administrator once the required checks are green. An agent can therefore change a module that runs on every node, open the pull request, wait for the - checks and merge it, and the controller walks the change out. The checks judge whether every machine still + checks and merge it, and the controller rolls the change out. The checks judge whether every machine still composes, not what the change is for. The operator's yes in a session is a convention, and nothing checks it. *Closes with:* a merge that needs the operator's own approval: a required approval on the forge from the operator's personal account only, or an approval through ADR 0259's asks. That holds only when the forge's @@ -98,7 +98,7 @@ note. Each gap says what is open, why it matters, and what would close it. 14. **Ways to root the node-engine's judge does not see.** ADR 0266 §3 lists them under *not judged, so not seen*: among them a root-run unit or script the agent account can write, a directory on root's `PATH` it can write, - root's ssh keys readable by it, and a setuid program on a filesystem the search does not walk. None is known + root's ssh keys readable by it, and a setuid program on a filesystem the search does not traverse. None is known on the control-node, but a healthy verdict says nothing about them. The record left them "for a later record" with nothing tracking it; added here on 2026-10-10. *Closes with:* each way judged by the node-engine beside the others, or a record that decides against judging diff --git a/04-ISSUES/348-a-resolver-that-stopped-answering-ran-on-unnoticed-and-builds-sent-after-it-were-blamed/01-diagnosis.md b/04-ISSUES/348-a-resolver-that-stopped-answering-ran-on-unnoticed-and-builds-sent-after-it-were-blamed/01-diagnosis.md index 8a6fd1c5..318413e3 100644 --- a/04-ISSUES/348-a-resolver-that-stopped-answering-ran-on-unnoticed-and-builds-sent-after-it-were-blamed/01-diagnosis.md +++ b/04-ISSUES/348-a-resolver-that-stopped-answering-ran-on-unnoticed-and-builds-sent-after-it-were-blamed/01-diagnosis.md @@ -16,14 +16,14 @@ stops answering while it runs. | 10:57:57 | the route proxy's first "lookup … on :53: … connection refused" | the route proxy's log | | 10:58:36 | `machine..network` raised: one of its two resolvers does not answer | conditions | | 10:58:45 | `machine..network` raised; the home server's cleared, held under it | conditions | -| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the walks | +| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the rollouts | | 10:59:37 | `module.step-ca..unhealthy` raised | conditions | | 10:59:42 | `machine..network` **cleared**, its newest evidence (10:59:37) still "connection refused" | conditions | | 11:00:23 | `machine..network` reopened, raised time 11:00:23 | conditions | | 11:06:00 | `seat.mesh-dns-resolver..wrong` raised by D2 | conditions | | 11:10:29 | both builds failed their gate: " as a whole: raised since it was sent: machine..network" — put back | conditions | | 11:18:12 | `dnsmasq.service` restarted by hand; every question answered again | the operator | -| 11:21:46 | `machine..core-behind` raised: "… and no plan is rolling them out", while a release walked the node-engine | conditions | +| 11:21:46 | `machine..core-behind` raised: "… and no plan is rolling them out", while a release rolled out the node-engine | conditions | ### What is measured, and what is guessed @@ -54,12 +54,12 @@ answered none of its questions for the next run (`Confirm`, issue 277), because question can be lost. Its first run after 10:57:57 held it; the next, at 11:06:00, raised it. Every one of those questions had been refused, which no load explains. -**Measured: why the second walk of the node-engine waited on its own first machine.** A release keeps no +**Measured: why the second rollout of the node-engine waited on its own first machine.** A release keeps no record per module — its modules are in its tier only — and D10 read only the records to know what a plan -is rolling out. So while a release walked the node-engine at 11:21, D10 raised `core-behind` on every +is rolling out. So while a release rolled out the node-engine at 11:21, D10 raised `core-behind` on every machine, "and no plan is rolling them out", and the gate on the home server, its first machine, waited on it (issue 281's settle window) until D10's next run five minutes later found the new engine there. -The walk then passed. It did not fail, but it waited five minutes for a condition its own send caused. +The rollout then passed. It did not fail, but it waited five minutes for a condition its own send caused. **Not explained: why the second resolver did not save the lookups.** Every machine's resolver file lists both resolvers (`options timeout:1 attempts:2`), and the control node's own file lists its private address @@ -153,7 +153,7 @@ registry, is the operator's decision. - An undecided part of a network statement — `unknown` or `starting`, as a node-engine just started says — holds the open network conditions whose newest evidence names that part, and only those. A statement undecided as a whole holds every part. The next statement decides. -- D10 counts what a release's tier names as rolling out, so a walked node-engine is not `core-behind` on +- D10 counts what a release's tier names as rolling out, so a rolled out node-engine is not `core-behind` on its own first machine. - D2 is unchanged: a resolver that answered nothing, refused or silent, is held for the next run, because a few hundred milliseconds of refusals is a resolver restarting as well as one that stopped. D2 says it diff --git a/04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md b/04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md index a884f253..43b8e57b 100644 --- a/04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md +++ b/04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md @@ -61,8 +61,8 @@ A replay in mesh-lab's register is still owed before this core issue resolves (A ## A sequel (2026-10-10) -Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a walk: -mesh-delivery matches a walk to the delivery merged as its commit, so two catalogue merges 18 seconds apart -left the earlier one held for a person and the walk carrying it waiting for a word nothing gave, for 58 +Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a rollout: +mesh-delivery matches a rollout to the delivery merged as its commit, so two catalogue merges 18 seconds apart +left the earlier one held for a person and the rollout carrying it waiting for a word nothing gave, for 58 minutes, with no condition raised. See [issue 362](../362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md). diff --git a/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md b/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md index e1040d11..94a444b0 100644 --- a/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md +++ b/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md @@ -2,7 +2,7 @@ opened: 2026-10-09 --- -# 351. A held delivery whose module a backlog walk carried to every machine is never covered +# 351. A held delivery whose module a backlog rollout carried to every machine is never covered ## Symptom @@ -19,12 +19,12 @@ What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `ha | time | what happened | |---|---| -| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word | -| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop | -| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded | -| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) | -| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 | -| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours | +| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its rollout waited for its word | +| 10-08 11:06 | three later catalogue merges took its rollout over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop | +| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the rollout ended `failed`, then superseded | +| 10-08 14:27 | #127 merged as bf542128; its rollout built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) | +| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog rollout sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 | +| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog rollout, as it had for 24 hours | No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever move it again (ADR 0267): @@ -32,22 +32,22 @@ left as it is, the delivery stays held for ever. ## Why it is an issue -ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running +ADR 0270 covers a held merge when a later delivered merge's rollout "sent the module to every machine running it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge commit, which the forge says contains a6385479. The decision's question — does this change go on? — was answered by the operator's own release of the build. Yet the record cannot end: -- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what +- **A backlog rollout's end is read by nothing.** While it judges, the controller's record of it names what its gate carried, and mesh-delivery credits the delivery that published each carried build one step per machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing from that: the steps stay `judging` for ever. -- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running - it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the +- **`covered` reads only a rollout's own aggregate steps.** A rollout's own record says "every machine running + it" or "the rest running it"; a step per machine, which is all a backlog rollout gives, never counts as the module sent everywhere. ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module whose build reached its machines on the operator's word, after the mesh held it back, rather than by the -walk of the merge that built it. A held delivery in that case has no end but a person's wrong act +rollout of the merge that built it. A held delivery in that case has no end but a person's wrong act ([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and the condition it raises every hour is one nobody can clear correctly. @@ -56,7 +56,7 @@ the condition it raises every hour is one nobody can clear correctly. - A forge that could not answer: the holder's journal has no "not yet covered" line for it, and `gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128. - A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps. -- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is +- The rollout that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is rightly no coverer. -- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module +- A person's `release` of #125 (the verb): it would roll out a6385479, an older commit than the one the module was built from, and is what ADR 0270 ruled out. diff --git a/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/01-diagnosis.md b/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/01-diagnosis.md index 51623a41..ca6c85ee 100644 --- a/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/01-diagnosis.md +++ b/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/01-diagnosis.md @@ -2,56 +2,56 @@ **2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`, at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127, -`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`, +`mesh-controller.delivery-walks` for #127's rollout and for the backlog rollout, `plans`, `hand-acts`, `module list`, the forge's `gitea_contains`, and the holder's journal on the control node. -**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a +**Where a backlog rollout's steps come from.** `walkMoved` (holder.go) hands a rollout whose record carries a `release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives the delivery whose merge commit each carried build was made from one step per machine: `judging` while the -gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk +gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog rollout holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`. `creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43 -and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the -module was not in that walk's tiers, and the backlog walk is a walk of its own.) +and 14:46 UTC were never moved on. (The controller's own record of #127's rollout never named the module: the +module was not in that rollout's tiers, and the backlog rollout is a rollout of its own.) **Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only -on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running +on a `sent` step whose machine is the rollout's own aggregate, "every machine running it" or "the rest running it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is found, and the forge is never asked — which is why the journal has no line. -**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a -backlog walk that failed a gate ends `FAILED at tier … failed its gate on ` and puts the builds back +**What "done on a machine" means for a backlog rollout.** Read from the controller's records, not its code: a +backlog rollout that failed a gate ends `FAILED at tier … failed its gate on ` and puts the builds back (seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done, with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis is wrong there. -**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb — +**Ruled out.** A forge that could not answer, a coverer forgotten, the failed rollout, and the `release` verb — see the report. -**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for +**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog rollout's `done` machines for nothing, and `carries`, which counts no per-machine step. ## The fix In mesh-catalog #163, one pull request, in `modules/mesh-delivery`: -- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that - walk (`carried by `), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A - step it failed on, and a step another walk carried, are left as they are. +- **A finished backlog rollout passes the steps it carried.** `creditReleased`: every step credited to that + rollout (`carried by `), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A + step it failed on, and a step another rollout carried, are left as they are. - **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when - the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held + the rollout's own aggregate step says so, as before, or when a `passed` step stands on every machine the held delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still judging, one failed or one never sent leave the delivery held. -- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its - open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was +- **A rollout that carried a step still judging is asked for by its id.** The controller lists back only its + open rollouts and the last fifty ended at a start; the backlog rollout of this case, ended the day before, was no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for - such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes + such a rollout by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes its record as said. Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own -walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's. +rollout of its merge commit on the same trunk, and the forge must still say that commit contains the held one's. The record's consequence that named the one-machine gap gains a progressive insight naming this one. **How it is checked**, in `covered_test.go`: @@ -59,15 +59,15 @@ The record's consequence that named the one-machine gap gains a progressive insi - `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered; still judging on one, failed on one, never sent to one, or a plan naming no machines stays held. - `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one - and leaves the other judging; a failed step stays failed; a step another walk carried is not touched. + and leaves the other judging; a failed step stays failed; a step another rollout carried is not touched. - `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held - while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the + while the backlog rollout judges, covered once it is done and read by its id, naming the later merge and the module. All three fail on the catalogue's main before the change and pass after it. **After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are -asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once +asked for by the backlog rollout's id and pass; #127 then carries the module for #125; the forge is asked once whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the acceptance of this issue, read with `mesh-delivery.show` of #125. diff --git a/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/00-report.md b/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/00-report.md index c7f3a8e0..84ca8758 100644 --- a/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/00-report.md +++ b/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/00-report.md @@ -21,7 +21,7 @@ On 2026-10-09, local times on the control node (its controller's and node-engine migrations than the store had applied, and served what it could. - 16:35:59 that older controller judged the newer plan's gate **passed** on the control node ("healthy 3 times over 4m17s") — from the health of the build it was itself, not of the build the plan had sent, - which the put-back had removed from the machine. The plan walked on, believing the controller delivered. + which the put-back had removed from the machine. The plan rolled out on, believing the controller delivered. ## Why it is an issue @@ -35,8 +35,8 @@ Four rules did not hold. gate of the build it had replaced. - **A put-back puts back something that can run the mesh** (ADR 0236: "put back by something other than itself"). A controller older than the store's schema starts behind its own records. Nothing refused it. -- **One walk of a module on a machine at a time** (issue 281, ADR 0236). A release's walk was invisible to - the merge plan's send: a release keeps no module records, and the check for a walk in progress read +- **One rollout of a module on a machine at a time** (issue 281, ADR 0236). A release's rollout was invisible to + the merge plan's send: a release keeps no module records, and the check for a rollout in progress read only those. The diagnosis and the fix are in [`01-diagnosis.md`](01-diagnosis.md). diff --git a/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/01-diagnosis.md b/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/01-diagnosis.md index 89e50d15..2e608ec9 100644 --- a/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/01-diagnosis.md +++ b/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/01-diagnosis.md @@ -25,10 +25,10 @@ read as a module's) are the nearest; neither is this. to the one the first machine ran before (`PreviousBuild`) and sends it. Nothing asked whether that build can read the store as the failed build left it. The controller says so itself when it starts ("this control plane is behind the mesh-controller row"), and serves anyway (ADR 0185). -- **The merge plan's send crossed the release's gate.** `walkedBy` finds a walk in progress by a plan's - module records (`p.Modules[module].First`). A release (the walk of the builds waiting for a gate) keeps its walk in its gate's carried moves, +- **The merge plan's send crossed the release's gate.** `walkedBy` finds a rollout in progress by a plan's + module records (`p.Modules[module].First`). A release (the rollout of the builds waiting for a gate) keeps its rollout in its gate's carried moves, not in module records, so the release's open gate on the control node held nothing back. And a plan's - own modules were never checked against another walk at all: the check ran over the moves a machine + own modules were never checked against another rollout at all: the check ran over the moves a machine had waiting, and skipped the plan's own. ### What is guessed @@ -62,13 +62,13 @@ In mesh-controller, pull request 185. never recorded what it reaches, is not put back; the current build is kept, and the condition (`rollback-failed`, urgent) says why and that a person decides. The lease holder's `Build` is that version too. -- **A walk of a module on a machine holds every other send of another build of it there.** `walkedBy` - reads a release's open gate as its walk, and a plan's own modules are held by it as any other move is: +- **A rollout of a module on a machine holds every other send of another build of it there.** `walkedBy` + reads a release's open gate as its rollout, and a plan's own modules are held by it as any other move is: a merge plan whose first machine is under a release's gate for the same module waits, said as "waiting for … which release-… is walking". The same build to the same machine is not held: it is already there. -Not changed: whether a release's walk and a merge plan's walk of the core should be one walk (the walk +Not changed: whether a release's rollout and a merge plan's rollout of the core should be one rollout (the rollout carrying over) is left to a design; serialising them is what this fixes. ## How it is checked @@ -79,7 +79,7 @@ carrying over) is left to a design; serialising them is what this fixes. | A send that moves the module supersedes the judging, puts nothing back | `TestReplay352`, `TestAPlansJudgingIsSupersededByASendThatMovesItsModule` | | A report is held against the gate's own send, by digest or sequence | `TestAReportIsHeldAgainstTheGatesOwnSend`, inventory `TestASendAndAReportAreKnownByTheirDeclaration` | | Only the build last sent is judged | `TestAGateJudgesOnlyTheBuildTheMachineWasLastSent` | -| A walk holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` | +| A rollout holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` | | No put-back to a controller behind the store, or one unproved | `TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema`, inventory `TestASchemaReachIsKeptByBuild`, catalogue `TestAProcessEnvCanNameTheBuildsOwnVersion`, `TestTheHolderNamesTheBuildTheDeclarationToldIt` | | Every condition of the change is tested | 31 mutations, each failing a test | diff --git a/04-ISSUES/353-the-controller-asked-before-the-bus-held-its-grant/00-report.md b/04-ISSUES/353-the-controller-asked-before-the-bus-held-its-grant/00-report.md index e62d916d..5a672c6b 100644 --- a/04-ISSUES/353-the-controller-asked-before-the-bus-held-its-grant/00-report.md +++ b/04-ISSUES/353-the-controller-asked-before-the-bus-held-its-grant/00-report.md @@ -9,7 +9,7 @@ opened: 2026-10-09 On 2026-10-09, local times on the control node, from its controller's journal (one process throughout): - 17:32 the router (the `operator-channel` holder) was unassigned, so that a merge gate of the catalogue had - nothing to send there; 17:52:54 a walk of another module pushed the control node and composed the bus's user + nothing to send there; 17:52:54 a rollout of another module pushed the control node and composed the bus's user list **without** the controller's grant to ask: the grant is composed from a holder of the seat that is *assigned*, and none was. - 17:54:05 the controller started asking (the router was assigned again by then, so its check of the record diff --git a/04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/01-diagnosis.md b/04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/01-diagnosis.md index 3a1c189e..b9a67668 100644 --- a/04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/01-diagnosis.md +++ b/04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/01-diagnosis.md @@ -73,9 +73,9 @@ one. The tool runner has the same shape at a smaller scale: the bus account and controller's user list and in every node's assignment. Hence: **one release in which both names are accepted, on every side, before any release in which the old -one is dropped**, each release walked one node first — the laptop, where a halted node is the operator at a +one is dropped**, each release rolled out one node first — the laptop, where a halted node is the operator at a keyboard — and then the rest. I think no stage should move before the delivery fixes of issues 351 and 352 -are live, because a core walk is exactly what those broke; that is a guess from their symptoms, not a +are live, because a core rollout is exactly what those broke; that is a guess from their symptoms, not a measurement. ### The staged plan @@ -90,7 +90,7 @@ measurement. tool runner's as another (runtime module, bus account, kind). The node-engine's launcher and upgrade package read the old and the new library and state directories, and the node-engine moves its state once, the first time it runs under the new name. Both unit names run the same launcher. The conditions - and the `node` verb say node-engine. The merge gate composes every node green; walked one node first. + and the `node` verb say node-engine. The merge gate composes every node green; rolled out one node first. 3. **The rename proper.** The new module names registered, assigned beside the old on the first node, the old unassigned there, the acceptance sentence checked on that node, then the rest. The repository renamed on the forge, with its build source and its Go module path; the packaging renamed. The tool diff --git a/04-ISSUES/355-a-held-delivery-of-a-module-that-runs-on-one-machine-only-is-never-covered/00-report.md b/04-ISSUES/355-a-held-delivery-of-a-module-that-runs-on-one-machine-only-is-never-covered/00-report.md index d35aeef5..c5e4ff28 100644 --- a/04-ISSUES/355-a-held-delivery-of-a-module-that-runs-on-one-machine-only-is-never-covered/00-report.md +++ b/04-ISSUES/355-a-held-delivery-of-a-module-that-runs-on-one-machine-only-is-never-covered/00-report.md @@ -9,7 +9,7 @@ opened: 2026-10-09 ADR 0270 (2026-10-09) ends a held delivery by itself when a later merge that contains it was delivered to every machine running its modules — the delivery is *covered*. Its consequences say: "A module that runs on one -machine only is never covered. A walk records 'the rest running it' only after its first machines passed +machine only is never covered. A rollout records 'the rest running it' only after its first machines passed and the rest were sent. A module on one machine has no rest, so its steps never say it reached every machine. mesh-delivery itself, on the control node alone, is such a module: a held merge of it stays held for a person, as before this record. Reading 'every machine running it' from the first machines when they @@ -29,8 +29,8 @@ done (issue 354). ## Open -- Whether a walk whose first machines are all the machines should record that it reached every machine +- Whether a rollout whose first machines are all the machines should record that it reached every machine (the record's own suggestion), or whether *covered* should read the machines running a module from the - placement rather than from the walk's steps. + placement rather than from the rollout's steps. - The owner is mesh-delivery, in the controller's repository, as the record says; `located-in:` is filled - when the diagnosis confirms where the walk's record is written. + when the diagnosis confirms where the rollout's record is written. diff --git a/04-ISSUES/358-a-mesh-directory-outside-a-home-is-changed-by-path-through-any-link/00-report.md b/04-ISSUES/358-a-mesh-directory-outside-a-home-is-changed-by-path-through-any-link/00-report.md index ec28facb..5faec83d 100644 --- a/04-ISSUES/358-a-mesh-directory-outside-a-home-is-changed-by-path-through-any-link/00-report.md +++ b/04-ISSUES/358-a-mesh-directory-outside-a-home-is-changed-by-path-through-any-link/00-report.md @@ -13,7 +13,7 @@ The node-engine sets a directory's mode and owner by path when the directory is an account's home: `applyDirectory` calls `chmodDirPath` and `own`, which call `os.Chmod` and `os.Chown` on the path. Both follow a link anywhere in the path. The directory applier refuses a link at the directory's own name, and issue 339's placement guard refuses the machine's own trees, but nothing refuses a link in a parent at the -moment of use. Below a home the engine already walks every component without following a link (ADR 0266). +moment of use. Below a home the engine already traverses every component without following a link (ADR 0266). Issue 356 closed this for a directory handed to the mesh by the operator: it is changed only through one descriptor opened from `/` with no link followed, on every apply. The same treatment for every other mesh diff --git a/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md index f2d329f8..123b8a6a 100644 --- a/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md +++ b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md @@ -19,8 +19,8 @@ the controller's `root-free` verb for that machine answered: > again later The control-node holds its root and home filesystems on a software RAID of spinning disks, with millions of -files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) walks every local -filesystem mounted without `nosuid`; that walk, cold, takes far longer than fifteen minutes there. Each attempt +files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) traverses every local +filesystem mounted without `nosuid`; that traversal, cold, takes far longer than fifteen minutes there. Each attempt was killed at its bound, judged nothing, and was tried again after a back-off, which ends the same way. ## Why it is a design fault, not one slow disk @@ -29,7 +29,7 @@ was killed at its bound, judged nothing, and was tried again after a back-off, w controller could tell "still running" from "stuck" (`rootsearch.Bound`, read by both). It turned a slow machine into one that is never judged, failing closed for good rather than for a while. - **Nothing was kept across a restart.** The search's result lived in the engine's memory only, so every - engine restart (every update of the engine) started from nothing and judged nothing until a whole walk + engine restart (every update of the engine) started from nothing and judged nothing until a whole traversal finished again. - **The controller's quiet was the same value.** The self-check stayed quiet while a search was "within its bound" — fifteen minutes — so a search that runs for hours would raise the urgent condition on every @@ -51,7 +51,7 @@ was killed at its bound, judged nothing, and was tried again after a back-off, w **Why a day-old search is sound.** Only root can make a setuid- or setgid-root file: the agent account can neither set the bit on a file root owns nor give a file of its own to root. So a complete search stays true until root acts, and root's acts through the mesh (applies) start a new search. A file that appears behind a running -walk's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age +traversal's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age bounds. The trail and the fix are in [`01-diagnosis.md`](01-diagnosis.md). diff --git a/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md index c55a296f..7636fde6 100644 --- a/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md +++ b/04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md @@ -15,25 +15,25 @@ health — which this issue does not change). - **Nothing else** used the bound: `SetuidCache` had one caller (`moreWays`), and the engine made one cache. - **Ruled out:** a stuck `find` (each attempt ended at exactly its bound) and a filesystem that should not be - searched (the walk covered only the local root, home and boot filesystems, as ADR 0266 asks). + searched (the traversal covered only the local root, home and boot filesystems, as ADR 0266 asks). ## The fix (`mesh-host` pull request 75, `mesh-controller` pull request 190) -**The walk is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as +**The traversal is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as `find -xdev` did, with the same prunes and the same match (a regular file, setuid and owned by root, or setgid and of root's group), never following a link. It visits each directory's entries in lexical order, so its -progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is walked before -`/a-c` although `-` sorts before `/`). Every minute the progress (when the walk started, its filesystems, those +progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is traversed before +`/a-c` although `-` sorts before `/`). Every minute the progress (when the traversal started, its filesystems, those done, the cursor, the matches so far) is written beside the engine's state as a new file renamed into place, -readable by root alone. After a restart the walk carries on from the cursor: what is before it is skipped -without being read, the directories leading back to it are read again. A walk older than the stated age, or one +readable by root alone. After a restart the traversal carries on from the cursor: what is before it is skipped +without being read, the directories leading back to it are read again. A traversal older than the stated age, or one an apply asked to redo, is thrown away and begun again. -**Low priority, one at a time, no bound.** The walk runs on a goroutine locked to an operating-system thread of +**Low priority, one at a time, no bound.** The traversal runs on a goroutine locked to an operating-system thread of its own, set to nice 19 and the idle I/O class; on Linux both are a thread's, so the engine's other work keeps its priority, and the thread ends with the goroutine. The package manager is run the same way. A priority that -cannot be lowered fails the search rather than walk at full priority. There is no deadline: the search ends when -it has walked everything, or with an error (a directory it could not read), which is incomplete. +cannot be lowered fails the search rather than traverse at full priority. There is no deadline: the search ends when +it has traversed everything, or with an error (a directory it could not read), which is incomplete. **A verdict only from a complete, fresh search.** The last complete search is kept beside the engine's state with its start and finish, and judges while it is younger than `rootsearch.FreshFor`, counted from its start. @@ -44,7 +44,7 @@ link, owned by the engine's account and writable by it alone, in a directory tha **When a search runs.** Once a day (`rootsearch.Every`, from the last complete one's start); after an apply whose outcomes moved a user, a package, an action, or a file of the user and group databases or of sudo's, doas's or polkit's rules (`apply.MovesRoot`), at most once an hour from the last search's end, the last complete -search judging meanwhile; and at once when a filesystem appears that the last search did not walk, which that +search judging meanwhile; and at once when a filesystem appears that the last search did not traverse, which that search does not judge. **The values, one place** (`rootsearch`, read by both repositories): @@ -57,29 +57,29 @@ search does not judge. `FreshFor` is `Every` plus `Quiet` so that a daily search which finishes within the controller's quiet never leaves an hour unjudged; the controller's test holds that relation. The operator's direction named 26 hours as an -example; 36 is chosen because a full walk at idle priority on the control-node is expected to take hours, and +example; 36 is chosen because a full traversal at idle priority on the control-node is expected to take hours, and with 26 hours a daily search longer than two hours would leave the account unjudged — and phone linking refused — every day until it finished. ## 2026-10-10 — the independent review, and what it changed An independent review of both pull requests asked four questions: can a stale or incomplete search yield "free"; -can the agent account steer the kept files or the walk; does the walk hurt the machine; do the controller and the +can the agent account steer the kept files or the traversal; does the traversal hurt the machine; do the controller and the engine agree. What it found, and what was done: -- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed walk +- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed traversal skipped every name that sorted between the real cursor and its replacement, and could finish "free" without - having read them. Any account can make such a name. The walk's paths are now kept as their bytes. -- **An ask to search again was lost to a restart**, and the restarted engine carried on the walk that started - before root acted. The ask now forgets the walk under way and stops it keeping progress. -- **A fresh search judged a filesystem it never walked.** A complete search now judges only while it walked every - filesystem a search would walk now; a filesystem mounted since makes the verdict "not judged yet" until walked. -- **A mount point inside a walked filesystem was read before it was known to be another filesystem.** A FUSE - mount root cannot read failed the walk, and a network mount whose server is away could hang it. Every mount + having read them. Any account can make such a name. The traversal's paths are now kept as their bytes. +- **An ask to search again was lost to a restart**, and the restarted engine carried on the traversal that started + before root acted. The ask now forgets the traversal under way and stops it keeping progress. +- **A fresh search judged a filesystem it never traversed.** A complete search now judges only while it traversed every + filesystem a search would traverse now; a filesystem mounted since makes the verdict "not judged yet" until traversed. +- **A mount point inside a traversed filesystem was read before it was known to be another filesystem.** A FUSE + mount root cannot read failed the traversal, and a network mount whose server is away could hang it. Every mount point the kernel lists is now left unread. Mount points are read with all the kernel's escapes. -- **A file found and gone before the package manager was asked failed every resumed search.** The walk looks at +- **A file found and gone before the package manager was asked failed every resumed search.** The traversal looks at what it found again when it ends, and drops what went or is no longer setuid-root. -- **Outcomes that moved nothing started a search**, and a search asked for could follow a walk of hours at once. +- **Outcomes that moved nothing started a search**, and a search asked for could follow a traversal of hours at once. Only created, updated, corrected and removed outcomes count now, and an ask waits an hour from the last search's end. - **Not changed, because the mesh cannot do it:** the review asked that a file or archive declared setuid start a @@ -87,53 +87,53 @@ engine agree. What it found, and what was done: ## Open, after the fix -- **How long the first walk on the control-node takes is not measured.** Until the first complete search, the +- **How long the first traversal on the control-node takes is not measured.** Until the first complete search, the verdict there is `not judged yet`, and linking stays refused. The engine says each search's end and how long - it ran in its journal; that line is the measurement. If a walk takes longer than `Quiet`, the self-check + it ran in its journal; that line is the measurement. If a traversal takes longer than `Quiet`, the self-check raises the urgent condition, which is right: it is then not known to finish. - **The idle I/O class is honoured only by a disk scheduler that has classes** (BFQ). Under another scheduler - the walk still runs at nice 19 but competes for the disk as any reader does. -- **An ask to search again is not kept across a restart.** The walk under way is forgotten when it is asked, - so a restarted engine does not carry an older walk on; the ask itself is lost, and the daily search covers it. + the traversal still runs at nice 19 but competes for the disk as any reader does. +- **An ask to search again is not kept across a restart.** The traversal under way is forgotten when it is asked, + so a restarted engine does not carry an older traversal on; the ask itself is lost, and the daily search covers it. The last complete search keeps judging meanwhile, on the reasoning in the report. - **The agent account can make the search fail, never pass.** A tree deeper than the longest path the kernel - takes, or a directory unreadable to root through FUSE, ends every walk incomplete; a directory of millions of + takes, or a directory unreadable to root through FUSE, ends every traversal incomplete; a directory of millions of entries makes the engine hold them all at once. Each is "not judged" — the urgent condition, and linking refused — never a pass. Reading by directory handle, as `find` did, would close the first. - **A filesystem mounted only on use** (an automounted boot partition) makes the verdict "not judged yet" each - time it appears unwalked, until a walk covers it. -- **Part of the walk's cost escapes its priority**: Go's garbage collector runs on threads of normal priority, + time it appears untraversed, until a traversal covers it. +- **Part of the traversal's cost escapes its priority**: Go's garbage collector runs on threads of normal priority, and a software RAID's own threads do not carry the idle class to the disks. - **ADR 0266 §3 item 10** still says "at most hourly" and "at most 15 minutes". On resolution it gains a progressive insight naming this issue and the values above. - **A later option, not built now:** a fanotify filesystem mark with `FAN_ATTRIB` (Linux 5.1 and later) on - every searched filesystem would report each change of mode or owner as it happens, turning the daily walk into + every searched filesystem would report each change of mode or owner as it happens, turning the daily traversal into a true incremental search. - **Resolution** of this core issue needs a replay in mesh-lab's register, or `replay-none:` with the reason. ## 2026-10-10 — the search shows its progress, and the operator can ask for a fresh one -- **The progress was kept and seen by nobody.** At 01:40 the operator asked how far the first walk on the - control-node had got. The walk kept its progress every minute beside the engine's state, but `root-free` said +- **The progress was kept and seen by nobody.** At 01:40 the operator asked how far the first traversal on the + control-node had got. The traversal kept its progress every minute beside the engine's state, but `root-free` said only "not judged yet (search running), started 01:27", and no tool read further. The node-engine serves no tools of its own, so the progress now travels where the verdict already does: the agent account's verdict carries it as fields, and its reason gains one line that `root-free` repeats. The line holds filesystems done - of the total, the mount point being walked, the entries visited, the setuid- or setgid-root files found so far, + of the total, the mount point being traversed, the entries visited, the setuid- or setgid-root files found so far, when the progress was last kept, and how long each finished filesystem took. Counts, times and mount points only: the cursor and every file found are often under an account's home, and stay in the engine's state and journal. The line moves at each checkpoint (once a minute) and is never part of what makes a verdict "changed", so the journal does not repeat it every minute; the statement is said every minute anyway while the account is not healthy. An entry is counted once across a resume, as it is kept with the cursor it - matches. The time a filesystem took counts only the time the engine walked it, not the time it was stopped. + matches. The time a filesystem took counts only the time the engine traversed it, not the time it was stopped. - **A change by hand was not searched for until the next day.** The first complete search on the control-node ended at about 01:48 and found two setuid-root programs no package owns, copies in the operator's home. When the operator removes them by hand, no apply says so, and nothing searches again until the daily search. The operator now asks for one at the controller's terminal: `nox node setuid-search ` on the control-node. The controller signs the ask with the mesh's key, as it signs a hand-over, under a signing context of its own. - The engine throws away the last complete search and any walk under way or kept, and the next look starts a + The engine throws away the last complete search and any traversal under way or kept, and the next look starts a full search. Until it completes, the verdict is "not judged yet", with its progress. - **Why at the terminal alone, not a verb.** A search can never make a machine root-free wrongly: it judges only - once complete. But every ask throws a walk away. Asked again and again, it would keep the machine unjudged + once complete. But every ask throws a traversal away. Asked again and again, it would keep the machine unjudged for good and its disks busy, and whoever may call a verb includes agents. The operator needs the line only after a change made by hand. - **A find counts only where the agent account can execute it.** The operator asked how the agent account could diff --git a/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md b/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md index 1607f6b6..eb56905c 100644 --- a/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md +++ b/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md @@ -11,17 +11,17 @@ Shortly after midnight on 2026-10-10 (local time) two pull requests of the catal their checks. `mesh-delivery.deliveries` then showed every delivery of the dunst pull request `superseded`, and the -claude-code delivery `held`, waiting for "a person: merged, and the controller opened no walk for it within +claude-code delivery `held`, waiting for "a person: merged, and the controller opened no rollout for it within 10m0s — nothing it holds follows that branch, or the merge was not heard: a person decides". The -controller's `plans` said of the walk for the catalogue's head, which carried both merges: "published; its -walk waits for mesh-delivery's word — `plans go … --why …` starts it by hand". +controller's `plans` said of the rollout for the catalogue's head, which carried both merges: "published; its +rollout waits for mesh-delivery's word — `plans go … --why …` starts it by hand". -Both reasons were wrong. A walk had been opened, and it contained the held merge. Nothing gave it the word. +Both reasons were wrong. A rollout had been opened, and it contained the held merge. Nothing gave it the word. The operator's change sat undelivered for 58 minutes, until the operator's hand-act `mesh-controller.plans -go` started the walk. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours -before `mesh-delivery.stalled` lists it, and the walk's own wait raised only a warning, at 00:51, which said +go` started the rollout. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours +before `mesh-delivery.stalled` lists it, and the rollout's own wait raised only a warning, at 00:51, which said there was nothing to do and pointed at the ended delivery (the diagnosis's second entry; this sentence said -before that the walk's wait raised nothing). +before that the rollout's wait raised nothing). ## Why it is an issue @@ -33,41 +33,41 @@ cause. ## Where it lives The diagnosis ([01-diagnosis.md](01-diagnosis.md)) traces it to two parts that each behave as written and -disagree on what a walk's commit means: +disagree on what a rollout's commit means: - The controller plans a merge it hears after a later merge of the same branch at the later merge's commit, - and supersedes the earlier walk (the fix for + and supersedes the earlier rollout (the fix for [issue 349](../349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md)). - So the claude-code merge got a walk at the dunst merge's commit. -- mesh-delivery finds the delivery a walk walks by its walk id or by the commit it was merged as. The new - walk matched the dunst delivery, which its first walk's supersession had already ended, so no live - delivery held the walk and nothing gave it its turn. The claude-code delivery, merged as a different - commit, never got a walk and was held after ten minutes. + So the claude-code merge got a rollout at the dunst merge's commit. +- mesh-delivery finds the delivery a rollout rollouts by its rollout id or by the commit it was merged as. The new + rollout matched the dunst delivery, which its first rollout's supersession had already ended, so no live + delivery held the rollout and nothing gave it its turn. The claude-code delivery, merged as a different + commit, never got a rollout and was held after ten minutes. ## Open -- How mesh-delivery learns which merges a walk carries: the controller's walk record naming every merge it +- How mesh-delivery learns which merges a rollout carries: the controller's rollout record naming every merge it answers (its commit and the commits planned into it), or mesh-delivery reading containment itself, as it already does to decide *covered*. -- A superseded walk whose successor matches only final deliveries: whether the successor should be adopted - as a delivery of its own, so that something always holds a waiting walk. -- A walk waiting for mesh-delivery's word past a bound, and a delivery held because no walk was opened, +- A superseded rollout whose successor matches only final deliveries: whether the successor should be adopted + as a delivery of its own, so that something always holds a waiting rollout. +- A rollout waiting for mesh-delivery's word past a bound, and a delivery held because no rollout was opened, raise a condition to the operator instead of waiting silently (failure must be loud). ## Decided The operator decided the model that removes this shape, not only its bookkeeping: -ADR 0276. Merges are collected in a rolling merge window and each batch is walked once, with at most one -commit per repository, one walk at a time. The bookkeeping that tied a walk to an ended delivery is repaired as +ADR 0276. Merges are collected in a rolling merge window and each batch is rolled out once, with at most one +commit per repository, one rollout at a time. The bookkeeping that tied a rollout to an ended delivery is repaired as well, and every wait of this shape has a bound and a condition. It is designed in to-be 47, *The merge window and the batch*. ## Resolved -2026-10-10. mesh-delivery's side is mesh-catalog#171 (merged): a walk is tied to every live delivery whose -merge it answers, never to one that has ended, and a waiting walk no live delivery holds is listed by `stalled` -at once. The controller's side is mesh-controller#197: the merge window and the batch, one walk per batch with -one commit per repository, the walk naming every merge it answers, and the bounds of decision 9. The replay +2026-10-10. mesh-delivery's side is mesh-catalog#171 (merged): a rollout is tied to every live delivery whose +merge it answers, never to one that has ended, and a waiting rollout no live delivery holds is listed by `stalled` +at once. The controller's side is mesh-controller#197: the merge window and the batch, one rollout per batch with +one commit per repository, the rollout naming every merge it answers, and the bounds of decision 9. The replay R362 is registered in mesh-lab's replays register (mesh-lab#75), proved to fail on the commit before the mesh-delivery fix and pass on it. The build raised one rule the record had not settled, decided during the build and written into ADR 0276: a merge on the controller's own path never shares a batch with a catalogue diff --git a/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/01-diagnosis.md b/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/01-diagnosis.md index 516e6ef1..d75d6b9d 100644 --- a/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/01-diagnosis.md +++ b/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/01-diagnosis.md @@ -9,38 +9,38 @@ request as 48bda475 at 00:21:09. 48bda475 is the branch's head and contains 553b **What the controller did** (its journal, 00:21:14 to 00:21:15): -1. It heard the dunst merge first and opened a walk at 48bda475 for dunst, waiting for mesh-delivery's word. +1. It heard the dunst merge first and opened a rollout at 48bda475 for dunst, waiting for mesh-delivery's word. 2. It then heard the claude-code merge, and printed "553b7191 was merged before 48bda475 (…, building): what it moved is built from 48bda475, which contains it". This is the rule that answered [issue 349](../349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md): `SourceMoved` replaces the merge's commit and merge time with the newest planned merge's, when that merge is later on the branch (`laterOnTheBranch`). -3. The second walk, also at 48bda475 and with the same merge time, superseded the first by when each was made +3. The second rollout, also at 48bda475 and with the same merge time, superseded the first by when each was made (`supersededBy`, `earlierOnTheBranch`), and carried claude-code and dunst. It waited for mesh-delivery's word. -So every walk the controller opened was at 48bda475. None was at 553b7191. +So every rollout the controller opened was at 48bda475. None was at 553b7191. **What mesh-delivery did** (`mesh-delivery.show` for both deliveries): -1. The dunst delivery (merged as 48bda475) took the first walk, matched by its commit, and became +1. The dunst delivery (merged as 48bda475) took the first rollout, matched by its commit, and became `published` at 00:21:14. -2. The first walk's record came back superseded, and the table's `superseded` transition ended the dunst +2. The first rollout's record came back superseded, and the table's `superseded` transition ended the dunst delivery at 00:21:15 ("a newer delivery to the same trunk took over its walk"). -3. The second walk was matched to a delivery by `deliveryWalking`, which looks first for a delivery holding - that walk id and then for any delivery whose `merged_as` is the walk's commit, final or not. That is the - dunst delivery again, already final. The walk was linked there, so it was never kept as unmatched and +3. The second rollout was matched to a delivery by `deliveryWalking`, which looks first for a delivery holding + that rollout id and then for any delivery whose `merged_as` is the rollout's commit, final or not. That is the + dunst delivery again, already final. The rollout was linked there, so it was never kept as unmatched and never adopted as a delivery of its own. A turn (`turn`) is given only to a `published` or `delivering` - delivery, so nothing ever gave the second walk its word. -4. The claude-code delivery, merged as 553b7191, matched no walk. Ten minutes after its merge + delivery, so nothing ever gave the second rollout its word. +4. The claude-code delivery, merged as 553b7191, matched no rollout. Ten minutes after its merge (`unwalkedPast`) it was held for a person at 00:30:58, with a reason that names two causes, neither of which happened. -After the operator's `plans go`, the second walk's machine steps (claude-code and dunst, each on a node) were recorded on the superseded dunst delivery, which confirms that mesh-delivery reads that walk as +After the operator's `plans go`, the second rollout's machine steps (claude-code and dunst, each on a node) were recorded on the superseded dunst delivery, which confirms that mesh-delivery reads that rollout as the dunst delivery's. **Why nothing was raised.** A delivery held for a person is bounded at 24 hours before -`mesh-delivery.stalled` lists it (`Bounds` in its table). The walk waiting for mesh-delivery's word is +`mesh-delivery.stalled` lists it (`Bounds` in its table). The rollout waiting for mesh-delivery's word is counted against the controller's plan wait bound of 30 minutes for `status`, but no condition reached the operator in the 58 minutes it waited. I did not trace why the controller's side stayed quiet; that belongs to the fix. @@ -48,28 +48,28 @@ to the fix. ## Ruled out - **The merge was not heard.** The controller's journal shows it was heard and planned within 23 seconds. -- **Nothing follows that branch.** claude-code follows the catalogue's trunk; the second walk built it. +- **Nothing follows that branch.** claude-code follows the catalogue's trunk; the second rollout built it. - **A failed check.** Both heads passed the merge gate and the repository's check. ## Located -Two parts, each correct by its own rule, disagree on what a walk's commit means. The controller (since issue -349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a walk's commit is the -merge commit of exactly one delivery. `located-in` names both. The fix is to make the walk say which merges -it answers, or to make mesh-delivery match by containment, and to let no waiting walk be held by nothing. +Two parts, each correct by its own rule, disagree on what a rollout's commit means. The controller (since issue +349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a rollout's commit is the +merge commit of exactly one delivery. `located-in` names both. The fix is to make the rollout say which merges +it answers, or to make mesh-delivery match by containment, and to let no waiting rollout be held by nothing. ## 2026-10-10 — what the controller's side said The first entry says no condition reached the operator in the 58 minutes. The controller's condition history -says more precisely what happened. S16 (a walk waiting for its delivery's word) raised -`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the walk opened, as its bound says. +says more precisely what happened. S16 (a rollout waiting for its delivery's word) raised +`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the rollout opened, as its bound says. It was a warning, its explanation was "Nothing for you to do … It becomes a question for you if it still waits after 4 hours", and its summary sent the reader to "`mesh-delivery.show` for the delivery that landed as 48bda475", which is the dunst delivery the supersession had already ended. It cleared at 01:20:32, when the -operator's `plans go` started the walk. +operator's `plans go` started the rollout. So the controller's side was not silent; it was mild and pointed at the wrong delivery. S16 cannot tell a -group member waiting its turn from a walk nobody holds; only mesh-delivery knows that. That is why ADR 0276 -decision 9 has mesh-delivery list a waiting walk no live delivery holds through `stalled` at once, and has -S16 name the merges a walk answers. +group member waiting its turn from a rollout nobody holds; only mesh-delivery knows that. That is why ADR 0276 +decision 9 has mesh-delivery list a waiting rollout no live delivery holds through `stalled` at once, and has +S16 name the merges a rollout answers. diff --git a/README.md b/README.md index f65c65ab..2279074b 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ and why. Today that is almost entirely **Novox Mesh**, the foundation everything | [`04-ISSUES`](04-ISSUES/) | The front door for "something is wrong" at the level of design or governance. | **The numbering is the flow.** Research produces a decision; the decision authorises a design. -That is why decisions are `02` and design is `03` — a reader following the numbers walks the +That is why decisions are `02` and design is `03` — a reader following the numbers rollouts the process in the order it happens. The research efforts and decision records themselves live in the knowledge base, the seat `knowledge-base`, read and written through the mesh MCP server from any node (ADR 0290).