novox/hq
8
0
Fork 0
Code Issues Pull Requests 2 Releases Wiki Activity

Say rollout, not walk, for the sending of a batch across nodes (issue 646)

Browse Source
This commit is contained in:
jschoubben
2026-10-15 19:57:20 +02:00
parent 99a9da0950
commit 437523554e
82 changed files with 561 additions and 561 deletions
Download Patch File Download Diff File Expand all files Collapse all files
00-META/glossary.md
+16 -16
View File
@@ -178,7 +178,7 @@ every other domain has something to run on. **Recorded by** the controller and t
(ADR 0229).
- **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its
records (nodes, assignments, walks, builds, settings), after each change commits in its store and only
records (nodes, assignments, rollouts, builds, settings), after each change commits in its store and only
when the value differs, with one event per write. The store stays the record; the self-check compares
the two and raises `mirror-drifted` on any difference
(ADR 0289). Shortened to *mirror* where the
@@ -306,7 +306,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences`
**Purpose.** To take one commit from its pull request to every node that should run it, and to put it
back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the
planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
planner, the rollout). **Downstream of** Module, Core, Health and repair and Data.
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323.
**Uses:** module, node, declaration and send (Placement), health (Health and repair), person.
@@ -348,7 +348,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
- **delivery** — one commit in one repository on its way to the nodes, from its pull request's head
being announced to one of its final states: one pull request, one status, one note
(ADR 0239).
From its merge on it is delivered by the walk of the batch that answers it, and ends with that walk
From its merge on it is delivered by the rollout of the batch that answers it, and ends with that rollout
(ADR 0276).
Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`,
`held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`;
@@ -357,9 +357,9 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
mechanism, which the as-is designs describe; it is never a word for a delivery.
*Not:* ~~deployment~~ (hq)
- **covered** — the end of a delivery that a later merge containing it delivered: every module it moves was
sent to every machine running it by the walk of a later merge into the same trunk, delivered on its own
sent to every machine running it by the rollout of a later merge into the same trunk, delivered on its own
passing check, whose merge commit the forge says contains this one. Not *superseded*, which is a delivery
whose walk a newer delivery took over before it ended; nor *delivered*, which its own walk or an empty plan
whose rollout a newer delivery took over before it ended; nor *delivered*, which its own rollout or an empty plan
ends in (ADR 0270).
- **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's
repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One
@@ -372,18 +372,18 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
- **merge window** — the rolling wait in which the controller collects merges before it plans them: each
merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest
10 minutes after its first merge (ADR 0276).
- **batch** — the merges one merge window collected, planned once as one walk. For a repository merged more
than once in it, the walk takes the latest commit, and the earlier merges are answered by it. Not *group*,
- **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more
than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*,
which is a delivery group.
- **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat
of the same name. It records and decides; the controller sends, judges and rolls back when it is asked.
- **walk** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
- **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236,
ADR 0276). One walk is open at a time; it names every merge it answers. A primitive the delivering stage
asks for, not an object anyone manages. A **tier** is one step of a walk: a module, then the modules
built against it. The controller's verb `plans` lists the walks. Walking is not a noun of its own:
a module *rolls out* by its `roll` policy.
*Not:* ~~rollout~~ (hq)
ADR 0276). One rollout is open at a time; it names every merge it answers. A primitive the delivering stage
asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules
built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll`
policy.
*Not:* ~~walk~~ (hq)
- **first-node gate** — the judgement on a delivery's first node: healthy three times, no witness put it
back, no condition raised since the send (ADR 0236 §2). Not *the gate* bare (see [Homonyms](#homonyms)).
- **core module** and **leaf module** — a core module is one on the controller's own path (the controller,
@@ -394,7 +394,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
the core modules add mesh-delivery and the resolver.
- **delivery time** — from a delivery's merge to every node of its deploy plan that is heard from running its
modules' builds and reporting them healthy. Its **delivery budget** is what it is held to: three minutes for
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the walk's time, which ends at the send. A
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the rollout's time, which ends at the send. A
measure inside the fix budget, not the operator's target (ADR 0313).
- **fix time** — from an issue's first move to `fixing` to the moment the last of its fixes is delivered: the
operator's measure of a fix, the agent's work, the review and the checks included, the delivery time its last
@@ -672,7 +672,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|---|---|---|---|
| plan | what the controller would send one node | that node's **declaration** (verb `plan`) | Placement |
| | what a delivery would do to the mesh | **delivery plan** | Change and delivery |
| | the sending of one batch of commits across nodes | **walk** (verb `plans`) | Change and delivery |
| | the sending of one batch of commits across nodes | **rollout** (verb `plans`) | Change and delivery |
| | a step a person starts, like the bus's | **planned step** | Core |
| membership | the subjects a bus account is issued | **membership** | Provisioning |
| | an account in a group of the machine | **account group** | Placement |
@@ -686,7 +686,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
| | the pull request status | **merge gate** | Change and delivery |
| | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement |
| tier | a level of the mesh | **layer** | Core |
| | a step of a walk | **tier** | Change and delivery |
| | a step of a rollout | **tier** | Change and delivery |
| | how much proof an ask needs | **assurance level** | Identity and access |
| ask | a request for the operator's input | **ask** | Operator and conversation |
| | an entry in the build queue | **build request** | Change and delivery |
03-DESIGN/01-to-be/00-work-breakdown.md
+1 -1
View File
@@ -340,7 +340,7 @@ once, and maintain for a year. The modules are the input; a person reads what on
writes what it declares tomorrow.
**It also changes what "safe" means for the system being retired.** A fix to it has to be safe on
its own, because there is no careful walk to sequence it into: the thing is being switched off
its own, because there is no careful rollout to sequence it into: the thing is being switched off
by hand, not managed into retirement. A change needing three steps in the right order is a change
that will be half-applied.
03-DESIGN/01-to-be/09-the-node-lifecycle.md
+3 -3
View File
@@ -54,7 +54,7 @@ rather than two kinds of thing
node** — it is a machine with a program on it that has not been told which mesh it belongs to.
There is no state for *the first node*. That is the point of
ADR 0004: the first node walks the
ADR 0004: the first node rollouts the
same path, in an unusual order.
---
@@ -239,7 +239,7 @@ carries, to the machine it is already on.
**After step 3 the first node is not special in any way**, which is the property `adopt.sh` and
the bootstrap script never had. Its specialness lasted two commands.
**And enrolment is exercised on node one.** The path every other node depends on is walked
**And enrolment is tried on node one.** The path every other node depends on is exercised
immediately, against a controller on the same machine, rather than being written and first
used months later on node two.
@@ -524,7 +524,7 @@ remains locally authoritative for *operating*; the copy exists only for this.
The node-engine is delivered like anything else
(ADR 0010), and this is worth
walking through because tier 0 looks like it should be special and is not.
going through step by step because tier 0 looks like it should be special and is not.
```
push to mesh-host
03-DESIGN/01-to-be/10-delivery.md
+5 -5
View File
@@ -277,15 +277,15 @@ A transition the table does not hold is refused. Two or more deliveries that sha
together as one future state of the mesh.
**This answers the first open question above.** A fit artifact does not declare itself. Off the trunk it is
only checked. On the trunk it is published, and its walk across the machines starts when its delivery says
only checked. On the trunk it is published, and its rollout across the machines starts when its delivery says
so: by itself for the core, which cannot wait for anything to be repaired, and by `mesh-delivery` for
everything else while that module is held. A person can always deliver by hand.
The controller keeps the comparison, the planner, the gate, sending and rollback. `mesh-delivery` keeps the
record and the order, and asks.
**Merges are walked in batches** (2026-10-10, by ADR 0276). The controller collects
**Merges are rolled out in batches** (2026-10-10, by ADR 0276). The controller collects
merges in a rolling merge window, 90 seconds by default and at most 10 minutes, and plans each batch once, as
one walk with at most one commit per repository: the latest of its branch, which contains the earlier ones.
One walk is open at a time, and a walk that started is never superseded. `plans` shows a batch while it
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's walk.
one rollout with at most one commit per repository: the latest of its branch, which contains the earlier ones.
One rollout is open at a time, and a rollout that started is never superseded. `plans` shows a batch while it
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's rollout.
03-DESIGN/01-to-be/20-writing-a-module.md
+1 -1
View File
@@ -215,7 +215,7 @@ and recreate a container naming a step after the step ran.
*Written 2026-10-02, from ADR 0189
and [issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).*
Some work cannot be done underneath a running service: an artifact store's collector walks the
Some work cannot be done underneath a running service: an artifact store's collector rollouts the
storage and requires every writer stopped. A `run-once` step runs *beside* containers and a
scheduled one is the same container again, so until this a module had no way to say it — and the
mesh inherited a store that has never collected anything, because the predecessor said it with a
03-DESIGN/01-to-be/24-the-secrets-vault.md
+1 -1
View File
@@ -163,7 +163,7 @@ again restarts nothing. The provider asks when it starts, when the software's va
consumer that binds in between holds a value the mesh made, which the software refuses, for at most the hour until the next ask. A change is recorded in the
hand-act log as a decision, not a repair, and `rotate` of such a provision is refused. *How it is checked:*
the tests ADR 0305 §13 names, in both repositories; the live row is `hand-acts` showing `secret issued` for
the media server's provision once both walks are done.
the media server's provision once both rollouts are done.
## Beyond generate and hold
03-DESIGN/01-to-be/25-the-bus-on-nats.md
+1 -1
View File
@@ -75,7 +75,7 @@ and uses, and the subjects, streams, consumers and permissions all follow from t
(ADR 0039).
**Added 2026-10-10** (ADR 0289): the controller's own records join
the bus as **record mirrors** — nodes, assignments, walks, builds and settings, each a bucket written from
the bus as **record mirrors** — nodes, assignments, rollouts, builds and settings, each a bucket written from
the controller's store only when the record changed, with one event per write, and audited against the
store on every self-check run. Designed in [to-be 54](54-the-controllers-records-on-the-bus.md). The
rule it states for every bucket, that a write is made only when what the bucket describes changed, holds
03-DESIGN/01-to-be/28-building-the-bus.md
+1 -1
View File
@@ -709,7 +709,7 @@ healthy while reacting to nothing.
Found on the way: **no build had ever recorded what it stood on.** A recipe reads its base from
a build argument, so the digest was never in the file the builder derived edges from, and every
order that says *bases first* — `build --on`, `build --behind`, the merge follow-up of
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — walked a
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — traversed a
graph with no edges. The builder now reports the bases it was handed, the controller records
them by artifact path, and the graph is read from the newest build of each module — a recorded
manifest carries no `build.on`, so the edge is derived from the build or it does not exist.
03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md
+2 -2
View File
@@ -141,7 +141,7 @@ Revision, ADR 0218. Three rules on how a plan delivers what it built.
A membership that could not be issued fails the send.
- **One machine first.** Unless a module's upgrade policy says *together*, a plan sends it to one machine,
the first by name, and to the rest only once that machine reports the new declaration applied and
current. A first machine that fails stops the module's walk there, with the reason in the plan.
current. A first machine that fails stops the module's rollout there, with the reason in the plan.
- **A newer merge takes over.** A merge's plan supersedes every older open plan for the same repository
and branch, and takes in the modules they had not yet built. A plan that waits on nothing can be closed
by hand, by its id.
@@ -182,7 +182,7 @@ policy or a plan is holding back
builds are not known is left too. The push names the machine, the module, the builds and the reason,
and says `push <node>` sends it.
- The machine a push names, a push of every machine, and `push --behind` send held builds as before.
Walking a change through the mesh under `record` is one `push <node>` per machine.
Rolling out a change through the mesh under `record` is one `push <node>` per machine.
## Why now, and why not yet
03-DESIGN/01-to-be/32-what-a-module-declares.md
+3 -3
View File
@@ -253,7 +253,7 @@ the controller creates from the catalogue, mesh-wide, existing from registration
watch before the owner runs anywhere ([design 25](25-the-bus-on-nats.md) §3). Every instance of the
owner writes; a reader reads and watches. A key may name a machine by the module's own convention;
the mesh keeps one bucket per name, not one per machine, because "every server, for every machine"
is then one list rather than a walk.
is then one list rather than a rollout.
What a module sees is what it named. Its assignment's membership lists its buckets by those names,
with whether it may write, and the runtime answers `get`, `put`, `delete`, `keys` and `watch` for
@@ -338,7 +338,7 @@ module, version, digest. An event again: a fact about the builder.
**The build cascade is that event fanning out through a graph the mesh already has.** A module
whose image is built *on* another's artifact declares that in `build.on`. So `built` reaches the
controller, which walks the declared graph and submits rebuild jobs for everything downstream. A
controller, which traverses the declared graph and submits rebuild jobs for everything downstream. A
dependency cascade is not special machinery — it is one event, one derived graph, and the same job
queue.
@@ -351,7 +351,7 @@ entrypoint that brings its state to the shape that version needs — the same vo
tools and its provisioner, and nothing about how a machine runs it. The mesh runs that entrypoint as it
runs the module's own code, to completion, in the module's own context, and a version whose preparation
did not succeed does not run: the step gates that module and nothing else on the machine
(ADR 0136), and the walk stops
(ADR 0136), and the rollout stops
at the first machine that did not take it
(ADR 0135, superseding
ADR 0133).
03-DESIGN/01-to-be/43-backups-against-mistakes.md
+2 -2
View File
@@ -53,9 +53,9 @@ bucket, runs a bad migration — and not for the day a disk dies (ADR 0214).
*Added 2026-10-06, ADR 0233.*
The mesh composes a second file for the holder: every data item every module on the machine declares,
with its class, its path, the path that covers it and its protection. Once an hour and after every
night the holder measures each item that is not a cache, by the item's `measure`: a bounded walk at
night the holder measures each item that is not a cache, by the item's `measure`: a bounded rollout at
most daily for a small item (stopped after ten minutes or two million files and said as partial), a ZFS
dataset's own counters hourly, or the top-level entries only — never an hourly walk of anything large —
dataset's own counters hourly, or the top-level entries only — never an hourly rollout of anything large —
and reads the redundant storage it is on (a ZFS pool's health, an md array's
members, a btrfs filesystem's error counters). `backed-up` says each item with that measurement and its
newest good backup. The holder judges nothing: the controller's self-check (to-be 45, D13) keeps the
03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
+10 -10
View File
@@ -101,7 +101,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps |
| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads |
| the record mirrors: nodes, assignments, walks, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
| the record mirrors: nodes, assignments, rollouts, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why; an ask is delivered again only when the controller made it itself, every other ask is only dropped (ADR 0293) |
A bus subject two components may publish on is refused when the controller composes grants, unless
@@ -176,7 +176,7 @@ process of its own reads them back from the store.
*Added 2026-10-08, ADR 0253:*
**a condition says itself to the operator in plain words, beside its summary, and says whether the operator
is needed.** The headline, needs, explanation, resolved line and actions are made where the condition is
made: by the producer when it knows more than the subject (a walk's modules, a failed unit's name, a
made: by the producer when it knows more than the subject (a rollout's modules, a failed unit's name, a
hand-act's cause), otherwise by the wording registered for the kind, otherwise from the scope, which is
reported. The explanation opens with its verdict. All of it is held to the plain rule when the controller
takes an observation: no hash, numbered id, dotted name, duration in code, clock time, flag, markup or line
@@ -267,10 +267,10 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — |
| S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | `facts-stale` | warning | — |
| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — |
| S16 | a walk waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose walk waits | 30 min; urgent after 4 h | `waiting` (names the walk and `plans go`) | warning | — |
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a walk out | warning | — |
| S18 | a batch of merges is cut into its walk (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no walk is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
| S19 | a batch waiting behind an open walk is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a walk is open | that walk's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the walk) | warning | — |
| S16 | a rollout waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose rollout waits | 30 min; urgent after 4 h | `waiting` (names the rollout and `plans go`) | warning | — |
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a rollout out | warning | — |
| S18 | a batch of merges is cut into its rollout (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no rollout is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
| S19 | a batch waiting behind an open rollout is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a rollout is open | that rollout's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the rollout) | warning | — |
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
controller translates each into a condition naming the thing in the mesh's words, as the refused reply
@@ -540,7 +540,7 @@ except an act recorded by a verb that is a person's decision by design, which is
counts, whatever cause it gives. The controller keeps one table of the verbs that write the log, and
each says whether it records a repair or a decision, and why: approving or rejecting a retirement and
deleting what was retired (ADR 0230); `bus upgrade`, because the bus is never rolled by the mesh, and
`upgrade release-backlog`, because after a failed walk the next opens only on a person's word
`upgrade release-backlog`, because after a failed rollout the next opens only on a person's word
(ADR 0236); and `secret rotate` when its cause is a leak (`leaked-in-logs`) — a person judges what was
disclosed, one leak rotates several values, and a leak that recurs is a defect of the module that
prints them, an issue against it, not a healer that rotates. A rotation for any other cause counts. A
@@ -619,7 +619,7 @@ the other machines follow. "Reported applied" is not enough.
**As built** (ADR 0236):
- **Every walk is gated, not only the core's.** A module is judged by its own health: it reported
- **Every rollout is gated, not only the core's.** A module is judged by its own health: it reported
applied, no witness put it back, no condition was raised since its send about the machine or about the
module there, and its tools are served on that machine where it has tools. Three healthy judgings at
least forty seconds apart and two minutes after the send, within ten minutes of it. A policy of
@@ -642,7 +642,7 @@ the other machines follow. "Reported applied" is not enough.
- **No build reaches a machine without a gate**: a gated send carries and judges everything waiting on
its machine; every other send — a plan's rest, a cascade, a healer's, a whole-mesh push — is refused
or leaves the machine while a build no gate has seen waits there; a rebuild with the same artifacts and
manifest is no move. What waits is moved by a **walk**, one machine at a time, the control
manifest is no move. What waits is moved by a **rollout**, one machine at a time, the control
node last, each judged; one that fails holds the next until `upgrade release-backlog --why`.
- **A wait for a person passes, and each module of a send keeps its own pass**
(ADR 0254): a module whose only fault is a new login the operator owes
@@ -938,7 +938,7 @@ and "the commit is the build at hand"), on branches, not yet merged:
| hq | its own `merge-check.sh`, running `records.py`, `index.py` and `cycle.py` |
**Not yet:** the delivery plan kept (id, inputs, result) and the release comparing its plan with it; the
state machine replacing the walks' states, with its table, its test, its conditions and healer H2;
state machine replacing the rollouts' states, with its table, its test, its conditions and healer H2;
the status's link to the kept plan; the notes under `refs/notes/mesh-plan`; check builds in a scratch
namespace (the check builds no module today, so there is none to keep apart yet); the statuses made
required — prepared as calls of the forge module's tool, applied by the operator.
03-DESIGN/01-to-be/46-the-conversation-with-the-operator.md
+19 -19
View File
@@ -327,36 +327,36 @@ Presence facts are events on the bus. The router keeps them as **current state o
machine and per intake kind, overwritten — never as a history, and never in a message's words. A
consumer that wants a timeline of the operator's day is a decision of its own.
### While a delivery walks
### While a delivery rollouts
*2026-10-12, ADR 0308; the build is tracked by issue 532.*
A delivery restarts what it moves, and those restarts raise conditions that clear once it is done; a router
that restarted used to say again what it had been told to keep quiet (issue 531). So the router holds what a
walk touches and tells the walk's result once.
rollout touches and tells the rollout's result once.
- **A walk names what it touches.** The controller's walk events say, when a walk starts and again when it
- **A rollout names what it touches.** The controller's rollout events say, when a rollout starts and again when it
ends with its outcome (delivered, failed or stopped), the modules it moves and the machines it sends them to.
What it touches is those modules on those machines, and every module a moved tool runner or node-engine
carries there.
- **The router holds what an open walk touches.** A message whose subject is a touched module or machine is
- **The router holds what an open rollout touches.** A message whose subject is a touched module or machine is
held instead of sent, kept with when it was held, and `history` says it was held. A message several open
walks touch is held until the last of them ends or its bound passes.
rollouts touch is held until the last of them ends or its bound passes.
- **Some messages are never held:** an ask (§6), a secret given or replaced, a message about anything no open
walk touches, and a walk's own failure.
- **A walk's end is told once:** one message with its outcome, the modules and machines it reached and how
rollout touches, and a rollout's own failure.
- **A rollout's end is told once:** one message with its outcome, the modules and machines it reached and how
long it took, then each held message whose condition is still open, one line each. One whose condition
cleared during the walk is only counted. A walk delivered with nothing held and nothing open is one line, or
cleared during the rollout is only counted. A rollout delivered with nothing held and nothing open is one line, or
nothing on a channel set to say only what needs the operator.
- **A hold has a bound:** twice the largest delivery budget in force among the walk's deliveries
- **A hold has a bound:** twice the largest delivery budget in force among the rollout's deliveries
([to-be 47](47-delivery-from-commit-to-delivered.md)): 20 minutes for a core module, 10 for leaf modules at
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a walk none of whose
deliveries has a budget. Past it, every held message still open is sent with a line saying the walk still
runs and since when, and the cleared ones are counted. A walk that hangs never silences the mesh.
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a rollout none of whose
deliveries has a budget. Past it, every held message still open is sent with a line saying the rollout still
runs and since when, and the cleared ones are counted. A rollout that hangs never silences the mesh.
- **A restart says nothing it has said:** a router that restarts says again only what is open and was never
answered, never a silenced message or one withheld for its content.
*How it is checked:* the controller's tests of the walk events, and the router's tests of holding, never
*How it is checked:* the controller's tests of the rollout events, and the router's tests of holding, never
holding, the end message, the bound and the restart, as ADR 0308 lists them; live, the next delivery of the
tool runner gives the operator one message at its end, and the channel's `history` shows what was held and
released.
@@ -467,13 +467,13 @@ the factors are the router's state.
### An approved settings change takes effect (ADR 0344)
*2026-10-13, issue 641.* Approving a settings proposal is the act that makes it take effect. The
serving controller sets the layer and, in the same act, opens a **settings walk**: every machine
serving controller sets the layer and, in the same act, opens a **settings rollout**: every machine
that runs the module and that the layer reaches, the control node last, each sent and its module
judged at the first-node gate before the next. A settings walk sends the layer and no build: every
judged at the first-node gate before the next. A settings rollout sends the layer and no build: every
module on the machine stays at the build it was last sent, so a build held for a person, or waiting
for a first-node gate, stays where it is. A send that would move a running module's data stops the
walk and names the `push --move` that sends it. The walk says its start and end like every walk, so
its end is told as every walk's is. A failure stops the walk and leaves the layer as approved; the
rollout and names the `push --move` that sends it. The rollout says its start and end like every rollout, so
its end is told as every rollout's is. A failure stops the rollout and leaves the layer as approved; the
previous layer is in the settings history. The ask says all this before the Approve, and no longer
names a push. A layer set by hand still waits for a push.
@@ -481,9 +481,9 @@ A delivered file is not yet an applied one: a module whose program reads its set
starts applies them itself (a service's `restart-on`, or a tool of its own). Betterbird is such a
module; issue 643 makes it restart Betterbird at a quiet moment.
How it is checked: the controller's tests open the walk on the warrant, send one machine at a time
How it is checked: the controller's tests open the rollout on the warrant, send one machine at a time
with every build kept and the module judged, stop before a data move and on a failure without
holding other walks back, and read the ask's words.
holding other rollouts back, and read the ask's words.
### Proofs and tiers
03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md
+122 -122
View File
@@ -30,9 +30,9 @@ delivery group is two or more deliveries that share a head branch name, delivere
are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.**
(ADR 0239.)
**From its merge on, a delivery is delivered by the walk of its batch.** The controller collects merges in a
rolling merge window and plans each batch once, as one walk with at most one commit per repository, one walk
at a time. Every delivery the walk answers ends with it
**From its merge on, a delivery is delivered by the rollout of its batch.** The controller collects merges in a
rolling merge window and plans each batch once, as one rollout with at most one commit per repository, one rollout
at a time. Every delivery the rollout answers ends with it
(ADR 0276).
This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison
@@ -53,7 +53,7 @@ commit's journey and in what order the steps are allowed.
- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants,
registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk
rule, a gated send, the rollback and the walk of one trunk commit's tiers across machines. It still answers
rule, a gated send, the rollback and the rollout of one trunk commit's tiers across machines. It still answers
every pull request's head with its check itself, so a check never depends on mesh-delivery.
- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the
group, the state table and their state, and nothing else.
@@ -61,9 +61,9 @@ commit's journey and in what order the steps are allowed.
request, and appends to the commit's note under `refs/notes/mesh-plan`.
- **The build seat** builds and checks, asked by the controller as before. **A change is built once** (ADR 0350):
a check that passes also builds what its merge would move into the artifact store's scratch namespace, under a
reuse key (the source fingerprint and the builder's version), and the walk of the merge commit copies a build
of the same key, which the check's verdict names, by digest instead of building. Otherwise the walk builds and
says why, per module, in its record. Check builds go when the walk answering the merge ends, when the pull
reuse key (the source fingerprint and the builder's version), and the rollout of the merge commit copies a build
of the same key, which the check's verdict names, by digest instead of building. Otherwise the rollout builds and
says why, per module, in its record. Check builds go when the rollout answering the merge ends, when the pull
request closes unmerged, and after seven days.
## A delivery
@@ -76,7 +76,7 @@ Its id is `<owner>/<repository>@<twelve characters of its commit>`. It holds:
its diffset (what ADR 0238 called the "change plan", under the glossary's name);
- its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its
**machine steps** while delivering;
- once merged, **the commit it landed on the trunk as**, and the walk that delivers it: the walk of the batch
- once merged, **the commit it landed on the trunk as**, and the rollout that delivers it: the rollout of the batch
that answers its merge, which may carry a later commit of its repository that contains it (see *The merge
window and the batch*).
@@ -93,9 +93,9 @@ allows.
| checked | a verdict arrived | 1 minute | nothing: it is decided at once |
| ready | it passed; it waits to be merged | none | — |
| rejected | it failed its check | none | — |
| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a walk that does not answer its merge took over its walk; `go` when its walk started |
| published | merged on the trunk its modules follow, its rollout opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a rollout that does not answer its merge took over its rollout; `go` when its rollout started |
| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
| delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says |
| delivering | its rollout runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the rollout's record says |
| delivered, failed, superseded, stopped, retired, covered | final | — | — |
**A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head
@@ -105,7 +105,7 @@ commits in the transition's reason; a `rejected` one stays `rejected`, its statu
of date. Nothing is asked for it: a recheck, a new head or the merge tool asks the check again. The row from
`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for
an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own walk.
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own rollout.
What this leaves as it is, said so nobody reads more into it:
@@ -125,7 +125,7 @@ What this leaves as it is, said so nobody reads more into it:
The transitions are those of ADR 0239 decision 2, ADR 0270 and ADR 0323. A delivery that is
merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A
delivery merged with no walk answering it within ten minutes is held too, with the true reason, and listed by
delivery merged with no rollout answering it within ten minutes is held too, with the true reason, and listed by
`stalled` at once. Nothing waits silently.
**What needs no person ends by itself** (ADR 0270):
@@ -133,37 +133,37 @@ delivery merged with no walk answering it within ten minutes is held too, with t
- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and
sends nowhere) is `delivered` as soon as it is merged, whatever its check said;
- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine
running it by the walk of a later merge into the same trunk, delivered on its own passing check, and the
running it by the rollout of a later merge into the same trunk, delivered on its own passing check, and the
forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge
order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery`
says `covered: a later merge that contains it was delivered` as a success. A module that runs on one
machine only is covered too: its walk records the empty rest as sent once its first machine passes
machine only is covered too: its rollout records the empty rest as sent once its first machine passes
(ADR 0270, as corrected 2026-10-12);
- *a merge whose walk said its modules unchanged* moves nothing too, and is `delivered` when that walk ends,
whichever commit the walk carried (issue 611). A delivery plan made while another walk built a module reads
that module whole and names it; the walk, cut later, reads the build source that build recorded (ADR 0267) and
may find none of the changed files in it. The walk's record then says so in `delivery.unchanged`: each such
- *a merge whose rollout said its modules unchanged* moves nothing too, and is `delivered` when that rollout ends,
whichever commit the rollout carried (issue 611). A delivery plan made while another rollout built a module reads
that module whole and names it; the rollout, cut later, reads the build source that build recorded (ADR 0267) and
may find none of the changed files in it. The rollout's record then says so in `delivery.unchanged`: each such
module with why, "its build source holds none of the changed files", and not in its modules, which keep
meaning what the walk builds. A delivery whose every module the walk built or said unchanged is delivered,
saying which matched what runs. One with a module the walk neither built nor said unchanged is read as below.
meaning what the rollout builds. A delivery whose every module the rollout built or said unchanged is delivered,
saying which matched what runs. One with a module the rollout neither built nor said unchanged is read as below.
Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's
`TestAMergeWhoseModulesMatchWhatRunsIsDelivered`;
- *a stop whose walk ended without building its modules* (issue 474) stays `stopped`, a final state, and
- *a stop whose rollout ended without building its modules* (issue 474) stays `stopped`, a final state, and
is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after
its stop` with the covering merge, once, as a success. No state or transition is added (issue 480);
- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes.
**History is retired, not held** (ADR 0250).
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no walk
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no rollout
of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes
to `retired` at adoption, before any check, hold or walk, and its pull request is not told. A held delivery that
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its walk
to `retired` at adoption, before any check, hold or rollout, and its pull request is not told. A held delivery that
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its rollout
starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a
build registered before its turn would be carried by somebody else's send.
**The machine steps.** While a delivery is `delivering`, each machine its walk reaches has a row with the
**The machine steps.** While a delivery is `delivering`, each machine its rollout reaches has a row with the
module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy
readings so far), `passed`, `failed`, `rolled-back`. They are read from the walk's own record in the
readings so far), `passed`, `failed`, `rolled-back`. They are read from the rollout's own record in the
controller's `plan-moved` event, never inferred from time.
## The merge window and the batch
@@ -174,95 +174,95 @@ controller's `plan-moved` event, never inferred from time.
Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by
default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its
merges and its times are kept in the controller's store and survive a restart.
- **The cut.** When the window closes and no walk is open, the batch becomes one walk: one delivery plan for
- **The cut.** When the window closes and no rollout is open, the batch becomes one rollout: one delivery plan for
every commit in it, tiered along the module graph across repositories, with at most one commit per
repository, the latest of its branch. The walk names every merge it answers: its own commits, and the
repository, the latest of its branch. The rollout names every merge it answers: its own commits, and the
earlier merges of a repository that its later commit contains.
- **One walk at a time.** Merges that arrive while a walk is open join the next batch. A walk that started
is never superseded. A walk still waiting for its word when the next batch is cut is folded into the new
walk, which answers every merge the folded one answered.
- **The controller's own path.** A merge that moves a module whose walk waits for nobody's word (the controller,
- **One rollout at a time.** Merges that arrive while a rollout is open join the next batch. A rollout that started
is never superseded. A rollout still waiting for its word when the next batch is cut is folded into the new
rollout, which answers every merge the folded one answered.
- **The controller's own path.** A merge that moves a module whose rollout waits for nobody's word (the controller,
the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for
mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its
walk like any other. An own-path walk folds no waiting catalogue walk; that walk's merges are batched again
rollout like any other. An own-path rollout folds no waiting catalogue rollout; that rollout's merges are batched again
behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10).
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a walk is
answered by that walk: the assembling batch, the open walk, or, when that walk was delivered, it ends
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a rollout is
answered by that rollout: the assembling batch, the open rollout, or, when that rollout was delivered, it ends
`covered` at once.
- **The ends.** A walk done ends every delivery it answers: `delivered` for its own commits, `covered` for
an earlier merge of a repository it carried later, naming that merge. A walk that failed fails its own
commits' deliveries and marks none delivered; each earlier merge it answered is then walked on its own
commit, the newest first, one walk each, until one is delivered. A walk stopped stops every delivery whose
merge it walks again nowhere (below).
- **A group inside a walk.** The pairs of a delivery group's order (ADR 0249) are edges of the walk's
tiers beside the module graph. mesh-delivery gives a walk its word only when every delivery it carries is
- **The ends.** A rollout done ends every delivery it answers: `delivered` for its own commits, `covered` for
an earlier merge of a repository it carried later, naming that merge. A rollout that failed fails its own
commits' deliveries and marks none delivered; each earlier merge it answered is then rolled out on its own
commit, the newest first, one rollout each, until one is delivered. A rollout stopped stops every delivery whose
merge it rollouts again nowhere (below).
- **A group inside a rollout.** The pairs of a delivery group's order (ADR 0249) are edges of the rollout's
tiers beside the module graph. mesh-delivery gives a rollout its word only when every delivery it carries is
at its turn.
- **The bookkeeping.** mesh-delivery ties a walk to every live delivery whose merge it answers, follows a
folded walk to the walk that took it over, and never ties a walk to a delivery that has ended. A waiting
walk no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
- **The bookkeeping.** mesh-delivery ties a rollout to every live delivery whose merge it answers, follows a
folded rollout to the rollout that took it over, and never ties a rollout to a delivery that has ended. A waiting
rollout no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped:
novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated",
and a batch waiting behind an open walk names that walk. The controller says the batch on the bus as
and a batch waiting behind an open rollout names that rollout. The controller says the batch on the bus as
`plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged
delivery it holds.
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no walk open, and a batch
waiting behind an open walk past that walk's bound, each raise a condition. A merged delivery that no walk
answers within ten minutes is held with the true reason (the open walks of its repository and that none
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no rollout open, and a batch
waiting behind an open rollout past that rollout's bound, each raise a condition. A merged delivery that no rollout
answers within ten minutes is held with the true reason (the open rollouts of its repository and that none
names its merge, or that the controller holds none) and listed by `stalled` at once.
## A stop on a shared walk
## A stop on a shared rollout
*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.*
One walk carries every merge of its batch, so ending a walk for one delivery ends it for every other
One rollout carries every merge of its batch, so ending a rollout for one delivery ends it for every other
delivery on it. A stop therefore ends only the change it is for:
- **A group's stop.** When a member after a failed one is stopped and its walk is open, mesh-delivery asks
`delivery-stop` for that walk with the stopped members' merges to leave out, every member the same pass
stops on that walk in one ask. The controller ends the walk, said as stopped, and walks every other merge
- **A group's stop.** When a member after a failed one is stopped and its rollout is open, mesh-delivery asks
`delivery-stop` for that rollout with the stopped members' merges to leave out, every member the same pass
stops on that rollout in one ask. The controller ends the rollout, said as stopped, and rolls out every other merge
it answered again.
- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A
module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too.
Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped
merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is
walked again.
- **Where the others go.** They join the open batch of their kind, where merges heard while the walk ran
already wait, so an older commit never walks after a newer one; only with none open do they form a new
rolled out again.
- **Where the others go.** They join the open batch of their kind, where merges heard while the rollout ran
already wait, so an older commit never rollouts after a newer one; only with none open do they form a new
batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop
before it is cut, are cut without the rest of their window. Its walk withholds the left-out merges' modules
before it is cut, are cut without the rest of their window. Its rollout withholds the left-out merges' modules
where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes
to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every
merge moved are written in one transaction.
- **The walk's record.** The stopped walk names the batch that walks its merges again, every merge it left
out, and each module it already sent that nothing walks again, with the machines it reached. A delivery
whose merge is walked again goes from `delivering` back to `published` (the row `rewalk`), or stays
`published`, and follows the fresh walk once it is cut. A delivery whose merge was left out, and every
delivery on a walk stopped whole, ends `stopped`, naming the walk and the stop.
- **A stopped merge never ships by a walk.** mesh-delivery asks every open walk or batch that names a stopped
delivery's merge to leave it out, and never gives such a walk its word. When the controller refuses, it
stops the whole walk.
- **A person's stop.** `stop` of a delivery whose open walk carries other deliveries does nothing at first: it
- **The rollout's record.** The stopped rollout names the batch that rolls out its merges again, every merge it left
out, and each module it already sent that nothing rollouts again, with the machines it reached. A delivery
whose merge is rolled out again goes from `delivering` back to `published` (the row `rewalk`), or stays
`published`, and follows the fresh rollout once it is cut. A delivery whose merge was left out, and every
delivery on a rollout stopped whole, ends `stopped`, naming the rollout and the stop.
- **A stopped merge never ships by a rollout.** mesh-delivery asks every open rollout or batch that names a stopped
delivery's merge to leave it out, and never gives such a rollout its word. When the controller refuses, it
stops the whole rollout.
- **A person's stop.** `stop` of a delivery whose open rollout carries other deliveries does nothing at first: it
answers with each of them, which of them contain this merge and end either way, and two ways given with
`way`: `again` (the walk ended and the others walked again without this merge) or `walk` (the whole walk
stopped, every delivery on it with it). A delivery alone on its walk is stopped at once. A delivery is
marked stopped only once its walk is; a stop the controller refuses leaves it as it was.
- **What it does not do.** What the stopped walk already sent stays on the machines it reached, and its
`way`: `again` (the rollout ended and the others rolled out again without this merge) or `walk` (the whole rollout
stopped, every delivery on it with it). A delivery alone on its rollout is stopped at once. A delivery is
marked stopped only once its rollout is; a stop the controller refuses leaves it as it was.
- **What it does not do.** What the stopped rollout already sent stays on the machines it reached, and its
asked builds still register, so a later send may carry them further; the stop names the machines, and
putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the
stopped merge's modules, contains the stopped change and delivers it when walked; the stop names each it
stopped merge's modules, contains the stopped change and delivers it when rolled out; the stop names each it
can see, and holding a merged change back for good is a revert's job.
## The delivery budget
*Decided 2026-10-10 by ADR 0282.
Where it differs, this section replaces the window for every merge and the one walk at a time above, slice by
Where it differs, this section replaces the window for every merge and the one rollout at a time above, slice by
slice as each slice below is built; until a slice is built, the text above is what runs.*
**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard
from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes;
a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is
shown and held to no budget. Measured on 51 walks on 2026-10-09 and 10, a leaf walk took a median of 6.0 minutes
(7 of 20 within five), a core walk of one tier 5.6 (worst 8.8), and a controller walk dragging two catalogue tiers
shown and held to no budget. Measured on 51 rollouts on 2026-10-09 and 10, a leaf rollout took a median of 6.0 minutes
(7 of 20 within five), a core rollout of one tier 5.6 (worst 8.8), and a controller rollout dragging two catalogue tiers
14.2 (issue 338).
*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget
@@ -273,56 +273,56 @@ phases — implementing, review, checks, and merge to running (the delivery time
budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget.
The delivery budget stays as written: a measure inside the fix budget, not replaced by it.
**What changes in the walk.**
**What changes in the rollout.**
- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged.
- **Independent walks run at once**: no shared repository, module or module-graph edge with any open walk. One
node judges one walk at a time; a send to a node under another walk's judgement waits for its verdict; no leaf
- **Independent rollouts run at once**: no shared repository, module or module-graph edge with any open rollout. One
node judges one rollout at a time; a send to a node under another rollout's judgement waits for its verdict; no leaf
build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged.
- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf
module with a long-running resource; one for a leaf module whose every resource is a tools bundle.
- **A tier's first nodes are judged at once.**
- **The bus's grants arrive before the gate** (ADR 0298): when a walk's builds change the bus's user list (a new
- **The bus's grants arrive before the gate** (ADR 0298): when a rollout's builds change the bus's user list (a new
tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before
anything is judged; a step that would carry anything else is refused unsent and said on the gate line.
- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy
report; the plan shows an expected landing time from the last seven days' medians.
*Walks beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
a change shares).* Building the walks at once found what those bullets left open, and ADR 0306 decides it:
*Rollouts beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
a change shares).* Building the rollouts at once found what those bullets left open, and ADR 0306 decides it:
- **A walk's builds are its own, never another's backlog.** A send composes a module whose build another open
walk owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
judged only by its own walk, so a failed walk puts back only its own builds. What waits on a node with no open
walk to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
for builds an open walk owns).
- **A rollout's builds are its own, never another's backlog.** A send composes a module whose build another open
rollout owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
judged only by its own rollout, so a failed rollout puts back only its own builds. What waits on a node with no open
rollout to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
for builds an open rollout owns).
- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the
merge window whether or not a batch is assembling, so a group's members walk in one batch, in their order.
- **Related means everything the walks would move.** A module in common is any module a walk plans to build or
merge window whether or not a batch is assembling, so a group's members roll out in one batch, in their order.
- **Related means everything the rollouts would move.** A module in common is any module a rollout plans to build or
send, the dependents of what its merges moved included; a change whose modules are not known yet is related to
every change. Two related walks are neither cut beside each other nor judged beside each other on one node.
every change. Two related rollouts are neither cut beside each other nor judged beside each other on one node.
- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a
warning naming the walk and the core walks that set it aside.
warning naming the rollout and the core rollouts that set it aside.
- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an
unhealthy network or engine) fails every judgement open on it, leaf and core, each walk putting back only its
unhealthy network or engine) fails every judgement open on it, leaf and core, each rollout putting back only its
own builds.
- **A rollback beside another walk's judgement** behaves as it does today; whether it should wait is tracked by
- **A rollback beside another rollout's judgement** behaves as it does today; whether it should wait is tracked by
issue 528.
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of walk A left out
of walk B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
overlapping dependents keeping two walks apart; the warning on the third set-aside; a failed apply failing every
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of rollout A left out
of rollout B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
overlapping dependents keeping two rollouts apart; the warning on the third set-aside; a failed apply failing every
open judgement on its node.
*A busy container on the first node (2026-10-13, ADR 0335, citing ADR 0282 §1).* A container may declare `busy`,
a command that says whether it holds work a replacement would lose; the build agent declares it, so a merge
into the controller's repository no longer throws away the checks in flight. When a walk's send would replace
into the controller's repository no longer throws away the checks in flight. When a rollout's send would replace
such a container while it says busy, the node-engine leaves it running, applies the rest, and replaces it once
it says idle, or after 15 minutes at the latest. Only that node's walk waits: the walk says it waits for a busy
it says idle, or after 15 minutes at the latest. Only that node's rollout waits: the rollout says it waits for a busy
container there, its first-node gate starts at the report that says the container replaced, and the wait is a
phase of the delivery time of its own, never exempted from the delivery budget. A delivery it puts over budget
is said over budget, with that wait as its longest phase. *How it is checked:* the controller's test that a
walk whose module's container waits as busy on its first node says so and starts that node's gate only once the
rollout whose module's container waits as busy on its first node says so and starts that node's gate only once the
node reports it replaced, and the walk-phase test that the wait is a measured phase between the send and the
judgement.
@@ -330,11 +330,11 @@ judgement.
| Part | Owner | What it adds |
|---|---|---|
| the walk's phases and readings | mesh-controller | each phase's start and end and each reading in the walk and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
| the rollout's phases and readings | mesh-controller | each phase's start and end and each reading in the rollout and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` |
| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery.<class>.over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` |
| the lone merge | mesh-controller | the cut at once when nothing waits to join |
| walks at once | mesh-controller | the independence test, one judgement per node, a core walk's ground, a repository's next batch |
| rollouts at once | mesh-controller | the independence test, one judgement per node, a core rollout's ground, a repository's next batch |
| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once |
| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` |
@@ -342,11 +342,11 @@ judgement.
| Slice | Delivers | Done when |
|---|---|---|
| 1 — the measure | the walk's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its walk's phases |
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its walk |
| 4 — walks at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
| 5 — the landing time | the expected landing time | `plans` shows it for an open walk, and the week's deliveries land within a minute of it at the median |
| 1 — the measure | the rollout's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its rollout's phases |
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its rollout |
| 4 — rollouts at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
| 5 — the landing time | the expected landing time | `plans` shows it for an open rollout, and the week's deliveries land within a minute of it at the median |
Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges
only once it is built.
@@ -374,9 +374,9 @@ only once it is built.
- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`.
A member published while its group's composed check is asked and not answered waits there for the answer;
one held because that check had not passed is *lifted* back to `published` once it passes or warns for the
heads that merged and its own walk is still open (ADR 0270).
heads that merged and its own rollout is still open (ADR 0270).
- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its
walk while the walk's other merges are walked again (ADR 0299). The members before it stay delivered.
rollout while the rollout's other merges are rolled out again (ADR 0299). The members before it stay delivered.
Its own builds were put back at its gate.
A group's state is derived from its members and its composed check, and is shown, never stored apart from
@@ -408,22 +408,22 @@ In this order, each by its owner:
| Verb | Answers or does |
|---|---|
| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group |
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the walk |
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the rollout |
| `groups` | every group with its members in order and its derived state |
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere |
| `table` | the state table and the machine-step table, with bounds and H2's repairs |
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting walk no live delivery holds |
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting rollout no live delivery holds |
| `recheck` | a rejected or ready delivery back to proposed, with why |
| `release` | a held delivery to delivering, with why: a person's word |
| `stop` | any delivery not final to stopped, with why; its walk is ended through the controller. On a walk that carries other deliveries it first answers what ending the walk would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
| `stop` | any delivery not final to stopped, with why; its rollout is ended through the controller. On a rollout that carries other deliveries it first answers what ending the rollout would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise |
| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing |
**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`,
`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which
takes the merges to leave out and walks the walk's other merges again without them, ADR 0299),
takes the merges to leave out and rolls out the rollout's other merges again without them, ADR 0299),
`delivery-walks`. A person has `plans go <plan> --why`, which does what `deliver` does under their name and
is recorded as a hand act. The controller says every walk it keeps as `plan-moved`, from a verb's command
is recorded as a hand act. The controller says every rollout it keeps as `plan-moved`, from a verb's command
too.
**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It
@@ -438,12 +438,12 @@ up.
## The bootstrap
- A walk that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
- A rollout that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
controller's own, started by the merge and witnessed on the machine. mesh-delivery records it.
- Any other walk waits for `deliver` only while the seat has a holder on record. With none on record, the
- Any other rollout waits for `deliver` only while the seat has a holder on record. With none on record, the
controller starts it as it always did.
- A waiting walk is said by the controller whatever mesh-delivery says of itself: S16 raises
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the walk answers and
- A waiting rollout is said by the controller whatever mesh-delivery says of itself: S16 raises
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the rollout answers and
`plans go <walk> --why` as the way on.
A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone.
@@ -453,22 +453,22 @@ up.
merged and rolls out as the core does. With no holder on record it behaves exactly as before.
2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built
and published.
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Walks that start
from then on wait for its word. Walks already open finish on their own and are adopted as deliveries in
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Rollouts that start
from then on wait for its word. Rollouts already open finish on their own and are adopted as deliveries in
`delivering`.
4. To step back, unassign it. Waiting walks are then started by the controller at its next pass.
4. To step back, unassign it. Waiting rollouts are then started by the controller at its next pass.
## Phases
| Phase | Delivers | Done when |
|---|---|---|
| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` |
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a walk never let go is raised by S16 |
| C — the walk itself | the per-tier walk moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a rollout never let go is raised by S16 |
| C — the rollout itself | the per-tier rollout moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
## What is not decided here
- A web view of deliveries beyond the pull request's comment.
- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge
module's protection tool.
- Moving the per-tier walk out of the controller (Phase C).
- Moving the per-tier rollout out of the controller (Phase C).
03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md
+1 -1
View File
@@ -201,7 +201,7 @@ build alike, so any build of the module is excused while the wait checks out.
**Each module of a send keeps its own pass** (ADR 0254). A send is still judged as one, with one verdict.
The gate also counts, per module, the judgings in a row that found it healthy or waiting for a person;
when the send fails, a module whose count reached the passes asked keeps a pass, said "on its own", and is
neither put back nor left as a move no gate has seen for every other walk to wait on. A module found broken
neither put back nor left as a move no gate has seen for every other rollout to wait on. A module found broken
is marked failed and put back at once, and the send's verdict waits until the modules beside it have their
own, or the bound, so every module of a failed send leaves it with a verdict: its own pass, or put back.
03-DESIGN/01-to-be/49-the-mesh-in-domains.md
+1 -1
View File
@@ -31,7 +31,7 @@ shape — what each is for, where its records live, and how they depend on one a
| **Placement** | to decide, send and apply what each node runs, and say why | the controller's `inventory` database |
| **Provisioning** | to resolve which module serves a provision for which consumer, and wire the two | the controller |
| **Identity and access** | to issue, hold, rotate and check credentials and grants | the controller's `identity` and `licences` databases, and the vault |
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and walk |
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and rollout |
| **Health and repair** | to notice what is wrong, say it once, repair what may be repaired unattended, and record what was done by hand | the controller's condition store |
| **Data** | to know every item of data a module holds, how precious it is, and keep it recoverable | the manifests' data declarations, and every node's `node-backup` seat |
| **Connectivity** | to make every node and module reachable by name where it should be, and unreachable where it should not | the controller |
03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md
+11 -11
View File
@@ -12,7 +12,7 @@ decisions:
# 54 — The controller's records on the bus
**What each node runs, how far each walk has got, what each build is doing and what each setting holds
**What each node runs, how far each rollout has got, what each build is doing and what each setting holds
can be watched on the bus, and what is watched never says something the controller's store does not**
(ADR 0289,
from gap G2 of research 036).
@@ -51,7 +51,7 @@ do not have: a check that the bucket says what the store says.
Each value is built from a named list of fields. A field not on the list is never in the value. A value
holds only what is built from its record: no epoch, no time of writing and no time of the last look,
because any lease holder must build the same bytes from the same record, and the audit must too. Times
that belong to the record itself, such as when a walk was opened or a resource's state began, are fields
that belong to the record itself, such as when a rollout was opened or a resource's state began, are fields
like any other. The epoch and the time of the write travel in the event.
### nodes — `mesh-controller_nodes`, key: the node's name
@@ -74,15 +74,15 @@ like any other. The epoch and the time of the write travel in the event.
| the build the node was last sent for it and the build its node reports running | — |
| the seats it holds there, and the provisions it is bound to | the bindings' credentials, which are secrets |
### walks — `mesh-controller_walks`, key: the walk's id
### rollouts — `mesh-controller_walks`, key: the rollout's id
| Holds | Leaves out, and why |
|---|---|
| the repository and commit, or the batch of merges it walks (ADR 0276) | — |
| the repository and commit, or the batch of merges it rollouts (ADR 0276) | — |
| its state (building, rolling, done, failed), its current tier, and each tier's modules and nodes with each step's state | the build and apply logs, which the `plans` and `builds` verbs answer |
| when it was opened, when it last moved, when it ended, and its note | — |
A walk is removed from the mirror 14 days after it ended.
A rollout is removed from the mirror 14 days after it ended.
### builds — `mesh-controller_builds`, key: the build's id
@@ -137,8 +137,8 @@ called).
caller today): a numbered migration of `inventory` and the consolidated schema, in the same change.
- **A removal is a purge** of the key, said as a `removed` event. A purge leaves one marker for the key.
Each purge gives its marker a time to live of one day (a per-message time to live, which the bus server
has offered since release 2.11 and the bucket enables), so the markers of removed walks and builds do
not pile up in a bucket that has no age. The controller's one-minute tick removes walks and builds 14
has offered since release 2.11 and the bucket enables), so the markers of removed rollouts and builds do
not pile up in a bucket that has no age. The controller's one-minute tick removes rollouts and builds 14
days after they ended.
- **A write that fails** sends no event, is logged, and is left to the audit. The writer does not retry
in a loop: a retry loop against a refusing bus is the reconnect loop of issue 327 in another place.
@@ -169,7 +169,7 @@ counts in §6 shows the gap.
**D17, the mirrors match their store**, is a probe in the self-check's registry (to-be 45 §4). Every run
does this for each mirror:
1. It reads every record the mirror should hold from the store (for walks and builds, those open or ended
1. It reads every record the mirror should hold from the store (for rollouts and builds, those open or ended
within 14 days) and builds each value as the writer would.
2. It lists the bucket's keys and reads their values.
3. It compares. A key missing, a value that differs, and a key with no record are each a candidate. A
@@ -225,10 +225,10 @@ remain open there.
| The bucket matches the store | D17 on every self-check run; its own test, which changes a key behind the writer's back and expects `mirror-drifted`, then H6's write, then the condition cleared on the next run; and a test that a change of lease alone raises nothing |
| Quiet means no writes | live: over ten minutes in which `plans`, `nodes`, `builds` and `settings` show no change, no mirror's write count rises |
| A watcher sees a change | live: a setting changed through `settings` is seen by a watch on `mesh-controller_settings` and as one `setting-changed` within a second |
| A value is under its cap, or cut and marked | the controller's test, which builds the largest walk the mesh could make (every module on every node) |
| A value is under its cap, or cut and marked | the controller's test, which builds the largest rollout the mesh could make (every module on every node) |
| Only the audit reads a mirror | a test over the controller's packages, which fails when any package but the audit's reads a mirror bucket |
| A write follows the commit | the controller's test that a rolled-back transaction makes no write and sends no event |
| Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals |
| Ended rollouts and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the rollouts bucket's keys including deleted ones at most its live keys plus a day's removals |
| A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run |
| A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set |
| A lost acknowledgement is no conflict | the controller's test in which the bus stores a write and drops its acknowledgement: the next write to that key succeeds and no condition is raised |
@@ -236,7 +236,7 @@ remain open there.
## What it leaves out
- **History.** A mirror is current state. What happened beyond the 14 days a walk or build is kept, and
- **History.** A mirror is current state. What happened beyond the 14 days a rollout or build is kept, and
beyond EVENTS' week, is gap G8 of research 036, which remains open there.
- **People reaching the mirrors**, through a WebSocket and an auth callout, is gap G1, open in the same
effort.
03-DESIGN/01-to-be/55-the-knowledge-base.md
+1 -1
View File
@@ -130,7 +130,7 @@ rules that judge the whole record set run (an accepted decision cited by nothing
undo the acceptance. It is a fault of the records, which their authors repair, not of the module: it is told to the
operator once, as a warning naming the rule and the records, and cleared once the rule holds again. The module's
health names it and stays healthy, because health says only whether the module works, and a finding never holds a
walk at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
rollout at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
reviewed).
The rule `cited` is also judged ahead, in the review, before the judgement: a decision that names no home and that no
03-DESIGN/01-to-be/57-building-config-and-provisioning.md
+7 -7
View File
@@ -32,10 +32,10 @@ reviewed independently at its head and merged only with its checks green.
**It ships as one release.** The operator's decision of 2026-10-14: one delivery, not one per package.
Each of mesh-controller, `mesh-host`, mesh-sdk, mesh-catalog and hq has the branch
`feat/57-config-and-provisioning`, and every package's pull request merges into that branch, never into
main. When the last package is merged, the whole is rehearsed in the lab: the walk, the moves of data in
main. When the last package is merged, the whole is rehearsed in the lab: the rollout, the moves of data in
their order, and every proof that is run live. Then each repository's branch reaches main in one pull
request, and the five, sharing their branch's name, are one delivery group (ADR 0239 §2), judged as one
future state. The five are merged within one merge window, so they are one batch and one walk (ADR 0276
future state. The five are merged within one merge window, so they are one batch and one rollout (ADR 0276
§1, §2), in an order declared with `after:` lines in each pull request (ADR 0239 §11): mesh-sdk, then
`mesh-host`, then mesh-controller, then mesh-catalog, then hq. The release holds every package but T6, R1 and R2:
the media catalogue and the applications live in repositories of their own and move after it, and only
@@ -52,9 +52,9 @@ database consumer connecting) is first seen in the lab rehearsal, and a fault fo
release, not one package. A merge's builds go to every machine at once (ADR 0360), so the release has no
first node to hold it: after the pull requests' checks, the lab rehearsal is the only live check before it
reaches every node. A machine that fails to apply it, or whose provisioner reports a module's files failed
(ADR 0362), is named in the walk's end and raises apply-failing, and is fixed forward with a new pull
(ADR 0362), is named in the rollout's end and raises apply-failing, and is fixed forward with a new pull
request into main; one that says nothing within fifteen minutes is named silent and gets its builds from
the backlog walk once it is heard from (ADR 0360 §10).
the backlog rollout once it is heard from (ADR 0360 §10).
**Nothing breaks on the way.** ADR 0349 clause 17: each file is filled the old way or the new way, never
both, by the form of its manifest, and the old forms stay accepted until the last module has moved.
@@ -247,11 +247,11 @@ back on drift, removes a module's files and needs when it leaves, fills from cur
comes back. Its first owners are placement (from the declaration) and config (from
`mesh-config_resolved`). **Proof:** the client's test list in ADR 0349, against a real bus.
**P6 (issue 672) — The controller hands a module to the provisioner, and the walk waits for it.** *mesh-controller.
**P6 (issue 672) — The controller hands a module to the provisioner, and the rollout waits for it.** *mesh-controller.
Two days.* For a module marked moved, the declaration carries placement, the templates' digest and the
filled-files list, and the controller runs no other filling pass for it; a walk counts the module applied
filled-files list, and the controller runs no other filling pass for it; a rollout counts the module applied
on a machine only after the node-engine's report and that node's `prov.applied` for the new build, and a
failed `prov.applied` is a failed apply (ADR 0362). dunst moves first. **Proof:** composition and walk
failed `prov.applied` is a failed apply (ADR 0362). dunst moves first. **Proof:** composition and rollout
tests; dunst on the first workstation filled by the client, live.
**P7 (issue 673) — The node-engine stops writing placed files from the declaration.** *`mesh-host`. A day.* For a moved
03-DESIGN/01-to-be/README.md
+2 -2
View File
@@ -47,13 +47,13 @@ document is written and this one's status becomes `implemented`.
| [`42-the-machines-modules-in-order.md`](42-the-machines-modules-in-order.md) | **In progress.** The order the machines' modules of research 026 and 027 are built and rolled out: every machine's first (sudo, localization, time sync, pacman, logrotate, avahi, systemd, docker, `~/.ssh`, scripts, kernel), then both workstations', then one machine model's; each proven on one workstation before the rest | ADR 0173, ADR 0182, ADR 0205 |
| [`45-a-core-that-cannot-fail-silently.md`](45-a-core-that-cannot-fail-silently.md) | **Designed.** The core says when it is wrong, refuses what is stale or unreadable, heals what it knows, upgrades one machine at a time with a witness that rolls it back, and is checked against the real mesh before merge: the writers and signals tables, the condition store, `doctor`, the minimal output channel, healers, the lease and report order, staged upgrades, the facts snapshot and replays, in six phases | ADR 0227, ADR 0224, ADR 0218 |
| [`46-the-conversation-with-the-operator.md`](46-the-conversation-with-the-operator.md) | **Designed.** The mesh tells and asks its operator over channels that are holders of two kinded benches, `channel` and `intake`, declaring capabilities from a fixed vocabulary; the router orders them by work context and never lowers the bar; an answer that performs an action is checked and performed by the controller, on a TOTP code or a verified Telegram sender, never on a desk click alone; operator messages addressed or answered by the mesh's responder, untrusted input kept as data, references for what words may not carry, and a recoverable factor; Telegram first, in six phases | ADR 0234, ADR 0227 |
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the walk, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the rollout, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
| [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of eleven domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | ADR 0244, ADR 0006, ADR 0008, ADR 0302 |
| [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | ADR 0247, ADR 0223 |
| [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | ADR 0251, ADR 0189 |
| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **In progress.** A node shares one folder at a time through `nfs-server`, which provides it as a provision to the private network only, mapping every client to the folder's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | ADR 0263, ADR 0051, ADR 0092 |
| [`53-the-issue-tracker.md`](53-the-issue-tracker.md) | **Designed.** The mesh keeps its issues in one module, `issues`, on the control-node: every number from one counter advanced by compare-and-set, the cycle's issue rules refused at the write, an event per change; an issue lives whole in the tracker, hq keeping no issue content (ADR 0291) | ADR 0275, ADR 0291 |
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, walks, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, rollouts, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
| [`55-the-knowledge-base.md`](55-the-knowledge-base.md) | **Designed.** The mesh keeps its decision records and research efforts whole in one module, `mesh-knowledge`, on the control-node: number, status, topic, typed links both ways, each decision's clauses with their own status, and the reasoning, with every version; a record is proposed, checked at the write, reviewed by the knowledge base's own judgement and accepted there, with the operator's Approve when it needs it; agents ask what holds in one bounded call | ADR 0290 |
| [`56-the-mail-client-set-up-by-the-mesh.md`](56-the-mail-client-set-up-by-the-mesh.md) | **Designed.** On every node with a display server, the `betterbird` module installs Betterbird from a pinned upstream release mirrored into the artifact store, ships the Owl add-on pinned and installed by Betterbird's enterprise policy, and declares the operator's mail accounts in a trusted setting without credentials; each account signs in once at a person's first use, and one the profile already holds is adopted by its address | ADR 0316, ADR 0317 |
| [`57-building-config-and-provisioning.md`](57-building-config-and-provisioning.md) | **Designed.** The work of ADR 0348 and ADR 0349 as 37 packages, each an issue with its proof: foundations, template files, config in its own module, the provisioner and its client, secrets in the vault, bindings from providers, collections, a module's runtime readers, and removing the old forms; at most five worked at once | ADR 0348, ADR 0349 |
04-ISSUES/021-a-consumer-on-the-providers-machine-is-given-no-credential/00-report.md
+1 -1
View File
@@ -28,7 +28,7 @@ for _, n := range nodes {
```
So a provider on the same machine is never a `Provider` in `world.Offered`, never becomes a
`Needed`, and the credential loop — which walks `resolved.Needs` — has nothing to walk. Every step
`Needed`, and the credential loop — which traverses `resolved.Needs` — has nothing to traverse. Every step
is individually reasonable and the sum is a silent gap.
## Why it was not noticed
04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md
+1 -1
View File
@@ -92,7 +92,7 @@ node that looks like a real one.
## Fixed
Needs fan out per consuming module in one place, after the resolution walk. The credential's key
Needs fan out per consuming module in one place, after the resolution rollout. The credential's key
gains the consuming module, the grant file is named after both halves of the consumer, needs are
matched by provision *and* module, and the provisioners name the role and the access key after the
module rather than the machine. The refusal is gone because there is nothing left to refuse.
04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md
+1 -1
View File
@@ -87,7 +87,7 @@ too new to have anybody watching it.
## Fixed
**The registry module names its image and is added, not built.** The lab's artifact-store test now
walks the only path open to a real first mesh — the manifest goes in directly, no builder and no
rollouts the only path open to a real first mesh — the manifest goes in directly, no builder and no
store involved — and passes.
**And the cycle is refused where it is written.** A manifest that provides `artifact-store` and
04-ISSUES/107-a-declaration-carries-no-order/00-report.md
+1 -1
View File
@@ -45,7 +45,7 @@ field it does not know — whole. That was
mesh now records which host each machine reports and `status` names every machine running an older one
than another does.
So the flag day is visible before it is walked into, which it was not when this was filed. It does not
So the flag day is visible before the mesh runs into it, which it was not when this was filed. It does not
make the field free: **the oldest host in the mesh is still what the mesh may send**, and one machine of
four is behind today. A sequence that an old host refuses takes that machine out of the mesh's reach
entirely — worse than the replay it prevents, which has never been observed.
04-ISSUES/107-a-declaration-carries-no-order/01-diagnosis.md
+1 -1
View File
@@ -18,7 +18,7 @@ half-understood declaration off a machine, and exactly what makes this expensive
[Issue 087](../087-the-controller-cannot-tell-a-host-is-too-old/00-report.md) is resolved: the mesh now
records the host version each machine reports and `status` names every machine running an older host
than another does. The flag day is visible before it is walked into, which it was not on 2026-09-23.
than another does. The flag day is visible before the mesh runs into it, which it was not on 2026-09-23.
That makes the cost measurable rather than hypothetical, and the measurement is the reason this is not
being built today.
04-ISSUES/117-a-modules-own-code-is-a-container-and-a-process/00-report.md
+1 -1
View File
@@ -72,7 +72,7 @@ what the sidecar is has nowhere in the design layer to look, which is how this w
resource type no decision records and contradicting an accepted one. The rule is "no design
without a decision"; the check is "no design without *a* decision." An unenforced rule is
indistinguishable from a wrong one, and these two documents are what that gap looks like when
something walks through it.
something rollouts through it.
## Open questions
04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md
+1 -1
View File
@@ -92,7 +92,7 @@ built from a repository elsewhere, or a mesh whose forge module is down — and
event performs, so the two cannot disagree about what "moved" means.
**Bases first needed edges, and there were none.** The order this report asked for was written and
walked a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
traversed a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
digest was never in the file the builder read edges from. A build now reports what it was handed, the
control plane records it by artifact path, and the order is read from each module's newest build.
04-ISSUES/178-a-routed-name-resolves-to-a-provider-merely-told-it/00-report.md
+1 -1
View File
@@ -38,7 +38,7 @@ Every node's resolution is read first, and the names are attributed across them
terminus serves the name**: among the providers a name reaches, the one that is not itself published
under a labelled name through another provider. An identity provider is routed through the proxy and
so is a consumer of names, not their end; the proxy contributes no label to anyone and is. The rule
knows nothing of what "route" means — it reads the graph the modules declared — and it walks nodes,
knows nothing of what "route" means — it reads the graph the modules declared — and it rollouts nodes,
requirements and contributions in order, so one mesh yields one region. Every shape of contribution
is counted, so a module routed under several names has every one of them resolved.
04-ISSUES/186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md
+1 -1
View File
@@ -73,5 +73,5 @@ give.
All three faults are closed on the controller's main: one ask in flight (8d4e940), a merge rebuilds what
stands on it (7e701c0), and a merge produces a tiered plan the mesh keeps, read with `plans` (d2ed6d5,
76f2756, ADR 0162). Plans were since carried into walks and deliveries (ADR 0236, ADR 0239); the order a
76f2756, ADR 0162). Plans were since carried into rollouts and deliveries (ADR 0236, ADR 0239); the order a
person kept is the order the tiers give.
04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/00-report.md
+1 -1
View File
@@ -86,6 +86,6 @@ machine. That is the path the console exists to replace, so it is a workaround a
Both pull requests named above are merged and live. Read through the console on 2026-10-08:
`mesh_describe mesh-controller.plan` declares `node` (required) and `files`; `mesh-controller.push`
declares `node`, `behind`, `cause` and `why`, and its description says that a push naming no machine
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` walks every verb it serves:
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` exercises every verb it serves:
an argument a verb does not declare is refused, and every flag of a verb's command is in its schema
or accounted for.
04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/01-diagnosis.md
+1 -1
View File
@@ -46,7 +46,7 @@ Pull requests: novox/mesh-tools#14 and novox/mesh-controller#71 — open, not me
first line of its answer, that it is a push of the whole mesh and which machines it is sending, and
its last line names the machines told. `plan` gains `files`, `push` gains `behind` (the whole mesh
said outright, refused beside `node`), and the listing verbs a `limit`.
- **Checked by tests that walk every verb the controller announces:** for every combination of a
- **Checked by tests that exercise every verb the controller announces:** for every combination of a
verb's declared arguments, the verb either refuses or every argument given changes the command it
runs; every verb refuses an argument it does not declare, and `node` where it takes none; and every
flag of the command a verb runs — read from the command's own source — is an argument of the verb's
04-ISSUES/259-a-named-push-sent-every-machine/00-report.md
+1 -1
View File
@@ -26,7 +26,7 @@ list alone. The flush has no such narrowing.
## What it costs
`record` is the policy for a change that must be walked through the mesh by hand. A named push cannot
`record` is the policy for a change that must be rolled out through the mesh by hand. A named push cannot
do that, so the policy does not hold the change back from anything but the merge. The command's name
says one machine, and the command's output is the only place that says otherwise.
04-ISSUES/290-a-delivery-adopted-while-it-waited-never-got-its-check/00-report.md
+1 -1
View File
@@ -61,7 +61,7 @@ The mesh-delivery tests `checkask_test.go` cover:
- An announcement that carries a decided gate status uses it.
- H2 re-asks once, a second re-ask is refused by the table, and a late verdict still settles the delivery.
The table tests walk the new row, and the pairs the table does not hold are refused by name. Live: once
The table tests exercise the new row, and the pairs the table does not hold are refused by name. Live: once
the fix is rolled out, the stalled delivery has its check asked at the owner's first tick. The controller
journal then shows its verdict, and `show` moves from proposed to checked to ready without a person
acting.
04-ISSUES/295-a-recorded-build-was-carried-by-a-plans-send/00-report.md
+3 -3
View File
@@ -39,7 +39,7 @@ The mesh's own record shows it:
- **What each machine reports.** `postgres.server`, `mongodb.server` and nine `mailu.*` containers
have run only since 18:57 on the control node, and `postgres.server` since 18:56 on the anchor.
No walk held back for later was involved. The sends were the plan's own sends to its first machines.
No rollout held back for later was involved. The sends were the plan's own sends to its first machines.
## Why it happened
@@ -53,7 +53,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
push". So the gate did not judge the recorded builds, the refusal of ungated sends did not see
them, and the gated send carried them. `record` held only against the end of a person's push to
another machine (issue 259's `heldBack`) and against the bus (the bus step). It did not hold
against a plan's walk, a rollback, a healer or a rotation.
against a plan's rollout, a rollback, a healer or a rotation.
3. **Nothing tells a person, before or during a send, that it will recreate every container of a
module at once.** A change that only alters how containers are declared reads as harmless. A
health check, an environment variable or a label each recreates the container all the same,
@@ -64,7 +64,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
## What is at stake
`record` is how the mesh keeps the providers, the network path and the irreplaceable data out of
unattended walks. Tonight showed that it holds only until some other module on the same machine
unattended rollouts. Tonight showed that it holds only until some other module on the same machine
rolls out. This was still true after the incident: plex (`record`, irreplaceable data) was rebuilt
by a media-catalogue merge at 19:01 and sent nowhere. The next gated send to the anchor, for any
module, would recreate it.
04-ISSUES/296-a-plans-clock-restarted-at-every-save/01-diagnosis.md
+5 -5
View File
@@ -13,8 +13,8 @@ the tier durations measured for the repository: three times their ninetieth perc
than 30 minutes. So the watchdog and the line a person reads used different start times and could use
different bounds. The line was the one that was wrong.
The line for a walk waiting on its delivery's word had the same flaw. It said "for" the time since the
last save. The watchdog for that wait (S16) counts from when the walk was opened.
The line for a rollout waiting on its delivery's word had the same flaw. It said "for" the time since the
last save. The watchdog for that wait (S16) counts from when the rollout was opened.
Ruled out: a clock difference between the machine running the verb and the store. The count was small
and kept growing again from zero, which only a column being rewritten explains.
@@ -44,8 +44,8 @@ nothing.
## Fix (mesh-controller)
- The line, the `since` and `late` fields of `status --json`, and the count of late plans all start from
when the plan entered its tier. If the delivery let the walk go after that, they start from the go. A
walk still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
when the plan entered its tier. If the delivery let the rollout go after that, they start from the go. A
rollout still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
called late. A plan read without the tier stamp falls back to its last save.
- LATE uses the bound the watchdog uses: the measured bound for the plan's repository, or the 30-minute
floor when nothing has been measured. The watchdog's start time and bound now come from the same two
@@ -53,4 +53,4 @@ nothing.
- The loop compares the plan before and after each step and saves only when they differ. The fields the
store stamps itself are left out of that comparison.
- A refused step is logged and saved only when its note is new. The refusal's wording called the gated
walk a "release plan", a name the glossary retires; it now says "walk".
rollout a "release plan", a name the glossary retires; it now says "walk" (since renamed rollout, issue 646).
04-ISSUES/300-a-new-module-is-not-built-at-its-merge/00-report.md
+1 -1
View File
@@ -23,7 +23,7 @@ another, with nothing on the mesh noting the difference.
ADR 0238 makes the delivery plan the planner's one answer for a pull request's check, the merge gate and
the merge. The plan counts a new module's directory as something the merge builds. The merge handler
read the same answer (`touchedBy`'s `added`) and dropped it: it only walks modules the catalogue already
read the same answer (`touchedBy`'s `added`) and dropped it: it only rollouts modules the catalogue already
holds, and a new one is not held until it is built. The code said so outright: "a new module, which a
check judges before it merges and a merge does not build". The take-in refuses a build off the trunk
with "merge it, and the merge builds it". So one part of the controller promised the build and another
04-ISSUES/300-a-new-module-is-not-built-at-its-merge/01-diagnosis.md
+1 -1
View File
@@ -29,7 +29,7 @@ already built from it are, and without waiting (`askNewModules`). The build's ta
module, as it does for a hand `build`. Nothing is sent: no machine is assigned a new module, and
assigning one stays a person's act.
The new module is outside the merge's plan. A plan walks modules the catalogue holds, tier by tier, and
The new module is outside the merge's plan. A plan rollouts modules the catalogue holds, tier by tier, and
sends each built one to the machines running it. A new module has no machine and nothing standing on it,
and the plan's ask for a module the catalogue does not hold fails the plan.
04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/00-report.md
+9 -9
View File
@@ -7,16 +7,16 @@ opened: 2026-10-08
## Symptom
On 2026-10-08 an agent proving the tunnel join (novox/mesh-lab PR #61) on the laptop could not run the
two-node walk, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
two-node rollout, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
reported by any check before someone tried to run a bed:
1. **The walk's scenario stocks a packet-filter runtime image that cannot be built.**
1. **The rollout's scenario stocks a packet-filter runtime image that cannot be built.**
`scripts/build-module-runtime.sh` copied the tool runner's compiled TypeScript and its `node_modules`
out of mesh-tools. mesh-tools has had no package at its root since the runtime became `node-tools`, a
Go binary that launches every bundle (ADR 0175,
ADR 0193),
so the script failed at its first step.
2. **The walk's builder dialled the old broker.** It was started with the old bus's guest account on
2. **The rollout's builder dialled the old broker.** It was started with the old bus's guest account on
loopback. The mesh moved its bus to NATS (ADR 0106,
ADR 0127,
ADR 0131), and the builder
@@ -27,8 +27,8 @@ reported by any check before someone tried to run a bed:
4. **The replays prover could not run a test in mesh-tools**, because that repository's Go module lives
in a subdirectory (`node-tools/`), and the prover ran `go test` only at a repository's root.
The same day, while trying to run the walk again, two more gaps surfaced: the object store's
provisioner image, which the walk also stocks, cannot be built, because the client image it copies
The same day, while trying to run the rollout again, two more gaps surfaced: the object store's
provisioner image, which the rollout also stocks, cannot be built, because the client image it copies
from is gone upstream; and the laptop could not reach the public container registry, which the lab's
egress check needs (an outside condition, not the lab's fault).
@@ -56,9 +56,9 @@ In novox/mesh-lab PR #63, stacked on #61:
builds `node-tools` from mesh-tools' Go module, and runs it as the image's entrypoint. The image serves
on the node's runtime credential: the Go runtime refuses to serve a module on that module's own
credential.
- **The walk no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
- **The rollout no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
names an image (its verbs are a tools bundle the node's runtime launches), so the image was never run.
- **The walk's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
- **The rollout's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
an own `broker` secret, as the catalogue's build agent does. Assigning it to the anchor issues the
account ([issue 203](../203-a-fresh-assignment-is-pushed-before-its-credential-exists/00-report.md)).
The bus's user list is placed, the push writes the sealed credential, and the builder starts on it.
@@ -71,12 +71,12 @@ In novox/mesh-lab PR #63, stacked on #61:
Still open:
- `builds.test.ts` still dials the old broker. Its bed enrols no machine, so the mesh has nowhere to
deliver a builder's credential. It needs the walk's enrolment, or it should be retired into the walk.
deliver a builder's credential. It needs the rollout's enrolment, or it should be retired into the rollout.
- Beds that run a per-module runtime container on the module's own credential (the tools beds for the
wiki and the code forge, and the audit bed) run into the runtime's refusal. They need the node's
runtime credential instead.
- The object store provisioner's image needs a client source that still exists.
- The walk has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
- The rollout has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
public container registry.
## How it is checked
04-ISSUES/307-the-labs-beds-still-dialled-the-bus-and-runtime-the-mesh-had-left/01-diagnosis.md
+5 -5
View File
@@ -14,25 +14,25 @@ script now does those steps by hand. Verified: the image for the packet filter b
answers `initialize` and `tools/list`; `node-tools` in it, started against a real bus with a plain URL,
launches the bundle and serves nothing until the mesh issues a membership, which is correct.
**Why the walk stocked that image at all.** The tunnel branch passed the held images to
**Why the rollout stocked that image at all.** The tunnel branch passed the held images to
`deriveTheFilterOn`, which turns the catalogue's manifest into the lab's form: build section gone, each
container's artifact replaced by a stocked image. The catalogue's nftables manifest has no container,
so no image is referenced. Removing the build section is what the bed needed; the stocked image was
not.
**The builder.** mesh-controller's `cmd/mesh-builder` reads `MESH_BROKER_FILE` and nothing else for the
bus. Without it, it says it has no credential. The walk set the old broker's variable, which the builder
bus. Without it, it says it has no credential. The rollout set the old broker's variable, which the builder
no longer reads. A builder's account is a module's account (`builder issue` and `module issue` both mint
through the inventory), composed into the bus's user list and sealed to the machine as the module's
`broker` own secret. Assigning a module that declares that secret now issues it in the same act (issue
203). The catalogue's build agent is that module, but it runs the builder as a container from an
artifact the mesh builds, and it needs the artifact store and package registry bound. The walk has
artifact the mesh builds, and it needs the artifact store and package registry bound. The rollout has
neither before it can build, so it declares a lab module with the same claim and the same secret. In
this bed the bus comes from the bundle, so its user list is placed by hand, the step the walk already
this bed the bus comes from the bundle, so its user list is placed by hand, the step the rollout already
takes at enrolment.
**The uplink and the prover** were each fixed by the agent that met them (mesh-lab #61 and #62), and are
listed here only so the lab's state on that day is in one place.
**Ruled out.** That the walk failed because of the tunnel changes. The builder and runtime faults are in
**Ruled out.** That the rollout failed because of the tunnel changes. The builder and runtime faults are in
code that #61 does not touch, and they reproduce on main.
04-ISSUES/309-two-order-rules-ordered-one-pair-both-ways/01-diagnosis.md
+1 -1
View File
@@ -21,7 +21,7 @@
the planner's sort stalled. Every two-way pair was a cycle, whatever the rules.
- The person's order, the controller first, was right for this change. The new node-engine calls a verb
the new controller adds, which is ADR 0246's case: a verb is promised before it is required.
*Engine before controller* comes from ADR 0236's order for its walks, where the node-engine had to
*Engine before controller* comes from ADR 0236's order for its rollouts, where the node-engine had to
accept something new first. The rule reads only module names, so it cannot tell the two cases apart. *Built
by* is read from the graph.
04-ISSUES/310-a-pull-request-ran-its-own-copy-of-its-check/01-diagnosis.md
+1 -1
View File
@@ -59,7 +59,7 @@ and fails only when it cannot read.
**Merged.** mesh-controller#135 (the build agent judges a pull request by the base branch's
`merge-check.sh`) merged on 2026-10-08 at 09:10 UTC, and mesh-lab#65 (R310) at 09:10 UTC. Every later
controller delivery carries it; the last, of mesh-controller#170, finished its walk on 2026-10-09 at
controller delivery carries it; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at
00:11 UTC, and `mesh-controller.status` named no machine behind.
**The open half is closed.** `gitea_branch_protection_get` for the media catalogue's main, read on
04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md
+5 -5
View File
@@ -32,11 +32,11 @@ was asked of a new, unrelated pull request.
## Why it is a design issue
ADR 0239
decision 10, *the switch*, adopts the walks open when mesh-delivery is assigned. It says nothing of the merges the
decision 10, *the switch*, adopts the rollouts open when mesh-delivery is assigned. It says nothing of the merges the
bus still holds from before: the forge's `pull.merged` is kept on the bus and heard again by a new consumer.
Each was taken as a merge just made whose head was never heard, so the table's row *merged unchecked → held*
held each for a person. The table had no state for "this already happened", and `held`'s only ways on were
a release, which would walk it, and a stop, which says a person stopped a delivery that was never running.
a release, which would roll it out, and a stop, which says a person stopped a delivery that was never running.
Healer H2's `close` has no transition from `held`, rightly: a held delivery is the operator's.
Decision 5 says what a transition owes outside is "retried until done". An effect the forge answers can never
@@ -45,14 +45,14 @@ be done was retried without end, every ten seconds, and nothing said so.
## Fix
- **A new final state, `retired`: history.** A delivery made from the forge's word at its merge, whose merge
was heard fifteen minutes or more after it happened and whose walk, if the controller keeps one, is not
was heard fifteen minutes or more after it happened and whose rollout, if the controller keeps one, is not
open, is history. At adoption it goes straight to `retired` by the observed row *history*, before any check,
hold or walk. Its pull request is not told: the forge may no longer hold it, or holds another under its
hold or rollout. Its pull request is not told: the forge may no longer hold it, or holds another under its
number. Its transitions are still said on the bus and kept on the commit's note.
- **A person's verb for the ones already held: `retire-history`**, with why. One delivery by id, refused when
the rule does not take it, or every held one (of one repository when named), answering how many it retired
and naming each one it left held. `dry` says what it would do and changes nothing. It is the act
*held → retired* of the table, never a release: nothing is walked, delivered or put back.
*held → retired* of the table, never a release: nothing is rolled out, delivered or put back.
- **Owed work ends.** The forge's answer that what an effect is for does not exist (a 404, or "does not
exist") is final: the effect is given up with one line, kept among the owner's refusals. Any other failure is
tried again later each time, from ten seconds doubling to an hour, and given up after sixty tries, saying so.
04-ISSUES/313-history-was-held-for-a-person-and-its-views-retried-for-ever/01-diagnosis.md
+5 -5
View File
@@ -5,20 +5,20 @@
- 699 deliveries were `held`. 688 say "merged without a passing check", all made between 23:05:27 and
23:05:29 on 2026-10-06, the owner's first start. A sample across repositories shows merges from
2026-10-02 to two hours before that start, each with the transition "its head was not heard by this owner,
so it is made from the forge's word", then *merged unchecked → held*. None has a walk.
- The other 11 are held for other reasons: a merge with no walk opened within ten minutes (10), and a group
so it is made from the forge's word", then *merged unchecked → held*. None has a rollout.
- The other 11 are held for other reasons: a merge with no rollout opened within ten minutes (10), and a group
whose composed check did not pass (1). They are not history, and this issue leaves them alone. One of them
was past its bound, which makes the 689th finding.
**Where it comes from.** The holder subscribes to the forge's `pull.merged` on the bus after reading back its
state and the controller's walks. The bus hands a new consumer the merges it still holds. For each, the holder
state and the controller's rollouts. The bus hands a new consumer the merges it still holds. For each, the holder
finds no delivery for the head, so it makes one from the forge's word. The pull request's statuses at its
merge do not include a passing `mesh/merge-gate` (the check did not exist for most of them), so the table's
*merged unchecked* row holds it.
**What would close them, and why not.**
- `release` walks a held delivery: wrong, these reached the machines already.
- `release` rollouts a held delivery: wrong, these reached the machines already.
- `stop` takes it to `stopped`, which says a person stopped a delivery and sets `mesh/delivery` to error on
the commit. Wrong in meaning, and it would ask the forge for a view on a pull request that is gone.
- Healer H2's `close` reads `held`'s bound, which names no H2 transition, and refuses: "the state is the
@@ -43,6 +43,6 @@ mesh-catalog#121 (the delivery's owner retires history and ends owed work the fo
under ADR 0250.
**Seen live.** `mesh-delivery.deliveries` on 2026-10-09 lists the held history as `retired`, all at
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The walk of
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The rollout of
mesh-catalog#121's own delivery reads `stopped`; the holder that answers today serves the fix all the
same, and `mesh-controller.status` names no machine behind.
04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md
+2 -2
View File
@@ -53,6 +53,6 @@ lift.
every other state in the table moves on its facts?
3. Should `mesh/delivery-group` be required on the trunks, so a group cannot merge before its composed
check passed? To-be 47 leaves this to the operator; this issue is the case that setting decides.
4. The group's order put the catalogue's member first, and the node-engine's member walked at once on the
controller's own path. Is a group's order meant to hold for a member whose walk never waits for
4. The group's order put the catalogue's member first, and the node-engine's member rolled out at once on the
controller's own path. Is a group's order meant to hold for a member whose rollout never waits for
mesh-delivery?
04-ISSUES/316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/01-diagnosis.md
+6 -6
View File
@@ -3,7 +3,7 @@
**2026-10-08.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`),
from `mesh-delivery.show` of both members, and from the holder's journal on the control node.
**Where the hold is set.** `groupTurns` (holder.go) walks a closed group's members in order. For a member
**Where the hold is set.** `groupTurns` (holder.go) rolls out a closed group's members in order. For a member
in `published` it computes `composed`: the group's check exists, its verdict is `pass` or `warning`, and it
names the members' current heads. When `composed` is false it writes the member's `HeldWhy` ("its group's
composed check did not pass for the heads that merged") and settles it, and the table's `hold` row takes it
@@ -20,7 +20,7 @@ to `held`. `composed` is false in three different cases, and the code does not t
look at the members.
- `groupTurns` skips a member in `held` (`default: return nil`), so the tick after the verdict sees the
passing check and does nothing with it.
- The table has two rows out of `held`: `go` (the walk started on a word that was not this owner's) and
- The table has two rows out of `held`: `go` (the rollout started on a word that was not this owner's) and
`release` (a person's act). There is no observed row that takes a held delivery back when the reason it
was held is gone, and `HeldWhy` is never cleared once set.
@@ -41,7 +41,7 @@ check was asked. To-be 47 lists whether to require it as not decided.
- A lost verdict. The holder logged it at 10:10:21 and the group's record holds it.
- The order. The group's order was asked and kept before the check; it plays no part in the hold.
**Not this issue, noted.** The node-engine's member walked at once (`published` → `delivering` at the
**Not this issue, noted.** The node-engine's member rolled out at once (`published` → `delivering` at the
same instant), because the controller's own path does not wait for mesh-delivery (ADR 0239 decision 8).
The group's order put it second. Open question 4 in the report asks whether that is intended.
@@ -57,9 +57,9 @@ ADR 0270
its group's composed check is asked for the current heads and has no verdict.
- **A hold whose reason is gone lifts, by observation.** A member held because its group's composed check had not
passed goes back to `published` (the event *lifted*) once the check passes or warns for the heads that merged,
while its own walk is still open. It then takes its turn in order.
- A held member whose own walk was since taken over by a later walk is not lifted, since its walk can no longer
deliver it. It is `covered` when that later walk was delivered and sent its module (issue 319).
while its own rollout is still open. It then takes its turn in order.
- A held member whose own rollout was since taken over by a later rollout is not lifted, since its rollout can no longer
deliver it. It is `covered` when that later rollout was delivered and sent its module (issue 319).
Open question 3, whether `mesh/delivery-group` is required on the trunks, stays the operator's setting. Open
question 4 is not touched.
04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md
+5 -5
View File
@@ -2,7 +2,7 @@
opened: 2026-10-08
---
# 317. A held delivery's module was walked by a later merge's plan
# 317. A held delivery's module was rolled out by a later merge's plan
## Symptom
@@ -22,7 +22,7 @@ took `openrazer` along. `mesh-controller.plans`:
| c78b5fc9 (#124's merge) | 11:07:02 | superseded at tier 0 by the plan at 86077780; docker, gitea, openrazer planned there again |
| 86077780 (#123's merge) | 11:07:03 | tier 0: distribution, docker, gitea and openrazer built from c78b5fc9; openrazer's gate on the laptop judging |
The controller's walk of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
The controller's rollout of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
c78b5fc9, beside `docker` and `node-tools`. From 11:10:58 UTC the laptop's condition for openrazer said
its account was *relogin needed*. That verdict exists only in the build #125 added, so the held change was
running on the laptop while its delivery said it waited for a person.
@@ -41,10 +41,10 @@ gone out. A person who releases it later releases nothing; a person who stops it
## Open questions
1. When a newer merge supersedes an older plan, does a module the older plan's walk was holding for its
1. When a newer merge supersedes an older plan, does a module the older plan's rollout was holding for its
delivery's word come along, or stay held with that delivery?
2. If it comes along, what does the held delivery become: superseded by the newer delivery, with the hold
carried over to it, or something else? Today the table has no row from `held` for a superseded walk.
carried over to it, or something else? Today the table has no row from `held` for a superseded rollout.
3. A trunk commit holds every earlier commit, so a later build of a module carries the held change in
any case. Is a hold on one delivery meaningful once a later merge of the same repository is walked,
any case. Is a hold on one delivery meaningful once a later merge of the same repository is rolled out,
or does a hold have to hold the module on that trunk until it is released?
04-ISSUES/317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/01-diagnosis.md
+9 -9
View File
@@ -3,7 +3,7 @@
**2026-10-08.** Read from `mesh-controller.plans` (the list, and each plan by id), `mesh-delivery.show`
and `deliveries`, and the code on the main branches of the controller and the catalogue.
**The held delivery's walk.** #125's merge made the plan at a6385479. Its walk waited for mesh-delivery's
**The held delivery's rollout.** #125's merge made the plan at a6385479. Its rollout waited for mesh-delivery's
word (`waited_for: mesh-delivery`, `waits: true`) and was never let go, because its delivery was held. Its
one module, `openrazer`, was "not yet asked": nothing was built.
@@ -11,32 +11,32 @@ one module, `openrazer`, was "not yet asked": nothing was built.
plan. For every open plan of the same repository and branch made before it, it takes each module the
older plan had not built (not asked, asked and not answered, or built and not yet sent), adds it to the
newer plan, and closes the older plan as superseded with the note "… planned there again". It does not
ask why the older plan had not built them. A walk waiting for a held delivery's word is open, so it is
ask why the older plan had not built them. A rollout waiting for a held delivery's word is open, so it is
superseded like any slow plan. This happened three times over (3f552274, c78b5fc9, 86077780), each newer
plan folding in what the one before it held.
The newest plan's own delivery (#123) was not held, so mesh-delivery let its walk go, and `openrazer` was
built from c78b5fc9. The controller's walk of the builds waiting for a gate then carried it to the laptop,
The newest plan's own delivery (#123) was not held, so mesh-delivery let its rollout go, and `openrazer` was
built from c78b5fc9. The controller's rollout of the builds waiting for a gate then carried it to the laptop,
since a send carries every move waiting on its machine (ADR 0236).
**Why the held delivery did not notice.** mesh-delivery's table has `superseded` rows only from
`published` and `delivering` ("a newer delivery to the same trunk took over its walk"). From `held` there
are only `go` and `release`. When the walk record said `superseded`, no row applied, so the delivery stayed
`held` with `held_why` unchanged, and its walk shows "superseded … openrazer planned there again". Two of
are only `go` and `release`. When the rollout record said `superseded`, no row applied, so the delivery stayed
`held` with `held_why` unchanged, and its rollout shows "superseded … openrazer planned there again". Two of
the three later deliveries (#122, #124) went to `superseded` correctly, through the `published` row.
**What the design says.** ADR 0218 decision 3 makes a newer merge take over an older plan's unfinished
work, and was written before deliveries could be held. ADR 0239 makes a held delivery wait for a person,
and to-be 47 says that nothing of a published delivery is registered before its walk starts, so that a
and to-be 47 says that nothing of a published delivery is registered before its rollout starts, so that a
send cannot carry it early. Neither says what a supersession does to a module a hold is keeping back.
**Ruled out.**
- A person's act. #125's transitions end at `hold`, with no `release`, and its own walk was superseded
- A person's act. #125's transitions end at `hold`, with no `release`, and its own rollout was superseded
while still waiting (`let_go` empty), so no `plans go` started it.
- Issue 295's path, a recorded build carried by a send: here the build was asked by a plan, not left
recorded.
**Located** in the controller's `supersededBy`, which folds a held walk's modules into a newer plan, and in
**Located** in the controller's `supersededBy`, which folds a held rollout's modules into a newer plan, and in
mesh-delivery's table, which has no way out of `held` when that happens, so the delivery's state stops
being true.
04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md
+8 -8
View File
@@ -2,11 +2,11 @@
opened: 2026-10-08
---
# 318. One module's wait for a new login held every other module's walk
# 318. One module's wait for a new login held every other module's rollout
## Symptom
On 2026-10-08 at 11:08 UTC the controller opened its walk of the builds waiting for a gate: the laptop
On 2026-10-08 at 11:08 UTC the controller opened its rollout of the builds waiting for a gate: the laptop
first, then the workstation, then the control node. On the laptop it sent three moves in one send and
judged them together:
@@ -23,14 +23,14 @@ account was *relogin needed* from 11:10:58 UTC, which
ADR 0252
expects until the operator logs out and in again.
Meanwhile three other walks could not send to the rest of the mesh. `mesh-controller.plans` said of the
Meanwhile three other rollouts could not send to the rest of the mesh. `mesh-controller.plans` said of the
tool runner's (`novox/mesh-tools` d4845f7c, `node-tools`), the catalogue's (86077780, `docker`) and the
controller's (950562af, `build-agent`): "a build waits there for its gate: … the workstation (…, openrazer
88135ad0 → c78b5fc9) — a walk sends them, one machine at a time, each judged".
88135ad0 → c78b5fc9) — a rollout sends them, one machine at a time, each judged".
At 11:21:06 UTC the laptop's gate failed: "not healthy within 10m0s of its apply: its unit
openrazer.daemon … is unhealthy", and `openrazer` was put back to 88135ad0 on the laptop
(`build.openrazer.<laptop>.rolled-back`). The walk ended `failed`. After it, all three walks still said
(`build.openrazer.<laptop>.rolled-back`). The rollout ended `failed`. After it, all three rollouts still said
the same: `openrazer` c78b5fc9 waits on the workstation for a gate, so nothing is sent there or to the
machines after it.
@@ -40,14 +40,14 @@ The wait was known, and it was a person's: ADR 0252 says a new login holds a del
that machine. It held every module on every machine after it instead, for changes that had nothing to do
with it: the tool runner, the container engine and the build agent. The end was worse than the wait. The
gate's bound turned "waiting for a person" into "failed", the build that added the account group was put
back, and the walk that carries the other modules failed with it. A failed walk of the waiting builds
back, and the rollout that carries the other modules failed with it. A failed rollout of the waiting builds
does not open again on its own: a person releases it.
Three things meet here, each sound alone:
- a send carries every move waiting on its machine, and the gate judges them all at once
(ADR 0236);
- no other send may go where a move waits for a gate, so every walk waits for the one that judges it;
- no other send may go where a move waits for a gate, so every rollout waits for the one that judges it;
- a module's health that waits for a person reads as *not yet healthy*, which fails at the gate's bound.
## Open questions
@@ -61,5 +61,5 @@ Three things meet here, each sound alone:
back the account group it added, so the login the gate waited for can no longer help.
4. The daemon's unit had failed since 10:13 UTC, almost an hour before the new build reached the laptop.
Should a gate fail a build for a failure that was there before it was sent?
5. While a move waits for a gate on one machine, should walks of unrelated modules still be refused there,
5. While a move waits for a gate on one machine, should rollouts of unrelated modules still be refused there,
or only those whose send would carry it?
04-ISSUES/318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/01-diagnosis.md
+11 -11
View File
@@ -1,6 +1,6 @@
# 318 — diagnosis
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the walk of the waiting builds by id) between
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the rollout of the waiting builds by id) between
11:17 and 11:22 UTC, `mesh-controller.conditions` and `status`, and the controller's code on its main
branch (`cmd/mesh-controller`).
@@ -9,15 +9,15 @@ gate all of them as carried moves. `judgeMoves` (gate.go) reads each module on e
worst reading. Any module *not yet healthy* resets the passes to zero for the whole send: `docker` and
`node-tools` were healthy, and the gate still said "0 of 3 passes" because `openrazer` was not. Past
`gateBound` (ten minutes after the apply) a *not yet healthy* gate fails. `g.Failing` names only the
modules found wanting, so only `openrazer` was put back, but the walk itself ended `failed`.
modules found wanting, so only `openrazer` was put back, but the rollout itself ended `failed`.
**Everything else waits for that verdict.** `ungatedIn` (release.go) refuses any send that is not a person's
and would carry a move no gate has seen, outside the send's own scope. A plan's sends to the rest of its
machines carry the machine's whole declaration, so while `openrazer` c78b5fc9 waited for a gate on the
workstation, the tool runner's, the catalogue's and the controller's walks were each refused there, and
they say so on every retry. They do not fail; they wait for the walk of the waiting builds, which goes one
workstation, the tool runner's, the catalogue's and the controller's rollouts were each refused there, and
they say so on every retry. They do not fail; they wait for the rollout of the waiting builds, which goes one
machine at a time and had stopped on the laptop. After it failed, the release code's rule applies: a
failed walk of the waiting builds stops the next one opening on its own until a person releases it.
failed rollout of the waiting builds stops the next one opening on its own until a person releases it.
**How *relogin needed* reached the gate.** ADR 0252 decision 5 says the node-engine states the account's
health as the module's resource of kind `account`, unhealthy with *relogin needed*, and that "the
@@ -33,8 +33,8 @@ running session lacks the group the daemon checks for. That is a guess, not chec
readings existed before or because of the person's step, and neither is a fault the build brought.
**What the design says.** ADR 0252's consequences say "a new login holds a delivery of that module on that
machine" and accept that a walk that starts on a workstation waits for the person. They do not consider the
other modules carried in the same send, the walks refused behind it, or the gate's ten-minute bound turning
machine" and accept that a rollout that starts on a workstation waits for the person. They do not consider the
other modules carried in the same send, the rollouts refused behind it, or the gate's ten-minute bound turning
the wait into a rollback. ADR 0236 makes one send judge everything waiting on a machine, to stop unjudged
builds from moving; it does not distinguish a module waiting for a person from one that is broken.
@@ -50,15 +50,15 @@ Whether the account actually left the group was not checked.
- A machine-wide fault on the laptop: its node-engine reported `applied` and current, and the gate's
wanting named one module's unit.
**Located** in the controller's walk and first-node gate: one verdict per send (`judgeMoves`), sends refused
**Located** in the controller's rollout and first-node gate: one verdict per send (`judgeMoves`), sends refused
on any machine where a move waits (`ungatedIn`), and no reading for a health that waits for a person, so a
known wait fails at the bound and stops every walk behind it.
known wait fails at the bound and stops every rollout behind it.
**Decided and fixed, 2026-10-08**, by ADR 0254.
Open questions 1 to 3: *relogin needed*, and a unit failing in that same account's own manager, read *waits
for a person*, which passes with the wait carried and is said as the module's condition `relogin-needed`;
nothing is put back for it. Question 2's other half: each module of a send counts its own passes, and, after
the settle time, in the walk path, keeps a pass when another fails beside it. Question 5: other walks are
the settle time, in the rollout path, keeps a pass when another fails beside it. Question 5: other rollouts are
still refused while a move waits for a gate (ADR 0236 §4a stands). With the wait a pass, 318's case leaves
nothing waiting for them. A fault decided before the settle time, and a plan's own tier path, still leave
the healthy modules of a failed send without a verdict (ADR 0254, consequences). Question 4, a unit that
@@ -71,4 +71,4 @@ between 12:41 and 13:03 UTC; the node-engine's and the controller's deliveries r
**Seen live.** On 2026-10-09 `mesh-controller.conditions` holds `module.openrazer.<machine>.relogin-needed`
for the laptop and the workstation, raised at 21:02 UTC: "Its update is in place and nothing was undone",
and no walk waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
and no rollout waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md
+7 -7
View File
@@ -15,8 +15,8 @@ What they are, read with `mesh-delivery.deliveries` and `show`:
| how many | repository | why held | what it builds |
|---|---|---|---|
| 4 | hq | merged without a passing check | nothing: "the change touches no module of the mesh's graph" |
| 2 | catalogue | no walk opened within ten minutes of the merge | a new module, sent nowhere |
| 8 | controller | no walk opened within ten minutes of the merge | the controller, the build agent, the route proxy |
| 2 | catalogue | no rollout opened within ten minutes of the merge | a new module, sent nowhere |
| 8 | controller | no rollout opened within ten minutes of the merge | the controller, the build agent, the route proxy |
| 2 | catalogue | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | one module each |
Two examples:
@@ -26,7 +26,7 @@ Two examples:
`proposed` to `held`: "merged without a passing check". Its plan builds nothing.
- `novox/mesh-controller@ec3769a8a669`, controller #105. Ready at 2026-10-07 00:26 UTC, merged at 00:27,
held at 00:37: "merged, and the controller opened no walk for it within 10m0s". The controller's modules
have been walked since from later commits of the same main, for example the group
have been rolled out since from later commits of the same main, for example the group
`feat/delivery-checks-verb`, whose controller member is `delivered`.
Five of them (the four hq merges and controller #105) are past the held bound of one day. The self-check's
@@ -39,7 +39,7 @@ A held delivery asks a person "does this go on?". For these sixteen the question
a later commit of the same trunk has already delivered what they held, or they hold nothing to deliver. The
state table offers a person four ways out of `held`, and none fits:
- `release` starts a walk of the old commit: wrong for a commit a later one superseded, and nothing for one
- `release` starts a rollout of the old commit: wrong for a commit a later one superseded, and nothing for one
that builds nothing.
- `retire-history` is for history made from the forge's word (ADR 0250),
and refuses every one of these because their heads were heard.
@@ -57,7 +57,7 @@ The issue states the need; the decision is for graduation.
- **An end for a held delivery that was superseded**, as `published` and `delivering` already have
(`superseded`, "a newer delivery to the same trunk took over its walk"): either because a later commit's
delivered walk carried the same modules on the same trunk, or because it builds nothing at all.
delivered rollout carried the same modules on the same trunk, or because it builds nothing at all.
- **Automatic where nothing is lost**: a held delivery whose plan builds nothing has nothing for a person to
decide.
- **A person's act, with why, otherwise**: a held delivery whose modules a later commit delivered is ended
@@ -70,5 +70,5 @@ The issue states the need; the decision is for graduation.
2. The four hq merges were heard at the owner's first start from the bus's history, as announcements rather
than merges. Are they history in the sense of ADR 0250, missed by its rule because their head was
replayed too?
3. Why did the controller open no walk for eight of its own merges within ten minutes? The controller's own
path never waits, so its walk may have been missed rather than absent. Not diagnosed here.
3. Why did the controller open no rollout for eight of its own merges within ten minutes? The controller's own
path never waits, so its rollout may have been missed rather than absent. Not diagnosed here.
04-ISSUES/319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/01-diagnosis.md
+30 -30
View File
@@ -7,14 +7,14 @@
| event | kind | needs |
|---|---|---|
| `go` | observed | its walk started on a word not this owner's (the controller's own path, or a person's `plans go`) |
| `release` | a person's act | why and who; it walks the delivery's own commit |
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no walk open |
| `go` | observed | its rollout started on a word not this owner's (the controller's own path, or a person's `plans go`) |
| `release` | a person's act | why and who; it rolls out the delivery's own commit |
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no rollout open |
| `stop` | a person's act | why and who; from any state not final |
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the walk's
record saying `superseded`. A held delivery whose walk never opened has no walk record to read, and one
whose walk was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the rollout's
record saying `superseded`. A held delivery whose rollout never opened has no rollout record to read, and one
whose rollout was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
does not match it either.
**Why `retire-history` refuses them.** `isHistory` requires `FromForgesWord()`: the delivery was made from
@@ -41,7 +41,7 @@ Neither considers a held delivery whose content a later commit delivered.
**Ruled out.**
- A missing release. Releasing any of these would walk an old commit over a newer one, or walk nothing.
- A missing release. Releasing any of these would roll out an old commit over a newer one, or roll out nothing.
- D14 itself. It reports the state truthfully; the state has no way to change.
**Located** in mesh-delivery's state table (no end for a superseded or empty held delivery, and history
@@ -52,14 +52,14 @@ read by how a delivery was made) and in to-be 47, whose state table carries the
**Eleven held deliveries, read again.** On 2026-10-09 eleven deliveries were `held`, each raising D14's
`delivery.<id>.stalled` and held for 24 to 48 hours. For each, `mesh-delivery.show` gave its state, why it was
held, its merge commit and its plan. A fresh clone of the repository then showed whether the merge commit is an
ancestor of the commit a finished walk delivered. `mesh-controller.status` listed none of the modules involved
ancestor of the commit a finished rollout delivered. `mesh-controller.status` listed none of the modules involved
as behind. All eleven were covered: none held a change missing where it should run.
**Why the controller opened no walk (open question 3).** This part comes from reading the code, and part of it
**Why the controller opened no rollout (open question 3).** This part comes from reading the code, and part of it
is still a guess.
- *A new module.* When a merge only adds a module, the controller's `SourceMoved`
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no walk ("built,
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no rollout ("built,
registered when the build lands, and sent nowhere", issue 300). mesh-delivery's `MovesNothing()`
(`model.go`) counts `Plan.New` as something to move, though. So a delivery whose plan is "builds new: …, sent
nowhere" is not let through to `published` → `delivered`. Instead the unwalked rule in `holder.go` holds it
@@ -67,25 +67,25 @@ is still a guess.
read from the code, and it matches `novox/mesh-catalog@60641176abaf`.
- *The controller's own merges (a guess, not proved).* `SourceMoved` also returns without a plan in two cases:
when the modules the merge touches are "already built from it" (`already > 0`), and when the merge is read as
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no walk
of its own when another walk, or a person's push, had already built from a later commit. Which of the two hit
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no rollout
of its own when another rollout, or a person's push, had already built from a later commit. Which of the two hit
these merges is not yet known. The controller's journal for the minutes after each merge would say.
- *The link from a walk to a delivery.* `deliveryWalking()` (`holder.go`) links a walk to the delivery whose
`merged_as` equals the walk's commit, and to no other delivery. A later walk that carried an earlier merge's
change therefore never reaches that earlier delivery, and nothing reads "this walk's commit contains your
- *The link from a rollout to a delivery.* `deliveryWalking()` (`holder.go`) links a rollout to the delivery whose
`merged_as` equals the rollout's commit, and to no other delivery. A later rollout that carried an earlier merge's
change therefore never reaches that earlier delivery, and nothing reads "this rollout's commit contains your
merge". That is the missing input the first open question asks about. mesh-delivery could answer it from the
forge (is the merge commit an ancestor of the delivered walk's commit) together with the walk's moved modules,
forge (is the merge commit an ancestor of the delivered rollout's commit) together with the rollout's moved modules,
without asking the controller what each machine runs.
**Four were closed with `stop`, before this issue was found.** On 2026-10-09 the operator agreed to close the
held deliveries. Four were stopped with why "covered: its merge … is contained in … which is delivered (walk …)":
| delivery | merge | contained in | walk |
| delivery | merge | contained in | rollout |
|---|---|---|---|
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier walk of the controller |
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same walk |
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same walk |
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue walk |
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier rollout of the controller |
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same rollout |
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same rollout |
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue rollout |
Their stalled conditions cleared at the next D14 observation. Open D14 conditions went from 11 to 7. As this
issue already said, `stop` is the wrong record for them: their merged pull requests now carry `mesh/delivery`
@@ -101,12 +101,12 @@ remaining seven were left `held` for this issue's fix.
| `novox/hq@2221be11a6ef` | merged without a passing check | merge on hq's main; builds nothing | the same |
| `novox/hq@897bcce50252` | merged without a passing check | merge on hq's main; builds nothing | the same |
| `novox/hq@ef6ab814b053` | merged without a passing check | merge on hq's main; builds nothing | the same |
| `novox/mesh-controller@b98fd0f396c8` | no walk opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
| `novox/mesh-controller@ea92af2a7d77` | no walk opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release walk that is done; the step looks stale but is not final |
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished walk of the same module | its group is still `delivering`, and its own walk is `superseded` |
| `novox/mesh-controller@b98fd0f396c8` | no rollout opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
| `novox/mesh-controller@ea92af2a7d77` | no rollout opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release rollout that is done; the step looks stale but is not final |
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished rollout of the same module | its group is still `delivering`, and its own rollout is `superseded` |
The seven show the two ends this issue asks for: four hold nothing to deliver, and three are covered by a
later walk.
later rollout.
## 2026-10-09 — the fix, proposed
@@ -115,12 +115,12 @@ ADR 0270
mesh-delivery:
- **A merge that moves nothing is delivered at once.** `MovesNothing()` no longer counts `Plan.New`: a module
the controller builds and sends nowhere opens no walk, so there is nothing to wait for. A merge whose plan moves
the controller builds and sends nowhere opens no rollout, so there is nothing to wait for. A merge whose plan moves
nothing goes from `proposed`, `checked`, `rejected` or `held` to `delivered`, before the row *merged unchecked
→ held*.
- **`covered`, a new final state: a later merge that contains it was delivered.** For every module its plan
moves, a delivered delivery of the same repository and trunk, merged after it, delivered on its own passing
check by the walk of its own merge commit, whose machine steps say the module reached every machine running
check by the rollout of its own merge commit, whose machine steps say the module reached every machine running
it. The forge then says, through a new read-only tool `gitea_contains` (its compare of the two commits),
whether the later merge commit contains this one; merge order is only the first filter. A forge that cannot
answer leaves it held. `mesh/delivery` says `covered: a later merge that contains it was delivered: …` as a
@@ -128,7 +128,7 @@ mesh-delivery:
- **Read on every tick**, so what is held now ends without an event.
This answers open question 1: mesh-delivery reads it from its own records and asks the forge for ancestry; the
controller is not asked. Open question 3, why the controller's own merges opened no walk, is not answered by this fix; it no
controller is not asked. Open question 3, why the controller's own merges opened no rollout, is not answered by this fix; it no
longer leaves anything held.
**How to verify after mesh-delivery is delivered with it.** Within a minute of its start:
@@ -137,13 +137,13 @@ longer leaves anything held.
merges (`builds new: …, sent nowhere`), and none of the five controller merges held for "no walk opened".
The hq and module-adding ones are `delivered`; the controller ones are `covered`.
2. `mesh-delivery.show` of one of them, for example `novox/mesh-controller@b98fd0f396c8`, ends with a
transition `held → covered (covered)` whose why names the later delivery and its walk.
transition `held → covered (covered)` whose why names the later delivery and its rollout.
3. `mesh-delivery.checks` for its pull request shows `mesh/delivery` as `success covered: …`.
4. The forge module is delivered in the same pull request. Until its `gitea_contains` answers, nothing is
covered, and the held controller merges stay held.
5. At the self-check's next reading, `mesh-controller.conditions` no longer lists their
`delivery.<id>.stalled`.
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered walk sent
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered rollout sent
their module. One that stays held is a real question: its change may not run anywhere.
**Accepted 2026-10-09.** The operator accepted the record named above; it is no longer proposed.
04-ISSUES/320-the-operator-could-not-read-the-meshs-notifications/00-report.md
+2 -2
View File
@@ -52,7 +52,7 @@ not addresses, paths or secrets — so they reached the operator untouched.
**Nothing said whether the operator was needed, or let them answer.** No field of a condition said whether
the operator was needed, so every warning was shown, and an explanation could imply an instruction. The
verbs that answer the common cases existed (release or stop a delivery, start a waiting walk, restart a
verbs that answer the common cases existed (release or stop a delivery, start a waiting rollout, restart a
service, silence a condition), but the desk could not take an answer: `node-notifier.send` had no actions,
and the conversation design that adds them (to-be 46) was not built. The one way to act was a session.
@@ -70,7 +70,7 @@ nothing is not sent. The desk offers the actions, and the operator channel's hol
through its seat verb. Every channel says the time in the operator's zone; a clearance is one line.
- novox/mesh-controller: the condition's three new fields, the plain rule, a wording for every kind the
controller raises, the richer words at the producers that know more (a walk's modules, a unit's name, a
controller raises, the richer words at the producers that know more (a rollout's modules, a unit's name, a
hand-act's cause, a delivery's repository), and a lint over every condition the suite raises.
- novox/mesh-catalog: the messenger shows the words, keeps quiet what needs nothing, offers and performs
the answers, says since when in the operator's zone (the new setting `time-zone`), says a clearance in
04-ISSUES/325-assign-said-no-such-module-while-its-build-ran/00-report.md
+1 -1
View File
@@ -88,5 +88,5 @@ To-be 45 §2 is amended: what `status` lists, and the new condition.
## Resolved, 2026-10-09
mesh-controller#150 and mesh-lab#68 (R325) merged on 2026-10-08 at 14:58 UTC. Every later controller
delivery carries #150; the last, of mesh-controller#170, finished its walk on 2026-10-09 at 00:11 UTC, and
delivery carries #150; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at 00:11 UTC, and
`mesh-controller.status` named no machine behind.
04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/00-report.md
+3 -3
View File
@@ -20,7 +20,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
1.4 a second, or about 43 every 30 seconds.
- Over the server's whole uptime (39 hours 49 minutes at 14:13 UTC) the average is 39.7 writes a second.
In that window one walk was in progress, so a few of those writes were real changes. The diagnosis
In that window one rollout was in progress, so a few of those writes were real changes. The diagnosis
shows why most were not.
## Why it is an issue
@@ -40,7 +40,7 @@ watcher, and the write rate cannot be used to see that deliveries are moving.
## How it is checked
- **In mesh-delivery's tests:** a reconcile pass over walks that did not change performs no write. The
- **In mesh-delivery's tests:** a reconcile pass over rollouts that did not change performs no write. The
test's fake store counts writes, and the test fails on any write.
- **Live:** over ten minutes in which no delivery changes state, the write count on
`$KV.mesh-delivery_deliveries.>` (`nats.nats_subscriptions`) rises by no more than the number of
@@ -56,6 +56,6 @@ build agent. So it resolves without a replay. The test above is still the proof.
- `keep` writes only when the delivery differs from the value it last wrote. Either compare the encoded
value with the last one written, or set a changed flag wherever the delivery is changed.
- `Reconcile` skips a walk whose revision the delivery already holds.
- `Reconcile` skips a rollout whose revision the delivery already holds.
- A retry of an owed effect that fails again changes only its try count and next time. Whether that
needs a write at every try is a choice for the fix: the count may stay in memory between writes.
04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/01-diagnosis.md
+8 -8
View File
@@ -12,16 +12,16 @@ runtime. `keep` does not compare the delivery with what it wrote before. Every c
of transitions is empty.
- `settleAndKeep` calls `settle` and passes the result to `moved`. A delivery with nothing to move is
still written.
- `walkMoved` takes one walk from the controller, records it on its delivery (`linkWalk`) and calls
`settleAndKeep`. A walk that has not changed since the last pass leaves the delivery as it was, and
- `walkMoved` takes one rollout from the controller, records it on its delivery (`linkWalk`) and calls
`settleAndKeep`. A rollout that has not changed since the last pass leaves the delivery as it was, and
the delivery is written anyway.
- `Reconcile` asks the controller's `delivery-walks` for the walks it keeps and calls `walkMoved` for each.
The answer holds every open walk and the last 50 that ended. The ticker in main.go runs `Reconcile`
- `Reconcile` asks the controller's `delivery-walks` for the rollouts it keeps and calls `walkMoved` for each.
The answer holds every open rollout and the last 50 that ended. The ticker in main.go runs `Reconcile`
every 30 seconds.
So each pass writes about one delivery per walk the controller returns. On 2026-10-08 that was 50 walks
(32 done, 14 superseded, 3 failed, 1 walking), against about 43 writes per pass measured. *I think* the
difference is walks that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
So each pass writes about one delivery per rollout the controller returns. On 2026-10-08 that was 50 rollouts
(32 done, 14 superseded, 3 failed, 1 `walking`), against about 43 writes per pass measured. *I think* the
difference is rollouts that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
**What made it 40 a second before today.** Before mesh-catalog #121 (the fix of
[issue 313](../313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md), merged
@@ -39,5 +39,5 @@ doubling to an hour) and ends them after 60 tries. A retry still writes its deli
- *The groups bucket.* `mesh-delivery_groups` has 26 keys, and its subscription counted 719 writes in the
same 39 hours.
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every walk the
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every rollout the
controller keeps through `settleAndKeep` every 30 seconds.
04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/00-report.md
+1 -1
View File
@@ -24,7 +24,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
The long-lived connections were young too. At 14:15 UTC the controller's two were 4 minutes old, the
build agents' 2 to 19 minutes, the route proxies' 13 to 17 minutes and the tool runners' 10 minutes to 3
hours. Two walks were sending new builds of the controller, the build agent and catalogue modules
hours. Two rollouts were sending new builds of the controller, the build agent and catalogue modules
between 14:00 and 14:20 UTC, and the build agent's container on the control node started at 14:19 UTC.
*I think* those young connections are restarts by deliveries; they are tens, not thousands.
04-ISSUES/327-every-controller-verb-opens-its-own-bus-connection-so-the-connection-count-hides-a-reconnect-loop/01-diagnosis.md
+2 -2
View File
@@ -17,7 +17,7 @@ controller's, so each place that needs the bus opens one, uses it and closes it:
command.
- `sayingOnTheBus` (delivery.go): a delivery's `check`, `go` and `stop`, which mesh-delivery asks for.
The serving controller dials too. Its watchdogs run every 30 seconds, and while a walk waits for a
The serving controller dials too. Its watchdogs run every 30 seconds, and while a rollout waits for a
build, `buildSeatPause` (plan_retry.go) opens a fresh connection to read whether the build seat's
holders paused, and closes it. `status` does the same.
@@ -41,7 +41,7 @@ its long-lived connections are named `mesh-controller` too and are told apart on
minute. The research's guess came from their age alone.
**Located** in the controller: one connection per command process that touches the bus, and one per
watchdog tick while a walk waits for a build.
watchdog tick while a rollout waits for a build.
## 2026-10-09, resolved
04-ISSUES/328-jetstream-api-errors-are-counted-and-attributed-to-nothing/01-diagnosis.md
+1 -1
View File
@@ -17,7 +17,7 @@ the fix is the attribution tool; with it, this diagnosis continues.
- **A holder that never said it paused.** `PausedSaid` (controller, internal/link/queue.go) asks for the
last message on each build holder's pause subject. A holder that never said one gets a not-found
answer, which the server counts as an API error. The serving controller's watchdog reads it every 30
seconds while a walk waits for a build (issue 327).
seconds while a rollout waits for a build (issue 327).
- **Short-lived consumers.** The controller heard 495 "consumer deleted" advisories in the first 8
minutes of its current connection, about one a second. They are short-lived consumers: the controller's
advisory reader (internal/link/advisories.go) ignores them and says why, "every watch of a bucket and
04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/00-report.md
+7 -7
View File
@@ -7,12 +7,12 @@ opened: 2026-10-08
## What was observed
On 2026-10-08 a person released a delivery of the catalogue that the group check (issue 316) had held. Its
walk had one tier with one module, `dunst`, which runs on the laptop and the workstation. The walk sent the
rollout had one tier with one module, `dunst`, which runs on the laptop and the workstation. The rollout sent the
new build to the laptop first, and the gate passed it there:
> dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent
Twenty-eight minutes later the same walk said the opposite, about the same gate:
Twenty-eight minutes later the same rollout said the opposite, about the same gate:
> dunst failed its gate on the laptop in tier 0: healthy 3 times over 2m5s; put back dunst to ea462666 on
> the laptop, in one send; the workstation left as it was
@@ -31,22 +31,22 @@ build that worked, recorded it as failed, and so blocked its delivery until a ne
## Fix
The controller judged the first machine's report again on every step of the walk, also after the gate had
The controller judged the first machine's report again on every step of the rollout, also after the gate had
passed (see `01-diagnosis.md`). The fix merged in mesh-controller pull request 165 (8170fc5), with its
replay in mesh-lab pull request 72.
- Once a module's gate has passed, the first machine's later reports no longer decide anything about it:
the walk sends the rest.
the rollout sends the rest.
- A module carried by another module's send takes over that pass before its step is read.
- A guard, not reached with the two changes above: should a later path still stop a walk at a module whose
gate passed, that module is not put back and its build is not marked failed. The walk stops with the
- A guard, not reached with the two changes above: should a later path still stop a rollout at a module whose
gate passed, that module is not put back and its build is not marked failed. The rollout stops with the
reason, and the module is left sent to its first node only, which `plans retry` takes up again.
## How it is checked
`TestReplay335` in the controller replays the live case with only what the controller had before the fix:
a passed gate on the laptop, the laptop's later report not about this build (or a failure of another
send), and the walk 30 minutes and 9 seconds after the first send. A second test drives the walk over a
send), and the rollout 30 minutes and 9 seconds after the first send. A second test drives the rollout over a
tier of two modules, the second carried by the first's send, and fails if the carried module's step is read
before it takes over the pass. On the commit before the fix it fails
with "laptop did not report it applied within 30m0s", and the passed build reaches the put-back. It is R335
04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/01-diagnosis.md
+8 -8
View File
@@ -6,15 +6,15 @@
found ADR 0236 (what a rollback is), issue 318 (a wait for a login failing a gate) and issue 279, none of
them about a gate that passed and was then failed.
**The walk's own lines.** The controller's journal on the control-node said, for this walk (times UTC):
**The rollout's own lines.** The controller's journal on the control-node said, for this rollout (times UTC):
| time | what the controller said |
|---|---|
| 16:44:45 | tier 0 built; sent dunst to the laptop first, the rest once its gate passes |
| 16:44:55 | the laptop applied what it was sent |
| 16:46:52 | dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent |
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another walk sends |
| 16:49:12 | the other walk (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another rollout sends |
| 16:49:12 | the other rollout (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
| 17:14:54 | dunst failed its gate on the laptop: healthy 3 times over 2m5s; put back on the laptop; the workstation left as it was |
The laptop's node-engine logged `applied` at 16:44:55 and nothing more until 18:01:58. It did not report
@@ -25,14 +25,14 @@ on the 16:49:12 send for over an hour. Why it did not is not yet known; the lapt
waits 30 minutes (`planWaitBound`) for a first machine to report what it was sent. The failure came at that
bound and not at the gate's.
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a walk reads
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a rollout reads
`nextRollout(state, …)` before anything else, for every module in the tier not yet sent to the rest. For a
module sent first, `nextRollout` looks at the first machine's last report. If that report is not about the
declaration the machine was last sent (`Current` false), it waits; past the bound it answers
`failed: <machine> did not report it applied within 30m0s`. Nothing in it looked at the gate's verdict.
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the walk tries again on
the next step). At 16:49:12 the other walk's send made the laptop's last report no longer current. From
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the rollout tries again on
the next step). At 16:49:12 the other rollout's send made the laptop's last report no longer current. From
then on `nextRollout` waited for a report it would never get about this build. At the bound it failed, and
`advanceOnce` called `failFirstSend`.
@@ -47,7 +47,7 @@ laptop. That is the line seen at 17:14:54.
- The person-wait of issue 318 (ADR 0254), `putBackBroken` and the carried-module paths: the plan had one
module, nothing was carried, and nothing was found broken.
- Sleep: the laptop did not sleep until 18:01, and the failure was at 17:14:54.
- The release itself (`mesh-delivery.release`): it let the walk go at 16:44, and the walk ran as any other.
- The release itself (`mesh-delivery.release`): it let the rollout go at 16:44, and the rollout ran as any other.
**Located.** mesh-controller, `cmd/mesh-controller/release_plan.go`: `nextRollout` judged a passed gate's
first machine again, and `failFirstSend` put back a module whose gate had passed. The path has been there
@@ -57,4 +57,4 @@ step as the pass and another send reaches the first machine before the 30-minute
**What it needs to deliver dunst.** The build is recorded as failed at its gate, and the controller does
not send a build marked that way again on its own. The registered build is back at the one before. Once the
fix is delivered, a new build of dunst (`rebuild dunst`, or a newer merge of it) is judged at the gate
again and walked to both machines.
again and rolled out to both machines.
04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/00-report.md
+10 -10
View File
@@ -20,10 +20,10 @@ From the merge on, every send to the control-node was held. `mesh-controller.pla
> planned step: sending control-node would replace the bus (nats 88135ad0 → 32307bd1), which is a planned
> step ... — tried again for 27m45s
The same refusal held the controller's own walk and the release of builds kept back for a gate.
The same refusal held the controller's own rollout and the release of builds kept back for a gate.
For those 28 minutes `mesh-controller.conditions` listed **no condition** naming the hold or saying it
waited for a person. The operator was not told. The hold was found only because a walk looked stuck, and
waited for a person. The operator was not told. The hold was found only because a rollout looked stuck, and
the operator asked: "where would you be waiting then? Where is my approve button?"
The operator then gave the word, the bus upgrade started at 18:52 and the bus answered healthy at 18:54.
@@ -33,7 +33,7 @@ The held sends then went through.
The design makes the bus wait for a person, and that part worked: nothing replaced the bus unasked. But a
wait that only a person can end is said nowhere a person looks. The refusal is kept as the note on each
held walk, which is read only by somebody already asking `plans`. Rule 4 of
held rollout, which is read only by somebody already asking `plans`. Rule 4 of
ADR 0227
(no failure is silent) is met in the letter and missed in effect: the mesh held every send to its
most important machine and said nothing until asked.
@@ -47,10 +47,10 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
## Impact
- Every send to the control-node stops: the controller's own walk, the node-engine's, every module's walk
- Every send to the control-node stops: the controller's own rollout, the node-engine's, every module's rollout
that includes the control-node, and gated builds waiting for release. Other machines in the same tier
wait with it.
- The operator learns of it only by noticing a stuck walk, which can take hours when nobody is watching.
- The operator learns of it only by noticing a stuck rollout, which can take hours when nobody is watching.
- The only condition that could have fired, a tier past its bound (watchdog S3, "stalled"), fires no
sooner than 30 minutes in, and its words ("stuck halfway ... longer than usual. Nothing is lost") name
neither the bus nor the verb a person runs. It never fired in this case, because the step was taken
@@ -62,9 +62,9 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
| Rule | Check |
|---|---|
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a walk wants to send raises a condition on the next watchdog tick, before any tier bound |
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held walks or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a walk whose only refusal is the bus's planned step |
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a rollout wants to send raises a condition on the next watchdog tick, before any tier bound |
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held rollouts or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a rollout whose only refusal is the bus's planned step |
| it clears when the step starts | the test ends with a bus step started: the condition clears, `bus-maintenance` opens |
| the merge says it before it happens | the merge gate's test: a change that moves the bus's build reports that a bus upgrade by a person is needed and which machine waits until then |
@@ -81,8 +81,8 @@ why none is possible.
with the operator as its resolver). Its headline says, in plain words, what waits ("sends to the
control-node: the controller, the node-engine, …"), behind what (the bus build, from and to), since
when, and the verb that ends it (`mesh-controller.bus upgrade --why … --reversible|--irreversible`). It
is raised on the first refusal, not after a bound. A walk held only by it is excluded from S3, as a
walk held by a paused build seat is.
is raised on the first refusal, not after a bound. A rollout held only by it is excluded from S3, as a
rollout held by a paused build seat is.
2. **An ask, once asks exist.** When the decision flow of ADR 0259 (an ask is a message on the bus, any
channel answers it; on an open pull request, not yet merged) is built, the condition becomes an ask
to the operator: "start the bus upgrade now?", whose yes runs the step with the operator's word
04-ISSUES/336-a-send-held-for-the-bus-step-raises-nothing/01-diagnosis.md
+8 -8
View File
@@ -13,13 +13,13 @@ The send is refused whole. `busHeld` (`bus_step.go`) gives the words "sending X
(nats from → to), which is a planned step: `bus upgrade --why …` …". So the remedy is in the words, but
only in the error.
**Where the error goes.** In `release_plan.go`, the walk's step loop catches the error, keeps it as the
**Where the error goes.** In `release_plan.go`, the rollout's step loop catches the error, keeps it as the
plan's note ("tier N: … — tried again"), saves the plan when the note changed, prints it to the
controller's log once, and tries again on the next tick. Nothing raises a condition from it. That is why
`plans` showed it and `conditions` did not.
**Which watchdog could have said it.** Only S3 (`watchPlans`, `signals.go`), "plan tier progress", raises
`plan.<id>.stalled` when a walk has been in one tier past its bound. The bound is
`plan.<id>.stalled` when a rollout has been in one tier past its bound. The bound is
`max(30 min, 3 × the p90 of that repository's measured tiers)` (`tierAtLeast = planWaitBound = 30 min`,
`watchdogs.go` `boundsOf`). So:
@@ -29,14 +29,14 @@ controller's log once, and tries again on the next tick. Nothing raises a condit
mesh keeps it where it is."). Its summary carries the plan's note, so the bus's reason would have
appeared there, but the headline a person reads names neither the bus nor the verb, and its resolver is
not the operator;
- a walk held by a paused build seat is excluded from S3 (`pausedWaiting`), and a walk waiting for its
delivery's word has its own watchdog (S16). A walk held for the bus step has neither.
- a rollout held by a paused build seat is excluded from S3 (`pausedWaiting`), and a rollout waiting for its
delivery's word has its own watchdog (S16). A rollout held for the bus step has neither.
**What the bus step does raise.** Probe DB (`probeBusStep`, `bus_step.go`) raises `bus-maintenance` only
while a step runs, and `bus-upgrade-failed` when one ended unhealthy. It reads only the latest bus step.
Before a step is started there is nothing for it to read, so a pending bus build raises nothing.
**Before the merge.** `delivery.go` lists the bus among the modules whose walk starts at the merge
**Before the merge.** `delivery.go` lists the bus among the modules whose rollout starts at the merge
(`onTheControllersPath`), and the merge gate (`merge_gate.go`) composes the machines with the change. Neither
reads `pendingBus` or says that the change will hold the bus's machine until a person runs the step. So a
merge that moves the bus's build passes green and blocks every later send to the control-node.
@@ -64,12 +64,12 @@ person, and the person must be told.
ADR 0271
(proposed) takes suggested fixes 1 and 3. mesh-controller #173 builds them:
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a walk's send was refused
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a rollout's send was refused
for the bus's planned step, while the new bus build would still replace the bus on its machine. It is a
warning, for the operator. It names the machines, each waiting walk and the modules of its tier, the bus build
warning, for the operator. It names the machines, each waiting rollout and the modules of its tier, the bus build
from and to, since when, and the `mesh-controller.bus` call with `upgrade`. It clears once the bus's machine has
been sent the build the mesh holds.
- **S3 leaves out a walk held only by it.**
- **S3 leaves out a rollout held only by it.**
- **The delivery plan says it before the merge**: "merging this needs a person's bus upgrade: … nothing else is
sent to <machine> either", and its summary, which `mesh/merge-gate` carries, says the same.
04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/00-report.md
+3 -3
View File
@@ -20,7 +20,7 @@ The plan rebuilt and moved them whatever the merge changed. Two examples:
- Mesh-controller PR #159, merged as `ca09a07f`, changed files under `cmd/mesh-controller`,
`internal/broker`, `internal/link`, `internal/catalogue/verbs.go` and the root `module.json`. Nothing
under `examples/route-proxy` changed. Its walk said, on the home server, "recreates 1 of route-proxy's
under `examples/route-proxy` changed. Its rollout said, on the home server, "recreates 1 of route-proxy's
3 container(s) (with a new image: server)" and "recreates 1 of build-agent's 1 container(s)".
- Mesh-controller PR #163, merged as `9b028b42`, did the same. The route proxy's program at that commit
is byte for byte the program at the commit before it (measured in the diagnosis), and it was still
@@ -43,8 +43,8 @@ The operator stated the rule the delivery plan should follow:
## Why it is an issue
- **Deliveries are slow.** A controller merge walked for about fifteen minutes instead of about five,
and the walks of one repository must not overlap (issue 335), so every merge after it waits too. The
- **Deliveries are slow.** A controller merge rolled out for about fifteen minutes instead of about five,
and the rollouts of one repository must not overlap (issue 335), so every merge after it waits too. The
operator considers this a major bug and a main reason deliveries are slow.
- **A public service restarts for nothing.** The route proxy on the home server was recreated at
merges that did not change its program.
04-ISSUES/338-a-module-built-from-a-shared-repository-moves-on-every-merge-to-it/01-diagnosis.md
+11 -11
View File
@@ -31,7 +31,7 @@ Read in mesh-controller at `8170fc58`.
4. `mesh-controller` itself is built from the repository's root (its manifest's bundle compiles
`cmd/mesh-controller`; its source path is empty), so every changed file in the repository touches
it as well.
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the walk serial here:
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the rollout serial here:
- a `packages` edge, from a module whose build read a repository to the module whose source that
repository is. So `route-proxy` and `build-agent` both depend on `mesh-controller`, because they
share its repository, not because either is built from the controller's artifacts;
@@ -108,9 +108,9 @@ change. The build source of the route proxy is `examples/route-proxy`, `internal
## Measured: what the extra tiers cost
From `mesh-controller.plans` (read only), the first-node gate of each tier of tonight's controller
walks:
rollouts:
| walk | mesh-controller | build-agent | route-proxy | three gates |
| rollout | mesh-controller | build-agent | route-proxy | three gates |
|---|---|---|---|---|
| `3f68a495` | 3m40s | 3m43s | 3m46s | 11m09s |
| `0cf5a3a5` | 3m39s | 3m26s | 3m34s | 10m39s |
@@ -120,12 +120,12 @@ walks:
| `ca09a07f` | 3m43s | 3m46s | 3m40s | 11m09s |
| `efcdd5dd` | 3m38s | 3m40s | 3m37s | 10m55s |
Two more walks (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still walking at
21:50. The gates alone are 10 to 11 minutes per walk, of which 7 minutes are the two packaging tiers.
Two more rollouts (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still rolling out at
21:50. The gates alone are 10 to 11 minutes per rollout, of which 7 minutes are the two packaging tiers.
Each tier also builds before it sends: the build seat's image took 53 seconds at 21:36. Estimated from
the plan's creation time and the build records, `ca09a07f` walked from 21:07 to about 21:23 and
the plan's creation time and the build records, `ca09a07f` rolled out from 21:07 to about 21:23 and
`efcdd5dd` from 21:32 to about 21:47, about 15 minutes each. The controller's own tier, alone, is about
4 to 5 minutes. Of the seven completed walks, five recreated `route-proxy` on the home server
4 to 5 minutes. Of the seven completed rollouts, five recreated `route-proxy` on the home server
(`3f68a495`, `e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`), and its program had changed in one of
them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`).
@@ -133,7 +133,7 @@ them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07
The deploy plan names a build of a packaging module by the module's own source commit, which is the
catalogue's. `moveFacts.identical` (cmd/mesh-controller/release.go) calls two builds identical when
their commits are equal. In `efcdd5dd`'s walk both modules went `7b307a06 → 7b307a06`, nothing was
their commits are equal. In `efcdd5dd`'s rollout both modules went `7b307a06 → 7b307a06`, nothing was
said to be recreated, and the gate still took about 3.5 minutes for each. Yet the route proxy's
program did change at `efcdd5dd` (table above). **I do not know whether that program reached the home
server.** Either it was sent and the plan said nothing, or it was not sent and the route proxy runs the
@@ -146,7 +146,7 @@ commit would also make issue 280's "repeat" rule register a different program as
- **The changed paths not reaching the planner.** PR #159's delivery record carries its paths and
`module_dirs_said: true`. They are read, and `touchedBy` uses them for the controller's own module.
They are never consulted for a packaging module.
- **The catalogue moving at the same time.** In some walks the catalogue's commit also changed
- **The catalogue moving at the same time.** In some rollouts the catalogue's commit also changed
(`32307bd1 → 7b307a06`), but the README-only what-if shows the controller's repository alone is enough
to plan both modules.
@@ -181,7 +181,7 @@ the commit into the program.
with the build; that is exact for Go and needs a reader per language.
2. **Reproducible builds**: `-buildvcs=false` (or a `.dockerignore` that leaves `.git` out of the
context) in both recipes, so an unchanged source gives an identical program. This alone does not
stop the walk: the image digest still differs (issue 280), and the planner would still plan and gate
stop the rollout: the image digest still differs (issue 280), and the planner would still plan and gate
three tiers. It is cheap and worth doing beside 1, so the program's identity no longer depends on the
commit.
3. **Split the route proxy and the build seat into repositories of their own.** Then the existing
@@ -191,5 +191,5 @@ the commit into the program.
and a change to that library then moves all three again, through a real build edge.
Recommendation: 1 with its check, and 2 beside it. A further question for the fix: whether `built-by`
should order a walk at all. It is a running-order edge for the build seat's worker (issue 206); the
should order a rollout at all. It is a running-order edge for the build seat's worker (issue 206); the
operator's rule 2 counts only modules built *from* a changed module.
04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/00-report.md
+2 -2
View File
@@ -23,7 +23,7 @@ What happened, in order (UTC):
**under the same JSON key `declared`**. The two edits touched different lines, so git merged them
without a conflict. The result does not compile:
`internal/store/store.go:145:2: Declared redeclared`.
4. **00:55.** The walk the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
4. **00:55.** The rollout the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
in tier 0". `mesh/delivery` on the head reads `failure`. Nothing was sent to a node, and every node kept
`d28ca95`.
5. **01:09.** Node-engine #67 repaired the trunk. It renamed #66's field to `DeclaredUnit`, key
@@ -33,7 +33,7 @@ The facts behind the "different lines" claim, read from the node-engine's reposi
contain `d28ca95`. Their common ancestor is `a2c2c35` (#63's merge). `dfb1415` holds one `Declared`
(store.go line 107), `d28ca95` holds the other (line 139), and `690966d` holds both (lines 107 and 145).
No node was harmed, because a walk builds before it sends. But the merge check said "this may merge" of
No node was harmed, because a rollout builds before it sends. But the merge check said "this may merge" of
a merge that did not build, and both required statuses were green when it merged.
## The cost before this: checks that went stale
04-ISSUES/341-a-merge-check-judges-the-head-not-what-merging-it-makes/01-diagnosis.md
+1 -1
View File
@@ -33,7 +33,7 @@ merging made, with two, was never built before the merge.
or ready delivery back to proposed. Its comment says "a person asked for it again, or its group
changed".
- `holder.go`, `PullMerged` (line 347): a merge records the merge commit on that pull request's own
delivery and links its walk. It does not touch the repository's other open deliveries into the same
delivery and links its rollout. It does not touch the repository's other open deliveries into the same
trunk.
- The forge marks a head pending only when it announces a new head (issue 305). A merge into the trunk is
not a new head of any other pull request.
04-ISSUES/344-known-gaps-in-the-agents-boundary-accepted-for-now/00-report.md
+4 -4
View File
@@ -60,14 +60,14 @@ note. Each gap says what is open, why it matters, and what would close it.
9. **On a large data pool the root search may never finish.** The node-engine's search for programs that run as
root now has one bound of fifteen minutes (`mesh-host` #61; the inner two-minute limit is gone). On a machine with
a very large data pool the walk can still run past it, and the verdict then stays "not judged". That is loud,
a very large data pool the traversal can still run past it, and the verdict then stays "not judged". That is loud,
not silent.
*Closes with:* setuid turned off on the data datasets, so the walk skips them.
*Closes with:* setuid turned off on the data datasets, so the traversal skips them.
10. **An agent can merge its own pull request.** No mesh repository's main requires an approval on the forge
(`required_approvals: 0`), and the forge tool merges as the forge's administrator once the required checks are
green. An agent can therefore change a module that runs on every node, open the pull request, wait for the
checks and merge it, and the controller walks the change out. The checks judge whether every machine still
checks and merge it, and the controller rolls the change out. The checks judge whether every machine still
composes, not what the change is for. The operator's yes in a session is a convention, and nothing checks it.
*Closes with:* a merge that needs the operator's own approval: a required approval on the forge from the
operator's personal account only, or an approval through ADR 0259's asks. That holds only when the forge's
@@ -98,7 +98,7 @@ note. Each gap says what is open, why it matters, and what would close it.
14. **Ways to root the node-engine's judge does not see.** ADR 0266 §3 lists them under *not judged, so not seen*:
among them a root-run unit or script the agent account can write, a directory on root's `PATH` it can write,
root's ssh keys readable by it, and a setuid program on a filesystem the search does not walk. None is known
root's ssh keys readable by it, and a setuid program on a filesystem the search does not traverse. None is known
on the control-node, but a healthy verdict says nothing about them. The record left them "for a later record"
with nothing tracking it; added here on 2026-10-10.
*Closes with:* each way judged by the node-engine beside the others, or a record that decides against judging
04-ISSUES/348-a-resolver-that-stopped-answering-ran-on-unnoticed-and-builds-sent-after-it-were-blamed/01-diagnosis.md
+6 -6
View File
@@ -16,14 +16,14 @@ stops answering while it runs.
| 10:57:57 | the route proxy's first "lookup … on <private address>:53: … connection refused" | the route proxy's log |
| 10:58:36 | `machine.<home server>.network` raised: one of its two resolvers does not answer | conditions |
| 10:58:45 | `machine.<control node>.network` raised; the home server's cleared, held under it | conditions |
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the walks |
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the rollouts |
| 10:59:37 | `module.step-ca.<control node>.unhealthy` raised | conditions |
| 10:59:42 | `machine.<control node>.network` **cleared**, its newest evidence (10:59:37) still "connection refused" | conditions |
| 11:00:23 | `machine.<control node>.network` reopened, raised time 11:00:23 | conditions |
| 11:06:00 | `seat.mesh-dns-resolver.<control node>.wrong` raised by D2 | conditions |
| 11:10:29 | both builds failed their gate: "<control node> as a whole: raised since it was sent: machine.<control node>.network" — put back | conditions |
| 11:18:12 | `dnsmasq.service` restarted by hand; every question answered again | the operator |
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release walked the node-engine | conditions |
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release rolled out the node-engine | conditions |
### What is measured, and what is guessed
@@ -54,12 +54,12 @@ answered none of its questions for the next run (`Confirm`, issue 277), because
question can be lost. Its first run after 10:57:57 held it; the next, at 11:06:00, raised it. Every one
of those questions had been refused, which no load explains.
**Measured: why the second walk of the node-engine waited on its own first machine.** A release keeps no
**Measured: why the second rollout of the node-engine waited on its own first machine.** A release keeps no
record per module — its modules are in its tier only — and D10 read only the records to know what a plan
is rolling out. So while a release walked the node-engine at 11:21, D10 raised `core-behind` on every
is rolling out. So while a release rolled out the node-engine at 11:21, D10 raised `core-behind` on every
machine, "and no plan is rolling them out", and the gate on the home server, its first machine, waited
on it (issue 281's settle window) until D10's next run five minutes later found the new engine there.
The walk then passed. It did not fail, but it waited five minutes for a condition its own send caused.
The rollout then passed. It did not fail, but it waited five minutes for a condition its own send caused.
**Not explained: why the second resolver did not save the lookups.** Every machine's resolver file lists
both resolvers (`options timeout:1 attempts:2`), and the control node's own file lists its private address
@@ -153,7 +153,7 @@ registry, is the operator's decision.
- An undecided part of a network statement — `unknown` or `starting`, as a node-engine just started says
— holds the open network conditions whose newest evidence names that part, and only those. A statement
undecided as a whole holds every part. The next statement decides.
- D10 counts what a release's tier names as rolling out, so a walked node-engine is not `core-behind` on
- D10 counts what a release's tier names as rolling out, so a rolled out node-engine is not `core-behind` on
its own first machine.
- D2 is unchanged: a resolver that answered nothing, refused or silent, is held for the next run, because
a few hundred milliseconds of refusals is a resolver restarting as well as one that stopped. D2 says it
04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md
+3 -3
View File
@@ -61,8 +61,8 @@ A replay in mesh-lab's register is still owed before this core issue resolves (A
## A sequel (2026-10-10)
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a walk:
mesh-delivery matches a walk to the delivery merged as its commit, so two catalogue merges 18 seconds apart
left the earlier one held for a person and the walk carrying it waiting for a word nothing gave, for 58
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a rollout:
mesh-delivery matches a rollout to the delivery merged as its commit, so two catalogue merges 18 seconds apart
left the earlier one held for a person and the rollout carrying it waiting for a word nothing gave, for 58
minutes, with no condition raised. See
[issue 362](../362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md).
04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md
+14 -14
View File
@@ -2,7 +2,7 @@
opened: 2026-10-09
---
# 351. A held delivery whose module a backlog walk carried to every machine is never covered
# 351. A held delivery whose module a backlog rollout carried to every machine is never covered
## Symptom
@@ -19,12 +19,12 @@ What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `ha
| time | what happened |
|---|---|
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word |
| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded |
| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours |
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its rollout waited for its word |
| 10-08 11:06 | three later catalogue merges took its rollout over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the rollout ended `failed`, then superseded |
| 10-08 14:27 | #127 merged as bf542128; its rollout built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog rollout sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog rollout, as it had for 24 hours |
No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever
move it again (ADR 0267):
@@ -32,22 +32,22 @@ left as it is, the delivery stays held for ever.
## Why it is an issue
ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running
ADR 0270 covers a held merge when a later delivered merge's rollout "sent the module to every machine running
it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge
commit, which the forge says contains a6385479. The decision's question — does this change go on? — was
answered by the operator's own release of the build. Yet the record cannot end:
- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what
- **A backlog rollout's end is read by nothing.** While it judges, the controller's record of it names what
its gate carried, and mesh-delivery credits the delivery that published each carried build one step per
machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing
from that: the steps stay `judging` for ever.
- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running
it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the
- **`covered` reads only a rollout's own aggregate steps.** A rollout's own record says "every machine running
it" or "the rest running it"; a step per machine, which is all a backlog rollout gives, never counts as the
module sent everywhere.
ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module
whose build reached its machines on the operator's word, after the mesh held it back, rather than by the
walk of the merge that built it. A held delivery in that case has no end but a person's wrong act
rollout of the merge that built it. A held delivery in that case has no end but a person's wrong act
([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and
the condition it raises every hour is one nobody can clear correctly.
@@ -56,7 +56,7 @@ the condition it raises every hour is one nobody can clear correctly.
- A forge that could not answer: the holder's journal has no "not yet covered" line for it, and
`gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128.
- A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps.
- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
- The rollout that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
rightly no coverer.
- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module
- A person's `release` of #125 (the verb): it would roll out a6385479, an older commit than the one the module
was built from, and is what ADR 0270 ruled out.
04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/01-diagnosis.md
+21 -21
View File
@@ -2,56 +2,56 @@
**2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`,
at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127,
`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`,
`mesh-controller.delivery-walks` for #127's rollout and for the backlog rollout, `plans`, `hand-acts`, `module list`,
the forge's `gitea_contains`, and the holder's journal on the control node.
**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a
**Where a backlog rollout's steps come from.** `walkMoved` (holder.go) hands a rollout whose record carries a
`release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives
the delivery whose merge commit each carried build was made from one step per machine: `judging` while the
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog rollout
holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`.
`creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43
and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the
module was not in that walk's tiers, and the backlog walk is a walk of its own.)
and 14:46 UTC were never moved on. (The controller's own record of #127's rollout never named the module: the
module was not in that rollout's tiers, and the backlog rollout is a rollout of its own.)
**Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a
later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only
on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running
on a `sent` step whose machine is the rollout's own aggregate, "every machine running it" or "the rest running
it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is
found, and the forge is never asked — which is why the journal has no line.
**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a
backlog walk that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
**What "done on a machine" means for a backlog rollout.** Read from the controller's records, not its code: a
backlog rollout that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
(seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done,
with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of
the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis
is wrong there.
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb —
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed rollout, and the `release` verb —
see the report.
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog rollout's `done` machines for
nothing, and `carries`, which counts no per-machine step.
## The fix
In mesh-catalog #163, one pull request, in `modules/mesh-delivery`:
- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that
walk (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
step it failed on, and a step another walk carried, are left as they are.
- **A finished backlog rollout passes the steps it carried.** `creditReleased`: every step credited to that
rollout (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
step it failed on, and a step another rollout carried, are left as they are.
- **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when
the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
the rollout's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still
judging, one failed or one never sent leave the delivery held.
- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its
open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was
- **A rollout that carried a step still judging is asked for by its id.** The controller lists back only its
open rollouts and the last fifty ended at a start; the backlog rollout of this case, ended the day before, was
no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for
such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
such a rollout by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
its record as said.
Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own
walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
rollout of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
The record's consequence that named the one-machine gap gains a progressive insight naming this one.
**How it is checked**, in `covered_test.go`:
@@ -59,15 +59,15 @@ The record's consequence that named the one-machine gap gains a progressive insi
- `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered;
still judging on one, failed on one, never sent to one, or a plan naming no machines stays held.
- `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one
and leaves the other judging; a failed step stays failed; a step another walk carried is not touched.
and leaves the other judging; a failed step stays failed; a step another rollout carried is not touched.
- `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held
while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the
while the backlog rollout judges, covered once it is done and read by its id, naming the later merge and the
module.
All three fail on the catalogue's main before the change and pass after it.
**After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are
asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once
asked for by the backlog rollout's id and pass; #127 then carries the module for #125; the forge is asked once
whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads
delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the
acceptance of this issue, read with `mesh-delivery.show` of #125.
04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/00-report.md
+3 -3
View File
@@ -21,7 +21,7 @@ On 2026-10-09, local times on the control node (its controller's and node-engine
migrations than the store had applied, and served what it could.
- 16:35:59 that older controller judged the newer plan's gate **passed** on the control node ("healthy 3
times over 4m17s") — from the health of the build it was itself, not of the build the plan had sent,
which the put-back had removed from the machine. The plan walked on, believing the controller delivered.
which the put-back had removed from the machine. The plan rolled out on, believing the controller delivered.
## Why it is an issue
@@ -35,8 +35,8 @@ Four rules did not hold.
gate of the build it had replaced.
- **A put-back puts back something that can run the mesh** (ADR 0236: "put back by something other than
itself"). A controller older than the store's schema starts behind its own records. Nothing refused it.
- **One walk of a module on a machine at a time** (issue 281, ADR 0236). A release's walk was invisible to
the merge plan's send: a release keeps no module records, and the check for a walk in progress read
- **One rollout of a module on a machine at a time** (issue 281, ADR 0236). A release's rollout was invisible to
the merge plan's send: a release keeps no module records, and the check for a rollout in progress read
only those.
The diagnosis and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/01-diagnosis.md
+7 -7
View File
@@ -25,10 +25,10 @@ read as a module's) are the nearest; neither is this.
to the one the first machine ran before (`PreviousBuild`) and sends it. Nothing asked whether that
build can read the store as the failed build left it. The controller says so itself when it starts
("this control plane is behind the mesh-controller row"), and serves anyway (ADR 0185).
- **The merge plan's send crossed the release's gate.** `walkedBy` finds a walk in progress by a plan's
module records (`p.Modules[module].First`). A release (the walk of the builds waiting for a gate) keeps its walk in its gate's carried moves,
- **The merge plan's send crossed the release's gate.** `walkedBy` finds a rollout in progress by a plan's
module records (`p.Modules[module].First`). A release (the rollout of the builds waiting for a gate) keeps its rollout in its gate's carried moves,
not in module records, so the release's open gate on the control node held nothing back. And a plan's
own modules were never checked against another walk at all: the check ran over the moves a machine
own modules were never checked against another rollout at all: the check ran over the moves a machine
had waiting, and skipped the plan's own.
### What is guessed
@@ -62,13 +62,13 @@ In mesh-controller, pull request 185.
never recorded what it reaches, is not put back; the current build is kept, and the condition
(`rollback-failed`, urgent) says why and that a person decides. The lease holder's `Build` is that
version too.
- **A walk of a module on a machine holds every other send of another build of it there.** `walkedBy`
reads a release's open gate as its walk, and a plan's own modules are held by it as any other move is:
- **A rollout of a module on a machine holds every other send of another build of it there.** `walkedBy`
reads a release's open gate as its rollout, and a plan's own modules are held by it as any other move is:
a merge plan whose first machine is under a release's gate for the same module waits, said as
"waiting for … which release-… is walking". The same build to the same machine is not held: it is
already there.
Not changed: whether a release's walk and a merge plan's walk of the core should be one walk (the walk
Not changed: whether a release's rollout and a merge plan's rollout of the core should be one rollout (the rollout
carrying over) is left to a design; serialising them is what this fixes.
## How it is checked
@@ -79,7 +79,7 @@ carrying over) is left to a design; serialising them is what this fixes.
| A send that moves the module supersedes the judging, puts nothing back | `TestReplay352`, `TestAPlansJudgingIsSupersededByASendThatMovesItsModule` |
| A report is held against the gate's own send, by digest or sequence | `TestAReportIsHeldAgainstTheGatesOwnSend`, inventory `TestASendAndAReportAreKnownByTheirDeclaration` |
| Only the build last sent is judged | `TestAGateJudgesOnlyTheBuildTheMachineWasLastSent` |
| A walk holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` |
| A rollout holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` |
| No put-back to a controller behind the store, or one unproved | `TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema`, inventory `TestASchemaReachIsKeptByBuild`, catalogue `TestAProcessEnvCanNameTheBuildsOwnVersion`, `TestTheHolderNamesTheBuildTheDeclarationToldIt` |
| Every condition of the change is tested | 31 mutations, each failing a test |
04-ISSUES/353-the-controller-asked-before-the-bus-held-its-grant/00-report.md
+1 -1
View File
@@ -9,7 +9,7 @@ opened: 2026-10-09
On 2026-10-09, local times on the control node, from its controller's journal (one process throughout):
- 17:32 the router (the `operator-channel` holder) was unassigned, so that a merge gate of the catalogue had
nothing to send there; 17:52:54 a walk of another module pushed the control node and composed the bus's user
nothing to send there; 17:52:54 a rollout of another module pushed the control node and composed the bus's user
list **without** the controller's grant to ask: the grant is composed from a holder of the seat that is
*assigned*, and none was.
- 17:54:05 the controller started asking (the router was assigned again by then, so its check of the record
04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/01-diagnosis.md
+3 -3
View File
@@ -73,9 +73,9 @@ one. The tool runner has the same shape at a smaller scale: the bus account and
controller's user list and in every node's assignment.
Hence: **one release in which both names are accepted, on every side, before any release in which the old
one is dropped**, each release walked one node first — the laptop, where a halted node is the operator at a
one is dropped**, each release rolled out one node first — the laptop, where a halted node is the operator at a
keyboard — and then the rest. I think no stage should move before the delivery fixes of issues 351 and 352
are live, because a core walk is exactly what those broke; that is a guess from their symptoms, not a
are live, because a core rollout is exactly what those broke; that is a guess from their symptoms, not a
measurement.
### The staged plan
@@ -90,7 +90,7 @@ measurement.
tool runner's as another (runtime module, bus account, kind). The node-engine's launcher and upgrade
package read the old and the new library and state directories, and the node-engine moves its state
once, the first time it runs under the new name. Both unit names run the same launcher. The conditions
and the `node` verb say node-engine. The merge gate composes every node green; walked one node first.
and the `node` verb say node-engine. The merge gate composes every node green; rolled out one node first.
3. **The rename proper.** The new module names registered, assigned beside the old on the first node, the
old unassigned there, the acceptance sentence checked on that node, then the rest. The repository
renamed on the forge, with its build source and its Go module path; the packaging renamed. The tool
04-ISSUES/355-a-held-delivery-of-a-module-that-runs-on-one-machine-only-is-never-covered/00-report.md
+4 -4
View File
@@ -9,7 +9,7 @@ opened: 2026-10-09
ADR 0270
(2026-10-09) ends a held delivery by itself when a later merge that contains it was delivered to every
machine running its modules — the delivery is *covered*. Its consequences say: "A module that runs on one
machine only is never covered. A walk records 'the rest running it' only after its first machines passed
machine only is never covered. A rollout records 'the rest running it' only after its first machines passed
and the rest were sent. A module on one machine has no rest, so its steps never say it reached every
machine. mesh-delivery itself, on the control node alone, is such a module: a held merge of it stays held
for a person, as before this record. Reading 'every machine running it' from the first machines when they
@@ -29,8 +29,8 @@ done (issue 354).
## Open
- Whether a walk whose first machines are all the machines should record that it reached every machine
- Whether a rollout whose first machines are all the machines should record that it reached every machine
(the record's own suggestion), or whether *covered* should read the machines running a module from the
placement rather than from the walk's steps.
placement rather than from the rollout's steps.
- The owner is mesh-delivery, in the controller's repository, as the record says; `located-in:` is filled
when the diagnosis confirms where the walk's record is written.
when the diagnosis confirms where the rollout's record is written.
04-ISSUES/358-a-mesh-directory-outside-a-home-is-changed-by-path-through-any-link/00-report.md
+1 -1
View File
@@ -13,7 +13,7 @@ The node-engine sets a directory's mode and owner by path when the directory is
an account's home: `applyDirectory` calls `chmodDirPath` and `own`, which call `os.Chmod` and `os.Chown` on the
path. Both follow a link anywhere in the path. The directory applier refuses a link at the directory's own name,
and issue 339's placement guard refuses the machine's own trees, but nothing refuses a link in a parent at the
moment of use. Below a home the engine already walks every component without following a link (ADR 0266).
moment of use. Below a home the engine already traverses every component without following a link (ADR 0266).
Issue 356 closed this for a directory handed to the mesh by the operator: it is changed only through one
descriptor opened from `/` with no link followed, on every apply. The same treatment for every other mesh
04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/00-report.md
+4 -4
View File
@@ -19,8 +19,8 @@ the controller's `root-free` verb for that machine answered:
> again later
The control-node holds its root and home filesystems on a software RAID of spinning disks, with millions of
files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) walks every local
filesystem mounted without `nosuid`; that walk, cold, takes far longer than fifteen minutes there. Each attempt
files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) traverses every local
filesystem mounted without `nosuid`; that traversal, cold, takes far longer than fifteen minutes there. Each attempt
was killed at its bound, judged nothing, and was tried again after a back-off, which ends the same way.
## Why it is a design fault, not one slow disk
@@ -29,7 +29,7 @@ was killed at its bound, judged nothing, and was tried again after a back-off, w
controller could tell "still running" from "stuck" (`rootsearch.Bound`, read by both). It turned a slow
machine into one that is never judged, failing closed for good rather than for a while.
- **Nothing was kept across a restart.** The search's result lived in the engine's memory only, so every
engine restart (every update of the engine) started from nothing and judged nothing until a whole walk
engine restart (every update of the engine) started from nothing and judged nothing until a whole traversal
finished again.
- **The controller's quiet was the same value.** The self-check stayed quiet while a search was "within its
bound" — fifteen minutes — so a search that runs for hours would raise the urgent condition on every
@@ -51,7 +51,7 @@ was killed at its bound, judged nothing, and was tried again after a back-off, w
**Why a day-old search is sound.** Only root can make a setuid- or setgid-root file: the agent account can
neither set the bit on a file root owns nor give a file of its own to root. So a complete search stays true until
root acts, and root's acts through the mesh (applies) start a new search. A file that appears behind a running
walk's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age
traversal's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age
bounds.
The trail and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
04-ISSUES/361-the-control-nodes-agent-account-is-never-judged/01-diagnosis.md
+37 -37
View File
@@ -15,25 +15,25 @@
health — which this issue does not change).
- **Nothing else** used the bound: `SetuidCache` had one caller (`moreWays`), and the engine made one cache.
- **Ruled out:** a stuck `find` (each attempt ended at exactly its bound) and a filesystem that should not be
searched (the walk covered only the local root, home and boot filesystems, as ADR 0266 asks).
searched (the traversal covered only the local root, home and boot filesystems, as ADR 0266 asks).
## The fix (`mesh-host` pull request 75, `mesh-controller` pull request 190)
**The walk is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as
**The traversal is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as
`find -xdev` did, with the same prunes and the same match (a regular file, setuid and owned by root, or setgid
and of root's group), never following a link. It visits each directory's entries in lexical order, so its
progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is walked before
`/a-c` although `-` sorts before `/`). Every minute the progress (when the walk started, its filesystems, those
progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is traversed before
`/a-c` although `-` sorts before `/`). Every minute the progress (when the traversal started, its filesystems, those
done, the cursor, the matches so far) is written beside the engine's state as a new file renamed into place,
readable by root alone. After a restart the walk carries on from the cursor: what is before it is skipped
without being read, the directories leading back to it are read again. A walk older than the stated age, or one
readable by root alone. After a restart the traversal carries on from the cursor: what is before it is skipped
without being read, the directories leading back to it are read again. A traversal older than the stated age, or one
an apply asked to redo, is thrown away and begun again.
**Low priority, one at a time, no bound.** The walk runs on a goroutine locked to an operating-system thread of
**Low priority, one at a time, no bound.** The traversal runs on a goroutine locked to an operating-system thread of
its own, set to nice 19 and the idle I/O class; on Linux both are a thread's, so the engine's other work keeps
its priority, and the thread ends with the goroutine. The package manager is run the same way. A priority that
cannot be lowered fails the search rather than walk at full priority. There is no deadline: the search ends when
it has walked everything, or with an error (a directory it could not read), which is incomplete.
cannot be lowered fails the search rather than traverse at full priority. There is no deadline: the search ends when
it has traversed everything, or with an error (a directory it could not read), which is incomplete.
**A verdict only from a complete, fresh search.** The last complete search is kept beside the engine's state
with its start and finish, and judges while it is younger than `rootsearch.FreshFor`, counted from its start.
@@ -44,7 +44,7 @@ link, owned by the engine's account and writable by it alone, in a directory tha
**When a search runs.** Once a day (`rootsearch.Every`, from the last complete one's start); after an apply
whose outcomes moved a user, a package, an action, or a file of the user and group databases or of sudo's,
doas's or polkit's rules (`apply.MovesRoot`), at most once an hour from the last search's end, the last complete
search judging meanwhile; and at once when a filesystem appears that the last search did not walk, which that
search judging meanwhile; and at once when a filesystem appears that the last search did not traverse, which that
search does not judge.
**The values, one place** (`rootsearch`, read by both repositories):
@@ -57,29 +57,29 @@ search does not judge.
`FreshFor` is `Every` plus `Quiet` so that a daily search which finishes within the controller's quiet never
leaves an hour unjudged; the controller's test holds that relation. The operator's direction named 26 hours as an
example; 36 is chosen because a full walk at idle priority on the control-node is expected to take hours, and
example; 36 is chosen because a full traversal at idle priority on the control-node is expected to take hours, and
with 26 hours a daily search longer than two hours would leave the account unjudged — and phone linking refused —
every day until it finished.
## 2026-10-10 — the independent review, and what it changed
An independent review of both pull requests asked four questions: can a stale or incomplete search yield "free";
can the agent account steer the kept files or the walk; does the walk hurt the machine; do the controller and the
can the agent account steer the kept files or the traversal; does the traversal hurt the machine; do the controller and the
engine agree. What it found, and what was done:
- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed walk
- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed traversal
skipped every name that sorted between the real cursor and its replacement, and could finish "free" without
having read them. Any account can make such a name. The walk's paths are now kept as their bytes.
- **An ask to search again was lost to a restart**, and the restarted engine carried on the walk that started
before root acted. The ask now forgets the walk under way and stops it keeping progress.
- **A fresh search judged a filesystem it never walked.** A complete search now judges only while it walked every
filesystem a search would walk now; a filesystem mounted since makes the verdict "not judged yet" until walked.
- **A mount point inside a walked filesystem was read before it was known to be another filesystem.** A FUSE
mount root cannot read failed the walk, and a network mount whose server is away could hang it. Every mount
having read them. Any account can make such a name. The traversal's paths are now kept as their bytes.
- **An ask to search again was lost to a restart**, and the restarted engine carried on the traversal that started
before root acted. The ask now forgets the traversal under way and stops it keeping progress.
- **A fresh search judged a filesystem it never traversed.** A complete search now judges only while it traversed every
filesystem a search would traverse now; a filesystem mounted since makes the verdict "not judged yet" until traversed.
- **A mount point inside a traversed filesystem was read before it was known to be another filesystem.** A FUSE
mount root cannot read failed the traversal, and a network mount whose server is away could hang it. Every mount
point the kernel lists is now left unread. Mount points are read with all the kernel's escapes.
- **A file found and gone before the package manager was asked failed every resumed search.** The walk looks at
- **A file found and gone before the package manager was asked failed every resumed search.** The traversal looks at
what it found again when it ends, and drops what went or is no longer setuid-root.
- **Outcomes that moved nothing started a search**, and a search asked for could follow a walk of hours at once.
- **Outcomes that moved nothing started a search**, and a search asked for could follow a traversal of hours at once.
Only created, updated, corrected and removed outcomes count now, and an ask waits an hour from the last
search's end.
- **Not changed, because the mesh cannot do it:** the review asked that a file or archive declared setuid start a
@@ -87,53 +87,53 @@ engine agree. What it found, and what was done:
## Open, after the fix
- **How long the first walk on the control-node takes is not measured.** Until the first complete search, the
- **How long the first traversal on the control-node takes is not measured.** Until the first complete search, the
verdict there is `not judged yet`, and linking stays refused. The engine says each search's end and how long
it ran in its journal; that line is the measurement. If a walk takes longer than `Quiet`, the self-check
it ran in its journal; that line is the measurement. If a traversal takes longer than `Quiet`, the self-check
raises the urgent condition, which is right: it is then not known to finish.
- **The idle I/O class is honoured only by a disk scheduler that has classes** (BFQ). Under another scheduler
the walk still runs at nice 19 but competes for the disk as any reader does.
- **An ask to search again is not kept across a restart.** The walk under way is forgotten when it is asked,
so a restarted engine does not carry an older walk on; the ask itself is lost, and the daily search covers it.
the traversal still runs at nice 19 but competes for the disk as any reader does.
- **An ask to search again is not kept across a restart.** The traversal under way is forgotten when it is asked,
so a restarted engine does not carry an older traversal on; the ask itself is lost, and the daily search covers it.
The last complete search keeps judging meanwhile, on the reasoning in the report.
- **The agent account can make the search fail, never pass.** A tree deeper than the longest path the kernel
takes, or a directory unreadable to root through FUSE, ends every walk incomplete; a directory of millions of
takes, or a directory unreadable to root through FUSE, ends every traversal incomplete; a directory of millions of
entries makes the engine hold them all at once. Each is "not judged" — the urgent condition, and linking
refused — never a pass. Reading by directory handle, as `find` did, would close the first.
- **A filesystem mounted only on use** (an automounted boot partition) makes the verdict "not judged yet" each
time it appears unwalked, until a walk covers it.
- **Part of the walk's cost escapes its priority**: Go's garbage collector runs on threads of normal priority,
time it appears untraversed, until a traversal covers it.
- **Part of the traversal's cost escapes its priority**: Go's garbage collector runs on threads of normal priority,
and a software RAID's own threads do not carry the idle class to the disks.
- **ADR 0266 §3 item 10** still says "at most hourly" and "at most 15 minutes". On resolution it gains a
progressive insight naming this issue and the values above.
- **A later option, not built now:** a fanotify filesystem mark with `FAN_ATTRIB` (Linux 5.1 and later) on
every searched filesystem would report each change of mode or owner as it happens, turning the daily walk into
every searched filesystem would report each change of mode or owner as it happens, turning the daily traversal into
a true incremental search.
- **Resolution** of this core issue needs a replay in mesh-lab's register, or `replay-none:` with the reason.
## 2026-10-10 — the search shows its progress, and the operator can ask for a fresh one
- **The progress was kept and seen by nobody.** At 01:40 the operator asked how far the first walk on the
control-node had got. The walk kept its progress every minute beside the engine's state, but `root-free` said
- **The progress was kept and seen by nobody.** At 01:40 the operator asked how far the first traversal on the
control-node had got. The traversal kept its progress every minute beside the engine's state, but `root-free` said
only "not judged yet (search running), started 01:27", and no tool read further. The node-engine serves no
tools of its own, so the progress now travels where the verdict already does: the agent account's verdict
carries it as fields, and its reason gains one line that `root-free` repeats. The line holds filesystems done
of the total, the mount point being walked, the entries visited, the setuid- or setgid-root files found so far,
of the total, the mount point being traversed, the entries visited, the setuid- or setgid-root files found so far,
when the progress was last kept, and how long each finished filesystem took. Counts, times and mount points
only: the cursor and every file found are often under an account's home, and stay in the engine's state and
journal. The line moves at each checkpoint (once a minute) and is never part of what makes a verdict
"changed", so the journal does not repeat it every minute; the statement is said every minute anyway while
the account is not healthy. An entry is counted once across a resume, as it is kept with the cursor it
matches. The time a filesystem took counts only the time the engine walked it, not the time it was stopped.
matches. The time a filesystem took counts only the time the engine traversed it, not the time it was stopped.
- **A change by hand was not searched for until the next day.** The first complete search on the control-node
ended at about 01:48 and found two setuid-root programs no package owns, copies in the operator's home. When the
operator removes them by hand, no apply says so, and nothing searches again until the daily search. The
operator now asks for one at the controller's terminal: `nox node setuid-search <node>` on the control-node.
The controller signs the ask with the mesh's key, as it signs a hand-over, under a signing context of its own.
The engine throws away the last complete search and any walk under way or kept, and the next look starts a
The engine throws away the last complete search and any traversal under way or kept, and the next look starts a
full search. Until it completes, the verdict is "not judged yet", with its progress.
- **Why at the terminal alone, not a verb.** A search can never make a machine root-free wrongly: it judges only
once complete. But every ask throws a walk away. Asked again and again, it would keep the machine unjudged
once complete. But every ask throws a traversal away. Asked again and again, it would keep the machine unjudged
for good and its disks busy, and whoever may call a verb includes agents. The operator needs the line only
after a change made by hand.
- **A find counts only where the agent account can execute it.** The operator asked how the agent account could
04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md
+24 -24
View File
@@ -11,17 +11,17 @@ Shortly after midnight on 2026-10-10 (local time) two pull requests of the catal
their checks.
`mesh-delivery.deliveries` then showed every delivery of the dunst pull request `superseded`, and the
claude-code delivery `held`, waiting for "a person: merged, and the controller opened no walk for it within
claude-code delivery `held`, waiting for "a person: merged, and the controller opened no rollout for it within
10m0s — nothing it holds follows that branch, or the merge was not heard: a person decides". The
controller's `plans` said of the walk for the catalogue's head, which carried both merges: "published; its
walk waits for mesh-delivery's word — `plans go … --why …` starts it by hand".
controller's `plans` said of the rollout for the catalogue's head, which carried both merges: "published; its
rollout waits for mesh-delivery's word — `plans go … --why …` starts it by hand".
Both reasons were wrong. A walk had been opened, and it contained the held merge. Nothing gave it the word.
Both reasons were wrong. A rollout had been opened, and it contained the held merge. Nothing gave it the word.
The operator's change sat undelivered for 58 minutes, until the operator's hand-act `mesh-controller.plans
go` started the walk. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours
before `mesh-delivery.stalled` lists it, and the walk's own wait raised only a warning, at 00:51, which said
go` started the rollout. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours
before `mesh-delivery.stalled` lists it, and the rollout's own wait raised only a warning, at 00:51, which said
there was nothing to do and pointed at the ended delivery (the diagnosis's second entry; this sentence said
before that the walk's wait raised nothing).
before that the rollout's wait raised nothing).
## Why it is an issue
@@ -33,41 +33,41 @@ cause.
## Where it lives
The diagnosis ([01-diagnosis.md](01-diagnosis.md)) traces it to two parts that each behave as written and
disagree on what a walk's commit means:
disagree on what a rollout's commit means:
- The controller plans a merge it hears after a later merge of the same branch at the later merge's commit,
and supersedes the earlier walk (the fix for
and supersedes the earlier rollout (the fix for
[issue 349](../349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md)).
So the claude-code merge got a walk at the dunst merge's commit.
- mesh-delivery finds the delivery a walk walks by its walk id or by the commit it was merged as. The new
walk matched the dunst delivery, which its first walk's supersession had already ended, so no live
delivery held the walk and nothing gave it its turn. The claude-code delivery, merged as a different
commit, never got a walk and was held after ten minutes.
So the claude-code merge got a rollout at the dunst merge's commit.
- mesh-delivery finds the delivery a rollout rollouts by its rollout id or by the commit it was merged as. The new
rollout matched the dunst delivery, which its first rollout's supersession had already ended, so no live
delivery held the rollout and nothing gave it its turn. The claude-code delivery, merged as a different
commit, never got a rollout and was held after ten minutes.
## Open
- How mesh-delivery learns which merges a walk carries: the controller's walk record naming every merge it
- How mesh-delivery learns which merges a rollout carries: the controller's rollout record naming every merge it
answers (its commit and the commits planned into it), or mesh-delivery reading containment itself, as it
already does to decide *covered*.
- A superseded walk whose successor matches only final deliveries: whether the successor should be adopted
as a delivery of its own, so that something always holds a waiting walk.
- A walk waiting for mesh-delivery's word past a bound, and a delivery held because no walk was opened,
- A superseded rollout whose successor matches only final deliveries: whether the successor should be adopted
as a delivery of its own, so that something always holds a waiting rollout.
- A rollout waiting for mesh-delivery's word past a bound, and a delivery held because no rollout was opened,
raise a condition to the operator instead of waiting silently (failure must be loud).
## Decided
The operator decided the model that removes this shape, not only its bookkeeping:
ADR 0276. Merges are collected in a rolling merge window and each batch is walked once, with at most one
commit per repository, one walk at a time. The bookkeeping that tied a walk to an ended delivery is repaired as
ADR 0276. Merges are collected in a rolling merge window and each batch is rolled out once, with at most one
commit per repository, one rollout at a time. The bookkeeping that tied a rollout to an ended delivery is repaired as
well, and every wait of this shape has a bound and a condition. It is designed in to-be 47, *The merge window
and the batch*.
## Resolved
2026-10-10. mesh-delivery's side is mesh-catalog#171 (merged): a walk is tied to every live delivery whose
merge it answers, never to one that has ended, and a waiting walk no live delivery holds is listed by `stalled`
at once. The controller's side is mesh-controller#197: the merge window and the batch, one walk per batch with
one commit per repository, the walk naming every merge it answers, and the bounds of decision 9. The replay
2026-10-10. mesh-delivery's side is mesh-catalog#171 (merged): a rollout is tied to every live delivery whose
merge it answers, never to one that has ended, and a waiting rollout no live delivery holds is listed by `stalled`
at once. The controller's side is mesh-controller#197: the merge window and the batch, one rollout per batch with
one commit per repository, the rollout naming every merge it answers, and the bounds of decision 9. The replay
R362 is registered in mesh-lab's replays register (mesh-lab#75), proved to fail on the commit before the
mesh-delivery fix and pass on it. The build raised one rule the record had not settled, decided during the
build and written into ADR 0276: a merge on the controller's own path never shares a batch with a catalogue
04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/01-diagnosis.md
+23 -23
View File
@@ -9,38 +9,38 @@ request as 48bda475 at 00:21:09. 48bda475 is the branch's head and contains 553b
**What the controller did** (its journal, 00:21:14 to 00:21:15):
1. It heard the dunst merge first and opened a walk at 48bda475 for dunst, waiting for mesh-delivery's word.
1. It heard the dunst merge first and opened a rollout at 48bda475 for dunst, waiting for mesh-delivery's word.
2. It then heard the claude-code merge, and printed "553b7191 was merged before 48bda475 (…, building): what
it moved is built from 48bda475, which contains it". This is the rule that answered
[issue 349](../349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md):
`SourceMoved` replaces the merge's commit and merge time with the newest planned merge's, when that merge
is later on the branch (`laterOnTheBranch`).
3. The second walk, also at 48bda475 and with the same merge time, superseded the first by when each was made
3. The second rollout, also at 48bda475 and with the same merge time, superseded the first by when each was made
(`supersededBy`, `earlierOnTheBranch`), and carried claude-code and dunst. It waited for mesh-delivery's
word.
So every walk the controller opened was at 48bda475. None was at 553b7191.
So every rollout the controller opened was at 48bda475. None was at 553b7191.
**What mesh-delivery did** (`mesh-delivery.show` for both deliveries):
1. The dunst delivery (merged as 48bda475) took the first walk, matched by its commit, and became
1. The dunst delivery (merged as 48bda475) took the first rollout, matched by its commit, and became
`published` at 00:21:14.
2. The first walk's record came back superseded, and the table's `superseded` transition ended the dunst
2. The first rollout's record came back superseded, and the table's `superseded` transition ended the dunst
delivery at 00:21:15 ("a newer delivery to the same trunk took over its walk").
3. The second walk was matched to a delivery by `deliveryWalking`, which looks first for a delivery holding
that walk id and then for any delivery whose `merged_as` is the walk's commit, final or not. That is the
dunst delivery again, already final. The walk was linked there, so it was never kept as unmatched and
3. The second rollout was matched to a delivery by `deliveryWalking`, which looks first for a delivery holding
that rollout id and then for any delivery whose `merged_as` is the rollout's commit, final or not. That is the
dunst delivery again, already final. The rollout was linked there, so it was never kept as unmatched and
never adopted as a delivery of its own. A turn (`turn`) is given only to a `published` or `delivering`
delivery, so nothing ever gave the second walk its word.
4. The claude-code delivery, merged as 553b7191, matched no walk. Ten minutes after its merge
delivery, so nothing ever gave the second rollout its word.
4. The claude-code delivery, merged as 553b7191, matched no rollout. Ten minutes after its merge
(`unwalkedPast`) it was held for a person at 00:30:58, with a reason that names two causes, neither of
which happened.
After the operator's `plans go`, the second walk's machine steps (claude-code and dunst, each on a node) were recorded on the superseded dunst delivery, which confirms that mesh-delivery reads that walk as
After the operator's `plans go`, the second rollout's machine steps (claude-code and dunst, each on a node) were recorded on the superseded dunst delivery, which confirms that mesh-delivery reads that rollout as
the dunst delivery's.
**Why nothing was raised.** A delivery held for a person is bounded at 24 hours before
`mesh-delivery.stalled` lists it (`Bounds` in its table). The walk waiting for mesh-delivery's word is
`mesh-delivery.stalled` lists it (`Bounds` in its table). The rollout waiting for mesh-delivery's word is
counted against the controller's plan wait bound of 30 minutes for `status`, but no condition reached the
operator in the 58 minutes it waited. I did not trace why the controller's side stayed quiet; that belongs
to the fix.
@@ -48,28 +48,28 @@ to the fix.
## Ruled out
- **The merge was not heard.** The controller's journal shows it was heard and planned within 23 seconds.
- **Nothing follows that branch.** claude-code follows the catalogue's trunk; the second walk built it.
- **Nothing follows that branch.** claude-code follows the catalogue's trunk; the second rollout built it.
- **A failed check.** Both heads passed the merge gate and the repository's check.
## Located
Two parts, each correct by its own rule, disagree on what a walk's commit means. The controller (since issue
349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a walk's commit is the
merge commit of exactly one delivery. `located-in` names both. The fix is to make the walk say which merges
it answers, or to make mesh-delivery match by containment, and to let no waiting walk be held by nothing.
Two parts, each correct by its own rule, disagree on what a rollout's commit means. The controller (since issue
349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a rollout's commit is the
merge commit of exactly one delivery. `located-in` names both. The fix is to make the rollout say which merges
it answers, or to make mesh-delivery match by containment, and to let no waiting rollout be held by nothing.
## 2026-10-10 — what the controller's side said
The first entry says no condition reached the operator in the 58 minutes. The controller's condition history
says more precisely what happened. S16 (a walk waiting for its delivery's word) raised
`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the walk opened, as its bound says.
says more precisely what happened. S16 (a rollout waiting for its delivery's word) raised
`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the rollout opened, as its bound says.
It was a warning, its explanation was "Nothing for you to do … It becomes a question for you if it still waits
after 4 hours", and its summary sent the reader to "`mesh-delivery.show` for the delivery that landed as
48bda475", which is the dunst delivery the supersession had already ended. It cleared at 01:20:32, when the
operator's `plans go` started the walk.
operator's `plans go` started the rollout.
So the controller's side was not silent; it was mild and pointed at the wrong delivery. S16 cannot tell a
group member waiting its turn from a walk nobody holds; only mesh-delivery knows that. That is why ADR 0276
decision 9 has mesh-delivery list a waiting walk no live delivery holds through `stalled` at once, and has
S16 name the merges a walk answers.
group member waiting its turn from a rollout nobody holds; only mesh-delivery knows that. That is why ADR 0276
decision 9 has mesh-delivery list a waiting rollout no live delivery holds through `stalled` at once, and has
S16 name the merges a rollout answers.
README.md
+1 -1
View File
@@ -14,7 +14,7 @@ and why. Today that is almost entirely **Novox Mesh**, the foundation everything
| [`04-ISSUES`](04-ISSUES/) | The front door for "something is wrong" at the level of design or governance. |
**The numbering is the flow.** Research produces a decision; the decision authorises a design.
That is why decisions are `02` and design is `03` — a reader following the numbers walks the
That is why decisions are `02` and design is `03` — a reader following the numbers rollouts the
process in the order it happens. The research efforts and decision records themselves live in the knowledge
base, the seat `knowledge-base`, read and written through the mesh MCP server from any node (ADR 0290).