Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ba7c9e7f17 | ||
|
|
9bfd57174f | ||
|
|
a227f98bdd | ||
|
|
409fa9bbc9 | ||
|
|
747fc546c1 | ||
|
|
046bbd3b30 | ||
|
|
99243ec47e | ||
|
|
0d66edb24c | ||
|
|
5ff97ec3aa | ||
|
|
75ac87c72b | ||
|
|
437523554e | ||
|
|
058d76ee8a | ||
|
|
db6430ccbf | ||
|
|
458f248c05 | ||
|
|
464833d0ef | ||
|
|
99a9da0950 | ||
|
|
de8a7176ab | ||
|
|
a7d4c39535 | ||
|
|
048f14540a | ||
|
|
c2ee44255e | ||
|
|
ddf23a7861 | ||
|
|
f3bbfd4bf6 | ||
|
|
893d90fb41 | ||
|
|
1e577168bf | ||
|
|
f349c940f0 | ||
|
|
1699375476 | ||
|
|
912fd428fb | ||
|
|
a75faad882 | ||
|
|
9f4757a8e4 | ||
|
|
93f8bbfc8e | ||
|
|
58edc4b4fb | ||
|
|
010a34cc33 | ||
|
|
35d5b2814f | ||
|
|
3c5edc3879 | ||
|
|
ab5025d508 | ||
|
|
6c8532ce70 | ||
|
|
c032cae100 | ||
|
|
071fc1c29d | ||
|
|
c2d5323dea | ||
|
|
21b79a04a2 | ||
|
|
bafc7b7d05 | ||
|
|
5e493934a5 | ||
|
|
c957fcc6d7 | ||
|
|
dcb5d5ce38 | ||
|
|
bb8075b37b | ||
|
|
40a39c8602 | ||
|
|
e5b98e3b0a | ||
|
|
2f3e6e6641 | ||
|
|
1405dd39e8 | ||
|
|
11294372b7 |
+16
-16
@@ -178,7 +178,7 @@ every other domain has something to run on. **Recorded by** the controller and t
|
||||
(ADR 0229).
|
||||
|
||||
- **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its
|
||||
records (nodes, assignments, walks, builds, settings), after each change commits in its store and only
|
||||
records (nodes, assignments, rollouts, builds, settings), after each change commits in its store and only
|
||||
when the value differs, with one event per write. The store stays the record; the self-check compares
|
||||
the two and raises `mirror-drifted` on any difference
|
||||
(ADR 0289). Shortened to *mirror* where the
|
||||
@@ -306,7 +306,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences`
|
||||
|
||||
**Purpose.** To take one commit from its pull request to every node that should run it, and to put it
|
||||
back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the
|
||||
planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
planner, the rollout). **Downstream of** Module, Core, Health and repair and Data.
|
||||
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323.
|
||||
**Uses:** module, node, declaration and send (Placement), health (Health and repair), person.
|
||||
|
||||
@@ -348,7 +348,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
- **delivery** — one commit in one repository on its way to the nodes, from its pull request's head
|
||||
being announced to one of its final states: one pull request, one status, one note
|
||||
(ADR 0239).
|
||||
From its merge on it is delivered by the walk of the batch that answers it, and ends with that walk
|
||||
From its merge on it is delivered by the rollout of the batch that answers it, and ends with that rollout
|
||||
(ADR 0276).
|
||||
Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`,
|
||||
`held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`;
|
||||
@@ -357,9 +357,9 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
mechanism, which the as-is designs describe; it is never a word for a delivery.
|
||||
*Not:* ~~deployment~~ (hq)
|
||||
- **covered** — the end of a delivery that a later merge containing it delivered: every module it moves was
|
||||
sent to every machine running it by the walk of a later merge into the same trunk, delivered on its own
|
||||
sent to every machine running it by the rollout of a later merge into the same trunk, delivered on its own
|
||||
passing check, whose merge commit the forge says contains this one. Not *superseded*, which is a delivery
|
||||
whose walk a newer delivery took over before it ended; nor *delivered*, which its own walk or an empty plan
|
||||
whose rollout a newer delivery took over before it ended; nor *delivered*, which its own rollout or an empty plan
|
||||
ends in (ADR 0270).
|
||||
- **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's
|
||||
repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One
|
||||
@@ -372,18 +372,18 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
- **merge window** — the rolling wait in which the controller collects merges before it plans them: each
|
||||
merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest
|
||||
10 minutes after its first merge (ADR 0276).
|
||||
- **batch** — the merges one merge window collected, planned once as one walk. For a repository merged more
|
||||
than once in it, the walk takes the latest commit, and the earlier merges are answered by it. Not *group*,
|
||||
- **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more
|
||||
than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*,
|
||||
which is a delivery group.
|
||||
- **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat
|
||||
of the same name. It records and decides; the controller sends, judges and rolls back when it is asked.
|
||||
- **walk** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
|
||||
- **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
|
||||
repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236,
|
||||
ADR 0276). One walk is open at a time; it names every merge it answers. A primitive the delivering stage
|
||||
asks for, not an object anyone manages. A **tier** is one step of a walk: a module, then the modules
|
||||
built against it. The controller's verb `plans` lists the walks. Walking is not a noun of its own:
|
||||
a module *rolls out* by its `roll` policy.
|
||||
*Not:* ~~rollout~~ (hq)
|
||||
ADR 0276). One rollout is open at a time; it names every merge it answers. A primitive the delivering stage
|
||||
asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules
|
||||
built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll`
|
||||
policy.
|
||||
*Not:* ~~walk~~ (hq)
|
||||
- **first-node gate** — the judgement on a delivery's first node: healthy three times, no witness put it
|
||||
back, no condition raised since the send (ADR 0236 §2). Not *the gate* bare (see [Homonyms](#homonyms)).
|
||||
- **core module** and **leaf module** — a core module is one on the controller's own path (the controller,
|
||||
@@ -394,7 +394,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
the core modules add mesh-delivery and the resolver.
|
||||
- **delivery time** — from a delivery's merge to every node of its deploy plan that is heard from running its
|
||||
modules' builds and reporting them healthy. Its **delivery budget** is what it is held to: three minutes for
|
||||
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the walk's time, which ends at the send. A
|
||||
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the rollout's time, which ends at the send. A
|
||||
measure inside the fix budget, not the operator's target (ADR 0313).
|
||||
- **fix time** — from an issue's first move to `fixing` to the moment the last of its fixes is delivered: the
|
||||
operator's measure of a fix, the agent's work, the review and the checks included, the delivery time its last
|
||||
@@ -672,7 +672,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|
||||
|---|---|---|---|
|
||||
| plan | what the controller would send one node | that node's **declaration** (verb `plan`) | Placement |
|
||||
| | what a delivery would do to the mesh | **delivery plan** | Change and delivery |
|
||||
| | the sending of one batch of commits across nodes | **walk** (verb `plans`) | Change and delivery |
|
||||
| | the sending of one batch of commits across nodes | **rollout** (verb `plans`) | Change and delivery |
|
||||
| | a step a person starts, like the bus's | **planned step** | Core |
|
||||
| membership | the subjects a bus account is issued | **membership** | Provisioning |
|
||||
| | an account in a group of the machine | **account group** | Placement |
|
||||
@@ -686,7 +686,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|
||||
| | the pull request status | **merge gate** | Change and delivery |
|
||||
| | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement |
|
||||
| tier | a level of the mesh | **layer** | Core |
|
||||
| | a step of a walk | **tier** | Change and delivery |
|
||||
| | a step of a rollout | **tier** | Change and delivery |
|
||||
| | how much proof an ask needs | **assurance level** | Identity and access |
|
||||
| ask | a request for the operator's input | **ask** | Operator and conversation |
|
||||
| | an entry in the build queue | **build request** | Change and delivery |
|
||||
|
||||
@@ -50,13 +50,14 @@ the expensive half.
|
||||
|
||||
| # | Playbook | Trigger |
|
||||
|---|---|---|
|
||||
| [01](01-research.md) | Research | An idea worth investigating before committing to design |
|
||||
| [02](02-graduation.md) | Graduation & design change | Research concludes, or a design must change |
|
||||
| [01](01-research.md) | Research | An idea from the operator, or one worth investigating before committing to design: setup questions, findings, a direction the operator confirms |
|
||||
| [02](02-graduation.md) | Graduation & design change | Research concludes (its direction confirmed), or a design must change: decision records, the to-be design |
|
||||
| [03](03-issues.md) | Issues | Something is wrong — often with the owner unknown |
|
||||
| [04](04-build-handoff.md) | Build handoff | A design is ready to be built |
|
||||
| [04](04-build-handoff.md) | Work breakdown and handoff | A design is ready to be built: milestones usable on their own, issues per repository, each milestone one feature step |
|
||||
| [05](05-constitution-sync.md) | Constitution sync | `how-we-build.md` changed a rule the mesh enforces |
|
||||
| [06](06-writing-a-module.md) | Writing a module | Something that runs today must run on the mesh |
|
||||
| [07](07-feature-branches.md) | Feature branches across repos | Work that changes code, in one repo or several at once |
|
||||
| [07](07-feature-branches.md) | Features | Work that changes code across issues or repositories: prepared context, local development, one review, one check, one delivery |
|
||||
| [08](08-small-fixes.md) | Small fixes | One issue in one repository, about an hour of work: one review, one push, one delivery |
|
||||
|
||||
## The cycle is checked
|
||||
|
||||
|
||||
@@ -1,66 +1,105 @@
|
||||
# Playbook 01 — Research
|
||||
|
||||
**Trigger.** An idea, technology or approach worth investigating before it is committed to
|
||||
design. Also: an as-is document that raises a question nobody can answer.
|
||||
**Trigger.** An idea the operator expresses, or an idea, technology or approach worth investigating before
|
||||
it is committed to design. Also: an as-is document that raises a question nobody can answer. A symptom (a
|
||||
bug, something broken) is not research: it follows playbook [03](03-issues.md).
|
||||
|
||||
**Who runs it.** Anyone.
|
||||
**Who runs it.** Anyone; in an agent session, the session that heard the idea leads it.
|
||||
|
||||
**Where it lives.** A research effort lives whole in the knowledge base, the seat `knowledge-base`, reached
|
||||
through the mesh MCP server from any node and any folder (ADR 0290). This repository holds none: an effort
|
||||
is written with the knowledge base's verbs, never as a folder here.
|
||||
through the mesh MCP server from any node and any folder (ADR 0290). This repository holds none.
|
||||
|
||||
## The cycle it starts
|
||||
|
||||
```
|
||||
idea ─► research (01) ─► decisions and design (02) ─► work breakdown in milestones (04)
|
||||
─► each milestone built and delivered as one feature step (07) ─► what is reported afterwards (03, 08)
|
||||
```
|
||||
|
||||
An idea arrives in one of two forms, and the first step differs:
|
||||
|
||||
- **A new idea**, where only the mesh's own context matters: read what the knowledge base holds on it.
|
||||
- **An idea for an existing application**: read that application's to-be and as-is designs, its decision
|
||||
records and its open issues first; the effort `touches` them.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Read what is known first.** `knowledge-base.ask` with the question in plain words returns the decision
|
||||
records and research efforts that bear on it; `knowledge-base.get` with `kind: research` and a number
|
||||
reads one effort, and `reasoning: true` (or `section` with a document's name) reads its documents.
|
||||
2. **Take a number.** `knowledge-base.reserve` with `kind: research` and `for`: the effort's title. The
|
||||
number comes from the issue tracker's counter and is spent whether or not the effort is ever proposed;
|
||||
one never proposed is withdrawn after a week. Never pick a number by hand.
|
||||
3. **Propose it.** `knowledge-base.propose` with `kind: research`, the `number` reserve gave, a one-line
|
||||
`title`, `status: active`, `date` (the day it began), and `sections`: its documents in order, each a
|
||||
`name` and its markdown `text`. The first is `00-overview.md`: **what** is being investigated, **why**,
|
||||
and **what it touches**. Name what it touches as `links` of kind `touches` (`"0276"`, `"research 045"`,
|
||||
`"issue 365"`, or a design's path), not only in prose, and what it names for another reason as `cites`.
|
||||
4. **Do the research** in further documents of the same effort: notes, option analyses, evidence, draft
|
||||
designs. Each change is `knowledge-base.propose` again with the effort whole and `why`, one line saying
|
||||
why this version. Every version is kept; `knowledge-base.history` lists them and gives the difference
|
||||
between two.
|
||||
1. **Read what is known.** `knowledge-base.ask` with the idea in plain words returns the decision records and
|
||||
research efforts that bear on it; `knowledge-base.get` reads one (`reasoning: true`, or `section`, for its
|
||||
documents). Read the issue tracker for open issues in the same area (`issue-tracker.list`).
|
||||
2. **Ask the operator to set it up.** A few questions, one at a time, each answerable with a choice and
|
||||
carrying a recommendation: what comes first, what is out of scope, which options must be tried, what
|
||||
done looks like. Three to five questions is usual; the answers are the effort's first findings.
|
||||
3. **Take a number.** `knowledge-base.reserve` with `kind: research` and `for`: the effort's title. The number
|
||||
is spent whether or not the effort is ever proposed; one never proposed is withdrawn after a week. Never
|
||||
pick one by hand.
|
||||
4. **Propose it.** `knowledge-base.propose` with `kind: research`, the `number`, a one-line `title`,
|
||||
`status: active`, `date` (the day it began), and `sections`, each a `name` and its markdown `text`:
|
||||
- `00-overview.md`: the idea in the operator's words, what exists, the operator's answers (dated), and the
|
||||
words the effort uses that the glossary lacks, each defined in one line;
|
||||
- `01-questions.md`: the questions to answer, the first ones first;
|
||||
- `02-method.md`: what will be tried and measured, and on what.
|
||||
|
||||
`propose` refuses an effort that breaks a rule that judges one record (an unknown status, a link to
|
||||
nothing, a retired word of the glossary, a daemon's line), naming every reason at once and writing nothing.
|
||||
The knowledge base then reviews the proposal itself, through a judgement that sees only the record, the
|
||||
records it links and the rules; on `changes` the effort stays proposed with the findings, and its author
|
||||
proposes it again. An effort that needs the operator's word is proposed with `needs-operator: true` and
|
||||
waits for the operator's Approve on a channel that proves who answered.
|
||||
Name what it touches as `links` of kind `touches` (`"0276"`, `"research 045"`, `"issue 365"`, or a
|
||||
design's path), and what it names for another reason as `cites`.
|
||||
5. **Do the research.** Try, measure and read, smallest question first. Each finding goes into
|
||||
`04-findings.md`: what was tried, what it showed (with numbers), what it means. Each change of the effort
|
||||
is `knowledge-base.propose` again, whole, with `why`; every version is kept, and `knowledge-base.history`
|
||||
lists them and gives the difference between two. Until issue 844 is delivered, the knowledge base
|
||||
takes a new version of an accepted, active effort only as a progressive insight: give `insight`, saying
|
||||
what the effort held before the new finding, and mark the finding in the text with its date.
|
||||
6. **Settle the direction with the operator.** When the questions are answered, write `05-direction.md`:
|
||||
the proposed decisions in a few lines each, the options rejected and why, what is left open. Ask the
|
||||
operator to confirm or change it, one point at a time. Then the effort graduates through playbook
|
||||
[02](02-graduation.md).
|
||||
|
||||
## Bounds and models
|
||||
|
||||
| Step | Bound | Model |
|
||||
|---|---|---|
|
||||
| reading and the setup questions (1, 2) | 30 minutes | the session's own, larger |
|
||||
| proposing (3, 4) | 15 minutes | larger |
|
||||
| each experiment or reading task (5) | 30 minutes | smaller, one agent at a time unless the operator allows more |
|
||||
| the direction (6) | 30 minutes | larger |
|
||||
|
||||
A step past its bound is said to the operator at once, with what is being done about it. The lead reports
|
||||
as playbook 07 does: every ten minutes and at once when something is found, stops or needs the operator,
|
||||
with the allowance used.
|
||||
|
||||
## What makes research worth reading
|
||||
|
||||
State evidence, not assertion. *"Zero of 124 modules declare `brain` as a dependency"*
|
||||
outranks *"the dependency rule is not followed"*. An effort that measured nothing has not
|
||||
finished.
|
||||
State evidence, not assertion. *"Zero of 124 modules declare `brain` as a dependency"* outranks *"the
|
||||
dependency rule is not followed"*. An effort that measured nothing has not finished.
|
||||
|
||||
Research describes real observations but never identifies the mesh it observed. The shape of a
|
||||
finding survives anonymisation intact — *a node publicly named but behind a household NAT*
|
||||
carries the whole lesson without naming anything.
|
||||
Research describes real observations but never identifies the mesh it observed. The shape of a finding
|
||||
survives anonymisation intact — *a node publicly named but behind a household NAT* carries the whole
|
||||
lesson without naming anything.
|
||||
|
||||
`propose` refuses an effort that breaks a rule that judges one record (an unknown status, a link to
|
||||
nothing, a retired word, a daemon's line), naming every reason at once and writing nothing. The knowledge
|
||||
base then reviews the proposal itself; on `changes` the effort stays proposed with the findings and its
|
||||
author proposes it again, at most three times before the findings go to the operator with a
|
||||
recommendation. An effort that needs the operator's word is proposed with `needs-operator: true` and waits
|
||||
for the operator's Approve on a channel that proves who answered.
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not put status in prose. It is the effort's `status`, and the prose must not restate it.
|
||||
- Do not give a `became` link while the effort is open — playbook 02 gives it at closure.
|
||||
- Do not write into `03-DESIGN` from an open effort.
|
||||
- Do not write an effort, or a copy of one, into this repository: the knowledge base is the only place one
|
||||
is read, and `records.py` refuses a new path under `01-RESEARCH/`.
|
||||
- Do not write an effort, or a copy of one, into this repository: `records.py` refuses a new path under
|
||||
`01-RESEARCH/`.
|
||||
|
||||
## Closing
|
||||
|
||||
An effort never just stops. It closes through playbook [02](02-graduation.md) as `graduated`
|
||||
or `abandoned`, always with a `became` link to what it turned into. Nothing is deleted —
|
||||
what was rejected, and why, is the more expensive half to rediscover.
|
||||
An effort never just stops. It closes through playbook [02](02-graduation.md) as `graduated` or
|
||||
`abandoned`, always with a `became` link to what it turned into. Nothing is deleted — what was rejected,
|
||||
and why, is the more expensive half to rediscover.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The rules on an effort are the knowledge base's: `propose` refuses one that breaks a rule that judges one
|
||||
record (the module's table test holds one case per rule), and the rules that judge the whole record set run
|
||||
after every acceptance and raise a condition when one fails. That this repository holds no effort is
|
||||
`records.py`'s `no-records` check, on every pull request.
|
||||
`records.py`'s `no-records` check, on every pull request. An active effort with no new version for seven
|
||||
days is the smell of research left behind: its lead closes or continues it.
|
||||
|
||||
@@ -13,11 +13,16 @@ clauses in force for a topic or a record).
|
||||
|
||||
## Graduating research
|
||||
|
||||
The effort's `05-direction.md`, confirmed by the operator (playbook [01](01-research.md), step 6), is what
|
||||
graduates. The lead writes the records and the design on the larger model; it is one sitting, bounded at
|
||||
about an hour, and said to the operator if it runs past.
|
||||
|
||||
1. **Check it against GENESIS.** An effort graduates only if its conclusion is traceable to
|
||||
[`mission.md`](../mission.md), [`context.md`](../context.md) and
|
||||
[`effect.md`](../effect.md). If it conflicts, either the effort is wrong or GENESIS is —
|
||||
say which, in writing, before proceeding.
|
||||
2. **Record the decision** (below, *Writing a decision record*).
|
||||
2. **Record the decisions** (below, *Writing a decision record*): one record per choice that could be
|
||||
reversed on its own, not one record for the whole effort.
|
||||
3. **Write the design.** Create the document under `03-DESIGN/01-to-be/` with frontmatter:
|
||||
|
||||
```yaml
|
||||
@@ -31,7 +36,9 @@ clauses in force for a topic or a record).
|
||||
---
|
||||
```
|
||||
|
||||
4. **Close the effort.** `knowledge-base.propose` the effort again, whole, with `status: graduated` and a
|
||||
4. **Hand it to the work breakdown.** The design goes on through playbook [04](04-build-handoff.md):
|
||||
milestones, issues, and each milestone built as one feature step (playbook [07](07-feature-branches.md)).
|
||||
5. **Close the effort.** `knowledge-base.propose` the effort again, whole, with `status: graduated` and a
|
||||
`became` link to the decision record (`"NNNN"`) and to the design document (its path). An effort that
|
||||
stops without a decision is proposed with `status: abandoned` and a `became` link to what replaced it.
|
||||
|
||||
@@ -66,7 +73,8 @@ clauses in force for a topic or a record).
|
||||
superseded record cited alone, an unknown topic, a retired word of the glossary, a daemon's line, a
|
||||
deferral no issue tracks. The knowledge base then reviews the proposal itself, through a judgement that
|
||||
sees only the record, the records it links and the rules, and answers `approve` or `changes`. On
|
||||
`changes` the record stays proposed with the findings: revise it by proposing it again, whole, with `why`.
|
||||
`changes` the record stays proposed with the findings: revise it by proposing it again, whole, with `why`;
|
||||
at most three rounds, then the findings go to the operator with a recommendation.
|
||||
On `approve` it is accepted, or, when it says `needs-operator`, it waits for the operator's Approve on a
|
||||
channel that proves who answered. Nothing proposed is in force. `knowledge-base.get` shows a record's
|
||||
reviews and the operator's ask when it waits for one.
|
||||
|
||||
@@ -1,35 +1,65 @@
|
||||
# Playbook 04 — Build handoff
|
||||
# Playbook 04 — Work breakdown and handoff
|
||||
|
||||
**Trigger.** A to-be design is settled and work is about to start in a code repository.
|
||||
**Trigger.** A to-be design is settled (playbook [02](02-graduation.md)) and its building is about to start.
|
||||
|
||||
**Who runs it.** Whoever starts the build.
|
||||
**Who runs it.** The lead who will run the feature, on the larger model. It is one sitting, bounded at about
|
||||
an hour, and said to the operator if it runs past.
|
||||
|
||||
## What it produces
|
||||
|
||||
A design becomes **milestones**, and each milestone becomes **issues**:
|
||||
|
||||
- **A milestone is usable on its own.** Once its delivery plan is done, the operator or the mesh can do
|
||||
something new, said in one sentence ("a Claude Code session is mirrored into a thread on the bus").
|
||||
A milestone that changes nothing anyone can use is not a milestone; fold it into the next.
|
||||
- **A milestone is one feature step** of playbook [07](07-feature-branches.md): prepared, developed in one to
|
||||
two hours, reviewed, checked and delivered as one. A milestone larger than that is split.
|
||||
- **An issue is one repository's part of a milestone,** sized so a fixing agent finishes it inside the
|
||||
milestone's development time. Issues that must land in order are linked `depends-on`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Confirm the design is settled.** Its frontmatter reads `status: designed`, and every
|
||||
claim in it traces to a decision record in the knowledge base (`knowledge-base.get` for each entry of
|
||||
its `decisions:`). An open question in the text is a
|
||||
reason to run playbook [01](01-research.md), not to start building around it.
|
||||
2. **Name the owner.** Set `code:` in the design's frontmatter to the repositories from
|
||||
[`repos.md`](../repos.md). If the repository does not exist yet, add it to `repos.md` in
|
||||
the same change.
|
||||
3. **Check the as-is.** Read the matching `03-DESIGN/00-as-is/` document. What is being
|
||||
replaced is stated there; if it is not, write it before changing it. Building against an
|
||||
undocumented as-is is how a shipped behaviour gets lost.
|
||||
4. **Flip the status.** `status: in-progress`, `updated:` today.
|
||||
5. **Build in the code repository.** HQ is not a code repository and never carries
|
||||
implementation.
|
||||
6. **On completion**, run the "when something ships" section of playbook
|
||||
1. **Confirm the design is settled.** Its frontmatter reads `status: designed`, and every claim in it traces
|
||||
to a decision record in the knowledge base (`knowledge-base.get` for each entry of its `decisions:`). An
|
||||
open question in the text is a reason to go back to playbook [01](01-research.md), not to build around it.
|
||||
2. **Name the owners.** Set `code:` in the design's frontmatter to the repositories from
|
||||
[`repos.md`](../repos.md). A repository that does not exist yet is added to `repos.md` in the same change.
|
||||
3. **Check the as-is.** Read the matching `03-DESIGN/00-as-is/` document. What is being replaced is stated
|
||||
there; if it is not, write it before changing it. Building against an undocumented as-is is how a shipped
|
||||
behaviour gets lost.
|
||||
4. **Break it into milestones.** Write them into the design, in a section *Milestones*: for each, its number,
|
||||
its one-sentence outcome, the repositories it touches, and what it depends on. Order them so each is
|
||||
usable as soon as it is delivered, the smallest useful one first.
|
||||
5. **Ask the operator to confirm the milestones**, in a short table: number, outcome, repositories, the
|
||||
parts each repository builds. Change what they change.
|
||||
6. **Open the issues.** For each confirmed milestone, `issue-tracker.open` one issue per repository part, its
|
||||
title the symptom or capability in plain words, its report naming the design, the milestone and the
|
||||
decisions it builds with; link the order with `depends-on`. Write the issue numbers into the milestone's
|
||||
entry in the design.
|
||||
7. **Flip the status.** `status: in-progress`, `updated:` today; the design change goes through playbook
|
||||
[08](08-small-fixes.md) (one hq pull request).
|
||||
8. **Hand each milestone to playbook 07, one at a time.** The milestone is the feature step; its issues are the
|
||||
step's issues; the next milestone starts when the previous one's delivery plan is done.
|
||||
9. **On completion of the last milestone**, run the "when something ships" section of playbook
|
||||
[02](02-graduation.md).
|
||||
|
||||
## Rules
|
||||
|
||||
- Every merge is a human checkpoint, without exception.
|
||||
- What the handoff leaves for later is an issue with an owner, named in the design or the decision that
|
||||
leaves it — never a sentence saying "later". The design's `code:` names who owns what is built; the issue
|
||||
names who owns what is not. the knowledge base's rule `deferral` checks the decision record when it is proposed
|
||||
- What the handoff leaves for later is an issue with an owner, named in the design or the decision that leaves
|
||||
it — never a sentence saying "later". The design's `code:` names who owns what is built; the issue names who
|
||||
owns what is not. The knowledge base's rule `deferral` checks the decision record when it is proposed
|
||||
([issue 354](../../04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/00-report.md)).
|
||||
- Never open a pull request unprompted. A permissions list saying it is allowed is not a
|
||||
request.
|
||||
- Work in an isolated worktree, never a shared checkout. A failed `cd` in a shared checkout
|
||||
commits to the wrong branch, and the error scrolls past.
|
||||
- HQ is not a code repository and never carries implementation.
|
||||
- How the building is done — worktrees, agents, review, checks, the ready mark and the delivery — is playbook
|
||||
07's, not this one's.
|
||||
|
||||
## How it is checked
|
||||
|
||||
Read by the lead and the reviewer today; a check in hq's merge-check is tracked by issue 845.
|
||||
|
||||
- A design in `status: in-progress` has a *Milestones* section, each milestone with its outcome and its
|
||||
issues; an in-progress design without one is the smell of building without a breakdown.
|
||||
- Every issue a milestone names exists in the tracker (`issue-tracker.get`), and an issue in `fixing` names
|
||||
its milestone in its report.
|
||||
- A milestone whose issues are all `resolved` has had its delivery plan done; its outcome sentence is proven on
|
||||
the mesh before the next milestone starts.
|
||||
|
||||
@@ -1,69 +1,213 @@
|
||||
# Playbook 07 — Feature branches across repos
|
||||
# Playbook 07 — Features
|
||||
|
||||
**Trigger.** Work that changes code — in one code repo or in several at once (`mesh-sdk`,
|
||||
`mesh-controller`, `mesh-catalog`, `mesh-host`, `mesh-lab`, and `hq` when a decision rides along).
|
||||
**Trigger.** A feature: work that changes code in one repository or several (`mesh-sdk`, `mesh-controller`,
|
||||
`mesh-catalog`, `mesh-host`, `mesh-lab`, and `hq` when a design rides along), too large to be one small fix.
|
||||
A small fix (one repository, one issue, about an hour of work) follows
|
||||
[playbook 08](08-small-fixes.md) instead.
|
||||
|
||||
**Who runs it.** Anyone who writes code, engineers and agents alike. Agents follow it exactly —
|
||||
it is the guard against the failure it was written for.
|
||||
**Who runs it.** Anyone who writes code, engineers and agents alike. Agents follow it exactly.
|
||||
|
||||
## The failure it prevents
|
||||
## The failures it prevents
|
||||
|
||||
A feature was worked as a branch-and-MR per *unit of thought* — one per decision, one per
|
||||
stacked increment — and each MR was treated as finished when it was *opened*, not when it was
|
||||
*merged*. Across repos the same feature took a different branch name in each. The MRs piled up
|
||||
unmerged: one session left **sixteen** stacked intermediate MRs that had to be consolidated and
|
||||
closed by hand. An MR is a review checkpoint, not a scratchpad.
|
||||
- **Pull requests as scratchpads.** A feature worked as one pull request per unit of thought left sixteen
|
||||
stacked pull requests unmerged, consolidated by hand afterwards. A pull request is the end of a feature,
|
||||
not a step in it.
|
||||
- **Waiting on checks mid-feature.** A feature worked as one pull request per issue into protected
|
||||
branches spent most of a day waiting for merge checks while the code was ready.
|
||||
- **Context bought again and again.** A fresh agent per small issue spends most of its run reading the
|
||||
playbook, the issue and the code before it writes a line. On 2026-10-14 an issue sized for five to ten
|
||||
minutes took forty-one.
|
||||
- **Finished work that never ships.** A feature that kept taking new issues held thirteen landed issues
|
||||
undelivered for a day: an issue done at 21:09 was still not on the machines at 15:00 the next day.
|
||||
- **Work that waits and nobody notices.** Most lost time was not work at all, but a check, a delivery plan
|
||||
or an agent waiting on something that had stopped, unseen.
|
||||
|
||||
## The rule
|
||||
## The shape
|
||||
|
||||
One feature is **one branch name**, **one worktree per repo**, **one MR per repo**, opened
|
||||
**once, at the end**.
|
||||
One feature is **one branch name**, **one worktree per repository**, **one pull request per repository**,
|
||||
opened **once**, after the review. It runs in five phases, and its time is bounded:
|
||||
|
||||
1. **Name the feature once.** `feat/<slug>`. The *same* branch name in every repo the feature
|
||||
touches — never a different name per repo, never a fresh branch per increment within the
|
||||
feature.
|
||||
2. **Isolate each repo.** One git worktree per touched repo under `.work/<slug>/<repo>`, branched
|
||||
off `main`:
|
||||
```
|
||||
git worktree add .work/<slug>/<repo> -b feat/<slug> origin/main
|
||||
```
|
||||
Parallel features never collide, and no shared checkout is edited.
|
||||
| Phase | What happens | Bound |
|
||||
|---|---|---|
|
||||
| 0. Prepare | the lead settles the open questions and writes one context file per repository | 15 minutes |
|
||||
| 1. Develop | one fixing agent per repository builds every issue locally, unit tests only | 1 to 2 hours |
|
||||
| 2. Review | one reviewer reads all of it; reports go both ways; at most 3 rounds | 15 minutes |
|
||||
| 3. Check | push; the forge's merge checks run the integration tests; no further review | one check run |
|
||||
| 4. Deliver | merge; the delivery plan builds, publishes and sends the whole feature | its delivery plan |
|
||||
|
||||
**Add the untouched siblings the lab reads.** The lab beds find the other repositories by
|
||||
sibling path from the lab checkout (`../mesh-tools/module.json`, `../mesh-sdk`, …), the way
|
||||
the main layout has them. A `.work/<slug>/` directory holding only the touched repos fails a
|
||||
bed at once with `no manifest for mesh-tools at …/.work/<slug>/mesh-tools/module.json`, after
|
||||
genesis has already passed. Give the directory those repos as **detached worktrees on `main`**
|
||||
— never symlinks:
|
||||
```
|
||||
git worktree add --detach .work/<slug>/mesh-tools main
|
||||
```
|
||||
3. **Commit as you go — locally.** Increments land on the one branch. Nothing is pushed and no
|
||||
MR is opened mid-feature.
|
||||
4. **Finish, then publish.** When the whole feature is done — every repo, tests green — push
|
||||
every branch and open **one MR per touched repo**, together.
|
||||
5. **Merge promptly, once approved.** Every merge into `main` is notified and approved
|
||||
(ADR 0023); once it is, merge —
|
||||
do not leave it sitting. The branch is deleted on merge.
|
||||
6. **Leave nothing behind.** After the MRs merge, no `feat/<slug>` branch and no `.work/<slug>`
|
||||
worktree survive.
|
||||
A feature that cannot be developed in about two hours is split into **steps**. Each step runs all five
|
||||
phases on its own; the next step starts only when the previous step's delivery plan is done. A step is the
|
||||
unit that ships, so nothing finished waits for work that is not.
|
||||
|
||||
## What this is not
|
||||
**Scope is fixed at phase 0.** An issue found during a feature goes to the next step or the next feature,
|
||||
never into the running one, unless it blocks the running one.
|
||||
|
||||
- **Not a licence to batch unbounded work.** A feature is a *bounded* unit; if it sprawls for
|
||||
days, end-of-feature bloat merely replaces per-increment bloat. Split it into features, each
|
||||
its own branch and MR.
|
||||
- **Not a second trunk.** Every repo branches off `main`. There is no longer an `initialization`
|
||||
trunk.
|
||||
## Phase 0 — Prepare (the lead, larger model)
|
||||
|
||||
1. **Name the feature once:** `feat/<slug>`, the same branch name in every repository it touches. The
|
||||
branch is never protected.
|
||||
2. **Make the worktrees:** one per touched repository under `.work/<slug>/<repo>`, branched off `main`
|
||||
(`git worktree add .work/<slug>/<repo> -b feat/<slug> origin/main`). The lab beds find the other
|
||||
repositories by sibling path, so add the untouched siblings the lab reads as detached worktrees on
|
||||
`main` (`git worktree add --detach .work/<slug>/mesh-tools main`), never symlinks.
|
||||
3. **Settle every open question** in the feature's issues before any agent starts: on the issue, through
|
||||
the knowledge base, or by asking the operator. A fixing agent that meets an undecided design question
|
||||
stops, and its run is spent again.
|
||||
4. **Write one context file per repository**, `.work/<slug>/<repo>.context.md`. It is what a fixing agent
|
||||
reads instead of rediscovering the code, and it is reused by every agent and every round:
|
||||
- the issues it builds, in order, each with: the files and functions it changes, the decisions it builds
|
||||
with (quoted, with where each was decided), the test to write first, done-when in one sentence;
|
||||
- the repository's map for this feature: the packages and files involved, with one line each on what they
|
||||
hold, and the patterns to copy (file:line);
|
||||
- what not to touch, and what lands beside it in other repositories;
|
||||
- the local facts: the worktree, the branch, the check commands, a temporary directory for tests outside
|
||||
the home (`TMPDIR`), a throwaway database server if the store tests need one.
|
||||
When the reading is large, a read-only agent on the smaller model answers the lead's exact questions with
|
||||
file:line, and the lead writes the file from its answers.
|
||||
5. **Move the issues to `fixing`**, `by` naming the feature's agent for that repository.
|
||||
|
||||
## Phase 1 — Develop (one fixing agent per repository, smaller model)
|
||||
|
||||
- **One agent per repository, kept for the whole feature.** It reads its context file once and builds the
|
||||
issues one after another, committing each locally with a message naming the issue ("Keep the assignments
|
||||
on the bus (issue 652)"). It is not replaced per issue: the lead continues the same agent (by its id) with
|
||||
the next instruction, so its context is paid for once.
|
||||
- **Unit tests only.** Each issue's first test fails before its change and passes after; `gofmt`, `go vet`
|
||||
and the repository's unit tests stay green. The integration tests (`merge-check.sh` with the store and the
|
||||
bus) wait for phase 3.
|
||||
- **Nothing is pushed** and no pull request is opened in this phase. Never land on a red check: run the
|
||||
checks, read the result, then land in a separate step.
|
||||
- **The fixing agent's brief** (the lead gives it with the context file): name your issue in every step and
|
||||
message ("issue 805: landing in mesh-catalog"); reach the mesh only through the mesh MCP server, never ssh;
|
||||
never push, open, merge or close a pull request; never write to a store's database by hand, a schema change
|
||||
is a numbered migration and the consolidated schema; Go only unless the context file says otherwise; never
|
||||
move another issue. Report back in under 150 words: commits, tests added and that they failed first, the
|
||||
check results, anything you stopped on.
|
||||
- **An agent that finds the code differs from its context file stops and reports** (file:line); the lead
|
||||
settles it, updates the context file, and continues the agent.
|
||||
- **Two repositories that depend on each other** keep their order in the context files: the reader before the
|
||||
writer, the controller's new manifest word before the catalogue manifest that uses it.
|
||||
- **Several agents in one repository** (only for a large repository with independent areas): each works on a
|
||||
local branch of its own from the feature branch's head and lands by fast-forward under a lock
|
||||
(`flock <feature worktree>.lock git -C <feature worktree> merge --ff-only <its branch>`), after its unit
|
||||
tests pass; if another landed first, it merges again and repeats. In a shared clone, fetch the feature
|
||||
branch by name before branching from it. Never resolve a conflict with a blind `git merge-file --union`.
|
||||
|
||||
## Phase 2 — Review (one reviewer, larger model; at most 3 rounds, about 15 minutes)
|
||||
|
||||
1. **Bring every feature branch up to the current trunk** (fetch by name, then merge), so the reviewed tree is
|
||||
the tree that is pushed.
|
||||
2. **One reviewer reads every repository's local commits** together, against the context files and the
|
||||
decisions. Its report is per repository: each finding as file:line, the failure it causes, and the fix.
|
||||
It says which round it is in ("round 1 of at most 3").
|
||||
3. **The report goes to the same fixing agents** that wrote the code, continued by their ids, so they fix
|
||||
with the context they already hold. Each answers the reviewer: what it changed, per finding, or why not.
|
||||
4. **The reviewer reads the answers and the new commits**, and approves or reports again. From round 2 it
|
||||
asks changes only for a finding that makes the feature wrong; smaller points become issues of their own.
|
||||
After round 3 the lead stops and brings the operator what is left, with a recommendation.
|
||||
|
||||
## Phase 3 — Check (no review)
|
||||
|
||||
1. **Push every feature branch and open one pull request per repository, together.** Pull requests in two or
|
||||
more repositories that share the branch name are one delivery group (ADR 0239 §2).
|
||||
Where a session pushes through the forge tool `gitea_commit` (one commit on the forge's current base), the
|
||||
reviewer first confirms that commit's tree equals the reviewed one.
|
||||
2. **The forge's merge checks run the integration tests once** (`mesh/merge-gate`, `mesh/repo-check`). No
|
||||
second review: phase 2 reviewed this tree.
|
||||
3. **A check that fails** goes back to the reviewer with the failure, for **one more cycle** of phase 2 with
|
||||
the same fixing agents; then push once again.
|
||||
4. **Mark each pull request ready** (`mesh/ready` and a comment `ready at <full sha>`) once the operator's word
|
||||
for the feature is given; mesh-delivery merges it when its checks pass (ADR 0364).
|
||||
|
||||
**One exception: a catalogue change that uses a new manifest word.** The catalogue's merge check reads
|
||||
manifests with the controller the mesh runs, so the controller's pull request is merged and delivered first;
|
||||
then `mesh-delivery.recheck` the catalogue's and merge it on the new verdict. Plan such a feature as two steps.
|
||||
|
||||
## Phase 4 — Deliver
|
||||
|
||||
The merge starts the delivery plan: build, publish, send. The lead follows it to done
|
||||
(`mesh-controller.plans` with `wait` and `until: done`), then resolves the feature's issues with `fixed-by`
|
||||
naming its pull requests, and removes what is left: no `feat/<slug>` branch and no `.work/<slug>` worktree
|
||||
survive.
|
||||
|
||||
## Models
|
||||
|
||||
| Who | Model | Why |
|
||||
|---|---|---|
|
||||
| The lead (phase 0 decisions, the context files, following and reporting) | larger | judgement |
|
||||
| Read-only questions about the code for the context files | smaller | reading, answered with file:line |
|
||||
| Fixing agents (phase 1, and the fixing in phases 2 and 3) | smaller | building to a written context |
|
||||
| The reviewer (phase 2, and the extra cycle in phase 3) | larger | judgement, once per round |
|
||||
| Decision records | larger | judgement |
|
||||
|
||||
Every agent the mesh runs draws on one weekly allowance shared by every session, and the larger model
|
||||
spends it many times faster. Up to five agents on the smaller model at once, at most two subagents on the
|
||||
larger one (the lead not counted).
|
||||
|
||||
## Following and reporting
|
||||
|
||||
Most lost time is waiting nobody saw. The lead therefore keeps a **wait list**: everything the feature waits
|
||||
on, with when it started and its bound.
|
||||
|
||||
| Waiting on | Bound | Past the bound |
|
||||
|---|---|---|
|
||||
| a fixing agent's issue | 30 minutes | ask it where it stands; split or take over |
|
||||
| the review | 15 minutes a round | ask the reviewer; stop at round 3 |
|
||||
| a merge check | 15 minutes | read the build queue (`mesh-controller.queue`) and the check's log |
|
||||
| a merge after ready | 5 minutes after the checks pass | read `mesh-delivery.show` for the delivery |
|
||||
| a delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
|
||||
|
||||
**Every ten minutes, and at once when something lands, stops or needs the operator,** the lead reports, in
|
||||
one of the two layouts below. Times are local and absolute; every number has its unit and its bound or
|
||||
baseline; an issue is named by its number and a few words, never by its number alone.
|
||||
|
||||
**The full layout** (in the session), the issues as a tree: grouped by repository, each issue under the one it
|
||||
waits for, marked ✓ landed, ▶ fixing, ⏸ waiting, ✗ stopped:
|
||||
|
||||
```
|
||||
Feature <slug> · step <n> of <m> · phase <name>, <minutes used> of <bound> min
|
||||
Issues: <landed> of <total> landed · <fixing> fixing · <waiting> waiting
|
||||
|
||||
mesh-controller (agent: fixing, 2 of 3 landed)
|
||||
├── ✓ 652 assignments on the bus landed 14:02
|
||||
├── ▶ 736 placement words leave the layers fixing since 13:51, 23 of 30 min
|
||||
│ └── ⏸ 751 config keys named (mesh-catalog) waits for 736
|
||||
└── ✓ 654 a file names its template landed 13:20
|
||||
mesh-catalog (agent: waiting)
|
||||
└── ⏸ 751 (above, under 736)
|
||||
|
||||
Past its bound: <the wait, how long, what is being done> — or "none"
|
||||
Needs you: <one line per question, with my recommendation> — or "nothing"
|
||||
Allowance: week <n> % (<+d> since the last report), session <n> %
|
||||
Next: <the next action, one line>
|
||||
```
|
||||
|
||||
**The short layout** (on a channel such as Telegram, or when nothing changed but time):
|
||||
|
||||
```
|
||||
<slug> step <n>/<m> · <phase> <used>/<bound> min · <landed>/<total> landed · past bound: <none | what> · needs you: <nothing | what> · week <n> %
|
||||
```
|
||||
|
||||
A report says what changed since the last one first; an unchanged table is not repeated, the short layout
|
||||
says it instead.
|
||||
|
||||
## The context of the lead's own session
|
||||
|
||||
At the end of each phase, and whenever the session's context grows large or spread over unrelated work, the
|
||||
lead prepares a `/clear` as the mesh's instruction "prepare before clear" says: a save point with the
|
||||
feature's phase, its branches and heads, the context files' paths, the agents' ids, the wait list and the
|
||||
next action; then the prompt the operator pastes. The context files make the feature resumable after a clear.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The end state is visible, and its absence is the smell:
|
||||
- A feature's issues show their commits on the feature branch (`git log --grep 'issue <n>'`), and no pull
|
||||
request into a feature branch exists. A pull request per issue, or an agent waiting on a merge check in
|
||||
phase 1, is the failure this playbook prevents.
|
||||
- A context file exists per touched repository under `.work/<slug>/` while the feature runs.
|
||||
- After a feature's delivery, `git branch -r | grep feat/<slug>` and `git worktree list` return nothing for it.
|
||||
- More than one open pull request in a repository that share no feature name, or a feature step whose issues
|
||||
stay in `fixing` for more than a working day, is the smell: stop and bring it to the operator.
|
||||
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it cannot find is a
|
||||
missing sibling worktree, not a mesh fault.
|
||||
|
||||
- After a feature merges, `git branch -r | grep feat/<slug>` and `git worktree list` return
|
||||
nothing for it. A surviving branch or worktree means step 6 was skipped.
|
||||
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it
|
||||
cannot find is a missing sibling worktree, not a mesh fault.
|
||||
- More than one open MR in a repo that share no feature name, or a stack of MRs none of which is
|
||||
merged, is the failure this playbook exists to prevent — stop and consolidate before opening
|
||||
more.
|
||||
When this playbook meets a problem, the lesson is added here, so the next feature avoids it.
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
# Playbook 08 — Small fixes
|
||||
|
||||
**Trigger.** One issue in one repository, small enough to build in about an hour: a bug with a known cause,
|
||||
a missing check, a wording, a playbook or design edit. Anything larger, or anything that needs a change in a
|
||||
second repository first, is a feature and follows [playbook 07](07-feature-branches.md).
|
||||
|
||||
**Related issues in one fix.** A small fix may cover two or three related issues when they are in the same
|
||||
repository, mostly in the same module, and together still fit in about an hour: they touch the same files, or
|
||||
one would be wrong or unsafe without the other (switching a clipboard's history on before its history tool stops
|
||||
handing out what was copied). Each issue is named in the commit and the pull request and moved to `fixing`,
|
||||
and each is resolved on its own once the fix is delivered. More issues, a second repository, or more than an
|
||||
hour is a feature.
|
||||
|
||||
**Who runs it.** Anyone who writes code, engineers and agents alike.
|
||||
|
||||
## The aim
|
||||
|
||||
A small fix spends its time on the fix, and runs the long part once: **one review, one push, one check run,
|
||||
one delivery**. Everything before the push is local and fast.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Read the issue and the code it names** (the session's own model, or one read-only agent on the smaller
|
||||
model answering exact questions with file:line). If a design question is open, it is settled before any
|
||||
code is written: on the issue, through the knowledge base, or by the operator. A fix that turns out to
|
||||
need a decision or a second repository stops here and becomes a feature.
|
||||
2. **Move the issue to `fixing`** with `by` as who@node.
|
||||
3. **Build on a local branch** `fix/<issue>-<slug>`, in a worktree of its own branched from `main`. Write the
|
||||
test that fails today first; make the change; keep `gofmt`, `go vet` and the repository's unit tests green.
|
||||
Commit, naming the issue. For a playbook or design edit, run the repository's own checks (in hq,
|
||||
`sh merge-check.sh`).
|
||||
4. **One review of the local commit, before anything is pushed** (a reviewer that did not write it, larger
|
||||
model; at most 3 rounds, as playbook 07; from round 2 only what is wrong). The reviewer's findings go back to the same agent that wrote the fix, continued by
|
||||
its id, so the fix keeps its context. A playbook or design edit of a few lines may skip the reviewer: the
|
||||
operator's word on the pull request is its review.
|
||||
5. **Push the branch, try it, then open the pull request.** `mesh-controller.try` with the repository and the
|
||||
branch runs the merge check in the check's own toolchain before any status is set; open the pull request
|
||||
only once it passes. A local run never stands in for it: the check's containers carry other tools than the
|
||||
nodes (a test that called util-linux `flock -E` passed on a node and failed on BusyBox's in the check). The
|
||||
forge's merge checks run the integration tests. A failed check
|
||||
goes back to the same agent with the failure, and the reviewer reads only the change it makes.
|
||||
6. **Mark it ready** (`mesh/ready` and `ready at <full sha>`) once the operator's word is given; mesh-delivery
|
||||
merges it when its checks pass.
|
||||
7. **Follow its delivery plan to done** (`mesh-controller.plans` with `wait` and `until: done`), prove the fix on
|
||||
the mesh (the symptom gone, measured), and resolve the issue with `located-in` and `fixed-by` the same hour.
|
||||
|
||||
## Bounds
|
||||
|
||||
| Waiting on | Bound | Past the bound |
|
||||
|---|---|---|
|
||||
| building the fix | 1 hour | it is a feature: stop and plan it under playbook 07 |
|
||||
| the review | 15 minutes | ask the reviewer where it stands |
|
||||
| the merge check | 15 minutes | read `mesh-controller.queue` and the check's log |
|
||||
| the merge after ready | 5 minutes after the checks pass | read `mesh-delivery.show` for the delivery |
|
||||
| the delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
|
||||
|
||||
A wait past its bound is said to the operator at once, with what is being done about it.
|
||||
|
||||
## Models
|
||||
|
||||
The fixing agent runs on the smaller model; the reviewer on the larger. A session fixing alone does the
|
||||
reading itself and uses one subagent only for the review.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- A small fix has one pull request with one or few commits naming its issue, opened after its review: a pull
|
||||
request with many heads is a fix reviewed after its push.
|
||||
- Its issue moves to `resolved` within the hour its delivery plan ends.
|
||||
@@ -340,7 +340,7 @@ once, and maintain for a year. The modules are the input; a person reads what on
|
||||
writes what it declares tomorrow.
|
||||
|
||||
**It also changes what "safe" means for the system being retired.** A fix to it has to be safe on
|
||||
its own, because there is no careful walk to sequence it into: the thing is being switched off
|
||||
its own, because there is no careful rollout to sequence it into: the thing is being switched off
|
||||
by hand, not managed into retirement. A change needing three steps in the right order is a change
|
||||
that will be half-applied.
|
||||
|
||||
|
||||
@@ -22,8 +22,10 @@ code:
|
||||
- mesh-host internal/apply (the service that reflects a rule set; a whole file handed to its new owner)
|
||||
- mesh-controller cmd/mesh-controller/nodes.go (a token issued for a machine's tunnel key, ADR 0169)
|
||||
- mesh-host cmd/mesh-host/join_tunnel.go (a machine joins through the tunnel, ADR 0169)
|
||||
updated: 2026-10-13
|
||||
updated: 2026-10-14
|
||||
decisions:
|
||||
- "ADR 0241"
|
||||
- "ADR 0117"
|
||||
- "ADR 0340"
|
||||
- "ADR 0247"
|
||||
- "ADR 0226"
|
||||
@@ -1119,6 +1121,113 @@ against today's behaviour, where both are always composed; an unrouted endpoint
|
||||
named; an assignment naming an endpoint the module does not declare refused where it is said; and a
|
||||
silent assignment rendering byte-identically to today.
|
||||
|
||||
## The five through the network layers
|
||||
|
||||
*2026-10-09. A reading aid: it decides nothing and states no rule. Each claim names the section or the
|
||||
record that decided it.*
|
||||
|
||||
The OSI model is the standard seven-layer picture of a network: every layer wraps what the layer above
|
||||
hands it, and the receiving side unwraps in reverse. The five contexts of this document act at
|
||||
different layers, and seeing which explains why they fail independently. A resolver that stopped
|
||||
answering leaves the private network working, and a tunnel that cannot handshake leaves every name
|
||||
resolving to an address nothing can reach.
|
||||
|
||||
| Layer | What it carries | Where the mesh acts there |
|
||||
|---|---|---|
|
||||
| 7 Application | the message itself | the bus (ADR 0106); resolution (§2); exposure, where the proxy chooses a route by the name a request asks for (§3) |
|
||||
| 6 Presentation | how what is said is encoded and encrypted | certificates (§5): TLS, which is usually drawn between 4 and 7 and placed here by the book |
|
||||
| 5 Session | a conversation kept open | a node's link to the bus, opened once and reopened after the machine sleeps |
|
||||
| 4 Transport | ports, TCP and UDP | filtering (§4), which opens a port to a source; the private network itself travels as UDP |
|
||||
| 3 Network | addresses and the path between them | the private network (§1): every node's address, the peer graph, the hub relaying |
|
||||
| 2 Data link | frames on one network segment | the machine's own uplink (ADR 0117), on the underlay |
|
||||
| 1 Physical | cable, fibre, radio | nothing of the mesh's |
|
||||
|
||||
### Two networks, one inside the other
|
||||
|
||||
**The private network is a layer 3 network carried at layer 4 of the underlay.** A packet for another
|
||||
node's private address is encrypted whole and becomes the payload of a UDP datagram addressed to a
|
||||
peer's underlay address. Every hop on the underlay reads only the outer envelope, up to its layer 3,
|
||||
and never sees inside.
|
||||
|
||||
```
|
||||
underlay │ L1 radio │ L2 frame │ L3 to the hub's public address │ L4 UDP │ ▓▓ encrypted ▓▓ │
|
||||
│
|
||||
decrypted at the hub │
|
||||
┌────────────────────────────────────────────────┘
|
||||
▼
|
||||
private network │ L3 to a private address │ L4 TCP │ L5–L7 a message on the bus │
|
||||
```
|
||||
|
||||
### The shape the layers are drawn on
|
||||
|
||||
The shape is §1's: a hub, direct peering between nodes at the same site, and a roaming node that goes
|
||||
through the hub only. In a small mesh one node is both the anchor and the hub, and holds the bus.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ hub and anchor — reachable from outside │
|
||||
│ holds the bus │
|
||||
└─────▲──────────────────▲─────────────▲───┘
|
||||
│ │ │
|
||||
│ each tunnel is dialled out by the node below it
|
||||
│ │ │
|
||||
┌─────┴─────┐ ┌──────┴──────┐ ┌────┴────────┐
|
||||
│ laptop │ │ home-server │ │ home-server │
|
||||
│ no site: │ └──────▲──────┘ └──────▲──────┘
|
||||
│ hub only │ └──── direct ───┘
|
||||
└───────────┘ same site, peered
|
||||
```
|
||||
|
||||
**Every arrow points at the hub.** A node with no public address still has a tunnel, because it dials
|
||||
out and the hub answers; only the hub's tunnel must be dialable, at a stable address
|
||||
(ADR 0169).
|
||||
|
||||
### One tool call, from the laptop to a home-server
|
||||
|
||||
The laptop wraps the call one layer at a time:
|
||||
|
||||
| Layer | On the laptop |
|
||||
|---|---|
|
||||
| 7 | a message on the bus naming the tool and its arguments |
|
||||
| 6 | the message encoded for the wire |
|
||||
| 5 | the laptop's link to the bus, already open |
|
||||
| 4 | TCP to the bus at its private address |
|
||||
| 3 | a packet from the laptop's private address to the bus's |
|
||||
| 3 inside 4 | the tunnel: that packet encrypted, in a UDP datagram to the hub's public address |
|
||||
| 2 | a frame to the gateway of whatever network the laptop is on |
|
||||
| 1 | radio |
|
||||
|
||||
**At the hub the outer envelope comes off** — frame, packet, datagram — and the tunnel decrypts what was
|
||||
inside. The inner packet is addressed to the bus, which is on this node, so the hub keeps unwrapping up
|
||||
to layer 7. The bus finds the home-server's subscription and sends the message down the home-server's
|
||||
own link, which that node opened earlier through its own tunnel. Neither spoke ever accepted a
|
||||
connection; both dialled the hub.
|
||||
|
||||
**How far the hub unwraps depends on what the packet is for.** A packet addressed to a home-server's
|
||||
private address, not to the bus, is opened only to layer 3: the hub relays it, encrypted again for the
|
||||
home-server's tunnel. That is the whole of the glossary's *a hub relays for nodes that cannot reach
|
||||
each other directly*. Two home-servers at one site skip the hub: their tunnel crosses the home network
|
||||
at layer 2, and nothing of theirs reaches the hub at all.
|
||||
|
||||
**Why a roaming node has one path is a layer 3 fact.** A more specific route to an endpoint that is
|
||||
gone takes the packets and drops them; it does not fall back (§1). A laptop that moves between networks
|
||||
therefore has exactly one route, the hub's.
|
||||
|
||||
### Reading a failure by layer
|
||||
|
||||
The machine's own network checks
|
||||
(ADR 0241)
|
||||
each look at one layer:
|
||||
|
||||
| Check | Layer | Failing alone means |
|
||||
|---|---|---|
|
||||
| **route** — a default route | 3, underlay | the machine has no way out; nothing above can work |
|
||||
| **tunnel** — a handshake with the hub within five minutes | 3 carried at 4 | the underlay works and the private network does not: a key, the hub's address, or a filter on the way |
|
||||
| **bus** — the link is open | 5 to 7 | the private network works and the bus, or the node's account on it, does not |
|
||||
| **resolv-conf** and **names** | 7 | addresses work and names do not, as when a VPN client rewrites the machine's resolver file |
|
||||
|
||||
Read them from the bottom. A layer that fails explains every failure above it, and never one below it.
|
||||
|
||||
## What this removes
|
||||
|
||||
The list is worth having in one place, because it is most of the argument:
|
||||
|
||||
@@ -54,7 +54,7 @@ rather than two kinds of thing
|
||||
node** — it is a machine with a program on it that has not been told which mesh it belongs to.
|
||||
|
||||
There is no state for *the first node*. That is the point of
|
||||
ADR 0004: the first node walks the
|
||||
ADR 0004: the first node rollouts the
|
||||
same path, in an unusual order.
|
||||
|
||||
---
|
||||
@@ -239,7 +239,7 @@ carries, to the machine it is already on.
|
||||
**After step 3 the first node is not special in any way**, which is the property `adopt.sh` and
|
||||
the bootstrap script never had. Its specialness lasted two commands.
|
||||
|
||||
**And enrolment is exercised on node one.** The path every other node depends on is walked
|
||||
**And enrolment is tried on node one.** The path every other node depends on is exercised
|
||||
immediately, against a controller on the same machine, rather than being written and first
|
||||
used months later on node two.
|
||||
|
||||
@@ -524,7 +524,7 @@ remains locally authoritative for *operating*; the copy exists only for this.
|
||||
|
||||
The node-engine is delivered like anything else
|
||||
(ADR 0010), and this is worth
|
||||
walking through because tier 0 looks like it should be special and is not.
|
||||
going through step by step because tier 0 looks like it should be special and is not.
|
||||
|
||||
```
|
||||
push to mesh-host
|
||||
|
||||
@@ -277,15 +277,15 @@ A transition the table does not hold is refused. Two or more deliveries that sha
|
||||
together as one future state of the mesh.
|
||||
|
||||
**This answers the first open question above.** A fit artifact does not declare itself. Off the trunk it is
|
||||
only checked. On the trunk it is published, and its walk across the machines starts when its delivery says
|
||||
only checked. On the trunk it is published, and its rollout across the machines starts when its delivery says
|
||||
so: by itself for the core, which cannot wait for anything to be repaired, and by `mesh-delivery` for
|
||||
everything else while that module is held. A person can always deliver by hand.
|
||||
|
||||
The controller keeps the comparison, the planner, the gate, sending and rollback. `mesh-delivery` keeps the
|
||||
record and the order, and asks.
|
||||
|
||||
**Merges are walked in batches** (2026-10-10, by ADR 0276). The controller collects
|
||||
**Merges are rolled out in batches** (2026-10-10, by ADR 0276). The controller collects
|
||||
merges in a rolling merge window, 90 seconds by default and at most 10 minutes, and plans each batch once, as
|
||||
one walk with at most one commit per repository: the latest of its branch, which contains the earlier ones.
|
||||
One walk is open at a time, and a walk that started is never superseded. `plans` shows a batch while it
|
||||
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's walk.
|
||||
one rollout with at most one commit per repository: the latest of its branch, which contains the earlier ones.
|
||||
One rollout is open at a time, and a rollout that started is never superseded. `plans` shows a batch while it
|
||||
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's rollout.
|
||||
|
||||
@@ -215,7 +215,7 @@ and recreate a container naming a step after the step ran.
|
||||
*Written 2026-10-02, from ADR 0189
|
||||
and [issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).*
|
||||
|
||||
Some work cannot be done underneath a running service: an artifact store's collector walks the
|
||||
Some work cannot be done underneath a running service: an artifact store's collector rollouts the
|
||||
storage and requires every writer stopped. A `run-once` step runs *beside* containers and a
|
||||
scheduled one is the same container again, so until this a module had no way to say it — and the
|
||||
mesh inherited a store that has never collected anything, because the predecessor said it with a
|
||||
|
||||
@@ -163,7 +163,7 @@ again restarts nothing. The provider asks when it starts, when the software's va
|
||||
consumer that binds in between holds a value the mesh made, which the software refuses, for at most the hour until the next ask. A change is recorded in the
|
||||
hand-act log as a decision, not a repair, and `rotate` of such a provision is refused. *How it is checked:*
|
||||
the tests ADR 0305 §13 names, in both repositories; the live row is `hand-acts` showing `secret issued` for
|
||||
the media server's provision once both walks are done.
|
||||
the media server's provision once both rollouts are done.
|
||||
|
||||
## Beyond generate and hold
|
||||
|
||||
|
||||
@@ -75,7 +75,7 @@ and uses, and the subjects, streams, consumers and permissions all follow from t
|
||||
(ADR 0039).
|
||||
|
||||
**Added 2026-10-10** (ADR 0289): the controller's own records join
|
||||
the bus as **record mirrors** — nodes, assignments, walks, builds and settings, each a bucket written from
|
||||
the bus as **record mirrors** — nodes, assignments, rollouts, builds and settings, each a bucket written from
|
||||
the controller's store only when the record changed, with one event per write, and audited against the
|
||||
store on every self-check run. Designed in [to-be 54](54-the-controllers-records-on-the-bus.md). The
|
||||
rule it states for every bucket, that a write is made only when what the bucket describes changed, holds
|
||||
|
||||
@@ -709,7 +709,7 @@ healthy while reacting to nothing.
|
||||
Found on the way: **no build had ever recorded what it stood on.** A recipe reads its base from
|
||||
a build argument, so the digest was never in the file the builder derived edges from, and every
|
||||
order that says *bases first* — `build --on`, `build --behind`, the merge follow-up of
|
||||
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — walked a
|
||||
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — traversed a
|
||||
graph with no edges. The builder now reports the bases it was handed, the controller records
|
||||
them by artifact path, and the graph is read from the newest build of each module — a recorded
|
||||
manifest carries no `build.on`, so the edge is derived from the build or it does not exist.
|
||||
|
||||
@@ -141,7 +141,7 @@ Revision, ADR 0218. Three rules on how a plan delivers what it built.
|
||||
A membership that could not be issued fails the send.
|
||||
- **One machine first.** Unless a module's upgrade policy says *together*, a plan sends it to one machine,
|
||||
the first by name, and to the rest only once that machine reports the new declaration applied and
|
||||
current. A first machine that fails stops the module's walk there, with the reason in the plan.
|
||||
current. A first machine that fails stops the module's rollout there, with the reason in the plan.
|
||||
- **A newer merge takes over.** A merge's plan supersedes every older open plan for the same repository
|
||||
and branch, and takes in the modules they had not yet built. A plan that waits on nothing can be closed
|
||||
by hand, by its id.
|
||||
@@ -182,7 +182,7 @@ policy or a plan is holding back
|
||||
builds are not known is left too. The push names the machine, the module, the builds and the reason,
|
||||
and says `push <node>` sends it.
|
||||
- The machine a push names, a push of every machine, and `push --behind` send held builds as before.
|
||||
Walking a change through the mesh under `record` is one `push <node>` per machine.
|
||||
Rolling out a change through the mesh under `record` is one `push <node>` per machine.
|
||||
|
||||
## Why now, and why not yet
|
||||
|
||||
|
||||
@@ -253,7 +253,7 @@ the controller creates from the catalogue, mesh-wide, existing from registration
|
||||
watch before the owner runs anywhere ([design 25](25-the-bus-on-nats.md) §3). Every instance of the
|
||||
owner writes; a reader reads and watches. A key may name a machine by the module's own convention;
|
||||
the mesh keeps one bucket per name, not one per machine, because "every server, for every machine"
|
||||
is then one list rather than a walk.
|
||||
is then one list rather than a rollout.
|
||||
|
||||
What a module sees is what it named. Its assignment's membership lists its buckets by those names,
|
||||
with whether it may write, and the runtime answers `get`, `put`, `delete`, `keys` and `watch` for
|
||||
@@ -338,7 +338,7 @@ module, version, digest. An event again: a fact about the builder.
|
||||
|
||||
**The build cascade is that event fanning out through a graph the mesh already has.** A module
|
||||
whose image is built *on* another's artifact declares that in `build.on`. So `built` reaches the
|
||||
controller, which walks the declared graph and submits rebuild jobs for everything downstream. A
|
||||
controller, which traverses the declared graph and submits rebuild jobs for everything downstream. A
|
||||
dependency cascade is not special machinery — it is one event, one derived graph, and the same job
|
||||
queue.
|
||||
|
||||
@@ -351,7 +351,7 @@ entrypoint that brings its state to the shape that version needs — the same vo
|
||||
tools and its provisioner, and nothing about how a machine runs it. The mesh runs that entrypoint as it
|
||||
runs the module's own code, to completion, in the module's own context, and a version whose preparation
|
||||
did not succeed does not run: the step gates that module and nothing else on the machine
|
||||
(ADR 0136), and the walk stops
|
||||
(ADR 0136), and the rollout stops
|
||||
at the first machine that did not take it
|
||||
(ADR 0135, superseding
|
||||
ADR 0133).
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-catalog, mesh-controller, mesh-host]
|
||||
updated: 2026-10-12
|
||||
updated: 2026-10-14
|
||||
decisions:
|
||||
- "ADR 0369"
|
||||
- "ADR 0303"
|
||||
- "ADR 0215"
|
||||
- "ADR 0173"
|
||||
@@ -75,7 +76,7 @@ In order:
|
||||
| 8 | `docker` | the runtime's packages, base configuration, group | four configurations, one owner on one machine |
|
||||
| 9 | `ssh-client` | everything under `~/.ssh` (research 027/03) | a predecessor's entries winning over the mesh's; stale keys |
|
||||
| 10 | scripts | the operator's own scripts, shared and per role (research 027/03) | under no version control, copied by hand |
|
||||
| 11 | `kernel` | kernel, microcode, boot entries | two machines without microcode |
|
||||
| 11 | `kernel` | kernel, microcode, boot entries, and the kernel's run-time settings per machine (transparent huge pages first) | two machines without microcode; huge pages left at the distribution's `always` on a machine whose slow disks they flood (issue 746) |
|
||||
|
||||
`systemd`, `pacman` and `docker` hold the three seats that apply resources
|
||||
(ADR 0207):
|
||||
@@ -83,7 +84,19 @@ every module that declares a service, a package or a container depends on them b
|
||||
They go on every machine before the rest, and once they have, an unmet dependency is refused rather
|
||||
than reported.
|
||||
|
||||
`kernel` is last because a mistake in it costs a boot. `docker` holds the runtime seat
|
||||
`kernel` is last because a mistake in it costs a boot. Its run-time settings are the exception and come
|
||||
first (ADR 0369, issue 746): they change nothing at boot that the kernel does not already accept at run time, and
|
||||
each is a per-machine setting whose default is what the kernel boots with, so assigning the module changes
|
||||
no machine until that machine is given another value. The first is transparent huge pages. On the
|
||||
control-node, with `always`, the journal's memory-mapped files were held in 2 MiB pages, so each small journal
|
||||
write cost a 2 MiB write-back; the journal daemon and a log-reading intrusion-prevention service wrote
|
||||
gigabytes in minutes onto a mirrored pair of spinning disks that also holds the bus's store, and the
|
||||
controller lost its lease. That machine is set to `madvise`. The module writes the mode at boot and, on a
|
||||
change, the new value directly; it reads it back and fails its unit if it did not take, gives back the mode
|
||||
it found when unassigned, and on a running machine rotates the journal once so the open journal files are
|
||||
mapped anew in small pages. How it is checked: the module's tests run its script against
|
||||
a stand-in directory, its unit's health is the unit's state, and its tool `kernel_huge_pages` shows the
|
||||
mode in force and the file pages each unit holds in huge pages. `docker` holds the runtime seat
|
||||
(ADR 0166), with its settings declared as
|
||||
ADR 0164 decides and the capability detected as
|
||||
ADR 0165 decides; what of the three is not built yet is tracked by issue 398.
|
||||
|
||||
@@ -53,9 +53,9 @@ bucket, runs a bad migration — and not for the day a disk dies (ADR 0214).
|
||||
*Added 2026-10-06, ADR 0233.*
|
||||
The mesh composes a second file for the holder: every data item every module on the machine declares,
|
||||
with its class, its path, the path that covers it and its protection. Once an hour and after every
|
||||
night the holder measures each item that is not a cache, by the item's `measure`: a bounded walk at
|
||||
night the holder measures each item that is not a cache, by the item's `measure`: a bounded rollout at
|
||||
most daily for a small item (stopped after ten minutes or two million files and said as partial), a ZFS
|
||||
dataset's own counters hourly, or the top-level entries only — never an hourly walk of anything large —
|
||||
dataset's own counters hourly, or the top-level entries only — never an hourly rollout of anything large —
|
||||
and reads the redundant storage it is on (a ZFS pool's health, an md array's
|
||||
members, a btrfs filesystem's error counters). `backed-up` says each item with that measurement and its
|
||||
newest good backup. The holder judges nothing: the controller's self-check (to-be 45, D13) keeps the
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
|
||||
updated: 2026-10-12
|
||||
updated: 2026-10-14
|
||||
decisions:
|
||||
- "ADR 0357"
|
||||
- "ADR 0325"
|
||||
- "ADR 0323"
|
||||
- "ADR 0318"
|
||||
@@ -100,7 +101,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
|
||||
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
|
||||
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps |
|
||||
| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads |
|
||||
| the record mirrors: nodes, assignments, walks, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
|
||||
| the record mirrors: nodes, assignments, rollouts, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
|
||||
| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why; an ask is delivered again only when the controller made it itself, every other ask is only dropped (ADR 0293) |
|
||||
|
||||
A bus subject two components may publish on is refused when the controller composes grants, unless
|
||||
@@ -175,7 +176,7 @@ process of its own reads them back from the store.
|
||||
*Added 2026-10-08, ADR 0253:*
|
||||
**a condition says itself to the operator in plain words, beside its summary, and says whether the operator
|
||||
is needed.** The headline, needs, explanation, resolved line and actions are made where the condition is
|
||||
made: by the producer when it knows more than the subject (a walk's modules, a failed unit's name, a
|
||||
made: by the producer when it knows more than the subject (a rollout's modules, a failed unit's name, a
|
||||
hand-act's cause), otherwise by the wording registered for the kind, otherwise from the scope, which is
|
||||
reported. The explanation opens with its verdict. All of it is held to the plain rule when the controller
|
||||
takes an observation: no hash, numbered id, dotted name, duration in code, clock time, flag, markup or line
|
||||
@@ -259,25 +260,39 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
|
||||
| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — |
|
||||
| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — |
|
||||
| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — |
|
||||
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
|
||||
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted; every user's connection the server closed as a slow consumer or for breaking the bus's rules (ADR 0357) | bus server's advisory subjects; the controller's connection; the bus module's watch of its server (`bus.slow-consumer`, `bus.connection-closed`) | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `connection-closed` (one per bus user), `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
|
||||
| S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — |
|
||||
| S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
|
||||
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
|
||||
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — |
|
||||
| S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | `facts-stale` | warning | — |
|
||||
| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — |
|
||||
| S16 | a walk waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose walk waits | 30 min; urgent after 4 h | `waiting` (names the walk and `plans go`) | warning | — |
|
||||
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a walk out | warning | — |
|
||||
| S18 | a batch of merges is cut into its walk (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no walk is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
|
||||
| S19 | a batch waiting behind an open walk is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a walk is open | that walk's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the walk) | warning | — |
|
||||
| S16 | a rollout waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose rollout waits | 30 min; urgent after 4 h | `waiting` (names the rollout and `plans go`) | warning | — |
|
||||
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a rollout out | warning | — |
|
||||
| S18 | a batch of merges is cut into its rollout (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no rollout is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
|
||||
| S19 | a batch waiting behind an open rollout is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a rollout is open | that rollout's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the rollout) | warning | — |
|
||||
|
||||
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
|
||||
controller translates each into a condition naming the thing in the mesh's words, as the refused reply
|
||||
of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer
|
||||
is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a
|
||||
system account the controller hears a slow consumer and a refused subject for its own connection only
|
||||
([issue 270](../../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md),
|
||||
open question 2).
|
||||
is said only for one the mesh names and expects, and clears when it exists again.
|
||||
|
||||
**Every user's slow consumer, from the bus module (ADR 0357).** The server closes every client it judges
|
||||
a slow consumer and keeps its user and reason in its record of closed connections; it publishes neither
|
||||
in the mesh's account. So the `nats` module's tool bundle reads that record on its own machine every 30
|
||||
seconds and publishes `bus.slow-consumer` for each connection closed as a slow consumer, and
|
||||
`bus.connection-closed` for each the server closed for something its client did against the bus's
|
||||
rules or limits — naming the bus user, the connection and the reason, never a credential. A closure by
|
||||
the client, by the network or by the server's own shutdown is not said: D15 says a user dropped in a
|
||||
loop. A closure — its user, connection id and opening time — is said at least once: again when publishing
|
||||
it failed, and again for the last hour when the bundle starts. The controller's watchdogs read the last
|
||||
hour of both events back from the events stream on every tick, count each closure once, and raise one
|
||||
condition per bus user and kind, naming the module and its machine; it clears an hour after the newest
|
||||
closure. A closure is a condition at most about a minute after the server closed it.
|
||||
|
||||
**A refused subject is still heard for the controller's own connection only.** The server tells the
|
||||
client and writes its log, and says it nowhere else — not in its monitoring endpoint and not to a system
|
||||
account (issue 702, open).
|
||||
|
||||
## 4. The self-check: `doctor` (rule 6)
|
||||
|
||||
@@ -525,7 +540,7 @@ except an act recorded by a verb that is a person's decision by design, which is
|
||||
counts, whatever cause it gives. The controller keeps one table of the verbs that write the log, and
|
||||
each says whether it records a repair or a decision, and why: approving or rejecting a retirement and
|
||||
deleting what was retired (ADR 0230); `bus upgrade`, because the bus is never rolled by the mesh, and
|
||||
`upgrade release-backlog`, because after a failed walk the next opens only on a person's word
|
||||
`upgrade release-backlog`, because after a failed rollout the next opens only on a person's word
|
||||
(ADR 0236); and `secret rotate` when its cause is a leak (`leaked-in-logs`) — a person judges what was
|
||||
disclosed, one leak rotates several values, and a leak that recurs is a defect of the module that
|
||||
prints them, an issue against it, not a healer that rotates. A rotation for any other cause counts. A
|
||||
@@ -604,7 +619,7 @@ the other machines follow. "Reported applied" is not enough.
|
||||
|
||||
**As built** (ADR 0236):
|
||||
|
||||
- **Every walk is gated, not only the core's.** A module is judged by its own health: it reported
|
||||
- **Every rollout is gated, not only the core's.** A module is judged by its own health: it reported
|
||||
applied, no witness put it back, no condition was raised since its send about the machine or about the
|
||||
module there, and its tools are served on that machine where it has tools. Three healthy judgings at
|
||||
least forty seconds apart and two minutes after the send, within ten minutes of it. A policy of
|
||||
@@ -627,7 +642,7 @@ the other machines follow. "Reported applied" is not enough.
|
||||
- **No build reaches a machine without a gate**: a gated send carries and judges everything waiting on
|
||||
its machine; every other send — a plan's rest, a cascade, a healer's, a whole-mesh push — is refused
|
||||
or leaves the machine while a build no gate has seen waits there; a rebuild with the same artifacts and
|
||||
manifest is no move. What waits is moved by a **walk**, one machine at a time, the control
|
||||
manifest is no move. What waits is moved by a **rollout**, one machine at a time, the control
|
||||
node last, each judged; one that fails holds the next until `upgrade release-backlog --why`.
|
||||
- **A wait for a person passes, and each module of a send keeps its own pass**
|
||||
(ADR 0254): a module whose only fault is a new login the operator owes
|
||||
@@ -787,10 +802,10 @@ has a week of entries; the durations are recorded for every machine.
|
||||
|
||||
| Repository | Delivers |
|
||||
|---|---|
|
||||
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
|
||||
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9), and every user's slow consumer and closed connection read back from the bus module's events (ADR 0357); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
|
||||
| `mesh-host` | the heartbeat carries its interval; D1's validator published as a package the controller imports |
|
||||
| mesh-tools | the tool runner's heartbeat (S11) |
|
||||
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) |
|
||||
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5); the `nats` module's watch of its server's closed connections, publishing `bus.slow-consumer` and `bus.connection-closed` (S9, ADR 0357) |
|
||||
| mesh-lab | R9: each signal suppressed in turn |
|
||||
|
||||
**Done when:** on a lab mesh, suppressing each signal raises its condition within its bound and sends a
|
||||
@@ -923,7 +938,7 @@ and "the commit is the build at hand"), on branches, not yet merged:
|
||||
| hq | its own `merge-check.sh`, running `records.py`, `index.py` and `cycle.py` |
|
||||
|
||||
**Not yet:** the delivery plan kept (id, inputs, result) and the release comparing its plan with it; the
|
||||
state machine replacing the walks' states, with its table, its test, its conditions and healer H2;
|
||||
state machine replacing the rollouts' states, with its table, its test, its conditions and healer H2;
|
||||
the status's link to the kept plan; the notes under `refs/notes/mesh-plan`; check builds in a scratch
|
||||
namespace (the check builds no module today, so there is none to keep apart yet); the statuses made
|
||||
required — prepared as calls of the forge module's tool, applied by the operator.
|
||||
|
||||
@@ -327,36 +327,36 @@ Presence facts are events on the bus. The router keeps them as **current state o
|
||||
machine and per intake kind, overwritten — never as a history, and never in a message's words. A
|
||||
consumer that wants a timeline of the operator's day is a decision of its own.
|
||||
|
||||
### While a delivery walks
|
||||
### While a delivery rollouts
|
||||
|
||||
*2026-10-12, ADR 0308; the build is tracked by issue 532.*
|
||||
|
||||
A delivery restarts what it moves, and those restarts raise conditions that clear once it is done; a router
|
||||
that restarted used to say again what it had been told to keep quiet (issue 531). So the router holds what a
|
||||
walk touches and tells the walk's result once.
|
||||
rollout touches and tells the rollout's result once.
|
||||
|
||||
- **A walk names what it touches.** The controller's walk events say, when a walk starts and again when it
|
||||
- **A rollout names what it touches.** The controller's rollout events say, when a rollout starts and again when it
|
||||
ends with its outcome (delivered, failed or stopped), the modules it moves and the machines it sends them to.
|
||||
What it touches is those modules on those machines, and every module a moved tool runner or node-engine
|
||||
carries there.
|
||||
- **The router holds what an open walk touches.** A message whose subject is a touched module or machine is
|
||||
- **The router holds what an open rollout touches.** A message whose subject is a touched module or machine is
|
||||
held instead of sent, kept with when it was held, and `history` says it was held. A message several open
|
||||
walks touch is held until the last of them ends or its bound passes.
|
||||
rollouts touch is held until the last of them ends or its bound passes.
|
||||
- **Some messages are never held:** an ask (§6), a secret given or replaced, a message about anything no open
|
||||
walk touches, and a walk's own failure.
|
||||
- **A walk's end is told once:** one message with its outcome, the modules and machines it reached and how
|
||||
rollout touches, and a rollout's own failure.
|
||||
- **A rollout's end is told once:** one message with its outcome, the modules and machines it reached and how
|
||||
long it took, then each held message whose condition is still open, one line each. One whose condition
|
||||
cleared during the walk is only counted. A walk delivered with nothing held and nothing open is one line, or
|
||||
cleared during the rollout is only counted. A rollout delivered with nothing held and nothing open is one line, or
|
||||
nothing on a channel set to say only what needs the operator.
|
||||
- **A hold has a bound:** twice the largest delivery budget in force among the walk's deliveries
|
||||
- **A hold has a bound:** twice the largest delivery budget in force among the rollout's deliveries
|
||||
([to-be 47](47-delivery-from-commit-to-delivered.md)): 20 minutes for a core module, 10 for leaf modules at
|
||||
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a walk none of whose
|
||||
deliveries has a budget. Past it, every held message still open is sent with a line saying the walk still
|
||||
runs and since when, and the cleared ones are counted. A walk that hangs never silences the mesh.
|
||||
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a rollout none of whose
|
||||
deliveries has a budget. Past it, every held message still open is sent with a line saying the rollout still
|
||||
runs and since when, and the cleared ones are counted. A rollout that hangs never silences the mesh.
|
||||
- **A restart says nothing it has said:** a router that restarts says again only what is open and was never
|
||||
answered, never a silenced message or one withheld for its content.
|
||||
|
||||
*How it is checked:* the controller's tests of the walk events, and the router's tests of holding, never
|
||||
*How it is checked:* the controller's tests of the rollout events, and the router's tests of holding, never
|
||||
holding, the end message, the bound and the restart, as ADR 0308 lists them; live, the next delivery of the
|
||||
tool runner gives the operator one message at its end, and the channel's `history` shows what was held and
|
||||
released.
|
||||
@@ -467,13 +467,13 @@ the factors are the router's state.
|
||||
### An approved settings change takes effect (ADR 0344)
|
||||
|
||||
*2026-10-13, issue 641.* Approving a settings proposal is the act that makes it take effect. The
|
||||
serving controller sets the layer and, in the same act, opens a **settings walk**: every machine
|
||||
serving controller sets the layer and, in the same act, opens a **settings rollout**: every machine
|
||||
that runs the module and that the layer reaches, the control node last, each sent and its module
|
||||
judged at the first-node gate before the next. A settings walk sends the layer and no build: every
|
||||
judged at the first-node gate before the next. A settings rollout sends the layer and no build: every
|
||||
module on the machine stays at the build it was last sent, so a build held for a person, or waiting
|
||||
for a first-node gate, stays where it is. A send that would move a running module's data stops the
|
||||
walk and names the `push --move` that sends it. The walk says its start and end like every walk, so
|
||||
its end is told as every walk's is. A failure stops the walk and leaves the layer as approved; the
|
||||
rollout and names the `push --move` that sends it. The rollout says its start and end like every rollout, so
|
||||
its end is told as every rollout's is. A failure stops the rollout and leaves the layer as approved; the
|
||||
previous layer is in the settings history. The ask says all this before the Approve, and no longer
|
||||
names a push. A layer set by hand still waits for a push.
|
||||
|
||||
@@ -481,9 +481,9 @@ A delivered file is not yet an applied one: a module whose program reads its set
|
||||
starts applies them itself (a service's `restart-on`, or a tool of its own). Betterbird is such a
|
||||
module; issue 643 makes it restart Betterbird at a quiet moment.
|
||||
|
||||
How it is checked: the controller's tests open the walk on the warrant, send one machine at a time
|
||||
How it is checked: the controller's tests open the rollout on the warrant, send one machine at a time
|
||||
with every build kept and the module judged, stop before a data move and on a failure without
|
||||
holding other walks back, and read the ask's words.
|
||||
holding other rollouts back, and read the ask's words.
|
||||
|
||||
### Proofs and tiers
|
||||
|
||||
|
||||
@@ -30,9 +30,9 @@ delivery group is two or more deliveries that share a head branch name, delivere
|
||||
are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.**
|
||||
(ADR 0239.)
|
||||
|
||||
**From its merge on, a delivery is delivered by the walk of its batch.** The controller collects merges in a
|
||||
rolling merge window and plans each batch once, as one walk with at most one commit per repository, one walk
|
||||
at a time. Every delivery the walk answers ends with it
|
||||
**From its merge on, a delivery is delivered by the rollout of its batch.** The controller collects merges in a
|
||||
rolling merge window and plans each batch once, as one rollout with at most one commit per repository, one rollout
|
||||
at a time. Every delivery the rollout answers ends with it
|
||||
(ADR 0276).
|
||||
|
||||
This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison
|
||||
@@ -53,7 +53,7 @@ commit's journey and in what order the steps are allowed.
|
||||
|
||||
- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants,
|
||||
registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk
|
||||
rule, a gated send, the rollback and the walk of one trunk commit's tiers across machines. It still answers
|
||||
rule, a gated send, the rollback and the rollout of one trunk commit's tiers across machines. It still answers
|
||||
every pull request's head with its check itself, so a check never depends on mesh-delivery.
|
||||
- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the
|
||||
group, the state table and their state, and nothing else.
|
||||
@@ -61,9 +61,9 @@ commit's journey and in what order the steps are allowed.
|
||||
request, and appends to the commit's note under `refs/notes/mesh-plan`.
|
||||
- **The build seat** builds and checks, asked by the controller as before. **A change is built once** (ADR 0350):
|
||||
a check that passes also builds what its merge would move into the artifact store's scratch namespace, under a
|
||||
reuse key (the source fingerprint and the builder's version), and the walk of the merge commit copies a build
|
||||
of the same key, which the check's verdict names, by digest instead of building. Otherwise the walk builds and
|
||||
says why, per module, in its record. Check builds go when the walk answering the merge ends, when the pull
|
||||
reuse key (the source fingerprint and the builder's version), and the rollout of the merge commit copies a build
|
||||
of the same key, which the check's verdict names, by digest instead of building. Otherwise the rollout builds and
|
||||
says why, per module, in its record. Check builds go when the rollout answering the merge ends, when the pull
|
||||
request closes unmerged, and after seven days.
|
||||
|
||||
## A delivery
|
||||
@@ -76,7 +76,7 @@ Its id is `<owner>/<repository>@<twelve characters of its commit>`. It holds:
|
||||
its diffset (what ADR 0238 called the "change plan", under the glossary's name);
|
||||
- its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its
|
||||
**machine steps** while delivering;
|
||||
- once merged, **the commit it landed on the trunk as**, and the walk that delivers it: the walk of the batch
|
||||
- once merged, **the commit it landed on the trunk as**, and the rollout that delivers it: the rollout of the batch
|
||||
that answers its merge, which may carry a later commit of its repository that contains it (see *The merge
|
||||
window and the batch*).
|
||||
|
||||
@@ -93,9 +93,9 @@ allows.
|
||||
| checked | a verdict arrived | 1 minute | nothing: it is decided at once |
|
||||
| ready | it passed; it waits to be merged | none | — |
|
||||
| rejected | it failed its check | none | — |
|
||||
| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a walk that does not answer its merge took over its walk; `go` when its walk started |
|
||||
| published | merged on the trunk its modules follow, its rollout opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a rollout that does not answer its merge took over its rollout; `go` when its rollout started |
|
||||
| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
|
||||
| delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says |
|
||||
| delivering | its rollout runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the rollout's record says |
|
||||
| delivered, failed, superseded, stopped, retired, covered | final | — | — |
|
||||
|
||||
**A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head
|
||||
@@ -105,7 +105,7 @@ commits in the transition's reason; a `rejected` one stays `rejected`, its statu
|
||||
of date. Nothing is asked for it: a recheck, a new head or the merge tool asks the check again. The row from
|
||||
`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for
|
||||
an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by
|
||||
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own walk.
|
||||
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own rollout.
|
||||
|
||||
What this leaves as it is, said so nobody reads more into it:
|
||||
|
||||
@@ -125,7 +125,7 @@ What this leaves as it is, said so nobody reads more into it:
|
||||
|
||||
The transitions are those of ADR 0239 decision 2, ADR 0270 and ADR 0323. A delivery that is
|
||||
merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A
|
||||
delivery merged with no walk answering it within ten minutes is held too, with the true reason, and listed by
|
||||
delivery merged with no rollout answering it within ten minutes is held too, with the true reason, and listed by
|
||||
`stalled` at once. Nothing waits silently.
|
||||
|
||||
**What needs no person ends by itself** (ADR 0270):
|
||||
@@ -133,37 +133,37 @@ delivery merged with no walk answering it within ten minutes is held too, with t
|
||||
- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and
|
||||
sends nowhere) is `delivered` as soon as it is merged, whatever its check said;
|
||||
- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine
|
||||
running it by the walk of a later merge into the same trunk, delivered on its own passing check, and the
|
||||
running it by the rollout of a later merge into the same trunk, delivered on its own passing check, and the
|
||||
forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge
|
||||
order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery`
|
||||
says `covered: a later merge that contains it was delivered` as a success. A module that runs on one
|
||||
machine only is covered too: its walk records the empty rest as sent once its first machine passes
|
||||
machine only is covered too: its rollout records the empty rest as sent once its first machine passes
|
||||
(ADR 0270, as corrected 2026-10-12);
|
||||
- *a merge whose walk said its modules unchanged* moves nothing too, and is `delivered` when that walk ends,
|
||||
whichever commit the walk carried (issue 611). A delivery plan made while another walk built a module reads
|
||||
that module whole and names it; the walk, cut later, reads the build source that build recorded (ADR 0267) and
|
||||
may find none of the changed files in it. The walk's record then says so in `delivery.unchanged`: each such
|
||||
- *a merge whose rollout said its modules unchanged* moves nothing too, and is `delivered` when that rollout ends,
|
||||
whichever commit the rollout carried (issue 611). A delivery plan made while another rollout built a module reads
|
||||
that module whole and names it; the rollout, cut later, reads the build source that build recorded (ADR 0267) and
|
||||
may find none of the changed files in it. The rollout's record then says so in `delivery.unchanged`: each such
|
||||
module with why, "its build source holds none of the changed files", and not in its modules, which keep
|
||||
meaning what the walk builds. A delivery whose every module the walk built or said unchanged is delivered,
|
||||
saying which matched what runs. One with a module the walk neither built nor said unchanged is read as below.
|
||||
meaning what the rollout builds. A delivery whose every module the rollout built or said unchanged is delivered,
|
||||
saying which matched what runs. One with a module the rollout neither built nor said unchanged is read as below.
|
||||
Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's
|
||||
`TestAMergeWhoseModulesMatchWhatRunsIsDelivered`;
|
||||
- *a stop whose walk ended without building its modules* (issue 474) stays `stopped`, a final state, and
|
||||
- *a stop whose rollout ended without building its modules* (issue 474) stays `stopped`, a final state, and
|
||||
is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after
|
||||
its stop` with the covering merge, once, as a success. No state or transition is added (issue 480);
|
||||
- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes.
|
||||
|
||||
**History is retired, not held** (ADR 0250).
|
||||
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no walk
|
||||
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no rollout
|
||||
of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes
|
||||
to `retired` at adoption, before any check, hold or walk, and its pull request is not told. A held delivery that
|
||||
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its walk
|
||||
to `retired` at adoption, before any check, hold or rollout, and its pull request is not told. A held delivery that
|
||||
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its rollout
|
||||
starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a
|
||||
build registered before its turn would be carried by somebody else's send.
|
||||
|
||||
**The machine steps.** While a delivery is `delivering`, each machine its walk reaches has a row with the
|
||||
**The machine steps.** While a delivery is `delivering`, each machine its rollout reaches has a row with the
|
||||
module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy
|
||||
readings so far), `passed`, `failed`, `rolled-back`. They are read from the walk's own record in the
|
||||
readings so far), `passed`, `failed`, `rolled-back`. They are read from the rollout's own record in the
|
||||
controller's `plan-moved` event, never inferred from time.
|
||||
|
||||
## The merge window and the batch
|
||||
@@ -174,95 +174,95 @@ controller's `plan-moved` event, never inferred from time.
|
||||
Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by
|
||||
default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its
|
||||
merges and its times are kept in the controller's store and survive a restart.
|
||||
- **The cut.** When the window closes and no walk is open, the batch becomes one walk: one delivery plan for
|
||||
- **The cut.** When the window closes and no rollout is open, the batch becomes one rollout: one delivery plan for
|
||||
every commit in it, tiered along the module graph across repositories, with at most one commit per
|
||||
repository, the latest of its branch. The walk names every merge it answers: its own commits, and the
|
||||
repository, the latest of its branch. The rollout names every merge it answers: its own commits, and the
|
||||
earlier merges of a repository that its later commit contains.
|
||||
- **One walk at a time.** Merges that arrive while a walk is open join the next batch. A walk that started
|
||||
is never superseded. A walk still waiting for its word when the next batch is cut is folded into the new
|
||||
walk, which answers every merge the folded one answered.
|
||||
- **The controller's own path.** A merge that moves a module whose walk waits for nobody's word (the controller,
|
||||
- **One rollout at a time.** Merges that arrive while a rollout is open join the next batch. A rollout that started
|
||||
is never superseded. A rollout still waiting for its word when the next batch is cut is folded into the new
|
||||
rollout, which answers every merge the folded one answered.
|
||||
- **The controller's own path.** A merge that moves a module whose rollout waits for nobody's word (the controller,
|
||||
the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for
|
||||
mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its
|
||||
walk like any other. An own-path walk folds no waiting catalogue walk; that walk's merges are batched again
|
||||
rollout like any other. An own-path rollout folds no waiting catalogue rollout; that rollout's merges are batched again
|
||||
behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10).
|
||||
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a walk is
|
||||
answered by that walk: the assembling batch, the open walk, or, when that walk was delivered, it ends
|
||||
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a rollout is
|
||||
answered by that rollout: the assembling batch, the open rollout, or, when that rollout was delivered, it ends
|
||||
`covered` at once.
|
||||
- **The ends.** A walk done ends every delivery it answers: `delivered` for its own commits, `covered` for
|
||||
an earlier merge of a repository it carried later, naming that merge. A walk that failed fails its own
|
||||
commits' deliveries and marks none delivered; each earlier merge it answered is then walked on its own
|
||||
commit, the newest first, one walk each, until one is delivered. A walk stopped stops every delivery whose
|
||||
merge it walks again nowhere (below).
|
||||
- **A group inside a walk.** The pairs of a delivery group's order (ADR 0249) are edges of the walk's
|
||||
tiers beside the module graph. mesh-delivery gives a walk its word only when every delivery it carries is
|
||||
- **The ends.** A rollout done ends every delivery it answers: `delivered` for its own commits, `covered` for
|
||||
an earlier merge of a repository it carried later, naming that merge. A rollout that failed fails its own
|
||||
commits' deliveries and marks none delivered; each earlier merge it answered is then rolled out on its own
|
||||
commit, the newest first, one rollout each, until one is delivered. A rollout stopped stops every delivery whose
|
||||
merge it rollouts again nowhere (below).
|
||||
- **A group inside a rollout.** The pairs of a delivery group's order (ADR 0249) are edges of the rollout's
|
||||
tiers beside the module graph. mesh-delivery gives a rollout its word only when every delivery it carries is
|
||||
at its turn.
|
||||
- **The bookkeeping.** mesh-delivery ties a walk to every live delivery whose merge it answers, follows a
|
||||
folded walk to the walk that took it over, and never ties a walk to a delivery that has ended. A waiting
|
||||
walk no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
|
||||
- **The bookkeeping.** mesh-delivery ties a rollout to every live delivery whose merge it answers, follows a
|
||||
folded rollout to the rollout that took it over, and never ties a rollout to a delivery that has ended. A waiting
|
||||
rollout no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
|
||||
- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped:
|
||||
novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated",
|
||||
and a batch waiting behind an open walk names that walk. The controller says the batch on the bus as
|
||||
and a batch waiting behind an open rollout names that rollout. The controller says the batch on the bus as
|
||||
`plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged
|
||||
delivery it holds.
|
||||
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no walk open, and a batch
|
||||
waiting behind an open walk past that walk's bound, each raise a condition. A merged delivery that no walk
|
||||
answers within ten minutes is held with the true reason (the open walks of its repository and that none
|
||||
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no rollout open, and a batch
|
||||
waiting behind an open rollout past that rollout's bound, each raise a condition. A merged delivery that no rollout
|
||||
answers within ten minutes is held with the true reason (the open rollouts of its repository and that none
|
||||
names its merge, or that the controller holds none) and listed by `stalled` at once.
|
||||
|
||||
|
||||
## A stop on a shared walk
|
||||
## A stop on a shared rollout
|
||||
|
||||
*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.*
|
||||
|
||||
One walk carries every merge of its batch, so ending a walk for one delivery ends it for every other
|
||||
One rollout carries every merge of its batch, so ending a rollout for one delivery ends it for every other
|
||||
delivery on it. A stop therefore ends only the change it is for:
|
||||
|
||||
- **A group's stop.** When a member after a failed one is stopped and its walk is open, mesh-delivery asks
|
||||
`delivery-stop` for that walk with the stopped members' merges to leave out, every member the same pass
|
||||
stops on that walk in one ask. The controller ends the walk, said as stopped, and walks every other merge
|
||||
- **A group's stop.** When a member after a failed one is stopped and its rollout is open, mesh-delivery asks
|
||||
`delivery-stop` for that rollout with the stopped members' merges to leave out, every member the same pass
|
||||
stops on that rollout in one ask. The controller ends the rollout, said as stopped, and rolls out every other merge
|
||||
it answered again.
|
||||
- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A
|
||||
module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too.
|
||||
Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped
|
||||
merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is
|
||||
walked again.
|
||||
- **Where the others go.** They join the open batch of their kind, where merges heard while the walk ran
|
||||
already wait, so an older commit never walks after a newer one; only with none open do they form a new
|
||||
rolled out again.
|
||||
- **Where the others go.** They join the open batch of their kind, where merges heard while the rollout ran
|
||||
already wait, so an older commit never rollouts after a newer one; only with none open do they form a new
|
||||
batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop
|
||||
before it is cut, are cut without the rest of their window. Its walk withholds the left-out merges' modules
|
||||
before it is cut, are cut without the rest of their window. Its rollout withholds the left-out merges' modules
|
||||
where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes
|
||||
to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every
|
||||
merge moved are written in one transaction.
|
||||
- **The walk's record.** The stopped walk names the batch that walks its merges again, every merge it left
|
||||
out, and each module it already sent that nothing walks again, with the machines it reached. A delivery
|
||||
whose merge is walked again goes from `delivering` back to `published` (the row `rewalk`), or stays
|
||||
`published`, and follows the fresh walk once it is cut. A delivery whose merge was left out, and every
|
||||
delivery on a walk stopped whole, ends `stopped`, naming the walk and the stop.
|
||||
- **A stopped merge never ships by a walk.** mesh-delivery asks every open walk or batch that names a stopped
|
||||
delivery's merge to leave it out, and never gives such a walk its word. When the controller refuses, it
|
||||
stops the whole walk.
|
||||
- **A person's stop.** `stop` of a delivery whose open walk carries other deliveries does nothing at first: it
|
||||
- **The rollout's record.** The stopped rollout names the batch that rolls out its merges again, every merge it left
|
||||
out, and each module it already sent that nothing rollouts again, with the machines it reached. A delivery
|
||||
whose merge is rolled out again goes from `delivering` back to `published` (the row `rewalk`), or stays
|
||||
`published`, and follows the fresh rollout once it is cut. A delivery whose merge was left out, and every
|
||||
delivery on a rollout stopped whole, ends `stopped`, naming the rollout and the stop.
|
||||
- **A stopped merge never ships by a rollout.** mesh-delivery asks every open rollout or batch that names a stopped
|
||||
delivery's merge to leave it out, and never gives such a rollout its word. When the controller refuses, it
|
||||
stops the whole rollout.
|
||||
- **A person's stop.** `stop` of a delivery whose open rollout carries other deliveries does nothing at first: it
|
||||
answers with each of them, which of them contain this merge and end either way, and two ways given with
|
||||
`way`: `again` (the walk ended and the others walked again without this merge) or `walk` (the whole walk
|
||||
stopped, every delivery on it with it). A delivery alone on its walk is stopped at once. A delivery is
|
||||
marked stopped only once its walk is; a stop the controller refuses leaves it as it was.
|
||||
- **What it does not do.** What the stopped walk already sent stays on the machines it reached, and its
|
||||
`way`: `again` (the rollout ended and the others rolled out again without this merge) or `walk` (the whole rollout
|
||||
stopped, every delivery on it with it). A delivery alone on its rollout is stopped at once. A delivery is
|
||||
marked stopped only once its rollout is; a stop the controller refuses leaves it as it was.
|
||||
- **What it does not do.** What the stopped rollout already sent stays on the machines it reached, and its
|
||||
asked builds still register, so a later send may carry them further; the stop names the machines, and
|
||||
putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the
|
||||
stopped merge's modules, contains the stopped change and delivers it when walked; the stop names each it
|
||||
stopped merge's modules, contains the stopped change and delivers it when rolled out; the stop names each it
|
||||
can see, and holding a merged change back for good is a revert's job.
|
||||
## The delivery budget
|
||||
|
||||
*Decided 2026-10-10 by ADR 0282.
|
||||
Where it differs, this section replaces the window for every merge and the one walk at a time above, slice by
|
||||
Where it differs, this section replaces the window for every merge and the one rollout at a time above, slice by
|
||||
slice as each slice below is built; until a slice is built, the text above is what runs.*
|
||||
|
||||
**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard
|
||||
from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes;
|
||||
a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is
|
||||
shown and held to no budget. Measured on 51 walks on 2026-10-09 and 10, a leaf walk took a median of 6.0 minutes
|
||||
(7 of 20 within five), a core walk of one tier 5.6 (worst 8.8), and a controller walk dragging two catalogue tiers
|
||||
shown and held to no budget. Measured on 51 rollouts on 2026-10-09 and 10, a leaf rollout took a median of 6.0 minutes
|
||||
(7 of 20 within five), a core rollout of one tier 5.6 (worst 8.8), and a controller rollout dragging two catalogue tiers
|
||||
14.2 (issue 338).
|
||||
|
||||
*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget
|
||||
@@ -273,56 +273,56 @@ phases — implementing, review, checks, and merge to running (the delivery time
|
||||
budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget.
|
||||
The delivery budget stays as written: a measure inside the fix budget, not replaced by it.
|
||||
|
||||
**What changes in the walk.**
|
||||
**What changes in the rollout.**
|
||||
|
||||
- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged.
|
||||
- **Independent walks run at once**: no shared repository, module or module-graph edge with any open walk. One
|
||||
node judges one walk at a time; a send to a node under another walk's judgement waits for its verdict; no leaf
|
||||
- **Independent rollouts run at once**: no shared repository, module or module-graph edge with any open rollout. One
|
||||
node judges one rollout at a time; a send to a node under another rollout's judgement waits for its verdict; no leaf
|
||||
build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged.
|
||||
- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf
|
||||
module with a long-running resource; one for a leaf module whose every resource is a tools bundle.
|
||||
- **A tier's first nodes are judged at once.**
|
||||
- **The bus's grants arrive before the gate** (ADR 0298): when a walk's builds change the bus's user list (a new
|
||||
- **The bus's grants arrive before the gate** (ADR 0298): when a rollout's builds change the bus's user list (a new
|
||||
tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before
|
||||
anything is judged; a step that would carry anything else is refused unsent and said on the gate line.
|
||||
- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy
|
||||
report; the plan shows an expected landing time from the last seven days' medians.
|
||||
|
||||
*Walks beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
|
||||
a change shares).* Building the walks at once found what those bullets left open, and ADR 0306 decides it:
|
||||
*Rollouts beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
|
||||
a change shares).* Building the rollouts at once found what those bullets left open, and ADR 0306 decides it:
|
||||
|
||||
- **A walk's builds are its own, never another's backlog.** A send composes a module whose build another open
|
||||
walk owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
|
||||
judged only by its own walk, so a failed walk puts back only its own builds. What waits on a node with no open
|
||||
walk to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
|
||||
for builds an open walk owns).
|
||||
- **A rollout's builds are its own, never another's backlog.** A send composes a module whose build another open
|
||||
rollout owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
|
||||
judged only by its own rollout, so a failed rollout puts back only its own builds. What waits on a node with no open
|
||||
rollout to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
|
||||
for builds an open rollout owns).
|
||||
- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the
|
||||
merge window whether or not a batch is assembling, so a group's members walk in one batch, in their order.
|
||||
- **Related means everything the walks would move.** A module in common is any module a walk plans to build or
|
||||
merge window whether or not a batch is assembling, so a group's members roll out in one batch, in their order.
|
||||
- **Related means everything the rollouts would move.** A module in common is any module a rollout plans to build or
|
||||
send, the dependents of what its merges moved included; a change whose modules are not known yet is related to
|
||||
every change. Two related walks are neither cut beside each other nor judged beside each other on one node.
|
||||
every change. Two related rollouts are neither cut beside each other nor judged beside each other on one node.
|
||||
- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a
|
||||
warning naming the walk and the core walks that set it aside.
|
||||
warning naming the rollout and the core rollouts that set it aside.
|
||||
- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an
|
||||
unhealthy network or engine) fails every judgement open on it, leaf and core, each walk putting back only its
|
||||
unhealthy network or engine) fails every judgement open on it, leaf and core, each rollout putting back only its
|
||||
own builds.
|
||||
- **A rollback beside another walk's judgement** behaves as it does today; whether it should wait is tracked by
|
||||
- **A rollback beside another rollout's judgement** behaves as it does today; whether it should wait is tracked by
|
||||
issue 528.
|
||||
|
||||
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of walk A left out
|
||||
of walk B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
|
||||
overlapping dependents keeping two walks apart; the warning on the third set-aside; a failed apply failing every
|
||||
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of rollout A left out
|
||||
of rollout B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
|
||||
overlapping dependents keeping two rollouts apart; the warning on the third set-aside; a failed apply failing every
|
||||
open judgement on its node.
|
||||
|
||||
*A busy container on the first node (2026-10-13, ADR 0335, citing ADR 0282 §1).* A container may declare `busy`,
|
||||
a command that says whether it holds work a replacement would lose; the build agent declares it, so a merge
|
||||
into the controller's repository no longer throws away the checks in flight. When a walk's send would replace
|
||||
into the controller's repository no longer throws away the checks in flight. When a rollout's send would replace
|
||||
such a container while it says busy, the node-engine leaves it running, applies the rest, and replaces it once
|
||||
it says idle, or after 15 minutes at the latest. Only that node's walk waits: the walk says it waits for a busy
|
||||
it says idle, or after 15 minutes at the latest. Only that node's rollout waits: the rollout says it waits for a busy
|
||||
container there, its first-node gate starts at the report that says the container replaced, and the wait is a
|
||||
phase of the delivery time of its own, never exempted from the delivery budget. A delivery it puts over budget
|
||||
is said over budget, with that wait as its longest phase. *How it is checked:* the controller's test that a
|
||||
walk whose module's container waits as busy on its first node says so and starts that node's gate only once the
|
||||
rollout whose module's container waits as busy on its first node says so and starts that node's gate only once the
|
||||
node reports it replaced, and the walk-phase test that the wait is a measured phase between the send and the
|
||||
judgement.
|
||||
|
||||
@@ -330,11 +330,11 @@ judgement.
|
||||
|
||||
| Part | Owner | What it adds |
|
||||
|---|---|---|
|
||||
| the walk's phases and readings | mesh-controller | each phase's start and end and each reading in the walk and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
|
||||
| the rollout's phases and readings | mesh-controller | each phase's start and end and each reading in the rollout and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
|
||||
| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` |
|
||||
| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery.<class>.over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` |
|
||||
| the lone merge | mesh-controller | the cut at once when nothing waits to join |
|
||||
| walks at once | mesh-controller | the independence test, one judgement per node, a core walk's ground, a repository's next batch |
|
||||
| rollouts at once | mesh-controller | the independence test, one judgement per node, a core rollout's ground, a repository's next batch |
|
||||
| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once |
|
||||
| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` |
|
||||
|
||||
@@ -342,11 +342,11 @@ judgement.
|
||||
|
||||
| Slice | Delivers | Done when |
|
||||
|---|---|---|
|
||||
| 1 — the measure | the walk's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
|
||||
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its walk's phases |
|
||||
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its walk |
|
||||
| 4 — walks at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
|
||||
| 5 — the landing time | the expected landing time | `plans` shows it for an open walk, and the week's deliveries land within a minute of it at the median |
|
||||
| 1 — the measure | the rollout's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
|
||||
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its rollout's phases |
|
||||
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its rollout |
|
||||
| 4 — rollouts at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
|
||||
| 5 — the landing time | the expected landing time | `plans` shows it for an open rollout, and the week's deliveries land within a minute of it at the median |
|
||||
|
||||
Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges
|
||||
only once it is built.
|
||||
@@ -374,9 +374,9 @@ only once it is built.
|
||||
- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`.
|
||||
A member published while its group's composed check is asked and not answered waits there for the answer;
|
||||
one held because that check had not passed is *lifted* back to `published` once it passes or warns for the
|
||||
heads that merged and its own walk is still open (ADR 0270).
|
||||
heads that merged and its own rollout is still open (ADR 0270).
|
||||
- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its
|
||||
walk while the walk's other merges are walked again (ADR 0299). The members before it stay delivered.
|
||||
rollout while the rollout's other merges are rolled out again (ADR 0299). The members before it stay delivered.
|
||||
Its own builds were put back at its gate.
|
||||
|
||||
A group's state is derived from its members and its composed check, and is shown, never stored apart from
|
||||
@@ -408,22 +408,22 @@ In this order, each by its owner:
|
||||
| Verb | Answers or does |
|
||||
|---|---|
|
||||
| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group |
|
||||
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the walk |
|
||||
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the rollout |
|
||||
| `groups` | every group with its members in order and its derived state |
|
||||
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere |
|
||||
| `table` | the state table and the machine-step table, with bounds and H2's repairs |
|
||||
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting walk no live delivery holds |
|
||||
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting rollout no live delivery holds |
|
||||
| `recheck` | a rejected or ready delivery back to proposed, with why |
|
||||
| `release` | a held delivery to delivering, with why: a person's word |
|
||||
| `stop` | any delivery not final to stopped, with why; its walk is ended through the controller. On a walk that carries other deliveries it first answers what ending the walk would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
|
||||
| `stop` | any delivery not final to stopped, with why; its rollout is ended through the controller. On a rollout that carries other deliveries it first answers what ending the rollout would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
|
||||
| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise |
|
||||
| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing |
|
||||
|
||||
**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`,
|
||||
`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which
|
||||
takes the merges to leave out and walks the walk's other merges again without them, ADR 0299),
|
||||
takes the merges to leave out and rolls out the rollout's other merges again without them, ADR 0299),
|
||||
`delivery-walks`. A person has `plans go <plan> --why`, which does what `deliver` does under their name and
|
||||
is recorded as a hand act. The controller says every walk it keeps as `plan-moved`, from a verb's command
|
||||
is recorded as a hand act. The controller says every rollout it keeps as `plan-moved`, from a verb's command
|
||||
too.
|
||||
|
||||
**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It
|
||||
@@ -438,12 +438,12 @@ up.
|
||||
|
||||
## The bootstrap
|
||||
|
||||
- A walk that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
|
||||
- A rollout that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
|
||||
controller's own, started by the merge and witnessed on the machine. mesh-delivery records it.
|
||||
- Any other walk waits for `deliver` only while the seat has a holder on record. With none on record, the
|
||||
- Any other rollout waits for `deliver` only while the seat has a holder on record. With none on record, the
|
||||
controller starts it as it always did.
|
||||
- A waiting walk is said by the controller whatever mesh-delivery says of itself: S16 raises
|
||||
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the walk answers and
|
||||
- A waiting rollout is said by the controller whatever mesh-delivery says of itself: S16 raises
|
||||
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the rollout answers and
|
||||
`plans go <walk> --why` as the way on.
|
||||
A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone.
|
||||
|
||||
@@ -453,22 +453,22 @@ up.
|
||||
merged and rolls out as the core does. With no holder on record it behaves exactly as before.
|
||||
2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built
|
||||
and published.
|
||||
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Walks that start
|
||||
from then on wait for its word. Walks already open finish on their own and are adopted as deliveries in
|
||||
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Rollouts that start
|
||||
from then on wait for its word. Rollouts already open finish on their own and are adopted as deliveries in
|
||||
`delivering`.
|
||||
4. To step back, unassign it. Waiting walks are then started by the controller at its next pass.
|
||||
4. To step back, unassign it. Waiting rollouts are then started by the controller at its next pass.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Delivers | Done when |
|
||||
|---|---|---|
|
||||
| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` |
|
||||
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a walk never let go is raised by S16 |
|
||||
| C — the walk itself | the per-tier walk moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
|
||||
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a rollout never let go is raised by S16 |
|
||||
| C — the rollout itself | the per-tier rollout moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
|
||||
|
||||
## What is not decided here
|
||||
|
||||
- A web view of deliveries beyond the pull request's comment.
|
||||
- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge
|
||||
module's protection tool.
|
||||
- Moving the per-tier walk out of the controller (Phase C).
|
||||
- Moving the per-tier rollout out of the controller (Phase C).
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-host, mesh-controller, mesh-lab, mesh-catalog, mesh-sdk]
|
||||
updated: 2026-10-11
|
||||
updated: 2026-10-14
|
||||
decisions:
|
||||
- "ADR 0292"
|
||||
- "ADR 0283"
|
||||
@@ -13,6 +13,7 @@ decisions:
|
||||
- "ADR 0236"
|
||||
- "ADR 0227"
|
||||
- "ADR 0256"
|
||||
- "ADR 0368"
|
||||
---
|
||||
|
||||
# 48 — A module says how it is healthy
|
||||
@@ -200,7 +201,7 @@ build alike, so any build of the module is excused while the wait checks out.
|
||||
**Each module of a send keeps its own pass** (ADR 0254). A send is still judged as one, with one verdict.
|
||||
The gate also counts, per module, the judgings in a row that found it healthy or waiting for a person;
|
||||
when the send fails, a module whose count reached the passes asked keeps a pass, said "on its own", and is
|
||||
neither put back nor left as a move no gate has seen for every other walk to wait on. A module found broken
|
||||
neither put back nor left as a move no gate has seen for every other rollout to wait on. A module found broken
|
||||
is marked failed and put back at once, and the send's verdict waits until the modules beside it have their
|
||||
own, or the bound, so every module of a failed send leaves it with a verdict: its own pass, or put back.
|
||||
|
||||
@@ -246,6 +247,11 @@ The runtime restarts what exits, as now. The node-engine never restarts, recreat
|
||||
being unhealthy. The condition reaches the operator through the conversation (to-be 46), or a healer once a
|
||||
record gives one that act.
|
||||
|
||||
One module repairs itself, by its own record: the dnsmasq module's watch restarts the resolver's unit when
|
||||
it stops answering, at most once every two minutes and three times an hour, and says through its declared
|
||||
health whether the restart helped (ADR 0368). The node-engine still restarts nothing; any other restart on
|
||||
health needs a record of its own.
|
||||
|
||||
## 8. A declaration is proved before it is trusted
|
||||
|
||||
- **`module check`** refuses a `health` field naming an endpoint the module does not declare under
|
||||
@@ -520,7 +526,8 @@ controller.
|
||||
|
||||
## What is not decided here
|
||||
|
||||
- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231).
|
||||
- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231). The resolver's own
|
||||
restart is decided, for that module alone, by ADR 0368.
|
||||
- Health of scheduled work, and of what a module's events do (issue 276).
|
||||
- A slow starter that needs more than five minutes to be ready — a change to the bound, recorded, when one
|
||||
exists.
|
||||
|
||||
@@ -31,7 +31,7 @@ shape — what each is for, where its records live, and how they depend on one a
|
||||
| **Placement** | to decide, send and apply what each node runs, and say why | the controller's `inventory` database |
|
||||
| **Provisioning** | to resolve which module serves a provision for which consumer, and wire the two | the controller |
|
||||
| **Identity and access** | to issue, hold, rotate and check credentials and grants | the controller's `identity` and `licences` databases, and the vault |
|
||||
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and walk |
|
||||
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and rollout |
|
||||
| **Health and repair** | to notice what is wrong, say it once, repair what may be repaired unattended, and record what was done by hand | the controller's condition store |
|
||||
| **Data** | to know every item of data a module holds, how precious it is, and keep it recoverable | the manifests' data declarations, and every node's `node-backup` seat |
|
||||
| **Connectivity** | to make every node and module reachable by name where it should be, and unreachable where it should not | the controller |
|
||||
|
||||
@@ -12,7 +12,7 @@ decisions:
|
||||
|
||||
# 54 — The controller's records on the bus
|
||||
|
||||
**What each node runs, how far each walk has got, what each build is doing and what each setting holds
|
||||
**What each node runs, how far each rollout has got, what each build is doing and what each setting holds
|
||||
can be watched on the bus, and what is watched never says something the controller's store does not**
|
||||
(ADR 0289,
|
||||
from gap G2 of research 036).
|
||||
@@ -51,7 +51,7 @@ do not have: a check that the bucket says what the store says.
|
||||
Each value is built from a named list of fields. A field not on the list is never in the value. A value
|
||||
holds only what is built from its record: no epoch, no time of writing and no time of the last look,
|
||||
because any lease holder must build the same bytes from the same record, and the audit must too. Times
|
||||
that belong to the record itself, such as when a walk was opened or a resource's state began, are fields
|
||||
that belong to the record itself, such as when a rollout was opened or a resource's state began, are fields
|
||||
like any other. The epoch and the time of the write travel in the event.
|
||||
|
||||
### nodes — `mesh-controller_nodes`, key: the node's name
|
||||
@@ -74,15 +74,15 @@ like any other. The epoch and the time of the write travel in the event.
|
||||
| the build the node was last sent for it and the build its node reports running | — |
|
||||
| the seats it holds there, and the provisions it is bound to | the bindings' credentials, which are secrets |
|
||||
|
||||
### walks — `mesh-controller_walks`, key: the walk's id
|
||||
### rollouts — `mesh-controller_walks`, key: the rollout's id
|
||||
|
||||
| Holds | Leaves out, and why |
|
||||
|---|---|
|
||||
| the repository and commit, or the batch of merges it walks (ADR 0276) | — |
|
||||
| the repository and commit, or the batch of merges it rollouts (ADR 0276) | — |
|
||||
| its state (building, rolling, done, failed), its current tier, and each tier's modules and nodes with each step's state | the build and apply logs, which the `plans` and `builds` verbs answer |
|
||||
| when it was opened, when it last moved, when it ended, and its note | — |
|
||||
|
||||
A walk is removed from the mirror 14 days after it ended.
|
||||
A rollout is removed from the mirror 14 days after it ended.
|
||||
|
||||
### builds — `mesh-controller_builds`, key: the build's id
|
||||
|
||||
@@ -137,8 +137,8 @@ called).
|
||||
caller today): a numbered migration of `inventory` and the consolidated schema, in the same change.
|
||||
- **A removal is a purge** of the key, said as a `removed` event. A purge leaves one marker for the key.
|
||||
Each purge gives its marker a time to live of one day (a per-message time to live, which the bus server
|
||||
has offered since release 2.11 and the bucket enables), so the markers of removed walks and builds do
|
||||
not pile up in a bucket that has no age. The controller's one-minute tick removes walks and builds 14
|
||||
has offered since release 2.11 and the bucket enables), so the markers of removed rollouts and builds do
|
||||
not pile up in a bucket that has no age. The controller's one-minute tick removes rollouts and builds 14
|
||||
days after they ended.
|
||||
- **A write that fails** sends no event, is logged, and is left to the audit. The writer does not retry
|
||||
in a loop: a retry loop against a refusing bus is the reconnect loop of issue 327 in another place.
|
||||
@@ -169,7 +169,7 @@ counts in §6 shows the gap.
|
||||
**D17, the mirrors match their store**, is a probe in the self-check's registry (to-be 45 §4). Every run
|
||||
does this for each mirror:
|
||||
|
||||
1. It reads every record the mirror should hold from the store (for walks and builds, those open or ended
|
||||
1. It reads every record the mirror should hold from the store (for rollouts and builds, those open or ended
|
||||
within 14 days) and builds each value as the writer would.
|
||||
2. It lists the bucket's keys and reads their values.
|
||||
3. It compares. A key missing, a value that differs, and a key with no record are each a candidate. A
|
||||
@@ -225,10 +225,10 @@ remain open there.
|
||||
| The bucket matches the store | D17 on every self-check run; its own test, which changes a key behind the writer's back and expects `mirror-drifted`, then H6's write, then the condition cleared on the next run; and a test that a change of lease alone raises nothing |
|
||||
| Quiet means no writes | live: over ten minutes in which `plans`, `nodes`, `builds` and `settings` show no change, no mirror's write count rises |
|
||||
| A watcher sees a change | live: a setting changed through `settings` is seen by a watch on `mesh-controller_settings` and as one `setting-changed` within a second |
|
||||
| A value is under its cap, or cut and marked | the controller's test, which builds the largest walk the mesh could make (every module on every node) |
|
||||
| A value is under its cap, or cut and marked | the controller's test, which builds the largest rollout the mesh could make (every module on every node) |
|
||||
| Only the audit reads a mirror | a test over the controller's packages, which fails when any package but the audit's reads a mirror bucket |
|
||||
| A write follows the commit | the controller's test that a rolled-back transaction makes no write and sends no event |
|
||||
| Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals |
|
||||
| Ended rollouts and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the rollouts bucket's keys including deleted ones at most its live keys plus a day's removals |
|
||||
| A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run |
|
||||
| A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set |
|
||||
| A lost acknowledgement is no conflict | the controller's test in which the bus stores a write and drops its acknowledgement: the next write to that key succeeds and no condition is raised |
|
||||
@@ -236,7 +236,7 @@ remain open there.
|
||||
|
||||
## What it leaves out
|
||||
|
||||
- **History.** A mirror is current state. What happened beyond the 14 days a walk or build is kept, and
|
||||
- **History.** A mirror is current state. What happened beyond the 14 days a rollout or build is kept, and
|
||||
beyond EVENTS' week, is gap G8 of research 036, which remains open there.
|
||||
- **People reaching the mirrors**, through a WebSocket and an auth callout, is gap G1, open in the same
|
||||
effort.
|
||||
|
||||
@@ -130,7 +130,7 @@ rules that judge the whole record set run (an accepted decision cited by nothing
|
||||
undo the acceptance. It is a fault of the records, which their authors repair, not of the module: it is told to the
|
||||
operator once, as a warning naming the rule and the records, and cleared once the rule holds again. The module's
|
||||
health names it and stays healthy, because health says only whether the module works, and a finding never holds a
|
||||
walk at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
|
||||
rollout at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
|
||||
reviewed).
|
||||
|
||||
The rule `cited` is also judged ahead, in the review, before the judgement: a decision that names no home and that no
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: [mesh-controller, mesh-host, mesh-sdk, mesh-catalog]
|
||||
updated: 2026-10-14
|
||||
decisions:
|
||||
- "ADR 0348"
|
||||
- "ADR 0349"
|
||||
- "ADR 0201"
|
||||
- "ADR 0113"
|
||||
- "ADR 0158"
|
||||
- "ADR 0360"
|
||||
- "ADR 0362"
|
||||
---
|
||||
|
||||
# 57. Building config and provisioning
|
||||
|
||||
**The work of ADR 0348 (config is a domain of its own) and ADR 0349 (provisioning), broken into
|
||||
packages small enough that each ends at something that can be seen to work, in the order their
|
||||
dependencies allow.** The two decision records are the authority on *what* is built; this document holds
|
||||
only the packages, their order, their sizes and their proofs, and is wrong the moment it disagrees with
|
||||
them. It is the shape [design 38](38-building-the-operators-machine.md) gave the operator's machine.
|
||||
|
||||
## How this is built
|
||||
|
||||
**Each package is one issue in the issue tracker, and one pull request per repository it touches.** The
|
||||
issue carries the package's number here in its title, links the packages it waits for with `depends-on`,
|
||||
and links ADR 0348 or ADR 0349. At most five packages are worked at once, each by its own agent in its
|
||||
own worktree; a package starts once every package it depends on is merged. Every pull request is
|
||||
reviewed independently at its head and merged only with its checks green.
|
||||
|
||||
**It ships as one release.** The operator's decision of 2026-10-14: one delivery, not one per package.
|
||||
Each of mesh-controller, `mesh-host`, mesh-sdk, mesh-catalog and hq has the branch
|
||||
`feat/57-config-and-provisioning`, and every package's pull request merges into that branch, never into
|
||||
main. When the last package is merged, the whole is rehearsed in the lab: the rollout, the moves of data in
|
||||
their order, and every proof that is run live. Then each repository's branch reaches main in one pull
|
||||
request, and the five, sharing their branch's name, are one delivery group (ADR 0239 §2), judged as one
|
||||
future state. The five are merged within one merge window, so they are one batch and one rollout (ADR 0276
|
||||
§1, §2), in an order declared with `after:` lines in each pull request (ADR 0239 §11): mesh-sdk, then
|
||||
`mesh-host`, then mesh-controller, then mesh-catalog, then hq. The release holds every package but T6, R1 and R2:
|
||||
the media catalogue and the applications live in repositories of their own and move after it, and only
|
||||
then can the old forms be refused (R1) and the record say what runs (R2).
|
||||
|
||||
**The moves of data run after the delivery, in this order, each a verb that says what it did:**
|
||||
placement out of the settings layers (F2), the settings layers into mesh-config (K5), the secrets into
|
||||
the vault, nothing rotated (V3), then the providers one by one (B2, B3). Each is rehearsed in the lab
|
||||
first, and each is checked by reading back through the module that owns the data before anything
|
||||
points at it.
|
||||
|
||||
**What a release costs.** A proof that is run live (dunst on a workstation, a secret reaching its file, a
|
||||
database consumer connecting) is first seen in the lab rehearsal, and a fault found then holds the
|
||||
release, not one package. A merge's builds go to every machine at once (ADR 0360), so the release has no
|
||||
first node to hold it: after the pull requests' checks, the lab rehearsal is the only live check before it
|
||||
reaches every node. A machine that fails to apply it, or whose provisioner reports a module's files failed
|
||||
(ADR 0362), is named in the rollout's end and raises apply-failing, and is fixed forward with a new pull
|
||||
request into main; one that says nothing within fifteen minutes is named silent and gets its builds from
|
||||
the backlog rollout once it is heard from (ADR 0360 §10).
|
||||
|
||||
**Nothing breaks on the way.** ADR 0349 clause 17: each file is filled the old way or the new way, never
|
||||
both, by the form of its manifest, and the old forms stay accepted until the last module has moved.
|
||||
Inside the release this still holds: the media catalogue and the applications kept in their own
|
||||
repositories move after it, so the old forms stay accepted until they have (R1).
|
||||
|
||||
## What exists already, measured
|
||||
|
||||
Counted 2026-10-14 at mesh-controller `1601cb1`, the node-engine (`mesh-host`) `1a26778`, mesh-sdk `a2d2af0`, mesh-catalog
|
||||
`ebf7f37`.
|
||||
|
||||
| Piece | Today | Becomes |
|
||||
|---|---|---|
|
||||
| the controller's composer | `catalogue.Resolution.compose` fills every namespace but `${secret:}` in one loop (`declaration.go` ~1000–1085); the namespaces and their fields are tabled in `fillers` | fills placement only, for a module that has moved |
|
||||
| settings | tables `settings`, `settings_history`; the `settings` verb (`modules.go` 407–760); proposals and asks (`proposals.go`, 975 lines); trusted files; `takes-settings` merging; about 4 100 lines and 2 500 of tests | the `mesh-config` module and its buckets; the verb `config` |
|
||||
| placement kept as settings | `places`, `accesses` and the kept networks are read from settings layers | facts of an assignment, the controller's own |
|
||||
| secrets | the controller mints, seals and accepts every secret (`internal/secrets`, `inventory/secrets.go` 1 125 lines; tables `secret`, `module_secret`, `secret_ask`, `operator_key`); the vault (`mesh-vault`, TypeScript) is read-only and never asked | the vault, in Go, holding and minting; handing out on request and reply |
|
||||
| bindings | grant files and binding files composed by the controller (`declaration.go` 750–818); `${bound:}` 117 uses | each provider answers its consumers; the vault mints the pair's credential |
|
||||
| contributions | `${contribution:}` (4 collectors), `${shell:}` (3), `${environment:}` (1); 22 contributors | collections, filled by the provisioner client |
|
||||
| file text | 225 file resources in 88 modules, all inline (about 193 KB); 12 modules keep a second copy in 34 files with a test that the two agree (ADR 0349's context counted eleven; the twelfth is lemurs) | template files under `files/`, packed by the module's build |
|
||||
| packing a directory | the builder's `archive` artifact packs a tree and publishes it by digest; the node-engine fetches and unpacks it | the same mechanism carries a module's templates |
|
||||
| the node-engine's file writing | `applyFile`, `applyBlock`, `applyInto`, create-once, drift written back, `Keep` backups, restart-on and reload-on, container reads | the same code, reached also through a narrow local write verb |
|
||||
| the node-engine's local verbs | one socket (`cli.sock`), peer-checked by uid, forwarding to the controller; no local write | a second, narrow socket for the provisioner client |
|
||||
| bus identities | one principal per node for the tool runner, one per module with a process of its own | the provisioner client gets one of its own |
|
||||
| the SDK | KV, watches, asks, caller proof; no sealing helper; the TypeScript `provisioner` is a database-provider harness | a seal box, provider helpers, and readers for config, bound values and secrets |
|
||||
|
||||
**Two measurements decide the order.** The builder can already pack a module's directory, and the
|
||||
controller already fills every placeholder at registration time from text; so template files can ship
|
||||
first, the controller reading the text back from the packed `files/`, and nothing on a node changes. And
|
||||
the node-engine's writing code needs no change in kind, only a second way in; so the provisioner client
|
||||
reuses it rather than writing as root itself.
|
||||
|
||||
## The order the work allows
|
||||
|
||||
```
|
||||
package waits for package waits for
|
||||
F1 — P1 —
|
||||
F2 — P2 P1
|
||||
F3 — P3 F3
|
||||
F4 — P4 P3
|
||||
T1 — P5 P2, P4, T1, K3
|
||||
T2 T1 P6 P5
|
||||
T3 T4 T5 T2 P7 P6
|
||||
T6 T1 V1 F1
|
||||
K1 T1 (same file) V2 V1
|
||||
K2 K1 V3 V2
|
||||
K3 F3, K2 V4 V3, P5
|
||||
K4 K3 V5 V3, V4
|
||||
K5 F2, K4 B1 P5, V5
|
||||
K6 K1 B2 B1
|
||||
K7 K5, P6 B3 B2
|
||||
L1 P6 L2 L1
|
||||
S1 P6 R1 T5, T6, K6, K7, P7, V4, B3, L2, S1
|
||||
R2 R1
|
||||
```
|
||||
The issue tracker carries these as `depends-on` links (thirteen issues also link what they wait for
|
||||
through another package), so the packages ready to start are those whose linked packages are all merged
|
||||
into the release's branches. An issue's "Waits for: … delivered" means merged into the release's
|
||||
branches: nothing is delivered before the release. The first wave touches different repositories and different files:
|
||||
**T1, F1, F3, P1 and F4**. K1 waits for T1 because both change `internal/catalogue/manifest.go`; P3
|
||||
waits for F3 because both change `internal/broker`; F2 joins as soon as a slot is free.
|
||||
|
||||
## 0 — Foundations
|
||||
|
||||
**F1 (issue 650) — A seal box in the SDK.** *mesh-sdk. Half a day.* The ADR 0206 box (X25519, HKDF-SHA256,
|
||||
AES-256-GCM) moves from the licence manager into `mesh-sdk/go/seal`, with a TypeScript twin, and the
|
||||
licence manager uses it. **Proof:** a byte-for-byte test against the licence manager's existing sealed
|
||||
values; the licence manager's tests pass unchanged.
|
||||
|
||||
**F2 (issue 651) — Placement leaves the settings layers.** *mesh-controller. Two days.* `places`, `accesses` and the
|
||||
kept networks become facts of an assignment, held by the controller beside the assignment, with a
|
||||
migration moving each existing layer's keys; `Settle` stops letting a setting replace a provider's
|
||||
served keys. **Proof:** composition of every node before and after the migration is byte-identical;
|
||||
the settings layers hold no placement key afterwards.
|
||||
|
||||
**F3 (issue 652) — The assignments and the modules' config declarations on the bus.** *mesh-controller. Two days.*
|
||||
The controller keeps the bucket of assignments design 54 decides (which module on which node), and a
|
||||
bucket of each registered module's declared config values, written when they change, with one event per
|
||||
write. mesh-config needs both to resolve. **Proof:** a test against a real bus: an assignment and a
|
||||
registration each change exactly one key and raise one event.
|
||||
|
||||
**F4 (issue 653) — The words.** *hq. Half a day.* The glossary gains *config value*, *predefined key*, *provision*
|
||||
and *provisioner client*, *collection*, and the ownership words *owned*, *seed* and *managed*. The word "setting" goes on *config
|
||||
value*'s *Not:* line only in R2, once the 67 documents still using it are reworded; until then the
|
||||
glossary says it is the older word. **Proof:** `words.py` passes.
|
||||
|
||||
## 1 — Template files (ADR 0349 clause 2)
|
||||
|
||||
**T1 (issue 654) — A file resource may name a template.** *mesh-controller. Three days; the largest package of the
|
||||
stage.* A file resource takes `"template": "files/<name>"` in place of `content`; the module's build packs
|
||||
`files/` into an archive artifact of its own (the existing `archive` packing); registration and `Resolve`
|
||||
read each template's text back from the packed tree into `content`, so composition and the node-engine
|
||||
are unchanged; `module check` and the merge gate read templates from the repository tree beside the
|
||||
manifest; a file naming both `template` and `content` is refused. **Proof:** composition tests: a module
|
||||
with its text in `files/` composes byte-identically to the same module with it inline; the merge gate
|
||||
judges a change to a template alone.
|
||||
|
||||
**T2 (issue 655) — The twelve modules with copies.** *mesh-catalog. A day.* adwaita, clipmenu, dunst, feh,
|
||||
gnome-keyring, i3, i3status-rust, lemurs, picom, rofi, screen-lock and xdg move to their templates; the
|
||||
34 inline copies and the `sameAsSource` tests go. **Proof:** each module's composed declaration on its
|
||||
nodes is unchanged (the merge gate's plan says no file changes).
|
||||
|
||||
**T3 (issue 656) — The desktop modules.** *mesh-catalog. A day.* Every remaining module assigned only to
|
||||
workstations. **Proof:** as T2.
|
||||
|
||||
**T4 (issue 657) — The servers' modules.** *mesh-catalog. Two days.* Every remaining module with a build. **Proof:**
|
||||
as T2.
|
||||
|
||||
**T5 (issue 658) — The ten modules without a build.** *mesh-catalog. A day.* ca-trust, de-spiegel, invoicing,
|
||||
local-model-consumer, matrix, node-env, only-office, sshd, step-ca and supabase gain an archive build for
|
||||
their templates. **Proof:** as T2; afterwards no manifest in the catalogue carries file text.
|
||||
|
||||
**T6 (issue 659) — The media catalogue and the applications.** *mesh-media-catalog and each application's
|
||||
repository. Two days.* **Proof:** as T2, per repository.
|
||||
|
||||
## 2 — Config (ADR 0348)
|
||||
|
||||
**K1 (issue 660) — A module declares its config.** *mesh-controller. Two days.* The manifest takes `config` (type,
|
||||
description, validation, trusted, `predefined`, `default` or `required`), `takes-config` and
|
||||
`${config:<key>}`, with `settings`, `takes-settings` and `${setting:}` read as aliases; registration
|
||||
refuses everything ADR 0348's "How it is checked" lists for registration: a value without type or
|
||||
description; a value with neither a default nor `required`; a default failing its validation; an
|
||||
object or list not trusted; a predefined key not on the list or of another type (with K2); an untrusted
|
||||
value falling back to a trusted key; a value neither in a template nor read at runtime; a placeholder
|
||||
naming an undeclared value; a `reload-on` or `restart-on` naming an undeclared file; a moved module's
|
||||
mergeable file not naming its keys; a mesh word or `puid`/`pgid` in `takes-config`. **Proof:** the registration refusal tests, one per
|
||||
rule; every manifest in the catalogue still registers.
|
||||
|
||||
**K2 (issue 661) — The list of predefined keys.** *mesh-controller and mesh-catalog. A day.* `predefined.json` at the
|
||||
catalogue's root, with `font-size` and `agent-managed-settings` first; the build validates it and every
|
||||
manifest naming a key; the merge check refuses removing a key still named or holding a value; the newest
|
||||
trunk build's list is published. **Proof:** the merge check refuses a planted removal; an older build's
|
||||
list does not replace a newer one.
|
||||
|
||||
**K3 (issue 662) — The mesh-config module.** *mesh-catalog. Four days.* A Go module holding the seat `mesh-config`:
|
||||
the buckets `mesh-config_values`, `_resolved`, `_predefined`, `_proposals`; the four sources resolved from
|
||||
F3's assignments and declarations; the verb `config` (set, clear, list, history, narrowed by module and
|
||||
node); validation; the hand-act log; `config-changed`; only its account writes its buckets. Terminal-only
|
||||
for trusted values until K4. **Proof:** its tests resolve each source in turn; a test against a real bus
|
||||
refuses a put from another account.
|
||||
|
||||
**K4 (issue 663) — Trusted values are proposed and approved.** *mesh-catalog. Two days.* mesh-config asks under its
|
||||
own name, with its own grant and cap of open asks, binds each option to the level, key and value, records
|
||||
the approver beside the value, and checks the proposal on Approve (ADR 0277's rules, ADR 0348 clause 8).
|
||||
**Proof:** a proposal test end to end with a stand-in channel; an Approve against a changed value is
|
||||
refused.
|
||||
|
||||
**K5 (issue 664) — The values move, and `settings` forwards.** *mesh-controller and mesh-catalog. Two days.* Each
|
||||
existing layer moves into mesh-config (a node layer to `module.<node>.<module>`; a whole-mesh layer to
|
||||
mesh config, a manifest default or each node, as ADR 0348's consequences say; claude-code's managed
|
||||
settings to `agent-managed-settings`), recorded with why; for a module not yet handed to the provisioner
|
||||
the controller reads `mesh-config_resolved` where it read its layers (ADR 0349 clause 17's old way,
|
||||
an exception to ADR 0348 §9 that R1 removes), and `settings` answers by forwarding to `config`. **Proof:** composition of
|
||||
every node before and after is byte-identical; `settings` and `config` list the same values.
|
||||
|
||||
**K6 (issue 665) — The catalogue says config.** *mesh-catalog. A day.* The 15 modules using `${setting:}`, the 9
|
||||
declaring settings and the 4 with `takes-settings` move to the new names and the full declaration.
|
||||
**Proof:** the catalogue's check finds no old name in a moved module.
|
||||
|
||||
**K7 (issue 666) — The operator's acceptance on dunst.** *live. An hour.* With every level cleared the first
|
||||
workstation's dunstrc says 10; a mesh font size, then that node's font size, then dunst's own value on
|
||||
that node each appear in its dunstrc within the call's answer, and the history shows each change with who and why. Needs P6.
|
||||
|
||||
## 3 — The provisioner (ADR 0349 clauses 3–5, 11, 13)
|
||||
|
||||
**P1 (issue 667) — The node-engine knows who owns a placed file.** *`mesh-host`. A day.* `declaration.File` gains
|
||||
`owner-at-runtime` (`owned`, `seed`, `managed`), mapped onto today's paths (drift written back,
|
||||
create-once, `into`), and the declaration gains the list of files the provisioner fills. **Proof:** the
|
||||
apply tests: each ownership behaves as its old form did.
|
||||
|
||||
**P2 (issue 668) — The narrow local write verb.** *`mesh-host`. Two days.* A second unix socket accepts a write only from
|
||||
the provisioner client's account (`SO_PEERCRED`) and only for a path the declaration names as filled,
|
||||
writes through `applyFile`, `applyBlock` or `applyInto` with `Keep`, and fires restart-on, reload-on and
|
||||
container reads as an apply does. **Proof:** tests: an undeclared path and another uid are refused; a
|
||||
write under `reload-on` reloads its program once.
|
||||
|
||||
**P3 (issue 669) — The provisioner client's own bus account.** *mesh-controller. A day.* A principal of its own per
|
||||
node for `node-provisioner`, granted to read its node's keys in each domain's bucket, publish
|
||||
`prov.applied.<node>.>` and ask the vault; nothing else. **Proof:** the grants test, and a test against a
|
||||
real bus refusing a read of another node's keys.
|
||||
|
||||
**P4 (issue 670) — The provisioner's server.** *mesh-catalog. Two days; after P3.* A Go module holding `mesh-provisioner` once:
|
||||
the registry of which domain answers which kind, the buckets `_registry`, `_needs`, `_applied` and
|
||||
`_collect`. **Proof:** its tests; the buckets listed live.
|
||||
|
||||
**P5 (issue 671) — The provisioner's client.** *mesh-catalog. Four days.* A Go service on every node, outside the tool
|
||||
runner: fetches a module's templates by digest, records needs, watches each owner's keys for its node,
|
||||
fills only when every value is there, writes through P2, reports `prov.applied`, writes an `owned` file
|
||||
back on drift, removes a module's files and needs when it leaves, fills from current values when its node
|
||||
comes back. Its first owners are placement (from the declaration) and config (from
|
||||
`mesh-config_resolved`). **Proof:** the client's test list in ADR 0349, against a real bus.
|
||||
|
||||
**P6 (issue 672) — The controller hands a module to the provisioner, and the rollout waits for it.** *mesh-controller.
|
||||
Two days.* For a module marked moved, the declaration carries placement, the templates' digest and the
|
||||
filled-files list, and the controller runs no other filling pass for it; a rollout counts the module applied
|
||||
on a machine only after the node-engine's report and that node's `prov.applied` for the new build, and a
|
||||
failed `prov.applied` is a failed apply (ADR 0362). dunst moves first. **Proof:** composition and rollout
|
||||
tests; dunst on the first workstation filled by the client, live.
|
||||
|
||||
**P7 (issue 673) — The node-engine stops writing placed files from the declaration.** *`mesh-host`. A day.* For a moved
|
||||
module the engine writes its filled files only through P2, and removes as an orphan only what the verb
|
||||
placed. **Proof:** the apply tests; no placed file written twice in a day's journal on the
|
||||
first workstation.
|
||||
|
||||
## 4 — Secrets (ADR 0349 clauses 6–11, 16; issue 647)
|
||||
|
||||
**V1 (issue 674) — The vault in Go, holding secrets.** *mesh-catalog. Four days.* `mesh-vault` is rewritten in Go:
|
||||
its own store, a copy of each secret sealed to the operator key, `get` on request and reply sealed to the
|
||||
asker's key (F1's box; the node-engine's key format is settled here and written down), `secret-changed`
|
||||
without a value, no secret in any bucket. **Proof:** the vault's test list in ADR 0349.
|
||||
|
||||
**V2 (issue 675) — The vault's verbs.** *mesh-catalog and mesh-controller. Three days.* `secret accept`, `secret-ask`
|
||||
(with ADR 0277 §24–§27's record and bounds in the vault's store), `issued` (ADR 0305, the provider's tool
|
||||
answering the vault alone) and `rotate` become the vault's; the controller's verbs forward. **Proof:** the
|
||||
`secret-ask` and issuing provider tests in ADR 0349.
|
||||
|
||||
**V3 (issue 676) — The secrets move to the vault.** *mesh-controller and mesh-catalog. Two days.* Every row of
|
||||
`secret`, `module_secret`, `secret_ask` and the operator key's public half are copied into the vault as they are,
|
||||
nothing rotated, and read back through the vault before anything points at it. **Proof:** for every
|
||||
secret, the vault's sealed answer opens to the same value the controller's does.
|
||||
|
||||
**V4 (issue 647) — A node's secrets come from the vault.** *mesh-catalog, mesh-controller, `mesh-host`.
|
||||
Three days.* The client fills `${secret:}` from the vault's `get`; for a moved module the controller seals
|
||||
nothing; the core's credentials travel sealed by the vault to their node and the node-engine opens them.
|
||||
**Proof:** a secret rotated in the vault reaches its file without a new declaration, live.
|
||||
|
||||
**V5 (issue 677) — The vault mints pair credentials.** *mesh-catalog. Two days.* One value per provider assignment and
|
||||
provision, remade for all holders when the vault sees a consumer's need recorded or deleted (ADR 0158
|
||||
§4–§6). A need recorded for a bind the vault already holds from V3 adopts that value and remakes
|
||||
nothing, so the move itself rotates no credential. **Proof:** the provider test in ADR 0349, with a stand-in provider.
|
||||
|
||||
## 5 — Bindings (ADR 0349 clause 12)
|
||||
|
||||
**B1 (issue 678) — A provider's helper in the SDK.** *mesh-sdk. Two days.* Go: watch the needs for a kind, write one
|
||||
entry per consumer, refuse a derived value for a multi-holder consumer, answer from the vault's pair
|
||||
credential. The TypeScript `provisioner` harness is renamed so the word means one thing. **Proof:** its
|
||||
tests with a stand-in bus.
|
||||
|
||||
**B2 (issue 679) — PostgreSQL answers its consumers.** *mesh-catalog and mesh-controller. Three days.* The first
|
||||
provider moves; its grant files and binding files are retired. **Proof:** every database consumer on
|
||||
every node connects after the move with the credential it had, adopted by the vault as V5 says, live.
|
||||
|
||||
**B3 (issue 680) — The other providers.** *mesh-catalog. A week, one pull request per provider.* **Proof:** as B2, per
|
||||
provider; afterwards the controller composes no grant file.
|
||||
|
||||
## 6 — Collections (ADR 0349 clause 13)
|
||||
|
||||
**L1 (issue 684) — A collection is declared and filled by the client.** *mesh-controller and mesh-catalog. Three days.*
|
||||
A collector requires a collection; a contributor provides an item from a template; the client fills the
|
||||
collector's template from `_collect`. `${contribution:}`, `${shell:}` and `${environment:}` read as
|
||||
collections. **Proof:** zsh's file holds every placed contributor's item and loses one whose contributor
|
||||
leaves.
|
||||
|
||||
**L2 (issue 681) — The collectors and contributors move.** *mesh-catalog. Two days.* zsh first, then the bar, the
|
||||
display session, the hotkeys, xorg, power and node-env, with their 22 contributors. **Proof:** each
|
||||
collector's composed file is unchanged at the move.
|
||||
|
||||
## 7 — A module's code asks at runtime (ADR 0349 clause 14)
|
||||
|
||||
**S1 (issue 685) — The SDK's readers.** *mesh-sdk and mesh-catalog. Two days.* `config`, `bound` and their watches,
|
||||
answered by the node's client for what the manifest declares. `secrets` refuses until the module runs
|
||||
under a bus account of its own, which waits on issue 569 and its decision. **Proof:** the SDK's test in
|
||||
ADR 0349; claude-code reads its managed settings through it.
|
||||
|
||||
## 8 — The old forms go (ADR 0349 clause 17)
|
||||
|
||||
**R1 (issue 682) — Refused and removed.** *mesh-controller, `mesh-host`, mesh-catalog. Three days.* Once every module of
|
||||
every catalogue has moved: the catalogue's check refuses inline text, `settings`, `${setting:}` and the
|
||||
old contribution forms; the controller's filling passes but placement, its settings, secret and binding
|
||||
code and tables (a new migration) go; the node-engine keeps opening sealed values for the core's files
|
||||
only; the controller stops reading `mesh-config_resolved`. **Proof:** the controller's test that its
|
||||
code opens no secret outside genesis and reads no config value; the absence of the
|
||||
removed code.
|
||||
|
||||
**R2 (issue 683) — The record says what runs.** *hq. Two days.* The as-is designs are written, designs 27 and 32 are
|
||||
amended, the 67 documents that still say "setting" in running prose are reworded so the glossary can
|
||||
retire the word (F4 left it off the *Not:* line for them), design 49 lists the Config domain, ADR 0348's and 0349's built status is said, and this document's status becomes `implemented`.
|
||||
|
||||
## What is deliberately not here
|
||||
|
||||
- **Secrets in a module's own code** wait on issue 569 (a module's tools run as the tool runner's
|
||||
account) and its decision; S1 ships without them.
|
||||
- **Release separated from delivery** stays issue 644's, and **the bus's replicas** issue 645's.
|
||||
- **Renaming "walk"** is issue 646's.
|
||||
|
||||
## How this list is kept true
|
||||
|
||||
Each package's proof that runs in its tests is run when it is merged into the release's branch, and its
|
||||
line here gains the date and the commit; a proof that runs live is run in the lab rehearsal and again
|
||||
after the release's delivery. A package whose proof fails is not reworded; the failure is recorded under it and the package
|
||||
stays open. The issue tracker says which packages are worked now; this document says only the order.
|
||||
@@ -47,15 +47,16 @@ document is written and this one's status becomes `implemented`.
|
||||
| [`42-the-machines-modules-in-order.md`](42-the-machines-modules-in-order.md) | **In progress.** The order the machines' modules of research 026 and 027 are built and rolled out: every machine's first (sudo, localization, time sync, pacman, logrotate, avahi, systemd, docker, `~/.ssh`, scripts, kernel), then both workstations', then one machine model's; each proven on one workstation before the rest | ADR 0173, ADR 0182, ADR 0205 |
|
||||
| [`45-a-core-that-cannot-fail-silently.md`](45-a-core-that-cannot-fail-silently.md) | **Designed.** The core says when it is wrong, refuses what is stale or unreadable, heals what it knows, upgrades one machine at a time with a witness that rolls it back, and is checked against the real mesh before merge: the writers and signals tables, the condition store, `doctor`, the minimal output channel, healers, the lease and report order, staged upgrades, the facts snapshot and replays, in six phases | ADR 0227, ADR 0224, ADR 0218 |
|
||||
| [`46-the-conversation-with-the-operator.md`](46-the-conversation-with-the-operator.md) | **Designed.** The mesh tells and asks its operator over channels that are holders of two kinded benches, `channel` and `intake`, declaring capabilities from a fixed vocabulary; the router orders them by work context and never lowers the bar; an answer that performs an action is checked and performed by the controller, on a TOTP code or a verified Telegram sender, never on a desk click alone; operator messages addressed or answered by the mesh's responder, untrusted input kept as data, references for what words may not carry, and a recoverable factor; Telegram first, in six phases | ADR 0234, ADR 0227 |
|
||||
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the walk, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
|
||||
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the rollout, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
|
||||
| [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of eleven domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | ADR 0244, ADR 0006, ADR 0008, ADR 0302 |
|
||||
| [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | ADR 0247, ADR 0223 |
|
||||
| [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | ADR 0251, ADR 0189 |
|
||||
| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **In progress.** A node shares one folder at a time through `nfs-server`, which provides it as a provision to the private network only, mapping every client to the folder's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | ADR 0263, ADR 0051, ADR 0092 |
|
||||
| [`53-the-issue-tracker.md`](53-the-issue-tracker.md) | **Designed.** The mesh keeps its issues in one module, `issues`, on the control-node: every number from one counter advanced by compare-and-set, the cycle's issue rules refused at the write, an event per change; an issue lives whole in the tracker, hq keeping no issue content (ADR 0291) | ADR 0275, ADR 0291 |
|
||||
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, walks, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
|
||||
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, rollouts, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
|
||||
| [`55-the-knowledge-base.md`](55-the-knowledge-base.md) | **Designed.** The mesh keeps its decision records and research efforts whole in one module, `mesh-knowledge`, on the control-node: number, status, topic, typed links both ways, each decision's clauses with their own status, and the reasoning, with every version; a record is proposed, checked at the write, reviewed by the knowledge base's own judgement and accepted there, with the operator's Approve when it needs it; agents ask what holds in one bounded call | ADR 0290 |
|
||||
| [`56-the-mail-client-set-up-by-the-mesh.md`](56-the-mail-client-set-up-by-the-mesh.md) | **Designed.** On every node with a display server, the `betterbird` module installs Betterbird from a pinned upstream release mirrored into the artifact store, ships the Owl add-on pinned and installed by Betterbird's enterprise policy, and declares the operator's mail accounts in a trusted setting without credentials; each account signs in once at a person's first use, and one the profile already holds is adopted by its address | ADR 0316, ADR 0317 |
|
||||
| [`57-building-config-and-provisioning.md`](57-building-config-and-provisioning.md) | **Designed.** The work of ADR 0348 and ADR 0349 as 37 packages, each an issue with its proof: foundations, template files, config in its own module, the provisioner and its client, secrets in the vault, bindings from providers, collections, a module's runtime readers, and removing the old forms; at most five worked at once | ADR 0348, ADR 0349 |
|
||||
|
||||
## Not yet written
|
||||
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ for _, n := range nodes {
|
||||
```
|
||||
|
||||
So a provider on the same machine is never a `Provider` in `world.Offered`, never becomes a
|
||||
`Needed`, and the credential loop — which walks `resolved.Needs` — has nothing to walk. Every step
|
||||
`Needed`, and the credential loop — which traverses `resolved.Needs` — has nothing to traverse. Every step
|
||||
is individually reasonable and the sum is a silent gap.
|
||||
|
||||
## Why it was not noticed
|
||||
|
||||
@@ -92,7 +92,7 @@ node that looks like a real one.
|
||||
|
||||
## Fixed
|
||||
|
||||
Needs fan out per consuming module in one place, after the resolution walk. The credential's key
|
||||
Needs fan out per consuming module in one place, after the resolution rollout. The credential's key
|
||||
gains the consuming module, the grant file is named after both halves of the consumer, needs are
|
||||
matched by provision *and* module, and the provisioners name the role and the access key after the
|
||||
module rather than the machine. The refusal is gone because there is nothing left to refuse.
|
||||
|
||||
+1
-1
@@ -87,7 +87,7 @@ too new to have anybody watching it.
|
||||
## Fixed
|
||||
|
||||
**The registry module names its image and is added, not built.** The lab's artifact-store test now
|
||||
walks the only path open to a real first mesh — the manifest goes in directly, no builder and no
|
||||
rollouts the only path open to a real first mesh — the manifest goes in directly, no builder and no
|
||||
store involved — and passes.
|
||||
|
||||
**And the cycle is refused where it is written.** A manifest that provides `artifact-store` and
|
||||
|
||||
@@ -45,7 +45,7 @@ field it does not know — whole. That was
|
||||
mesh now records which host each machine reports and `status` names every machine running an older one
|
||||
than another does.
|
||||
|
||||
So the flag day is visible before it is walked into, which it was not when this was filed. It does not
|
||||
So the flag day is visible before the mesh runs into it, which it was not when this was filed. It does not
|
||||
make the field free: **the oldest host in the mesh is still what the mesh may send**, and one machine of
|
||||
four is behind today. A sequence that an old host refuses takes that machine out of the mesh's reach
|
||||
entirely — worse than the replay it prevents, which has never been observed.
|
||||
|
||||
@@ -18,7 +18,7 @@ half-understood declaration off a machine, and exactly what makes this expensive
|
||||
|
||||
[Issue 087](../087-the-controller-cannot-tell-a-host-is-too-old/00-report.md) is resolved: the mesh now
|
||||
records the host version each machine reports and `status` names every machine running an older host
|
||||
than another does. The flag day is visible before it is walked into, which it was not on 2026-09-23.
|
||||
than another does. The flag day is visible before the mesh runs into it, which it was not on 2026-09-23.
|
||||
|
||||
That makes the cost measurable rather than hypothetical, and the measurement is the reason this is not
|
||||
being built today.
|
||||
|
||||
@@ -72,7 +72,7 @@ what the sidecar is has nowhere in the design layer to look, which is how this w
|
||||
resource type no decision records and contradicting an accepted one. The rule is "no design
|
||||
without a decision"; the check is "no design without *a* decision." An unenforced rule is
|
||||
indistinguishable from a wrong one, and these two documents are what that gap looks like when
|
||||
something walks through it.
|
||||
something rollouts through it.
|
||||
|
||||
## Open questions
|
||||
|
||||
|
||||
@@ -92,7 +92,7 @@ built from a repository elsewhere, or a mesh whose forge module is down — and
|
||||
event performs, so the two cannot disagree about what "moved" means.
|
||||
|
||||
**Bases first needed edges, and there were none.** The order this report asked for was written and
|
||||
walked a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
|
||||
traversed a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
|
||||
digest was never in the file the builder read edges from. A build now reports what it was handed, the
|
||||
control plane records it by artifact path, and the order is read from each module's newest build.
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ Every node's resolution is read first, and the names are attributed across them
|
||||
terminus serves the name**: among the providers a name reaches, the one that is not itself published
|
||||
under a labelled name through another provider. An identity provider is routed through the proxy and
|
||||
so is a consumer of names, not their end; the proxy contributes no label to anyone and is. The rule
|
||||
knows nothing of what "route" means — it reads the graph the modules declared — and it walks nodes,
|
||||
knows nothing of what "route" means — it reads the graph the modules declared — and it rollouts nodes,
|
||||
requirements and contributions in order, so one mesh yields one region. Every shape of contribution
|
||||
is counted, so a module routed under several names has every one of them resolved.
|
||||
|
||||
|
||||
+1
-1
@@ -73,5 +73,5 @@ give.
|
||||
|
||||
All three faults are closed on the controller's main: one ask in flight (8d4e940), a merge rebuilds what
|
||||
stands on it (7e701c0), and a merge produces a tiered plan the mesh keeps, read with `plans` (d2ed6d5,
|
||||
76f2756, ADR 0162). Plans were since carried into walks and deliveries (ADR 0236, ADR 0239); the order a
|
||||
76f2756, ADR 0162). Plans were since carried into rollouts and deliveries (ADR 0236, ADR 0239); the order a
|
||||
person kept is the order the tiers give.
|
||||
|
||||
+1
-1
@@ -86,6 +86,6 @@ machine. That is the path the console exists to replace, so it is a workaround a
|
||||
Both pull requests named above are merged and live. Read through the console on 2026-10-08:
|
||||
`mesh_describe mesh-controller.plan` declares `node` (required) and `files`; `mesh-controller.push`
|
||||
declares `node`, `behind`, `cause` and `why`, and its description says that a push naming no machine
|
||||
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` walks every verb it serves:
|
||||
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` exercises every verb it serves:
|
||||
an argument a verb does not declare is refused, and every flag of a verb's command is in its schema
|
||||
or accounted for.
|
||||
|
||||
+1
-1
@@ -46,7 +46,7 @@ Pull requests: novox/mesh-tools#14 and novox/mesh-controller#71 — open, not me
|
||||
first line of its answer, that it is a push of the whole mesh and which machines it is sending, and
|
||||
its last line names the machines told. `plan` gains `files`, `push` gains `behind` (the whole mesh
|
||||
said outright, refused beside `node`), and the listing verbs a `limit`.
|
||||
- **Checked by tests that walk every verb the controller announces:** for every combination of a
|
||||
- **Checked by tests that exercise every verb the controller announces:** for every combination of a
|
||||
verb's declared arguments, the verb either refuses or every argument given changes the command it
|
||||
runs; every verb refuses an argument it does not declare, and `node` where it takes none; and every
|
||||
flag of the command a verb runs — read from the command's own source — is an argument of the verb's
|
||||
|
||||
@@ -26,7 +26,7 @@ list alone. The flush has no such narrowing.
|
||||
|
||||
## What it costs
|
||||
|
||||
`record` is the policy for a change that must be walked through the mesh by hand. A named push cannot
|
||||
`record` is the policy for a change that must be rolled out through the mesh by hand. A named push cannot
|
||||
do that, so the policy does not hold the change back from anything but the merge. The command's name
|
||||
says one machine, and the command's output is the only place that says otherwise.
|
||||
|
||||
|
||||
@@ -61,7 +61,7 @@ The mesh-delivery tests `checkask_test.go` cover:
|
||||
- An announcement that carries a decided gate status uses it.
|
||||
- H2 re-asks once, a second re-ask is refused by the table, and a late verdict still settles the delivery.
|
||||
|
||||
The table tests walk the new row, and the pairs the table does not hold are refused by name. Live: once
|
||||
The table tests exercise the new row, and the pairs the table does not hold are refused by name. Live: once
|
||||
the fix is rolled out, the stalled delivery has its check asked at the owner's first tick. The controller
|
||||
journal then shows its verdict, and `show` moves from proposed to checked to ready without a person
|
||||
acting.
|
||||
|
||||
@@ -39,7 +39,7 @@ The mesh's own record shows it:
|
||||
- **What each machine reports.** `postgres.server`, `mongodb.server` and nine `mailu.*` containers
|
||||
have run only since 18:57 on the control node, and `postgres.server` since 18:56 on the anchor.
|
||||
|
||||
No walk held back for later was involved. The sends were the plan's own sends to its first machines.
|
||||
No rollout held back for later was involved. The sends were the plan's own sends to its first machines.
|
||||
|
||||
## Why it happened
|
||||
|
||||
@@ -53,7 +53,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
|
||||
push". So the gate did not judge the recorded builds, the refusal of ungated sends did not see
|
||||
them, and the gated send carried them. `record` held only against the end of a person's push to
|
||||
another machine (issue 259's `heldBack`) and against the bus (the bus step). It did not hold
|
||||
against a plan's walk, a rollback, a healer or a rotation.
|
||||
against a plan's rollout, a rollback, a healer or a rotation.
|
||||
3. **Nothing tells a person, before or during a send, that it will recreate every container of a
|
||||
module at once.** A change that only alters how containers are declared reads as harmless. A
|
||||
health check, an environment variable or a label each recreates the container all the same,
|
||||
@@ -64,7 +64,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
|
||||
## What is at stake
|
||||
|
||||
`record` is how the mesh keeps the providers, the network path and the irreplaceable data out of
|
||||
unattended walks. Tonight showed that it holds only until some other module on the same machine
|
||||
unattended rollouts. Tonight showed that it holds only until some other module on the same machine
|
||||
rolls out. This was still true after the incident: plex (`record`, irreplaceable data) was rebuilt
|
||||
by a media-catalogue merge at 19:01 and sent nowhere. The next gated send to the anchor, for any
|
||||
module, would recreate it.
|
||||
|
||||
@@ -13,8 +13,8 @@ the tier durations measured for the repository: three times their ninetieth perc
|
||||
than 30 minutes. So the watchdog and the line a person reads used different start times and could use
|
||||
different bounds. The line was the one that was wrong.
|
||||
|
||||
The line for a walk waiting on its delivery's word had the same flaw. It said "for" the time since the
|
||||
last save. The watchdog for that wait (S16) counts from when the walk was opened.
|
||||
The line for a rollout waiting on its delivery's word had the same flaw. It said "for" the time since the
|
||||
last save. The watchdog for that wait (S16) counts from when the rollout was opened.
|
||||
|
||||
Ruled out: a clock difference between the machine running the verb and the store. The count was small
|
||||
and kept growing again from zero, which only a column being rewritten explains.
|
||||
@@ -44,8 +44,8 @@ nothing.
|
||||
## Fix (mesh-controller)
|
||||
|
||||
- The line, the `since` and `late` fields of `status --json`, and the count of late plans all start from
|
||||
when the plan entered its tier. If the delivery let the walk go after that, they start from the go. A
|
||||
walk still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
|
||||
when the plan entered its tier. If the delivery let the rollout go after that, they start from the go. A
|
||||
rollout still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
|
||||
called late. A plan read without the tier stamp falls back to its last save.
|
||||
- LATE uses the bound the watchdog uses: the measured bound for the plan's repository, or the 30-minute
|
||||
floor when nothing has been measured. The watchdog's start time and bound now come from the same two
|
||||
@@ -53,4 +53,4 @@ nothing.
|
||||
- The loop compares the plan before and after each step and saves only when they differ. The fields the
|
||||
store stamps itself are left out of that comparison.
|
||||
- A refused step is logged and saved only when its note is new. The refusal's wording called the gated
|
||||
walk a "release plan", a name the glossary retires; it now says "walk".
|
||||
rollout a "release plan", a name the glossary retires; it now says "walk" (since renamed rollout, issue 646).
|
||||
|
||||
@@ -23,7 +23,7 @@ another, with nothing on the mesh noting the difference.
|
||||
|
||||
ADR 0238 makes the delivery plan the planner's one answer for a pull request's check, the merge gate and
|
||||
the merge. The plan counts a new module's directory as something the merge builds. The merge handler
|
||||
read the same answer (`touchedBy`'s `added`) and dropped it: it only walks modules the catalogue already
|
||||
read the same answer (`touchedBy`'s `added`) and dropped it: it only rollouts modules the catalogue already
|
||||
holds, and a new one is not held until it is built. The code said so outright: "a new module, which a
|
||||
check judges before it merges and a merge does not build". The take-in refuses a build off the trunk
|
||||
with "merge it, and the merge builds it". So one part of the controller promised the build and another
|
||||
|
||||
@@ -29,7 +29,7 @@ already built from it are, and without waiting (`askNewModules`). The build's ta
|
||||
module, as it does for a hand `build`. Nothing is sent: no machine is assigned a new module, and
|
||||
assigning one stays a person's act.
|
||||
|
||||
The new module is outside the merge's plan. A plan walks modules the catalogue holds, tier by tier, and
|
||||
The new module is outside the merge's plan. A plan rollouts modules the catalogue holds, tier by tier, and
|
||||
sends each built one to the machines running it. A new module has no machine and nothing standing on it,
|
||||
and the plan's ask for a module the catalogue does not hold fails the plan.
|
||||
|
||||
|
||||
+9
-9
@@ -7,16 +7,16 @@ opened: 2026-10-08
|
||||
## Symptom
|
||||
|
||||
On 2026-10-08 an agent proving the tunnel join (novox/mesh-lab PR #61) on the laptop could not run the
|
||||
two-node walk, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
|
||||
two-node rollout, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
|
||||
reported by any check before someone tried to run a bed:
|
||||
|
||||
1. **The walk's scenario stocks a packet-filter runtime image that cannot be built.**
|
||||
1. **The rollout's scenario stocks a packet-filter runtime image that cannot be built.**
|
||||
`scripts/build-module-runtime.sh` copied the tool runner's compiled TypeScript and its `node_modules`
|
||||
out of mesh-tools. mesh-tools has had no package at its root since the runtime became `node-tools`, a
|
||||
Go binary that launches every bundle (ADR 0175,
|
||||
ADR 0193),
|
||||
so the script failed at its first step.
|
||||
2. **The walk's builder dialled the old broker.** It was started with the old bus's guest account on
|
||||
2. **The rollout's builder dialled the old broker.** It was started with the old bus's guest account on
|
||||
loopback. The mesh moved its bus to NATS (ADR 0106,
|
||||
ADR 0127,
|
||||
ADR 0131), and the builder
|
||||
@@ -27,8 +27,8 @@ reported by any check before someone tried to run a bed:
|
||||
4. **The replays prover could not run a test in mesh-tools**, because that repository's Go module lives
|
||||
in a subdirectory (`node-tools/`), and the prover ran `go test` only at a repository's root.
|
||||
|
||||
The same day, while trying to run the walk again, two more gaps surfaced: the object store's
|
||||
provisioner image, which the walk also stocks, cannot be built, because the client image it copies
|
||||
The same day, while trying to run the rollout again, two more gaps surfaced: the object store's
|
||||
provisioner image, which the rollout also stocks, cannot be built, because the client image it copies
|
||||
from is gone upstream; and the laptop could not reach the public container registry, which the lab's
|
||||
egress check needs (an outside condition, not the lab's fault).
|
||||
|
||||
@@ -56,9 +56,9 @@ In novox/mesh-lab PR #63, stacked on #61:
|
||||
builds `node-tools` from mesh-tools' Go module, and runs it as the image's entrypoint. The image serves
|
||||
on the node's runtime credential: the Go runtime refuses to serve a module on that module's own
|
||||
credential.
|
||||
- **The walk no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
|
||||
- **The rollout no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
|
||||
names an image (its verbs are a tools bundle the node's runtime launches), so the image was never run.
|
||||
- **The walk's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
|
||||
- **The rollout's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
|
||||
an own `broker` secret, as the catalogue's build agent does. Assigning it to the anchor issues the
|
||||
account ([issue 203](../203-a-fresh-assignment-is-pushed-before-its-credential-exists/00-report.md)).
|
||||
The bus's user list is placed, the push writes the sealed credential, and the builder starts on it.
|
||||
@@ -71,12 +71,12 @@ In novox/mesh-lab PR #63, stacked on #61:
|
||||
Still open:
|
||||
|
||||
- `builds.test.ts` still dials the old broker. Its bed enrols no machine, so the mesh has nowhere to
|
||||
deliver a builder's credential. It needs the walk's enrolment, or it should be retired into the walk.
|
||||
deliver a builder's credential. It needs the rollout's enrolment, or it should be retired into the rollout.
|
||||
- Beds that run a per-module runtime container on the module's own credential (the tools beds for the
|
||||
wiki and the code forge, and the audit bed) run into the runtime's refusal. They need the node's
|
||||
runtime credential instead.
|
||||
- The object store provisioner's image needs a client source that still exists.
|
||||
- The walk has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
|
||||
- The rollout has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
|
||||
public container registry.
|
||||
|
||||
## How it is checked
|
||||
|
||||
+5
-5
@@ -14,25 +14,25 @@ script now does those steps by hand. Verified: the image for the packet filter b
|
||||
answers `initialize` and `tools/list`; `node-tools` in it, started against a real bus with a plain URL,
|
||||
launches the bundle and serves nothing until the mesh issues a membership, which is correct.
|
||||
|
||||
**Why the walk stocked that image at all.** The tunnel branch passed the held images to
|
||||
**Why the rollout stocked that image at all.** The tunnel branch passed the held images to
|
||||
`deriveTheFilterOn`, which turns the catalogue's manifest into the lab's form: build section gone, each
|
||||
container's artifact replaced by a stocked image. The catalogue's nftables manifest has no container,
|
||||
so no image is referenced. Removing the build section is what the bed needed; the stocked image was
|
||||
not.
|
||||
|
||||
**The builder.** mesh-controller's `cmd/mesh-builder` reads `MESH_BROKER_FILE` and nothing else for the
|
||||
bus. Without it, it says it has no credential. The walk set the old broker's variable, which the builder
|
||||
bus. Without it, it says it has no credential. The rollout set the old broker's variable, which the builder
|
||||
no longer reads. A builder's account is a module's account (`builder issue` and `module issue` both mint
|
||||
through the inventory), composed into the bus's user list and sealed to the machine as the module's
|
||||
`broker` own secret. Assigning a module that declares that secret now issues it in the same act (issue
|
||||
203). The catalogue's build agent is that module, but it runs the builder as a container from an
|
||||
artifact the mesh builds, and it needs the artifact store and package registry bound. The walk has
|
||||
artifact the mesh builds, and it needs the artifact store and package registry bound. The rollout has
|
||||
neither before it can build, so it declares a lab module with the same claim and the same secret. In
|
||||
this bed the bus comes from the bundle, so its user list is placed by hand, the step the walk already
|
||||
this bed the bus comes from the bundle, so its user list is placed by hand, the step the rollout already
|
||||
takes at enrolment.
|
||||
|
||||
**The uplink and the prover** were each fixed by the agent that met them (mesh-lab #61 and #62), and are
|
||||
listed here only so the lab's state on that day is in one place.
|
||||
|
||||
**Ruled out.** That the walk failed because of the tunnel changes. The builder and runtime faults are in
|
||||
**Ruled out.** That the rollout failed because of the tunnel changes. The builder and runtime faults are in
|
||||
code that #61 does not touch, and they reproduce on main.
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
the planner's sort stalled. Every two-way pair was a cycle, whatever the rules.
|
||||
- The person's order, the controller first, was right for this change. The new node-engine calls a verb
|
||||
the new controller adds, which is ADR 0246's case: a verb is promised before it is required.
|
||||
*Engine before controller* comes from ADR 0236's order for its walks, where the node-engine had to
|
||||
*Engine before controller* comes from ADR 0236's order for its rollouts, where the node-engine had to
|
||||
accept something new first. The rule reads only module names, so it cannot tell the two cases apart. *Built
|
||||
by* is read from the graph.
|
||||
|
||||
|
||||
@@ -59,7 +59,7 @@ and fails only when it cannot read.
|
||||
|
||||
**Merged.** mesh-controller#135 (the build agent judges a pull request by the base branch's
|
||||
`merge-check.sh`) merged on 2026-10-08 at 09:10 UTC, and mesh-lab#65 (R310) at 09:10 UTC. Every later
|
||||
controller delivery carries it; the last, of mesh-controller#170, finished its walk on 2026-10-09 at
|
||||
controller delivery carries it; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at
|
||||
00:11 UTC, and `mesh-controller.status` named no machine behind.
|
||||
|
||||
**The open half is closed.** `gitea_branch_protection_get` for the media catalogue's main, read on
|
||||
|
||||
+5
-5
@@ -32,11 +32,11 @@ was asked of a new, unrelated pull request.
|
||||
## Why it is a design issue
|
||||
|
||||
ADR 0239
|
||||
decision 10, *the switch*, adopts the walks open when mesh-delivery is assigned. It says nothing of the merges the
|
||||
decision 10, *the switch*, adopts the rollouts open when mesh-delivery is assigned. It says nothing of the merges the
|
||||
bus still holds from before: the forge's `pull.merged` is kept on the bus and heard again by a new consumer.
|
||||
Each was taken as a merge just made whose head was never heard, so the table's row *merged unchecked → held*
|
||||
held each for a person. The table had no state for "this already happened", and `held`'s only ways on were
|
||||
a release, which would walk it, and a stop, which says a person stopped a delivery that was never running.
|
||||
a release, which would roll it out, and a stop, which says a person stopped a delivery that was never running.
|
||||
Healer H2's `close` has no transition from `held`, rightly: a held delivery is the operator's.
|
||||
|
||||
Decision 5 says what a transition owes outside is "retried until done". An effect the forge answers can never
|
||||
@@ -45,14 +45,14 @@ be done was retried without end, every ten seconds, and nothing said so.
|
||||
## Fix
|
||||
|
||||
- **A new final state, `retired`: history.** A delivery made from the forge's word at its merge, whose merge
|
||||
was heard fifteen minutes or more after it happened and whose walk, if the controller keeps one, is not
|
||||
was heard fifteen minutes or more after it happened and whose rollout, if the controller keeps one, is not
|
||||
open, is history. At adoption it goes straight to `retired` by the observed row *history*, before any check,
|
||||
hold or walk. Its pull request is not told: the forge may no longer hold it, or holds another under its
|
||||
hold or rollout. Its pull request is not told: the forge may no longer hold it, or holds another under its
|
||||
number. Its transitions are still said on the bus and kept on the commit's note.
|
||||
- **A person's verb for the ones already held: `retire-history`**, with why. One delivery by id, refused when
|
||||
the rule does not take it, or every held one (of one repository when named), answering how many it retired
|
||||
and naming each one it left held. `dry` says what it would do and changes nothing. It is the act
|
||||
*held → retired* of the table, never a release: nothing is walked, delivered or put back.
|
||||
*held → retired* of the table, never a release: nothing is rolled out, delivered or put back.
|
||||
- **Owed work ends.** The forge's answer that what an effect is for does not exist (a 404, or "does not
|
||||
exist") is final: the effect is given up with one line, kept among the owner's refusals. Any other failure is
|
||||
tried again later each time, from ten seconds doubling to an hour, and given up after sixty tries, saying so.
|
||||
|
||||
+5
-5
@@ -5,20 +5,20 @@
|
||||
- 699 deliveries were `held`. 688 say "merged without a passing check", all made between 23:05:27 and
|
||||
23:05:29 on 2026-10-06, the owner's first start. A sample across repositories shows merges from
|
||||
2026-10-02 to two hours before that start, each with the transition "its head was not heard by this owner,
|
||||
so it is made from the forge's word", then *merged unchecked → held*. None has a walk.
|
||||
- The other 11 are held for other reasons: a merge with no walk opened within ten minutes (10), and a group
|
||||
so it is made from the forge's word", then *merged unchecked → held*. None has a rollout.
|
||||
- The other 11 are held for other reasons: a merge with no rollout opened within ten minutes (10), and a group
|
||||
whose composed check did not pass (1). They are not history, and this issue leaves them alone. One of them
|
||||
was past its bound, which makes the 689th finding.
|
||||
|
||||
**Where it comes from.** The holder subscribes to the forge's `pull.merged` on the bus after reading back its
|
||||
state and the controller's walks. The bus hands a new consumer the merges it still holds. For each, the holder
|
||||
state and the controller's rollouts. The bus hands a new consumer the merges it still holds. For each, the holder
|
||||
finds no delivery for the head, so it makes one from the forge's word. The pull request's statuses at its
|
||||
merge do not include a passing `mesh/merge-gate` (the check did not exist for most of them), so the table's
|
||||
*merged unchecked* row holds it.
|
||||
|
||||
**What would close them, and why not.**
|
||||
|
||||
- `release` walks a held delivery: wrong, these reached the machines already.
|
||||
- `release` rollouts a held delivery: wrong, these reached the machines already.
|
||||
- `stop` takes it to `stopped`, which says a person stopped a delivery and sets `mesh/delivery` to error on
|
||||
the commit. Wrong in meaning, and it would ask the forge for a view on a pull request that is gone.
|
||||
- Healer H2's `close` reads `held`'s bound, which names no H2 transition, and refuses: "the state is the
|
||||
@@ -43,6 +43,6 @@ mesh-catalog#121 (the delivery's owner retires history and ends owed work the fo
|
||||
under ADR 0250.
|
||||
|
||||
**Seen live.** `mesh-delivery.deliveries` on 2026-10-09 lists the held history as `retired`, all at
|
||||
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The walk of
|
||||
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The rollout of
|
||||
mesh-catalog#121's own delivery reads `stopped`; the holder that answers today serves the fix all the
|
||||
same, and `mesh-controller.status` names no machine behind.
|
||||
|
||||
+2
-2
@@ -53,6 +53,6 @@ lift.
|
||||
every other state in the table moves on its facts?
|
||||
3. Should `mesh/delivery-group` be required on the trunks, so a group cannot merge before its composed
|
||||
check passed? To-be 47 leaves this to the operator; this issue is the case that setting decides.
|
||||
4. The group's order put the catalogue's member first, and the node-engine's member walked at once on the
|
||||
controller's own path. Is a group's order meant to hold for a member whose walk never waits for
|
||||
4. The group's order put the catalogue's member first, and the node-engine's member rolled out at once on the
|
||||
controller's own path. Is a group's order meant to hold for a member whose rollout never waits for
|
||||
mesh-delivery?
|
||||
|
||||
+6
-6
@@ -3,7 +3,7 @@
|
||||
**2026-10-08.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`),
|
||||
from `mesh-delivery.show` of both members, and from the holder's journal on the control node.
|
||||
|
||||
**Where the hold is set.** `groupTurns` (holder.go) walks a closed group's members in order. For a member
|
||||
**Where the hold is set.** `groupTurns` (holder.go) rolls out a closed group's members in order. For a member
|
||||
in `published` it computes `composed`: the group's check exists, its verdict is `pass` or `warning`, and it
|
||||
names the members' current heads. When `composed` is false it writes the member's `HeldWhy` ("its group's
|
||||
composed check did not pass for the heads that merged") and settles it, and the table's `hold` row takes it
|
||||
@@ -20,7 +20,7 @@ to `held`. `composed` is false in three different cases, and the code does not t
|
||||
look at the members.
|
||||
- `groupTurns` skips a member in `held` (`default: return nil`), so the tick after the verdict sees the
|
||||
passing check and does nothing with it.
|
||||
- The table has two rows out of `held`: `go` (the walk started on a word that was not this owner's) and
|
||||
- The table has two rows out of `held`: `go` (the rollout started on a word that was not this owner's) and
|
||||
`release` (a person's act). There is no observed row that takes a held delivery back when the reason it
|
||||
was held is gone, and `HeldWhy` is never cleared once set.
|
||||
|
||||
@@ -41,7 +41,7 @@ check was asked. To-be 47 lists whether to require it as not decided.
|
||||
- A lost verdict. The holder logged it at 10:10:21 and the group's record holds it.
|
||||
- The order. The group's order was asked and kept before the check; it plays no part in the hold.
|
||||
|
||||
**Not this issue, noted.** The node-engine's member walked at once (`published` → `delivering` at the
|
||||
**Not this issue, noted.** The node-engine's member rolled out at once (`published` → `delivering` at the
|
||||
same instant), because the controller's own path does not wait for mesh-delivery (ADR 0239 decision 8).
|
||||
The group's order put it second. Open question 4 in the report asks whether that is intended.
|
||||
|
||||
@@ -57,9 +57,9 @@ ADR 0270
|
||||
its group's composed check is asked for the current heads and has no verdict.
|
||||
- **A hold whose reason is gone lifts, by observation.** A member held because its group's composed check had not
|
||||
passed goes back to `published` (the event *lifted*) once the check passes or warns for the heads that merged,
|
||||
while its own walk is still open. It then takes its turn in order.
|
||||
- A held member whose own walk was since taken over by a later walk is not lifted, since its walk can no longer
|
||||
deliver it. It is `covered` when that later walk was delivered and sent its module (issue 319).
|
||||
while its own rollout is still open. It then takes its turn in order.
|
||||
- A held member whose own rollout was since taken over by a later rollout is not lifted, since its rollout can no longer
|
||||
deliver it. It is `covered` when that later rollout was delivered and sent its module (issue 319).
|
||||
|
||||
Open question 3, whether `mesh/delivery-group` is required on the trunks, stays the operator's setting. Open
|
||||
question 4 is not touched.
|
||||
|
||||
+5
-5
@@ -2,7 +2,7 @@
|
||||
opened: 2026-10-08
|
||||
---
|
||||
|
||||
# 317. A held delivery's module was walked by a later merge's plan
|
||||
# 317. A held delivery's module was rolled out by a later merge's plan
|
||||
|
||||
## Symptom
|
||||
|
||||
@@ -22,7 +22,7 @@ took `openrazer` along. `mesh-controller.plans`:
|
||||
| c78b5fc9 (#124's merge) | 11:07:02 | superseded at tier 0 by the plan at 86077780; docker, gitea, openrazer planned there again |
|
||||
| 86077780 (#123's merge) | 11:07:03 | tier 0: distribution, docker, gitea and openrazer built from c78b5fc9; openrazer's gate on the laptop judging |
|
||||
|
||||
The controller's walk of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
|
||||
The controller's rollout of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
|
||||
c78b5fc9, beside `docker` and `node-tools`. From 11:10:58 UTC the laptop's condition for openrazer said
|
||||
its account was *relogin needed*. That verdict exists only in the build #125 added, so the held change was
|
||||
running on the laptop while its delivery said it waited for a person.
|
||||
@@ -41,10 +41,10 @@ gone out. A person who releases it later releases nothing; a person who stops it
|
||||
|
||||
## Open questions
|
||||
|
||||
1. When a newer merge supersedes an older plan, does a module the older plan's walk was holding for its
|
||||
1. When a newer merge supersedes an older plan, does a module the older plan's rollout was holding for its
|
||||
delivery's word come along, or stay held with that delivery?
|
||||
2. If it comes along, what does the held delivery become: superseded by the newer delivery, with the hold
|
||||
carried over to it, or something else? Today the table has no row from `held` for a superseded walk.
|
||||
carried over to it, or something else? Today the table has no row from `held` for a superseded rollout.
|
||||
3. A trunk commit holds every earlier commit, so a later build of a module carries the held change in
|
||||
any case. Is a hold on one delivery meaningful once a later merge of the same repository is walked,
|
||||
any case. Is a hold on one delivery meaningful once a later merge of the same repository is rolled out,
|
||||
or does a hold have to hold the module on that trunk until it is released?
|
||||
|
||||
+9
-9
@@ -3,7 +3,7 @@
|
||||
**2026-10-08.** Read from `mesh-controller.plans` (the list, and each plan by id), `mesh-delivery.show`
|
||||
and `deliveries`, and the code on the main branches of the controller and the catalogue.
|
||||
|
||||
**The held delivery's walk.** #125's merge made the plan at a6385479. Its walk waited for mesh-delivery's
|
||||
**The held delivery's rollout.** #125's merge made the plan at a6385479. Its rollout waited for mesh-delivery's
|
||||
word (`waited_for: mesh-delivery`, `waits: true`) and was never let go, because its delivery was held. Its
|
||||
one module, `openrazer`, was "not yet asked": nothing was built.
|
||||
|
||||
@@ -11,32 +11,32 @@ one module, `openrazer`, was "not yet asked": nothing was built.
|
||||
plan. For every open plan of the same repository and branch made before it, it takes each module the
|
||||
older plan had not built (not asked, asked and not answered, or built and not yet sent), adds it to the
|
||||
newer plan, and closes the older plan as superseded with the note "… planned there again". It does not
|
||||
ask why the older plan had not built them. A walk waiting for a held delivery's word is open, so it is
|
||||
ask why the older plan had not built them. A rollout waiting for a held delivery's word is open, so it is
|
||||
superseded like any slow plan. This happened three times over (3f552274, c78b5fc9, 86077780), each newer
|
||||
plan folding in what the one before it held.
|
||||
|
||||
The newest plan's own delivery (#123) was not held, so mesh-delivery let its walk go, and `openrazer` was
|
||||
built from c78b5fc9. The controller's walk of the builds waiting for a gate then carried it to the laptop,
|
||||
The newest plan's own delivery (#123) was not held, so mesh-delivery let its rollout go, and `openrazer` was
|
||||
built from c78b5fc9. The controller's rollout of the builds waiting for a gate then carried it to the laptop,
|
||||
since a send carries every move waiting on its machine (ADR 0236).
|
||||
|
||||
**Why the held delivery did not notice.** mesh-delivery's table has `superseded` rows only from
|
||||
`published` and `delivering` ("a newer delivery to the same trunk took over its walk"). From `held` there
|
||||
are only `go` and `release`. When the walk record said `superseded`, no row applied, so the delivery stayed
|
||||
`held` with `held_why` unchanged, and its walk shows "superseded … openrazer planned there again". Two of
|
||||
are only `go` and `release`. When the rollout record said `superseded`, no row applied, so the delivery stayed
|
||||
`held` with `held_why` unchanged, and its rollout shows "superseded … openrazer planned there again". Two of
|
||||
the three later deliveries (#122, #124) went to `superseded` correctly, through the `published` row.
|
||||
|
||||
**What the design says.** ADR 0218 decision 3 makes a newer merge take over an older plan's unfinished
|
||||
work, and was written before deliveries could be held. ADR 0239 makes a held delivery wait for a person,
|
||||
and to-be 47 says that nothing of a published delivery is registered before its walk starts, so that a
|
||||
and to-be 47 says that nothing of a published delivery is registered before its rollout starts, so that a
|
||||
send cannot carry it early. Neither says what a supersession does to a module a hold is keeping back.
|
||||
|
||||
**Ruled out.**
|
||||
|
||||
- A person's act. #125's transitions end at `hold`, with no `release`, and its own walk was superseded
|
||||
- A person's act. #125's transitions end at `hold`, with no `release`, and its own rollout was superseded
|
||||
while still waiting (`let_go` empty), so no `plans go` started it.
|
||||
- Issue 295's path, a recorded build carried by a send: here the build was asked by a plan, not left
|
||||
recorded.
|
||||
|
||||
**Located** in the controller's `supersededBy`, which folds a held walk's modules into a newer plan, and in
|
||||
**Located** in the controller's `supersededBy`, which folds a held rollout's modules into a newer plan, and in
|
||||
mesh-delivery's table, which has no way out of `held` when that happens, so the delivery's state stops
|
||||
being true.
|
||||
|
||||
+8
-8
@@ -2,11 +2,11 @@
|
||||
opened: 2026-10-08
|
||||
---
|
||||
|
||||
# 318. One module's wait for a new login held every other module's walk
|
||||
# 318. One module's wait for a new login held every other module's rollout
|
||||
|
||||
## Symptom
|
||||
|
||||
On 2026-10-08 at 11:08 UTC the controller opened its walk of the builds waiting for a gate: the laptop
|
||||
On 2026-10-08 at 11:08 UTC the controller opened its rollout of the builds waiting for a gate: the laptop
|
||||
first, then the workstation, then the control node. On the laptop it sent three moves in one send and
|
||||
judged them together:
|
||||
|
||||
@@ -23,14 +23,14 @@ account was *relogin needed* from 11:10:58 UTC, which
|
||||
ADR 0252
|
||||
expects until the operator logs out and in again.
|
||||
|
||||
Meanwhile three other walks could not send to the rest of the mesh. `mesh-controller.plans` said of the
|
||||
Meanwhile three other rollouts could not send to the rest of the mesh. `mesh-controller.plans` said of the
|
||||
tool runner's (`novox/mesh-tools` d4845f7c, `node-tools`), the catalogue's (86077780, `docker`) and the
|
||||
controller's (950562af, `build-agent`): "a build waits there for its gate: … the workstation (…, openrazer
|
||||
88135ad0 → c78b5fc9) — a walk sends them, one machine at a time, each judged".
|
||||
88135ad0 → c78b5fc9) — a rollout sends them, one machine at a time, each judged".
|
||||
|
||||
At 11:21:06 UTC the laptop's gate failed: "not healthy within 10m0s of its apply: its unit
|
||||
openrazer.daemon … is unhealthy", and `openrazer` was put back to 88135ad0 on the laptop
|
||||
(`build.openrazer.<laptop>.rolled-back`). The walk ended `failed`. After it, all three walks still said
|
||||
(`build.openrazer.<laptop>.rolled-back`). The rollout ended `failed`. After it, all three rollouts still said
|
||||
the same: `openrazer` c78b5fc9 waits on the workstation for a gate, so nothing is sent there or to the
|
||||
machines after it.
|
||||
|
||||
@@ -40,14 +40,14 @@ The wait was known, and it was a person's: ADR 0252 says a new login holds a del
|
||||
that machine. It held every module on every machine after it instead, for changes that had nothing to do
|
||||
with it: the tool runner, the container engine and the build agent. The end was worse than the wait. The
|
||||
gate's bound turned "waiting for a person" into "failed", the build that added the account group was put
|
||||
back, and the walk that carries the other modules failed with it. A failed walk of the waiting builds
|
||||
back, and the rollout that carries the other modules failed with it. A failed rollout of the waiting builds
|
||||
does not open again on its own: a person releases it.
|
||||
|
||||
Three things meet here, each sound alone:
|
||||
|
||||
- a send carries every move waiting on its machine, and the gate judges them all at once
|
||||
(ADR 0236);
|
||||
- no other send may go where a move waits for a gate, so every walk waits for the one that judges it;
|
||||
- no other send may go where a move waits for a gate, so every rollout waits for the one that judges it;
|
||||
- a module's health that waits for a person reads as *not yet healthy*, which fails at the gate's bound.
|
||||
|
||||
## Open questions
|
||||
@@ -61,5 +61,5 @@ Three things meet here, each sound alone:
|
||||
back the account group it added, so the login the gate waited for can no longer help.
|
||||
4. The daemon's unit had failed since 10:13 UTC, almost an hour before the new build reached the laptop.
|
||||
Should a gate fail a build for a failure that was there before it was sent?
|
||||
5. While a move waits for a gate on one machine, should walks of unrelated modules still be refused there,
|
||||
5. While a move waits for a gate on one machine, should rollouts of unrelated modules still be refused there,
|
||||
or only those whose send would carry it?
|
||||
|
||||
+11
-11
@@ -1,6 +1,6 @@
|
||||
# 318 — diagnosis
|
||||
|
||||
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the walk of the waiting builds by id) between
|
||||
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the rollout of the waiting builds by id) between
|
||||
11:17 and 11:22 UTC, `mesh-controller.conditions` and `status`, and the controller's code on its main
|
||||
branch (`cmd/mesh-controller`).
|
||||
|
||||
@@ -9,15 +9,15 @@ gate all of them as carried moves. `judgeMoves` (gate.go) reads each module on e
|
||||
worst reading. Any module *not yet healthy* resets the passes to zero for the whole send: `docker` and
|
||||
`node-tools` were healthy, and the gate still said "0 of 3 passes" because `openrazer` was not. Past
|
||||
`gateBound` (ten minutes after the apply) a *not yet healthy* gate fails. `g.Failing` names only the
|
||||
modules found wanting, so only `openrazer` was put back, but the walk itself ended `failed`.
|
||||
modules found wanting, so only `openrazer` was put back, but the rollout itself ended `failed`.
|
||||
|
||||
**Everything else waits for that verdict.** `ungatedIn` (release.go) refuses any send that is not a person's
|
||||
and would carry a move no gate has seen, outside the send's own scope. A plan's sends to the rest of its
|
||||
machines carry the machine's whole declaration, so while `openrazer` c78b5fc9 waited for a gate on the
|
||||
workstation, the tool runner's, the catalogue's and the controller's walks were each refused there, and
|
||||
they say so on every retry. They do not fail; they wait for the walk of the waiting builds, which goes one
|
||||
workstation, the tool runner's, the catalogue's and the controller's rollouts were each refused there, and
|
||||
they say so on every retry. They do not fail; they wait for the rollout of the waiting builds, which goes one
|
||||
machine at a time and had stopped on the laptop. After it failed, the release code's rule applies: a
|
||||
failed walk of the waiting builds stops the next one opening on its own until a person releases it.
|
||||
failed rollout of the waiting builds stops the next one opening on its own until a person releases it.
|
||||
|
||||
**How *relogin needed* reached the gate.** ADR 0252 decision 5 says the node-engine states the account's
|
||||
health as the module's resource of kind `account`, unhealthy with *relogin needed*, and that "the
|
||||
@@ -33,8 +33,8 @@ running session lacks the group the daemon checks for. That is a guess, not chec
|
||||
readings existed before or because of the person's step, and neither is a fault the build brought.
|
||||
|
||||
**What the design says.** ADR 0252's consequences say "a new login holds a delivery of that module on that
|
||||
machine" and accept that a walk that starts on a workstation waits for the person. They do not consider the
|
||||
other modules carried in the same send, the walks refused behind it, or the gate's ten-minute bound turning
|
||||
machine" and accept that a rollout that starts on a workstation waits for the person. They do not consider the
|
||||
other modules carried in the same send, the rollouts refused behind it, or the gate's ten-minute bound turning
|
||||
the wait into a rollback. ADR 0236 makes one send judge everything waiting on a machine, to stop unjudged
|
||||
builds from moving; it does not distinguish a module waiting for a person from one that is broken.
|
||||
|
||||
@@ -50,15 +50,15 @@ Whether the account actually left the group was not checked.
|
||||
- A machine-wide fault on the laptop: its node-engine reported `applied` and current, and the gate's
|
||||
wanting named one module's unit.
|
||||
|
||||
**Located** in the controller's walk and first-node gate: one verdict per send (`judgeMoves`), sends refused
|
||||
**Located** in the controller's rollout and first-node gate: one verdict per send (`judgeMoves`), sends refused
|
||||
on any machine where a move waits (`ungatedIn`), and no reading for a health that waits for a person, so a
|
||||
known wait fails at the bound and stops every walk behind it.
|
||||
known wait fails at the bound and stops every rollout behind it.
|
||||
|
||||
**Decided and fixed, 2026-10-08**, by ADR 0254.
|
||||
Open questions 1 to 3: *relogin needed*, and a unit failing in that same account's own manager, read *waits
|
||||
for a person*, which passes with the wait carried and is said as the module's condition `relogin-needed`;
|
||||
nothing is put back for it. Question 2's other half: each module of a send counts its own passes, and, after
|
||||
the settle time, in the walk path, keeps a pass when another fails beside it. Question 5: other walks are
|
||||
the settle time, in the rollout path, keeps a pass when another fails beside it. Question 5: other rollouts are
|
||||
still refused while a move waits for a gate (ADR 0236 §4a stands). With the wait a pass, 318's case leaves
|
||||
nothing waiting for them. A fault decided before the settle time, and a plan's own tier path, still leave
|
||||
the healthy modules of a failed send without a verdict (ADR 0254, consequences). Question 4, a unit that
|
||||
@@ -71,4 +71,4 @@ between 12:41 and 13:03 UTC; the node-engine's and the controller's deliveries r
|
||||
|
||||
**Seen live.** On 2026-10-09 `mesh-controller.conditions` holds `module.openrazer.<machine>.relogin-needed`
|
||||
for the laptop and the workstation, raised at 21:02 UTC: "Its update is in place and nothing was undone",
|
||||
and no walk waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
|
||||
and no rollout waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
|
||||
|
||||
+7
-7
@@ -15,8 +15,8 @@ What they are, read with `mesh-delivery.deliveries` and `show`:
|
||||
| how many | repository | why held | what it builds |
|
||||
|---|---|---|---|
|
||||
| 4 | hq | merged without a passing check | nothing: "the change touches no module of the mesh's graph" |
|
||||
| 2 | catalogue | no walk opened within ten minutes of the merge | a new module, sent nowhere |
|
||||
| 8 | controller | no walk opened within ten minutes of the merge | the controller, the build agent, the route proxy |
|
||||
| 2 | catalogue | no rollout opened within ten minutes of the merge | a new module, sent nowhere |
|
||||
| 8 | controller | no rollout opened within ten minutes of the merge | the controller, the build agent, the route proxy |
|
||||
| 2 | catalogue | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | one module each |
|
||||
|
||||
Two examples:
|
||||
@@ -26,7 +26,7 @@ Two examples:
|
||||
`proposed` to `held`: "merged without a passing check". Its plan builds nothing.
|
||||
- `novox/mesh-controller@ec3769a8a669`, controller #105. Ready at 2026-10-07 00:26 UTC, merged at 00:27,
|
||||
held at 00:37: "merged, and the controller opened no walk for it within 10m0s". The controller's modules
|
||||
have been walked since from later commits of the same main, for example the group
|
||||
have been rolled out since from later commits of the same main, for example the group
|
||||
`feat/delivery-checks-verb`, whose controller member is `delivered`.
|
||||
|
||||
Five of them (the four hq merges and controller #105) are past the held bound of one day. The self-check's
|
||||
@@ -39,7 +39,7 @@ A held delivery asks a person "does this go on?". For these sixteen the question
|
||||
a later commit of the same trunk has already delivered what they held, or they hold nothing to deliver. The
|
||||
state table offers a person four ways out of `held`, and none fits:
|
||||
|
||||
- `release` starts a walk of the old commit: wrong for a commit a later one superseded, and nothing for one
|
||||
- `release` starts a rollout of the old commit: wrong for a commit a later one superseded, and nothing for one
|
||||
that builds nothing.
|
||||
- `retire-history` is for history made from the forge's word (ADR 0250),
|
||||
and refuses every one of these because their heads were heard.
|
||||
@@ -57,7 +57,7 @@ The issue states the need; the decision is for graduation.
|
||||
|
||||
- **An end for a held delivery that was superseded**, as `published` and `delivering` already have
|
||||
(`superseded`, "a newer delivery to the same trunk took over its walk"): either because a later commit's
|
||||
delivered walk carried the same modules on the same trunk, or because it builds nothing at all.
|
||||
delivered rollout carried the same modules on the same trunk, or because it builds nothing at all.
|
||||
- **Automatic where nothing is lost**: a held delivery whose plan builds nothing has nothing for a person to
|
||||
decide.
|
||||
- **A person's act, with why, otherwise**: a held delivery whose modules a later commit delivered is ended
|
||||
@@ -70,5 +70,5 @@ The issue states the need; the decision is for graduation.
|
||||
2. The four hq merges were heard at the owner's first start from the bus's history, as announcements rather
|
||||
than merges. Are they history in the sense of ADR 0250, missed by its rule because their head was
|
||||
replayed too?
|
||||
3. Why did the controller open no walk for eight of its own merges within ten minutes? The controller's own
|
||||
path never waits, so its walk may have been missed rather than absent. Not diagnosed here.
|
||||
3. Why did the controller open no rollout for eight of its own merges within ten minutes? The controller's own
|
||||
path never waits, so its rollout may have been missed rather than absent. Not diagnosed here.
|
||||
|
||||
+30
-30
@@ -7,14 +7,14 @@
|
||||
|
||||
| event | kind | needs |
|
||||
|---|---|---|
|
||||
| `go` | observed | its walk started on a word not this owner's (the controller's own path, or a person's `plans go`) |
|
||||
| `release` | a person's act | why and who; it walks the delivery's own commit |
|
||||
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no walk open |
|
||||
| `go` | observed | its rollout started on a word not this owner's (the controller's own path, or a person's `plans go`) |
|
||||
| `release` | a person's act | why and who; it rolls out the delivery's own commit |
|
||||
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no rollout open |
|
||||
| `stop` | a person's act | why and who; from any state not final |
|
||||
|
||||
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the walk's
|
||||
record saying `superseded`. A held delivery whose walk never opened has no walk record to read, and one
|
||||
whose walk was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
|
||||
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the rollout's
|
||||
record saying `superseded`. A held delivery whose rollout never opened has no rollout record to read, and one
|
||||
whose rollout was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
|
||||
does not match it either.
|
||||
|
||||
**Why `retire-history` refuses them.** `isHistory` requires `FromForgesWord()`: the delivery was made from
|
||||
@@ -41,7 +41,7 @@ Neither considers a held delivery whose content a later commit delivered.
|
||||
|
||||
**Ruled out.**
|
||||
|
||||
- A missing release. Releasing any of these would walk an old commit over a newer one, or walk nothing.
|
||||
- A missing release. Releasing any of these would roll out an old commit over a newer one, or roll out nothing.
|
||||
- D14 itself. It reports the state truthfully; the state has no way to change.
|
||||
|
||||
**Located** in mesh-delivery's state table (no end for a superseded or empty held delivery, and history
|
||||
@@ -52,14 +52,14 @@ read by how a delivery was made) and in to-be 47, whose state table carries the
|
||||
**Eleven held deliveries, read again.** On 2026-10-09 eleven deliveries were `held`, each raising D14's
|
||||
`delivery.<id>.stalled` and held for 24 to 48 hours. For each, `mesh-delivery.show` gave its state, why it was
|
||||
held, its merge commit and its plan. A fresh clone of the repository then showed whether the merge commit is an
|
||||
ancestor of the commit a finished walk delivered. `mesh-controller.status` listed none of the modules involved
|
||||
ancestor of the commit a finished rollout delivered. `mesh-controller.status` listed none of the modules involved
|
||||
as behind. All eleven were covered: none held a change missing where it should run.
|
||||
|
||||
**Why the controller opened no walk (open question 3).** This part comes from reading the code, and part of it
|
||||
**Why the controller opened no rollout (open question 3).** This part comes from reading the code, and part of it
|
||||
is still a guess.
|
||||
|
||||
- *A new module.* When a merge only adds a module, the controller's `SourceMoved`
|
||||
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no walk ("built,
|
||||
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no rollout ("built,
|
||||
registered when the build lands, and sent nowhere", issue 300). mesh-delivery's `MovesNothing()`
|
||||
(`model.go`) counts `Plan.New` as something to move, though. So a delivery whose plan is "builds new: …, sent
|
||||
nowhere" is not let through to `published` → `delivered`. Instead the unwalked rule in `holder.go` holds it
|
||||
@@ -67,25 +67,25 @@ is still a guess.
|
||||
read from the code, and it matches `novox/mesh-catalog@60641176abaf`.
|
||||
- *The controller's own merges (a guess, not proved).* `SourceMoved` also returns without a plan in two cases:
|
||||
when the modules the merge touches are "already built from it" (`already > 0`), and when the merge is read as
|
||||
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no walk
|
||||
of its own when another walk, or a person's push, had already built from a later commit. Which of the two hit
|
||||
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no rollout
|
||||
of its own when another rollout, or a person's push, had already built from a later commit. Which of the two hit
|
||||
these merges is not yet known. The controller's journal for the minutes after each merge would say.
|
||||
- *The link from a walk to a delivery.* `deliveryWalking()` (`holder.go`) links a walk to the delivery whose
|
||||
`merged_as` equals the walk's commit, and to no other delivery. A later walk that carried an earlier merge's
|
||||
change therefore never reaches that earlier delivery, and nothing reads "this walk's commit contains your
|
||||
- *The link from a rollout to a delivery.* `deliveryWalking()` (`holder.go`) links a rollout to the delivery whose
|
||||
`merged_as` equals the rollout's commit, and to no other delivery. A later rollout that carried an earlier merge's
|
||||
change therefore never reaches that earlier delivery, and nothing reads "this rollout's commit contains your
|
||||
merge". That is the missing input the first open question asks about. mesh-delivery could answer it from the
|
||||
forge (is the merge commit an ancestor of the delivered walk's commit) together with the walk's moved modules,
|
||||
forge (is the merge commit an ancestor of the delivered rollout's commit) together with the rollout's moved modules,
|
||||
without asking the controller what each machine runs.
|
||||
|
||||
**Four were closed with `stop`, before this issue was found.** On 2026-10-09 the operator agreed to close the
|
||||
held deliveries. Four were stopped with why "covered: its merge … is contained in … which is delivered (walk …)":
|
||||
|
||||
| delivery | merge | contained in | walk |
|
||||
| delivery | merge | contained in | rollout |
|
||||
|---|---|---|---|
|
||||
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier walk of the controller |
|
||||
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same walk |
|
||||
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same walk |
|
||||
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue walk |
|
||||
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier rollout of the controller |
|
||||
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same rollout |
|
||||
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same rollout |
|
||||
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue rollout |
|
||||
|
||||
Their stalled conditions cleared at the next D14 observation. Open D14 conditions went from 11 to 7. As this
|
||||
issue already said, `stop` is the wrong record for them: their merged pull requests now carry `mesh/delivery`
|
||||
@@ -101,12 +101,12 @@ remaining seven were left `held` for this issue's fix.
|
||||
| `novox/hq@2221be11a6ef` | merged without a passing check | merge on hq's main; builds nothing | the same |
|
||||
| `novox/hq@897bcce50252` | merged without a passing check | merge on hq's main; builds nothing | the same |
|
||||
| `novox/hq@ef6ab814b053` | merged without a passing check | merge on hq's main; builds nothing | the same |
|
||||
| `novox/mesh-controller@b98fd0f396c8` | no walk opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
|
||||
| `novox/mesh-controller@ea92af2a7d77` | no walk opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release walk that is done; the step looks stale but is not final |
|
||||
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished walk of the same module | its group is still `delivering`, and its own walk is `superseded` |
|
||||
| `novox/mesh-controller@b98fd0f396c8` | no rollout opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
|
||||
| `novox/mesh-controller@ea92af2a7d77` | no rollout opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release rollout that is done; the step looks stale but is not final |
|
||||
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished rollout of the same module | its group is still `delivering`, and its own rollout is `superseded` |
|
||||
|
||||
The seven show the two ends this issue asks for: four hold nothing to deliver, and three are covered by a
|
||||
later walk.
|
||||
later rollout.
|
||||
|
||||
## 2026-10-09 — the fix, proposed
|
||||
|
||||
@@ -115,12 +115,12 @@ ADR 0270
|
||||
mesh-delivery:
|
||||
|
||||
- **A merge that moves nothing is delivered at once.** `MovesNothing()` no longer counts `Plan.New`: a module
|
||||
the controller builds and sends nowhere opens no walk, so there is nothing to wait for. A merge whose plan moves
|
||||
the controller builds and sends nowhere opens no rollout, so there is nothing to wait for. A merge whose plan moves
|
||||
nothing goes from `proposed`, `checked`, `rejected` or `held` to `delivered`, before the row *merged unchecked
|
||||
→ held*.
|
||||
- **`covered`, a new final state: a later merge that contains it was delivered.** For every module its plan
|
||||
moves, a delivered delivery of the same repository and trunk, merged after it, delivered on its own passing
|
||||
check by the walk of its own merge commit, whose machine steps say the module reached every machine running
|
||||
check by the rollout of its own merge commit, whose machine steps say the module reached every machine running
|
||||
it. The forge then says, through a new read-only tool `gitea_contains` (its compare of the two commits),
|
||||
whether the later merge commit contains this one; merge order is only the first filter. A forge that cannot
|
||||
answer leaves it held. `mesh/delivery` says `covered: a later merge that contains it was delivered: …` as a
|
||||
@@ -128,7 +128,7 @@ mesh-delivery:
|
||||
- **Read on every tick**, so what is held now ends without an event.
|
||||
|
||||
This answers open question 1: mesh-delivery reads it from its own records and asks the forge for ancestry; the
|
||||
controller is not asked. Open question 3, why the controller's own merges opened no walk, is not answered by this fix; it no
|
||||
controller is not asked. Open question 3, why the controller's own merges opened no rollout, is not answered by this fix; it no
|
||||
longer leaves anything held.
|
||||
|
||||
**How to verify after mesh-delivery is delivered with it.** Within a minute of its start:
|
||||
@@ -137,13 +137,13 @@ longer leaves anything held.
|
||||
merges (`builds new: …, sent nowhere`), and none of the five controller merges held for "no walk opened".
|
||||
The hq and module-adding ones are `delivered`; the controller ones are `covered`.
|
||||
2. `mesh-delivery.show` of one of them, for example `novox/mesh-controller@b98fd0f396c8`, ends with a
|
||||
transition `held → covered (covered)` whose why names the later delivery and its walk.
|
||||
transition `held → covered (covered)` whose why names the later delivery and its rollout.
|
||||
3. `mesh-delivery.checks` for its pull request shows `mesh/delivery` as `success covered: …`.
|
||||
4. The forge module is delivered in the same pull request. Until its `gitea_contains` answers, nothing is
|
||||
covered, and the held controller merges stay held.
|
||||
5. At the self-check's next reading, `mesh-controller.conditions` no longer lists their
|
||||
`delivery.<id>.stalled`.
|
||||
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered walk sent
|
||||
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered rollout sent
|
||||
their module. One that stays held is a real question: its change may not run anywhere.
|
||||
|
||||
**Accepted 2026-10-09.** The operator accepted the record named above; it is no longer proposed.
|
||||
|
||||
@@ -52,7 +52,7 @@ not addresses, paths or secrets — so they reached the operator untouched.
|
||||
|
||||
**Nothing said whether the operator was needed, or let them answer.** No field of a condition said whether
|
||||
the operator was needed, so every warning was shown, and an explanation could imply an instruction. The
|
||||
verbs that answer the common cases existed (release or stop a delivery, start a waiting walk, restart a
|
||||
verbs that answer the common cases existed (release or stop a delivery, start a waiting rollout, restart a
|
||||
service, silence a condition), but the desk could not take an answer: `node-notifier.send` had no actions,
|
||||
and the conversation design that adds them (to-be 46) was not built. The one way to act was a session.
|
||||
|
||||
@@ -70,7 +70,7 @@ nothing is not sent. The desk offers the actions, and the operator channel's hol
|
||||
through its seat verb. Every channel says the time in the operator's zone; a clearance is one line.
|
||||
|
||||
- novox/mesh-controller: the condition's three new fields, the plain rule, a wording for every kind the
|
||||
controller raises, the richer words at the producers that know more (a walk's modules, a unit's name, a
|
||||
controller raises, the richer words at the producers that know more (a rollout's modules, a unit's name, a
|
||||
hand-act's cause, a delivery's repository), and a lint over every condition the suite raises.
|
||||
- novox/mesh-catalog: the messenger shows the words, keeps quiet what needs nothing, offers and performs
|
||||
the answers, says since when in the operator's zone (the new setting `time-zone`), says a clearance in
|
||||
|
||||
@@ -88,5 +88,5 @@ To-be 45 §2 is amended: what `status` lists, and the new condition.
|
||||
## Resolved, 2026-10-09
|
||||
|
||||
mesh-controller#150 and mesh-lab#68 (R325) merged on 2026-10-08 at 14:58 UTC. Every later controller
|
||||
delivery carries #150; the last, of mesh-controller#170, finished its walk on 2026-10-09 at 00:11 UTC, and
|
||||
delivery carries #150; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at 00:11 UTC, and
|
||||
`mesh-controller.status` named no machine behind.
|
||||
|
||||
+3
-3
@@ -20,7 +20,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
|
||||
1.4 a second, or about 43 every 30 seconds.
|
||||
- Over the server's whole uptime (39 hours 49 minutes at 14:13 UTC) the average is 39.7 writes a second.
|
||||
|
||||
In that window one walk was in progress, so a few of those writes were real changes. The diagnosis
|
||||
In that window one rollout was in progress, so a few of those writes were real changes. The diagnosis
|
||||
shows why most were not.
|
||||
|
||||
## Why it is an issue
|
||||
@@ -40,7 +40,7 @@ watcher, and the write rate cannot be used to see that deliveries are moving.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **In mesh-delivery's tests:** a reconcile pass over walks that did not change performs no write. The
|
||||
- **In mesh-delivery's tests:** a reconcile pass over rollouts that did not change performs no write. The
|
||||
test's fake store counts writes, and the test fails on any write.
|
||||
- **Live:** over ten minutes in which no delivery changes state, the write count on
|
||||
`$KV.mesh-delivery_deliveries.>` (`nats.nats_subscriptions`) rises by no more than the number of
|
||||
@@ -56,6 +56,6 @@ build agent. So it resolves without a replay. The test above is still the proof.
|
||||
|
||||
- `keep` writes only when the delivery differs from the value it last wrote. Either compare the encoded
|
||||
value with the last one written, or set a changed flag wherever the delivery is changed.
|
||||
- `Reconcile` skips a walk whose revision the delivery already holds.
|
||||
- `Reconcile` skips a rollout whose revision the delivery already holds.
|
||||
- A retry of an owed effect that fails again changes only its try count and next time. Whether that
|
||||
needs a write at every try is a choice for the fix: the count may stay in memory between writes.
|
||||
|
||||
+8
-8
@@ -12,16 +12,16 @@ runtime. `keep` does not compare the delivery with what it wrote before. Every c
|
||||
of transitions is empty.
|
||||
- `settleAndKeep` calls `settle` and passes the result to `moved`. A delivery with nothing to move is
|
||||
still written.
|
||||
- `walkMoved` takes one walk from the controller, records it on its delivery (`linkWalk`) and calls
|
||||
`settleAndKeep`. A walk that has not changed since the last pass leaves the delivery as it was, and
|
||||
- `walkMoved` takes one rollout from the controller, records it on its delivery (`linkWalk`) and calls
|
||||
`settleAndKeep`. A rollout that has not changed since the last pass leaves the delivery as it was, and
|
||||
the delivery is written anyway.
|
||||
- `Reconcile` asks the controller's `delivery-walks` for the walks it keeps and calls `walkMoved` for each.
|
||||
The answer holds every open walk and the last 50 that ended. The ticker in main.go runs `Reconcile`
|
||||
- `Reconcile` asks the controller's `delivery-walks` for the rollouts it keeps and calls `walkMoved` for each.
|
||||
The answer holds every open rollout and the last 50 that ended. The ticker in main.go runs `Reconcile`
|
||||
every 30 seconds.
|
||||
|
||||
So each pass writes about one delivery per walk the controller returns. On 2026-10-08 that was 50 walks
|
||||
(32 done, 14 superseded, 3 failed, 1 walking), against about 43 writes per pass measured. *I think* the
|
||||
difference is walks that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
|
||||
So each pass writes about one delivery per rollout the controller returns. On 2026-10-08 that was 50 rollouts
|
||||
(32 done, 14 superseded, 3 failed, 1 `walking`), against about 43 writes per pass measured. *I think* the
|
||||
difference is rollouts that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
|
||||
|
||||
**What made it 40 a second before today.** Before mesh-catalog #121 (the fix of
|
||||
[issue 313](../313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md), merged
|
||||
@@ -39,5 +39,5 @@ doubling to an hour) and ends them after 60 tries. A retry still writes its deli
|
||||
- *The groups bucket.* `mesh-delivery_groups` has 26 keys, and its subscription counted 719 writes in the
|
||||
same 39 hours.
|
||||
|
||||
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every walk the
|
||||
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every rollout the
|
||||
controller keeps through `settleAndKeep` every 30 seconds.
|
||||
|
||||
+1
-1
@@ -24,7 +24,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
|
||||
|
||||
The long-lived connections were young too. At 14:15 UTC the controller's two were 4 minutes old, the
|
||||
build agents' 2 to 19 minutes, the route proxies' 13 to 17 minutes and the tool runners' 10 minutes to 3
|
||||
hours. Two walks were sending new builds of the controller, the build agent and catalogue modules
|
||||
hours. Two rollouts were sending new builds of the controller, the build agent and catalogue modules
|
||||
between 14:00 and 14:20 UTC, and the build agent's container on the control node started at 14:19 UTC.
|
||||
*I think* those young connections are restarts by deliveries; they are tens, not thousands.
|
||||
|
||||
|
||||
+2
-2
@@ -17,7 +17,7 @@ controller's, so each place that needs the bus opens one, uses it and closes it:
|
||||
command.
|
||||
- `sayingOnTheBus` (delivery.go): a delivery's `check`, `go` and `stop`, which mesh-delivery asks for.
|
||||
|
||||
The serving controller dials too. Its watchdogs run every 30 seconds, and while a walk waits for a
|
||||
The serving controller dials too. Its watchdogs run every 30 seconds, and while a rollout waits for a
|
||||
build, `buildSeatPause` (plan_retry.go) opens a fresh connection to read whether the build seat's
|
||||
holders paused, and closes it. `status` does the same.
|
||||
|
||||
@@ -41,7 +41,7 @@ its long-lived connections are named `mesh-controller` too and are told apart on
|
||||
minute. The research's guess came from their age alone.
|
||||
|
||||
**Located** in the controller: one connection per command process that touches the bus, and one per
|
||||
watchdog tick while a walk waits for a build.
|
||||
watchdog tick while a rollout waits for a build.
|
||||
|
||||
## 2026-10-09, resolved
|
||||
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ the fix is the attribution tool; with it, this diagnosis continues.
|
||||
- **A holder that never said it paused.** `PausedSaid` (controller, internal/link/queue.go) asks for the
|
||||
last message on each build holder's pause subject. A holder that never said one gets a not-found
|
||||
answer, which the server counts as an API error. The serving controller's watchdog reads it every 30
|
||||
seconds while a walk waits for a build (issue 327).
|
||||
seconds while a rollout waits for a build (issue 327).
|
||||
- **Short-lived consumers.** The controller heard 495 "consumer deleted" advisories in the first 8
|
||||
minutes of its current connection, about one a second. They are short-lived consumers: the controller's
|
||||
advisory reader (internal/link/advisories.go) ignores them and says why, "every watch of a bucket and
|
||||
|
||||
+7
-7
@@ -7,12 +7,12 @@ opened: 2026-10-08
|
||||
## What was observed
|
||||
|
||||
On 2026-10-08 a person released a delivery of the catalogue that the group check (issue 316) had held. Its
|
||||
walk had one tier with one module, `dunst`, which runs on the laptop and the workstation. The walk sent the
|
||||
rollout had one tier with one module, `dunst`, which runs on the laptop and the workstation. The rollout sent the
|
||||
new build to the laptop first, and the gate passed it there:
|
||||
|
||||
> dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent
|
||||
|
||||
Twenty-eight minutes later the same walk said the opposite, about the same gate:
|
||||
Twenty-eight minutes later the same rollout said the opposite, about the same gate:
|
||||
|
||||
> dunst failed its gate on the laptop in tier 0: healthy 3 times over 2m5s; put back dunst to ea462666 on
|
||||
> the laptop, in one send; the workstation left as it was
|
||||
@@ -31,22 +31,22 @@ build that worked, recorded it as failed, and so blocked its delivery until a ne
|
||||
|
||||
## Fix
|
||||
|
||||
The controller judged the first machine's report again on every step of the walk, also after the gate had
|
||||
The controller judged the first machine's report again on every step of the rollout, also after the gate had
|
||||
passed (see `01-diagnosis.md`). The fix merged in mesh-controller pull request 165 (8170fc5), with its
|
||||
replay in mesh-lab pull request 72.
|
||||
|
||||
- Once a module's gate has passed, the first machine's later reports no longer decide anything about it:
|
||||
the walk sends the rest.
|
||||
the rollout sends the rest.
|
||||
- A module carried by another module's send takes over that pass before its step is read.
|
||||
- A guard, not reached with the two changes above: should a later path still stop a walk at a module whose
|
||||
gate passed, that module is not put back and its build is not marked failed. The walk stops with the
|
||||
- A guard, not reached with the two changes above: should a later path still stop a rollout at a module whose
|
||||
gate passed, that module is not put back and its build is not marked failed. The rollout stops with the
|
||||
reason, and the module is left sent to its first node only, which `plans retry` takes up again.
|
||||
|
||||
## How it is checked
|
||||
|
||||
`TestReplay335` in the controller replays the live case with only what the controller had before the fix:
|
||||
a passed gate on the laptop, the laptop's later report not about this build (or a failure of another
|
||||
send), and the walk 30 minutes and 9 seconds after the first send. A second test drives the walk over a
|
||||
send), and the rollout 30 minutes and 9 seconds after the first send. A second test drives the rollout over a
|
||||
tier of two modules, the second carried by the first's send, and fails if the carried module's step is read
|
||||
before it takes over the pass. On the commit before the fix it fails
|
||||
with "laptop did not report it applied within 30m0s", and the passed build reaches the put-back. It is R335
|
||||
|
||||
+8
-8
@@ -6,15 +6,15 @@
|
||||
found ADR 0236 (what a rollback is), issue 318 (a wait for a login failing a gate) and issue 279, none of
|
||||
them about a gate that passed and was then failed.
|
||||
|
||||
**The walk's own lines.** The controller's journal on the control-node said, for this walk (times UTC):
|
||||
**The rollout's own lines.** The controller's journal on the control-node said, for this rollout (times UTC):
|
||||
|
||||
| time | what the controller said |
|
||||
|---|---|
|
||||
| 16:44:45 | tier 0 built; sent dunst to the laptop first, the rest once its gate passes |
|
||||
| 16:44:55 | the laptop applied what it was sent |
|
||||
| 16:46:52 | dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent |
|
||||
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another walk sends |
|
||||
| 16:49:12 | the other walk (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
|
||||
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another rollout sends |
|
||||
| 16:49:12 | the other rollout (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
|
||||
| 17:14:54 | dunst failed its gate on the laptop: healthy 3 times over 2m5s; put back on the laptop; the workstation left as it was |
|
||||
|
||||
The laptop's node-engine logged `applied` at 16:44:55 and nothing more until 18:01:58. It did not report
|
||||
@@ -25,14 +25,14 @@ on the 16:49:12 send for over an hour. Why it did not is not yet known; the lapt
|
||||
waits 30 minutes (`planWaitBound`) for a first machine to report what it was sent. The failure came at that
|
||||
bound and not at the gate's.
|
||||
|
||||
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a walk reads
|
||||
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a rollout reads
|
||||
`nextRollout(state, …)` before anything else, for every module in the tier not yet sent to the rest. For a
|
||||
module sent first, `nextRollout` looks at the first machine's last report. If that report is not about the
|
||||
declaration the machine was last sent (`Current` false), it waits; past the bound it answers
|
||||
`failed: <machine> did not report it applied within 30m0s`. Nothing in it looked at the gate's verdict.
|
||||
|
||||
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the walk tries again on
|
||||
the next step). At 16:49:12 the other walk's send made the laptop's last report no longer current. From
|
||||
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the rollout tries again on
|
||||
the next step). At 16:49:12 the other rollout's send made the laptop's last report no longer current. From
|
||||
then on `nextRollout` waited for a report it would never get about this build. At the bound it failed, and
|
||||
`advanceOnce` called `failFirstSend`.
|
||||
|
||||
@@ -47,7 +47,7 @@ laptop. That is the line seen at 17:14:54.
|
||||
- The person-wait of issue 318 (ADR 0254), `putBackBroken` and the carried-module paths: the plan had one
|
||||
module, nothing was carried, and nothing was found broken.
|
||||
- Sleep: the laptop did not sleep until 18:01, and the failure was at 17:14:54.
|
||||
- The release itself (`mesh-delivery.release`): it let the walk go at 16:44, and the walk ran as any other.
|
||||
- The release itself (`mesh-delivery.release`): it let the rollout go at 16:44, and the rollout ran as any other.
|
||||
|
||||
**Located.** mesh-controller, `cmd/mesh-controller/release_plan.go`: `nextRollout` judged a passed gate's
|
||||
first machine again, and `failFirstSend` put back a module whose gate had passed. The path has been there
|
||||
@@ -57,4 +57,4 @@ step as the pass and another send reaches the first machine before the 30-minute
|
||||
**What it needs to deliver dunst.** The build is recorded as failed at its gate, and the controller does
|
||||
not send a build marked that way again on its own. The registered build is back at the one before. Once the
|
||||
fix is delivered, a new build of dunst (`rebuild dunst`, or a newer merge of it) is judged at the gate
|
||||
again and walked to both machines.
|
||||
again and rolled out to both machines.
|
||||
|
||||
@@ -20,10 +20,10 @@ From the merge on, every send to the control-node was held. `mesh-controller.pla
|
||||
> planned step: sending control-node would replace the bus (nats 88135ad0 → 32307bd1), which is a planned
|
||||
> step ... — tried again for 27m45s
|
||||
|
||||
The same refusal held the controller's own walk and the release of builds kept back for a gate.
|
||||
The same refusal held the controller's own rollout and the release of builds kept back for a gate.
|
||||
|
||||
For those 28 minutes `mesh-controller.conditions` listed **no condition** naming the hold or saying it
|
||||
waited for a person. The operator was not told. The hold was found only because a walk looked stuck, and
|
||||
waited for a person. The operator was not told. The hold was found only because a rollout looked stuck, and
|
||||
the operator asked: "where would you be waiting then? Where is my approve button?"
|
||||
|
||||
The operator then gave the word, the bus upgrade started at 18:52 and the bus answered healthy at 18:54.
|
||||
@@ -33,7 +33,7 @@ The held sends then went through.
|
||||
|
||||
The design makes the bus wait for a person, and that part worked: nothing replaced the bus unasked. But a
|
||||
wait that only a person can end is said nowhere a person looks. The refusal is kept as the note on each
|
||||
held walk, which is read only by somebody already asking `plans`. Rule 4 of
|
||||
held rollout, which is read only by somebody already asking `plans`. Rule 4 of
|
||||
ADR 0227
|
||||
(no failure is silent) is met in the letter and missed in effect: the mesh held every send to its
|
||||
most important machine and said nothing until asked.
|
||||
@@ -47,10 +47,10 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
|
||||
|
||||
## Impact
|
||||
|
||||
- Every send to the control-node stops: the controller's own walk, the node-engine's, every module's walk
|
||||
- Every send to the control-node stops: the controller's own rollout, the node-engine's, every module's rollout
|
||||
that includes the control-node, and gated builds waiting for release. Other machines in the same tier
|
||||
wait with it.
|
||||
- The operator learns of it only by noticing a stuck walk, which can take hours when nobody is watching.
|
||||
- The operator learns of it only by noticing a stuck rollout, which can take hours when nobody is watching.
|
||||
- The only condition that could have fired, a tier past its bound (watchdog S3, "stalled"), fires no
|
||||
sooner than 30 minutes in, and its words ("stuck halfway ... longer than usual. Nothing is lost") name
|
||||
neither the bus nor the verb a person runs. It never fired in this case, because the step was taken
|
||||
@@ -62,9 +62,9 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
|
||||
|
||||
| Rule | Check |
|
||||
|---|---|
|
||||
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a walk wants to send raises a condition on the next watchdog tick, before any tier bound |
|
||||
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held walks or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
|
||||
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a walk whose only refusal is the bus's planned step |
|
||||
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a rollout wants to send raises a condition on the next watchdog tick, before any tier bound |
|
||||
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held rollouts or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
|
||||
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a rollout whose only refusal is the bus's planned step |
|
||||
| it clears when the step starts | the test ends with a bus step started: the condition clears, `bus-maintenance` opens |
|
||||
| the merge says it before it happens | the merge gate's test: a change that moves the bus's build reports that a bus upgrade by a person is needed and which machine waits until then |
|
||||
|
||||
@@ -81,8 +81,8 @@ why none is possible.
|
||||
with the operator as its resolver). Its headline says, in plain words, what waits ("sends to the
|
||||
control-node: the controller, the node-engine, …"), behind what (the bus build, from and to), since
|
||||
when, and the verb that ends it (`mesh-controller.bus upgrade --why … --reversible|--irreversible`). It
|
||||
is raised on the first refusal, not after a bound. A walk held only by it is excluded from S3, as a
|
||||
walk held by a paused build seat is.
|
||||
is raised on the first refusal, not after a bound. A rollout held only by it is excluded from S3, as a
|
||||
rollout held by a paused build seat is.
|
||||
2. **An ask, once asks exist.** When the decision flow of ADR 0259 (an ask is a message on the bus, any
|
||||
channel answers it; on an open pull request, not yet merged) is built, the condition becomes an ask
|
||||
to the operator: "start the bus upgrade now?", whose yes runs the step with the operator's word
|
||||
|
||||
@@ -13,13 +13,13 @@ The send is refused whole. `busHeld` (`bus_step.go`) gives the words "sending X
|
||||
(nats from → to), which is a planned step: `bus upgrade --why …` …". So the remedy is in the words, but
|
||||
only in the error.
|
||||
|
||||
**Where the error goes.** In `release_plan.go`, the walk's step loop catches the error, keeps it as the
|
||||
**Where the error goes.** In `release_plan.go`, the rollout's step loop catches the error, keeps it as the
|
||||
plan's note ("tier N: … — tried again"), saves the plan when the note changed, prints it to the
|
||||
controller's log once, and tries again on the next tick. Nothing raises a condition from it. That is why
|
||||
`plans` showed it and `conditions` did not.
|
||||
|
||||
**Which watchdog could have said it.** Only S3 (`watchPlans`, `signals.go`), "plan tier progress", raises
|
||||
`plan.<id>.stalled` when a walk has been in one tier past its bound. The bound is
|
||||
`plan.<id>.stalled` when a rollout has been in one tier past its bound. The bound is
|
||||
`max(30 min, 3 × the p90 of that repository's measured tiers)` (`tierAtLeast = planWaitBound = 30 min`,
|
||||
`watchdogs.go` `boundsOf`). So:
|
||||
|
||||
@@ -29,14 +29,14 @@ controller's log once, and tries again on the next tick. Nothing raises a condit
|
||||
mesh keeps it where it is."). Its summary carries the plan's note, so the bus's reason would have
|
||||
appeared there, but the headline a person reads names neither the bus nor the verb, and its resolver is
|
||||
not the operator;
|
||||
- a walk held by a paused build seat is excluded from S3 (`pausedWaiting`), and a walk waiting for its
|
||||
delivery's word has its own watchdog (S16). A walk held for the bus step has neither.
|
||||
- a rollout held by a paused build seat is excluded from S3 (`pausedWaiting`), and a rollout waiting for its
|
||||
delivery's word has its own watchdog (S16). A rollout held for the bus step has neither.
|
||||
|
||||
**What the bus step does raise.** Probe DB (`probeBusStep`, `bus_step.go`) raises `bus-maintenance` only
|
||||
while a step runs, and `bus-upgrade-failed` when one ended unhealthy. It reads only the latest bus step.
|
||||
Before a step is started there is nothing for it to read, so a pending bus build raises nothing.
|
||||
|
||||
**Before the merge.** `delivery.go` lists the bus among the modules whose walk starts at the merge
|
||||
**Before the merge.** `delivery.go` lists the bus among the modules whose rollout starts at the merge
|
||||
(`onTheControllersPath`), and the merge gate (`merge_gate.go`) composes the machines with the change. Neither
|
||||
reads `pendingBus` or says that the change will hold the bus's machine until a person runs the step. So a
|
||||
merge that moves the bus's build passes green and blocks every later send to the control-node.
|
||||
@@ -64,12 +64,12 @@ person, and the person must be told.
|
||||
ADR 0271
|
||||
(proposed) takes suggested fixes 1 and 3. mesh-controller #173 builds them:
|
||||
|
||||
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a walk's send was refused
|
||||
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a rollout's send was refused
|
||||
for the bus's planned step, while the new bus build would still replace the bus on its machine. It is a
|
||||
warning, for the operator. It names the machines, each waiting walk and the modules of its tier, the bus build
|
||||
warning, for the operator. It names the machines, each waiting rollout and the modules of its tier, the bus build
|
||||
from and to, since when, and the `mesh-controller.bus` call with `upgrade`. It clears once the bus's machine has
|
||||
been sent the build the mesh holds.
|
||||
- **S3 leaves out a walk held only by it.**
|
||||
- **S3 leaves out a rollout held only by it.**
|
||||
- **The delivery plan says it before the merge**: "merging this needs a person's bus upgrade: … nothing else is
|
||||
sent to <machine> either", and its summary, which `mesh/merge-gate` carries, says the same.
|
||||
|
||||
|
||||
+3
-3
@@ -20,7 +20,7 @@ The plan rebuilt and moved them whatever the merge changed. Two examples:
|
||||
|
||||
- Mesh-controller PR #159, merged as `ca09a07f`, changed files under `cmd/mesh-controller`,
|
||||
`internal/broker`, `internal/link`, `internal/catalogue/verbs.go` and the root `module.json`. Nothing
|
||||
under `examples/route-proxy` changed. Its walk said, on the home server, "recreates 1 of route-proxy's
|
||||
under `examples/route-proxy` changed. Its rollout said, on the home server, "recreates 1 of route-proxy's
|
||||
3 container(s) (with a new image: server)" and "recreates 1 of build-agent's 1 container(s)".
|
||||
- Mesh-controller PR #163, merged as `9b028b42`, did the same. The route proxy's program at that commit
|
||||
is byte for byte the program at the commit before it (measured in the diagnosis), and it was still
|
||||
@@ -43,8 +43,8 @@ The operator stated the rule the delivery plan should follow:
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
- **Deliveries are slow.** A controller merge walked for about fifteen minutes instead of about five,
|
||||
and the walks of one repository must not overlap (issue 335), so every merge after it waits too. The
|
||||
- **Deliveries are slow.** A controller merge rolled out for about fifteen minutes instead of about five,
|
||||
and the rollouts of one repository must not overlap (issue 335), so every merge after it waits too. The
|
||||
operator considers this a major bug and a main reason deliveries are slow.
|
||||
- **A public service restarts for nothing.** The route proxy on the home server was recreated at
|
||||
merges that did not change its program.
|
||||
|
||||
+11
-11
@@ -31,7 +31,7 @@ Read in mesh-controller at `8170fc58`.
|
||||
4. `mesh-controller` itself is built from the repository's root (its manifest's bundle compiles
|
||||
`cmd/mesh-controller`; its source path is empty), so every changed file in the repository touches
|
||||
it as well.
|
||||
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the walk serial here:
|
||||
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the rollout serial here:
|
||||
- a `packages` edge, from a module whose build read a repository to the module whose source that
|
||||
repository is. So `route-proxy` and `build-agent` both depend on `mesh-controller`, because they
|
||||
share its repository, not because either is built from the controller's artifacts;
|
||||
@@ -108,9 +108,9 @@ change. The build source of the route proxy is `examples/route-proxy`, `internal
|
||||
## Measured: what the extra tiers cost
|
||||
|
||||
From `mesh-controller.plans` (read only), the first-node gate of each tier of tonight's controller
|
||||
walks:
|
||||
rollouts:
|
||||
|
||||
| walk | mesh-controller | build-agent | route-proxy | three gates |
|
||||
| rollout | mesh-controller | build-agent | route-proxy | three gates |
|
||||
|---|---|---|---|---|
|
||||
| `3f68a495` | 3m40s | 3m43s | 3m46s | 11m09s |
|
||||
| `0cf5a3a5` | 3m39s | 3m26s | 3m34s | 10m39s |
|
||||
@@ -120,12 +120,12 @@ walks:
|
||||
| `ca09a07f` | 3m43s | 3m46s | 3m40s | 11m09s |
|
||||
| `efcdd5dd` | 3m38s | 3m40s | 3m37s | 10m55s |
|
||||
|
||||
Two more walks (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still walking at
|
||||
21:50. The gates alone are 10 to 11 minutes per walk, of which 7 minutes are the two packaging tiers.
|
||||
Two more rollouts (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still rolling out at
|
||||
21:50. The gates alone are 10 to 11 minutes per rollout, of which 7 minutes are the two packaging tiers.
|
||||
Each tier also builds before it sends: the build seat's image took 53 seconds at 21:36. Estimated from
|
||||
the plan's creation time and the build records, `ca09a07f` walked from 21:07 to about 21:23 and
|
||||
the plan's creation time and the build records, `ca09a07f` rolled out from 21:07 to about 21:23 and
|
||||
`efcdd5dd` from 21:32 to about 21:47, about 15 minutes each. The controller's own tier, alone, is about
|
||||
4 to 5 minutes. Of the seven completed walks, five recreated `route-proxy` on the home server
|
||||
4 to 5 minutes. Of the seven completed rollouts, five recreated `route-proxy` on the home server
|
||||
(`3f68a495`, `e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`), and its program had changed in one of
|
||||
them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`).
|
||||
|
||||
@@ -133,7 +133,7 @@ them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07
|
||||
|
||||
The deploy plan names a build of a packaging module by the module's own source commit, which is the
|
||||
catalogue's. `moveFacts.identical` (cmd/mesh-controller/release.go) calls two builds identical when
|
||||
their commits are equal. In `efcdd5dd`'s walk both modules went `7b307a06 → 7b307a06`, nothing was
|
||||
their commits are equal. In `efcdd5dd`'s rollout both modules went `7b307a06 → 7b307a06`, nothing was
|
||||
said to be recreated, and the gate still took about 3.5 minutes for each. Yet the route proxy's
|
||||
program did change at `efcdd5dd` (table above). **I do not know whether that program reached the home
|
||||
server.** Either it was sent and the plan said nothing, or it was not sent and the route proxy runs the
|
||||
@@ -146,7 +146,7 @@ commit would also make issue 280's "repeat" rule register a different program as
|
||||
- **The changed paths not reaching the planner.** PR #159's delivery record carries its paths and
|
||||
`module_dirs_said: true`. They are read, and `touchedBy` uses them for the controller's own module.
|
||||
They are never consulted for a packaging module.
|
||||
- **The catalogue moving at the same time.** In some walks the catalogue's commit also changed
|
||||
- **The catalogue moving at the same time.** In some rollouts the catalogue's commit also changed
|
||||
(`32307bd1 → 7b307a06`), but the README-only what-if shows the controller's repository alone is enough
|
||||
to plan both modules.
|
||||
|
||||
@@ -181,7 +181,7 @@ the commit into the program.
|
||||
with the build; that is exact for Go and needs a reader per language.
|
||||
2. **Reproducible builds**: `-buildvcs=false` (or a `.dockerignore` that leaves `.git` out of the
|
||||
context) in both recipes, so an unchanged source gives an identical program. This alone does not
|
||||
stop the walk: the image digest still differs (issue 280), and the planner would still plan and gate
|
||||
stop the rollout: the image digest still differs (issue 280), and the planner would still plan and gate
|
||||
three tiers. It is cheap and worth doing beside 1, so the program's identity no longer depends on the
|
||||
commit.
|
||||
3. **Split the route proxy and the build seat into repositories of their own.** Then the existing
|
||||
@@ -191,5 +191,5 @@ the commit into the program.
|
||||
and a change to that library then moves all three again, through a real build edge.
|
||||
|
||||
Recommendation: 1 with its check, and 2 beside it. A further question for the fix: whether `built-by`
|
||||
should order a walk at all. It is a running-order edge for the build seat's worker (issue 206); the
|
||||
should order a rollout at all. It is a running-order edge for the build seat's worker (issue 206); the
|
||||
operator's rule 2 counts only modules built *from* a changed module.
|
||||
|
||||
@@ -23,7 +23,7 @@ What happened, in order (UTC):
|
||||
**under the same JSON key `declared`**. The two edits touched different lines, so git merged them
|
||||
without a conflict. The result does not compile:
|
||||
`internal/store/store.go:145:2: Declared redeclared`.
|
||||
4. **00:55.** The walk the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
|
||||
4. **00:55.** The rollout the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
|
||||
in tier 0". `mesh/delivery` on the head reads `failure`. Nothing was sent to a node, and every node kept
|
||||
`d28ca95`.
|
||||
5. **01:09.** Node-engine #67 repaired the trunk. It renamed #66's field to `DeclaredUnit`, key
|
||||
@@ -33,7 +33,7 @@ The facts behind the "different lines" claim, read from the node-engine's reposi
|
||||
contain `d28ca95`. Their common ancestor is `a2c2c35` (#63's merge). `dfb1415` holds one `Declared`
|
||||
(store.go line 107), `d28ca95` holds the other (line 139), and `690966d` holds both (lines 107 and 145).
|
||||
|
||||
No node was harmed, because a walk builds before it sends. But the merge check said "this may merge" of
|
||||
No node was harmed, because a rollout builds before it sends. But the merge check said "this may merge" of
|
||||
a merge that did not build, and both required statuses were green when it merged.
|
||||
|
||||
## The cost before this: checks that went stale
|
||||
|
||||
+1
-1
@@ -33,7 +33,7 @@ merging made, with two, was never built before the merge.
|
||||
or ready delivery back to proposed. Its comment says "a person asked for it again, or its group
|
||||
changed".
|
||||
- `holder.go`, `PullMerged` (line 347): a merge records the merge commit on that pull request's own
|
||||
delivery and links its walk. It does not touch the repository's other open deliveries into the same
|
||||
delivery and links its rollout. It does not touch the repository's other open deliveries into the same
|
||||
trunk.
|
||||
- The forge marks a head pending only when it announces a new head (issue 305). A merge into the trunk is
|
||||
not a new head of any other pull request.
|
||||
|
||||
@@ -60,14 +60,14 @@ note. Each gap says what is open, why it matters, and what would close it.
|
||||
|
||||
9. **On a large data pool the root search may never finish.** The node-engine's search for programs that run as
|
||||
root now has one bound of fifteen minutes (`mesh-host` #61; the inner two-minute limit is gone). On a machine with
|
||||
a very large data pool the walk can still run past it, and the verdict then stays "not judged". That is loud,
|
||||
a very large data pool the traversal can still run past it, and the verdict then stays "not judged". That is loud,
|
||||
not silent.
|
||||
*Closes with:* setuid turned off on the data datasets, so the walk skips them.
|
||||
*Closes with:* setuid turned off on the data datasets, so the traversal skips them.
|
||||
|
||||
10. **An agent can merge its own pull request.** No mesh repository's main requires an approval on the forge
|
||||
(`required_approvals: 0`), and the forge tool merges as the forge's administrator once the required checks are
|
||||
green. An agent can therefore change a module that runs on every node, open the pull request, wait for the
|
||||
checks and merge it, and the controller walks the change out. The checks judge whether every machine still
|
||||
checks and merge it, and the controller rolls the change out. The checks judge whether every machine still
|
||||
composes, not what the change is for. The operator's yes in a session is a convention, and nothing checks it.
|
||||
*Closes with:* a merge that needs the operator's own approval: a required approval on the forge from the
|
||||
operator's personal account only, or an approval through ADR 0259's asks. That holds only when the forge's
|
||||
@@ -98,7 +98,7 @@ note. Each gap says what is open, why it matters, and what would close it.
|
||||
|
||||
14. **Ways to root the node-engine's judge does not see.** ADR 0266 §3 lists them under *not judged, so not seen*:
|
||||
among them a root-run unit or script the agent account can write, a directory on root's `PATH` it can write,
|
||||
root's ssh keys readable by it, and a setuid program on a filesystem the search does not walk. None is known
|
||||
root's ssh keys readable by it, and a setuid program on a filesystem the search does not traverse. None is known
|
||||
on the control-node, but a healthy verdict says nothing about them. The record left them "for a later record"
|
||||
with nothing tracking it; added here on 2026-10-10.
|
||||
*Closes with:* each way judged by the node-engine beside the others, or a record that decides against judging
|
||||
|
||||
+6
-6
@@ -16,14 +16,14 @@ stops answering while it runs.
|
||||
| 10:57:57 | the route proxy's first "lookup … on <private address>:53: … connection refused" | the route proxy's log |
|
||||
| 10:58:36 | `machine.<home server>.network` raised: one of its two resolvers does not answer | conditions |
|
||||
| 10:58:45 | `machine.<control node>.network` raised; the home server's cleared, held under it | conditions |
|
||||
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the walks |
|
||||
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the rollouts |
|
||||
| 10:59:37 | `module.step-ca.<control node>.unhealthy` raised | conditions |
|
||||
| 10:59:42 | `machine.<control node>.network` **cleared**, its newest evidence (10:59:37) still "connection refused" | conditions |
|
||||
| 11:00:23 | `machine.<control node>.network` reopened, raised time 11:00:23 | conditions |
|
||||
| 11:06:00 | `seat.mesh-dns-resolver.<control node>.wrong` raised by D2 | conditions |
|
||||
| 11:10:29 | both builds failed their gate: "<control node> as a whole: raised since it was sent: machine.<control node>.network" — put back | conditions |
|
||||
| 11:18:12 | `dnsmasq.service` restarted by hand; every question answered again | the operator |
|
||||
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release walked the node-engine | conditions |
|
||||
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release rolled out the node-engine | conditions |
|
||||
|
||||
### What is measured, and what is guessed
|
||||
|
||||
@@ -54,12 +54,12 @@ answered none of its questions for the next run (`Confirm`, issue 277), because
|
||||
question can be lost. Its first run after 10:57:57 held it; the next, at 11:06:00, raised it. Every one
|
||||
of those questions had been refused, which no load explains.
|
||||
|
||||
**Measured: why the second walk of the node-engine waited on its own first machine.** A release keeps no
|
||||
**Measured: why the second rollout of the node-engine waited on its own first machine.** A release keeps no
|
||||
record per module — its modules are in its tier only — and D10 read only the records to know what a plan
|
||||
is rolling out. So while a release walked the node-engine at 11:21, D10 raised `core-behind` on every
|
||||
is rolling out. So while a release rolled out the node-engine at 11:21, D10 raised `core-behind` on every
|
||||
machine, "and no plan is rolling them out", and the gate on the home server, its first machine, waited
|
||||
on it (issue 281's settle window) until D10's next run five minutes later found the new engine there.
|
||||
The walk then passed. It did not fail, but it waited five minutes for a condition its own send caused.
|
||||
The rollout then passed. It did not fail, but it waited five minutes for a condition its own send caused.
|
||||
|
||||
**Not explained: why the second resolver did not save the lookups.** Every machine's resolver file lists
|
||||
both resolvers (`options timeout:1 attempts:2`), and the control node's own file lists its private address
|
||||
@@ -153,7 +153,7 @@ registry, is the operator's decision.
|
||||
- An undecided part of a network statement — `unknown` or `starting`, as a node-engine just started says
|
||||
— holds the open network conditions whose newest evidence names that part, and only those. A statement
|
||||
undecided as a whole holds every part. The next statement decides.
|
||||
- D10 counts what a release's tier names as rolling out, so a walked node-engine is not `core-behind` on
|
||||
- D10 counts what a release's tier names as rolling out, so a rolled out node-engine is not `core-behind` on
|
||||
its own first machine.
|
||||
- D2 is unchanged: a resolver that answered nothing, refused or silent, is held for the next run, because
|
||||
a few hundred milliseconds of refusals is a resolver restarting as well as one that stopped. D2 says it
|
||||
|
||||
+3
-3
@@ -61,8 +61,8 @@ A replay in mesh-lab's register is still owed before this core issue resolves (A
|
||||
|
||||
## A sequel (2026-10-10)
|
||||
|
||||
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a walk:
|
||||
mesh-delivery matches a walk to the delivery merged as its commit, so two catalogue merges 18 seconds apart
|
||||
left the earlier one held for a person and the walk carrying it waiting for a word nothing gave, for 58
|
||||
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a rollout:
|
||||
mesh-delivery matches a rollout to the delivery merged as its commit, so two catalogue merges 18 seconds apart
|
||||
left the earlier one held for a person and the rollout carrying it waiting for a word nothing gave, for 58
|
||||
minutes, with no condition raised. See
|
||||
[issue 362](../362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md).
|
||||
|
||||
+14
-14
@@ -2,7 +2,7 @@
|
||||
opened: 2026-10-09
|
||||
---
|
||||
|
||||
# 351. A held delivery whose module a backlog walk carried to every machine is never covered
|
||||
# 351. A held delivery whose module a backlog rollout carried to every machine is never covered
|
||||
|
||||
## Symptom
|
||||
|
||||
@@ -19,12 +19,12 @@ What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `ha
|
||||
|
||||
| time | what happened |
|
||||
|---|---|
|
||||
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word |
|
||||
| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
|
||||
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded |
|
||||
| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
|
||||
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
|
||||
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours |
|
||||
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its rollout waited for its word |
|
||||
| 10-08 11:06 | three later catalogue merges took its rollout over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
|
||||
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the rollout ended `failed`, then superseded |
|
||||
| 10-08 14:27 | #127 merged as bf542128; its rollout built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
|
||||
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog rollout sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
|
||||
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog rollout, as it had for 24 hours |
|
||||
|
||||
No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever
|
||||
move it again (ADR 0267):
|
||||
@@ -32,22 +32,22 @@ left as it is, the delivery stays held for ever.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running
|
||||
ADR 0270 covers a held merge when a later delivered merge's rollout "sent the module to every machine running
|
||||
it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge
|
||||
commit, which the forge says contains a6385479. The decision's question — does this change go on? — was
|
||||
answered by the operator's own release of the build. Yet the record cannot end:
|
||||
|
||||
- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what
|
||||
- **A backlog rollout's end is read by nothing.** While it judges, the controller's record of it names what
|
||||
its gate carried, and mesh-delivery credits the delivery that published each carried build one step per
|
||||
machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing
|
||||
from that: the steps stay `judging` for ever.
|
||||
- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running
|
||||
it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the
|
||||
- **`covered` reads only a rollout's own aggregate steps.** A rollout's own record says "every machine running
|
||||
it" or "the rest running it"; a step per machine, which is all a backlog rollout gives, never counts as the
|
||||
module sent everywhere.
|
||||
|
||||
ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module
|
||||
whose build reached its machines on the operator's word, after the mesh held it back, rather than by the
|
||||
walk of the merge that built it. A held delivery in that case has no end but a person's wrong act
|
||||
rollout of the merge that built it. A held delivery in that case has no end but a person's wrong act
|
||||
([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and
|
||||
the condition it raises every hour is one nobody can clear correctly.
|
||||
|
||||
@@ -56,7 +56,7 @@ the condition it raises every hour is one nobody can clear correctly.
|
||||
- A forge that could not answer: the holder's journal has no "not yet covered" line for it, and
|
||||
`gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128.
|
||||
- A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps.
|
||||
- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
|
||||
- The rollout that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
|
||||
rightly no coverer.
|
||||
- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module
|
||||
- A person's `release` of #125 (the verb): it would roll out a6385479, an older commit than the one the module
|
||||
was built from, and is what ADR 0270 ruled out.
|
||||
|
||||
+21
-21
@@ -2,56 +2,56 @@
|
||||
|
||||
**2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`,
|
||||
at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127,
|
||||
`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`,
|
||||
`mesh-controller.delivery-walks` for #127's rollout and for the backlog rollout, `plans`, `hand-acts`, `module list`,
|
||||
the forge's `gitea_contains`, and the holder's journal on the control node.
|
||||
|
||||
**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a
|
||||
**Where a backlog rollout's steps come from.** `walkMoved` (holder.go) hands a rollout whose record carries a
|
||||
`release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives
|
||||
the delivery whose merge commit each carried build was made from one step per machine: `judging` while the
|
||||
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk
|
||||
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog rollout
|
||||
holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`.
|
||||
`creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43
|
||||
and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the
|
||||
module was not in that walk's tiers, and the backlog walk is a walk of its own.)
|
||||
and 14:46 UTC were never moved on. (The controller's own record of #127's rollout never named the module: the
|
||||
module was not in that rollout's tiers, and the backlog rollout is a rollout of its own.)
|
||||
|
||||
**Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a
|
||||
later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only
|
||||
on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running
|
||||
on a `sent` step whose machine is the rollout's own aggregate, "every machine running it" or "the rest running
|
||||
it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is
|
||||
found, and the forge is never asked — which is why the journal has no line.
|
||||
|
||||
**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a
|
||||
backlog walk that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
|
||||
**What "done on a machine" means for a backlog rollout.** Read from the controller's records, not its code: a
|
||||
backlog rollout that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
|
||||
(seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done,
|
||||
with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of
|
||||
the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis
|
||||
is wrong there.
|
||||
|
||||
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb —
|
||||
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed rollout, and the `release` verb —
|
||||
see the report.
|
||||
|
||||
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for
|
||||
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog rollout's `done` machines for
|
||||
nothing, and `carries`, which counts no per-machine step.
|
||||
|
||||
## The fix
|
||||
|
||||
In mesh-catalog #163, one pull request, in `modules/mesh-delivery`:
|
||||
|
||||
- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that
|
||||
walk (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
|
||||
step it failed on, and a step another walk carried, are left as they are.
|
||||
- **A finished backlog rollout passes the steps it carried.** `creditReleased`: every step credited to that
|
||||
rollout (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
|
||||
step it failed on, and a step another rollout carried, are left as they are.
|
||||
- **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when
|
||||
the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
|
||||
the rollout's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
|
||||
delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still
|
||||
judging, one failed or one never sent leave the delivery held.
|
||||
- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its
|
||||
open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was
|
||||
- **A rollout that carried a step still judging is asked for by its id.** The controller lists back only its
|
||||
open rollouts and the last fifty ended at a start; the backlog rollout of this case, ended the day before, was
|
||||
no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for
|
||||
such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
|
||||
such a rollout by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
|
||||
its record as said.
|
||||
|
||||
Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own
|
||||
walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
|
||||
rollout of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
|
||||
The record's consequence that named the one-machine gap gains a progressive insight naming this one.
|
||||
|
||||
**How it is checked**, in `covered_test.go`:
|
||||
@@ -59,15 +59,15 @@ The record's consequence that named the one-machine gap gains a progressive insi
|
||||
- `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered;
|
||||
still judging on one, failed on one, never sent to one, or a plan naming no machines stays held.
|
||||
- `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one
|
||||
and leaves the other judging; a failed step stays failed; a step another walk carried is not touched.
|
||||
and leaves the other judging; a failed step stays failed; a step another rollout carried is not touched.
|
||||
- `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held
|
||||
while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the
|
||||
while the backlog rollout judges, covered once it is done and read by its id, naming the later merge and the
|
||||
module.
|
||||
|
||||
All three fail on the catalogue's main before the change and pass after it.
|
||||
|
||||
**After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are
|
||||
asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once
|
||||
asked for by the backlog rollout's id and pass; #127 then carries the module for #125; the forge is asked once
|
||||
whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads
|
||||
delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the
|
||||
acceptance of this issue, read with `mesh-delivery.show` of #125.
|
||||
|
||||
+3
-3
@@ -21,7 +21,7 @@ On 2026-10-09, local times on the control node (its controller's and node-engine
|
||||
migrations than the store had applied, and served what it could.
|
||||
- 16:35:59 that older controller judged the newer plan's gate **passed** on the control node ("healthy 3
|
||||
times over 4m17s") — from the health of the build it was itself, not of the build the plan had sent,
|
||||
which the put-back had removed from the machine. The plan walked on, believing the controller delivered.
|
||||
which the put-back had removed from the machine. The plan rolled out on, believing the controller delivered.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
@@ -35,8 +35,8 @@ Four rules did not hold.
|
||||
gate of the build it had replaced.
|
||||
- **A put-back puts back something that can run the mesh** (ADR 0236: "put back by something other than
|
||||
itself"). A controller older than the store's schema starts behind its own records. Nothing refused it.
|
||||
- **One walk of a module on a machine at a time** (issue 281, ADR 0236). A release's walk was invisible to
|
||||
the merge plan's send: a release keeps no module records, and the check for a walk in progress read
|
||||
- **One rollout of a module on a machine at a time** (issue 281, ADR 0236). A release's rollout was invisible to
|
||||
the merge plan's send: a release keeps no module records, and the check for a rollout in progress read
|
||||
only those.
|
||||
|
||||
The diagnosis and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
|
||||
|
||||
+7
-7
@@ -25,10 +25,10 @@ read as a module's) are the nearest; neither is this.
|
||||
to the one the first machine ran before (`PreviousBuild`) and sends it. Nothing asked whether that
|
||||
build can read the store as the failed build left it. The controller says so itself when it starts
|
||||
("this control plane is behind the mesh-controller row"), and serves anyway (ADR 0185).
|
||||
- **The merge plan's send crossed the release's gate.** `walkedBy` finds a walk in progress by a plan's
|
||||
module records (`p.Modules[module].First`). A release (the walk of the builds waiting for a gate) keeps its walk in its gate's carried moves,
|
||||
- **The merge plan's send crossed the release's gate.** `walkedBy` finds a rollout in progress by a plan's
|
||||
module records (`p.Modules[module].First`). A release (the rollout of the builds waiting for a gate) keeps its rollout in its gate's carried moves,
|
||||
not in module records, so the release's open gate on the control node held nothing back. And a plan's
|
||||
own modules were never checked against another walk at all: the check ran over the moves a machine
|
||||
own modules were never checked against another rollout at all: the check ran over the moves a machine
|
||||
had waiting, and skipped the plan's own.
|
||||
|
||||
### What is guessed
|
||||
@@ -62,13 +62,13 @@ In mesh-controller, pull request 185.
|
||||
never recorded what it reaches, is not put back; the current build is kept, and the condition
|
||||
(`rollback-failed`, urgent) says why and that a person decides. The lease holder's `Build` is that
|
||||
version too.
|
||||
- **A walk of a module on a machine holds every other send of another build of it there.** `walkedBy`
|
||||
reads a release's open gate as its walk, and a plan's own modules are held by it as any other move is:
|
||||
- **A rollout of a module on a machine holds every other send of another build of it there.** `walkedBy`
|
||||
reads a release's open gate as its rollout, and a plan's own modules are held by it as any other move is:
|
||||
a merge plan whose first machine is under a release's gate for the same module waits, said as
|
||||
"waiting for … which release-… is walking". The same build to the same machine is not held: it is
|
||||
already there.
|
||||
|
||||
Not changed: whether a release's walk and a merge plan's walk of the core should be one walk (the walk
|
||||
Not changed: whether a release's rollout and a merge plan's rollout of the core should be one rollout (the rollout
|
||||
carrying over) is left to a design; serialising them is what this fixes.
|
||||
|
||||
## How it is checked
|
||||
@@ -79,7 +79,7 @@ carrying over) is left to a design; serialising them is what this fixes.
|
||||
| A send that moves the module supersedes the judging, puts nothing back | `TestReplay352`, `TestAPlansJudgingIsSupersededByASendThatMovesItsModule` |
|
||||
| A report is held against the gate's own send, by digest or sequence | `TestAReportIsHeldAgainstTheGatesOwnSend`, inventory `TestASendAndAReportAreKnownByTheirDeclaration` |
|
||||
| Only the build last sent is judged | `TestAGateJudgesOnlyTheBuildTheMachineWasLastSent` |
|
||||
| A walk holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` |
|
||||
| A rollout holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` |
|
||||
| No put-back to a controller behind the store, or one unproved | `TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema`, inventory `TestASchemaReachIsKeptByBuild`, catalogue `TestAProcessEnvCanNameTheBuildsOwnVersion`, `TestTheHolderNamesTheBuildTheDeclarationToldIt` |
|
||||
| Every condition of the change is tested | 31 mutations, each failing a test |
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ opened: 2026-10-09
|
||||
On 2026-10-09, local times on the control node, from its controller's journal (one process throughout):
|
||||
|
||||
- 17:32 the router (the `operator-channel` holder) was unassigned, so that a merge gate of the catalogue had
|
||||
nothing to send there; 17:52:54 a walk of another module pushed the control node and composed the bus's user
|
||||
nothing to send there; 17:52:54 a rollout of another module pushed the control node and composed the bus's user
|
||||
list **without** the controller's grant to ask: the grant is composed from a holder of the seat that is
|
||||
*assigned*, and none was.
|
||||
- 17:54:05 the controller started asking (the router was assigned again by then, so its check of the record
|
||||
|
||||
+3
-3
@@ -73,9 +73,9 @@ one. The tool runner has the same shape at a smaller scale: the bus account and
|
||||
controller's user list and in every node's assignment.
|
||||
|
||||
Hence: **one release in which both names are accepted, on every side, before any release in which the old
|
||||
one is dropped**, each release walked one node first — the laptop, where a halted node is the operator at a
|
||||
one is dropped**, each release rolled out one node first — the laptop, where a halted node is the operator at a
|
||||
keyboard — and then the rest. I think no stage should move before the delivery fixes of issues 351 and 352
|
||||
are live, because a core walk is exactly what those broke; that is a guess from their symptoms, not a
|
||||
are live, because a core rollout is exactly what those broke; that is a guess from their symptoms, not a
|
||||
measurement.
|
||||
|
||||
### The staged plan
|
||||
@@ -90,7 +90,7 @@ measurement.
|
||||
tool runner's as another (runtime module, bus account, kind). The node-engine's launcher and upgrade
|
||||
package read the old and the new library and state directories, and the node-engine moves its state
|
||||
once, the first time it runs under the new name. Both unit names run the same launcher. The conditions
|
||||
and the `node` verb say node-engine. The merge gate composes every node green; walked one node first.
|
||||
and the `node` verb say node-engine. The merge gate composes every node green; rolled out one node first.
|
||||
3. **The rename proper.** The new module names registered, assigned beside the old on the first node, the
|
||||
old unassigned there, the acceptance sentence checked on that node, then the rest. The repository
|
||||
renamed on the forge, with its build source and its Go module path; the packaging renamed. The tool
|
||||
|
||||
+4
-4
@@ -9,7 +9,7 @@ opened: 2026-10-09
|
||||
ADR 0270
|
||||
(2026-10-09) ends a held delivery by itself when a later merge that contains it was delivered to every
|
||||
machine running its modules — the delivery is *covered*. Its consequences say: "A module that runs on one
|
||||
machine only is never covered. A walk records 'the rest running it' only after its first machines passed
|
||||
machine only is never covered. A rollout records 'the rest running it' only after its first machines passed
|
||||
and the rest were sent. A module on one machine has no rest, so its steps never say it reached every
|
||||
machine. mesh-delivery itself, on the control node alone, is such a module: a held merge of it stays held
|
||||
for a person, as before this record. Reading 'every machine running it' from the first machines when they
|
||||
@@ -29,8 +29,8 @@ done (issue 354).
|
||||
|
||||
## Open
|
||||
|
||||
- Whether a walk whose first machines are all the machines should record that it reached every machine
|
||||
- Whether a rollout whose first machines are all the machines should record that it reached every machine
|
||||
(the record's own suggestion), or whether *covered* should read the machines running a module from the
|
||||
placement rather than from the walk's steps.
|
||||
placement rather than from the rollout's steps.
|
||||
- The owner is mesh-delivery, in the controller's repository, as the record says; `located-in:` is filled
|
||||
when the diagnosis confirms where the walk's record is written.
|
||||
when the diagnosis confirms where the rollout's record is written.
|
||||
|
||||
+1
-1
@@ -13,7 +13,7 @@ The node-engine sets a directory's mode and owner by path when the directory is
|
||||
an account's home: `applyDirectory` calls `chmodDirPath` and `own`, which call `os.Chmod` and `os.Chown` on the
|
||||
path. Both follow a link anywhere in the path. The directory applier refuses a link at the directory's own name,
|
||||
and issue 339's placement guard refuses the machine's own trees, but nothing refuses a link in a parent at the
|
||||
moment of use. Below a home the engine already walks every component without following a link (ADR 0266).
|
||||
moment of use. Below a home the engine already traverses every component without following a link (ADR 0266).
|
||||
|
||||
Issue 356 closed this for a directory handed to the mesh by the operator: it is changed only through one
|
||||
descriptor opened from `/` with no link followed, on every apply. The same treatment for every other mesh
|
||||
|
||||
@@ -19,8 +19,8 @@ the controller's `root-free` verb for that machine answered:
|
||||
> again later
|
||||
|
||||
The control-node holds its root and home filesystems on a software RAID of spinning disks, with millions of
|
||||
files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) walks every local
|
||||
filesystem mounted without `nosuid`; that walk, cold, takes far longer than fifteen minutes there. Each attempt
|
||||
files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) traverses every local
|
||||
filesystem mounted without `nosuid`; that traversal, cold, takes far longer than fifteen minutes there. Each attempt
|
||||
was killed at its bound, judged nothing, and was tried again after a back-off, which ends the same way.
|
||||
|
||||
## Why it is a design fault, not one slow disk
|
||||
@@ -29,7 +29,7 @@ was killed at its bound, judged nothing, and was tried again after a back-off, w
|
||||
controller could tell "still running" from "stuck" (`rootsearch.Bound`, read by both). It turned a slow
|
||||
machine into one that is never judged, failing closed for good rather than for a while.
|
||||
- **Nothing was kept across a restart.** The search's result lived in the engine's memory only, so every
|
||||
engine restart (every update of the engine) started from nothing and judged nothing until a whole walk
|
||||
engine restart (every update of the engine) started from nothing and judged nothing until a whole traversal
|
||||
finished again.
|
||||
- **The controller's quiet was the same value.** The self-check stayed quiet while a search was "within its
|
||||
bound" — fifteen minutes — so a search that runs for hours would raise the urgent condition on every
|
||||
@@ -51,7 +51,7 @@ was killed at its bound, judged nothing, and was tried again after a back-off, w
|
||||
**Why a day-old search is sound.** Only root can make a setuid- or setgid-root file: the agent account can
|
||||
neither set the bit on a file root owns nor give a file of its own to root. So a complete search stays true until
|
||||
root acts, and root's acts through the mesh (applies) start a new search. A file that appears behind a running
|
||||
walk's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age
|
||||
traversal's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age
|
||||
bounds.
|
||||
|
||||
The trail and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
|
||||
|
||||
@@ -15,25 +15,25 @@
|
||||
health — which this issue does not change).
|
||||
- **Nothing else** used the bound: `SetuidCache` had one caller (`moreWays`), and the engine made one cache.
|
||||
- **Ruled out:** a stuck `find` (each attempt ended at exactly its bound) and a filesystem that should not be
|
||||
searched (the walk covered only the local root, home and boot filesystems, as ADR 0266 asks).
|
||||
searched (the traversal covered only the local root, home and boot filesystems, as ADR 0266 asks).
|
||||
|
||||
## The fix (`mesh-host` pull request 75, `mesh-controller` pull request 190)
|
||||
|
||||
**The walk is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as
|
||||
**The traversal is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as
|
||||
`find -xdev` did, with the same prunes and the same match (a regular file, setuid and owned by root, or setgid
|
||||
and of root's group), never following a link. It visits each directory's entries in lexical order, so its
|
||||
progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is walked before
|
||||
`/a-c` although `-` sorts before `/`). Every minute the progress (when the walk started, its filesystems, those
|
||||
progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is traversed before
|
||||
`/a-c` although `-` sorts before `/`). Every minute the progress (when the traversal started, its filesystems, those
|
||||
done, the cursor, the matches so far) is written beside the engine's state as a new file renamed into place,
|
||||
readable by root alone. After a restart the walk carries on from the cursor: what is before it is skipped
|
||||
without being read, the directories leading back to it are read again. A walk older than the stated age, or one
|
||||
readable by root alone. After a restart the traversal carries on from the cursor: what is before it is skipped
|
||||
without being read, the directories leading back to it are read again. A traversal older than the stated age, or one
|
||||
an apply asked to redo, is thrown away and begun again.
|
||||
|
||||
**Low priority, one at a time, no bound.** The walk runs on a goroutine locked to an operating-system thread of
|
||||
**Low priority, one at a time, no bound.** The traversal runs on a goroutine locked to an operating-system thread of
|
||||
its own, set to nice 19 and the idle I/O class; on Linux both are a thread's, so the engine's other work keeps
|
||||
its priority, and the thread ends with the goroutine. The package manager is run the same way. A priority that
|
||||
cannot be lowered fails the search rather than walk at full priority. There is no deadline: the search ends when
|
||||
it has walked everything, or with an error (a directory it could not read), which is incomplete.
|
||||
cannot be lowered fails the search rather than traverse at full priority. There is no deadline: the search ends when
|
||||
it has traversed everything, or with an error (a directory it could not read), which is incomplete.
|
||||
|
||||
**A verdict only from a complete, fresh search.** The last complete search is kept beside the engine's state
|
||||
with its start and finish, and judges while it is younger than `rootsearch.FreshFor`, counted from its start.
|
||||
@@ -44,7 +44,7 @@ link, owned by the engine's account and writable by it alone, in a directory tha
|
||||
**When a search runs.** Once a day (`rootsearch.Every`, from the last complete one's start); after an apply
|
||||
whose outcomes moved a user, a package, an action, or a file of the user and group databases or of sudo's,
|
||||
doas's or polkit's rules (`apply.MovesRoot`), at most once an hour from the last search's end, the last complete
|
||||
search judging meanwhile; and at once when a filesystem appears that the last search did not walk, which that
|
||||
search judging meanwhile; and at once when a filesystem appears that the last search did not traverse, which that
|
||||
search does not judge.
|
||||
|
||||
**The values, one place** (`rootsearch`, read by both repositories):
|
||||
@@ -57,29 +57,29 @@ search does not judge.
|
||||
|
||||
`FreshFor` is `Every` plus `Quiet` so that a daily search which finishes within the controller's quiet never
|
||||
leaves an hour unjudged; the controller's test holds that relation. The operator's direction named 26 hours as an
|
||||
example; 36 is chosen because a full walk at idle priority on the control-node is expected to take hours, and
|
||||
example; 36 is chosen because a full traversal at idle priority on the control-node is expected to take hours, and
|
||||
with 26 hours a daily search longer than two hours would leave the account unjudged — and phone linking refused —
|
||||
every day until it finished.
|
||||
|
||||
## 2026-10-10 — the independent review, and what it changed
|
||||
|
||||
An independent review of both pull requests asked four questions: can a stale or incomplete search yield "free";
|
||||
can the agent account steer the kept files or the walk; does the walk hurt the machine; do the controller and the
|
||||
can the agent account steer the kept files or the traversal; does the traversal hurt the machine; do the controller and the
|
||||
engine agree. What it found, and what was done:
|
||||
|
||||
- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed walk
|
||||
- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed traversal
|
||||
skipped every name that sorted between the real cursor and its replacement, and could finish "free" without
|
||||
having read them. Any account can make such a name. The walk's paths are now kept as their bytes.
|
||||
- **An ask to search again was lost to a restart**, and the restarted engine carried on the walk that started
|
||||
before root acted. The ask now forgets the walk under way and stops it keeping progress.
|
||||
- **A fresh search judged a filesystem it never walked.** A complete search now judges only while it walked every
|
||||
filesystem a search would walk now; a filesystem mounted since makes the verdict "not judged yet" until walked.
|
||||
- **A mount point inside a walked filesystem was read before it was known to be another filesystem.** A FUSE
|
||||
mount root cannot read failed the walk, and a network mount whose server is away could hang it. Every mount
|
||||
having read them. Any account can make such a name. The traversal's paths are now kept as their bytes.
|
||||
- **An ask to search again was lost to a restart**, and the restarted engine carried on the traversal that started
|
||||
before root acted. The ask now forgets the traversal under way and stops it keeping progress.
|
||||
- **A fresh search judged a filesystem it never traversed.** A complete search now judges only while it traversed every
|
||||
filesystem a search would traverse now; a filesystem mounted since makes the verdict "not judged yet" until traversed.
|
||||
- **A mount point inside a traversed filesystem was read before it was known to be another filesystem.** A FUSE
|
||||
mount root cannot read failed the traversal, and a network mount whose server is away could hang it. Every mount
|
||||
point the kernel lists is now left unread. Mount points are read with all the kernel's escapes.
|
||||
- **A file found and gone before the package manager was asked failed every resumed search.** The walk looks at
|
||||
- **A file found and gone before the package manager was asked failed every resumed search.** The traversal looks at
|
||||
what it found again when it ends, and drops what went or is no longer setuid-root.
|
||||
- **Outcomes that moved nothing started a search**, and a search asked for could follow a walk of hours at once.
|
||||
- **Outcomes that moved nothing started a search**, and a search asked for could follow a traversal of hours at once.
|
||||
Only created, updated, corrected and removed outcomes count now, and an ask waits an hour from the last
|
||||
search's end.
|
||||
- **Not changed, because the mesh cannot do it:** the review asked that a file or archive declared setuid start a
|
||||
@@ -87,53 +87,53 @@ engine agree. What it found, and what was done:
|
||||
|
||||
## Open, after the fix
|
||||
|
||||
- **How long the first walk on the control-node takes is not measured.** Until the first complete search, the
|
||||
- **How long the first traversal on the control-node takes is not measured.** Until the first complete search, the
|
||||
verdict there is `not judged yet`, and linking stays refused. The engine says each search's end and how long
|
||||
it ran in its journal; that line is the measurement. If a walk takes longer than `Quiet`, the self-check
|
||||
it ran in its journal; that line is the measurement. If a traversal takes longer than `Quiet`, the self-check
|
||||
raises the urgent condition, which is right: it is then not known to finish.
|
||||
- **The idle I/O class is honoured only by a disk scheduler that has classes** (BFQ). Under another scheduler
|
||||
the walk still runs at nice 19 but competes for the disk as any reader does.
|
||||
- **An ask to search again is not kept across a restart.** The walk under way is forgotten when it is asked,
|
||||
so a restarted engine does not carry an older walk on; the ask itself is lost, and the daily search covers it.
|
||||
the traversal still runs at nice 19 but competes for the disk as any reader does.
|
||||
- **An ask to search again is not kept across a restart.** The traversal under way is forgotten when it is asked,
|
||||
so a restarted engine does not carry an older traversal on; the ask itself is lost, and the daily search covers it.
|
||||
The last complete search keeps judging meanwhile, on the reasoning in the report.
|
||||
- **The agent account can make the search fail, never pass.** A tree deeper than the longest path the kernel
|
||||
takes, or a directory unreadable to root through FUSE, ends every walk incomplete; a directory of millions of
|
||||
takes, or a directory unreadable to root through FUSE, ends every traversal incomplete; a directory of millions of
|
||||
entries makes the engine hold them all at once. Each is "not judged" — the urgent condition, and linking
|
||||
refused — never a pass. Reading by directory handle, as `find` did, would close the first.
|
||||
- **A filesystem mounted only on use** (an automounted boot partition) makes the verdict "not judged yet" each
|
||||
time it appears unwalked, until a walk covers it.
|
||||
- **Part of the walk's cost escapes its priority**: Go's garbage collector runs on threads of normal priority,
|
||||
time it appears untraversed, until a traversal covers it.
|
||||
- **Part of the traversal's cost escapes its priority**: Go's garbage collector runs on threads of normal priority,
|
||||
and a software RAID's own threads do not carry the idle class to the disks.
|
||||
- **ADR 0266 §3 item 10** still says "at most hourly" and "at most 15 minutes". On resolution it gains a
|
||||
progressive insight naming this issue and the values above.
|
||||
- **A later option, not built now:** a fanotify filesystem mark with `FAN_ATTRIB` (Linux 5.1 and later) on
|
||||
every searched filesystem would report each change of mode or owner as it happens, turning the daily walk into
|
||||
every searched filesystem would report each change of mode or owner as it happens, turning the daily traversal into
|
||||
a true incremental search.
|
||||
- **Resolution** of this core issue needs a replay in mesh-lab's register, or `replay-none:` with the reason.
|
||||
|
||||
## 2026-10-10 — the search shows its progress, and the operator can ask for a fresh one
|
||||
|
||||
- **The progress was kept and seen by nobody.** At 01:40 the operator asked how far the first walk on the
|
||||
control-node had got. The walk kept its progress every minute beside the engine's state, but `root-free` said
|
||||
- **The progress was kept and seen by nobody.** At 01:40 the operator asked how far the first traversal on the
|
||||
control-node had got. The traversal kept its progress every minute beside the engine's state, but `root-free` said
|
||||
only "not judged yet (search running), started 01:27", and no tool read further. The node-engine serves no
|
||||
tools of its own, so the progress now travels where the verdict already does: the agent account's verdict
|
||||
carries it as fields, and its reason gains one line that `root-free` repeats. The line holds filesystems done
|
||||
of the total, the mount point being walked, the entries visited, the setuid- or setgid-root files found so far,
|
||||
of the total, the mount point being traversed, the entries visited, the setuid- or setgid-root files found so far,
|
||||
when the progress was last kept, and how long each finished filesystem took. Counts, times and mount points
|
||||
only: the cursor and every file found are often under an account's home, and stay in the engine's state and
|
||||
journal. The line moves at each checkpoint (once a minute) and is never part of what makes a verdict
|
||||
"changed", so the journal does not repeat it every minute; the statement is said every minute anyway while
|
||||
the account is not healthy. An entry is counted once across a resume, as it is kept with the cursor it
|
||||
matches. The time a filesystem took counts only the time the engine walked it, not the time it was stopped.
|
||||
matches. The time a filesystem took counts only the time the engine traversed it, not the time it was stopped.
|
||||
- **A change by hand was not searched for until the next day.** The first complete search on the control-node
|
||||
ended at about 01:48 and found two setuid-root programs no package owns, copies in the operator's home. When the
|
||||
operator removes them by hand, no apply says so, and nothing searches again until the daily search. The
|
||||
operator now asks for one at the controller's terminal: `nox node setuid-search <node>` on the control-node.
|
||||
The controller signs the ask with the mesh's key, as it signs a hand-over, under a signing context of its own.
|
||||
The engine throws away the last complete search and any walk under way or kept, and the next look starts a
|
||||
The engine throws away the last complete search and any traversal under way or kept, and the next look starts a
|
||||
full search. Until it completes, the verdict is "not judged yet", with its progress.
|
||||
- **Why at the terminal alone, not a verb.** A search can never make a machine root-free wrongly: it judges only
|
||||
once complete. But every ask throws a walk away. Asked again and again, it would keep the machine unjudged
|
||||
once complete. But every ask throws a traversal away. Asked again and again, it would keep the machine unjudged
|
||||
for good and its disks busy, and whoever may call a verb includes agents. The operator needs the line only
|
||||
after a change made by hand.
|
||||
- **A find counts only where the agent account can execute it.** The operator asked how the agent account could
|
||||
|
||||
+24
-24
@@ -11,17 +11,17 @@ Shortly after midnight on 2026-10-10 (local time) two pull requests of the catal
|
||||
their checks.
|
||||
|
||||
`mesh-delivery.deliveries` then showed every delivery of the dunst pull request `superseded`, and the
|
||||
claude-code delivery `held`, waiting for "a person: merged, and the controller opened no walk for it within
|
||||
claude-code delivery `held`, waiting for "a person: merged, and the controller opened no rollout for it within
|
||||
10m0s — nothing it holds follows that branch, or the merge was not heard: a person decides". The
|
||||
controller's `plans` said of the walk for the catalogue's head, which carried both merges: "published; its
|
||||
walk waits for mesh-delivery's word — `plans go … --why …` starts it by hand".
|
||||
controller's `plans` said of the rollout for the catalogue's head, which carried both merges: "published; its
|
||||
rollout waits for mesh-delivery's word — `plans go … --why …` starts it by hand".
|
||||
|
||||
Both reasons were wrong. A walk had been opened, and it contained the held merge. Nothing gave it the word.
|
||||
Both reasons were wrong. A rollout had been opened, and it contained the held merge. Nothing gave it the word.
|
||||
The operator's change sat undelivered for 58 minutes, until the operator's hand-act `mesh-controller.plans
|
||||
go` started the walk. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours
|
||||
before `mesh-delivery.stalled` lists it, and the walk's own wait raised only a warning, at 00:51, which said
|
||||
go` started the rollout. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours
|
||||
before `mesh-delivery.stalled` lists it, and the rollout's own wait raised only a warning, at 00:51, which said
|
||||
there was nothing to do and pointed at the ended delivery (the diagnosis's second entry; this sentence said
|
||||
before that the walk's wait raised nothing).
|
||||
before that the rollout's wait raised nothing).
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
@@ -33,41 +33,41 @@ cause.
|
||||
## Where it lives
|
||||
|
||||
The diagnosis ([01-diagnosis.md](01-diagnosis.md)) traces it to two parts that each behave as written and
|
||||
disagree on what a walk's commit means:
|
||||
disagree on what a rollout's commit means:
|
||||
|
||||
- The controller plans a merge it hears after a later merge of the same branch at the later merge's commit,
|
||||
and supersedes the earlier walk (the fix for
|
||||
and supersedes the earlier rollout (the fix for
|
||||
[issue 349](../349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md)).
|
||||
So the claude-code merge got a walk at the dunst merge's commit.
|
||||
- mesh-delivery finds the delivery a walk walks by its walk id or by the commit it was merged as. The new
|
||||
walk matched the dunst delivery, which its first walk's supersession had already ended, so no live
|
||||
delivery held the walk and nothing gave it its turn. The claude-code delivery, merged as a different
|
||||
commit, never got a walk and was held after ten minutes.
|
||||
So the claude-code merge got a rollout at the dunst merge's commit.
|
||||
- mesh-delivery finds the delivery a rollout rollouts by its rollout id or by the commit it was merged as. The new
|
||||
rollout matched the dunst delivery, which its first rollout's supersession had already ended, so no live
|
||||
delivery held the rollout and nothing gave it its turn. The claude-code delivery, merged as a different
|
||||
commit, never got a rollout and was held after ten minutes.
|
||||
|
||||
## Open
|
||||
|
||||
- How mesh-delivery learns which merges a walk carries: the controller's walk record naming every merge it
|
||||
- How mesh-delivery learns which merges a rollout carries: the controller's rollout record naming every merge it
|
||||
answers (its commit and the commits planned into it), or mesh-delivery reading containment itself, as it
|
||||
already does to decide *covered*.
|
||||
- A superseded walk whose successor matches only final deliveries: whether the successor should be adopted
|
||||
as a delivery of its own, so that something always holds a waiting walk.
|
||||
- A walk waiting for mesh-delivery's word past a bound, and a delivery held because no walk was opened,
|
||||
- A superseded rollout whose successor matches only final deliveries: whether the successor should be adopted
|
||||
as a delivery of its own, so that something always holds a waiting rollout.
|
||||
- A rollout waiting for mesh-delivery's word past a bound, and a delivery held because no rollout was opened,
|
||||
raise a condition to the operator instead of waiting silently (failure must be loud).
|
||||
|
||||
## Decided
|
||||
|
||||
The operator decided the model that removes this shape, not only its bookkeeping:
|
||||
ADR 0276. Merges are collected in a rolling merge window and each batch is walked once, with at most one
|
||||
commit per repository, one walk at a time. The bookkeeping that tied a walk to an ended delivery is repaired as
|
||||
ADR 0276. Merges are collected in a rolling merge window and each batch is rolled out once, with at most one
|
||||
commit per repository, one rollout at a time. The bookkeeping that tied a rollout to an ended delivery is repaired as
|
||||
well, and every wait of this shape has a bound and a condition. It is designed in to-be 47, *The merge window
|
||||
and the batch*.
|
||||
|
||||
## Resolved
|
||||
|
||||
2026-10-10. mesh-delivery's side is mesh-catalog#171 (merged): a walk is tied to every live delivery whose
|
||||
merge it answers, never to one that has ended, and a waiting walk no live delivery holds is listed by `stalled`
|
||||
at once. The controller's side is mesh-controller#197: the merge window and the batch, one walk per batch with
|
||||
one commit per repository, the walk naming every merge it answers, and the bounds of decision 9. The replay
|
||||
2026-10-10. mesh-delivery's side is mesh-catalog#171 (merged): a rollout is tied to every live delivery whose
|
||||
merge it answers, never to one that has ended, and a waiting rollout no live delivery holds is listed by `stalled`
|
||||
at once. The controller's side is mesh-controller#197: the merge window and the batch, one rollout per batch with
|
||||
one commit per repository, the rollout naming every merge it answers, and the bounds of decision 9. The replay
|
||||
R362 is registered in mesh-lab's replays register (mesh-lab#75), proved to fail on the commit before the
|
||||
mesh-delivery fix and pass on it. The build raised one rule the record had not settled, decided during the
|
||||
build and written into ADR 0276: a merge on the controller's own path never shares a batch with a catalogue
|
||||
|
||||
+23
-23
@@ -9,38 +9,38 @@ request as 48bda475 at 00:21:09. 48bda475 is the branch's head and contains 553b
|
||||
|
||||
**What the controller did** (its journal, 00:21:14 to 00:21:15):
|
||||
|
||||
1. It heard the dunst merge first and opened a walk at 48bda475 for dunst, waiting for mesh-delivery's word.
|
||||
1. It heard the dunst merge first and opened a rollout at 48bda475 for dunst, waiting for mesh-delivery's word.
|
||||
2. It then heard the claude-code merge, and printed "553b7191 was merged before 48bda475 (…, building): what
|
||||
it moved is built from 48bda475, which contains it". This is the rule that answered
|
||||
[issue 349](../349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md):
|
||||
`SourceMoved` replaces the merge's commit and merge time with the newest planned merge's, when that merge
|
||||
is later on the branch (`laterOnTheBranch`).
|
||||
3. The second walk, also at 48bda475 and with the same merge time, superseded the first by when each was made
|
||||
3. The second rollout, also at 48bda475 and with the same merge time, superseded the first by when each was made
|
||||
(`supersededBy`, `earlierOnTheBranch`), and carried claude-code and dunst. It waited for mesh-delivery's
|
||||
word.
|
||||
|
||||
So every walk the controller opened was at 48bda475. None was at 553b7191.
|
||||
So every rollout the controller opened was at 48bda475. None was at 553b7191.
|
||||
|
||||
**What mesh-delivery did** (`mesh-delivery.show` for both deliveries):
|
||||
|
||||
1. The dunst delivery (merged as 48bda475) took the first walk, matched by its commit, and became
|
||||
1. The dunst delivery (merged as 48bda475) took the first rollout, matched by its commit, and became
|
||||
`published` at 00:21:14.
|
||||
2. The first walk's record came back superseded, and the table's `superseded` transition ended the dunst
|
||||
2. The first rollout's record came back superseded, and the table's `superseded` transition ended the dunst
|
||||
delivery at 00:21:15 ("a newer delivery to the same trunk took over its walk").
|
||||
3. The second walk was matched to a delivery by `deliveryWalking`, which looks first for a delivery holding
|
||||
that walk id and then for any delivery whose `merged_as` is the walk's commit, final or not. That is the
|
||||
dunst delivery again, already final. The walk was linked there, so it was never kept as unmatched and
|
||||
3. The second rollout was matched to a delivery by `deliveryWalking`, which looks first for a delivery holding
|
||||
that rollout id and then for any delivery whose `merged_as` is the rollout's commit, final or not. That is the
|
||||
dunst delivery again, already final. The rollout was linked there, so it was never kept as unmatched and
|
||||
never adopted as a delivery of its own. A turn (`turn`) is given only to a `published` or `delivering`
|
||||
delivery, so nothing ever gave the second walk its word.
|
||||
4. The claude-code delivery, merged as 553b7191, matched no walk. Ten minutes after its merge
|
||||
delivery, so nothing ever gave the second rollout its word.
|
||||
4. The claude-code delivery, merged as 553b7191, matched no rollout. Ten minutes after its merge
|
||||
(`unwalkedPast`) it was held for a person at 00:30:58, with a reason that names two causes, neither of
|
||||
which happened.
|
||||
|
||||
After the operator's `plans go`, the second walk's machine steps (claude-code and dunst, each on a node) were recorded on the superseded dunst delivery, which confirms that mesh-delivery reads that walk as
|
||||
After the operator's `plans go`, the second rollout's machine steps (claude-code and dunst, each on a node) were recorded on the superseded dunst delivery, which confirms that mesh-delivery reads that rollout as
|
||||
the dunst delivery's.
|
||||
|
||||
**Why nothing was raised.** A delivery held for a person is bounded at 24 hours before
|
||||
`mesh-delivery.stalled` lists it (`Bounds` in its table). The walk waiting for mesh-delivery's word is
|
||||
`mesh-delivery.stalled` lists it (`Bounds` in its table). The rollout waiting for mesh-delivery's word is
|
||||
counted against the controller's plan wait bound of 30 minutes for `status`, but no condition reached the
|
||||
operator in the 58 minutes it waited. I did not trace why the controller's side stayed quiet; that belongs
|
||||
to the fix.
|
||||
@@ -48,28 +48,28 @@ to the fix.
|
||||
## Ruled out
|
||||
|
||||
- **The merge was not heard.** The controller's journal shows it was heard and planned within 23 seconds.
|
||||
- **Nothing follows that branch.** claude-code follows the catalogue's trunk; the second walk built it.
|
||||
- **Nothing follows that branch.** claude-code follows the catalogue's trunk; the second rollout built it.
|
||||
- **A failed check.** Both heads passed the merge gate and the repository's check.
|
||||
|
||||
## Located
|
||||
|
||||
Two parts, each correct by its own rule, disagree on what a walk's commit means. The controller (since issue
|
||||
349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a walk's commit is the
|
||||
merge commit of exactly one delivery. `located-in` names both. The fix is to make the walk say which merges
|
||||
it answers, or to make mesh-delivery match by containment, and to let no waiting walk be held by nothing.
|
||||
Two parts, each correct by its own rule, disagree on what a rollout's commit means. The controller (since issue
|
||||
349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a rollout's commit is the
|
||||
merge commit of exactly one delivery. `located-in` names both. The fix is to make the rollout say which merges
|
||||
it answers, or to make mesh-delivery match by containment, and to let no waiting rollout be held by nothing.
|
||||
|
||||
## 2026-10-10 — what the controller's side said
|
||||
|
||||
The first entry says no condition reached the operator in the 58 minutes. The controller's condition history
|
||||
says more precisely what happened. S16 (a walk waiting for its delivery's word) raised
|
||||
`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the walk opened, as its bound says.
|
||||
says more precisely what happened. S16 (a rollout waiting for its delivery's word) raised
|
||||
`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the rollout opened, as its bound says.
|
||||
It was a warning, its explanation was "Nothing for you to do … It becomes a question for you if it still waits
|
||||
after 4 hours", and its summary sent the reader to "`mesh-delivery.show` for the delivery that landed as
|
||||
48bda475", which is the dunst delivery the supersession had already ended. It cleared at 01:20:32, when the
|
||||
operator's `plans go` started the walk.
|
||||
operator's `plans go` started the rollout.
|
||||
|
||||
So the controller's side was not silent; it was mild and pointed at the wrong delivery. S16 cannot tell a
|
||||
group member waiting its turn from a walk nobody holds; only mesh-delivery knows that. That is why ADR 0276
|
||||
decision 9 has mesh-delivery list a waiting walk no live delivery holds through `stalled` at once, and has
|
||||
S16 name the merges a walk answers.
|
||||
group member waiting its turn from a rollout nobody holds; only mesh-delivery knows that. That is why ADR 0276
|
||||
decision 9 has mesh-delivery list a waiting rollout no live delivery holds through `stalled` at once, and has
|
||||
S16 name the merges a rollout answers.
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ and why. Today that is almost entirely **Novox Mesh**, the foundation everything
|
||||
| [`04-ISSUES`](04-ISSUES/) | The front door for "something is wrong" at the level of design or governance. |
|
||||
|
||||
**The numbering is the flow.** Research produces a decision; the decision authorises a design.
|
||||
That is why decisions are `02` and design is `03` — a reader following the numbers walks the
|
||||
That is why decisions are `02` and design is `03` — a reader following the numbers rollouts the
|
||||
process in the order it happens. The research efforts and decision records themselves live in the knowledge
|
||||
base, the seat `knowledge-base`, read and written through the mesh MCP server from any node (ADR 0290).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user