|
|
|
@@ -0,0 +1,430 @@
|
|
|
|
|
---
|
|
|
|
|
layer: to-be
|
|
|
|
|
status: in-progress
|
|
|
|
|
design: 47
|
|
|
|
|
milestone: H
|
|
|
|
|
updated: 2026-11-20
|
|
|
|
|
decisions:
|
|
|
|
|
- "ADR 0666"
|
|
|
|
|
- "ADR 0667"
|
|
|
|
|
- "ADR 0681"
|
|
|
|
|
- "ADR 0548"
|
|
|
|
|
---
|
|
|
|
|
# 47, milestone H — the plan to deliver the build-and-delivery system properly
|
|
|
|
|
|
|
|
|
|
The operator, 2026-11-19 about 23:30 CET: "This mess has been going on for too long and I want to fix it properly.
|
|
|
|
|
Write out a plan to deliver our mesh build-and-delivery system. Thoroughly. The research and design 47 should have
|
|
|
|
|
all the details of what it should look like and what it should do. Unfortunately, the code is very very poor. So we
|
|
|
|
|
need to fix it. What is our best approach? I'm willing to do some hand-deploys to get us into a clean state." And:
|
|
|
|
|
"Nobody should 'take in a merge', only commits to the configured branch of registered modules."
|
|
|
|
|
|
|
|
|
|
This file is that plan. It was written by the lead (lead-47) on 2026-11-20 between 00:00 and 01:00 CET from seven
|
|
|
|
|
readings by agents: one specification per part of the design (what design 47, its milestone G, research 086 and their
|
|
|
|
|
records say the part does; no code read) and one gap table per part (main's code of 2026-11-19 held line by line
|
|
|
|
|
against that specification, with what runs live on each machine). The readings are in the working folder
|
|
|
|
|
`.work/plan-47/` of the lead's hq working copy (`spec/`, `gaps/`, `live-state.md`); their findings are summarised
|
|
|
|
|
here, and every number below comes from them unless it says otherwise. The plan is the lead's; the operator reads it
|
|
|
|
|
before any code is written.
|
|
|
|
|
|
|
|
|
|
## 1. Where the system stands (read 2026-11-20 00:05 to 00:30 CET)
|
|
|
|
|
|
|
|
|
|
**What works.** One pipeline delivered one module end to end with no hand act on 2026-11-19 at 22:48 (the image
|
|
|
|
|
registry, main.1039). The shape of the design is in the code: of 619 table rows of design lines held against main, 413 are built by their contract: 78
|
|
|
|
|
per cent of the 527 rows judged (the rest belong to another part's code) (per part: runner 87 of 100 judged, gate and bridge 71 of 93, indexer 70 of
|
|
|
|
|
89, assigner and node-engine 56 of 76). No part needs to be written again whole.
|
|
|
|
|
|
|
|
|
|
**What does not work.** The system cannot deliver itself today:
|
|
|
|
|
|
|
|
|
|
| symptom (live) | count | cause (from the gap tables) |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| wants held or failed on the four machines | 205 held, 66 failed | the catalog half holds no entry for their fingerprints; the node-engines compose nothing new |
|
|
|
|
|
| compositions fail on shanks | entries `asked` since 2026-11-18 03:09, unready answers seen from 2026-11-19 11:56 | an entry ask that failed in transport is never asked again; the node-engine's shadow waits on it without bound (the surveyor's reading); other compositions failed on the removed revision trunk.510 and on a call-token refusal |
|
|
|
|
|
| delivery records open, none ended | 833 open, 0 ended since 2026-11-14 | the gate hears only `take-in.taken`; nothing writes a record's trace, so its difference is never known |
|
|
|
|
|
| pipelines about to start at the next take-in | about 60 (reasoned from the sweep's rule; no tool reads the runner's waiting starts) | the runner's sweep starts a pipeline for every module whose fingerprint has none (b9c50ef0, 2026-11-16); no design line allows it |
|
|
|
|
|
| source revision moved twice | both paths on main | A-464 (per merged message) wins live over A-34 (on the take-in): a group's members become separate revisions; a pushed commit reaches the source only through the take-in or the gate's comparison at its next start |
|
|
|
|
|
| catalog revision removed while pipelines name it | trunk.510 at 22:51 | pins are judged at restart against a key written once; only the take-in pins pipelines |
|
|
|
|
|
| stale bus state | 753 merges, 6,951 owed, 898 asks, 1,256 pulls keys; 5 pipelines running since midday; 31 stale pin records | buckets of the pre-G model never dropped; no tool reads their contents |
|
|
|
|
|
| bus load | 98 slow consumers, 117 consumers behind; the watchman 39,037 events behind | mesh-config passes every 5 minutes and polls every minute on main (fixed on the held pull requests); the forge module polls every minute |
|
|
|
|
|
| hand-placed programs | every delivery module but the indexer, the image registry and the tool runner (on ace and g14) | G's live start of 2026-11-16 and after; 16 drop-in files under `/run/systemd/system` on the four machines (4 of them `.pre-g` copies), lost at a reboot |
|
|
|
|
|
|
|
|
|
|
**Every node, every module** (the assigner's assignments at revision 339, unchanged since 2026-11-18 02:08 UTC; the
|
|
|
|
|
node-engines' haves; read 00:47 to 00:54 CET):
|
|
|
|
|
|
|
|
|
|
| machine | placed | running | held | failed |
|
|
|
|
|
|---|---|---|---|---|
|
|
|
|
|
| the control node | 90 | 10 | 53 (entry missing) | 24 |
|
|
|
|
|
| the build node | 80 | 12 | 59 (entry asked) | 7 |
|
|
|
|
|
| the desktop | 83 | 24 | 33 (entry missing) | 26 |
|
|
|
|
|
| the laptop | 76 | 6 | 60 (45 entry asked, 15 entry missing) | 9 |
|
|
|
|
|
|
|
|
|
|
65 of the 66 failed wants fail for one reason: the want names no digest for the module's files, code or tools (the
|
|
|
|
|
A-430 stand-in, a want naming its parts, left half built). The one other is the desktop's node-engine ("operation not
|
|
|
|
|
permitted" unpacking a version). No service is failed on any machine; the 25 open conditions are warnings. Every
|
|
|
|
|
delivery module's want is held by the assigner's rule 2 (no passed pipeline at the graph's fingerprint); before it,
|
|
|
|
|
rule 8 held the 23 bus-state modules because the bus module's have is held although the bus runs. The haves are wrong
|
|
|
|
|
about what runs for three modules (the tool runner on two machines, the indexer): `node-engine.process-builds` is the
|
|
|
|
|
tool that says it, found only in this second reading.
|
|
|
|
|
|
|
|
|
|
**The module graph** (the indexer, trunk revision 512, source 1038): 193 modules; 269 package edges (published before
|
|
|
|
|
it builds), 178 toolchain edges, 208 provider and requirement edges, 212 served-verb edges (19 verbs served by no
|
|
|
|
|
module), 30 await edges; the acting-part and contribution edges are not in the graph's state. The other watched branches
|
|
|
|
|
stand at revision 1 since 2026-11-14. No tool answers a build order or a deploy order for a set of modules, the
|
|
|
|
|
difference between two revisions, a revision narrowed by module or edge kind, or which source commit a fingerprint was
|
|
|
|
|
built from; the whole revision is 1.3 MB, over the bus's message size. Those reads are tools milestone H4 builds in the
|
|
|
|
|
indexer and the assigner (one-question-one-tool; section 4, item 12).
|
|
|
|
|
|
|
|
|
|
**No pull request is checked today.** The checker's verdict verb asks the indexer `indexer_read_candidates`, which no
|
|
|
|
|
running indexer answers (503 no responders), so no `mesh/check` status is ever set; the forge calls every pull request
|
|
|
|
|
mergeable because main requires no status. Nine open conditions ask `indexer_read_modules` the same way. Tools were
|
|
|
|
|
removed or renamed while their callers still name them; by design 47 G the pipeline makes the check and the checker
|
|
|
|
|
goes (G-2), which milestone H1 builds for hq too (hq is a registered module whose one step is its Go checks).
|
|
|
|
|
|
|
|
|
|
**The root of it, in one sentence.** The design settled on one model (a change is a commit on a watched branch of a
|
|
|
|
|
registered module; the forge bridge says it, the indexer takes it in per branch, the runner starts exactly the movers,
|
|
|
|
|
the gate follows), but the code carries that model and the earlier per-pull-request model side by side, and the live
|
|
|
|
|
patches of 2026-11-16 and 2026-11-17 (the sweep, the source per merged message, the catch-up by comparison) were
|
|
|
|
|
written on the earlier one. Every list above is a consequence.
|
|
|
|
|
|
|
|
|
|
## 2. The approach, decided
|
|
|
|
|
|
|
|
|
|
1. **One model, no leftovers.** A change is a commit on a watched branch of a registered module (design 47 G §11,
|
|
|
|
|
ADR 0666 §6). The forge bridge says every move of a watched branch's head, and at its start reads every watched
|
|
|
|
|
branch's head and says it (G §11, A-236). The indexer takes a move in per watched branch (A-476: the commit, the
|
|
|
|
|
fingerprints of that repository's modules, a catalog revision; `take-in.taken` names the movers by A-476a). The
|
|
|
|
|
runner starts one pipeline per mover and nothing else (G §12, §112: never by a sweep). The gate moves the
|
|
|
|
|
source revision on `take-in.taken` alone (A-34, G §592). Everything of the earlier model goes: the gate's and the
|
|
|
|
|
bridge's pull-request paths, the source per merged message (A-464 series), the catch-up by comparison (A-463,
|
|
|
|
|
A-490k), the runner's sweep, the start-time `outdated` by digest comparison (G §375; a kind-1 dependent is started
|
|
|
|
|
by `package-published`, which the runner's durable consumer delivers also after a stop), the forge module's
|
|
|
|
|
merge-check machinery and its polls. Principles 1 and 7 settle this; design 10 names a change found by comparison
|
|
|
|
|
as a failure mode.
|
|
|
|
|
2. **Fix in place, every module; write one part again.** Each gap table judged its modules by the share of lines
|
|
|
|
|
built and by whether their events, verbs and kept data are the design's. All say fix in place, mostly by deletion,
|
|
|
|
|
but two parts: the assigner's want rule (about 2,600 lines of `wants.go` and the want half of its pass, written again
|
|
|
|
|
to A-478) and the indexer's take-in part (about 3,400 of its 34,000 lines: pending, source numbering, behind, the
|
|
|
|
|
branch take-in, the pull-request candidates and previews) is written again as one take-in per watched branch fed
|
|
|
|
|
by the forge stream alone; its storage, entries, pins, versions and verbs stay.
|
|
|
|
|
3. **The design first, then code.** Where the operator's words and the lead's live answers departed from the
|
|
|
|
|
written lines and records, the lines and records are brought in line before any code (implement-the-design,
|
|
|
|
|
design-it-as-it-should-be): section 4 lists them. No lead answer patches a design question this time.
|
|
|
|
|
4. **One clean state by hand, once.** The system cannot deliver the fixes to itself, so milestone H1's programs are
|
|
|
|
|
built from main after its merges and placed by hand on the four machines as one mutually consistent set, the
|
|
|
|
|
stale bus state is dropped, and from that moment nothing is placed by hand again: every later milestone is
|
|
|
|
|
delivered by a pipeline, and that delivery is its proof (principle 5). Section 5 is the step list.
|
|
|
|
|
5. **Milestones in dependency order, one release each**, each proven by one pipeline delivering it, with the
|
|
|
|
|
guards of section 7 so it does not rot again.
|
|
|
|
|
|
|
|
|
|
## 3. The six parts: findings and the decision per module
|
|
|
|
|
|
|
|
|
|
Each part's specification and gap table hold the line-by-line detail; this is the summary a reader needs to judge
|
|
|
|
|
the plan.
|
|
|
|
|
|
|
|
|
|
### 3.1 The forge bridge and the gate (mesh-delivery: the bridge and both seat contracts; mesh-catalog: the gate and the forge module)
|
|
|
|
|
|
|
|
|
|
Built 71, built otherwise 5, contradicts 10, missing 7, not judged 15 (of 108). No gap is large alone.
|
|
|
|
|
|
|
|
|
|
- **No delivery record ever ends.** The gate consumes only `take-in.taken`. It must consume the runner's
|
|
|
|
|
`pipeline-ended`, the node-engines' `running`, the `wait-held` and `wait-ended` events and the assigner's families
|
|
|
|
|
(G §5, §146) to write a record's trace and end it (G-3 "The delivery record").
|
|
|
|
|
- **Two paths move the source revision**; the per-merge one (A-464, about 330 lines) acts live and the design's
|
|
|
|
|
(A-34, about 75 lines) is left without effect. A-464 goes.
|
|
|
|
|
- **The per-pull-request model** is about 1,300 of the gate's 5,475 lines and about 1,050 of the bridge's 4,281.
|
|
|
|
|
It goes, apart from what the operator's answer on groups keeps (section 4, question 1).
|
|
|
|
|
- **The forge module** holds no failed pipeline step (`failed-steps` has 0 entries, so `merge-pull` refuses
|
|
|
|
|
nothing), and still runs the old merge-check machinery: `mesh/merge-gate` statuses, weighing kept in files on disk,
|
|
|
|
|
a one-minute poll and a ten-minute timer. All of it goes.
|
|
|
|
|
- **Decision:** fix in place, mostly deletion. Gate: consume the two events, end records, one source path, drop the
|
|
|
|
|
pull-request model. Bridge: drop the pull-message path, the checker's owed acts and the merge act. Forge module:
|
|
|
|
|
drop the merge-check machinery and the polls; find why `seen` is empty (the surveyor's guess: the hand-placed
|
|
|
|
|
runner says `step-ended` without commit and fingerprint). Seat contracts: drop `asks` and `turns` and the callers
|
|
|
|
|
lists (small).
|
|
|
|
|
|
|
|
|
|
### 3.2 The indexer and mesh-config (mesh-delivery: the indexer; mesh-catalog: mesh-config)
|
|
|
|
|
|
|
|
|
|
Built 70, built otherwise 8, contradicts 7, missing 4, not judged 18 (of 107). Seven medium gaps in the indexer are
|
|
|
|
|
one problem: the take-in's shape.
|
|
|
|
|
|
|
|
|
|
- **The take-in is the trunk's alone, fed by two paths**: a `merged` or `pushed` message of any branch lands in the
|
|
|
|
|
trunk's one view (a `pushed` message has no base), other watched branches get revisions only by comparing against
|
|
|
|
|
the gate's `source-moved`, one `pending` holds every branch, and the catch-up takes heads in outside `pending`.
|
|
|
|
|
- **Revisions are made at every keep, not only at a take-in**: "27 changed" counts entries whose stored form differs
|
|
|
|
|
from the newest kept revision; two revisions were kept at the 22:51 restart with no `take-in.taken`.
|
|
|
|
|
- **Pins** are judged at restart against a `made` key written once in the state's life and against the server's
|
|
|
|
|
pending count, which is 0 at start, so the last restatement counts at once and trunk.510 was removed while
|
|
|
|
|
pipelines named it. Source numbers live in memory and restart from the gate's number while later pipelines exist.
|
|
|
|
|
- **mesh-config** on main passes every 5 minutes, polls every minute and reads its states key by key (issue 1982);
|
|
|
|
|
the held pull requests mesh-catalog #708, mesh-contracts #16, mesh-sdk #200 and mesh-delivery #261 make it wake
|
|
|
|
|
on events and one deadline timer, reading from watched copies, and remove its pull-request path (ADR 0681).
|
|
|
|
|
- **Decision:** write the take-in part again (one take-in per watched branch, from the forge stream alone, movers by
|
|
|
|
|
A-476a, source numbers on the bus, pins restated by every asker and judged against what is pinned now, never
|
|
|
|
|
against a count); keep storage, entries, pins' state, versions and verbs. mesh-config: merge the four held pull
|
|
|
|
|
requests; its reads of the indexer and the assigner through module tools (principle 9) and its full pass per
|
|
|
|
|
event are a later milestone's.
|
|
|
|
|
|
|
|
|
|
### 3.3 The pipeline runner (mesh-delivery: mesh-delivery-pipeline)
|
|
|
|
|
|
|
|
|
|
Built 87, built otherwise 5, contradicts 2, missing 6, not judged 19 (of 119). No large gap; 4 medium, 9 small.
|
|
|
|
|
|
|
|
|
|
- **The sweep** (`StartUnstarted`) runs after every take-in and at every start; removing it is small. The held pull
|
|
|
|
|
request mesh-delivery #262 adds a third start path (`StartHeld`); it is closed, not merged.
|
|
|
|
|
- **The refused step ends naming an empty pipeline** are the assigner's composition runs on the Go build seat: the
|
|
|
|
|
runner reads the pipeline field before the asker field. Skip an end whose asker is not the runner (small).
|
|
|
|
|
- **Missing, medium each:** `toolchain-agreed` starts no kind-3 rebuild (G §376); nothing consumes `approval-carried`
|
|
|
|
|
(G §542); each repository's own checks are not a step (G-2); the start-time `outdated` and `runner_catch_up` (G §375)
|
|
|
|
|
do not exist, and by decision 1 they are not built: a kind-1 dependent is started by `package-published`, which the
|
|
|
|
|
durable consumer delivers also after a stop (section 4, item 7 says it as a line).
|
|
|
|
|
- **A bound on pipelines at once** has no design line; principle 11 makes it a setting (`pipelines-at-once`). A start
|
|
|
|
|
over the bound is written with no steps and placed when the runner's own `pipeline-ended` comes, no timer
|
|
|
|
|
(medium). What a waiting pipeline reports against G §12's `pipeline-stands` is a design line (section 4).
|
|
|
|
|
- **Timers found:** mesh-sdk's view catch-up asks again on a timer (against A-490k's "no timer"); a one-minute
|
|
|
|
|
ticker while the step log is full. Both go in the class fix of section 3.6.
|
|
|
|
|
- **Decision:** fix in place. The self-hold A-446 removed is already gone from main (commit 070a5dd3).
|
|
|
|
|
|
|
|
|
|
### 3.4 Builds, the builder and the registries (mesh-delivery: the builder, the artifact holder; mesh-catalog: the build modules, the forge's package registry, the image registry)
|
|
|
|
|
|
|
|
|
|
Built 73, built otherwise 3, contradicts 8, missing 2, not judged 24 (of 110). 85 per cent of the judged rows
|
|
|
|
|
built; no large gap, five medium.
|
|
|
|
|
|
|
|
|
|
- **A failed key taken again skips a version** (G §369): the reserver treats a key's own standing reservation as its
|
|
|
|
|
floor unless the source says rebuild, and only the hand tool `builder_build_again` says so; a failed key taken again
|
|
|
|
|
by a look, an event or the start moves its number on by one. This is the 0.1.19 skip. Small.
|
|
|
|
|
- **The pre-G numbering still runs beside the reservations** (design 51's versions, replaced): it numbers every trunk
|
|
|
|
|
build from its own state and floor, can hold a build waiting for that floor, and its late look writes the live "done
|
|
|
|
|
build no version reserved" line. Medium; it goes.
|
|
|
|
|
- **The builder is not entered only through the runner's `build` step** (G §473, A-97): it builds every catalog entry
|
|
|
|
|
it hears and still follows the gate's `source`. Medium; it goes.
|
|
|
|
|
- **`build.ended` carries no version** (A-430b), and builds of other branches reserve nothing.
|
|
|
|
|
- **The build modules** pull their three queues in turns, not deliveries first (ADR 0525 §4, G §86; a code comment
|
|
|
|
|
cites an operator's yes of 2026-11-15; the surveyor did not search the knowledge base for it, so H2's prepare does: if
|
|
|
|
|
the yes was recorded, the record and the lines are out of date, not the code), and when idle re-pull every 5 seconds, a poll. The artifact
|
|
|
|
|
store is read by raw HTTP in group runs; the registries are reached through bus verbs. Two copies of the client that
|
|
|
|
|
asks `reserved` differ (the forge's and the image registry's).
|
|
|
|
|
- **The publish again** runs in one shape, A-457's retried items with A-473's ask at build end as their first try;
|
|
|
|
|
§588's event path is gone and its queue remains as plumbing. Live, all 42 publish retries ended after 8 tries,
|
|
|
|
|
unreleased, and 37 reservations built since 2026-11-18 were never published (cause a guess, in the table).
|
|
|
|
|
- **From the specification:** a module with no registration has no pipeline and is never built, so
|
|
|
|
|
mesh-delivery-sdk's missing registration blocks mesh-delivery-contracts by design; a `required` step ending
|
|
|
|
|
`nothing to run` holds the delivery and no line works it out again later; no line has the builder reserve an old done
|
|
|
|
|
build by itself. Each is a design line to write (section 4, item 11).
|
|
|
|
|
- **Decision:** fix in place, all three. Builder: remove the numbering, the gate-source follow, the builds without a
|
|
|
|
|
step ask and the other pre-G mechanisms the table lists; fix the reserver's floor and the version on `build.ended`.
|
|
|
|
|
Build modules: deliveries first and no idle poll in the shared step runner; the store read over the bus. Registries:
|
|
|
|
|
one `reserved` client, the forge's form moved into `mesh-delivery-contracts` as the surveyor recommends. The delivery
|
|
|
|
|
budget (G-2) is checked by nothing; medium, H2.
|
|
|
|
|
|
|
|
|
|
### 3.5 The assigner, the node-engine and the tool runner (mesh-delivery: the assigner; `mesh-host`: the node-engine; `mesh-tools`: the tool runner)
|
|
|
|
|
|
|
|
|
|
Built 56, built otherwise 8, contradicts 9, missing 3, not judged 7 (of 83). Assigner 31 of 44 built, node-engine
|
|
|
|
|
17 of 27, tool runner 8 of 12.
|
|
|
|
|
|
|
|
|
|
- **The want rule A-478 is not built** (the newest released version whose pipeline passed, on every machine that
|
|
|
|
|
places the module; the operator, 2026-11-17). Every old rule still holds wants: canary, grant, provisions, served
|
|
|
|
|
verbs, bus state, the acting-part waits (about 450 lines), the `version-delivered` holds (about 500), the parts
|
|
|
|
|
check (about 250), the rest about 300. All of it goes; the backup, the yes on a failed backup or a held revert, and
|
|
|
|
|
A-478b's order waits on one machine stay (the order waits reshaped to order one machine's apply, not hold wants).
|
|
|
|
|
- **The assigner reads every have** whole and with history (§520 says it goes); the node-engine reads every node's
|
|
|
|
|
record key (§459 forbids it); wants are not written from the kept assignment graph revision. Medium each, H1.
|
|
|
|
|
- **The shanks shadow fault**: the entries client leaves an entry `asked` after a transport failure and never asks
|
|
|
|
|
again; about 110 entries on shanks have stood `asked` since 2026-11-18 03:09. Ask again on failure, with a bound
|
|
|
|
|
(small). The surveyor's reading, to be confirmed by `entry-again` on one entry during the clean state.
|
|
|
|
|
- **The tool runner:** A-490m is ready on mesh-tools #77 (merge); it lacks a test that starts it whole.
|
|
|
|
|
- **Tests:** the node-engine has 1,719 unit tests against a budget of 500; every change to it shrinks that.
|
|
|
|
|
- **Decision:** fix in place, all three. The want rule is rewritten to A-478 in `wants.go` and the want half of the
|
|
|
|
|
pass; the old holds are deleted.
|
|
|
|
|
|
|
|
|
|
### 3.6 The status view, the watchman, health, the tools and the grants part (mesh-delivery: the view, the watchman, the grants part; mesh-catalog: the bus's module; mesh-tools; mesh-sdk)
|
|
|
|
|
|
|
|
|
|
Built 56, built otherwise 11, contradicts 5, missing 11, not judged 9 (of 92).
|
|
|
|
|
|
|
|
|
|
- **The status view, the watchman and the grants part read other modules' buckets directly**, against ADR 0596: the
|
|
|
|
|
view watches twelve such buckets, retries each every 5 seconds, and still follows the teller's answers and the gate's
|
|
|
|
|
source, which §512 retires. Large, with the want rule, the take-in and the SDK's generic state tools (the fill against
|
|
|
|
|
a cap in every module's tools, the grants part's `filling` and `room`) the large gaps of the survey.
|
|
|
|
|
- **The kind of fault behind issue 1982 is in more modules than mesh-config.** The grants part recomposes everything
|
|
|
|
|
and reads its own states key by key on every change; the indexer re-reads every module, every graph key and every
|
|
|
|
|
manifest after each verdict, import or source move (its modules state stands at revision 1,789,962); the view
|
|
|
|
|
retries twelve watches every 5 seconds; the forge module polls every minute; the build modules re-pull every 5
|
|
|
|
|
seconds when idle; mesh-sdk's view catch-up asks again on a timer. The gap table's section 5 has the table of
|
|
|
|
|
modules, functions and intervals. The fix belongs in mesh-sdk, for every module at once (fix-a-bug-met-twice).
|
|
|
|
|
- **`builder_stuck` times out** for a fixed cost, measured in the tool runner's journal on shanks: 1,010 values took
|
|
|
|
|
52.5 seconds and 53 values took 44.5 seconds. Each call starts a tool process that lists every key of `builds` and of
|
|
|
|
|
every build seat's `steps`, then reads one value per request. The builder keeps its own index of what is building
|
|
|
|
|
and answers from it. Medium.
|
|
|
|
|
- **The runner's live seat serves five read verbs and no act**; main has them all (commit 8fa1b8ae). The surveyors'
|
|
|
|
|
guess is a hand-placed build from before it, or an older tools bundle; the clean state ends either.
|
|
|
|
|
- **Who may call what** stands in two forms: §610's class defaults and A-434's stand-in that every account may do
|
|
|
|
|
everything. **The lead's answer: principle 6 settles it** (inside the mesh every module is trusted, who may do what
|
|
|
|
|
is not a question): A-434 stands, the callers lists go from every seat contract, and tests 40, 51 and 85 are not
|
|
|
|
|
checked while it holds.
|
|
|
|
|
- **No line names rates on the bus**; the bus's module gives connections and consumers, not rates. A tool to build in
|
|
|
|
|
the bus's module (principle 4), named in section 4 so the design says it.
|
|
|
|
|
- **Decision:** fix in place, every module. View: replace the bucket watches with events (the large gap, H4).
|
|
|
|
|
Watchman (16 of 23 built) and condition keeper (14 of 18): fix in place; their live builds predate milestone G. Grants
|
|
|
|
|
part: the re-read through the SDK shape in H4, its move from reads to events a release of its own (H4b). Bus module and tool runner: the
|
|
|
|
|
rate read and the SDK's generic state tools.
|
|
|
|
|
- **The wake-up rule is a class fix:** one shared shape in mesh-sdk (watched copies of a module's own states, a pass
|
|
|
|
|
only on a changed input, one deadline timer for its real deadlines, no poll), every delivery module moved onto it in
|
|
|
|
|
H4, and the review refuses a timer, a poll or a full re-read in any delivery module from H1 on.
|
|
|
|
|
|
|
|
|
|
## 4. What the design must say before code (milestone H0)
|
|
|
|
|
|
|
|
|
|
The design's lines and the records carry the operator's later words only in the lead's answers, and in places not
|
|
|
|
|
at all. Before any code, `hq-design-writer` writes these as design lines in this milestone's part and
|
|
|
|
|
`hq-record-writer` proposes the records, in one batch read by the four reviewers by architectural domain (playbook
|
|
|
|
|
02). The lead decides none of them.
|
|
|
|
|
|
|
|
|
|
**Questions for the operator** (each changes how the operator works or removes data; the lead's recommendation first):
|
|
|
|
|
|
|
|
|
|
1. **How a change across repositories lands together.** The design forms a group from pull requests' head branch
|
|
|
|
|
names (G §42, §716); the operator's rule is that the mesh knows only commits on watched branches. Recommendation:
|
|
|
|
|
a group is a trailer in each member commit's message (`Mesh-Group: <name>`, the name chosen by the author); the
|
|
|
|
|
forge bridge reads it from the commit and the gate forms the group from it; nothing is read from a pull request.
|
|
|
|
|
The alternative is no groups at all, with the order between modules left to the artifact dependency edges, which
|
|
|
|
|
does not cover a seat contract change that several modules must follow at once.
|
|
|
|
|
2. **Data the clean state removes:** the gate's `merges`, `owed`, `asks` and `pulls` buckets, its 833 open records,
|
|
|
|
|
the indexer's stale pins and the 5 pipelines running since 2026-11-19 midday, every held or failed want, and the
|
|
|
|
|
19 drop-in files. Recommendation: remove all of it; none is a record anyone reads (no tool reads what the buckets
|
|
|
|
|
hold), and the delivery records of the one model start from the clean state.
|
|
|
|
|
3. **Pipelines one at a time** until the operator raises it: the setting `pipelines-at-once` with default 1.
|
|
|
|
|
Recommendation: yes for the clean state and the first milestones; the operator raises it when the chain has proven
|
|
|
|
|
itself on about ten deliveries.
|
|
|
|
|
4. **The one hand placement** of milestone H1 at the operator's terminal, one command at a time as before, after
|
|
|
|
|
which no pack is run again except the ones this plan names. Recommendation: yes; it is what the operator offered.
|
|
|
|
|
|
|
|
|
|
**The operator's answers (2026-11-20 about 01:10 CET).** Question 2: "I don't care about old data at this moment, I want
|
|
|
|
|
a working mesh again": the stale state goes. Question 3: "for my part, pipelines don't exist, or a group of commits
|
|
|
|
|
form one big pipeline, but parallel pipelines sounds like a very bad idea": no two pipelines run at once, and one
|
|
|
|
|
take-in is one pipeline (design 47 G §44 already says it for a group: one source revision, one pipeline per watched
|
|
|
|
|
branch; the single commit is the case of one member); the setting of item 7 keeps the bound at one by its default and
|
|
|
|
|
the design says serial. Question 4: "I don't mind doing hand placements": yes. Question 1: the operator's first idea was a
|
|
|
|
|
build plan and a deploy plan per change (ADR 0238 §2, "one commit, one change plan", the glossary's *delivery plan*,
|
|
|
|
|
deleted at release 6 of ADR 0429 and replaced by ADR 0666's pipelines per module ordered by edges, from research 086);
|
|
|
|
|
what remained open was how the mesh learns which commits belong together, since ADR 0666 §13 reads a shared branch
|
|
|
|
|
name through pull requests. The operator, 01:15 CET: since every version is deployed by a merge into main, an action,
|
|
|
|
|
they have no objection to kicking the build-and-delivery process off by hand instead of listening to the forge's
|
|
|
|
|
hooks; the lead named principle 3 (a kick-off a person forgets is a silent non-delivery) and proposed the merge through
|
|
|
|
|
the mesh as the kick-off with the forge's event kept as the watchdog; the operator chose it (01:20 CET, "A, go
|
|
|
|
|
ahead"): one call of the seat `git`'s verb `merge-pull` naming one or more pull requests is the delivery's start and
|
|
|
|
|
the group; nothing reads a pull request after it; a head moved outside the mesh raises a condition and delivers
|
|
|
|
|
nothing. Item 1 below is answered so; item 6 is written from it. The operator also said they would not mind a start from scratch; the lead's answer is in
|
|
|
|
|
section 2 (fix in place, two parts written again) with the reason: 78 per cent of the lines are built by their contract,
|
|
|
|
|
and the parts that are the earlier model are named and bounded.
|
|
|
|
|
|
|
|
|
|
**Design lines and records to write** (no operator decision; `hq-design-writer` and `hq-record-writer`):
|
|
|
|
|
|
|
|
|
|
5. The take-in by A-476, as a record: it changes ADR 0666 §6 and ends ADR 0633 §1's hold for manifest validation
|
|
|
|
|
and the "pending revision"; the manifest validation becomes a pipeline step with a name, a server and a policy
|
|
|
|
|
default (A-476 names none), and `manifest-refused` and the cycle refusal move with it.
|
|
|
|
|
6. The forge bridge's one message for a watched branch's head moving, replacing the `merged`/`pushed` pair where
|
|
|
|
|
they differ (a `pushed` message carries no base today), and the end of A-464 to A-464e, A-463 and A-490k.
|
|
|
|
|
7. One take-in, one pipeline, and no two at once: the pipeline is per source revision of a watched branch, its steps per
|
|
|
|
|
mover in the graph's build order, delivered in the deploy order (the operator, 2026-11-20); the setting
|
|
|
|
|
`pipelines-at-once` with default 1 (principle 11), the order waiting take-ins are placed in, and that a waiting
|
|
|
|
|
pipeline is not owed progress under G §12. A record, since ADR 0666 §1 and §7 speak per module.
|
|
|
|
|
8. The want rule A-478 and A-478b as design lines, with the list of holds they end (§266, §268, §269, §376, A-189,
|
|
|
|
|
§524) and the reshaped order wait on one machine.
|
|
|
|
|
9. The wake-up rule for every delivery part (section 3.6) as a design line in design 47's main file, with how it is
|
|
|
|
|
checked (the review and the integration test that starts the module whole).
|
|
|
|
|
10. The small contradictions the specifications list, corrected in place: G §5's unprotected-branch refusal (removed
|
|
|
|
|
by the operator, still in ADR 0666 §3), G §654 (replaced by ADR 0681 §1), A-449's `pinned`/`freed` publisher,
|
|
|
|
|
test 53's `forge-made`, G §476's "one entry per module of that revision", ADR 0666 §12's three open asks (A-424),
|
|
|
|
|
§386's per-module durable consumer (A-397).
|
|
|
|
|
11. The builder's lines the specification found missing: a done build with no reservation is reserved by the builder
|
|
|
|
|
itself (not only by a hand tool); a failed fingerprint built again keeps its reservation; a `required` step with
|
|
|
|
|
nothing to run is worked out again at every pipeline, not only at the seed; a registered module whose pipeline has
|
|
|
|
|
no steps is refused at registration, naming the step (test 67); and the record of the build modules' queue order
|
|
|
|
|
(deliveries first, ADR 0525, or the turns the code takes on an operator's yes no record holds).
|
|
|
|
|
12. The reads the status view, the watchman and the grants part make of other parts' buckets, each turned into the
|
|
|
|
|
event or verb of the part that owns the data (ADR 0596), and the rate read on the seat `mesh-broker`.
|
|
|
|
|
13. The read tools the second live reading found missing (one-question-one-tool): in the indexer, one revision
|
|
|
|
|
narrowed by module or edge kind, a non-trunk branch's revision, the edges by kind, the difference between two
|
|
|
|
|
revisions, a build order for a set of modules, the source commit a fingerprint was built from; in the assigner, a
|
|
|
|
|
deploy order per machine; in the node-engine, its own build. Also the checker's and the conditions' calls of
|
|
|
|
|
indexer tools no indexer serves (`indexer_read_candidates`, `indexer_read_modules`), removed with their callers.
|
|
|
|
|
|
|
|
|
|
## 5. The clean state, by hand once (milestone H1's delivery)
|
|
|
|
|
|
|
|
|
|
Milestone H1 is "the one chain": the smallest set of fixes that makes a commit on a watched branch reach a running
|
|
|
|
|
module through the design's path with no hand act, in every module of the chain. It is built, reviewed and merged
|
|
|
|
|
like any milestone, but its delivery is by hand, because the running system cannot carry it. The steps, in order;
|
|
|
|
|
the operator runs each pack one command at a time and says "done"; the lead checks each with the tools before the
|
|
|
|
|
next:
|
|
|
|
|
|
|
|
|
|
1. **Freeze.** No merge to any delivery repository but H1's own. The runner on novox is stopped through the service
|
|
|
|
|
seat so no take-in starts a pipeline during the freeze; the assigner stays, so nothing moves.
|
|
|
|
|
2. **Merge H1's set to main**, in this order, each reviewed: mesh-contracts #16 and mesh-sdk #200 (held), then
|
|
|
|
|
mesh-catalog #708 and mesh-delivery #261 (held, mesh-config), mesh-tools #77 (held), and H1's own pull requests
|
|
|
|
|
(indexer take-in, runner, gate, bridge, assigner, node-engine). mesh-delivery #262 is closed as superseded. The
|
|
|
|
|
stopped runner starts nothing; the indexer takes the commits in and the gate follows.
|
|
|
|
|
3. **Build H1's programs from main** with the packs (the g-start phases, one per program), on shanks and novox as
|
|
|
|
|
before, from main's head after step 2 so that main is what runs.
|
|
|
|
|
4. **Drop the stale state** (question 2's yes): the buckets, the records, the pins, the stuck pipelines, the held and
|
|
|
|
|
failed wants, through each module's tools where one exists (the surveyor found none for the bucket contents; the
|
|
|
|
|
pack carries the one-off deletion and says what it deleted, counted). The drop-ins are removed and the units the
|
|
|
|
|
node-engine writes take their place as the wants compose.
|
|
|
|
|
5. **Place the programs** in the order the graph gives (derived by the reader of 00:54 CET from the package,
|
|
|
|
|
toolchain and provision edges; no tool answers it yet), each step read with the tools before the next:
|
|
|
|
|
1. keep the bus, the forge and the image registry running; make the bus module's have `running` (else the
|
|
|
|
|
assigner's rule 8 holds the 23 bus-state modules);
|
|
|
|
|
2. the packages first: `mesh-sdk`, `mesh-contracts`, `mesh-delivery-sdk` published (every Go module needs them;
|
|
|
|
|
`mesh-delivery-sdk` needs its registration first, section 3.4);
|
|
|
|
|
3. build with the running Go build module (it is what breaks the cycle "every Go module, the build module itself
|
|
|
|
|
included, is built by a running build module"), the builder first, then the indexer, the forge bridge, the
|
|
|
|
|
runner, the assigner; then the three build modules (they need the builder's package), the gate (needs the
|
|
|
|
|
indexer's and the bridge's packages) and the grants part; then the node-engine;
|
|
|
|
|
4. place on the control node: the indexer, the forge bridge, the gate, the forge module, mesh-config; on the build
|
|
|
|
|
node the builder; on the desktop the assigner;
|
|
|
|
|
5. place the node-engines: the control node, then the build node, the desktop, the laptop;
|
|
|
|
|
6. place the watchman, the status view, the teller, the grants part and the hand-act log on the control node;
|
|
|
|
|
the checker is not placed (it goes, section 1);
|
|
|
|
|
7. after each machine, read its health, its first views, its `process-builds` against its haves, and the bus's
|
|
|
|
|
consumers.
|
|
|
|
|
6. **Start the runner last**, with `pipelines-at-once` at 1.
|
|
|
|
|
7. **Prove it.** One single-module change (a comment in a module with lint, unit and composition steps only), merged
|
|
|
|
|
to main, followed from `take-in.taken` through `pipeline-ended` to the node-engine's `running` and the gate's
|
|
|
|
|
ended record, with no hand act; then one two-module change along a dependency edge (mesh-sdk and a module that
|
|
|
|
|
requires it published), the same way. Each proof read with the tools and quoted with its times.
|
|
|
|
|
8. **Lift the freeze.** From here every merge is delivered by its pipeline; a hand pack is a breach the lead names.
|
|
|
|
|
|
|
|
|
|
## 6. The milestones after the clean state
|
|
|
|
|
|
|
|
|
|
Each is one release of playbook 07 (prepare by `mesh-preparer`, develop, one review, one check, delivered by its
|
|
|
|
|
own pipeline), in this order, the next starting when the previous one's delivery is read on the machines:
|
|
|
|
|
|
|
|
|
|
| # | outcome | repositories | proof |
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
| H0 | the design says the one model whole (section 4) | hq, the knowledge base | the four reviewers by architectural domain approve the batch; the operator's four answers are in |
|
|
|
|
|
| H1 | the one chain works with no hand act (section 5) | `mesh-delivery`, `mesh-catalog`, `mesh-host`, `mesh-tools`, `mesh-sdk`, `mesh-contracts` | the two proofs of step 7 |
|
|
|
|
|
| H2 | the builder, the build modules and the registries follow the latest lines (section 3.4; versions, publish again, the floor, a step with nothing to run, a module with no registration) | mesh-delivery, mesh-catalog | a version is skipped by nothing; every registered module has a pipeline with steps |
|
|
|
|
|
| H3 | the gate and the forge module carry only the one model; groups by the operator's answer; the forge module without its merge-check machinery | mesh-catalog, mesh-delivery | a group of two repositories delivers as one; every delivery record ends |
|
|
|
|
|
| H4 | every delivery part wakes on its inputs alone and is steered by its tools (sections 3.6, 7): the shared wake-up shape (mesh-config's full pass per event and its reads through module tools, the indexer's re-read after every verdict, the node-engine's retry clocks, the build modules' idle pull), the view and the watchman without reads of another part's state, the SDK's generic state tools with the fill against a cap, the missing tools per module, rates on the bus | `mesh-sdk`, `mesh-delivery`, `mesh-catalog`, `mesh-host` | the bus's slow consumers and behind counts at 0 for an hour; every module's tools pass the principle-4 table |
|
|
|
|
|
| H4b | the grants part says what it holds as events and reads nothing of another part's bus state (ADR 0596) | `mesh-delivery` | the grants part's own `filling` and `room` and no bucket watch in its code |
|
|
|
|
|
| H5 | tests to budget and one integration test per module that starts it whole (ADR 0548); the catalog check that refuses a module whose state has no read tool | every delivery repository, mesh-catalog's check | the check reports each module's counts within budget |
|
|
|
|
|
|
|
|
|
|
Milestone H1 is large for one release (six repositories); it is not cut (how-work-is-done), its subtasks are split
|
|
|
|
|
at the prepare, one developer per repository on the larger model for the indexer and the gate (their parts meet at
|
|
|
|
|
`take-in.taken` and the source revision), the smaller model for the rest.
|
|
|
|
|
|
|
|
|
|
## 7. The guards, so it does not rot again
|
|
|
|
|
|
|
|
|
|
- **An integration test per module that starts it whole** against a real bus and store (ADR 0548), run in the one
|
|
|
|
|
check per release, and the change to how a module runs carries it in the same commit.
|
|
|
|
|
- **The review reads against the one model:** a sweep, a poll, a timer for another part, a comparison in place of an
|
|
|
|
|
event, a read of another part's state or a per-pull-request path is a finding that fails the review.
|
|
|
|
|
- **The tools table per module** (principle 4) in every pull request of a delivery module, and the catalog check of
|
|
|
|
|
H5 that refuses a module without its read tool.
|
|
|
|
|
- **No hand pack after the clean state.** A delivery that fails is fixed in its repository and delivered by its
|
|
|
|
|
pipeline; a pack is a breach the lead names, with the one exception the operator grants in words.
|
|
|
|
|
- **The test budget** per module (50 integration, 500 unit) reported by the check; a module over it shrinks with
|
|
|
|
|
every change.
|
|
|
|
|
- **Live answers become design lines the same day** (implement-the-design): a lead answer that patches a model
|
|
|
|
|
question is a breach the reviewer reports.
|
|
|
|
|
|
|
|
|
|
## 8. What the plan does not do
|
|
|
|
|
|
|
|
|
|
- It does not merge the six held pull requests now; they merge in H1's step 2, and one (#262) is closed.
|
|
|
|
|
- It does not touch the controller, the vault or the recovery module beyond what the chain needs; the recovery
|
|
|
|
|
module is not placed and is a later milestone's.
|
|
|
|
|
- It does not change the catalog's module set or the build nodes (novox's build agent stays paused).
|
|
|
|
|
- It leaves what the surveyors could not read as named guesses, each with how to check it, in the gap tables'
|
|
|
|
|
last sections; the prepare of each milestone checks them.
|