Merge pull request 'ADR 0218 and issues 250-254: delivery in order, and a store that keeps what it says' (#102) from records/delivery-and-store into main

2026-10-05 15:55:45 +00:00
parent 400ed8d396 f000288147
commit 44808ac563
12 changed files with 363 additions and 5 deletions
@@ -60,6 +60,14 @@ with the server held still for its duration. Plain collection, not `--delete-unt
 mesh keeps is still a manifest in the store, so it is still referenced, so its blobs stay — the
 dangerous flag is not needed at all once the mesh is the one deciding.

+> **Progressive insight — 2026-10-05.** "What the mesh keeps is still a manifest in the store" was true
+> of images and false of archives: the builder published every archive as a bare blob no manifest names,
+> and the store's collector keeps only what a manifest names. Its first night would have deleted every
+> archive the mesh keeps ([issue 253](../04-ISSUES/253-the-stores-collector-would-delete-every-archive-the-mesh-keeps/00-report.md)).
+> The decision stands — the mesh decides, the store reclaims with plain collection. What changes is how an
+> archive is published: with a manifest that holds it, so the sentence becomes true of archives too. Until
+> every kept archive is held, the collector runs as a dry run.
+
 **3. What the mesh keeps, stated as three reasons rather than a number.** A digest is kept because:

 - **a definition names it** — every artifact reference in any module's current recorded manifest,
@@ -0,0 +1,100 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-05
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md
+---
+
+# 218. A plan sends grants before code, rolls a module out one machine first, and a newer merge takes over an older plan
+
+## Context
+
+On 2026-10-05 the delivery path was watched through a day of merges, by several sessions at once. Three
+things went wrong, each recorded as an issue with its evidence.
+
+- **Code arrived before the right to use it** ([issue 249](../04-ISSUES/249-a-modules-new-state-is-refused-until-a-push-the-merge-did-not-make/00-report.md)).
+  A merge gave a module a new key-value state. The plan sent the new bundle to every machine, and only
+  then issued the memberships that grant the state. On three machines the module's new state was refused
+  for two minutes, until a push made by hand. The order is written into the code on purpose: memberships
+  "after the declaration, because the runtime it is for arrives with it". That reason holds only for a
+  first assignment, and even then a membership is kept on the bus for the runtime that connects later
+  ([ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)).
+- **No machine went first.** The module's upgrade policy sends one machine at a time, but a plan's rollout
+  ignores it and sends every machine running the module at once. One at a time also never waited for the
+  first machine to come up healthy: it stopped only if the publish itself failed. A change was therefore
+  everywhere before anything had seen it run.
+- **Plans for successive merges ran over each other** ([issue 254](../04-ISSUES/254-plans-for-successive-merges-run-over-each-other-and-one-was-left-open/00-report.md)).
+  Three merges to the catalogue within four minutes made three plans. Each sent the build agent to every
+  machine and asked for the same builds. One was still "building" hours later, with nothing left for it to
+  wait on. [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) decides one plan per merge
+  and says nothing about the next merge arriving while one is open. [Issue 219](../04-ISSUES/219-an-older-build-that-finishes-later-replaces-a-newer-one/00-report.md)
+  settled only which build's output wins.
+
+## Considered Options
+
+1. **Debounce merges:** wait a window before planning, so close merges make one plan. Rejected: it only
+   delays the overlap, does nothing for merges further apart than the window, and makes every merge slower.
+2. **Queue plans:** a new plan waits until the older one is done. Rejected: the older plan builds what the
+   newer merge is about to replace, then the newer one builds it again.
+3. **A newer merge's plan takes over the older plan's unfinished work, a plan rolls a module out one
+   machine first, and grants travel before code.** Chosen.
+
+## Decision
+
+**1. Grants before code.** Every send — a plan's rollout and a push alike — issues the memberships for
+the machines it is about to send to before it sends their declarations, after raising the buckets they
+name. When the composed list of bus users changes, the machine that holds the bus is sent first, because
+that list travels in its declaration. A membership that could not be issued fails the send, and the send
+is tried again. It is never reported as done "until the next push".
+
+**2. One machine first.** A plan rolls a module out according to the module's upgrade policy
+([ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) §3). Unless the policy says
+*together*:
+
+- the module is sent to one machine first, the first by name of the machines running it;
+- the rest are sent only once that machine has reported the new declaration applied and current;
+- a first machine that reports a failure, or does not report in time, stops the module's rollout there.
+  The plan names the machine and the reason, and the other machines keep what they ran.
+
+The plan records which machine went first, so a controller replaced mid-rollout resumes from there. A
+policy of *together* keeps today's behaviour.
+
+**3. A newer merge takes over an older plan.** When a merge into a repository's branch makes a plan,
+every open plan for the same repository and branch made before it is superseded, ordered by when each
+plan was made, never by commit:
+
+- the modules the older plan had not yet built join the newer plan's set, before its tiers are computed;
+- the older plan ends in a state of its own, *superseded*, naming the plan that took it over.
+
+Builds the older plan already asked for still finish and register; issue 219's ordering keeps the newer
+one current. A person can also close a plan that waits on nothing, by its id. The plan is marked closed
+by hand and never resumed.
+
+## Consequences
+
+- A module that gains a state, an event or a tool can use it from its first start on every machine.
+- A change reaches one machine before the rest. A change that breaks its first machine stops there, with
+  the reason in the plan.
+- Successive merges build each module once, for the newest commit. The build agent is sent to the
+  machines once per run of merges, not once per merge.
+- **What got harder:** a rollout takes one machine's report longer than before. A module that must change
+  everywhere at once says *together* in its policy. A plan's record now has a superseded state that
+  readers of the plans must know.
+
+## How it is checked
+
+| Rule | Checked by |
+|---|---|
+| grants before code | the controller's test: a send records memberships issued before any declaration; the machine holding the bus is sent first when the user list changes; a failed membership fails the send |
+| one machine first | the controller's test: with a one-at-a-time policy, one machine is sent, the rest only after its applied and current report; a failed first machine stops the module; *together* sends all at once |
+| a newer merge takes over | the controller's test: an older open plan for the same repository and branch is superseded, its unbuilt modules folded in; a plan for another repository is left alone; a superseded plan is not open |
+| live | the next merge to the catalogue that gives a module a new state: no refusal of that state on any machine, the first machine named in the plan, one plan open per repository |
+
+## References
+
+- [Issue 249](../04-ISSUES/249-a-modules-new-state-is-refused-until-a-push-the-merge-did-not-make/00-report.md), [issue 254](../04-ISSUES/254-plans-for-successive-merges-run-over-each-other-and-one-was-left-open/00-report.md)
+- [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) — plans and tiers, extended here
+- [ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md) — memberships, kept on the bus
+- [to-be 30](../03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md) — the design this amends
@@ -194,6 +194,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0207** — [A module depends on the node seats that apply its resources](0207-a-module-depends-on-the-node-seats-that-apply-its-resources.md)
 - **0210** — [A tool's configuration is its seat holder's, and every other module extends it through the seat](0210-a-tools-configuration-is-its-seat-holders-and-every-other-module-extends-it-through-the-seat.md)
 - **0212** — [A seat says what it receives, and the machine's hotkeys are a seat](0212-a-seat-says-what-it-receives-and-the-machines-hotkeys-are-a-seat.md)
+- **0218** — [A plan sends grants before code, rolls a module out one machine first, and a newer merge takes over an older plan](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md)

 ### Its tiers, from the bottom up

@@ -5,7 +5,7 @@ code:
  - mesh-controller cmd/mesh-builder
  - mesh-controller internal/builder
  - mesh-catalog modules/build-agent
-updated: 2026-10-04
+updated: 2026-10-05
 decisions:
  - 02-DECISIONS/0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md
  - 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
@@ -346,6 +346,21 @@ scheduled step with the server held still — which is what `while-stopped` exis
 keeps is still a manifest and so still referenced, and the dangerous flag is not needed once the
 mesh is the one deciding.

+**An archive is held by a manifest of its own** (2026-10-05,
+[issue 253](../../04-ISSUES/253-the-stores-collector-would-delete-every-archive-the-mesh-keeps/00-report.md)).
+The sentence above held for images and not for archives: an archive was published as a bare blob that no
+manifest names, and the store's collector keeps only what a manifest names, so a nightly collection
+would have removed every archive the mesh keeps, the current ones included. So:
+
+- an archive is published with a manifest that names it and nothing else, built from the archive's digest
+  and size alone so it can be computed again from the record;
+- the sweep makes sure every archive it keeps is held that way before it lets anything go, and lets go of
+  an archive by removing its manifest first;
+- the reference a machine fetches is unchanged.
+
+The collector runs as a dry run until the controller reports no kept archive unheld; only then does it
+collect for real.
+
 A machine behind by more than five builds of a module, recreating a container, cannot pull what it
 was running. It is already a machine the mesh reports as behind, and the answer is the current
 declaration.
@@ -2,8 +2,9 @@
 layer: to-be
 status: proposed
 code: []
-updated: 2026-10-01
+updated: 2026-10-05
 decisions:
+  - 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md
  - 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md
  - 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
  - 02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md
@@ -129,6 +130,25 @@ controller replaced mid-plan resumes from the store. `status` lists open plans a
 has waited too long. The transition discipline for breaking changes in the list above is still
 unwritten, and still the next thing.

+## How a plan sends (2026-10-05)
+
+Revision, [ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md). Three rules on how a plan delivers what it built.
+
+- **Grants travel before code.** A send issues the memberships for the machines it is about to send to
+  before their declarations. The machine holding the bus is sent first when the list of bus users changes.
+  A membership that could not be issued fails the send.
+- **One machine first.** Unless a module's upgrade policy says *together*, a plan sends it to one machine,
+  the first by name, and to the rest only once that machine reports the new declaration applied and
+  current. A first machine that fails stops the module's rollout there, with the reason in the plan.
+- **A newer merge takes over.** A merge's plan supersedes every older open plan for the same repository
+  and branch, and takes in the modules they had not yet built. A plan that waits on nothing can be closed
+  by hand, by its id.
+
+What a merge changed is read from the forge whole, page by page
+([issue 252](../../04-ISSUES/252-a-merges-changed-modules-were-read-wrong/00-report.md)). A changed path in a
+module directory the mesh does not hold yet is that module's own, not shared code, when its definition is
+among the changed paths.
+
 ## Why now, and why not yet

 **Why it matters:** self-update is the difference between a mesh a person maintains by typing
@@ -1,9 +1,9 @@
 ---
-status: open
+status: located
 opened: 2026-10-05
-located-in: []
+located-in: [mesh-controller cmd/mesh-controller]
 fixed-by:
-amended-design:
+amended-design: 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md
 ---

 # 249 — A module's new state is refused until a push the merge did not make
@@ -0,0 +1,20 @@
+# 249 — Diagnosis
+
+## 2026-10-05
+
+**Grants after code.** Both a plan's rollout and a push send every machine its declaration first, and
+issue the memberships afterwards. The order is written into the code on purpose, "because the runtime it is
+for arrives with it". That reason holds only for a first assignment, and a membership is kept on the bus for
+a runtime that connects later anyway. A membership that failed was only printed, and left "until the next
+push". The list of bus users travels in the declaration of the machine that holds the bus, which a module's
+rollout reaches only if that machine runs the module.
+
+**No machine first.** The module's upgrade policy sends one machine at a time, but a plan's rollout ignored
+it and sent every machine at once. One at a time did not wait for the first machine to come up either: it
+stopped only if the publish failed.
+
+**Not answered by the open decision on unseen changes** (a removal, a move or a replacement, shown
+before it takes effect). That decision leaves an add-only change alone on purpose, and a new state is one.
+
+**Decided** in [ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md):
+grants before code, and one machine first unless a module's policy says *together*.
@@ -0,0 +1,45 @@
+---
+status: located
+opened: 2026-10-05
+located-in: [mesh-catalog modules/gitea]
+fixed-by:
+amended-design:
+---
+
+# 250 — A merge made through the forge's tool is announced twice
+
+## What was observed
+
+The controller logged the record repository's merges twice, seconds apart, with the same commit, even with
+its event consumer freshly made ([issue 248](../248-the-controllers-event-consumer-replayed-a-week-and-held-every-merge-behind-it/00-report.md)).
+Counted over one day:
+
+- 8 of the record's merges were logged twice, against 4 once;
+- 10 of the catalogue's, against 7 once;
+- 2 of the controller's.
+
+A code repository's second line reads differently — "it changed nothing any module the mesh holds is built
+from" — so it was taken for a different message.
+
+## Diagnosis
+
+The forge's module announces a merge from two places in the same process:
+
+1. its merge tool, the moment it merges;
+2. the poll added for [issue 131](../131-nothing-tells-the-mesh-a-source-moved/00-report.md), which
+   announces every merged pull request it has not recorded as announced.
+
+The tool never records what it announced, so the poll announces it again 0.5 to 16 seconds later.
+Merges made in the forge's web interface or by a plain API call are seen by the poll alone, and those are
+the ones logged once. The module's own header comments still say merges are announced "from the tools …
+one process only".
+
+No harm was done this time, but only by luck. The second event is absorbed because the first one moved
+the controller's record of the source. A repository read only by packaging modules has no such record, so
+it would get a second plan. The record module synced twice for each merge.
+
+## Fix
+
+The poll is the only emitter: it sees every path and carries the clone address. The tool merges and
+answers the merge commit. A merge made through the tool is heard up to thirty seconds later, which the
+module already accepts ("an event a minute late is still an event").
@@ -0,0 +1,32 @@
+---
+status: located
+opened: 2026-10-05
+located-in: [mesh-catalog modules/records]
+fixed-by:
+amended-design:
+---
+
+# 251 — The record's checkout could not sync after it ran as another account
+
+## What was observed
+
+Every sync of the record module failed, on every merge and every timer: git refused the checkout as
+"dubious ownership". The record tools answered all the while, from the checkout as it last stood, and
+nothing said it was stale.
+
+## Diagnosis
+
+The module ran in a container, as the superuser, until its code moved into the machine's tool runtime
+([ADR 0198](../../02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)).
+The runtime launches it as the operator account. The checkout's git directory and 2 598 of its files still
+belonged to the superuser. Git refuses a repository owned by another user, and the operator account could
+change none of those files.
+
+The module also still declared that it needs a container runtime, a leftover of the same move.
+
+## Fix
+
+The module, ported to Go as part of the fix, clones into a directory of its own inside the one it is given:
+a directory it makes, and so owns. What the old layout left behind is removed where it is the module's.
+Where it is not, the module names it in its status, with the one command that deletes it. The
+container-runtime capability is dropped. A sync that fails is still said in the status, as before.
@@ -0,0 +1,34 @@
+---
+status: located
+opened: 2026-10-05
+located-in: [mesh-catalog modules/gitea, mesh-controller cmd/mesh-controller]
+fixed-by:
+amended-design: 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md
+---
+
+# 252 — A merge's changed modules were read wrong, in both directions
+
+## What was observed
+
+Each of three merges to the catalogue within four minutes planned 99 to 100 modules, and rebuilt modules
+they did not touch. The outputs were identical, so nothing was redeployed. The rebuilds cost the build
+machine minutes for every merge.
+
+## Diagnosis
+
+**Too many.** The controller treats a changed path outside every module directory it knows as shared code,
+and rebuilds every module built from the repository. A new module's directory, or one being removed,
+counts: the module is registered only after the merge is planned. Each of the three merges added or
+removed a module. The build agent, built from the same repository, then joins the set and becomes tier 0.
+
+**Too few.** The forge's module asked for a hundred changed files and was given fifty, the forge's page
+size, and reported the list as whole. A merge of 59 files reached the controller with 50. Had the full
+rebuild not hidden it, a module whose own files changed would have stayed unbuilt.
+
+## Fix
+
+- The forge's module reads every page of a pull request's files.
+- The controller counts a changed path in a sibling of known module directories as that module's own, not
+  shared, when that module's definition is among the changed paths. A sibling without a definition, such
+  as a shared library, still means everything, which is the safe direction. Root files still mean
+  everything.
@@ -0,0 +1,51 @@
+---
+status: located
+opened: 2026-10-05
+located-in: [mesh-controller internal/builder, mesh-controller internal/artifacts, mesh-catalog modules/distribution]
+fixed-by:
+amended-design: 03-DESIGN/01-to-be/18-building-a-module.md
+---
+
+# 253 — The store's collector would delete every archive the mesh keeps
+
+## What was observed
+
+The store's nightly collector ([ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md))
+was installed the same day, its first run due that night. Measured read-only beforehand:
+
+| | |
+|---|---|
+| blobs in the store | 8 186 |
+| blobs a manifest names, kept by the collector | 2 090 |
+| blobs it would delete | 6 096 |
+| repositories holding only archives, none named by any manifest | 105 |
+
+Among the archives it would delete were the current bundles of the agent module, the machine host, the
+controller and the tool runtime. Each was named by no manifest, though the controller's records keep them.
+
+## Why it matters
+
+Machines keep their unpacked copies, so nothing would have stopped at once. But any fresh fetch of an
+unchanged module would have failed: a machine joining, a reinstall, an apply that fetches again, the
+controller's own next rollout.
+
+## Diagnosis
+
+Images are pushed with manifests. Archives were published as bare blobs, which no manifest names. The
+store's stock collector marks only from manifests, so every bare blob is unmarked, kept or not. ADR 0189's
+sentence "what the mesh keeps is still a manifest in the store" was true of images only.
+
+Found alongside: the "five most recent builds" reason kept builds of modules the mesh no longer holds,
+forever.
+
+## Fix
+
+- **That night, before the first run:** the collector was changed to a dry run, in its module's
+  definition, and delivered.
+- **Then:**
+  - every archive is published with a manifest that holds it;
+  - the controller's sweep holds every kept archive before it lets anything go, which backfills those
+    already published;
+  - letting an archive go removes its manifest first;
+  - a forgotten module keeps nothing.
+- Real collection returns once the controller reports no kept archive unheld.
@@ -0,0 +1,32 @@
+---
+status: located
+opened: 2026-10-05
+located-in: [mesh-controller cmd/mesh-controller, mesh-controller internal/inventory]
+fixed-by:
+amended-design: 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md
+---
+
+# 254 — Plans for successive merges run over each other, and one was left open
+
+## What was observed
+
+Three merges to the catalogue within four minutes made three plans, and all three ran at once:
+
+- each sent the build agent to every machine;
+- each asked for the same tier of builds — one module was asked for 32 seconds apart by two plans;
+- the plan for the middle merge still showed "building" hours later, waiting on nothing.
+
+## Diagnosis
+
+A merge's plan is saved without looking at the open plans
+([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md): one plan per merge).
+Each open plan advances on its own. Nothing ends a plan whose work a newer merge has taken over, and
+nothing lets a person close a plan that waits on nothing.
+[Issue 219](../219-an-older-build-that-finishes-later-replaces-a-newer-one/00-report.md) settled only which
+build's output wins.
+
+## Fix
+
+[ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md)
+§3. A newer merge's plan supersedes the older open plans for the same repository and branch, and takes in
+the modules they had not built. A person can close a plan by its id.