Author SHA1 Message Date
mesh-admin ab1fbe3202 Merge pull request 'Issues 224, 248-252, 254, 255 resolved; 256 opened and resolved; 253 where it stands' (#104) from records/resolved-2026-10-05 into main 2026-10-05 16:36:27 +00:00
jochen 0fdcb15200 Issues 224, 248-252, 254, 255 resolved, 256 opened and resolved, 253 where it stands
Each resolved with the pull request that fixed it and what was seen live;
253 waits for real collection, the operator's word to give.
2026-10-05 18:36:19 +02:00
mesh-admin 91aa481269 Merge pull request 'Issue 255: the journal verb read nothing for a system service' (#103) from issue/255-the-journal-verb-read-nothing into main 2026-10-05 16:10:02 +00:00
jochen 0627b7f1f3 Issue 255: the journal verb read nothing for a system service 2026-10-05 18:09:44 +02:00
mesh-admin 049b50b66c Merge pull request 'Issue 248: delivery restored; a merge still heard twice has another cause' (#101) from issue/248-a-merge-heard-twice into main 2026-10-05 15:55:48 +00:00
mesh-admin 44808ac563 Merge pull request 'ADR 0218 and issues 250-254: delivery in order, and a store that keeps what it says' (#102) from records/delivery-and-store into main 2026-10-05 15:55:45 +00:00
jochen f000288147 ADR 0218 and issues 250-254: delivery in order, and a store that keeps what it says
ADR 0218: grants before code, one machine first, a newer merge takes over an
older plan (to-be 30 amended). Issues 250 (a merge announced twice), 251 (the
record's checkout owned by another account), 252 (a merge's changed modules
read wrong), 253 (the collector would delete every kept archive; to-be 18
amended, ADR 0189 corrected as a progressive insight), 254 (plans run over
each other); 249 located.
2026-10-05 17:51:03 +02:00
jochen 80aff0f457 Issue 248: delivery restored; a merge still heard twice has another cause 2026-10-05 17:30:18 +02:00
mesh-admin 400ed8d396 Merge pull request 'Issues 248, 249: the controller's event consumer replayed a week; a new state is refused until a push' (#100) from issues/248-249-delivery into main 2026-10-05 15:27:24 +00:00
jochen df2072aed5 Issues 248 and 249: the controller's event consumer replayed a week; a new state is refused until a push
248, located: a consumer made with the server's default replays the whole
stream, and the controller's held every new merge and build behind a week of
old ones. 249, open: a module's new state reaches its bundle before the
grants that let it use it.
2026-10-05 17:16:35 +02:00
mesh-admin ae58570209 Merge pull request 'Issue 247: a module cannot put the operator's account in a group' (#99) from issues/247-a-module-cannot-put-the-operator-in-a-group into main 2026-10-05 13:45:52 +00:00
mesh-admin 855c3f9b33 Merge pull request 'Issue 246: the console says a module runs nowhere when a runtime answers late' (#98) from issues/246-the-console-loses-the-largest-runtime into main 2026-10-05 13:45:49 +00:00
jochen f93b02900c Issue 247: a module cannot put the operator's account in a group 2026-10-05 15:45:24 +02:00
jochen 9306192f93 Issue 246: the console says a module runs nowhere when a runtime answers late 2026-10-05 15:40:14 +02:00
mesh-admin 8bd56a2820 Merge pull request 'Issue 243: resolved' (#87) from issue/243-resolved into main 2026-10-05 13:17:44 +00:00
mesh-admin 7497149eed Merge pull request 'Issue 245: behind is what a plan decided and has not yet done' (#96) from issues/245-the-plan-is-the-truth into main 2026-10-05 13:17:04 +00:00
jochen 1b9196b807 Issue 245: behind is what a plan decided and has not yet done (the operator's direction) 2026-10-05 15:16:51 +02:00
mesh-admin b3e7191ddc Merge pull request 'Issue 245: status calls a module behind when only its repository moved' (#95) from issues/245-behind-means-its-files-changed into main 2026-10-05 13:15:57 +00:00
jochen 87f80573b8 Issue 245: status calls a module behind when only its repository moved 2026-10-05 15:15:44 +02:00
mesh-admin 8aeef9969d Merge pull request 'ADR 0189 collects again; issue 244: a verb with an empty schema cannot be called' (#93) from decision/0189-collects-again into main 2026-10-05 12:38:40 +00:00
jochen 9197b99153 Issue 243: resolved by nodes applying any differing binding and the manager never repeating a generation 2026-10-05 10:03:49 +02:00
22 changed files with 789 additions and 6 deletions
@@ -60,6 +60,14 @@ with the server held still for its duration. Plain collection, not `--delete-unt
mesh keeps is still a manifest in the store, so it is still referenced, so its blobs stay — the
dangerous flag is not needed at all once the mesh is the one deciding.
> **Progressive insight — 2026-10-05.** "What the mesh keeps is still a manifest in the store" was true
> of images and false of archives: the builder published every archive as a bare blob no manifest names,
> and the store's collector keeps only what a manifest names. Its first night would have deleted every
> archive the mesh keeps ([issue 253](../04-ISSUES/253-the-stores-collector-would-delete-every-archive-the-mesh-keeps/00-report.md)).
> The decision stands — the mesh decides, the store reclaims with plain collection. What changes is how an
> archive is published: with a manifest that holds it, so the sentence becomes true of archives too. Until
> every kept archive is held, the collector runs as a dry run.
**3. What the mesh keeps, stated as three reasons rather than a number.** A digest is kept because:
- **a definition names it** — every artifact reference in any module's current recorded manifest,
@@ -0,0 +1,100 @@
---
topic: the mesh
status: accepted
date: 2026-10-05
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md
---
# 218. A plan sends grants before code, rolls a module out one machine first, and a newer merge takes over an older plan
## Context
On 2026-10-05 the delivery path was watched through a day of merges, by several sessions at once. Three
things went wrong, each recorded as an issue with its evidence.
- **Code arrived before the right to use it** ([issue 249](../04-ISSUES/249-a-modules-new-state-is-refused-until-a-push-the-merge-did-not-make/00-report.md)).
A merge gave a module a new key-value state. The plan sent the new bundle to every machine, and only
then issued the memberships that grant the state. On three machines the module's new state was refused
for two minutes, until a push made by hand. The order is written into the code on purpose: memberships
"after the declaration, because the runtime it is for arrives with it". That reason holds only for a
first assignment, and even then a membership is kept on the bus for the runtime that connects later
([ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)).
- **No machine went first.** The module's upgrade policy sends one machine at a time, but a plan's rollout
ignores it and sends every machine running the module at once. One at a time also never waited for the
first machine to come up healthy: it stopped only if the publish itself failed. A change was therefore
everywhere before anything had seen it run.
- **Plans for successive merges ran over each other** ([issue 254](../04-ISSUES/254-plans-for-successive-merges-run-over-each-other-and-one-was-left-open/00-report.md)).
Three merges to the catalogue within four minutes made three plans. Each sent the build agent to every
machine and asked for the same builds. One was still "building" hours later, with nothing left for it to
wait on. [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) decides one plan per merge
and says nothing about the next merge arriving while one is open. [Issue 219](../04-ISSUES/219-an-older-build-that-finishes-later-replaces-a-newer-one/00-report.md)
settled only which build's output wins.
## Considered Options
1. **Debounce merges:** wait a window before planning, so close merges make one plan. Rejected: it only
delays the overlap, does nothing for merges further apart than the window, and makes every merge slower.
2. **Queue plans:** a new plan waits until the older one is done. Rejected: the older plan builds what the
newer merge is about to replace, then the newer one builds it again.
3. **A newer merge's plan takes over the older plan's unfinished work, a plan rolls a module out one
machine first, and grants travel before code.** Chosen.
## Decision
**1. Grants before code.** Every send — a plan's rollout and a push alike — issues the memberships for
the machines it is about to send to before it sends their declarations, after raising the buckets they
name. When the composed list of bus users changes, the machine that holds the bus is sent first, because
that list travels in its declaration. A membership that could not be issued fails the send, and the send
is tried again. It is never reported as done "until the next push".
**2. One machine first.** A plan rolls a module out according to the module's upgrade policy
([ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) §3). Unless the policy says
*together*:
- the module is sent to one machine first, the first by name of the machines running it;
- the rest are sent only once that machine has reported the new declaration applied and current;
- a first machine that reports a failure, or does not report in time, stops the module's rollout there.
The plan names the machine and the reason, and the other machines keep what they ran.
The plan records which machine went first, so a controller replaced mid-rollout resumes from there. A
policy of *together* keeps today's behaviour.
**3. A newer merge takes over an older plan.** When a merge into a repository's branch makes a plan,
every open plan for the same repository and branch made before it is superseded, ordered by when each
plan was made, never by commit:
- the modules the older plan had not yet built join the newer plan's set, before its tiers are computed;
- the older plan ends in a state of its own, *superseded*, naming the plan that took it over.
Builds the older plan already asked for still finish and register; issue 219's ordering keeps the newer
one current. A person can also close a plan that waits on nothing, by its id. The plan is marked closed
by hand and never resumed.
## Consequences
- A module that gains a state, an event or a tool can use it from its first start on every machine.
- A change reaches one machine before the rest. A change that breaks its first machine stops there, with
the reason in the plan.
- Successive merges build each module once, for the newest commit. The build agent is sent to the
machines once per run of merges, not once per merge.
- **What got harder:** a rollout takes one machine's report longer than before. A module that must change
everywhere at once says *together* in its policy. A plan's record now has a superseded state that
readers of the plans must know.
## How it is checked
| Rule | Checked by |
|---|---|
| grants before code | the controller's test: a send records memberships issued before any declaration; the machine holding the bus is sent first when the user list changes; a failed membership fails the send |
| one machine first | the controller's test: with a one-at-a-time policy, one machine is sent, the rest only after its applied and current report; a failed first machine stops the module; *together* sends all at once |
| a newer merge takes over | the controller's test: an older open plan for the same repository and branch is superseded, its unbuilt modules folded in; a plan for another repository is left alone; a superseded plan is not open |
| live | the next merge to the catalogue that gives a module a new state: no refusal of that state on any machine, the first machine named in the plan, one plan open per repository |
## References
- [Issue 249](../04-ISSUES/249-a-modules-new-state-is-refused-until-a-push-the-merge-did-not-make/00-report.md), [issue 254](../04-ISSUES/254-plans-for-successive-merges-run-over-each-other-and-one-was-left-open/00-report.md)
- [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) — plans and tiers, extended here
- [ADR 0160](0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md) — memberships, kept on the bus
- [to-be 30](../03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md) — the design this amends
+1
View File
@@ -194,6 +194,7 @@ python3 00-META/checks/index.py fail if stale
- **0207** — [A module depends on the node seats that apply its resources](0207-a-module-depends-on-the-node-seats-that-apply-its-resources.md)
- **0210** — [A tool's configuration is its seat holder's, and every other module extends it through the seat](0210-a-tools-configuration-is-its-seat-holders-and-every-other-module-extends-it-through-the-seat.md)
- **0212** — [A seat says what it receives, and the machine's hotkeys are a seat](0212-a-seat-says-what-it-receives-and-the-machines-hotkeys-are-a-seat.md)
- **0218** — [A plan sends grants before code, rolls a module out one machine first, and a newer merge takes over an older plan](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md)
### Its tiers, from the bottom up
+16 -1
View File
@@ -5,7 +5,7 @@ code:
- mesh-controller cmd/mesh-builder
- mesh-controller internal/builder
- mesh-catalog modules/build-agent
updated: 2026-10-04
updated: 2026-10-05
decisions:
- 02-DECISIONS/0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md
- 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
@@ -346,6 +346,21 @@ scheduled step with the server held still — which is what `while-stopped` exis
keeps is still a manifest and so still referenced, and the dangerous flag is not needed once the
mesh is the one deciding.
**An archive is held by a manifest of its own** (2026-10-05,
[issue 253](../../04-ISSUES/253-the-stores-collector-would-delete-every-archive-the-mesh-keeps/00-report.md)).
The sentence above held for images and not for archives: an archive was published as a bare blob that no
manifest names, and the store's collector keeps only what a manifest names, so a nightly collection
would have removed every archive the mesh keeps, the current ones included. So:
- an archive is published with a manifest that names it and nothing else, built from the archive's digest
and size alone so it can be computed again from the record;
- the sweep makes sure every archive it keeps is held that way before it lets anything go, and lets go of
an archive by removing its manifest first;
- the reference a machine fetches is unchanged.
The collector runs as a dry run until the controller reports no kept archive unheld; only then does it
collect for real.
A machine behind by more than five builds of a module, recreating a container, cannot pull what it
was running. It is already a machine the mesh reports as behind, and the answer is the current
declaration.
@@ -2,8 +2,9 @@
layer: to-be
status: proposed
code: []
updated: 2026-10-01
updated: 2026-10-05
decisions:
- 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md
- 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md
- 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
- 02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md
@@ -129,6 +130,25 @@ controller replaced mid-plan resumes from the store. `status` lists open plans a
has waited too long. The transition discipline for breaking changes in the list above is still
unwritten, and still the next thing.
## How a plan sends (2026-10-05)
Revision, [ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md). Three rules on how a plan delivers what it built.
- **Grants travel before code.** A send issues the memberships for the machines it is about to send to
before their declarations. The machine holding the bus is sent first when the list of bus users changes.
A membership that could not be issued fails the send.
- **One machine first.** Unless a module's upgrade policy says *together*, a plan sends it to one machine,
the first by name, and to the rest only once that machine reports the new declaration applied and
current. A first machine that fails stops the module's rollout there, with the reason in the plan.
- **A newer merge takes over.** A merge's plan supersedes every older open plan for the same repository
and branch, and takes in the modules they had not yet built. A plan that waits on nothing can be closed
by hand, by its id.
What a merge changed is read from the forge whole, page by page
([issue 252](../../04-ISSUES/252-a-merges-changed-modules-were-read-wrong/00-report.md)). A changed path in a
module directory the mesh does not hold yet is that module's own, not shared code, when its definition is
among the changed paths.
## Why now, and why not yet
**Why it matters:** self-update is the difference between a mesh a person maintains by typing
@@ -1,8 +1,8 @@
---
status: open
status: resolved
opened: 2026-10-04
located-in: [mesh-host internal/apply/apply.go, mesh-host internal/apply/schedule.go]
fixed-by:
fixed-by: mesh-host PR #23
amended-design:
---
@@ -62,3 +62,9 @@ is a hole in something new rather than something that broke.
at the cost of a second source for "is this container meant to be running".
- Either way: should the *report* say a window is open, so a machine that looks half-stopped at
03:31 reads as working rather than broken?
## Resolved — 2026-10-05
An apply that arrives during a window now leaves the containers the window holds alone, and reports them `held-still`, naming the step. The first apply after the window converges them. The window is recorded under the host's state directory, the one place every applier on a machine shares: the daemon, a hand-run apply and the installer. It is released on every path, and lapses after six hours or when its process is gone. The machine's report lists open windows. Live on all four machines the same evening. Of the issue's two options, the narrower was taken: nothing blocks, so a push is never held for the length of a window.
Accepted and said in the change: an apply that inspected a container as running in the instant a window opens can still recreate it.
@@ -1,8 +1,8 @@
---
status: located
status: resolved
opened: 2026-10-05
located-in: [mesh-catalog modules/claude-code, mesh-catalog modules/claude-licence-manager]
fixed-by:
fixed-by: mesh-catalog PR #45
amended-design:
---
@@ -31,3 +31,9 @@
**Ruled out.** The bus delivered every binding: each machine that took the fourth binding did so in the
same second it was published. The licences themselves were sound. The remaining licence refreshed
on every attempt, and the second account's login was adopted from its first report.
## Resolved — 2026-10-05
Live on all four machines and the manager the same morning. After the restart every machine reported
the generation it held, and the manager's bindings matched them. The manager logged no failure while
moving its sequence past them.
@@ -0,0 +1,50 @@
---
status: open
opened: 2026-10-05
located-in: []
fixed-by:
amended-design:
---
# 245 — `status` calls a module behind when only its repository moved
## What was observed
2026-10-05. After three catalogue merges that changed 11 modules, `status` listed **69 modules
behind their source**, each as `holds <older commit>, source has <newer commit>`, with the advice
"`build --behind` builds them; `push --behind` sends them on".
Between the two commits, `git diff --name-only` shows changes under 11 module directories only. For
58 of the 69 modules listed, for example a Bluetooth module, the container runtime's, the forge's,
the package manager's and the bus's, the diff of the module's own path is empty. Their sources did
not change; only the repository's commit did.
The merges' plans were right: they rebuilt the changed modules and those that depend on them, by
tier. Only the report was wrong. An agent following the report's own advice ran `build --behind`,
which rebuilt all 69. The rebuilt bus module was rolled out, and its container was replaced on the
control node. Every node's runtime lost the bus for about a minute.
## Why it matters beyond this instance
"Behind" is the word a person and an agent act on, and `status` attaches a command to it. A module
is held to a commit of its repository, so after any merge almost every module of that repository
reads behind. The list then says nothing about what needs building: it hides the few modules that
really are behind among the many that are not, and it invites a rebuild of everything, which is not
a harmless act (above).
## The operator's direction (2026-10-05)
The only truth is the outcome of the build plan. A plan already decides, from a change, which modules
it affects: those whose sources changed and those that depend on them, tier by tier. "Behind" means
a module that a plan has decided to rebuild and has not yet rebuilt or rolled out, and nothing else.
No second comparison beside the plan is made, whether of commits, of folders or of files, because a
second answer to the same question is how the two came to disagree.
## Open questions
1. Where does `status` read "behind" from today, and what replaces it: the open plans' remaining
tiers?
2. What does a module's recorded commit mean once a plan that leaves it untouched has run? Does it
move forward, or does the record stop carrying a commit that only says when it was last built?
3. Should `build --behind` and `push --behind` take their lists from the same place, so that they can
never act on a module no plan named?
@@ -0,0 +1,65 @@
---
status: resolved
opened: 2026-10-05
located-in: [mesh-tools]
fixed-by: mesh-tools pull request 13 — discovery waits for every runtime that answered PING, names the ones it missed, and mesh_runtimes says who answered
amended-design:
---
# 246 — The console says a module runs nowhere when a runtime answers late
## What was observed
2026-10-05. On the laptop, the console answered `mesh_machine` for the laptop with no modules and no
seats, and a call to one of the laptop's modules with "nothing in the mesh is called slack". The
laptop's own runtime said at the same moment that it served 275 tools for 48 modules. A minute
earlier, a call to another laptop module had been answered with "it does not run on the laptop; it
runs on the workstation", and a retry of the same call worked. Nothing was logged anywhere.
An agent worked around it by calling the runtime's local MCP port directly, which gives the right
answer and bypasses everything the console stands for: one way in, one account, one record of what
was called. The operator asked for a tool instead.
## What was measured
A read-only probe on the laptop's own runtime credential timed the discovery answers over 25 rounds
against the live bus, whose round trip from the laptop was about 40 ms:
- The laptop's runtime answer was the largest on the mesh, at about 164 kB with 341 endpoints. That
is far below the bus's message limit, and it was never shortened.
- It arrived last in every round: a median of about 365 ms, and once 813 ms. The other runtimes
answered within 180 to 275 ms, and the controller within 50 ms.
- The console gathered discovery answers for a fixed 750 ms. Inside the console, the gather runs
beside two controller calls and about 570 kB of answers on the same link, and it is slower still
while a runtime re-serves after a restart.
## Root cause
Discovery decided who was there by who answered in a fixed window. A late answer was not a failure
to anyone, so nobody said it. The index simply lacked that runtime, and every answer built on the
index then stated as fact that the runtime's modules did not exist, or ran only elsewhere.
Ruled out by measurement or by reading the code: an answer too large for the bus, subscriptions lost
when the bus reconnects, the console not counting its own machine's answer, and the merge of two
answers dropping a machine.
## Resolution
- Discovery asks who is there (PING, a hundred bytes, answered at once) beside what each serves
(INFO). It waits at least the old window, and then up to five seconds for every instance that said
it is there, so a large answer is waited for and a quiet mesh costs nothing extra.
- A runtime that said it is there and did not say what it serves in time, or that answered recently
and not now, is named. While one is unheard, the console never says an address is missing or runs
elsewhere: it says which runtime was not heard, and where the controller's records place the module.
- A new console tool, `mesh_runtimes`, says for every runtime how long its answer took, how large it
was, how many modules and tools it announced, whether it was shortened, and when it was last heard,
and which runtimes or machines were not heard.
- An announcement still too large after its descriptions are cut to their first line now leaves the
descriptions out, and says so.
## How it is checked
The fix ships with tests against a real bus: a runtime that answers after the old window is found and
called (the same test fails with the fixed window), a runtime that answers PING and never INFO is
named, and a restarted runtime, which answers under a new instance, is not reported as missed. Live,
`mesh_runtimes` shows every machine's answer and its time.
@@ -0,0 +1,52 @@
---
status: open
opened: 2026-10-05
located-in: []
fixed-by:
amended-design:
---
# 247 — A module cannot put the operator's account in a group
## What was observed
2026-10-05. The module for a peripheral-lighting daemon was assigned to the laptop. Its package
installs the daemon and creates the daemon's group. The daemon then refuses to start: "User is not a
member of the openrazer group". The device files are the group's, so the daemon cannot reach the
devices.
The module's own check names the fix, which is to add the account to the group and log in again. No
module can declare that fix:
- The account is a `user` resource, and the login-shell module already declares it, to set its shell.
- A second module that declares the same account, only to add one group, is refused as a duplicate
name.
- There is no resource for one membership on its own. A whole-account declaration that lists groups
would also take from the account every group it does not list, including the operator's own.
So the step is done by hand, with `sudo`, outside the mesh, and nothing records why the account is in
the group.
## Why it matters beyond this instance
More modules need this than this one: input devices (`input`), serial ports (`uucp`), the container
runtime (`docker`), virtual machines (`libvirt`, `kvm`), and capture or scanner hardware. Each is a
fact a module knows and the operator's account needs. Today every one is a hand step that survives a
reinstall only by memory. A membership added by hand is also never taken away when the module that
needed it is unassigned.
## Open questions
1. Is a membership its own resource (account, group), held by the module that needs it and given
back on undeclare? Or is it a contribution to the account's holder, in the way ADR 0212 lets a
module contribute to a seat?
2. A membership takes effect at the next login. How does the module say so: a finding, or a
moment the power or session seat already knows?
3. What does undeclare do with a membership the account already had before any module declared it?
The host keeps what it found and gives it back, as it does with a whole file it wrote over.
## How it is checked
When fixed, assigning the lighting module to a machine whose account is not in the group puts the
account in the group, says that a new login is needed, and leaves the account's other groups as they
were. Unassigning it removes only a membership the module added.
@@ -0,0 +1,58 @@
---
status: resolved
opened: 2026-10-05
located-in: [mesh-controller internal/broker]
fixed-by: mesh-controller PR #51
amended-design:
---
# 248 — The controller's event consumer replayed a week, and held every new merge behind it
## What was observed
A merge to the catalogue at 15:17 never reached the controller. No plan was made and no build was asked
for it. Every machine kept running the build from before it. The merge just before, to the record, was
logged twice.
The bus showed why. The controller's durable consumer on the event stream was set to deliver
**everything the stream holds**, not from where it last was:
| | |
|---|---|
| the stream | a week of events, from message 1 514 to message 356 004 |
| the consumer delivered to | message 1 517, later 4 898 |
| acknowledged to | 0, later 1 569 |
| still to deliver | 6 955 merges and build outcomes, a week old |
| allowed outstanding | one at a time ([issue 175](../175-an-announcement-behind-a-long-build-comes-back/00-report.md)) |
So the controller was working through a week of past merges and build outcomes, one at a time, slowly. Every
new one — a merge, a build asked by hand — waited behind them. A build asked by hand finished on its
machine and was never registered. While this went on, the controller's client dropped messages
("slow consumer") several times, because heartbeats and reports share the loop with these events. From 15:17
its event loop did nothing more: no line logged, the one delivered event never acknowledged.
How the consumer came to deliver everything is not certain. The bus and the stores were rebuilt the night
before ([issue 241](../241-one-unreadable-grants-file-dropped-every-database-on-the-control-node/00-report.md)).
A consumer that is missing is made again by the controller's own assertion, and that made it with the
server's default, which is everything. [Issue 207](../207-a-re-made-worker-replayed-every-ask-the-stream-kept/00-report.md)
closed this for a consumer re-made because its type changed. It did not close it for one that is simply
not there.
## Why it matters
**Delivery stops, and nothing says so.** Status showed every plan done and no machine behind. The merge
that was missed is not "behind", because no plan was ever made for it. A replay of past merges can also
act on them again. Issue 207 records nine modules re-registered from the past the same way.
**The way out was a hand on the bus.** No verb resets a consumer. On the operator's explicit word, the
consumer was re-made from now with a one-off program run as the controller, otherwise unchanged. The
controller's loop still held the old event afterwards, so the new consumer's first delivery went
unacknowledged; the loop needs a restart of the controller to let go of it.
## Noticed alongside, not this issue
- Each merge to the catalogue planned 99 to 100 modules in two tiers and rebuilt modules it did not touch.
The output was byte-identical, so nothing was redeployed, but it costs minutes of the build machine
per merge.
- Three plans for three merges ran over each other. Each sent the build agent to every machine and asked
for the same builds. Nothing supersedes a plan for an older commit.
@@ -0,0 +1,54 @@
# 248 — Diagnosis
## 2026-10-05
1. A merge was not in the controller's log. The forge's own log showed nothing about delivering it, so the
question moved to the bus.
2. The bus's backlog tool named the controller's event consumer: 6 955 pending, redeliveries, one
unacknowledged. Its configuration was read from the server's monitoring endpoint: deliver policy
*all*, one outstanding, 30 seconds to acknowledge, five deliveries.
3. Sampled three times over a minute it did not move, and over the following hour it crawled forward
through week-old events. The controller's log held only its client's warnings: dropped messages,
and one refused reply.
4. The controller's code makes a missing consumer with the configuration it asserts, which sets no
deliver policy, so the server's default applies: everything. Issue 207's fix sets *from now* only on
the path where an existing consumer's type changes.
**Unblocked**, on the operator's explicit word: the consumer re-made from now with its configuration
otherwise unchanged (nothing pending afterwards). The controller's loop still held the old event, so
it takes a restart of the controller to let go of it; that restart waits for the operator's word.
**Fix** (mesh-controller, branch `fix/a-consumer-on-a-history-stream-starts-from-now`):
- a consumer may say it starts **from now** when it is made. The controller's event consumer does. A
consumer that exists keeps where it is. The server would refuse a changed start anyway.
- `broker consumer-reset <stream> <consumer>` re-makes a stuck consumer from now, its configuration
otherwise kept, and refuses a work queue, where what is pending is work. It is the person's act, said by
a command, rather than a one-off program.
Checked by a live test against a throwaway bus:
- made from now, a consumer holds none of the stream's past and does hold the next announcement;
- asserted again, it keeps its place;
- the default replays all of it;
- a reset leaves nothing pending and keeps every other setting;
- a work queue's consumer is refused.
**Not fixed here:** the client dropping messages while the loop acts on a long merge. Reports and
heartbeats are redelivered or replaced, so nothing is lost for good, but the loop holding everything while
it builds is the shape issue 175 already describes.
## 2026-10-05, after the restart
The consumer re-made from now held nothing, and the restarted controller acknowledged what it was handed.
Two merges made right after reached it within two seconds, and the fixed controller was built, delivered and
took over by its own plan within two minutes. The re-asked build of the agent module was registered and
reached all four machines.
**Not explained by this issue:** the record's merge was still logged twice, with the consumer fresh and
nothing replayed. The duplication has a cause of its own, still to be found. It may be that the forge
announces a merge on two paths, or that one event is handled twice.
## Resolved — 2026-10-05
The controller's event consumer is made from now when it is made, and `broker consumer-reset` re-makes a stuck one from now. Live: after the restart, merges reached the controller within seconds. Two tests main then failed, both skipped without a store, were fixed in mesh-controller PR #52. The merge heard twice was a separate cause: [issue 250](../250-a-merge-made-through-the-forges-tool-is-announced-twice/00-report.md).
@@ -0,0 +1,37 @@
---
status: resolved
opened: 2026-10-05
located-in: [mesh-controller cmd/mesh-controller]
fixed-by: mesh-controller PR #54
amended-design: 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md
---
# 249 — A module's new state is refused until a push the merge did not make
## What was observed
A merge gave the agent module a new state, a key-value bucket
([ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)).
The module's new bundle reached all four machines within a minute of the build, at once. The machines'
bus permissions did not include the new state until a push was made by hand afterwards.
On three machines the module's watch of the new state was refused for about two minutes: "claude-code keeps
and reads no state called config". It recovered only because the module asks again with a back-off, and
the hand-made pushes issued the permissions.
## Why it matters
**The code arrives before the right to use it.** A module that does not retry stays broken until someone
pushes. A module whose first act on start is to read its new state fails its start. Nothing in the plan
says the two must travel together.
**No machine went first.** The bundle reached every machine at the same moment. The rollout the operator
was told — one machine first, then the rest — could not be followed, because the merge had already
delivered it everywhere.
## Open questions
- Should a plan send a machine its membership, the grants that come with a module's new
declarations, in the same push as the bundle, and before it?
- Should a merge that changes a module's declarations (state, events, tools) be delivered to one machine
first, and to the rest only once that one reports it healthy?
@@ -0,0 +1,30 @@
# 249 — Diagnosis
## 2026-10-05
**Grants after code.** Both a plan's rollout and a push send every machine its declaration first, and
issue the memberships afterwards. The order is written into the code on purpose, "because the runtime it is
for arrives with it". That reason holds only for a first assignment, and a membership is kept on the bus for
a runtime that connects later anyway. A membership that failed was only printed, and left "until the next
push". The list of bus users travels in the declaration of the machine that holds the bus, which a module's
rollout reaches only if that machine runs the module.
**No machine first.** The module's upgrade policy sends one machine at a time, but a plan's rollout ignored
it and sent every machine at once. One at a time did not wait for the first machine to come up either: it
stopped only if the publish failed.
**Not answered by the open decision on unseen changes** (a removal, a move or a replacement, shown
before it takes effect). That decision leaves an add-only change alone on purpose, and a new state is one.
**Decided** in [ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md):
grants before code, and one machine first unless a module's policy says *together*.
## Resolved — 2026-10-05
[ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md), live the same evening:
- every push now issues memberships before it sends declarations;
- the bus's machine goes first when its list of users moved;
- a plan sent the build agent to one machine first and to the rest once that machine reported.
The first live rollout exposed a fault in the tier gate. The first machine's report, made between the two sends, was read as stale ([issue 256](../256-a-first-machines-report-read-as-stale-between-the-two-sends/00-report.md)).
@@ -0,0 +1,49 @@
---
status: resolved
opened: 2026-10-05
located-in: [mesh-catalog modules/gitea]
fixed-by: mesh-catalog PR #63, PR #66
amended-design:
---
# 250 — A merge made through the forge's tool is announced twice
## What was observed
The controller logged the record repository's merges twice, seconds apart, with the same commit, even with
its event consumer freshly made ([issue 248](../248-the-controllers-event-consumer-replayed-a-week-and-held-every-merge-behind-it/00-report.md)).
Counted over one day:
- 8 of the record's merges were logged twice, against 4 once;
- 10 of the catalogue's, against 7 once;
- 2 of the controller's.
A code repository's second line reads differently — "it changed nothing any module the mesh holds is built
from" — so it was taken for a different message.
## Diagnosis
The forge's module announces a merge from two places in the same process:
1. its merge tool, the moment it merges;
2. the poll added for [issue 131](../131-nothing-tells-the-mesh-a-source-moved/00-report.md), which
announces every merged pull request it has not recorded as announced.
The tool never records what it announced, so the poll announces it again 0.5 to 16 seconds later.
Merges made in the forge's web interface or by a plain API call are seen by the poll alone, and those are
the ones logged once. The module's own header comments still say merges are announced "from the tools …
one process only".
No harm was done this time, but only by luck. The second event is absorbed because the first one moved
the controller's record of the source. A repository read only by packaging modules has no such record, so
it would get a second plan. The record module synced twice for each merge.
## Fix
The poll is the only emitter: it sees every path and carries the clone address. The tool merges and
answers the merge commit. A merge made through the tool is heard up to thirty seconds later, which the
module already accepts ("an event a minute late is still an event").
## Resolved — 2026-10-05
The forge module's poll is the only announcer of a merge. It asks only the repositories that moved since its last look, and one pass at a time. Live: three merges were each heard once, and a later merge was planned within a minute.
@@ -0,0 +1,36 @@
---
status: resolved
opened: 2026-10-05
located-in: [mesh-catalog modules/records]
fixed-by: mesh-catalog PR #64
amended-design:
---
# 251 — The record's checkout could not sync after it ran as another account
## What was observed
Every sync of the record module failed, on every merge and every timer: git refused the checkout as
"dubious ownership". The record tools answered all the while, from the checkout as it last stood, and
nothing said it was stale.
## Diagnosis
The module ran in a container, as the superuser, until its code moved into the machine's tool runtime
([ADR 0198](../../02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)).
The runtime launches it as the operator account. The checkout's git directory and 2 598 of its files still
belonged to the superuser. Git refuses a repository owned by another user, and the operator account could
change none of those files.
The module also still declared that it needs a container runtime, a leftover of the same move.
## Fix
The module, ported to Go as part of the fix, clones into a directory of its own inside the one it is given:
a directory it makes, and so owns. What the old layout left behind is removed where it is the module's.
Where it is not, the module names it in its status, with the one command that deletes it. The
container-runtime capability is dropped. A sync that fails is still said in the status, as before.
## Resolved — 2026-10-05
The record module, ported to Go, clones into a directory it makes and owns. Live: it synced to the newest commit, with 692 documents. It names seven leftovers of the old checkout that it cannot remove, with the command that removes them; that is the operator's act.
@@ -0,0 +1,38 @@
---
status: resolved
opened: 2026-10-05
located-in: [mesh-catalog modules/gitea, mesh-controller cmd/mesh-controller]
fixed-by: mesh-catalog PR #63, mesh-controller PR #54
amended-design: 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md
---
# 252 — A merge's changed modules were read wrong, in both directions
## What was observed
Each of three merges to the catalogue within four minutes planned 99 to 100 modules, and rebuilt modules
they did not touch. The outputs were identical, so nothing was redeployed. The rebuilds cost the build
machine minutes for every merge.
## Diagnosis
**Too many.** The controller treats a changed path outside every module directory it knows as shared code,
and rebuilds every module built from the repository. A new module's directory, or one being removed,
counts: the module is registered only after the merge is planned. Each of the three merges added or
removed a module. The build agent, built from the same repository, then joins the set and becomes tier 0.
**Too few.** The forge's module asked for a hundred changed files and was given fifty, the forge's page
size, and reported the list as whole. A merge of 59 files reached the controller with 50. Had the full
rebuild not hidden it, a module whose own files changed would have stayed unbuilt.
## Fix
- The forge's module reads every page of a pull request's files.
- The controller counts a changed path in a sibling of known module directories as that module's own, not
shared, when that module's definition is among the changed paths. A sibling without a definition, such
as a shared library, still means everything, which is the safe direction. Root files still mean
everything.
## Resolved — 2026-10-05
The forge's module reads every page of a pull request's files. The controller counts a new module's own directory as that module's when its definition is among the changed paths. Live: catalogue merges planned one module each.
@@ -0,0 +1,55 @@
---
status: located
opened: 2026-10-05
located-in: [mesh-controller internal/builder, mesh-controller internal/artifacts, mesh-catalog modules/distribution]
fixed-by: mesh-controller PR #53, mesh-host PR #23, mesh-catalog PR #62
amended-design: 03-DESIGN/01-to-be/18-building-a-module.md
---
# 253 — The store's collector would delete every archive the mesh keeps
## What was observed
The store's nightly collector ([ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md))
was installed the same day, its first run due that night. Measured read-only beforehand:
| | |
|---|---|
| blobs in the store | 8 186 |
| blobs a manifest names, kept by the collector | 2 090 |
| blobs it would delete | 6 096 |
| repositories holding only archives, none named by any manifest | 105 |
Among the archives it would delete were the current bundles of the agent module, the machine host, the
controller and the tool runtime. Each was named by no manifest, though the controller's records keep them.
## Why it matters
Machines keep their unpacked copies, so nothing would have stopped at once. But any fresh fetch of an
unchanged module would have failed: a machine joining, a reinstall, an apply that fetches again, the
controller's own next rollout.
## Diagnosis
Images are pushed with manifests. Archives were published as bare blobs, which no manifest names. The
store's stock collector marks only from manifests, so every bare blob is unmarked, kept or not. ADR 0189's
sentence "what the mesh keeps is still a manifest in the store" was true of images only.
Found alongside: the "five most recent builds" reason kept builds of modules the mesh no longer holds,
forever.
## Fix
- **That night, before the first run:** the collector was changed to a dry run, in its module's
definition, and delivered.
- **Then:**
- every archive is published with a manifest that holds it;
- the controller's sweep holds every kept archive before it lets anything go, which backfills those
already published;
- letting an archive go removes its manifest first;
- a forgotten module keeps nothing.
- Real collection returns once the controller reports no kept archive unheld.
## Where it stands — 2026-10-05
Every kept archive is held: the controller's collection command reports 134 of 134 held, none missing. The window the collector needs is no longer reopened by an apply ([issue 224](../224-an-apply-reopens-a-maintenance-window-by-recreating-what-it-held-still/00-report.md)). The collector still runs as a dry run. Turning it to real collection deletes the layers nothing keeps, which is the operator's word to give; this issue resolves when that change lands.
@@ -0,0 +1,36 @@
---
status: resolved
opened: 2026-10-05
located-in: [mesh-controller cmd/mesh-controller, mesh-controller internal/inventory]
fixed-by: mesh-controller PR #54
amended-design: 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md
---
# 254 — Plans for successive merges run over each other, and one was left open
## What was observed
Three merges to the catalogue within four minutes made three plans, and all three ran at once:
- each sent the build agent to every machine;
- each asked for the same tier of builds — one module was asked for 32 seconds apart by two plans;
- the plan for the middle merge still showed "building" hours later, waiting on nothing.
## Diagnosis
A merge's plan is saved without looking at the open plans
([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md): one plan per merge).
Each open plan advances on its own. Nothing ends a plan whose work a newer merge has taken over, and
nothing lets a person close a plan that waits on nothing.
[Issue 219](../219-an-older-build-that-finishes-later-replaces-a-newer-one/00-report.md) settled only which
build's output wins.
## Fix
[ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md)
§3. A newer merge's plan supersedes the older open plans for the same repository and branch, and takes in
the modules they had not built. A person can close a plan by its id.
## Resolved — 2026-10-05
A newer merge's plan supersedes the older open plans of its repository and branch, and a person can close a plan by its id. Live: the plan left open since the afternoon was closed by hand, and its note said what it had built and never sent. A merge made while the controller's own plan was open took it over: the older plan reads "superseded at tier 0 by" the newer, which planned its three modules again.
@@ -0,0 +1,36 @@
---
status: resolved
opened: 2026-10-05
located-in: [mesh-catalog modules/systemd]
fixed-by: mesh-catalog PR #65
amended-design:
---
# 255 — The journal verb read nothing for a system service
## What was observed
The service manager seat's `journal` verb answered "-- No entries --" for the controller's service on the
control node, while the service was logging steadily. Twice in one day a session reading the controller's
log fell back to a shell on the machine: the verb that exists for exactly that question answered nothing.
On one machine of four the same verb did answer: the only one whose operator account is in the journal's
group.
## Diagnosis
The module runs as the operator account and escalates the five acts on the system manager with `sudo -n`.
It ran `journalctl` unescalated. journalctl shows an account outside the journal's group only that
account's own entries, and says "-- No entries --" for everything else. That reads as a quiet service, not
as a refusal.
The mesh grants the operator account passwordless escalation on every machine through its own drop-in, as
the sudo module's check confirms.
## Fix
A read of the system journal escalates like an act does. The account's own journal, in the user scope,
does not. The module was ported to Go as part of the fix, its tests with it.
## Resolved — 2026-10-05
The journal verb reads a system unit's journal escalated, in the module ported to Go. Live: the controller's and the host's journals read through the verb on the control node.
@@ -0,0 +1,31 @@
---
status: resolved
opened: 2026-10-05
located-in: [mesh-controller cmd/mesh-controller]
fixed-by: mesh-controller PR #55
amended-design:
---
# 256 — A first machine's report read as stale between the two sends
## What was observed
The first plan to roll out under [ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md)
sent the build agent to one machine first, and to the other two once that machine reported it applied, 27
seconds later. All three applied it within seconds. The plan then said, for seven minutes, that it was
"waiting for build-agent" on the first machine to be applied. It went on only when that machine happened to
report again for another reason.
## Diagnosis
The tier gate asked every machine for a report made after the module's last send. With one machine first,
a module is sent twice, and the second send is the later one. The first machine's report came between the
two sends, so it read as older than the build. The plan waited for that machine's next report, one report
cycle, and never knew why.
## Fix
The gate judges each machine from its own send: the first machine from the first send, the rest from the
second. A plan's wait is also printed to the second rather than the minute, where it had read "0s", and the
first send is printed in the machine's own time rather than in UTC. Checked by the controller's test: a first
machine's report between the two sends opens the gate, and a report from before its send does not.