Triage issues 279-298: resolve the fixed with replays R292 and R298
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered

292 and 298 resolve with replays now registered in mesh-lab. The other fixed
core issues say why no replay can be laid over the commit before their fix. The
rest are re-checked against main.
This commit is contained in:
jochen
2026-10-08 01:31:13 +02:00
parent 990414c9b8
commit 64e97d0de1
16 changed files with 108 additions and 24 deletions
@@ -61,3 +61,5 @@ manifest passed, and the failure showed only once applied.
account: is it a fact the mesh should know, or an access it only mounts?
- Separately: should one module's failed step fail every gate in the rollout on that node, and should a
build with nothing kept to put back be judged as a failed rollback?
Re-checked 2026-10-08: still holds — a declared directory's owner is still literal, and the media managers' recycle-bin folders on the catalogue's main are still the stopgap's mode 0777.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-06
located-in: [mesh-controller cmd/mesh-controller, mesh-controller internal/builder, mesh-controller internal/inventory]
fixed-by: mesh-controller PR #99
fixed-by: novox/mesh-controller PR #99, novox/mesh-controller PR #101
replay-none: fixed on 2026-10-06, before ADR 0237's replay rule began; the controller's fingerprint tests hold it, and a faithful replay would need the image builder's non-reproducible digests, which a laid-over test cannot raise
amended-design:
---
@@ -86,3 +87,7 @@ planning rule (issue 278's area) and is not done here.
| no bus step for an unchanged bus | the controller's test: the bus rebuilt from an unchanged source with a new digest holds no push to its machine, `bus upgrade` has nothing to do and takes no snapshot, and a real change to its source demands the planned step again |
| a plan sends nothing for an unchanged source | the controller's test: a plan's build made from the source every machine runs is marked sent with "no move", with no send and no gate; a changed source is sent to its first machine and gated |
| live | the next catalogue merge that rebuilds the bus without touching its directory demands no bus step; `bus` says every machine runs the build the mesh holds |
## Resolved — 2026-10-08
Re-checked against the controller's main. PR #99 (merged 2026-10-06) records each build's source fingerprint and treats a rebuild of an unchanged source as no move, so a bus rebuilt from an unchanged directory asks for no bus step. PR #101 (ADR 0238) closed the *left open*: a file at the repository's root rebuilds nothing. No bus step has been demanded for an unchanged bus since.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-06
located-in: [mesh-controller cmd/mesh-controller]
fixed-by: mesh-controller PR #100
fixed-by: novox/mesh-controller PR #100
replay-none: fixed on 2026-10-06, before ADR 0237's replay rule began; the controller's tier and gate tests in the fix hold it, and none was registered in the lab
amended-design:
---
@@ -88,3 +89,7 @@ module's first build on the machine.
| a machine unhealthy as a whole fails its send together | the controller's test: any other machine-level condition fails every module of the send with "as a whole", put back in one send |
| a send that changed nothing is no verdict | the controller's test: a module already carried to its first machine is left as it was, unmarked, with no rollback condition, and `plans retry` asks it again |
| live | the next merge that rebuilds a wide tier logs one "sent N module(s) to <machine> first in one send" line per first machine, and no gate fails for `core-behind` |
## Resolved — 2026-10-08
Re-checked against the controller's main. PR #100 (merged 2026-10-06) sends a tier to each machine once, judges it by one gate, reads a machine-level condition as the machine's, and gives no verdict on a send that changed nothing of a module. ADR 0240 and ADR 0241 build on that sorting of conditions. The *left open* items are not this symptom.
@@ -135,3 +135,5 @@ Where the old value went:
| `docker_events` never answers a secret an exec carried | Module tests: a broker's admin password after `-P` is redacted and named, and so are a client password after `createClient -p`, `setClientPassword`'s password, `redis-cli -a`, `PGPASSWORD=`, `--password=` and a URI password. A port, a path and a plain command are left as they are. |
| `docker_secrets_in_events` names, never quotes | Module test: the scan counts each exec once, names container, module, what it was and the program, and its answer carries no value. |
| keycloak's repair passes no password as an argument | Module test: the script contains no `--password "$…"`, no `--new-password` and no `-p "$…"`. |
Re-checked 2026-10-08: still holds in part — catalogue PR #100 (merged 2026-10-06) fixed every row marked fixed; on main `minio` still passes `--secret-key` to `mc admin user svcacct add` and `gitea`'s run-once step still passes `--password` to `gitea admin user create`. Whether the leaked broker admin value was rotated is not recorded here.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-controller cmd/mesh-controller, mesh-controller internal/facts, mesh-controller internal/builder]
fixed-by: mesh-controller PR #104
fixed-by: novox/mesh-controller PR #104, novox/mesh-controller PR #106
replay-none: the fix's own tests (TestIssue282… in the controller's merge_gate_test.go, named for the issue's number before renumbering) use what the fix added, and on 2026-10-08 mesh-lab's prover found they do not build at the commit before it, so they cannot be laid over it. A replay written only with the older gate would have to raise the gate's store for a module on the bus, and none was written.
amended-design:
---
@@ -77,3 +78,7 @@ and the facts package's `TestAWithheldPathStaysAPath` cover the path stand-ins.
- The snapshot does not say whether a machine holds a bus membership, so the gate composes none. It is
one resource, the same with the change and without.
- The snapshot's `sources` still name each repository by the forge's address and owner.
## Resolved — 2026-10-08
PR #104 (merged 2026-10-06) raises the mesh as it is in the gate and makes a baseline that does not compose an error, never a pass; PR #106 (merged 2026-10-07) makes the build seat call a gate that raised no machine an error whatever judged it. A verdict of "0 of N compose" can no longer read PASS.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-controller internal/builder, mesh-controller cmd/mesh-controller, mesh-host internal/bootstrap]
fixed-by: mesh-controller PR #104, mesh-host PR #46
fixed-by: novox/mesh-controller PR #104, novox/mesh-host PR #46
replay-none: the cause was the environment — another Go release's gofmt, siblings at another ref, another user — not a code path, so no commit-before/commit-after replay can hold it; the controller's check-here runs a check as the seat does instead
amended-design:
---
@@ -69,3 +70,7 @@ cover the summary and the refs.
heads beside it.
- The toolchain's Go is older than the agents'. Raising it is a change to the toolchain module, not to
the check.
## Resolved — 2026-10-08
PR #104 (merged 2026-10-06) adds `check-here`, sets `safe.directory` for a check's checkouts and summarises a failed script by what failed; node-engine PR #46 (merged 2026-10-07) lays the publishing test out as both releases' gofmt agree. The *left open* items (a delivery group's members' scripts, the toolchain's Go release) are not this symptom.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-tools node-tools/internal/console]
fixed-by: mesh-tools PR #19
fixed-by: novox/mesh-tools PR #19
replay-none: the fix's test, TestASeatAndAModuleOfOneNameAreEachReached in the console, calls the index the fix introduced and does not build at the commit before it (checked 2026-10-08). The lab's prover also cannot yet run a test in the node tools, whose Go module is a subdirectory of its repository.
amended-design:
---
@@ -47,3 +48,7 @@ announcement (the seat's holder is heard, which is why the overview names the ma
The mesh MCP server's `TestASeatAndAModuleOfOneNameAreEachReached` covers this: the seat listed with its verbs,
the module with its tools, and each address resolved to the seat or the module. The test launches no
bundle, so the repository's `merge-check.sh` runs it even where the mesh MCP server's other tests cannot run.
## Resolved — 2026-10-08
The mesh MCP server's PR #19 in mesh-tools (merged 2026-10-07) keys a seat's verbs and a module's tools apart. Live on 2026-10-08: the mesh MCP server describes `mesh-delivery.deliveries` with its arguments, and the delivery seat's verbs are called through it.
@@ -50,3 +50,5 @@ rewritten manifest, would judge a different mesh.
scrub that leaves composition unchanged. That is a question for the design, not the scrub.
- A login in a setting is caught only once the scrub knows the operator's logins on every machine,
not only the one the mesh records as its account.
Re-checked 2026-10-08: still holds in part — controller PR #105 (merged 2026-10-07) names repositories `owner/repository`; owners, manifests and a login in a setting still pass the scrub unchanged, as *left open* says.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-controller cmd/mesh-controller, mesh-controller internal/link, mesh-host internal/witness, mesh-tools node-tools/internal/bus]
fixed-by: mesh-controller PR #108, mesh-host PR #48, mesh-tools PR #20
fixed-by: novox/mesh-controller PR #108, novox/mesh-host PR #48, novox/mesh-tools PR #20
replay-none: the fix's tests (TestAVerbRunsWhileItsBuildIsMovedOrDeleted and its neighbours in the controller's selfexec_test.go) use the self-execution the fix added, and on 2026-10-08 mesh-lab's prover found they do not build at the commit before it. The incident needs a build moved or deleted under a running verb on a real machine, which the lab cannot raise.
amended-design:
---
@@ -88,3 +89,7 @@ starts), prove, retire. What changes is that every step is safe for a process th
- A process with no witness has its directory deleted in place when a new build arrives. None of those
run themselves today. A process that someday does would meet the same fault, and would need the same
retirement.
## Resolved — 2026-10-08
All three fixes merged on 2026-10-07: the controller runs its verbs from its own running image and refuses what it cannot run as a handover (mesh-controller PR #108); the node-engine keeps a replaced build reachable until no process runs from it (novox/`mesh-host` PR #48); the tool runner asks a handover refusal once more (mesh-tools PR #20). The design amendment named under *left open* (to-be 45 §8: a kept build is deleted only once no process runs from it) is not yet written.
@@ -1,8 +1,8 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-catalog modules/mesh-delivery]
fixed-by: mesh-catalog PR #101
fixed-by: novox/mesh-catalog PR #101
amended-design:
---
@@ -69,3 +69,7 @@ The table tests walk the new row, and the pairs the table does not hold are refu
the fix is rolled out, the stalled delivery has its check asked at the owner's first tick. The controller
journal then shows its verdict, and `show` moves from proposed to checked to ready without a person
acting.
## Resolved — 2026-10-08
Catalogue PR #101 (merged 2026-10-07) has the delivery owner ask the check of a proposed delivery nobody asked for, after its grace period, and counts the `proposed` bound from the ask.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-host internal/apply, mesh-host internal/store]
fixed-by: mesh-host PR #49
fixed-by: novox/mesh-host PR #49
replay-none: the node-engine's internal/apply/store_away_test.go uses the bounds the fix added, and on 2026-10-08 mesh-lab's prover found it does not build at the commit before. A faithful replay needs a store's maintenance window to meet an apply in flight on a real runtime, which the lab does not raise.
amended-design:
---
@@ -80,3 +81,7 @@ while the server is held still:
Live: the nightly collection on the store's machine no longer leaves `failed … connection refused` lines
in the node-engine's apply. A window that meets an apply in flight says "an apply is in flight … the
window opens once it ends".
## Resolved — 2026-10-08
Node-engine PR #49 (merged 2026-10-07) opens the store's window only once no apply is in flight, makes an apply that meets the window wait for it, and gives a store that does not answer one bounded retry per apply on every other machine.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-controller cmd/mesh-controller (handacts.go, signals.go S15)]
fixed-by: mesh-controller PR #109
fixed-by: novox/mesh-controller PR #109
replay: R292
amended-design: 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
---
@@ -50,3 +51,11 @@ mesh-controller tests:
Live: once the build is rolled out, `mesh.hand-acts.drill.healer-wanted` closes within a watchdog tick.
Later drills are recorded with `mesh-controller.drill`.
## Resolved — 2026-10-08
Fixed by novox/mesh-controller PR #109, merged on 2026-10-07. The replay is R292 in mesh-lab's replays
register (novox/mesh-lab PR #60). It is a test in the controller (novox/mesh-controller PR #129), written only
with what the controller had before the fix. It puts the two drills of 2026-10-07 in the hand-act log, as
`hand-act record` held them, and asks S15. On the commit before the fix S15 says "drill" was repaired by hand
2 times and wants a healer. On the fix it wants none. mesh-lab's prover says it is proved.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-catalog modules/gitea (pulls.ts, delivery.ts, index.ts)]
fixed-by: mesh-catalog PR #102
fixed-by: novox/mesh-catalog PR #102
replay-none: the prover in mesh-lab runs replays with go test only, and the fix is in the forge module's TypeScript. Its test/pulls.test.ts runs every verdict against every repository-check fact and asserts that none sets warning.
amended-design: 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
---
@@ -57,3 +58,10 @@ It asserts that none sets `warning`, and that each case above lands where this r
Live: the next pull request whose gate notes a wide rebuild shows `mesh/merge-gate` as success, and its
combined status is success.
## Resolved — 2026-10-08
Fixed by novox/mesh-catalog PR #102, merged on 2026-10-07. On main the forge module maps a note to
`success` ("pass, with a note:") and an error to `error`. It never sets `warning` on either merge-check
context, and its status tool refuses both. No replay is registered (see `replay-none:`). The module's own
tests are the check.
@@ -31,3 +31,5 @@ and can in another is a trap, and the failure names the wrong cause.
Whether `command` should accept an escape inside quoted values (as a shell does), refuse a line
with an unbalanced quote naming the position, or both; and whether every verb that a `command` line
reaches should also say which verb takes the value as its own field.
Re-checked 2026-10-08: still holds — the controller's `command` line splitter is unchanged since it was written: a single-quoted value takes no escape (as in a shell, `'\''` or double quotes with `\"` would pass one), and an odd quote is refused only as an unclosed quote, naming no position and no verb that takes the value as a field.
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-controller (cmd/mesh-controller release.go, plan.go, push.go), mesh-catalog modules/mailu]
fixed-by: mesh-controller PR #115; mesh-catalog PR #106
fixed-by: novox/mesh-controller PR #115, novox/mesh-catalog PR #106
replay-none: the fix lives inside the whole send (sendToEach). A replay written only with what the controller had before would have to drive a real send over a bus, with an identity store and grants, and the controller's tests raise neither for a send. The fix's own test, TestARecordedBuildIsCarriedOnlyByAPersonsPush, replays the case through the composition marker the fix introduced, so it cannot be laid over the commit before.
amended-design:
---
@@ -90,3 +91,10 @@ Located and fixed in pull requests, not merged. Decision: [ADR 0242](../../02-DE
A replay in mesh-lab's register is still owed before this resolves (ADR 0237). The controller test
above shows the replay's shape: a recorded provider and a rolled module on one machine, both
rebuilt, and the plan's gated send.
## Resolved — 2026-10-08
Fixed by novox/mesh-controller PR #115, merged on 2026-10-07 (its title says "hq issue 294", the number
before renumbering). Every send except a person's push composes a recorded module at the build its machine
runs, and a send says what it recreates. novox/mesh-catalog PR #106, merged after it, makes mail record
rather than roll out. No replay is registered (see `replay-none:`).
@@ -1,8 +1,9 @@
---
status: located
status: resolved
opened: 2026-10-07
located-in: [mesh-controller internal/catalogue (seats.go, verbs.go), mesh-catalog (the seats' holders)]
fixed-by: mesh-controller PR #117, mesh-controller PR #114
fixed-by: novox/mesh-controller PR #117, novox/mesh-controller PR #114
replay: R298
amended-design: 03-DESIGN/01-to-be/33-the-tools-the-mesh-answers.md
---
@@ -69,3 +70,14 @@ controller first promises the verb as optional, the holder serves it, and the co
The trail is in [01-diagnosis.md](01-diagnosis.md). This is a core issue: at resolution it names its
replay, or says in `replay-none:` why none is possible (ADR 0237).
## Resolved — 2026-10-08
Fixed by novox/mesh-controller PR #117, which added the optional verb, and PR #114, which keeps the mark when
a seat's row is read back. Both are merged, and the rule is in design 33 §7 by ADR 0246. The replay is R298
in mesh-lab's replays register (novox/mesh-lab PR #60). It is a test in the controller (novox/mesh-controller
PR #129), written only with what the catalogue package had before #117. It asserts that the delivery seat's
holder holds the seat both without `checks` and with it. On the commit before #117 the holder serving
`checks` is refused ("which that seat's protocol does not promise"). On #117 both hold the seat. mesh-lab's
prover says it is proved. The replay covers #117's half. #114's half, the mark read back from the store,
is held by that pull request's own tests.