Keep a condition's looks in the store, so a look that changes nothing writes no bucket (issue 420) #249

Merged
mesh-admin merged 1 commits from fix/420-conditions-bucket-on-transition into main 2026-10-12 10:27:26 +00:00
Contributor

Fixes issue 420 (the conditions bucket is rewritten on every observation).

Why. Every look rewrote a condition's value in mesh-controller_conditions, because the value carried the count of observations, the last observed time and the newest evidence. Measured on 2026-10-10: 131 647 writes in four days for twelve open conditions (mesh.hand-acts.disk-load.healer-wanted alone 7 003). That breaks novox/hq ADR 0289 §1 (a bucket is written only when what it describes changed) and §4 (no idle-changing fields). The decision on the issue (2026-10-11, shanks session): the bucket holds what changes on a transition; the looks live in the controller's store and the verb and the events still carry them.

What changes

  • internal/conditions/looks.go: Looks (observations, last-observed, evidence) and a LookStore. BucketValue writes a condition without its looks, keys in one order.
  • Keeper.Observe compares the new bucket value with the stored bytes and writes only when they differ; a fresh look is kept in the LookStore after the bucket write (so a lost compare-and-set does not double-count). Clear forgets the looks. Every read (Open, Get, Read/ReadWith, ReadOne/ReadOneWith) and every event puts the looks back.
  • A condition written before this (its value still carries the looks) keeps its count: the first look counts on from the value and rewrites it once without them.
  • Looks that cannot be read or kept never stop a raise (a store going away is exactly what must still be raised): the keeper says it, does not write over the count, and the condition carries looks-unread with why, never a count of none.
  • internal/inventory: migration 0095 condition_looks (key, observations, last_observed, evidence jsonb, epoch, kept_at); written under the lease (actingEpoch), forgotten on a clear. There is no consolidated schema in this repository; migrations are the schema.
  • cmd/mesh-controller/condition_looks.go: the store adapter. The serving controller and doctor run locally use it; the conditions verb, which runs as a process of its own, opens the store to read the looks (looksHere), or says them unread.
  • Writers table line says where the looks are kept.

Readers. The messenger reads the conditions verb answer and the condition events, not the bucket (checked in mesh-catalog main); both still carry observations, last-observed and evidence. No other reader of the bucket was found. The hq side: novox/hq branch fix/420-condition-looks-beside-the-bucket amends to-be 45 §2's field table.

Tests (each fails without the fix: replacing the compare with an unconditional write fails all three)

  • TestALookThatChangesNothingWritesNothingToTheBucket: five looks write the bucket once; the verb reads 5 observations and 5 evidence lines; the bucket's value holds no look field; a severity change writes once more and its event carries the looks; the clearing forgets them.
  • TestAConditionWrittenWithItsLooksMovesThemOutOnce: an old-shaped value with 7 003 observations is rewritten once and counts on to 7 006.
  • TestLooksThatCannotBeKeptAreSaidNotCountedAsNone: with the store away the condition is still raised and says looks-unread.
  • TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds (store + real bus): four looks leave the bucket stream at sequence 1; conditions show and the list, as a process other than the serving controller, read 4 observations from the store.
  • TestAConditionsLooksAreKeptBesideTheBucket (store): kept under the lease only, replaced, listed, forgotten.
  • Two existing tests adjusted to the new rule: the race test now races a write that changes severity (an unchanged look no longer writes, so there is nothing to race); the bus test uses a wrong revision rather than revision-1, which is 0 now that the second look writes nothing.
  • Both store tests are listed in testdata/store-tests. sh merge-check.sh passed locally with a throwaway PostgreSQL 18.6, race on, every listed store test PASS.

Reviewers' checklist

  • A look that changes nothing makes no bucket write (bytes.Equal in Observe), and a lost compare-and-set does not count a look twice (looks kept after the write).
  • Every path that writes the bucket uses BucketValue (raise, observe, amend/tried/escalate, silence, end of silence).
  • The events and the verb still carry observations, last-observed and evidence; nothing reads them from the bucket.
  • A looks store away never blocks a raise and never reads as zero.
  • Migration 0095 is the next number on main; no other open PR here adds one (#247, #246, #203, #202, #188, #182 checked).

Overlap. Does not touch any file #247 changes (walk_said.go, walksaid.go, plans.go, delivery.go, release_plan.go). A controller rollout: merge only while mesh-controller.plans says no plan is mid-gate, and not while #247 is mid-gate.

By fix-420@g14. Not merged; waits for review.

Fixes issue 420 (the conditions bucket is rewritten on every observation). **Why.** Every look rewrote a condition's value in `mesh-controller_conditions`, because the value carried the count of observations, the last observed time and the newest evidence. Measured on 2026-10-10: 131 647 writes in four days for twelve open conditions (`mesh.hand-acts.disk-load.healer-wanted` alone 7 003). That breaks novox/hq ADR 0289 §1 (a bucket is written only when what it describes changed) and §4 (no idle-changing fields). The decision on the issue (2026-10-11, shanks session): the bucket holds what changes on a transition; the looks live in the controller's store and the verb and the events still carry them. **What changes** - `internal/conditions/looks.go`: `Looks` (observations, last-observed, evidence) and a `LookStore`. `BucketValue` writes a condition without its looks, keys in one order. - `Keeper.Observe` compares the new bucket value with the stored bytes and **writes only when they differ**; a fresh look is kept in the `LookStore` after the bucket write (so a lost compare-and-set does not double-count). Clear forgets the looks. Every read (`Open`, `Get`, `Read`/`ReadWith`, `ReadOne`/`ReadOneWith`) and every event puts the looks back. - A condition written before this (its value still carries the looks) keeps its count: the first look counts on from the value and rewrites it once without them. - Looks that cannot be read or kept never stop a raise (a store going away is exactly what must still be raised): the keeper says it, does not write over the count, and the condition carries `looks-unread` with why, never a count of none. - `internal/inventory`: migration **0095** `condition_looks` (key, observations, last_observed, evidence jsonb, epoch, kept_at); written under the lease (`actingEpoch`), forgotten on a clear. There is no consolidated schema in this repository; migrations are the schema. - `cmd/mesh-controller/condition_looks.go`: the store adapter. The serving controller and `doctor` run locally use it; the `conditions` verb, which runs as a process of its own, opens the store to read the looks (`looksHere`), or says them unread. - Writers table line says where the looks are kept. **Readers.** The messenger reads the `conditions` verb answer and the condition events, not the bucket (checked in mesh-catalog main); both still carry `observations`, `last-observed` and `evidence`. No other reader of the bucket was found. The hq side: novox/hq branch `fix/420-condition-looks-beside-the-bucket` amends to-be 45 §2's field table. **Tests** (each fails without the fix: replacing the compare with an unconditional write fails all three) - `TestALookThatChangesNothingWritesNothingToTheBucket`: five looks write the bucket once; the verb reads 5 observations and 5 evidence lines; the bucket's value holds no look field; a severity change writes once more and its event carries the looks; the clearing forgets them. - `TestAConditionWrittenWithItsLooksMovesThemOutOnce`: an old-shaped value with 7 003 observations is rewritten once and counts on to 7 006. - `TestLooksThatCannotBeKeptAreSaidNotCountedAsNone`: with the store away the condition is still raised and says `looks-unread`. - `TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds` (store + real bus): four looks leave the bucket stream at sequence 1; `conditions show` and the list, as a process other than the serving controller, read 4 observations from the store. - `TestAConditionsLooksAreKeptBesideTheBucket` (store): kept under the lease only, replaced, listed, forgotten. - Two existing tests adjusted to the new rule: the race test now races a write that changes severity (an unchanged look no longer writes, so there is nothing to race); the bus test uses a wrong revision rather than revision-1, which is 0 now that the second look writes nothing. - Both store tests are listed in `testdata/store-tests`. `sh merge-check.sh` passed locally with a throwaway PostgreSQL 18.6, race on, every listed store test PASS. **Reviewers' checklist** - [ ] A look that changes nothing makes no bucket write (`bytes.Equal` in `Observe`), and a lost compare-and-set does not count a look twice (looks kept after the write). - [ ] Every path that writes the bucket uses `BucketValue` (raise, observe, amend/tried/escalate, silence, end of silence). - [ ] The events and the verb still carry observations, last-observed and evidence; nothing reads them from the bucket. - [ ] A looks store away never blocks a raise and never reads as zero. - [ ] Migration 0095 is the next number on main; no other open PR here adds one (#247, #246, #203, #202, #188, #182 checked). **Overlap.** Does not touch any file #247 changes (`walk_said.go`, `walksaid.go`, `plans.go`, `delivery.go`, `release_plan.go`). A controller rollout: merge only while `mesh-controller.plans` says no plan is mid-gate, and not while #247 is mid-gate. By fix-420@g14. Not merged; waits for review.
mesh-admin added 1 commit 2026-10-12 10:12:14 +00:00
Keep a condition's looks in the store, so a look that changes nothing writes no bucket (issue 420)
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
dcdafd37f0
Every look rewrote the condition's value in the conditions bucket, because the
value carried the count of looks, the last look and the newest evidence: 131 647
writes in four days for twelve open conditions, against ADR 0289's rule that a
bucket is written only when what it describes changed. The looks now live in the
controller's store (migration 0095, condition_looks), the bucket holds what a
condition says and is written only when that changes, and the conditions verb and
the condition events put the looks back.
Author
Contributor

Delivery novox/mesh-controller@dcdafd37f008 — delivered since 2026-10-12T10:41:33Z

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
  • grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine

Machines

  • mesh-controller on novox: passed — healthy 3 times over 3m39s
  • mesh-controller on the rest running it: sent
  • build-agent on ace: passed — healthy 1 times over 1m33s
  • build-agent on the rest running it: sent
  • route-proxy on ace: passed — healthy 1 times over 1m39s
  • route-proxy on the rest running it: sent

Transitions

  • 2026-10-12 10:12 (new) → proposed (announced): novox/mesh-controller#249's head announced
  • 2026-10-12 10:18 proposed → checked (checked): the verdict names this commit
  • 2026-10-12 10:18 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
  • 2026-10-12 10:29 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
  • 2026-10-12 10:29 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit
  • 2026-10-12 10:41 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-controller@dcdafd37f008` — **delivered** since 2026-10-12T10:41:33Z **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine **Machines** - mesh-controller on novox: passed — healthy 3 times over 3m39s - mesh-controller on the rest running it: sent - build-agent on ace: passed — healthy 1 times over 1m33s - build-agent on the rest running it: sent - route-proxy on ace: passed — healthy 1 times over 1m39s - route-proxy on the rest running it: sent **Transitions** - 2026-10-12 10:12 (new) → proposed (announced): novox/mesh-controller#249's head announced - 2026-10-12 10:18 proposed → checked (checked): the verdict names this commit - 2026-10-12 10:18 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-12 10:29 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-12 10:29 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit - 2026-10-12 10:41 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
Author
Contributor

Merge check at dcdafd37

  • mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
  • mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

  • tier 0: mesh-controller
  • tier 1: build-agent
  • tier 2: route-proxy
  • ace: receives build-agent, route-proxy
  • g14: receives build-agent
  • novox: receives mesh-controller, build-agent, route-proxy
  • shanks: receives build-agent
  • build-agent keeps data (ADR 0233): what it holds is backed up before it moves
  • route-proxy provides route: its consumers are sent again after it
  • route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
  • grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine
  • read by no module's build: cmd/mesh-controller/condition_looks_test.go, cmd/mesh-controller/replay327_test.go, internal/conditions/bus_test.go, internal/conditions/looks_test.go, internal/conditions/store_test.go, internal/inventory/looks_test.go, testdata/store-tests

Run by the build seat on ace as build-1791799936294115889 (builds --log build-1791799936294115889).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 57 tool(s); WARNING: merge-window ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-12T10:08:43Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-12T10:08:43Z

modules the change moves:
  - mesh-controller (changed)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy

notes:
  - cmd/mesh-controller/condition_looks_test.go, cmd/mesh-controller/replay327_test.go, internal/conditions/bus_test.go, internal/conditions/looks_test.go, internal/conditions/store_test.go, internal/inventory/looks_test.go and testdata/store-tests read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.038s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	225.308s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.025s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.034s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.028s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.143s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.071s
?   	github.com/novox/mesh-controller/internal/beside	[no test files]
ok  	github.com/novox/mesh-controller/internal/broker	4.133s
ok  	github.com/novox/mesh-controller/internal/builder	16.061s
ok  	github.com/novox/mesh-controller/internal/catalogue	7.809s
ok  	github.com/novox/mesh-controller/internal/conditions	2.661s
ok  	github.com/novox/mesh-controller/internal/envfile	1.071s
ok  	github.com/novox/mesh-controller/internal/facts	1.069s
ok  	github.com/novox/mesh-controller/internal/identity	4.682s
ok  	github.com/novox/mesh-controller/internal/inventory	87.982s
ok  	github.com/novox/mesh-controller/internal/lease	17.735s
ok  	github.com/novox/mesh-controller/internal/licences	4.287s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.087s
ok  	github.com/novox/mesh-controller/internal/link	59.428s
ok  	github.com/novox/mesh-controller/internal/lint	2.355s
ok  	github.com/novox/mesh-controller/internal/outward	1.090s
ok  	github.com/novox/mesh-controller/internal/overlay	1.081s
ok  	github.com/novox/mesh-controller/internal/secrets	1.486s
ok  	github.com/novox/mesh-controller/internal/store	3.502s
ok  	github.com/novox/mesh-controller/internal/testbus	1.257s
ok  	github.com/novox/mesh-controller/internal/token	1.020s
store tests, each run against the store:
  ./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeWalksTheOthersAgain (0.39s)
  ./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeEndsTheMergesThatContainIt (0.45s)
  ./cmd/mesh-controller --- PASS: TestAStopWalksTheOthersAgainInTheOpenBatch (0.37s)
  ./cmd/mesh-controller --- PASS: TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt (0.40s)
  ./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeLeavesItOutOfABatch (0.38s)
  ./cmd/mesh-controller --- PASS: TestSavingPlansAndMovingMergesIsOneAct (0.36s)
  ./cmd/mesh-controller --- PASS: TestNodeShowAndStatusReadTheLastSendFromTheStore (0.41s)
  ./cmd/mesh-controller --- PASS: TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds (0.28s)
  ./internal/inventory --- PASS: TestTheLastSendSaysWhatTheMachineAnswered (0.25s)
  ./internal/inventory --- PASS: TestAReportIsTheLastSendsAnswerOnlyWhenItIsAboutThatSend (0.24s)
  ./internal/inventory --- PASS: TestAnIssuedValueIsSealedToEveryHolderOnceAndNotAgain (0.25s)
  ./internal/inventory --- PASS: TestAnIssuedValueIsSealedAgainAfterAKeyChangeOrAnAcceptance (0.25s)
  ./internal/inventory --- PASS: TestAConditionsLooksAreKeptBesideTheBucket (0.25s)
**Merge check** at `dcdafd37` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine - read by no module's build: cmd/mesh-controller/condition_looks_test.go, cmd/mesh-controller/replay327_test.go, internal/conditions/bus_test.go, internal/conditions/looks_test.go, internal/conditions/store_test.go, internal/inventory/looks_test.go, testdata/store-tests Run by the build seat on ace as `build-1791799936294115889` (`builds --log build-1791799936294115889`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 57 tool(s); WARNING: merge-window ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 files asking for a setting without saying whether it is trusted: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-12T10:08:43Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-12T10:08:43Z modules the change moves: - mesh-controller (changed) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy notes: - cmd/mesh-controller/condition_looks_test.go, cmd/mesh-controller/replay327_test.go, internal/conditions/bus_test.go, internal/conditions/looks_test.go, internal/conditions/store_test.go, internal/inventory/looks_test.go and testdata/store-tests read by no module's build, and rebuild nothing (issue 280) --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.038s ok github.com/novox/mesh-controller/cmd/mesh-controller 225.308s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.025s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.034s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.028s ok github.com/novox/mesh-controller/examples/route-proxy 1.143s ok github.com/novox/mesh-controller/internal/artifacts 1.071s ? github.com/novox/mesh-controller/internal/beside [no test files] ok github.com/novox/mesh-controller/internal/broker 4.133s ok github.com/novox/mesh-controller/internal/builder 16.061s ok github.com/novox/mesh-controller/internal/catalogue 7.809s ok github.com/novox/mesh-controller/internal/conditions 2.661s ok github.com/novox/mesh-controller/internal/envfile 1.071s ok github.com/novox/mesh-controller/internal/facts 1.069s ok github.com/novox/mesh-controller/internal/identity 4.682s ok github.com/novox/mesh-controller/internal/inventory 87.982s ok github.com/novox/mesh-controller/internal/lease 17.735s ok github.com/novox/mesh-controller/internal/licences 4.287s ok github.com/novox/mesh-controller/internal/licences/adapters 1.087s ok github.com/novox/mesh-controller/internal/link 59.428s ok github.com/novox/mesh-controller/internal/lint 2.355s ok github.com/novox/mesh-controller/internal/outward 1.090s ok github.com/novox/mesh-controller/internal/overlay 1.081s ok github.com/novox/mesh-controller/internal/secrets 1.486s ok github.com/novox/mesh-controller/internal/store 3.502s ok github.com/novox/mesh-controller/internal/testbus 1.257s ok github.com/novox/mesh-controller/internal/token 1.020s store tests, each run against the store: ./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeWalksTheOthersAgain (0.39s) ./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeEndsTheMergesThatContainIt (0.45s) ./cmd/mesh-controller --- PASS: TestAStopWalksTheOthersAgainInTheOpenBatch (0.37s) ./cmd/mesh-controller --- PASS: TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt (0.40s) ./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeLeavesItOutOfABatch (0.38s) ./cmd/mesh-controller --- PASS: TestSavingPlansAndMovingMergesIsOneAct (0.36s) ./cmd/mesh-controller --- PASS: TestNodeShowAndStatusReadTheLastSendFromTheStore (0.41s) ./cmd/mesh-controller --- PASS: TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds (0.28s) ./internal/inventory --- PASS: TestTheLastSendSaysWhatTheMachineAnswered (0.25s) ./internal/inventory --- PASS: TestAReportIsTheLastSendsAnswerOnlyWhenItIsAboutThatSend (0.24s) ./internal/inventory --- PASS: TestAnIssuedValueIsSealedToEveryHolderOnceAndNotAgain (0.25s) ./internal/inventory --- PASS: TestAnIssuedValueIsSealedAgainAfterAKeyChangeOrAnAcceptance (0.25s) ./internal/inventory --- PASS: TestAConditionsLooksAreKeptBesideTheBucket (0.25s) ```
mesh-admin left a comment
Author
Contributor

Verdict: approve at dcdafd37f0

Independent review for issue 420 (the conditions bucket is rewritten on every observation). It follows the decision note on the issue (2026-10-11): the bucket holds what changes on a transition, the looks live in the controller's store, and the verb and the events still carry them.

Checklist

  1. Schema rule. This repository has no consolidated schema, and I verified how a fresh install builds it. internal/inventory/inventory.go:20 embeds migrations/*.sql, and internal/store/migrate.go:201 (Migrate) applies every pending migration in order, each in a transaction under an advisory lock, recorded in a ledger. A fresh store therefore runs 0001 to 0095. No schema.sql or other non-migration .sql exists. Not blocking.
  2. Migration 0095. The number is unique: main's highest is 0094, and main moved only by #246, which adds no migration. It is plain create table with no if not exists, like all 92 other migrations here. It runs once because of the ledger and the transaction, and a changed or duplicate file is refused (migrate_test.go). During rollout, old code never touches condition_looks. New code reading an old-shape bucket value keeps the value's looks (looks.go attachOne, the LastObserved non-zero branch) and rewrites the value once without them. TestAConditionWrittenWithItsLooksMovesThemOutOnce proves this: 7 003 goes to 7 006. On a rollback, old code would read observations 0 and count on from 1, which is acceptable.
  3. Readers of the bucket. In mesh-catalog main, the only references to mesh-controller_conditions are the nats snapshot tests (modules/nats/snapshot/live_test.go, which backs up and restores the stream, independent of field shape) and a messenger test fixture. The messenger's condition.go reads last-observed and observations from the verb answer and the events, and both still carry them. mesh-tools, mesh-sdk, mesh-host (only the NATS permission line in an example lock) and mesh-lab do not read the bucket's fields. I found no reader that loses data.
  4. Lease. KeepConditionLooks (internal/inventory/looks.go) records actingEpoch, following the same pattern as plans, so a process without the lease keeps nothing. A clear forgets the row (store.go, after the bucket delete). If that forget fails, a re-raise overwrites the row through keepLooks on the Create path, so no stale count survives. Evidence is capped at KeptEvidence = 10, and rows exist only for open conditions, so growth is bounded.
  5. Tests. I ran sh merge-check.sh on this head against a throwaway PostgreSQL 18.6 with gcc present, so under -race. It passed: gofmt, vet, all 26 packages ok, and every listed store test PASS, including TestAConditionsLooksAreKeptBesideTheBucket and TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds. Mutation: I replaced !bytes.Equal(body, entry.Value) in Observe with true ||. That fails TestALookThatChangesNothingWritesNothingToTheBucket, TestAConditionWrittenWithItsLooksMovesThemOutOnce and the store-backed verb test. The tests bite.
  6. Checks at this head: mesh/merge-gate success (builds build-agent, mesh-controller, route-proxy for ace, g14, novox, shanks; every machine composes) and mesh/repo-check success. mesh-controller.plans shows no plan mid-gate at 12:25.

Non-blocking observations. These are follow-ups, not reasons to hold the merge.

  • An old-shape condition loses its count if the store is away on its first look after rollout (internal/conditions/store.go, Observe). In that case attach returns false and c keeps the value's looks. BucketValue strips them and the value differs, so the bucket is rewritten without them, but keepLooks is skipped because looked is false. The count then exists nowhere, and the next look starts from 1. The keeper says that the looks could not be read, but the loss itself goes unreported. A small fix: when !looked, skip the bucket write if the only difference is the looks fields, that is, compare BucketValue(c) with BucketValue of the value as read.
  • Concurrent looks at the same key can lose a count. Before this change, the compare-and-set on the bucket serialised two Observe calls on one key. Now an unchanged look writes no bucket, so two concurrent looks can both read N and both keep N+1. The condition_looks upsert has no revision guard. I think this is rare, because one watchdog owns one key, so it is not blocking, but the count can now undercount. If it matters, an observations = condition_looks.observations + 1-style increment or a version column would fix it.
  • The word "looks" is not in the glossary. The hq pull request (#321) proposes it. That is for the operator or a glossary pull request to settle.

Not merged by me.

Verdict: approve at dcdafd37f008690935a1560c79145025e4c3cd88 Independent review for issue 420 (the conditions bucket is rewritten on every observation). It follows the decision note on the issue (2026-10-11): the bucket holds what changes on a transition, the looks live in the controller's store, and the verb and the events still carry them. Checklist 1. Schema rule. This repository has no consolidated schema, and I verified how a fresh install builds it. `internal/inventory/inventory.go:20` embeds `migrations/*.sql`, and `internal/store/migrate.go:201` (`Migrate`) applies every pending migration in order, each in a transaction under an advisory lock, recorded in a ledger. A fresh store therefore runs 0001 to 0095. No `schema.sql` or other non-migration `.sql` exists. Not blocking. 2. Migration 0095. The number is unique: main's highest is 0094, and main moved only by #246, which adds no migration. It is plain `create table` with no `if not exists`, like all 92 other migrations here. It runs once because of the ledger and the transaction, and a changed or duplicate file is refused (`migrate_test.go`). During rollout, old code never touches `condition_looks`. New code reading an old-shape bucket value keeps the value's looks (`looks.go` `attachOne`, the `LastObserved` non-zero branch) and rewrites the value once without them. `TestAConditionWrittenWithItsLooksMovesThemOutOnce` proves this: 7 003 goes to 7 006. On a rollback, old code would read observations 0 and count on from 1, which is acceptable. 3. Readers of the bucket. In mesh-catalog main, the only references to `mesh-controller_conditions` are the nats snapshot tests (`modules/nats/snapshot/live_test.go`, which backs up and restores the stream, independent of field shape) and a messenger test fixture. The messenger's `condition.go` reads `last-observed` and `observations` from the verb answer and the events, and both still carry them. mesh-tools, mesh-sdk, mesh-host (only the NATS permission line in an example lock) and mesh-lab do not read the bucket's fields. I found no reader that loses data. 4. Lease. `KeepConditionLooks` (`internal/inventory/looks.go`) records `actingEpoch`, following the same pattern as plans, so a process without the lease keeps nothing. A clear forgets the row (`store.go`, after the bucket delete). If that forget fails, a re-raise overwrites the row through `keepLooks` on the Create path, so no stale count survives. Evidence is capped at `KeptEvidence = 10`, and rows exist only for open conditions, so growth is bounded. 5. Tests. I ran `sh merge-check.sh` on this head against a throwaway PostgreSQL 18.6 with gcc present, so under `-race`. It passed: gofmt, vet, all 26 packages ok, and every listed store test PASS, including `TestAConditionsLooksAreKeptBesideTheBucket` and `TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds`. Mutation: I replaced `!bytes.Equal(body, entry.Value)` in `Observe` with `true ||`. That fails `TestALookThatChangesNothingWritesNothingToTheBucket`, `TestAConditionWrittenWithItsLooksMovesThemOutOnce` and the store-backed verb test. The tests bite. 7. Checks at this head: mesh/merge-gate success (builds build-agent, mesh-controller, route-proxy for ace, g14, novox, shanks; every machine composes) and mesh/repo-check success. `mesh-controller.plans` shows no plan mid-gate at 12:25. Non-blocking observations. These are follow-ups, not reasons to hold the merge. - **An old-shape condition loses its count if the store is away on its first look after rollout** (`internal/conditions/store.go`, `Observe`). In that case `attach` returns false and `c` keeps the value's looks. `BucketValue` strips them and the value differs, so the bucket is rewritten without them, but `keepLooks` is skipped because `looked` is false. The count then exists nowhere, and the next look starts from 1. The keeper says that the looks could not be read, but the loss itself goes unreported. A small fix: when `!looked`, skip the bucket write if the only difference is the looks fields, that is, compare `BucketValue(c)` with `BucketValue` of the value as read. - **Concurrent looks at the same key can lose a count.** Before this change, the compare-and-set on the bucket serialised two `Observe` calls on one key. Now an unchanged look writes no bucket, so two concurrent looks can both read N and both keep N+1. The `condition_looks` upsert has no revision guard. I think this is rare, because one watchdog owns one key, so it is not blocking, but the count can now undercount. If it matters, an `observations = condition_looks.observations + 1`-style increment or a version column would fix it. - The word "looks" is not in the glossary. The hq pull request (#321) proposes it. That is for the operator or a glossary pull request to settle. Not merged by me.
mesh-admin merged commit d881f4764d into main 2026-10-12 10:27:26 +00:00
mesh-admin deleted branch fix/420-conditions-bucket-on-transition 2026-10-12 10:27:26 +00:00
This repo is archived. You cannot comment on pull requests.
No Reviewers
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-controller#249