Keep the epoch out of a mirror's value, age purge markers, say a lost event, and check every rule (review of #303)

02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md

@@ -155,8 +155,10 @@ same change, and says how it differs from the builder's mirrored upstream image.
 What each value holds, and what it leaves out, is in to-be 54. These limits hold for all five:
 
 - **Written by the lease holder alone**, as a row of to-be 45 §1's writers table. A controller without
-  the lease writes no mirror. Each value carries the epoch it was written under and the time its record
-  last changed in the store.
+  the lease writes no mirror. Each write is a compare-and-set on the key's revision the writer last saw,
+  so a former holder that has not yet noticed it lost the lease cannot overwrite a newer value. A value
+  carries only fields built from the record: no epoch and no time of writing, because the next lease
+  holder must build the same bytes from the same record. The epoch travels in the event.
 - **No secret and no credential**, ever (ADR 0201 §5). Each value is built from a list of named fields,
   and a test fails on a field the list does not name.
 - **The settings mirror holds no setting's value**, only the names a layer sets, when it was set and by
@@ -164,14 +166,18 @@ What each value holds, and what it leaves out, is in to-be 54. These limits hold
   is a stream the bus's snapshots back up (ADR 0235). A digest of the value is refused too, because a
   digest of a short password can be guessed back. A watcher asks the `settings` verb for the values.
 - **No age on the bucket.** A walk or build that ended is removed from its mirror 14 days after it
-  ended, by the controller, and the removal is a change with its event. The store keeps it.
+  ended, by the controller, and the removal is a change with its event. The store keeps it. A removal is
+  a purge, which leaves one marker for the key; each marker is given a time to live of one day, so
+  removed keys do not pile up in a bucket with no age.
 - **Each value is at most 64 KiB.** A value that would be larger is cut to its summary and says so.
-  The verb still answers the whole record.
+  The verb still answers the whole record. Checked by the controller's test, which builds the largest
+  walk the mesh could make (every module on every node) and expects a value under the cap, marked cut.
 - **Declared as the controller's state** in its manifest, so a module names one in `reads` and receives
   its grant as ADR 0201 gives it. The controller asserts the options at start, as it does for its other
   buckets, and a test holds the declaration and the assertion equal.
 - **The controller never reads a mirror to decide anything.** Only the audit in §4 reads it. A mirror
-  that fed the controller's own decisions would become a second record.
+  that fed the controller's own decisions would become a second record. Checked by a test over the
+  controller's packages, which fails when any package but the audit's reads a mirror bucket.
 
 ### 3. Every write to a mirror is one event
 
@@ -180,6 +186,13 @@ and carries the key, the change (`added`, `changed` or `removed`), the names of
 the bucket revision of the write, the epoch, and whether the healer of §4 made it. `setting-changed`
 also carries who changed the setting, as the runtime proves the caller. A failed write sends no event.
 
+**The bucket is the authority and the event is a notice.** An event can be lost after its write
+succeeded: the controller stops, or the bus refuses the publish. A watcher that must not miss a change
+watches the bucket, whose watch delivers every revision. The writer tries a refused event once more,
+then counts it as lost. D17's verdict carries the count since the last run, and any lost event raises
+`mirror-event-lost` (warning), which clears on the next run that counts none. A controller that stops
+between a write and its event loses that event without a count; the bucket still holds the change.
+
 The existing events keep their meaning: `plan-moved` says a walk moved, `built` says a build ended,
 `applied` says a node applied its declaration. A mirror's event says that the record now reads
 differently. Some changes are therefore said twice, once as the act and once as the record.
@@ -193,14 +206,16 @@ counts both over a run of changes.
 - **The probe D17** (the mirrors match their store) runs with the self-check, every five minutes. For each
   mirror it builds the value of every record from the store and compares it, byte for byte, with the
   bucket's key. A key missing, a key whose value differs and a key with no record are each a difference.
-  A record that changed in the last 30 seconds is skipped by that run, so the probe never races the
-  write that follows a commit.
+  A key counts as different only when it differs on two reads 30 seconds apart and its record did not
+  change between them, so the probe never races the write that follows a commit. A record that keeps
+  changing for longer is not judged in that run, and the verdict names it.
 - **Any difference raises `mirror-drifted`**, a warning keyed `core.mirror.<mirror>.drifted`. Its
-  evidence names the keys and what kind of difference each is. The difference is definite, so it is
-  raised at once, not on a second look (to-be 45 §4).
+  evidence names the keys and what kind of difference each is. The second read inside the run is its
+  second look, so it is raised at once (to-be 45 §4). A key past its 14 days is a key with no record, so
+  the removal of §2 is checked here too.
 - **The healer H6 writes the store's value** for each key the condition names, removes a key with no
   record, and sends each write's event marked as the healer's. It may do this twice per mirror in 24
-  hours. Then the condition's resolver becomes the operator, because a mirror that keeps drifting means
+  hours. Then the condition's resolver becomes the operator and its severity urgent, because a mirror that keeps drifting means
   the controller's write path is broken. The next run of D17 clears the condition when it finds no
   difference.
 - **When the controller takes the lease, it runs D17 before it serves.** A difference found then is
@@ -208,9 +223,8 @@ counts both over a run of changes.
 - **A mirror created empty** by this start is filled whole before the controller serves, and that fill
   is not a difference.
 
-**The bound a reader may rely on:** a mirror differs from its store for at most the self-check's
-interval plus the probe's 30 seconds of grace before a condition says so, and the value names the epoch
-and the time of the record it shows.
+**The bound a reader may rely on:** for a record that was quiet for 30 seconds, a mirror differs from
+its store for at most the self-check's interval plus those 30 seconds before a condition says so.
 
 ## Consequences
 
@@ -225,7 +239,8 @@ and the time of the record it shows.
   keeps a list of fields per mirror. Every new column of a mirrored table is a choice to mirror it or
   not, made in the list, where the test sees it.
 - The self-check reads every mirrored record every five minutes. At the sizes measured (a few hundred
-  keys outside builds, a few thousand builds in 14 days) that is well inside a probe's 30 seconds; the
+  keys outside builds, a few thousand builds in 14 days) that is well inside the probe's bound of 90
+  seconds, which covers its second read; the
   probe's own duration is in its verdict, so a growing mirror is seen.
 - The conditions bucket stays as it is until issue 420 is fixed, and so breaks the rule of §1 in the
   meantime.

03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md

@@ -297,7 +297,7 @@ producer that wrote it.
 | D12 | every consumer of a provision that keeps its data is bound where it was last sent, or moves by a pin (`binding-kept`, `binding-moving`, `binding-moved`) | ADR 0232 |
 | D13 | every item of data a machine declares is measured, is there, holds what it held, is written where it says it is, is backed up within its bound or sits on healthy redundant storage, and is no empty replacement of a copy kept elsewhere; an irreplaceable or valuable item a machine no longer declares is retired, not forgotten. Each machine's `node-backup` holder is asked `backed-up`; every provider of kept consumer data `provisioner_retirement` (with each held consumer's size) — see §2 for the kinds | ADR 0233, issue 273 |
 | D15 | no user of the bus had its connection dropped more than twelve times in the last hour (*provisional*): the bus module's `nats_closed_connections`, from the server's record of closed connections; `bus-reconnects` (ADR 0265) | 327 |
-| D17 | every record mirror holds exactly what its store says: no key missing, none differing, none without a record, records changed in the last 30 seconds skipped; `mirror-drifted`, raised at once (H6). *Added 2026-10-10* ([to-be 54](54-the-controllers-records-on-the-bus.md) §4) | ADR 0289 |
+| D17 | every record mirror holds exactly what its store says: no key missing, none differing, none without a record, a difference counted only when a second read 30 seconds later still finds it and the record did not change between them; `mirror-drifted` (H6), and `mirror-event-lost` for an event lost after its write. *Added 2026-10-10* ([to-be 54](54-the-controllers-records-on-the-bus.md) §4) | ADR 0289 |
 | DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 |
 | H-* | the health probes of §8, run for every core component on every machine: H-controller, H-engine, H-tools, H-bus (as built, [ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)) | rule 8 |
 | DG | every build that failed its gate, and every core build a witness put back, is said — `build.<module>.<machine>.rolled-back`, `core.<component>.<machine>.<outcome>`, `rollback-failed` — until a newer build passes or the reports stop saying it (ADR 0236) | rule 8 |

03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md

@@ -48,9 +48,11 @@ do not have: a check that the bucket says what the store says.
 
 ## 1. The five mirrors
 
-Each value is built from a named list of fields. A field not on the list is never in the value. Every
-value also carries `epoch` (the lease epoch it was written under) and `changed` (when the record last
-changed in the store, which is the time of the change and not of the write).
+Each value is built from a named list of fields. A field not on the list is never in the value. A value
+holds only what is built from its record: no epoch, no time of writing and no time of the last look,
+because any lease holder must build the same bytes from the same record, and the audit must too. Times
+that belong to the record itself, such as when a walk was opened or a resource's state began, are fields
+like any other. The epoch and the time of the write travel in the event.
 
 ### nodes — `mesh-controller_nodes`, key: the node's name
 
@@ -94,7 +96,8 @@ A walk is removed from the mirror 14 days after it ended.
 A build is removed from the mirror 14 days after it ended. A build that failed before its module was
 known is kept by its id like any other.
 
-### settings — `mesh-controller_settings`, key: `<module>` for the mesh-wide layer, `<module>.<node>` for one node's layer
+### settings — `mesh-controller_settings`, key: `<module>` for the mesh-wide layer, `<node>.<module>` for one node's layer, in the order the
+assignments mirror uses
 
 | Holds | Leaves out, and why |
 |---|---|
@@ -116,14 +119,20 @@ called).
   every other writer of the controller's buckets (to-be 45 §6). The five mirrors are rows of to-be 45
   §1's writers table, so a grant that lets anything else publish under them is refused at composition.
 - **After the commit, never before.** A value written before its commit could describe a change the
-  store then rolled back.
+  store then rolled back. The writer is handed only a committed change; a rolled-back one never reaches it.
+- **Compare-and-set.** Each write names the key's revision the writer last saw. A write refused for a
+  newer revision means another writer was there, which only a former lease holder can be; the writer
+  stops and the refusal is logged and counted with the lost events below.
 - **Only a difference is written.** The writer keeps the last value it wrote for each key, filled from the
   bucket when the controller takes the lease, and compares the new value byte for byte. The encoding is
   fixed (fields in their list's order, maps sorted), so the same record always encodes the same.
 - **Who set a settings layer** is kept in the store beside when (`settings` has the time and not the
   caller today): a numbered migration of `inventory` and the consolidated schema, in the same change.
-- **A removal is a purge** of the key, so the bucket keeps no marker per removed key, and it is said as
-  a `removed` event. The controller's one-minute tick removes walks and builds 14 days after they ended.
+- **A removal is a purge** of the key, said as a `removed` event. A purge leaves one marker for the key.
+  Each purge gives its marker a time to live of one day (a per-message time to live, which the bus server
+  has offered since release 2.11 and the bucket enables), so the markers of removed walks and builds do
+  not pile up in a bucket that has no age. The controller's one-minute tick removes walks and builds 14
+  days after they ended.
 - **A write that fails** sends no event, is logged, and is left to the audit. The writer does not retry
   in a loop: a retry loop against a refusing bus is the reconnect loop of issue 327 in another place.
 
@@ -134,12 +143,20 @@ Each mirror has its own kind on the controller's seat, `mesh.seat.mesh-controlle
 the seat's emits, as every event of the controller is.
 
 Each carries the mirror, the key, the change (`added`, `changed`, `removed`), the names of the fields that
-changed, the bucket revision of the write, the epoch, `changed`, and `repaired` when the healer H6 made the
-write. A watcher that wants the value reads it from the bucket at that revision or later.
+changed, the bucket revision of the write, the epoch, the time of the write, and `repaired` when the
+healer H6 made the write. A watcher that wants the value reads it from the bucket at that revision or later.
 
 **One write is one event, and one event is one write.** The event is sent after the write succeeds, so
 there is no event for a value that is not in the bucket.
 
+**The bucket is the authority, and the event is a notice.** An event can be lost after its write: the
+controller stops, or the bus refuses the publish. A watcher that must not miss a change watches the bucket,
+whose watch delivers every revision. The writer tries a refused event once more, then counts it as lost.
+D17's verdict carries the count since the last run, and any lost event raises `mirror-event-lost`
+(warning), cleared by the next run that counts none. A controller that stops between a write and its event
+loses that event uncounted; the bucket still holds the change, and the live comparison of write and event
+counts in §6 shows the gap.
+
 ## 4. The audit: D17 and H6
 
 **D17, the mirrors match their store**, is a probe in the self-check's registry (to-be 45 §4). Every run
@@ -148,10 +165,18 @@ does this for each mirror:
 1. It reads every record the mirror should hold from the store (for walks and builds, those open or ended
    within 14 days) and builds each value as the writer would.
 2. It lists the bucket's keys and reads their values.
-3. It compares. A key missing, a value that differs, and a key with no record are each a difference. A
-   record whose `changed` is less than 30 seconds old is skipped, so the probe never races the write
-   that follows a commit, and so is a walk or build within 30 seconds of its 14-day removal.
-4. Its verdict names, per mirror, the keys compared, the differences and how long it took.
+3. It compares. A key missing, a value that differs, and a key with no record are each a candidate. A
+   key past its 14 days is a key with no record, so the removal of §2 is checked here.
+4. After 30 seconds it reads each candidate's record and key again. A candidate is a difference only
+   when it still differs and its record did not change between the two reads. So the probe never races
+   the write that follows a commit, and the second read is its second look (to-be 45 §4). A record that
+   kept changing is not judged in that run and is named in the verdict.
+5. Its verdict names, per mirror, the keys compared, the differences, the records not judged, the events
+   counted lost and how long it took. Its bound is 90 seconds rather than a probe's usual 30, to hold the
+   second read.
+
+D17 is the next free number: D16 is taken by the delivery-time probe the controller already runs (the
+`over-budget` condition, ADR 0282), which to-be 45's table does not list yet.
 
 **A difference raises `mirror-drifted`**, warning, keyed `core.mirror.<mirror>.drifted`, at once. Its
 evidence lists up to ten differing keys, each with what kind of difference it is, and the count of the
@@ -164,8 +189,9 @@ a key with no record, and each write sends its event with `repaired`. Its budget
 that keeps drifting means the writer is broken, and an issue is the next step. H6 writes only a mirror,
 never the store, so it neither deletes nor recreates data.
 
-**When the controller takes the lease**, it fills the writer's memory from the buckets and runs D17 once
-before it serves. A bucket created empty by this start is filled whole first; that fill is not a
+**When the controller takes the lease**, it fills the writer's memory (each key's value and revision) from
+the buckets and runs D17 once before it serves. Because a value holds no epoch, a change of lease alone
+is never a difference. A bucket created empty by this start is filled whole first; that fill is not a
 difference. Anything else found then is raised, because it means a controller stopped between a commit
 and its write.
 
@@ -188,9 +214,15 @@ remain open there.
 | A value holds only its listed fields, and never a secret | the controller's test per mirror, which builds every value from a store holding every column filled and fails on a field not on the list |
 | One write, one event | the controller's test over a run of changes, counting both; live, the server's write count for `$KV.mesh-controller_<mirror>.>` (`nats.nats_subscriptions`) and the count of that mirror's events in EVENTS rise by the same number over any window |
 | Only the lease holder writes | the writers table's compiled test (to-be 45 §1); D5 |
-| The bucket matches the store | D17 on every self-check run; its own test, which changes a key behind the writer's back and expects `mirror-drifted`, then H6's write, then the condition cleared on the next run |
+| The bucket matches the store | D17 on every self-check run; its own test, which changes a key behind the writer's back and expects `mirror-drifted`, then H6's write, then the condition cleared on the next run; and a test that a change of lease alone raises nothing |
 | Quiet means no writes | live: over ten minutes in which `plans`, `nodes`, `builds` and `settings` show no change, no mirror's write count rises |
 | A watcher sees a change | live: a setting changed through `settings` is seen by a watch on `mesh-controller_settings` and as one `setting-changed` within a second |
+| A value is under its cap, or cut and marked | the controller's test, which builds the largest walk the mesh could make (every module on every node) |
+| Only the audit reads a mirror | a test over the controller's packages, which fails when any package but the audit's reads a mirror bucket |
+| A write follows the commit | the controller's test that a rolled-back transaction makes no write and sends no event |
+| Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals |
+| A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run |
+| A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set |
 | The runtime makes no write for an equal `put` | the runtime's test over a real bus, which puts the same value twice and expects one write and the answer `unchanged` |
 
 ## What it leaves out