From 9be659b18cb4a88a4b0ec314e2f87e2ea8810460 Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 10 Oct 2026 22:02:34 +0200 Subject: [PATCH] Say how the writer tells its own lost acknowledgement from a former holder, list D16, and fix the settings heading (review of #303) --- ...-a-change-and-audited-against-its-store.md | 7 +++-- .../45-a-core-that-cannot-fail-silently.md | 1 + .../54-the-controllers-records-on-the-bus.md | 27 ++++++++++++------- 3 files changed, 24 insertions(+), 11 deletions(-) diff --git a/02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md b/02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md index 0011bc1d..7e2dad90 100644 --- a/02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md +++ b/02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md @@ -156,7 +156,9 @@ What each value holds, and what it leaves out, is in to-be 54. These limits hold - **Written by the lease holder alone**, as a row of to-be 45 §1's writers table. A controller without the lease writes no mirror. Each write is a compare-and-set on the key's revision the writer last saw, - so a former holder that has not yet noticed it lost the lease cannot overwrite a newer value. A value + so a former holder that has not yet noticed it lost the lease cannot overwrite a newer value. A refusal + can also be the writer's own write whose acknowledgement was lost; to-be 54 §2 says how the writer tells + the two apart. A value carries only fields built from the record: no epoch and no time of writing, because the next lease holder must build the same bytes from the same record. The epoch travels in the event. - **No secret and no credential**, ever (ADR 0201 §5). Each value is built from a list of named fields, @@ -218,7 +220,8 @@ counts both over a run of changes. hours. Then the condition's resolver becomes the operator and its severity urgent, because a mirror that keeps drifting means the controller's write path is broken. The next run of D17 clears the condition when it finds no difference. -- **When the controller takes the lease, it runs D17 before it serves.** A difference found then is +- **When the controller takes the lease, it runs D17's first read before it serves**, and the second read + while it serves. A difference found then is raised like any other, because it means a controller stopped between a commit and its write. - **A mirror created empty** by this start is filled whole before the controller serves, and that fill is not a difference. diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index 7646d76c..e64a10a4 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -297,6 +297,7 @@ producer that wrote it. | D12 | every consumer of a provision that keeps its data is bound where it was last sent, or moves by a pin (`binding-kept`, `binding-moving`, `binding-moved`) | ADR 0232 | | D13 | every item of data a machine declares is measured, is there, holds what it held, is written where it says it is, is backed up within its bound or sits on healthy redundant storage, and is no empty replacement of a copy kept elsewhere; an irreplaceable or valuable item a machine no longer declares is retired, not forgotten. Each machine's `node-backup` holder is asked `backed-up`; every provider of kept consumer data `provisioner_retirement` (with each held consumer's size) — see §2 for the kinds | ADR 0233, issue 273 | | D15 | no user of the bus had its connection dropped more than twelve times in the last hour (*provisional*): the bus module's `nats_closed_connections`, from the server's record of closed connections; `bus-reconnects` (ADR 0265) | 327 | +| D16 | every delivery runs healthy on every node within its class's budget (five minutes for a leaf module, ten for a core one); `delivery..over-budget`, naming the delivery and its longest phase. *Listed 2026-10-10*: the controller already runs it under this id | ADR 0282 | | D17 | every record mirror holds exactly what its store says: no key missing, none differing, none without a record, a difference counted only when a second read 30 seconds later still finds it and the record did not change between them; `mirror-drifted` (H6), and `mirror-event-lost` for an event lost after its write. *Added 2026-10-10* ([to-be 54](54-the-controllers-records-on-the-bus.md) §4) | ADR 0289 | | DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 | | H-* | the health probes of §8, run for every core component on every machine: H-controller, H-engine, H-tools, H-bus (as built, [ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)) | rule 8 | diff --git a/03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md b/03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md index a8b6b216..a4055712 100644 --- a/03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md +++ b/03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md @@ -96,8 +96,10 @@ A walk is removed from the mirror 14 days after it ended. A build is removed from the mirror 14 days after it ended. A build that failed before its module was known is kept by its id like any other. -### settings — `mesh-controller_settings`, key: `` for the mesh-wide layer, `.` for one node's layer, in the order the -assignments mirror uses +### settings — `mesh-controller_settings`, key: ``, or `.` for one node's layer + +The mesh-wide layer is keyed by the module, and one node's layer by `.`, the order the +assignments mirror uses. | Holds | Leaves out, and why | |---|---| @@ -120,9 +122,14 @@ called). §1's writers table, so a grant that lets anything else publish under them is refused at composition. - **After the commit, never before.** A value written before its commit could describe a change the store then rolled back. The writer is handed only a committed change; a rolled-back one never reaches it. -- **Compare-and-set.** Each write names the key's revision the writer last saw. A write refused for a - newer revision means another writer was there, which only a former lease holder can be; the writer - stops and the refusal is logged and counted with the lost events below. +- **Compare-and-set.** Each write names the key's revision the writer last saw. A refusal has more than + one cause: a former lease holder that wrote in between, or the writer's own earlier write whose + acknowledgement was lost. After a refusal the writer reads the key's value and revision again. If the + bucket already holds the value it meant to write, that was its own write, and it only takes the new + revision. Otherwise it checks that it still holds the lease: if it does, it writes again against the + revision it just read; if it does not, it stops writing every mirror, as every writer without the lease + does. Each refusal is logged and counted in D17's verdict. +- **H6 writes through the writer**, so the revision the writer keeps is current after a repair. - **Only a difference is written.** The writer keeps the last value it wrote for each key, filled from the bucket when the controller takes the lease, and compares the new value byte for byte. The encoding is fixed (fields in their list's order, maps sorted), so the same record always encodes the same. @@ -175,8 +182,8 @@ does this for each mirror: counted lost and how long it took. Its bound is 90 seconds rather than a probe's usual 30, to hold the second read. -D17 is the next free number: D16 is taken by the delivery-time probe the controller already runs (the -`over-budget` condition, ADR 0282), which to-be 45's table does not list yet. +D17 is the next free number: D16 is the delivery-time probe the controller already runs (the +`over-budget` condition, ADR 0282), listed in to-be 45's table in the same change. **A difference raises `mirror-drifted`**, warning, keyed `core.mirror..drifted`, at once. Its evidence lists up to ten differing keys, each with what kind of difference it is, and the count of the @@ -190,8 +197,9 @@ that keeps drifting means the writer is broken, and an issue is the next step. H never the store, so it neither deletes nor recreates data. **When the controller takes the lease**, it fills the writer's memory (each key's value and revision) from -the buckets and runs D17 once before it serves. Because a value holds no epoch, a change of lease alone -is never a difference. A bucket created empty by this start is filled whole first; that fill is not a +the buckets and runs D17's first read before it serves; the second read follows while it serves, and +raises what it confirms. So a change of lease does not wait 30 seconds for the audit. Because a value +holds no epoch, a change of lease alone is never a difference. A bucket created empty by this start is filled whole first; that fill is not a difference. Anything else found then is raised, because it means a controller stopped between a commit and its write. @@ -223,6 +231,7 @@ remain open there. | Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals | | A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run | | A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set | +| A lost acknowledgement is no conflict | the controller's test in which the bus stores a write and drops its acknowledgement: the next write to that key succeeds and no condition is raised | | The runtime makes no write for an equal `put` | the runtime's test over a real bus, which puts the same value twice and expects one write and the answer `unchanged` | ## What it leaves out