Say how the writer tells its own lost acknowledgement from a former holder, list D16, and fix the settings heading (review of #303)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered

This commit is contained in:
jochen
2026-10-10 22:05:17 +02:00
parent 5a3513b819
commit 9be659b18c
3 changed files with 24 additions and 11 deletions
@@ -156,7 +156,9 @@ What each value holds, and what it leaves out, is in to-be 54. These limits hold
- **Written by the lease holder alone**, as a row of to-be 45 §1's writers table. A controller without
the lease writes no mirror. Each write is a compare-and-set on the key's revision the writer last saw,
so a former holder that has not yet noticed it lost the lease cannot overwrite a newer value. A value
so a former holder that has not yet noticed it lost the lease cannot overwrite a newer value. A refusal
can also be the writer's own write whose acknowledgement was lost; to-be 54 §2 says how the writer tells
the two apart. A value
carries only fields built from the record: no epoch and no time of writing, because the next lease
holder must build the same bytes from the same record. The epoch travels in the event.
- **No secret and no credential**, ever (ADR 0201 §5). Each value is built from a list of named fields,
@@ -218,7 +220,8 @@ counts both over a run of changes.
hours. Then the condition's resolver becomes the operator and its severity urgent, because a mirror that keeps drifting means
the controller's write path is broken. The next run of D17 clears the condition when it finds no
difference.
- **When the controller takes the lease, it runs D17 before it serves.** A difference found then is
- **When the controller takes the lease, it runs D17's first read before it serves**, and the second read
while it serves. A difference found then is
raised like any other, because it means a controller stopped between a commit and its write.
- **A mirror created empty** by this start is filled whole before the controller serves, and that fill
is not a difference.
@@ -297,6 +297,7 @@ producer that wrote it.
| D12 | every consumer of a provision that keeps its data is bound where it was last sent, or moves by a pin (`binding-kept`, `binding-moving`, `binding-moved`) | ADR 0232 |
| D13 | every item of data a machine declares is measured, is there, holds what it held, is written where it says it is, is backed up within its bound or sits on healthy redundant storage, and is no empty replacement of a copy kept elsewhere; an irreplaceable or valuable item a machine no longer declares is retired, not forgotten. Each machine's `node-backup` holder is asked `backed-up`; every provider of kept consumer data `provisioner_retirement` (with each held consumer's size) — see §2 for the kinds | ADR 0233, issue 273 |
| D15 | no user of the bus had its connection dropped more than twelve times in the last hour (*provisional*): the bus module's `nats_closed_connections`, from the server's record of closed connections; `bus-reconnects` (ADR 0265) | 327 |
| D16 | every delivery runs healthy on every node within its class's budget (five minutes for a leaf module, ten for a core one); `delivery.<class>.over-budget`, naming the delivery and its longest phase. *Listed 2026-10-10*: the controller already runs it under this id | ADR 0282 |
| D17 | every record mirror holds exactly what its store says: no key missing, none differing, none without a record, a difference counted only when a second read 30 seconds later still finds it and the record did not change between them; `mirror-drifted` (H6), and `mirror-event-lost` for an event lost after its write. *Added 2026-10-10* ([to-be 54](54-the-controllers-records-on-the-bus.md) §4) | ADR 0289 |
| DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 |
| H-* | the health probes of §8, run for every core component on every machine: H-controller, H-engine, H-tools, H-bus (as built, [ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)) | rule 8 |
@@ -96,8 +96,10 @@ A walk is removed from the mirror 14 days after it ended.
A build is removed from the mirror 14 days after it ended. A build that failed before its module was
known is kept by its id like any other.
### settings — `mesh-controller_settings`, key: `<module>` for the mesh-wide layer, `<node>.<module>` for one node's layer, in the order the
assignments mirror uses
### settings — `mesh-controller_settings`, key: `<module>`, or `<node>.<module>` for one node's layer
The mesh-wide layer is keyed by the module, and one node's layer by `<node>.<module>`, the order the
assignments mirror uses.
| Holds | Leaves out, and why |
|---|---|
@@ -120,9 +122,14 @@ called).
§1's writers table, so a grant that lets anything else publish under them is refused at composition.
- **After the commit, never before.** A value written before its commit could describe a change the
store then rolled back. The writer is handed only a committed change; a rolled-back one never reaches it.
- **Compare-and-set.** Each write names the key's revision the writer last saw. A write refused for a
newer revision means another writer was there, which only a former lease holder can be; the writer
stops and the refusal is logged and counted with the lost events below.
- **Compare-and-set.** Each write names the key's revision the writer last saw. A refusal has more than
one cause: a former lease holder that wrote in between, or the writer's own earlier write whose
acknowledgement was lost. After a refusal the writer reads the key's value and revision again. If the
bucket already holds the value it meant to write, that was its own write, and it only takes the new
revision. Otherwise it checks that it still holds the lease: if it does, it writes again against the
revision it just read; if it does not, it stops writing every mirror, as every writer without the lease
does. Each refusal is logged and counted in D17's verdict.
- **H6 writes through the writer**, so the revision the writer keeps is current after a repair.
- **Only a difference is written.** The writer keeps the last value it wrote for each key, filled from the
bucket when the controller takes the lease, and compares the new value byte for byte. The encoding is
fixed (fields in their list's order, maps sorted), so the same record always encodes the same.
@@ -175,8 +182,8 @@ does this for each mirror:
counted lost and how long it took. Its bound is 90 seconds rather than a probe's usual 30, to hold the
second read.
D17 is the next free number: D16 is taken by the delivery-time probe the controller already runs (the
`over-budget` condition, ADR 0282), which to-be 45's table does not list yet.
D17 is the next free number: D16 is the delivery-time probe the controller already runs (the
`over-budget` condition, ADR 0282), listed in to-be 45's table in the same change.
**A difference raises `mirror-drifted`**, warning, keyed `core.mirror.<mirror>.drifted`, at once. Its
evidence lists up to ten differing keys, each with what kind of difference it is, and the count of the
@@ -190,8 +197,9 @@ that keeps drifting means the writer is broken, and an issue is the next step. H
never the store, so it neither deletes nor recreates data.
**When the controller takes the lease**, it fills the writer's memory (each key's value and revision) from
the buckets and runs D17 once before it serves. Because a value holds no epoch, a change of lease alone
is never a difference. A bucket created empty by this start is filled whole first; that fill is not a
the buckets and runs D17's first read before it serves; the second read follows while it serves, and
raises what it confirms. So a change of lease does not wait 30 seconds for the audit. Because a value
holds no epoch, a change of lease alone is never a difference. A bucket created empty by this start is filled whole first; that fill is not a
difference. Anything else found then is raised, because it means a controller stopped between a commit
and its write.
@@ -223,6 +231,7 @@ remain open there.
| Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals |
| A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run |
| A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set |
| A lost acknowledgement is no conflict | the controller's test in which the bus stores a write and drops its acknowledgement: the next write to that key succeeds and no condition is raised |
| The runtime makes no write for an equal `put` | the runtime's test over a real bus, which puts the same value twice and expects one write and the answer `unchanged` |
## What it leaves out