novox/hq
6
0
Fork 0
Code Issues Pull Requests 5 Releases Wiki Activity

Merge pull request 'ADR 0289: the controller mirrors its records into buckets written only on a change (issue 419, research 036 G2)' (#303) from decisions/0289-the-controller-mirrors-its-records-on-the-bus into main

Browse Source
This commit was merged in pull request #303.
This commit is contained in:
mesh-admin
2026-10-10 20:07:05 +00:00
parent 750a1d54d4 9be659b18c
commit 1b84960179
8 changed files with 565 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
00-META/glossary.md
+9 -1
View File
@@ -116,7 +116,7 @@ are a published vocabulary, fixed by the manifest's schema. **Decided by** ADR 0
**Purpose.** To keep the controller, the bus, the store and every node's engine running and agreed, so
every other domain has something to run on. **Recorded by** the controller and the store.
**Upstream of** every domain but Module; the others use the bus and the store as they are.
**Decided by** ADR 0005, 0006, 0067, 0106, 0141, 0175, 0227, 0229.
**Decided by** ADR 0005, 0006, 0067, 0106, 0141, 0175, 0227, 0229, 0289.
**Uses:** node, module, seat.
- **controller** — the component that decides what each node should be, holds the mesh's records, and
@@ -177,6 +177,14 @@ every other domain has something to run on. **Recorded by** the controller and t
order of what it sent (the epoch, which a node reads before it accepts a declaration)
([ADR 0229](../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)).
- **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its
records (nodes, assignments, walks, builds, settings), after each change commits in its store and only
when the value differs, with one event per write. The store stays the record; the self-check compares
the two and raises `mirror-drifted` on any difference
([ADR 0289](../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md)). Shortened to *mirror* where the
text is about the controller's records; the builder's *mirrored* `upstream` image (Change and delivery) is
a different thing, a copy of an outside image in the artifact store.
## Placement — what runs on which node
**Purpose.** To decide, send and apply what each node runs, and to say why.
01-RESEARCH/036-everything-over-nats/00-overview.md
+20
View File
@@ -11,6 +11,7 @@ touches:
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- the controller's records
- the catalogue's module manifests
- 03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md
became: []
---
@@ -117,6 +118,25 @@ API through a provision are acceptable by the principle.
- **What a module declares:** state, history streams, and tools in place of side doors.
- **The conversation with the operator**, where the approval flow meets a view.
## Decided so far, and what stays open
**This effort stays active.** It graduates in parts, each part a decision that extends it without
closing it:
- **G7, the bus watching itself:** [ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md)
(the dead-letter stream) and [ADR 0265](../../02-DECISIONS/0265-a-verb-that-reads-is-answered-on-the-serving-controllers-own-connection-and-every-connection-says-what-it-is.md)
(one connection per controller, named), with issues 326 to 330.
- **G2, the controller's state:** [ADR 0289](../../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md), *2026-10-10*. The controller
keeps five record mirrors (nodes, assignments, walks, builds and settings), written only on a change and
audited against its store, designed in [to-be 54](../../03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md)
and built under issue 419. Its rule that a bucket is written only when what it describes changed holds
for every bucket; the conditions bucket breaks it today, which issue 420 tracks. This answers open
question 1 below: G2 came first.
Still open here: G1 (people reaching the bus, open questions 2 and 3), G3 (one bus server, open question
4) and the views, which the operator has parked; and G4, G5, G6 and G8 to G10, which no decision has taken
yet. The effort is graduated when each has a decision or is dropped with why.
## Open questions for the next step
1. **Order.** Which comes first: G2 for the controller, which makes views possible, or G1, which
02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
+6
View File
@@ -61,6 +61,12 @@ reason — a bucket the module was not issued and a write to one it only reads.
current values first, without deletions, then an end-of-current marker, then every change, each as a
`mesh/state` request the bundle answers.
> **The mechanism changed — 2026-10-10, by [ADR 0289](0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md).** What stands: the runtime serves
> `put` and the other verbs on the bundle's channel from the issued list. What moved: a `put` whose value
> equals the key's current value makes no write and answers `unchanged` with the key's current revision,
> because a bucket is written only when what it describes changed ([issue 326](../04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/00-report.md)
> rewrote one about 40 times a second).
**5. No secret is stored in a bucket, sealed or not.** A bucket is a stream, and design 32 §10 keeps
every secret off streams. A value that needs a secret names it; the secret travels on request/reply.
02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md
+264
View File
@@ -0,0 +1,264 @@
---
topic: the tiers
status: accepted
date: 2026-10-10
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
---
# 289. The controller mirrors its records into buckets written only on a change, and audits each against its store
This record decides gap G2 of [research 036](../01-RESEARCH/036-everything-over-nats/00-overview.md)
(everything over NATS): most of the controller's state cannot be watched on the bus. The effort stays
active for its other gaps. Its design is [to-be 54](../03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md),
and its build is tracked by issue 419 (the controller's records cannot be watched on the bus).
The operator delegated the choices ("I trust your judgement"): this is plumbing the operator does not
see. The choices below are therefore the record's own, each with its evidence.
## Context
**What already works.** The controller's conditions are the one complete example of the pattern a
watcher needs ([research 036](../01-RESEARCH/036-everything-over-nats/01-evidence.md) §1): current state in
the bucket `mesh-controller_conditions`, one key per open condition, and an event for each change
(`condition-raised`, `condition-changed`, `condition-cleared`). The messenger hears the events and has
told the operator about conditions since 2026-10-06, without ever asking a verb.
**What cannot be watched.** Measured on 2026-10-10 with `nats.nats_buckets` and `mesh-store.query`:
- The bus holds 28 buckets, 6 of them the controller's: `asked`, `calls`, `conditions`,
`condition-history`, `hand-acts` and `lease`. None holds a node, an assignment, a walk, a build or a
setting.
- Those live only in the controller's `inventory` database. The store's statistics estimate 4 nodes,
254 assignments, 613 walks (`release_plan`), 7 462 builds and 142 modules. The settings tables
themselves are not readable through `mesh-store.query`; the statistics estimate 19 settings rows
and 3 rows of `settings_history`, which held 0 rows when the effort measured it on 2026-10-08.
- A module or a person learns any of them only by calling a verb again and again (`nodes`, `node`,
`plans`, `delivery-walks`, `builds`, `settings`). A verb's answer is bounded by the bus's 1 MiB
message limit and is paged above it (issue 314).
- A setting changes with no event at all. `plan-moved` (ADR 0239) says a walk moved; nothing says an
assignment, a node's state or a build changed.
**A bucket written when nothing changed lies about change.** [Issue 326](../04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/00-report.md)
found mesh-delivery rewriting its `deliveries` bucket on every pass: 5 689 690 writes in the bus
server's first 39 hours, about 40 a second for about 1 000 keys. A watcher receives every write, and a
key's revision no longer meant that the delivery changed. The fix (mesh-catalog #187 and #188, resolved
2026-10-10 19:36 UTC) writes only when the value differs; in a quiet window of 612 seconds it wrote
nothing. The same shape is in the controller's own conditions bucket today: the server had stored
131 647 writes to `$KV.mesh-controller_conditions.>` in 3 days 21 hours (about 0.39 a second), and 29
more between 19:50:33 and 19:52:36 UTC, while 12 conditions were open. One of them,
`mesh.hand-acts.disk-load.healer-wanted`, counted 7 003 observations since 2026-10-08. Each observation
rewrites the condition, because its value carries the time of the last look, the count of looks and
the newest evidence.
**What the store is for.** The controller's databases are its record: they keep history, join tables,
change in transactions, migrate by number, and are backed up with the store
([ADR 0008](0008-a-context-owns-its-store.md)). A bucket keeps one value per key, at most 256 KiB a value
and 64 MiB a bucket ([ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md) §6).
The record was searched first ("bucket", "mirror", "watch", "written on change", "settings_history"):
ADR 0201, to-be 45 §1 and §2, design 25 §1, issue 326 and research 036.
## Considered Options
### Where the current state lives
1. *Move the records into buckets, and the bus becomes the record.* Refused. A bucket keeps one value,
no history, no join and no transaction, and the store's migrations and backups would have no
equivalent. Research 036's target keeps the store as the owner's record (02-utopia, "Every
important state is watchable").
2. *Events only, from which a reader rebuilds the state.* Refused. EVENTS keeps 7 days and 10 000
messages per subject, so a reader that starts later cannot rebuild a node that has not changed in a
week. The conditions show that a bucket and an event together work.
3. *Copy the store's tables onto the bus by change data capture.* Refused. It copies tables, not records
in the mesh's words, so every column would need a filter to keep a secret off a stream (ADR 0201 §5).
It also adds a second writer of the controller's state, which the writers table forbids (to-be 45
§1), and a new moving part to the core.
4. **A record mirror: a bucket the controller writes from its store after each change commits, with an
event for each write.** Chosen. The store stays the record, and the bucket is a copy that anything
granted may watch.
### Which records are mirrored
1. *Every table research 036 lists under G2.* Refused. Durations (28 870 rows) and data readings
(1 158) are history, which is gap G8 and stays with the effort. Credentials, secrets, bus users,
enrolment tokens and persons never go on a stream (ADR 0201 §5).
2. **Five records: nodes, assignments, walks, builds and settings.** Chosen. They answer what research
036 found a watcher would show first: what each node runs, how far a walk has got and what is being
built (evidence §3). Settings are added because a setting changed with no event at all. Data items
and heals are not mirrored, because what a person needs from them already reaches the bus: the data
conditions of ADR 0233 and the `healer-acted` event.
### How a mirror is kept true
1. *Trust the write after the commit.* Refused. A controller that stops between its commit and its write
to the bucket, or a bus that refuses the write, leaves the bucket stale, and nothing would say so.
That is the mesh's worst fault: work that reports success and did nothing.
2. *Rewrite every key on a timer.* Refused. It is issue 326 on purpose, and it would also hide the
fault rather than say it.
3. **Compare every key with the store on the self-check, raise a condition on any difference, and let a
healer write the store's value.** Chosen. A difference is said before it is repaired, and only the
next comparison says that the repair worked ([ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)).
### What an event carries
1. *The whole value, as a condition event does.* Refused. A walk's value can approach 64 KiB, and EVENTS
would keep a second copy of every write for a week. The bucket already holds the value.
2. *One event kind for every mirror.* Refused. A grant and a consumer filter by subject, so a module that
wants only settings would receive every change of every node and build.
3. **One event kind per mirror, naming the key, the change, the fields that changed and the bucket
revision of the write.** Chosen.
### Where "written only on a change" is enforced
1. *By each owner's care.* Refused. Issue 326 happened under care.
2. **In the writer: the controller's mirror code compares before it writes, and the runtime compares
before it performs a module's `put`.** Chosen. Every bucket in the mesh is written either by the
controller or through the runtime, so the two places cover all of them.
## Decision
### 1. A bucket is written only when what it describes changed
This is a rule for every bucket in the mesh, the controller's and every module's.
- **A write whose value equals the key's current value is not made.** The controller compares before
every write to its own buckets. The runtime compares before a module's `mesh/state.put`, makes no
write when the value is equal, and answers `unchanged` with the key's current revision. This changes
ADR 0201 §4's `put`, and that record carries a note saying so.
- **A value carries no field that changes when the thing it describes did not.** That rules out the time
of the last look, a count of looks, a "checked at" time and evidence that rolls on every look. Those
belong in a verb's answer or a history, not in current state.
- **Checked by:** the controller's tests, where a pass over unchanged records makes no write (the fake
bucket counts writes, as issue 326's test does); the runtime's test that an equal `put` makes no
write; and live, where over ten minutes in which nothing a bucket describes changes, the server's
write count for that bucket (`nats.nats_subscriptions` on `$KV.<bucket>.>`) does not rise.
- **The controller's conditions bucket breaks the second part today**, as the context measured. Moving the
count of looks, the last look and the rolling evidence out of the bucket changes to-be 45 §2's fields
and is tracked by issue 420 (the conditions bucket is rewritten on every observation).
### 2. The controller keeps five record mirrors
A **record mirror** is a bucket the controller writes from one kind of its records, after each change
to that record commits in its store. The store stays the record. The glossary gains the word in the
same change, and says how it differs from the builder's mirrored upstream image.
| Mirror | Bucket | One key per | Event |
|---|---|---|---|
| nodes | `mesh-controller_nodes` | node | `node-changed` |
| assignments | `mesh-controller_assignments` | module on a node | `assignment-changed` |
| walks | `mesh-controller_walks` | walk | `walk-changed` |
| builds | `mesh-controller_builds` | build | `build-changed` |
| settings | `mesh-controller_settings` | settings layer: a module mesh-wide, or a module on one node | `setting-changed` |
What each value holds, and what it leaves out, is in to-be 54. These limits hold for all five:
- **Written by the lease holder alone**, as a row of to-be 45 §1's writers table. A controller without
the lease writes no mirror. Each write is a compare-and-set on the key's revision the writer last saw,
so a former holder that has not yet noticed it lost the lease cannot overwrite a newer value. A refusal
can also be the writer's own write whose acknowledgement was lost; to-be 54 §2 says how the writer tells
the two apart. A value
carries only fields built from the record: no epoch and no time of writing, because the next lease
holder must build the same bytes from the same record. The epoch travels in the event.
- **No secret and no credential**, ever (ADR 0201 §5). Each value is built from a list of named fields,
and a test fails on a field the list does not name.
- **The settings mirror holds no setting's value**, only the names a layer sets, when it was set and by
whom. Some settings still carry a password, which issue 342 tracks moving into the vault, and a bucket
is a stream the bus's snapshots back up (ADR 0235). A digest of the value is refused too, because a
digest of a short password can be guessed back. A watcher asks the `settings` verb for the values.
- **No age on the bucket.** A walk or build that ended is removed from its mirror 14 days after it
ended, by the controller, and the removal is a change with its event. The store keeps it. A removal is
a purge, which leaves one marker for the key; each marker is given a time to live of one day, so
removed keys do not pile up in a bucket with no age.
- **Each value is at most 64 KiB.** A value that would be larger is cut to its summary and says so.
The verb still answers the whole record. Checked by the controller's test, which builds the largest
walk the mesh could make (every module on every node) and expects a value under the cap, marked cut.
- **Declared as the controller's state** in its manifest, so a module names one in `reads` and receives
its grant as ADR 0201 gives it. The controller asserts the options at start, as it does for its other
buckets, and a test holds the declaration and the assertion equal.
- **The controller never reads a mirror to decide anything.** Only the audit in §4 reads it. A mirror
that fed the controller's own decisions would become a second record. Checked by a test over the
controller's packages, which fails when any package but the audit's reads a mirror bucket.
### 3. Every write to a mirror is one event
The write is made first. The event follows on the controller's seat (`mesh.seat.mesh-controller.event.<kind>`)
and carries the key, the change (`added`, `changed` or `removed`), the names of the fields that changed,
the bucket revision of the write, the epoch, and whether the healer of §4 made it. `setting-changed`
also carries who changed the setting, as the runtime proves the caller. A failed write sends no event.
**The bucket is the authority and the event is a notice.** An event can be lost after its write
succeeded: the controller stops, or the bus refuses the publish. A watcher that must not miss a change
watches the bucket, whose watch delivers every revision. The writer tries a refused event once more,
then counts it as lost. D17's verdict carries the count since the last run, and any lost event raises
`mirror-event-lost` (warning), which clears on the next run that counts none. A controller that stops
between a write and its event loses that event without a count; the bucket still holds the change.
The existing events keep their meaning: `plan-moved` says a walk moved, `built` says a build ended,
`applied` says a node applied its declaration. A mirror's event says that the record now reads
differently. Some changes are therefore said twice, once as the act and once as the record.
**Checked by:** live, the count of writes to `$KV.mesh-controller_<mirror>.>` and the count of that
mirror's events in EVENTS rise by the same number over any window; and the controller's test, which
counts both over a run of changes.
### 4. A mirror is audited against its store, and a difference is said before it is repaired
- **The probe D17** (the mirrors match their store) runs with the self-check, every five minutes. For each
mirror it builds the value of every record from the store and compares it, byte for byte, with the
bucket's key. A key missing, a key whose value differs and a key with no record are each a difference.
A key counts as different only when it differs on two reads 30 seconds apart and its record did not
change between them, so the probe never races the write that follows a commit. A record that keeps
changing for longer is not judged in that run, and the verdict names it.
- **Any difference raises `mirror-drifted`**, a warning keyed `core.mirror.<mirror>.drifted`. Its
evidence names the keys and what kind of difference each is. The second read inside the run is its
second look, so it is raised at once (to-be 45 §4). A key past its 14 days is a key with no record, so
the removal of §2 is checked here too.
- **The healer H6 writes the store's value** for each key the condition names, removes a key with no
record, and sends each write's event marked as the healer's. It may do this twice per mirror in 24
hours. Then the condition's resolver becomes the operator and its severity urgent, because a mirror that keeps drifting means
the controller's write path is broken. The next run of D17 clears the condition when it finds no
difference.
- **When the controller takes the lease, it runs D17's first read before it serves**, and the second read
while it serves. A difference found then is
raised like any other, because it means a controller stopped between a commit and its write.
- **A mirror created empty** by this start is filled whole before the controller serves, and that fill
is not a difference.
**The bound a reader may rely on:** for a record that was quiet for 30 seconds, a mirror differs from
its store for at most the self-check's interval plus those 30 seconds before a condition says so.
## Consequences
- Any module granted a mirror can react to a node, an assignment, a walk, a build or a setting changing,
by watching the bucket or hearing the event, without calling a verb. Research 036's views, once
people can reach the bus (G1), read them the same way.
- A setting's change is said for the first time. Who made it is the caller as the runtime proves it,
which reads `unproven` until the runtime tells a module who called (issue 365 in the issue tracker).
- The store gains when and by whom a settings layer was set, as a numbered migration of `inventory` and
in its consolidated schema, built under issue 419.
- **What got harder:** the controller writes each change twice, to the store and to the bucket, and
keeps a list of fields per mirror. Every new column of a mirrored table is a choice to mirror it or
not, made in the list, where the test sees it.
- The self-check reads every mirrored record every five minutes. At the sizes measured (a few hundred
keys outside builds, a few thousand builds in 14 days) that is well inside the probe's bound of 90
seconds, which covers its second read; the
probe's own duration is in its verdict, so a growing mirror is seen.
- The conditions bucket stays as it is until issue 420 is fixed, and so breaks the rule of §1 in the
meantime.
- The runtime's `put` gains one read before each write.
## References
- [Research 036](../01-RESEARCH/036-everything-over-nats/00-overview.md), gap G2 and §3 of its evidence.
- [Issue 326](../04-ISSUES/326-mesh-delivery-writes-a-delivery-whether-or-not-it-changed/00-report.md),
mesh-catalog #187 and #188.
- [ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)
(a module's state in buckets), which this record extends.
- [ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md) (healers),
[ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md) (`plan-moved`).
- [To-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §1, §2, §4 and §7, which gain the
mirrors' writer row, probe and healer.
- [To-be 54](../03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md), the design.
- Issue 419 (the build) and issue 420 (the conditions bucket), in the issue tracker.
03-DESIGN/01-to-be/25-the-bus-on-nats.md
+9 -1
View File
@@ -8,8 +8,9 @@ code:
- mesh-catalog modules/nats (to be written)
- mesh-sdk src (the protocol's NATS binding, step 3)
- mesh-tools node-tools/internal/bus (a module's state, ADR 0201)
updated: 2026-10-08
updated: 2026-10-10
decisions:
- 02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md
- 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md
- 02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
@@ -73,6 +74,13 @@ and uses, and the subjects, streams, consumers and permissions all follow from t
`subscribe`, `close` — is the whole surface, and it does not change
([ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)).
**Added 2026-10-10** ([ADR 0289](../../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md)): the controller's own records join
the bus as **record mirrors** — nodes, assignments, walks, builds and settings, each a bucket written from
the controller's store only when the record changed, with one event per write, and audited against the
store on every self-check run. Designed in [to-be 54](54-the-controllers-records-on-the-bus.md). The
rule it states for every bucket, that a write is made only when what the bucket describes changed, holds
for a module's state too.
## 2. Subjects
NATS addresses everything by subject. The mesh's subject space is one tree, and every account's
03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
+10
View File
@@ -4,6 +4,7 @@ status: in-progress
code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
updated: 2026-10-10
decisions:
- 02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md
- 02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md
- 02-DECISIONS/0271-a-send-held-for-the-buss-planned-step-is-told-to-the-operator-at-once-and-a-change-that-needs-it-says-so-before-it-merges.md
- 02-DECISIONS/0265-a-verb-that-reads-is-answered-on-the-serving-controllers-own-connection-and-every-connection-says-what-it-is.md
@@ -92,6 +93,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps |
| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads |
| the record mirrors: nodes, assignments, walks, builds, settings ([ADR 0289](../../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md), *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why |
A bus subject two components may publish on is refused when the controller composes grants, unless
@@ -175,6 +177,11 @@ controller's suite raises, which fails a kind without words.
- **Nobody resolves a condition by hand.** It clears when the signal returns or the probe passes.
- **A condition cleared and raised again within ten minutes** reopens with its count increased; it is
not a new message.
- *Added 2026-10-10, [ADR 0289](../../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md) §1:* a bucket is written only when what it
describes changed, and a value carries no field that changes on every look. This bucket breaks the second
part: `last observed`, the count of observations and the rolling evidence rewrite an open condition on every
observation (131 647 writes in 3 days 21 hours for a dozen open conditions, measured 2026-10-10). Moving them
out of the bucket is tracked by issue 420 (the conditions bucket is rewritten on every observation).
- **The verbs:** `conditions` (open ones, filtered by scope, severity or machine), `conditions show
<key>`, `conditions silence <key> --for <duration> --why <text>`, `conditions history`.
- **The events**, emitted by the controller on its own subjects: `condition-raised`,
@@ -290,6 +297,8 @@ producer that wrote it.
| D12 | every consumer of a provision that keeps its data is bound where it was last sent, or moves by a pin (`binding-kept`, `binding-moving`, `binding-moved`) | ADR 0232 |
| D13 | every item of data a machine declares is measured, is there, holds what it held, is written where it says it is, is backed up within its bound or sits on healthy redundant storage, and is no empty replacement of a copy kept elsewhere; an irreplaceable or valuable item a machine no longer declares is retired, not forgotten. Each machine's `node-backup` holder is asked `backed-up`; every provider of kept consumer data `provisioner_retirement` (with each held consumer's size) — see §2 for the kinds | ADR 0233, issue 273 |
| D15 | no user of the bus had its connection dropped more than twelve times in the last hour (*provisional*): the bus module's `nats_closed_connections`, from the server's record of closed connections; `bus-reconnects` (ADR 0265) | 327 |
| D16 | every delivery runs healthy on every node within its class's budget (five minutes for a leaf module, ten for a core one); `delivery.<class>.over-budget`, naming the delivery and its longest phase. *Listed 2026-10-10*: the controller already runs it under this id | ADR 0282 |
| D17 | every record mirror holds exactly what its store says: no key missing, none differing, none without a record, a difference counted only when a second read 30 seconds later still finds it and the record did not change between them; `mirror-drifted` (H6), and `mirror-event-lost` for an event lost after its write. *Added 2026-10-10* ([to-be 54](54-the-controllers-records-on-the-bus.md) §4) | ADR 0289 |
| DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 |
| H-* | the health probes of §8, run for every core component on every machine: H-controller, H-engine, H-tools, H-bus (as built, [ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)) | rule 8 |
| DG | every build that failed its gate, and every core build a witness put back, is said — `build.<module>.<machine>.rolled-back`, `core.<component>.<machine>.<outcome>`, `rollback-failed` — until a newer build passes or the reports stop saying it (ADR 0236) | rule 8 |
@@ -439,6 +448,7 @@ operator.
| H3 | a seat holder that does not answer (D3, `holder-silent`), or a consumer the mesh expects missing (D6, S9, `consumer-lost`) | assert the bus's objects again — the assertion every send makes (208) | once per object per hour |
| H4 | `consumer-behind` (D6) on a consumer the stream table marks *resettable* — the controller's own events consumer alone | `broker consumer-reset` (248) | once a day per consumer, then condition |
| H5 | `provider-failing` with the administrator refusing the mesh's secret | ADR 0224 §5 (exists, in the module) | as ADR 0224 |
| H6 | `mirror-drifted` (D17) | write the store's value for each key the condition names, purge a key with no record, each write said as its mirror's event marked `repaired`; never writes the store (ADR 0289, *added 2026-10-10*) | twice per mirror in 24 h, then resolver `operator`, urgent |
As built ([ADR 0231](../../02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)): the registry is compiled into the controller with a test generated
from it; each act is begun in the controller's store before it is made, under the lease, and the
03-DESIGN/01-to-be/54-the-controllers-records-on-the-bus.md
+246
View File
@@ -0,0 +1,246 @@
---
layer: to-be
status: designed
code: []
updated: 2026-10-10
decisions:
- 02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- 02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md
- 02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md
---
# 54 — The controller's records on the bus
**What each node runs, how far each walk has got, what each build is doing and what each setting holds
can be watched on the bus, and what is watched never says something the controller's store does not**
([ADR 0289](../../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md),
from gap G2 of [research 036](../../01-RESEARCH/036-everything-over-nats/00-overview.md)).
The design follows the controller's conditions ([to-be 45](45-a-core-that-cannot-fail-silently.md) §2):
current state in a bucket, one key per thing, and an event for each change. It adds what the conditions
do not have: a check that the bucket says what the store says.
## The parts
```
a verb or a reconcile changes a record
│
▼
the controller's store (inventory) ── commit ──► the mirror writer ── equal? ──► nothing
▲ │ differs
│ ▼
│ bucket mesh-controller_<mirror> (put, or purge on removal)
│ │ written
│ ▼
│ event <kind>-changed on the controller's seat
│
D17, every self-check run ── reads the store and the bucket, compares ──► mirror-drifted ──► H6
```
- **The store** stays the record. Nothing in this design changes what the controller decides from, or
where it keeps it.
- **The mirror writer** is one place in the controller that every change to a mirrored record passes
through after its commit. It builds the record's value, compares it with the value the key holds, and
writes only a difference.
- **The bucket** holds the value. **The event** says that it changed.
- **The audit** (probe D17) and **its healer** (H6) keep the bucket honest.
## 1. The five mirrors
Each value is built from a named list of fields. A field not on the list is never in the value. A value
holds only what is built from its record: no epoch, no time of writing and no time of the last look,
because any lease holder must build the same bytes from the same record, and the audit must too. Times
that belong to the record itself, such as when a walk was opened or a resource's state began, are fields
like any other. The epoch and the time of the write travel in the event.
### nodes — `mesh-controller_nodes`, key: the node's name
| Holds | Leaves out, and why |
|---|---|
| the node's name and role; its operating system, kernel, architecture and hardware summary from its last profile | the profile's whole document, which belongs to the node-engine and can be large; the `node` verb answers it |
| the node-engine and tool runner builds it reports running | — |
| whether it said it is asleep or shut down (ADR 0211) | its heartbeat times, which change on every beat while nothing else does; a silent node is the condition `machine.<node>.silent` |
| the sequence of the last declaration sent and of the last one applied, and whether that apply was refused | the time a report was heard |
| per resource it watches: module, resource, kind, state and since when (ADR 0240) | the reason text, the streak and the restart count, which change on every look (the reason itself counts looks today) |
| per network check (ADR 0241): its state and since when | when it was last checked |
### assignments — `mesh-controller_assignments`, key: `<node>.<module>`
| Holds | Leaves out, and why |
|---|---|
| the module and the node, and since when it is assigned | — |
| whether the assignment is made or pending (ADR 0261), and what ended a pending one | — |
| the build the node was last sent for it and the build its node reports running | — |
| the seats it holds there, and the provisions it is bound to | the bindings' credentials, which are secrets |
### walks — `mesh-controller_walks`, key: the walk's id
| Holds | Leaves out, and why |
|---|---|
| the repository and commit, or the batch of merges it walks (ADR 0276) | — |
| its state (building, rolling, done, failed), its current tier, and each tier's modules and nodes with each step's state | the build and apply logs, which the `plans` and `builds` verbs answer |
| when it was opened, when it last moved, when it ended, and its note | — |
A walk is removed from the mirror 14 days after it ended.
### builds — `mesh-controller_builds`, key: the build's id
| Holds | Leaves out, and why |
|---|---|
| the repository, ref and commit asked; the module once known | — |
| the machine that builds it, its state, and its outcome with the artifact's digest when there is one | the build's output, which the `builds` verb answers |
| its gate on its first node, when there is one (ADR 0236) | — |
| when it was asked and when it ended | — |
A build is removed from the mirror 14 days after it ended. A build that failed before its module was
known is kept by its id like any other.
### settings — `mesh-controller_settings`, key: `<module>`, or `<node>.<module>` for one node's layer
The mesh-wide layer is keyed by the module, and one node's layer by `<node>.<module>`, the order the
assignments mirror uses.
| Holds | Leaves out, and why |
|---|---|
| the names of the settings the layer sets | the values, and any digest of them, since a digest of a short password can be guessed back: some settings still carry a password ([issue 342](../../04-ISSUES/342-a-password-is-taken-as-a-setting-and-outranks-the-vaults/00-report.md)), and a bucket is a stream that the bus's own snapshots back up (ADR 0235); the `settings` verb answers the values to whoever may read them |
| when it was set, and who set it | the layer's history, which `settings_history` keeps and the verb answers |
A watcher therefore learns which settings changed (the event names them, from the layer's previous row,
which `settings_history` keeps), who changed them and when, and asks the verb for the values. A layer set
again is a change, because its time of setting moved. The values join the mirror only once no setting
holds a secret, by a decision of its own; issue 342 tracks the move of every password out of settings.
The event `setting-changed` carries who changed the setting, as the runtime proves the caller, or
`unproven` while the runtime cannot say (issue 365 in the issue tracker, the tool runner not told who
called).
## 2. Writing
- **Only the lease holder writes.** The mirror writer refuses to write without the lease, and so does
every other writer of the controller's buckets (to-be 45 §6). The five mirrors are rows of to-be 45
§1's writers table, so a grant that lets anything else publish under them is refused at composition.
- **After the commit, never before.** A value written before its commit could describe a change the
store then rolled back. The writer is handed only a committed change; a rolled-back one never reaches it.
- **Compare-and-set.** Each write names the key's revision the writer last saw. A refusal has more than
one cause: a former lease holder that wrote in between, or the writer's own earlier write whose
acknowledgement was lost. After a refusal the writer reads the key's value and revision again. If the
bucket already holds the value it meant to write, that was its own write, and it only takes the new
revision. Otherwise it checks that it still holds the lease: if it does, it writes again against the
revision it just read; if it does not, it stops writing every mirror, as every writer without the lease
does. Each refusal is logged and counted in D17's verdict.
- **H6 writes through the writer**, so the revision the writer keeps is current after a repair.
- **Only a difference is written.** The writer keeps the last value it wrote for each key, filled from the
bucket when the controller takes the lease, and compares the new value byte for byte. The encoding is
fixed (fields in their list's order, maps sorted), so the same record always encodes the same.
- **Who set a settings layer** is kept in the store beside when (`settings` has the time and not the
caller today): a numbered migration of `inventory` and the consolidated schema, in the same change.
- **A removal is a purge** of the key, said as a `removed` event. A purge leaves one marker for the key.
Each purge gives its marker a time to live of one day (a per-message time to live, which the bus server
has offered since release 2.11 and the bucket enables), so the markers of removed walks and builds do
not pile up in a bucket that has no age. The controller's one-minute tick removes walks and builds 14
days after they ended.
- **A write that fails** sends no event, is logged, and is left to the audit. The writer does not retry
in a loop: a retry loop against a refusing bus is the reconnect loop of issue 327 in another place.
## 3. The events
Each mirror has its own kind on the controller's seat, `mesh.seat.mesh-controller.event.<kind>`:
`node-changed`, `assignment-changed`, `walk-changed`, `build-changed`, `setting-changed`. They are added to
the seat's emits, as every event of the controller is.
Each carries the mirror, the key, the change (`added`, `changed`, `removed`), the names of the fields that
changed, the bucket revision of the write, the epoch, the time of the write, and `repaired` when the
healer H6 made the write. A watcher that wants the value reads it from the bucket at that revision or later.
**One write is one event, and one event is one write.** The event is sent after the write succeeds, so
there is no event for a value that is not in the bucket.
**The bucket is the authority, and the event is a notice.** An event can be lost after its write: the
controller stops, or the bus refuses the publish. A watcher that must not miss a change watches the bucket,
whose watch delivers every revision. The writer tries a refused event once more, then counts it as lost.
D17's verdict carries the count since the last run, and any lost event raises `mirror-event-lost`
(warning), cleared by the next run that counts none. A controller that stops between a write and its event
loses that event uncounted; the bucket still holds the change, and the live comparison of write and event
counts in §6 shows the gap.
## 4. The audit: D17 and H6
**D17, the mirrors match their store**, is a probe in the self-check's registry (to-be 45 §4). Every run
does this for each mirror:
1. It reads every record the mirror should hold from the store (for walks and builds, those open or ended
within 14 days) and builds each value as the writer would.
2. It lists the bucket's keys and reads their values.
3. It compares. A key missing, a value that differs, and a key with no record are each a candidate. A
key past its 14 days is a key with no record, so the removal of §2 is checked here.
4. After 30 seconds it reads each candidate's record and key again. A candidate is a difference only
when it still differs and its record did not change between the two reads. So the probe never races
the write that follows a commit, and the second read is its second look (to-be 45 §4). A record that
kept changing is not judged in that run and is named in the verdict.
5. Its verdict names, per mirror, the keys compared, the differences, the records not judged, the events
counted lost and how long it took. Its bound is 90 seconds rather than a probe's usual 30, to hold the
second read.
D17 is the next free number: D16 is the delivery-time probe the controller already runs (the
`over-budget` condition, ADR 0282), listed in to-be 45's table in the same change.
**A difference raises `mirror-drifted`**, warning, keyed `core.mirror.<mirror>.drifted`, at once. Its
evidence lists up to ten differing keys, each with what kind of difference it is, and the count of the
rest. Its plain words say that what other modules see of this record on the bus was out of date and is
being corrected, and that nothing is needed from the operator.
**H6 repairs it** (to-be 45 §7): for each key the condition names, it writes the store's value, or purges
a key with no record, and each write sends its event with `repaired`. Its budget is twice per mirror in
24 hours. Then the condition's resolver becomes the operator and its severity urgent, because a mirror
that keeps drifting means the writer is broken, and an issue is the next step. H6 writes only a mirror,
never the store, so it neither deletes nor recreates data.
**When the controller takes the lease**, it fills the writer's memory (each key's value and revision) from
the buckets and runs D17's first read before it serves; the second read follows while it serves, and
raises what it confirms. So a change of lease does not wait 30 seconds for the audit. Because a value
holds no epoch, a change of lease alone is never a difference. A bucket created empty by this start is filled whole first; that fill is not a
difference. Anything else found then is raised, because it means a controller stopped between a commit
and its write.
## 5. Readers
A module that wants a mirror names it in `reads` as `mesh-controller.<mirror>` (ADR 0201 §1), and its
grant follows. The five are declared under `state` in the controller's own manifest, with the same
options the controller asserts at start (one value per key, no age, 64 KiB a value, 64 MiB a bucket); a
test holds the declaration and the assertion equal.
The first readers are expected to be mesh-delivery, which today asks `plans` and reads `plan-moved`, and
the operator's channel, once it shows more than conditions. People's views are research 036's, which
remain open there.
## 6. How it is checked
| Rule | Checked by |
|---|---|
| A pass over unchanged records writes nothing | the controller's test with a fake bucket that counts writes and fails on any |
| A value holds only its listed fields, and never a secret | the controller's test per mirror, which builds every value from a store holding every column filled and fails on a field not on the list |
| One write, one event | the controller's test over a run of changes, counting both; live, the server's write count for `$KV.mesh-controller_<mirror>.>` (`nats.nats_subscriptions`) and the count of that mirror's events in EVENTS rise by the same number over any window |
| Only the lease holder writes | the writers table's compiled test (to-be 45 §1); D5 |
| The bucket matches the store | D17 on every self-check run; its own test, which changes a key behind the writer's back and expects `mirror-drifted`, then H6's write, then the condition cleared on the next run; and a test that a change of lease alone raises nothing |
| Quiet means no writes | live: over ten minutes in which `plans`, `nodes`, `builds` and `settings` show no change, no mirror's write count rises |
| A watcher sees a change | live: a setting changed through `settings` is seen by a watch on `mesh-controller_settings` and as one `setting-changed` within a second |
| A value is under its cap, or cut and marked | the controller's test, which builds the largest walk the mesh could make (every module on every node) |
| Only the audit reads a mirror | a test over the controller's packages, which fails when any package but the audit's reads a mirror bucket |
| A write follows the commit | the controller's test that a rolled-back transaction makes no write and sends no event |
| Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals |
| A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run |
| A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set |
| A lost acknowledgement is no conflict | the controller's test in which the bus stores a write and drops its acknowledgement: the next write to that key succeeds and no condition is raised |
| The runtime makes no write for an equal `put` | the runtime's test over a real bus, which puts the same value twice and expects one write and the answer `unchanged` |
## What it leaves out
- **History.** A mirror is current state. What happened beyond the 14 days a walk or build is kept, and
beyond EVENTS' week, is gap G8 of research 036, which remains open there.
- **People reaching the mirrors**, through a WebSocket and an auth callout, is gap G1, open in the same
effort.
- **The licence manager's state, and other modules' databases**, are gap G6: each module mirrors its own,
by ADR 0201, under the rule of ADR 0289 §1.
- **The conditions bucket's per-look writes** are tracked by issue 420 (the conditions bucket is
rewritten on every observation).
03-DESIGN/01-to-be/README.md
+1
View File
@@ -53,6 +53,7 @@ document is written and this one's status becomes `implemented`.
| [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | [ADR 0251](../../02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md), [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md) |
| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **In progress.** A node shares one folder at a time through `nfs-server`, which provides it as a provision to the private network only, mapping every client to the folder's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | [ADR 0263](../../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md), [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md), [ADR 0092](../../02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md) |
| [`53-the-issue-tracker.md`](53-the-issue-tracker.md) | **Designed.** The mesh keeps its issues in one module, `issues`, on the control-node: every number from one counter advanced by compare-and-set, the cycle's issue rules refused at the write, an event per change; hq keeps a folder only for a design or governance issue | [ADR 0275](../../02-DECISIONS/0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md) |
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, walks, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | [ADR 0289](../../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md) |
## Not yet written