diff --git a/02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md b/02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md new file mode 100644 index 0000000..16e5e7f --- /dev/null +++ b/02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md @@ -0,0 +1,136 @@ +--- +topic: what runs on it +status: accepted +date: 2026-10-06 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0110-a-seat-is-a-module-assignment-from-a-closed-set.md +--- + +# 232. A binding to a consumer's data moves only by a person + +## Context + +**A rule written for the resolver moved a machine's databases.** [Issue 258](../04-ISSUES/258-every-machine-bound-the-resolver-to-itself/00-report.md) +found that a machine running its own provider of a mesh-wide provision bound its consumers to that +provider, even when the mesh's seat for the provision was held elsewhere. Its fix made the seat's +holder, or a pin naming another machine, win over the local provider. That is right for the mesh's +resolver: every resolver gives the same answer, so a consumer moved between two loses nothing. + +The fix applied to every seat that delivers a provision. The store's seat delivers the relational +database. On the home server, which runs its own store with five applications' databases in it, the +fix's first push re-bound all five to the store on the control node, which holds the seat. That store +did what a provider does for a new consumer: it made each one an empty database. The applications +started, ran their first migrations, and served empty data for about twenty hours. Nothing warned +([issue 273](../04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md)). +Nothing was lost, only because the old store kept everything. + +The incident has two parts: + +1. **Issue 258's rule applies to two kinds of provision as if they were one.** It does not say which + kind it is for. +2. **Nothing in the mesh knows where a consumer's data is.** A resolution answers one question: + *which provider would I choose now?* Every input to that answer can change under a consumer without + anybody meaning to move it: the seat's holder (a handover, ADR 0131), a pin added or removed, a + provider assigned or unassigned. Only the pair secrets in the store held a trace of the old + binding, and only because a new pair was minted beside it. + +## Considered Options + +1. **Revert issue 258's fix.** Rejected. It was correct for the resolver, and the resolver would + break again. +2. **Make the store's seat not deliver its provision.** Rejected. That fixes one seat, and the next + seat that delivers a provision which keeps data repeats the incident. +3. **Refuse any resolution that changes a binding.** Rejected. It would refuse the resolver's moves, + which are the point of 258, and a person moving a database on purpose would have no way to do it. +4. **Know which provisions keep their consumers' data. For those, keep each consumer where it was + last sent, say every move the mesh would have made, and let only a pin move it.** Chosen. + +## Decision + +**1. An offer says whether it keeps its consumers' data.** The field is `keeps-consumer-data` on +`provides`. + +- If unsaid, a provider that **grants** each consumer a credential of its own keeps that consumer's + data. The grant makes an account (a role and its database, a key and its bucket, a client), and + what the consumer writes under it stays with that provider. +- A provider that grants nothing keeps nothing of anybody's. Examples: the resolver, a CA, the + artifact store, a route. +- The property belongs to the provision's name, as brokering does (ADR 0009): if any provider says + the provision keeps data, it keeps data. + +**2. Issue 258's rule is for provisions that keep nothing.** For a provision that keeps data, the +seat's holder on another machine does **not** overrule a provider beside the consumer. A pin naming +another machine still does, because a pin is a person. + +**3. Where each such consumer was sent is recorded, and a resolution keeps it there.** One row per +consumer and provision is written when a declaration carrying the binding is sent. The provider is +recorded by name, so a provider's machine leaving the mesh does not erase where the data is. + +- **Recorded binding, different provider chosen, no pin naming the new one:** the recorded provider + keeps answering. The move is said. +- **Recorded provider no longer provides the provision:** the machine's set is **refused**, naming the + pin that would confirm the move. It is never answered by the provider chosen instead. This is ADR + 0009's stance, already applied to a pin naming a provider that is gone. +- **New consumer with no record:** it binds as resolved, and is recorded when first sent. + +**4. Only a pin moves it.** The pin names the provider, for that machine and provision. The send +that carries the move records the new provider and keeps where it was. The data is moved by a person +before the push; the mesh does not move data. + +**5. Nothing is silent.** A kept move is printed by the push that composed it and raised at once as +an **urgent** condition: + +> would move X's P from A to B — its data is on A; kept there. `pin …` to confirm a move (and move +> the data first) + +A self-check probe ([to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §4) +checks every machine every run. It raises: + +- a kept move, as urgent; +- a pinned move not yet sent, as a **warning**, so the data goes first; +- any consumer about to be sent another provider than the one on record with no pin naming it, as + **urgent**. The resolver makes this impossible; the probe exists for the day it is not. + +## Consequences + +- **The first push after rollout records what each machine is bound to then.** A consumer moved + silently before the rollout and never moved back would be recorded where it was moved to. The + rollout therefore starts by checking that every binding to data points where the data is. On + 2026-10-06 every one did: the five moved consumers had already been pinned back. +- **A pin is per machine and provision, not per consumer.** Pinning one consumer of a machine + elsewhere pins all its consumers of that provision. That is how pins have always worked, and the + warning names every consumer that would move. +- **Unassigning a store beside its consumers now refuses their machine** until a person pins them + elsewhere. Before this, they moved silently to whichever store answered. +- **A retired consumer's binding stays on record** ([ADR 0230](0230-a-consumer-the-mesh-stops-asking-for-is-retired-and-deleted-only-by-a-person.md)). + Assigned again, the consumer returns to the provider that kept its retired data. +- **No module definition changes.** Every provider in the catalogue that keeps data already grants + credentials, and none that keeps nothing does. A definition states the field only when the default + is wrong for it. + +## How it is checked + +| What | Checked by | +|---|---| +| the incident: a machine running its own store and its consumers, the store's seat held elsewhere — the consumers stay on their own store, and the resolver beside them still follows its seat | mesh-controller `internal/catalogue/bound_test.go`, and `cmd/mesh-controller/bindings_test.go` through the real stores; both fail without the fix | +| a recorded consumer is kept when the seat's holder changes, and the move is said with its pin | `bound_test.go` | +| a pin moves it; a gone provider, or a store beside it unassigned, is refused and never answered elsewhere | `bound_test.go`, `bindings_test.go` | +| an offer's `keeps-consumer-data` is read and written, and unsaid follows the grant | `bound_test.go` | +| a binding is recorded on send, a move keeps where it was, and a provider's machine leaving keeps the record | `internal/inventory/bindings_test.go` | +| a push raises a kept move at once; the probe says kept (urgent), pinned and not yet sent (warning), and unasked (urgent) | `bindings_test.go` | +| live, after rollout | the self-check passes its binding probe on every run, and `conditions` holds no `binding-kept` on a mesh nobody is changing | + +## References + +- [Issue 273](../04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md): + the incident. +- [Issue 258](../04-ISSUES/258-every-machine-bound-the-resolver-to-itself/00-report.md): the rule + this record limits. +- [ADR 0110](0110-a-seat-is-a-module-assignment-from-a-closed-set.md): a seat's holder answers. This + record extends it, because the holder now answers only where nothing is kept. +- [ADR 0009](0009-modules-and-the-graph.md): refuse rather than guess. +- [ADR 0230](0230-a-consumer-the-mesh-stops-asking-for-is-retired-and-deleted-only-by-a-person.md): + the provider keeps a consumer's data until a person deletes it. +- mesh-controller PR #86: `catalogue/bound.go`, the `binding` table (migration 0071), and the + self-check's binding probe. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 334a0de..eaa86a3 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -330,6 +330,7 @@ python3 00-META/checks/index.py fail if stale - **0220** — [What a machine asks needs its uplink held, and the retired resolver pieces go](0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md) - **0225** — [A consumer's identity is bounded by the provision it requires, judged before merge, and never refuses its provider](0225-a-consumers-identity-is-bounded-by-the-provision-it-requires.md) - **0228** — [A value given by hand lives only until its module's first good start](0228-a-value-given-by-hand-lives-only-until-its-modules-first-good-start.md) +- **0232** — [A binding to a consumer's data moves only by a person](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md) ### How it is built diff --git a/04-ISSUES/258-every-machine-bound-the-resolver-to-itself/00-report.md b/04-ISSUES/258-every-machine-bound-the-resolver-to-itself/00-report.md index 32f3c5f..9525d34 100644 --- a/04-ISSUES/258-every-machine-bound-the-resolver-to-itself/00-report.md +++ b/04-ISSUES/258-every-machine-bound-the-resolver-to-itself/00-report.md @@ -46,6 +46,14 @@ This is checked by three resolve tests in mesh-controller: The tests fail without the fix. +**The fix broke something else** ([issue 273](../273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md)). +It applied to every seat that delivers a provision, and the store's seat delivers the relational +database. Its first push re-bound every database consumer on a machine running its own store to the +store holding the seat. Each was made an empty database there, and nothing said so for twenty hours. +[ADR 0232](../../02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md) limits +this rule to provisions that keep nothing of their consumers'. For one that keeps data, only a pin +moves a consumer away from a provider beside it. + ## Verified 2026-10-05, after the fix rolled out: all four machines' resolver configuration names the anchor's diff --git a/04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md b/04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md new file mode 100644 index 0000000..dfd9f74 --- /dev/null +++ b/04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md @@ -0,0 +1,113 @@ +--- +status: located +opened: 2026-10-06 +located-in: [mesh-controller] +fixed-by: novox/mesh-controller PR #86 +amended-design: +--- + +# 273. A rule for the resolver moved a machine's databases + +## Symptom + +On 2026-10-06 the five applications on the home server were found serving empty data. Each is a +consumer of the relational database provision and has its own database. The home server runs its +own store, and that is where their data is: one application's hundreds of tables, another's +hundred-and-seventy thousand rows, a workflow engine's workflows, an agent server's agents and +messages, and a game's players. + +Their binding files named the store on the control node, which holds the mesh's store seat. That +store had each application's database, made new and empty on 2026-10-05 at 19:16 UTC. The first +application ran its first migration there at 19:17. All five then ran against the empty databases +for about twenty hours. + +Nothing warned: + +- the push that moved them reported success; +- `status` read them as applied and current; +- no condition was raised. + +The real data stayed untouched on the home server's store. Pinning the home server to its own store +for the provision, then pushing, put every application back on its data. **No data was lost.** + +The evidence is in the store's pair secrets: each of the five consumers holds one credential from +the home server, minted in the days before, and a second from the control node. All five second +credentials were minted in the same second, 19:16:21 UTC on 2026-10-05, two minutes after the +controller carrying [issue 258](../258-every-machine-bound-the-resolver-to-itself/00-report.md)'s fix +was merged and rolled out. + +## Cause + +Issue 258's fix made the holder of a provision's mesh seat on another machine, or a pin naming +another machine, answer **before** a provider on the consumer's own machine. It was written for the +resolver, which every holder answers alike. But the check ran for every provision a seat delivers, +and the store's seat delivers the relational database. On a machine running its own store beside its +consumers, every consumer was re-bound to the seat's holder. The holder did what a provider does for +a consumer it has not seen: it made an empty database. + +Two gaps let this happen, and only the first was the change: + +1. **The rule did not know which provisions it was for.** Nothing in a module definition, or in the + resolver, separated a provision every provider answers alike from one whose provider keeps what + the consumer wrote. +2. **Nothing knew where a consumer's data was.** A resolution chooses a provider from inputs that + change: seat holders, pins, assignments. A consumer bound to its data was re-bound like any other, + with nothing to compare against and nothing to say. Before this incident, a handover of the + store's seat, or a pin removed, could have done the same. + +## Other machines + +All four machines were checked on 2026-10-06, read-only. Each machine's planned binding files were +compared with the pair secrets in the store and with where each database is. **No other machine's +consumers were moved.** + +- Every other database consumer, and every consumer of the document store, object store, mail, + identity provider, package registry and broker topics, has one credential, from the provider it is + bound to now. +- The control node's consumers were never at risk: the control node holds the seat itself. + +Three findings remain on the home server: + +- **A sixth consumer there** first asked for a database after the change, at 00:28 on 2026-10-06. It + was bound to the control node from its first send. Its data, made before the mesh bound it, is on + the home server, and it is bound there now through the pin. +- **The control node's store still holds the six databases made for the home server's consumers.** + Four of them grew past an empty database's size while the applications ran against them. What was + written there in those twenty hours should be reviewed before they are retired + ([ADR 0230](../../02-DECISIONS/0230-a-consumer-the-mesh-stops-asking-for-is-retired-and-deleted-only-by-a-person.md) + makes that a person's act). +- **The control-node pair credentials for those consumers are still on record**, though nothing is + sent them. + +## Fix + +The fix is mesh-controller PR #86, decided by +[ADR 0232](../../02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md): + +- An offer says whether it keeps its consumers' data. If it does not say, a provider that grants + each consumer a credential does. +- For such a provision, issue 258's seat rule no longer overrules a provider beside the consumer. A + pin still does. +- Where each such consumer was sent is recorded. A resolution that would bind it elsewhere keeps it + where it is, and raises an urgent condition naming the move and the pin that would confirm it. If + the recorded provider is gone, the machine's set is refused; the consumer is never moved to another + provider. +- A self-check probe raises: + - every kept move; + - every pinned move not yet sent, so its data is moved first; + - any move that slips past the resolver. + +## How it is checked + +The resolver's tests reproduce the incident: a machine running its own store and its consumers, with +the store's seat held on another machine. Every consumer stays on its own store, and the resolver +beside them still follows its seat. A second test runs the same incident through the controller's +real stores. It checks: + +- the binding is recorded on send; +- the probe is silent; +- the store taken away is refused; +- a pin moves the binding and is said as a warning; +- once sent, it is quiet again. + +Both tests fail without the fix. The rest of the checks are listed in ADR 0232.