diff --git a/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md b/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md new file mode 100644 index 0000000..bce1367 --- /dev/null +++ b/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md @@ -0,0 +1,134 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-29 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0010-delivery.md +--- + +# 143. A consumer verifies the grant it is given + +## Context + +A **grant** is what the mesh writes on a consumer's machine so it can reach a provider. The real one +the forge receives for its database, as it arrives: + +``` +provision postgres-database +at +port the machine port the provider is published on +as the role the provider created for this consumer +``` + +with the credential sealed in a separate file. Four facts and a password, and they are the whole +mechanism by which anything in the mesh reaches anything else. + +**The mesh asserts that claim and never finds out whether it is true.** +[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md): +converging a machine dropped the path from a container to a port on its own machine, and for eleven +hours the mesh answered *all doing what they were told, all heard from, every module current with its +source* while a web application logged, six thousand times: + +``` +connection to server at "" (10.10.0.1), port 6852 failed: timeout expired +``` + +Every check the mesh makes passed, because every check it makes is about the relationship between the +mesh and a machine: the declaration was applied, the digest matched, every container named was running. +None of them asks whether a consumer can reach what it requires — though the mesh composed the grant +and therefore knows the consumer, the machine, the address, the port and the credential. + +**And where the check runs decides whether it catches anything.** The rule in force admitted the +machines' own addresses on the private network. A dial from the *machine* to its own address carries +exactly such a source address, so a check run by the host on its own behalf would have matched that rule +and passed — while every container on the machine was refused. This is inference from the rule that was +loaded, not a measurement: the fault was found and fixed before anyone thought to dial from the host. +It is enough to decide the question, because a check whose position differs from the consumer's is +testing something nobody asked about. + +## Considered Options + +1. **The control plane dials each provision.** Rejected, and it is the tempting one because the control + plane holds every fact. It sits on the provider's machine for most provisions here and reaches the + address by a path no consumer uses; in the measured outage it would have passed throughout. +2. **The host dials on the consumer's behalf, from the machine.** Rejected for the reason above: the + machine's network position is not the consumer's, and the one outage this exists to catch is exactly + a difference between them. +3. **Ask the module.** Rejected: a module is arbitrary software that the mesh does not write. Some could + report on their provisions and most cannot, and a check that covers the modules that opted in tells + nobody anything about the rest. +4. **Read the module's logs.** Rejected: the failure was in a log the whole time, and reading a module's + logs makes the mesh depend on the wording of software it does not control. +5. **The consumer verifies it, from its own network position.** Adopted. + +## Decision + +**A consumer verifies each grant it is given, from its own network position.** After a reconcile has +applied a grant, the machine opens a connection to the address and port that grant names, from inside +the consumer's own network namespace — the same position the consumer's software dials from, which is +the only position that answers the question the grant asks. + +**It is a connection, not a conversation.** Whether the port accepts a connection is what a grant +claims; whether the credential is right, the role exists or the schema is current is the provider's to +answer and the consumer's to discover. A check that spoke each provision's protocol would be a second +implementation of every provision, and would fail for reasons that are not the mesh's. + +**One failure is not news.** A provider restarting is ordinary, and so is a consumer between containers. +A grant is reported unreachable only after it has failed on **consecutive** reconciles, and the count is +what the machine reports rather than the last attempt — so a reader can tell "it was briefly away" from +"it has never worked". + +**A grant that cannot be checked is said to be unchecked, never assumed good.** A consumer that is not +running has no network position to dial from; that is not a broken grant and must not read as one. It is +also not a verified grant, and the two are different sentences. + +**What it costs to be wrong is the constraint on all of it.** A check that reports a working provision +broken trains a reader to ignore the report, which is worse than having none — the fault this +repository keeps finding, one level up. So the threshold is consecutive failures, the check is the +cheapest thing that answers the question, and an unknown is reported as unknown. + +**The mesh says it where it says everything else.** A machine's report carries its unreachable grants, +and `status` names them beside what is out of date — so "every module current with its source" stops +being the whole of what the mesh will tell you about a machine whose modules cannot reach each other. + +## Consequences + +- **The mesh can be wrong out loud.** It has been able to assert a grant and not check it; now a grant + that does not work is a thing the mesh says, and the eleven hours of issue 145 become minutes. +- **The host gains the ability to act from a container's network position**, which it has not needed + before. That is a real capability and the only one this needs. +- **A machine reports something that is not about the declaration.** Everything it reports today is + what it applied and what it holds; this is the first thing it says about whether what it applied + works. +- **A provision with no port is not checked**, because there is nothing to dial. Several are files and + secrets, and saying "checked" about those would be the appearance of verification that this record + exists to remove. +- **What got harder:** a reconcile does more than apply. Every grant adds a connection attempt on a + cadence, which is cheap individually and worth naming: a machine with many consumers dials once per + grant per reconcile. + +## How it is checked + +- **The outage is caught.** A bed drops the path from a consumer's network position to a provider's + port while leaving the machine's own path to it open — the exact shape of issue 145 — and the grant + reads unreachable. This fails against the previous behaviour, where nothing reported anything, and + against a check run from the machine, which passes while the consumer cannot reach it. +- **A restarting provider is not an outage.** One failed reconcile reports nothing; the count rises and + falls, and the grant reads reachable again without anybody acting. +- **A consumer that is not running reads unchecked, not broken**, asserted separately from the + unreachable case because they are different sentences. +- **A provision with no port is not claimed to be checked.** +- **The report carries the count, not the last attempt**, so "briefly away" and "never worked" are + distinguishable by a reader who sees only the report. +- **`status` names an unreachable grant**, asserted on the output, since a check nothing surfaces is + the same as no check. + +## References + +- [ADR 0010](0010-delivery.md) — the declaration is owned resources; a grant is one of them +- [ADR 0009](0009-modules-and-the-graph.md) — what a provision and a consumer are +- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md) + — the eleven hours +- [issue 136](../04-ISSUES/136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md) — the + same distance between a declaration and a machine, one level down diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 4c2b005..54a1778 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -223,6 +223,7 @@ python3 00-META/checks/index.py fail if stale - **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md) *(superseded)* - **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md) - **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md) +- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) ### How it is built diff --git a/03-DESIGN/01-to-be/10-delivery.md b/03-DESIGN/01-to-be/10-delivery.md index 13a4f78..ad80cdf 100644 --- a/03-DESIGN/01-to-be/10-delivery.md +++ b/03-DESIGN/01-to-be/10-delivery.md @@ -5,8 +5,9 @@ code: - mesh-controller internal/builder - mesh-controller cmd/mesh-controller (build, build --behind, push, status) - mesh-controller internal/inventory/builds.go -updated: 2026-09-21 +updated: 2026-09-29 decisions: + - 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md - 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md - 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md - 02-DECISIONS/0010-delivery.md @@ -226,3 +227,38 @@ when the current failure began and how many reports in a row have said it — th id, whatever the words; three make the machine stuck, and `status` says so beside the failure. The host keeps trying — stuck is what the mesh knows, not what the machine is told. *How it is checked:* an inventory test counts three identical reports, a different one, and a clean apply; the status test asserts the word appears. + +## A consumer verifies the grant it is given + +*2026-09-29, from an outage that ran eleven hours — +[issue 145](../../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md), +settled by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md).* + +A grant is four facts and a credential: the provision, the machine, the port, and who the consumer is +when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and the +mesh asserted it without ever finding out whether it was true. + +What that cost: a machine was converged, the path from a container to a port on its own machine closed, +and for eleven hours the mesh answered *all heard from, every module current with its source* while a +module logged a connection timeout to its database six thousand times. Every check the mesh makes passed, +because every one is about the relationship between the mesh and a machine — applied, current, running. +None asks whether a consumer can reach what it requires. + +**So a consumer verifies its own grants, from its own network position.** Not the control plane, which +reaches the address by a path no consumer uses; not the machine, whose own packets carried a source +address the filter admitted while every container's was refused. The position is the point: a check +somewhere else is testing something nobody asked about. + +It opens a connection and nothing more. Whether the credential is right or the schema current is the +provider's to answer and the consumer's to discover; a check that spoke each provision's protocol would +be a second implementation of every provision. + +One failure is not news — a provider restarting is ordinary — so a grant reads unreachable only after +consecutive reconciles, and the machine reports the count rather than the last attempt, which is what +separates "briefly away" from "never worked". A consumer that is not running has no position to dial +from: that grant reads unchecked, which is a different sentence from broken and must not be written as +one. + +*How it is checked* is stated with the decision, and the first of them is the outage itself: a bed drops +the path from a consumer's position while leaving the machine's own open, and the grant must read +unreachable — which fails both against reporting nothing and against a check run from the machine. diff --git a/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md b/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md index 65afda0..0ffddcc 100644 --- a/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md +++ b/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md @@ -5,7 +5,7 @@ located-in: - mesh-controller internal/catalogue/filtering.go (fixed for this instance) - mesh-controller (what status reports, and what it does not ask) fixed-by: -amended-design: +amended-design: 03-DESIGN/01-to-be/10-delivery.md --- # 145 — A machine reads healthy while its modules cannot reach each other @@ -72,6 +72,18 @@ distance between a declaration and the machine, and in both cases the report was **And the eleven hours are the measurement, not the bug.** The filter fault was one line and is fixed. What is not fixed is that nothing in the mesh would have told anybody. +## What was decided + +*2026-09-29, the same day.* The first open question below — should a grant be checked, and from where — +is answered by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md): the +consumer verifies it, from its own network position, because that is the only position that answers what +a grant claims. A check run by the control plane or by the machine would have passed throughout this +outage, since the rule in force admitted the machines' own addresses and it was the containers that were +refused. + +The remaining questions below stand, and the record answers two of them: one failure is not news, only +consecutive ones, and a grant that cannot be checked is reported unchecked rather than assumed good. + ## Open questions - Should a grant be checked? The mesh knows the consumer, the provider, the address and the port, so a