Merge pull request 'ADR 0143: a consumer verifies the grant it is given' (#180) from decision/0143-a-consumer-verifies-its-grant into main

2026-09-29 11:07:20 +00:00
parent f65664640a ddfd62edf6
commit 78d4873f4f
4 changed files with 185 additions and 2 deletions
@@ -0,0 +1,134 @@
+---
+topic: what runs on it
+status: accepted
+date: 2026-09-29
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0010-delivery.md
+---
+
+# 143. A consumer verifies the grant it is given
+
+## Context
+
+A **grant** is what the mesh writes on a consumer's machine so it can reach a provider. The real one
+the forge receives for its database, as it arrives:
+
+```
+provision  postgres-database
+at         <the provider's machine, by name>
+port       the machine port the provider is published on
+as         the role the provider created for this consumer
+```
+
+with the credential sealed in a separate file. Four facts and a password, and they are the whole
+mechanism by which anything in the mesh reaches anything else.
+
+**The mesh asserts that claim and never finds out whether it is true.**
+[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md):
+converging a machine dropped the path from a container to a port on its own machine, and for eleven
+hours the mesh answered *all doing what they were told, all heard from, every module current with its
+source* while a web application logged, six thousand times:
+
+```
+connection to server at "<the machine>" (10.10.0.1), port 6852 failed: timeout expired
+```
+
+Every check the mesh makes passed, because every check it makes is about the relationship between the
+mesh and a machine: the declaration was applied, the digest matched, every container named was running.
+None of them asks whether a consumer can reach what it requires — though the mesh composed the grant
+and therefore knows the consumer, the machine, the address, the port and the credential.
+
+**And where the check runs decides whether it catches anything.** The rule in force admitted the
+machines' own addresses on the private network. A dial from the *machine* to its own address carries
+exactly such a source address, so a check run by the host on its own behalf would have matched that rule
+and passed — while every container on the machine was refused. This is inference from the rule that was
+loaded, not a measurement: the fault was found and fixed before anyone thought to dial from the host.
+It is enough to decide the question, because a check whose position differs from the consumer's is
+testing something nobody asked about.
+
+## Considered Options
+
+1. **The control plane dials each provision.** Rejected, and it is the tempting one because the control
+   plane holds every fact. It sits on the provider's machine for most provisions here and reaches the
+   address by a path no consumer uses; in the measured outage it would have passed throughout.
+2. **The host dials on the consumer's behalf, from the machine.** Rejected for the reason above: the
+   machine's network position is not the consumer's, and the one outage this exists to catch is exactly
+   a difference between them.
+3. **Ask the module.** Rejected: a module is arbitrary software that the mesh does not write. Some could
+   report on their provisions and most cannot, and a check that covers the modules that opted in tells
+   nobody anything about the rest.
+4. **Read the module's logs.** Rejected: the failure was in a log the whole time, and reading a module's
+   logs makes the mesh depend on the wording of software it does not control.
+5. **The consumer verifies it, from its own network position.** Adopted.
+
+## Decision
+
+**A consumer verifies each grant it is given, from its own network position.** After a reconcile has
+applied a grant, the machine opens a connection to the address and port that grant names, from inside
+the consumer's own network namespace — the same position the consumer's software dials from, which is
+the only position that answers the question the grant asks.
+
+**It is a connection, not a conversation.** Whether the port accepts a connection is what a grant
+claims; whether the credential is right, the role exists or the schema is current is the provider's to
+answer and the consumer's to discover. A check that spoke each provision's protocol would be a second
+implementation of every provision, and would fail for reasons that are not the mesh's.
+
+**One failure is not news.** A provider restarting is ordinary, and so is a consumer between containers.
+A grant is reported unreachable only after it has failed on **consecutive** reconciles, and the count is
+what the machine reports rather than the last attempt — so a reader can tell "it was briefly away" from
+"it has never worked".
+
+**A grant that cannot be checked is said to be unchecked, never assumed good.** A consumer that is not
+running has no network position to dial from; that is not a broken grant and must not read as one. It is
+also not a verified grant, and the two are different sentences.
+
+**What it costs to be wrong is the constraint on all of it.** A check that reports a working provision
+broken trains a reader to ignore the report, which is worse than having none — the fault this
+repository keeps finding, one level up. So the threshold is consecutive failures, the check is the
+cheapest thing that answers the question, and an unknown is reported as unknown.
+
+**The mesh says it where it says everything else.** A machine's report carries its unreachable grants,
+and `status` names them beside what is out of date — so "every module current with its source" stops
+being the whole of what the mesh will tell you about a machine whose modules cannot reach each other.
+
+## Consequences
+
+- **The mesh can be wrong out loud.** It has been able to assert a grant and not check it; now a grant
+  that does not work is a thing the mesh says, and the eleven hours of issue 145 become minutes.
+- **The host gains the ability to act from a container's network position**, which it has not needed
+  before. That is a real capability and the only one this needs.
+- **A machine reports something that is not about the declaration.** Everything it reports today is
+  what it applied and what it holds; this is the first thing it says about whether what it applied
+  works.
+- **A provision with no port is not checked**, because there is nothing to dial. Several are files and
+  secrets, and saying "checked" about those would be the appearance of verification that this record
+  exists to remove.
+- **What got harder:** a reconcile does more than apply. Every grant adds a connection attempt on a
+  cadence, which is cheap individually and worth naming: a machine with many consumers dials once per
+  grant per reconcile.
+
+## How it is checked
+
+- **The outage is caught.** A bed drops the path from a consumer's network position to a provider's
+  port while leaving the machine's own path to it open — the exact shape of issue 145 — and the grant
+  reads unreachable. This fails against the previous behaviour, where nothing reported anything, and
+  against a check run from the machine, which passes while the consumer cannot reach it.
+- **A restarting provider is not an outage.** One failed reconcile reports nothing; the count rises and
+  falls, and the grant reads reachable again without anybody acting.
+- **A consumer that is not running reads unchecked, not broken**, asserted separately from the
+  unreachable case because they are different sentences.
+- **A provision with no port is not claimed to be checked.**
+- **The report carries the count, not the last attempt**, so "briefly away" and "never worked" are
+  distinguishable by a reader who sees only the report.
+- **`status` names an unreachable grant**, asserted on the output, since a check nothing surfaces is
+  the same as no check.
+
+## References
+
+- [ADR 0010](0010-delivery.md) — the declaration is owned resources; a grant is one of them
+- [ADR 0009](0009-modules-and-the-graph.md) — what a provision and a consumer are
+- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
+  — the eleven hours
+- [issue 136](../04-ISSUES/136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md) — the
+  same distance between a declaration and a machine, one level down
@@ -223,6 +223,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md)  *(superseded)*
 - **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md)
 - **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)
+- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md)

 ### How it is built

@@ -5,8 +5,9 @@ code:
  - mesh-controller internal/builder
  - mesh-controller cmd/mesh-controller (build, build --behind, push, status)
  - mesh-controller internal/inventory/builds.go
-updated: 2026-09-21
+updated: 2026-09-29
 decisions:
+  - 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md
  - 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
  - 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
  - 02-DECISIONS/0010-delivery.md
@@ -226,3 +227,38 @@ when the current failure began and how many reports in a row have said it — th
 id, whatever the words; three make the machine stuck, and `status` says so beside the failure. The host keeps trying — stuck is what the mesh
 knows, not what the machine is told. *How it is checked:* an inventory test counts three identical
 reports, a different one, and a clean apply; the status test asserts the word appears.
+
+## A consumer verifies the grant it is given
+
+*2026-09-29, from an outage that ran eleven hours —
+[issue 145](../../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
+settled by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md).*
+
+A grant is four facts and a credential: the provision, the machine, the port, and who the consumer is
+when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and the
+mesh asserted it without ever finding out whether it was true.
+
+What that cost: a machine was converged, the path from a container to a port on its own machine closed,
+and for eleven hours the mesh answered *all heard from, every module current with its source* while a
+module logged a connection timeout to its database six thousand times. Every check the mesh makes passed,
+because every one is about the relationship between the mesh and a machine — applied, current, running.
+None asks whether a consumer can reach what it requires.
+
+**So a consumer verifies its own grants, from its own network position.** Not the control plane, which
+reaches the address by a path no consumer uses; not the machine, whose own packets carried a source
+address the filter admitted while every container's was refused. The position is the point: a check
+somewhere else is testing something nobody asked about.
+
+It opens a connection and nothing more. Whether the credential is right or the schema current is the
+provider's to answer and the consumer's to discover; a check that spoke each provision's protocol would
+be a second implementation of every provision.
+
+One failure is not news — a provider restarting is ordinary — so a grant reads unreachable only after
+consecutive reconciles, and the machine reports the count rather than the last attempt, which is what
+separates "briefly away" from "never worked". A consumer that is not running has no position to dial
+from: that grant reads unchecked, which is a different sentence from broken and must not be written as
+one.
+
+*How it is checked* is stated with the decision, and the first of them is the outage itself: a bed drops
+the path from a consumer's position while leaving the machine's own open, and the grant must read
+unreachable — which fails both against reporting nothing and against a check run from the machine.
@@ -5,7 +5,7 @@ located-in:
  - mesh-controller internal/catalogue/filtering.go (fixed for this instance)
  - mesh-controller (what status reports, and what it does not ask)
 fixed-by:
-amended-design:
+amended-design: 03-DESIGN/01-to-be/10-delivery.md
 ---

 # 145 — A machine reads healthy while its modules cannot reach each other
@@ -72,6 +72,18 @@ distance between a declaration and the machine, and in both cases the report was
 **And the eleven hours are the measurement, not the bug.** The filter fault was one line and is fixed.
 What is not fixed is that nothing in the mesh would have told anybody.

+## What was decided
+
+*2026-09-29, the same day.* The first open question below — should a grant be checked, and from where —
+is answered by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md): the
+consumer verifies it, from its own network position, because that is the only position that answers what
+a grant claims. A check run by the control plane or by the machine would have passed throughout this
+outage, since the rule in force admitted the machines' own addresses and it was the containers that were
+refused.
+
+The remaining questions below stand, and the record answers two of them: one failure is not news, only
+consecutive ones, and a grant that cannot be checked is reported unchecked rather than assumed good.
+
 ## Open questions

 - Should a grant be checked? The mesh knows the consumer, the provider, the address and the port, so a