Issue 267: a reconcile's report overtook the apply that followed it

Record why a release plan waited on a report it had already been given, and point issue 264 at it, since 264's fix was suspected and is not the cause.
2026-10-06 01:46:33 +02:00
parent 3ba75a312b
commit d77399055c
2 changed files with 97 additions and 0 deletions
@@ -70,3 +70,10 @@ These tests guard the edges:

 Once the fix is rolled out, the live check is the next declaration that carries a new host version.
 The first machine's report should reach the plan without anyone pushing by hand.
+
+## Later
+
+**2026-10-06.** The next stalled plan after this fix was rolled out looked like this issue coming
+back, and was suspected of being caused by its kept report. It was not. A reconcile's report about
+an older declaration reached the controller after the newer apply's report, and replaced it:
+[issue 267](../267-a-reconciles-report-overtook-the-apply-that-followed-it/00-report.md).
@@ -0,0 +1,90 @@
+---
+status: located
+opened: 2026-10-06
+located-in: [mesh-host, mesh-controller]
+fixed-by:
+amended-design:
+---
+
+# 267. A reconcile's report overtook the apply that followed it
+
+## Symptom
+
+A release plan sent the home server a declaration and waited for the home server to report it. The
+host's journal showed the declaration applied nine seconds later, with one line,
+`applied 659 resource(s)`. The controller's journal showed that line twice in the same second. The
+report the controller kept for the home server then named a different, older declaration than the
+one it had recorded sending. The plan requires those two to match for its first machine, so it
+waited until a person pushed again by hand.
+
+The same doubled line appears earlier that night, in a log written before the fix for
+[issue 264](../264-a-self-updating-engine-lost-the-report-of-the-apply-that-delivered-it/00-report.md)
+existed.
+
+## Cause
+
+Two reports, and the older one arrived last.
+
+The host's five-minute reconcile was due three seconds before the declaration arrived. It took the
+apply lock first. It read the declaration kept at that moment, which was the older one, held the
+machine to it and made its report. A reconcile on an adopted node, or on any node that can say
+which links face outside, offers that report to the link to send without being asked
+([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md),
+[ADR 0140](../../02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md)). The link was busy:
+the delivery's apply was waiting for the same lock. When the apply finished, the link sent its
+report about the new declaration, and then went back to its loop and sent the queued reconcile
+report about the old one.
+
+The controller keeps one report per machine, and the last one written wins. It already sets aside a
+report about a declaration it has moved past
+([design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §3), but only a report that is
+nothing more than an apply's account. A report that also says what the machine is (its links, what
+it holds, its firewall) is always acted on, because that half is never stale. The reconcile's report
+said both, so its account of the old declaration replaced the account of the new one.
+
+[Issue 261](../261-a-module-was-applied-and-given-back-half-a-minute-later/00-report.md) fixed
+which declaration a reconcile applies when it waits for a delivery. This is the other order: the
+reconcile goes first, and only its report is late.
+
+**It was not introduced by the fix for issue 264**, which was suspected first because the stall
+followed its rollout. That fix re-sends a kept report only when the link is opened, and says so in
+the journal. The host had linked once, an hour earlier, and its journal has no such line. Its kept
+report was absent, because the broker had taken the last apply's report. And the same doubled
+report appears in the controller's log before that fix was written.
+
+## Fix
+
+Both ends, so that the order of arrival cannot decide which account is kept:
+
+1. **The host.** The link remembers the declaration it last applied, or last re-sent from what was
+   kept. A report made without being asked is set aside, and counted as not sent, when it names a
+   different declaration. The apply's own report has already described the machine, and it is more
+   recent. The next reconcile sends anything that is still news. A report that names no declaration
+   is sent as before, and so is one made before this link has applied anything.
+2. **The controller.** A report about a declaration other than the one the mesh last sent a machine
+   still records what it says about the machine, as before. It no longer replaces the stored account
+   of the apply, or the list of what the machine owns. The controller logs that it kept the report's
+   facts and not its account.
+
+The report does not carry the declaration's sequence number, so the digest the mesh recorded sending
+is what decides. This is the same check that already applies to a report that is only an account of
+an apply.
+
+## How it is checked
+
+These tests fail without the fix:
+
+- mesh-host `TestAReconcileReportOlderThanTheApplyIsNotSaidAfterIt`. A reconcile report about the
+  old declaration is queued while the new one is applied. The mesh hears only the report of the
+  new apply, and the reconcile report is counted as not said.
+- mesh-controller `TestAnAccountOfAnOlderDeclarationDoesNotReplaceTheNewer`. The mesh sent the new
+  declaration and heard its report, then a report about the old one. The kept account still names
+  the new declaration, and the old report's links are recorded.
+
+These tests guard the edges:
+
+- mesh-host `TestAReconcileReportAboutTheAppliedDeclarationIsSaid` and `TestOvertaken`.
+- mesh-controller `TestAnAccountOfTheSentDeclarationReplacesTheKeptOne`.
+
+Once the fix is rolled out, the live check is a push that arrives while a reconcile is running. The
+controller should log the report once, and the plan should move on without anyone pushing by hand.