diff --git a/04-ISSUES/264-a-self-updating-engine-lost-the-report-of-the-apply-that-delivered-it/00-report.md b/04-ISSUES/264-a-self-updating-engine-lost-the-report-of-the-apply-that-delivered-it/00-report.md index 8da7bea..feff708 100644 --- a/04-ISSUES/264-a-self-updating-engine-lost-the-report-of-the-apply-that-delivered-it/00-report.md +++ b/04-ISSUES/264-a-self-updating-engine-lost-the-report-of-the-apply-that-delivered-it/00-report.md @@ -70,3 +70,10 @@ These tests guard the edges: Once the fix is rolled out, the live check is the next declaration that carries a new host version. The first machine's report should reach the plan without anyone pushing by hand. + +## Later + +**2026-10-06.** The next stalled plan after this fix was rolled out looked like this issue coming +back, and was suspected of being caused by its kept report. It was not. A reconcile's report about +an older declaration reached the controller after the newer apply's report, and replaced it: +[issue 267](../267-a-reconciles-report-overtook-the-apply-that-followed-it/00-report.md). diff --git a/04-ISSUES/267-a-reconciles-report-overtook-the-apply-that-followed-it/00-report.md b/04-ISSUES/267-a-reconciles-report-overtook-the-apply-that-followed-it/00-report.md new file mode 100644 index 0000000..a4f58c7 --- /dev/null +++ b/04-ISSUES/267-a-reconciles-report-overtook-the-apply-that-followed-it/00-report.md @@ -0,0 +1,90 @@ +--- +status: located +opened: 2026-10-06 +located-in: [mesh-host, mesh-controller] +fixed-by: +amended-design: +--- + +# 267. A reconcile's report overtook the apply that followed it + +## Symptom + +A release plan sent the home server a declaration and waited for the home server to report it. The +host's journal showed the declaration applied nine seconds later, with one line, +`applied 659 resource(s)`. The controller's journal showed that line twice in the same second. The +report the controller kept for the home server then named a different, older declaration than the +one it had recorded sending. The plan requires those two to match for its first machine, so it +waited until a person pushed again by hand. + +The same doubled line appears earlier that night, in a log written before the fix for +[issue 264](../264-a-self-updating-engine-lost-the-report-of-the-apply-that-delivered-it/00-report.md) +existed. + +## Cause + +Two reports, and the older one arrived last. + +The host's five-minute reconcile was due three seconds before the declaration arrived. It took the +apply lock first. It read the declaration kept at that moment, which was the older one, held the +machine to it and made its report. A reconcile on an adopted node, or on any node that can say +which links face outside, offers that report to the link to send without being asked +([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md), +[ADR 0140](../../02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md)). The link was busy: +the delivery's apply was waiting for the same lock. When the apply finished, the link sent its +report about the new declaration, and then went back to its loop and sent the queued reconcile +report about the old one. + +The controller keeps one report per machine, and the last one written wins. It already sets aside a +report about a declaration it has moved past +([design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) ยง3), but only a report that is +nothing more than an apply's account. A report that also says what the machine is (its links, what +it holds, its firewall) is always acted on, because that half is never stale. The reconcile's report +said both, so its account of the old declaration replaced the account of the new one. + +[Issue 261](../261-a-module-was-applied-and-given-back-half-a-minute-later/00-report.md) fixed +which declaration a reconcile applies when it waits for a delivery. This is the other order: the +reconcile goes first, and only its report is late. + +**It was not introduced by the fix for issue 264**, which was suspected first because the stall +followed its rollout. That fix re-sends a kept report only when the link is opened, and says so in +the journal. The host had linked once, an hour earlier, and its journal has no such line. Its kept +report was absent, because the broker had taken the last apply's report. And the same doubled +report appears in the controller's log before that fix was written. + +## Fix + +Both ends, so that the order of arrival cannot decide which account is kept: + +1. **The host.** The link remembers the declaration it last applied, or last re-sent from what was + kept. A report made without being asked is set aside, and counted as not sent, when it names a + different declaration. The apply's own report has already described the machine, and it is more + recent. The next reconcile sends anything that is still news. A report that names no declaration + is sent as before, and so is one made before this link has applied anything. +2. **The controller.** A report about a declaration other than the one the mesh last sent a machine + still records what it says about the machine, as before. It no longer replaces the stored account + of the apply, or the list of what the machine owns. The controller logs that it kept the report's + facts and not its account. + +The report does not carry the declaration's sequence number, so the digest the mesh recorded sending +is what decides. This is the same check that already applies to a report that is only an account of +an apply. + +## How it is checked + +These tests fail without the fix: + +- mesh-host `TestAReconcileReportOlderThanTheApplyIsNotSaidAfterIt`. A reconcile report about the + old declaration is queued while the new one is applied. The mesh hears only the report of the + new apply, and the reconcile report is counted as not said. +- mesh-controller `TestAnAccountOfAnOlderDeclarationDoesNotReplaceTheNewer`. The mesh sent the new + declaration and heard its report, then a report about the old one. The kept account still names + the new declaration, and the old report's links are recorded. + +These tests guard the edges: + +- mesh-host `TestAReconcileReportAboutTheAppliedDeclarationIsSaid` and `TestOvertaken`. +- mesh-controller `TestAnAccountOfTheSentDeclarationReplacesTheKeptOne`. + +Once the fix is rolled out, the live check is a push that arrives while a reconcile is running. The +controller should log the report once, and the plan should move on without anyone pushing by hand.