Keep the core over an older record instead of refusing the send (issue 201, review 90)

Refusing the whole send wedged a machine whose gate put back the
controller and whose put-back send was refused: every later automatic
send there was refused, retried each tick, and held up every other walk.

A send now composes the controller and the node-engine at the build the
machine was last sent when the mesh's record is older, and carries the
rest; nothing is refused or retried, and a plan whose own build is the
older one ends with its gate superseded. A sent build that failed its
gate is never kept, so the next send completes a put-back. The condition
names the two ways out and clears on any send that carries the record,
a person's push included.

cmd/mesh-controller/core_downgrade.go

@@ -2,7 +2,6 @@ package main
 
 import (
 	"context"
-	"errors"
 	"fmt"
 	"sort"
 	"time"
@@ -12,28 +11,31 @@ import (
 	"github.com/novox/mesh-controller/internal/lease"
 )
 
-// **A send never moves the mesh's own core to a build older than the one its machine runs** (novox/hq issue
-// 201). On 2026-10-02 two merges a minute apart made two plans; the controller's plan rolled the new controller,
-// which widened its seat's row, and the node-engine's plan then sent the control node the controller build
-// before it. The older binary read the row it could not serve and the mesh lost its voice for ten minutes — the
-// thing that would have corrected it was the thing that was down. ADR 0185 took the outage out of it (a
-// controller behind its row serves what it can); the race is closed here, without needing its trigger: whatever
-// composed the older build, the send that would carry it is refused.
+// **No send but a person's push and a gate's own put-back moves the mesh's own core to a build older than the
+// one its machine was last sent** (novox/hq issue 201). On 2026-10-02 two merges a minute apart made two plans;
+// the controller's plan rolled the new controller, which widened its seat's row, and the node-engine's plan then
+// sent the control node the controller build before it. The older binary read the row it could not serve and the
+// mesh lost its voice for ten minutes — the thing that would have corrected it was the thing that was down. ADR
+// 0185 took the outage out of it (a controller behind its row serves what it can); the race is closed here,
+// without needing its trigger: whatever made the mesh's record older, the send keeps the build.
 //
-// Older is by the builds' registration order — when each was asked (novox/hq issue 219) — never by the
-// commits' dates. A gate's own put-back is the one send that may carry it, for the modules it puts back: it says
-// so (sendScope.puttingBack), and the controller's put-back is held to the store's schema already (issue 352).
-// A person's push carries what it carries, as every other refusal of a send leaves it.
+// **Kept, not refused.** The send composes the controller or the node-engine at the build the machine was last
+// sent, as a recorded module is kept (ADR 0245), and carries everything else: no other module, secret, healer or
+// walk waits on it, and nothing is retried. A plan whose own build is the older one finds its gate superseded by
+// the build kept there (issue 352) and ends. The mesh's record stays older, and a warning condition says so until
+// a send carries the record — a person's `push <node>` accepts the older build — or the record moves forward: a
+// newer merge, or `rebuild mesh-controller`.
 //
-// The refusal is said in the plan's words — the send's error, which the plan keeps as its note and tries again
-// — and as a warning condition on the machine, cleared by the first send there that carries no such move.
+// Older is by the builds' registration order — when each was asked (novox/hq issue 219) — never by the commits'
+// dates. So a retry or a rebuild of an old commit, asked anew, counts as newer: it is registered as the module's
+// current build by the same order, and the guard does not stop that move. A build that failed its gate is never
+// ahead: the machine was sent it and the record was put back before it (gateFailed), and if the put-back's own
+// send was refused, the next send completes it rather than keeping the failed build there. A gate's own put-back
+// carries its modules back whatever the order (sendScope.puttingBack); the controller's is held to the store's
+// schema already (issue 352).
 
-// errCoreDowngrade is a send refused because it would put a core module on a machine back behind the build that
-// machine runs.
-var errCoreDowngrade = errors.New("a send would put the mesh's own core back behind the build its machine runs")
-
-// kindCoreDowngrade is that refusal, as a condition.
-const kindCoreDowngrade = "core-downgrade-refused"
+// kindCoreDowngrade is a core module kept at the build its machine was last sent, over an older record.
+const kindCoreDowngrade = "core-downgrade-kept"
 
 // coreDowngradeSource is the condition's source: the sends, which raise and clear it.
 const coreDowngradeSource = "sends"
@@ -42,67 +44,84 @@ const coreDowngradeSource = "sends"
 // store and its seat's row, and the node-engine, which reads what the controller declares.
 var downgradeGuarded = []string{lease.ComponentController, lease.ComponentEngine}
 
-// buildAsked is when the newest build of a module from a commit that worked was asked — or recorded, when the
-// request time is not known — and false when the records hold none. A variable, so a test reads its own order.
-var buildAsked = func(ctx context.Context, inv *inventory.Inventory, module, commit string) (time.Time, bool, error) {
+// askedBuild is what the order needs of one build of a module: when it was asked, and whether it failed its gate.
+type askedBuild struct {
+	asked      time.Time
+	failedGate bool
+}
+
+// buildAsked is the newest build of a module from a commit that worked — when it was asked (or recorded, when
+// the request time is not known), and whether it failed its gate — and false when the records hold none. A
+// variable, so a test reads its own order.
+var buildAsked = func(ctx context.Context, inv *inventory.Inventory, module, commit string) (askedBuild, bool, error) {
 	builds, err := inv.Builds(ctx, module, askedLookBack)
 	if err != nil {
-		return time.Time{}, false, err
+		return askedBuild{}, false, err
 	}
 	for _, b := range builds {
 		if b.Worked() && sameCommit(b.Commit, commit) {
-			return b.AskedOrAt(), true, nil
+			failed, err := inv.GateFailed(ctx, b.ID)
+			if err != nil {
+				return askedBuild{}, false, err
+			}
+			return askedBuild{asked: b.AskedOrAt(), failedGate: failed}, true, nil
 		}
 	}
-	return time.Time{}, false, nil
+	return askedBuild{}, false, nil
 }
 
 // askedLookBack is how many of a module's newest builds are read for when a build was asked.
 const askedLookBack = 50
 
-// coreDowngrade is one move of a core module to an older build than its machine runs.
+// coreDowngrade is one core module a machine was last sent a build of that is newer than the mesh's record.
 type coreDowngrade struct {
-	move             inventory.CarriedMove
-	component        string
-	fromAsked, asked time.Time
+	module, node, component string
+	// sent is the build the machine was last sent, record the one the mesh holds.
+	sent, record           string
+	sentAsked, recordAsked time.Time
 }
 
 func (d coreDowngrade) String() string {
-	return fmt.Sprintf("%s would be sent %s %s, older than the %s it runs (asked %s, before %s) — no send but a "+
-		"gate's own put-back moves the %s back (novox/hq issue 201)", d.move.Node, d.move.Module, short(d.move.To),
-		short(d.move.From), d.asked.UTC().Format(time.RFC3339), d.fromAsked.UTC().Format(time.RFC3339), d.component)
+	return fmt.Sprintf("%s was last sent %s %s, and the mesh holds the older %s (asked %s, before %s): a send there "+
+		"keeps %s; `push %s` sends the older build, and a newer merge or `rebuild %s` moves the record forward "+
+		"(novox/hq issue 201)", d.node, d.module, short(d.sent), short(d.record),
+		d.recordAsked.UTC().Format(time.RFC3339), d.sentAsked.UTC().Format(time.RFC3339), short(d.sent), d.node,
+		d.module)
 }
 
-// coreDowngrades is the moves of a machine's send that put a core module back to a build asked before the one it
-// runs, leaving out what a put-back puts back. A move whose either build the records do not hold is not proved
-// older, and is left to the gate that judges it. Pure but for asked.
-func coreDowngrades(moves []inventory.CarriedMove, scope sendScope,
-	asked func(module, commit string) (time.Time, bool, error)) ([]coreDowngrade, error) {
+// coreDowngrades is, for one machine, each guarded core module whose record is older than the build the machine
+// was last sent, leaving out what a gate's own put-back puts back and a sent build that failed its gate. A move
+// whose either build the records do not hold is not proved older and is left to the gate that judges it. Pure
+// but for asked.
+func coreDowngrades(node string, sent map[string]string, record map[string]string, scope sendScope,
+	asked func(module, commit string) (askedBuild, bool, error)) ([]coreDowngrade, error) {
 	var out []coreDowngrade
-	for _, mv := range moves {
-		component := coreComponent(mv.Module)
-		if !guarded(component) || mv.From == "" || mv.To == "" || sameCommit(mv.From, mv.To) {
+	for module, was := range sent {
+		component := coreComponent(module)
+		now := record[module]
+		if !guarded(component) || was == "" || now == "" || sameCommit(was, now) {
 			continue
 		}
-		if scope.puttingBack && scope.modules[mv.Module] {
+		if scope.puttingBack && scope.modules[module] {
 			continue
 		}
-		from, known, err := asked(mv.Module, mv.From)
-		if err != nil || !known {
-			if err != nil {
-				return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", mv.Module, short(mv.From), err)
-			}
-			continue
-		}
-		to, known, err := asked(mv.Module, mv.To)
+		from, known, err := asked(module, was)
 		if err != nil {
-			return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", mv.Module, short(mv.To), err)
+			return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(was), err)
 		}
-		if known && to.Before(from) {
-			out = append(out, coreDowngrade{move: mv, component: component, fromAsked: from, asked: to})
+		if !known || from.failedGate {
+			continue
+		}
+		to, known, err := asked(module, now)
+		if err != nil {
+			return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(now), err)
+		}
+		if known && to.asked.Before(from.asked) {
+			out = append(out, coreDowngrade{module: module, node: node, component: component, sent: was, record: now,
+				sentAsked: from.asked, recordAsked: to.asked})
 		}
 	}
-	sort.Slice(out, func(i, j int) bool { return out[i].move.Module < out[j].move.Module })
+	sort.Slice(out, func(i, j int) bool { return out[i].module < out[j].module })
 	return out, nil
 }
 
@@ -115,32 +134,29 @@ func guarded(component string) bool {
 	return false
 }
 
-// refuseCoreDowngrades says, for one machine's moves, each that would put a core module back, raising its
-// condition; and clears the condition of each guarded component that machine's send does not move back.
-func refuseCoreDowngrades(ctx context.Context, inv *inventory.Inventory, scope sendScope, node string,
-	moves []inventory.CarriedMove) ([]string, error) {
-	found, err := coreDowngrades(moves, scope, func(module, commit string) (time.Time, bool, error) {
+// coreKept is, for a send that keeps (sendKeeps), each guarded core module of a machine to keep at the build it
+// was last sent, module → commit, each said and raised as a condition.
+func coreKept(ctx context.Context, inv *inventory.Inventory, node string, sent map[string]string,
+	current map[string]inventory.CurrentBuild) (map[string]string, error) {
+	record := map[string]string{}
+	for m, c := range current {
+		record[m] = c.Commit
+	}
+	found, err := coreDowngrades(node, sent, record, scopeOf(ctx), func(module, commit string) (askedBuild, bool, error) {
 		return buildAsked(ctx, inv, module, commit)
 	})
-	if err != nil {
+	if err != nil || len(found) == 0 {
 		return nil, err
 	}
-	var said []string
-	refused := map[string]bool{}
+	out := map[string]string{}
 	for _, d := range found {
-		said = append(said, d.String())
-		refused[d.component] = true
+		out[d.module] = d.sent
 		sayCoreDowngrade(ctx, d)
 	}
-	for _, c := range downgradeGuarded {
-		if !refused[c] {
-			clearCoreDowngrade(ctx, c, node)
-		}
-	}
-	return said, nil
+	return out, nil
 }
 
-// coreDowngradeObservation is a refused downgrade as a condition: `core.<component>.<machine>.<kind>`.
+// coreDowngradeObservation is a kept core build as a condition: `core.<component>.<machine>.<kind>`.
 func coreDowngradeObservation(component, node, summary string) conditions.Observation {
 	return conditions.Observation{Scope: conditions.ScopeCore, ID: component + "." + node, Kind: kindCoreDowngrade,
 		Token: kindCoreDowngrade, Machine: node, Severity: conditions.Warning, Source: coreDowngradeSource,
@@ -148,24 +164,49 @@ func coreDowngradeObservation(component, node, summary string) conditions.Observ
 }
 
 func sayCoreDowngrade(ctx context.Context, d coreDowngrade) {
-	fmt.Printf("NOT SENT: %s\n", d)
+	fmt.Printf("KEPT: %s\n", d)
 	keeper := conditionsFrom
 	if keeper == nil {
 		return
 	}
-	if _, err := keeper.Observe(ctx, coreDowngradeObservation(d.component, d.move.Node, d.String())); err != nil {
-		fmt.Printf("the condition for %s's refused downgrade on %s could not be raised: %v\n", d.move.Module,
-			d.move.Node, err)
+	if _, err := keeper.Observe(ctx, coreDowngradeObservation(d.component, d.node, d.String())); err != nil {
+		fmt.Printf("the condition for %s kept on %s could not be raised: %v\n", d.module, d.node, err)
 	}
 }
 
-func clearCoreDowngrade(ctx context.Context, component, node string) {
+// settleCoreDowngrades clears, after a send to a machine is away, the condition of each guarded core module the
+// send carried at the mesh's record — a person's push that accepted the older build, or a send after the record
+// moved forward — or did not carry at all. A send that kept the newer build leaves it open.
+func settleCoreDowngrades(ctx context.Context, inv *inventory.Inventory, node string, carried map[string]string) {
 	keeper := conditionsFrom
-	if keeper == nil {
+	if keeper == nil || carried == nil {
 		return
 	}
-	key := coreDowngradeObservation(component, node, "").Key()
-	if _, err := keeper.Clear(ctx, key, "a send to "+node+" carries no older "+component); err != nil {
-		fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
+	var open []string
+	for _, c := range downgradeGuarded {
+		if _, found, err := keeper.Get(ctx, coreDowngradeObservation(c, node, "").Key()); err == nil && found {
+			open = append(open, c)
+		}
+	}
+	if len(open) == 0 {
+		return
+	}
+	current, err := inv.CurrentBuilds(ctx)
+	if err != nil {
+		fmt.Printf("whether %s still keeps a core build over the mesh's record cannot be read: %v\n", node, err)
+		return
+	}
+	for module, now := range current {
+		component := coreComponent(module)
+		if !guarded(component) {
+			continue
+		}
+		if was, has := carried[module]; has && !sameCommit(was, now.Commit) {
+			continue
+		}
+		key := coreDowngradeObservation(component, node, "").Key()
+		if _, err := keeper.Clear(ctx, key, node+" was sent the "+component+" the mesh holds"); err != nil {
+			fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
+		}
 	}
 }