Keep the core over an older record instead of refusing the send (issue 201, review 90)
mesh/merge-gate out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Refusing the whole send wedged a machine whose gate put back the controller and whose put-back send was refused: every later automatic send there was refused, retried each tick, and held up every other walk. A send now composes the controller and the node-engine at the build the machine was last sent when the mesh's record is older, and carries the rest; nothing is refused or retried, and a plan whose own build is the older one ends with its gate superseded. A sent build that failed its gate is never kept, so the next send completes a put-back. The condition names the two ways out and clears on any send that carries the record, a person's push included.
This commit is contained in:
@@ -2,7 +2,6 @@ package main
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"sort"
|
"sort"
|
||||||
"time"
|
"time"
|
||||||
@@ -12,28 +11,31 @@ import (
|
|||||||
"github.com/novox/mesh-controller/internal/lease"
|
"github.com/novox/mesh-controller/internal/lease"
|
||||||
)
|
)
|
||||||
|
|
||||||
// **A send never moves the mesh's own core to a build older than the one its machine runs** (novox/hq issue
|
// **No send but a person's push and a gate's own put-back moves the mesh's own core to a build older than the
|
||||||
// 201). On 2026-10-02 two merges a minute apart made two plans; the controller's plan rolled the new controller,
|
// one its machine was last sent** (novox/hq issue 201). On 2026-10-02 two merges a minute apart made two plans;
|
||||||
// which widened its seat's row, and the node-engine's plan then sent the control node the controller build
|
// the controller's plan rolled the new controller, which widened its seat's row, and the node-engine's plan then
|
||||||
// before it. The older binary read the row it could not serve and the mesh lost its voice for ten minutes — the
|
// sent the control node the controller build before it. The older binary read the row it could not serve and the
|
||||||
// thing that would have corrected it was the thing that was down. ADR 0185 took the outage out of it (a
|
// mesh lost its voice for ten minutes — the thing that would have corrected it was the thing that was down. ADR
|
||||||
// controller behind its row serves what it can); the race is closed here, without needing its trigger: whatever
|
// 0185 took the outage out of it (a controller behind its row serves what it can); the race is closed here,
|
||||||
// composed the older build, the send that would carry it is refused.
|
// without needing its trigger: whatever made the mesh's record older, the send keeps the build.
|
||||||
//
|
//
|
||||||
// Older is by the builds' registration order — when each was asked (novox/hq issue 219) — never by the
|
// **Kept, not refused.** The send composes the controller or the node-engine at the build the machine was last
|
||||||
// commits' dates. A gate's own put-back is the one send that may carry it, for the modules it puts back: it says
|
// sent, as a recorded module is kept (ADR 0245), and carries everything else: no other module, secret, healer or
|
||||||
// so (sendScope.puttingBack), and the controller's put-back is held to the store's schema already (issue 352).
|
// walk waits on it, and nothing is retried. A plan whose own build is the older one finds its gate superseded by
|
||||||
// A person's push carries what it carries, as every other refusal of a send leaves it.
|
// the build kept there (issue 352) and ends. The mesh's record stays older, and a warning condition says so until
|
||||||
|
// a send carries the record — a person's `push <node>` accepts the older build — or the record moves forward: a
|
||||||
|
// newer merge, or `rebuild mesh-controller`.
|
||||||
//
|
//
|
||||||
// The refusal is said in the plan's words — the send's error, which the plan keeps as its note and tries again
|
// Older is by the builds' registration order — when each was asked (novox/hq issue 219) — never by the commits'
|
||||||
// — and as a warning condition on the machine, cleared by the first send there that carries no such move.
|
// dates. So a retry or a rebuild of an old commit, asked anew, counts as newer: it is registered as the module's
|
||||||
|
// current build by the same order, and the guard does not stop that move. A build that failed its gate is never
|
||||||
|
// ahead: the machine was sent it and the record was put back before it (gateFailed), and if the put-back's own
|
||||||
|
// send was refused, the next send completes it rather than keeping the failed build there. A gate's own put-back
|
||||||
|
// carries its modules back whatever the order (sendScope.puttingBack); the controller's is held to the store's
|
||||||
|
// schema already (issue 352).
|
||||||
|
|
||||||
// errCoreDowngrade is a send refused because it would put a core module on a machine back behind the build that
|
// kindCoreDowngrade is a core module kept at the build its machine was last sent, over an older record.
|
||||||
// machine runs.
|
const kindCoreDowngrade = "core-downgrade-kept"
|
||||||
var errCoreDowngrade = errors.New("a send would put the mesh's own core back behind the build its machine runs")
|
|
||||||
|
|
||||||
// kindCoreDowngrade is that refusal, as a condition.
|
|
||||||
const kindCoreDowngrade = "core-downgrade-refused"
|
|
||||||
|
|
||||||
// coreDowngradeSource is the condition's source: the sends, which raise and clear it.
|
// coreDowngradeSource is the condition's source: the sends, which raise and clear it.
|
||||||
const coreDowngradeSource = "sends"
|
const coreDowngradeSource = "sends"
|
||||||
@@ -42,67 +44,84 @@ const coreDowngradeSource = "sends"
|
|||||||
// store and its seat's row, and the node-engine, which reads what the controller declares.
|
// store and its seat's row, and the node-engine, which reads what the controller declares.
|
||||||
var downgradeGuarded = []string{lease.ComponentController, lease.ComponentEngine}
|
var downgradeGuarded = []string{lease.ComponentController, lease.ComponentEngine}
|
||||||
|
|
||||||
// buildAsked is when the newest build of a module from a commit that worked was asked — or recorded, when the
|
// askedBuild is what the order needs of one build of a module: when it was asked, and whether it failed its gate.
|
||||||
// request time is not known — and false when the records hold none. A variable, so a test reads its own order.
|
type askedBuild struct {
|
||||||
var buildAsked = func(ctx context.Context, inv *inventory.Inventory, module, commit string) (time.Time, bool, error) {
|
asked time.Time
|
||||||
|
failedGate bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildAsked is the newest build of a module from a commit that worked — when it was asked (or recorded, when
|
||||||
|
// the request time is not known), and whether it failed its gate — and false when the records hold none. A
|
||||||
|
// variable, so a test reads its own order.
|
||||||
|
var buildAsked = func(ctx context.Context, inv *inventory.Inventory, module, commit string) (askedBuild, bool, error) {
|
||||||
builds, err := inv.Builds(ctx, module, askedLookBack)
|
builds, err := inv.Builds(ctx, module, askedLookBack)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return time.Time{}, false, err
|
return askedBuild{}, false, err
|
||||||
}
|
}
|
||||||
for _, b := range builds {
|
for _, b := range builds {
|
||||||
if b.Worked() && sameCommit(b.Commit, commit) {
|
if b.Worked() && sameCommit(b.Commit, commit) {
|
||||||
return b.AskedOrAt(), true, nil
|
failed, err := inv.GateFailed(ctx, b.ID)
|
||||||
|
if err != nil {
|
||||||
|
return askedBuild{}, false, err
|
||||||
|
}
|
||||||
|
return askedBuild{asked: b.AskedOrAt(), failedGate: failed}, true, nil
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return time.Time{}, false, nil
|
return askedBuild{}, false, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// askedLookBack is how many of a module's newest builds are read for when a build was asked.
|
// askedLookBack is how many of a module's newest builds are read for when a build was asked.
|
||||||
const askedLookBack = 50
|
const askedLookBack = 50
|
||||||
|
|
||||||
// coreDowngrade is one move of a core module to an older build than its machine runs.
|
// coreDowngrade is one core module a machine was last sent a build of that is newer than the mesh's record.
|
||||||
type coreDowngrade struct {
|
type coreDowngrade struct {
|
||||||
move inventory.CarriedMove
|
module, node, component string
|
||||||
component string
|
// sent is the build the machine was last sent, record the one the mesh holds.
|
||||||
fromAsked, asked time.Time
|
sent, record string
|
||||||
|
sentAsked, recordAsked time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
func (d coreDowngrade) String() string {
|
func (d coreDowngrade) String() string {
|
||||||
return fmt.Sprintf("%s would be sent %s %s, older than the %s it runs (asked %s, before %s) — no send but a "+
|
return fmt.Sprintf("%s was last sent %s %s, and the mesh holds the older %s (asked %s, before %s): a send there "+
|
||||||
"gate's own put-back moves the %s back (novox/hq issue 201)", d.move.Node, d.move.Module, short(d.move.To),
|
"keeps %s; `push %s` sends the older build, and a newer merge or `rebuild %s` moves the record forward "+
|
||||||
short(d.move.From), d.asked.UTC().Format(time.RFC3339), d.fromAsked.UTC().Format(time.RFC3339), d.component)
|
"(novox/hq issue 201)", d.node, d.module, short(d.sent), short(d.record),
|
||||||
|
d.recordAsked.UTC().Format(time.RFC3339), d.sentAsked.UTC().Format(time.RFC3339), short(d.sent), d.node,
|
||||||
|
d.module)
|
||||||
}
|
}
|
||||||
|
|
||||||
// coreDowngrades is the moves of a machine's send that put a core module back to a build asked before the one it
|
// coreDowngrades is, for one machine, each guarded core module whose record is older than the build the machine
|
||||||
// runs, leaving out what a put-back puts back. A move whose either build the records do not hold is not proved
|
// was last sent, leaving out what a gate's own put-back puts back and a sent build that failed its gate. A move
|
||||||
// older, and is left to the gate that judges it. Pure but for asked.
|
// whose either build the records do not hold is not proved older and is left to the gate that judges it. Pure
|
||||||
func coreDowngrades(moves []inventory.CarriedMove, scope sendScope,
|
// but for asked.
|
||||||
asked func(module, commit string) (time.Time, bool, error)) ([]coreDowngrade, error) {
|
func coreDowngrades(node string, sent map[string]string, record map[string]string, scope sendScope,
|
||||||
|
asked func(module, commit string) (askedBuild, bool, error)) ([]coreDowngrade, error) {
|
||||||
var out []coreDowngrade
|
var out []coreDowngrade
|
||||||
for _, mv := range moves {
|
for module, was := range sent {
|
||||||
component := coreComponent(mv.Module)
|
component := coreComponent(module)
|
||||||
if !guarded(component) || mv.From == "" || mv.To == "" || sameCommit(mv.From, mv.To) {
|
now := record[module]
|
||||||
|
if !guarded(component) || was == "" || now == "" || sameCommit(was, now) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if scope.puttingBack && scope.modules[mv.Module] {
|
if scope.puttingBack && scope.modules[module] {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
from, known, err := asked(mv.Module, mv.From)
|
from, known, err := asked(module, was)
|
||||||
if err != nil || !known {
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", mv.Module, short(mv.From), err)
|
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(was), err)
|
||||||
}
|
}
|
||||||
|
if !known || from.failedGate {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
to, known, err := asked(mv.Module, mv.To)
|
to, known, err := asked(module, now)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", mv.Module, short(mv.To), err)
|
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(now), err)
|
||||||
}
|
}
|
||||||
if known && to.Before(from) {
|
if known && to.asked.Before(from.asked) {
|
||||||
out = append(out, coreDowngrade{move: mv, component: component, fromAsked: from, asked: to})
|
out = append(out, coreDowngrade{module: module, node: node, component: component, sent: was, record: now,
|
||||||
|
sentAsked: from.asked, recordAsked: to.asked})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
sort.Slice(out, func(i, j int) bool { return out[i].move.Module < out[j].move.Module })
|
sort.Slice(out, func(i, j int) bool { return out[i].module < out[j].module })
|
||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -115,32 +134,29 @@ func guarded(component string) bool {
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
// refuseCoreDowngrades says, for one machine's moves, each that would put a core module back, raising its
|
// coreKept is, for a send that keeps (sendKeeps), each guarded core module of a machine to keep at the build it
|
||||||
// condition; and clears the condition of each guarded component that machine's send does not move back.
|
// was last sent, module → commit, each said and raised as a condition.
|
||||||
func refuseCoreDowngrades(ctx context.Context, inv *inventory.Inventory, scope sendScope, node string,
|
func coreKept(ctx context.Context, inv *inventory.Inventory, node string, sent map[string]string,
|
||||||
moves []inventory.CarriedMove) ([]string, error) {
|
current map[string]inventory.CurrentBuild) (map[string]string, error) {
|
||||||
found, err := coreDowngrades(moves, scope, func(module, commit string) (time.Time, bool, error) {
|
record := map[string]string{}
|
||||||
|
for m, c := range current {
|
||||||
|
record[m] = c.Commit
|
||||||
|
}
|
||||||
|
found, err := coreDowngrades(node, sent, record, scopeOf(ctx), func(module, commit string) (askedBuild, bool, error) {
|
||||||
return buildAsked(ctx, inv, module, commit)
|
return buildAsked(ctx, inv, module, commit)
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil || len(found) == 0 {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
var said []string
|
out := map[string]string{}
|
||||||
refused := map[string]bool{}
|
|
||||||
for _, d := range found {
|
for _, d := range found {
|
||||||
said = append(said, d.String())
|
out[d.module] = d.sent
|
||||||
refused[d.component] = true
|
|
||||||
sayCoreDowngrade(ctx, d)
|
sayCoreDowngrade(ctx, d)
|
||||||
}
|
}
|
||||||
for _, c := range downgradeGuarded {
|
return out, nil
|
||||||
if !refused[c] {
|
|
||||||
clearCoreDowngrade(ctx, c, node)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return said, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// coreDowngradeObservation is a refused downgrade as a condition: `core.<component>.<machine>.<kind>`.
|
// coreDowngradeObservation is a kept core build as a condition: `core.<component>.<machine>.<kind>`.
|
||||||
func coreDowngradeObservation(component, node, summary string) conditions.Observation {
|
func coreDowngradeObservation(component, node, summary string) conditions.Observation {
|
||||||
return conditions.Observation{Scope: conditions.ScopeCore, ID: component + "." + node, Kind: kindCoreDowngrade,
|
return conditions.Observation{Scope: conditions.ScopeCore, ID: component + "." + node, Kind: kindCoreDowngrade,
|
||||||
Token: kindCoreDowngrade, Machine: node, Severity: conditions.Warning, Source: coreDowngradeSource,
|
Token: kindCoreDowngrade, Machine: node, Severity: conditions.Warning, Source: coreDowngradeSource,
|
||||||
@@ -148,24 +164,49 @@ func coreDowngradeObservation(component, node, summary string) conditions.Observ
|
|||||||
}
|
}
|
||||||
|
|
||||||
func sayCoreDowngrade(ctx context.Context, d coreDowngrade) {
|
func sayCoreDowngrade(ctx context.Context, d coreDowngrade) {
|
||||||
fmt.Printf("NOT SENT: %s\n", d)
|
fmt.Printf("KEPT: %s\n", d)
|
||||||
keeper := conditionsFrom
|
keeper := conditionsFrom
|
||||||
if keeper == nil {
|
if keeper == nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if _, err := keeper.Observe(ctx, coreDowngradeObservation(d.component, d.move.Node, d.String())); err != nil {
|
if _, err := keeper.Observe(ctx, coreDowngradeObservation(d.component, d.node, d.String())); err != nil {
|
||||||
fmt.Printf("the condition for %s's refused downgrade on %s could not be raised: %v\n", d.move.Module,
|
fmt.Printf("the condition for %s kept on %s could not be raised: %v\n", d.module, d.node, err)
|
||||||
d.move.Node, err)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func clearCoreDowngrade(ctx context.Context, component, node string) {
|
// settleCoreDowngrades clears, after a send to a machine is away, the condition of each guarded core module the
|
||||||
|
// send carried at the mesh's record — a person's push that accepted the older build, or a send after the record
|
||||||
|
// moved forward — or did not carry at all. A send that kept the newer build leaves it open.
|
||||||
|
func settleCoreDowngrades(ctx context.Context, inv *inventory.Inventory, node string, carried map[string]string) {
|
||||||
keeper := conditionsFrom
|
keeper := conditionsFrom
|
||||||
if keeper == nil {
|
if keeper == nil || carried == nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
var open []string
|
||||||
|
for _, c := range downgradeGuarded {
|
||||||
|
if _, found, err := keeper.Get(ctx, coreDowngradeObservation(c, node, "").Key()); err == nil && found {
|
||||||
|
open = append(open, c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(open) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
current, err := inv.CurrentBuilds(ctx)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Printf("whether %s still keeps a core build over the mesh's record cannot be read: %v\n", node, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for module, now := range current {
|
||||||
|
component := coreComponent(module)
|
||||||
|
if !guarded(component) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if was, has := carried[module]; has && !sameCommit(was, now.Commit) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
key := coreDowngradeObservation(component, node, "").Key()
|
key := coreDowngradeObservation(component, node, "").Key()
|
||||||
if _, err := keeper.Clear(ctx, key, "a send to "+node+" carries no older "+component); err != nil {
|
if _, err := keeper.Clear(ctx, key, node+" was sent the "+component+" the mesh holds"); err != nil {
|
||||||
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
|
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package main
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -15,104 +14,76 @@ import (
|
|||||||
|
|
||||||
// novox/hq issue 201: on 2026-10-02 a plan sent the control node a controller older than the one running there,
|
// novox/hq issue 201: on 2026-10-02 a plan sent the control node a controller older than the one running there,
|
||||||
// whose seat row the newer one had already widened; the older binary refused the row and the mesh went quiet for
|
// whose seat row the newer one had already widened; the older binary refused the row and the mesh went quiet for
|
||||||
// ten minutes. No send but a gate's own put-back moves the controller or the node-engine back behind the build its
|
// ten minutes. No send but a person's push and a gate's own put-back moves the controller or the node-engine back
|
||||||
// machine runs, by the builds' registration order.
|
// behind the build its machine was last sent: the send keeps it there, and carries the rest.
|
||||||
|
|
||||||
// askedIn answers when a build was asked from a fixed table: module@commit → when.
|
// askedIn answers a build from a fixed table: module@commit → when it was asked; failed names those that failed
|
||||||
func askedIn(table map[string]time.Time) func(module, commit string) (time.Time, bool, error) {
|
// their gate.
|
||||||
return func(module, commit string) (time.Time, bool, error) {
|
func askedIn(table map[string]time.Time, failed ...string) func(module, commit string) (askedBuild, bool, error) {
|
||||||
|
return func(module, commit string) (askedBuild, bool, error) {
|
||||||
at, known := table[module+"@"+commit]
|
at, known := table[module+"@"+commit]
|
||||||
return at, known, nil
|
b := askedBuild{asked: at}
|
||||||
|
for _, f := range failed {
|
||||||
|
if f == module+"@"+commit {
|
||||||
|
b.failedGate = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return b, known, nil
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestACoreModuleIsNeverMovedBehindTheBuildItsMachineRuns(t *testing.T) {
|
func TestACoreModuleIsKeptOverAnOlderRecord(t *testing.T) {
|
||||||
old, now := t0, t0.Add(time.Minute)
|
old, now := t0, t0.Add(time.Minute)
|
||||||
asked := askedIn(map[string]time.Time{
|
table := map[string]time.Time{
|
||||||
"mesh-controller@c1": old, "mesh-controller@c2": now,
|
"mesh-controller@c1": old, "mesh-controller@c2": now,
|
||||||
"mesh-host@h1": old, "mesh-host@h2": now,
|
"mesh-host@h1": old, "mesh-host@h2": now,
|
||||||
"app@a1": old, "app@a2": now,
|
"app@a1": old, "app@a2": now,
|
||||||
})
|
|
||||||
back := []inventory.CarriedMove{
|
|
||||||
{Module: "mesh-controller", Node: "anchor", From: "c2", To: "c1"},
|
|
||||||
{Module: "mesh-host", Node: "anchor", From: "h2", To: "h1"},
|
|
||||||
{Module: "app", Node: "anchor", From: "a2", To: "a1"},
|
|
||||||
}
|
}
|
||||||
found, err := coreDowngrades(back, sendScope{judged: map[string]bool{"anchor": true}}, asked)
|
sent := map[string]string{"mesh-controller": "c2", "mesh-host": "h2", "app": "a2"}
|
||||||
|
record := map[string]string{"mesh-controller": "c1", "mesh-host": "h1", "app": "a1"}
|
||||||
|
found, err := coreDowngrades("anchor", sent, record, sendScope{judged: map[string]bool{"anchor": true}}, askedIn(table))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if len(found) != 2 || found[0].move.Module != "mesh-controller" || found[0].component != lease.ComponentController ||
|
if len(found) != 2 || found[0].module != "mesh-controller" || found[0].component != lease.ComponentController ||
|
||||||
found[1].move.Module != "mesh-host" || found[1].component != lease.ComponentEngine {
|
found[0].sent != "c2" || found[1].module != "mesh-host" || found[1].component != lease.ComponentEngine {
|
||||||
t.Fatalf("the controller and the node-engine sent back, and nothing else, are refused: %+v", found)
|
t.Fatalf("the controller and the node-engine over an older record are kept, and nothing else: %+v", found)
|
||||||
}
|
}
|
||||||
if said := found[0].String(); !strings.Contains(said, "anchor would be sent mesh-controller c1") ||
|
if said := found[0].String(); !strings.Contains(said, "anchor was last sent mesh-controller c2") ||
|
||||||
!strings.Contains(said, "issue 201") {
|
!strings.Contains(said, "`push anchor`") || !strings.Contains(said, "`rebuild mesh-controller`") {
|
||||||
t.Fatalf("the refusal says the machine, the module and the build: %s", said)
|
t.Fatalf("it says the machine, the build kept and the two ways out: %s", said)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Forward, the same build, or a build the records do not hold: not refused.
|
// The record forward, the same build, or a build the records do not hold: nothing kept.
|
||||||
for _, mv := range []inventory.CarriedMove{
|
for _, c := range [][2]string{{"c1", "c2"}, {"c2", "c2"}, {"c2", "c9"}, {"c9", "c1"}} {
|
||||||
{Module: "mesh-controller", Node: "anchor", From: "c1", To: "c2"},
|
found, err := coreDowngrades("anchor", map[string]string{"mesh-controller": c[0]},
|
||||||
{Module: "mesh-controller", Node: "anchor", From: "c2", To: "c2"},
|
map[string]string{"mesh-controller": c[1]}, sendScope{}, askedIn(table))
|
||||||
{Module: "mesh-controller", Node: "anchor", From: "c2", To: "c9"},
|
if err != nil || len(found) != 0 {
|
||||||
{Module: "mesh-controller", Node: "anchor", From: "c9", To: "c1"},
|
t.Fatalf("sent %s, record %s: no downgrade proved, nothing kept: %+v %v", c[0], c[1], found, err)
|
||||||
} {
|
|
||||||
if found, err := coreDowngrades([]inventory.CarriedMove{mv}, sendScope{}, asked); err != nil || len(found) != 0 {
|
|
||||||
t.Fatalf("%+v is no downgrade proved: %+v %v", mv, found, err)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A sent build that failed its gate is never ahead: the record put back before it is sent.
|
||||||
|
if found, _ := coreDowngrades("anchor", sent, record, sendScope{}, askedIn(table, "mesh-controller@c2")); len(found) != 1 ||
|
||||||
|
found[0].module != "mesh-host" {
|
||||||
|
t.Fatalf("a controller that failed its gate was kept over the build put back: %+v", found)
|
||||||
|
}
|
||||||
|
|
||||||
// A gate's own put-back carries its own modules back, and nothing else.
|
// A gate's own put-back carries its own modules back, and nothing else.
|
||||||
putBack := sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true}
|
putBack := sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true}
|
||||||
found, err = coreDowngrades(back, putBack, asked)
|
if found, _ := coreDowngrades("anchor", sent, record, putBack, askedIn(table)); len(found) != 1 || found[0].module != "mesh-host" {
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if len(found) != 1 || found[0].move.Module != "mesh-host" {
|
|
||||||
t.Fatalf("a put-back of the controller carries it back, and not the node-engine: %+v", found)
|
t.Fatalf("a put-back of the controller carries it back, and not the node-engine: %+v", found)
|
||||||
}
|
}
|
||||||
// Naming a module is not a put-back.
|
if found, _ := coreDowngrades("anchor", sent, record, sendScope{modules: map[string]bool{"mesh-controller": true}},
|
||||||
if found, _ := coreDowngrades(back[:1], sendScope{modules: map[string]bool{"mesh-controller": true}}, asked); len(found) != 1 {
|
askedIn(table)); len(found) != 2 {
|
||||||
t.Fatalf("a send for the controller that is no put-back carries it back: %+v", found)
|
t.Fatalf("a send for the controller that is no put-back carries it back: %+v", found)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The refusal is a warning condition on the machine, which the next send there carrying no such move clears.
|
// core201 is a mesh whose control node, anchor, was sent the newer controller c2 while the mesh's record of the
|
||||||
func TestARefusedCoreDowngradeIsSaidAndClearedByTheNextSend(t *testing.T) {
|
// controller is the build before it, c1, which passed its gate when it rolled out.
|
||||||
k, _ := withConditionsInMemory(t)
|
func core201(t *testing.T) *stores {
|
||||||
ctx := t.Context()
|
t.Helper()
|
||||||
was := buildAsked
|
|
||||||
buildAsked = func(_ context.Context, _ *inventory.Inventory, module, commit string) (time.Time, bool, error) {
|
|
||||||
return askedIn(map[string]time.Time{"mesh-controller@c1": t0, "mesh-controller@c2": t0.Add(time.Minute)})(module, commit)
|
|
||||||
}
|
|
||||||
t.Cleanup(func() { buildAsked = was })
|
|
||||||
|
|
||||||
said, err := refuseCoreDowngrades(ctx, nil, sendScope{}, "anchor",
|
|
||||||
[]inventory.CarriedMove{{Module: "mesh-controller", Node: "anchor", From: "c2", To: "c1"}})
|
|
||||||
if err != nil || len(said) != 1 {
|
|
||||||
t.Fatalf("the downgrade is refused: %v %v", said, err)
|
|
||||||
}
|
|
||||||
key := coreDowngradeObservation(lease.ComponentController, "anchor", "").Key()
|
|
||||||
c, open, err := k.Get(ctx, key)
|
|
||||||
if err != nil || !open || !strings.Contains(c.Summary, "mesh-controller c1") {
|
|
||||||
t.Fatalf("the condition %s is raised, saying the build: %+v %v %v", key, c, open, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
said, err = refuseCoreDowngrades(ctx, nil, sendScope{}, "anchor",
|
|
||||||
[]inventory.CarriedMove{{Module: "mesh-controller", Node: "anchor", From: "c1", To: "c2"}})
|
|
||||||
if err != nil || len(said) != 0 {
|
|
||||||
t.Fatalf("a move forward is sent: %v %v", said, err)
|
|
||||||
}
|
|
||||||
if _, open, err := k.Get(ctx, key); err != nil || open {
|
|
||||||
t.Fatalf("the condition is cleared by a send carrying no older controller: %v %v", open, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestReplay201 replays the race on a store: the control node runs the newer controller, the mesh's record of
|
|
||||||
// the controller is the build before it, and a plan's gated send there — the node-engine's plan of 2026-10-02 —
|
|
||||||
// is refused, said as the plan's words; the gate's own put-back of the controller is not.
|
|
||||||
func TestReplay201(t *testing.T) {
|
|
||||||
open := aMesh(t)
|
open := aMesh(t)
|
||||||
ctx := t.Context()
|
ctx := t.Context()
|
||||||
inv := open.inventory
|
inv := open.inventory
|
||||||
@@ -128,6 +99,10 @@ func TestReplay201(t *testing.T) {
|
|||||||
return b
|
return b
|
||||||
}
|
}
|
||||||
before := record("c1aaaaaaaaaa", old)
|
before := record("c1aaaaaaaaaa", old)
|
||||||
|
if err := inv.RecordGate(ctx, inventory.GateVerdict{Build: before.ID, Module: "mesh-controller", Commit: before.Commit,
|
||||||
|
Machines: []string{"anchor"}, Verdict: inventory.GatePassed}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
record("c2bbbbbbbbbb", now)
|
record("c2bbbbbbbbbb", now)
|
||||||
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"}, inventory.Source{
|
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"}, inventory.Source{
|
||||||
Repository: "novox/mesh-controller", BuiltFrom: "c2bbbbbbbbbb", Head: "c2bbbbbbbbbb", Asked: now}); err != nil {
|
Repository: "novox/mesh-controller", BuiltFrom: "c2bbbbbbbbbb", Head: "c2bbbbbbbbbb", Asked: now}); err != nil {
|
||||||
@@ -139,7 +114,6 @@ func TestReplay201(t *testing.T) {
|
|||||||
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"mesh-controller": "c2bbbbbbbbbb"}); err != nil {
|
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"mesh-controller": "c2bbbbbbbbbb"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// The mesh's record goes back to the build before, as the record went on 2026-10-02.
|
|
||||||
if err := inv.RestoreModule(ctx, before); err != nil {
|
if err := inv.RestoreModule(ctx, before); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -156,14 +130,70 @@ func TestReplay201(t *testing.T) {
|
|||||||
return f.moves(node, modules, sent, known, all), nil
|
return f.moves(node, modules, sent, known, all), nil
|
||||||
}
|
}
|
||||||
t.Cleanup(func() { machineMoves = wasMoves })
|
t.Cleanup(func() { machineMoves = wasMoves })
|
||||||
|
return open
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReplay201 replays the race on a store: the control node runs the newer controller and the mesh's record is
|
||||||
|
// the build before it. A plan's send there — the node-engine's plan of 2026-10-02 — goes, keeps the newer
|
||||||
|
// controller, and says so as a condition; the gate's own put-back carries it back; a send that carries the record,
|
||||||
|
// as a person's push does, clears the condition.
|
||||||
|
func TestReplay201(t *testing.T) {
|
||||||
|
open := core201(t)
|
||||||
|
ctx := t.Context()
|
||||||
|
k, _ := withConditionsInMemory(t)
|
||||||
|
|
||||||
judged := withScope(ctx, sendScope{judged: map[string]bool{"anchor": true}})
|
judged := withScope(ctx, sendScope{judged: map[string]bool{"anchor": true}})
|
||||||
if _, err := ungatedIn(judged, open, []string{"anchor"}, ""); !errors.Is(err, errCoreDowngrade) ||
|
if kept, err := ungatedIn(judged, open, []string{"anchor"}, ""); err != nil || len(kept) != 1 {
|
||||||
!strings.Contains(err.Error(), "mesh-controller c1aaaaaa") {
|
t.Fatalf("a plan's send to the machine is made, the rest of it with it: %v %v", kept, err)
|
||||||
t.Fatalf("a plan's send of the older controller to the machine running the newer is refused: %v", err)
|
|
||||||
}
|
}
|
||||||
putBack := withScope(ctx, sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true})
|
keeps, err := recordedKept(keepingRecorded(judged), open, "anchor")
|
||||||
if kept, err := ungatedIn(putBack, open, []string{"anchor"}, ""); err != nil || len(kept) != 1 {
|
if err != nil || keeps["mesh-controller"] != "c2bbbbbbbbbb" {
|
||||||
t.Fatalf("the gate's own put-back of the controller is sent: %v %v", kept, err)
|
t.Fatalf("the send keeps the newer controller there: %v %v", keeps, err)
|
||||||
|
}
|
||||||
|
shelf := map[string]catalogue.Manifest{"mesh-controller": {Module: "mesh-controller", Version: "old"}}
|
||||||
|
if _, err := keepRecorded(keepingRecorded(judged), open, "anchor", shelf); err != nil || shelf["mesh-controller"].Version != "1" {
|
||||||
|
t.Fatalf("the send composes the kept build's manifest: %+v %v", shelf, err)
|
||||||
|
}
|
||||||
|
key := coreDowngradeObservation(lease.ComponentController, "anchor", "").Key()
|
||||||
|
if c, found, err := k.Get(ctx, key); err != nil || !found || !strings.Contains(c.Summary, "c2bbbbbb") {
|
||||||
|
t.Fatalf("the condition %s is raised, saying the build kept: %+v %v %v", key, c, found, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
putBack := keepingRecorded(withScope(ctx, sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true}))
|
||||||
|
if keeps, err := recordedKept(putBack, open, "anchor"); err != nil || len(keeps) != 0 {
|
||||||
|
t.Fatalf("the gate's own put-back of the controller carries it back: %v %v", keeps, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
settleCoreDowngrades(ctx, open.inventory, "anchor", map[string]string{"mesh-controller": "c2bbbbbbbbbb"})
|
||||||
|
if _, found, _ := k.Get(ctx, key); !found {
|
||||||
|
t.Fatal("a send that kept the newer controller cleared the condition")
|
||||||
|
}
|
||||||
|
settleCoreDowngrades(ctx, open.inventory, "anchor", map[string]string{"mesh-controller": "c1aaaaaaaaaa"})
|
||||||
|
if _, found, _ := k.Get(ctx, key); found {
|
||||||
|
t.Fatal("a send carrying the mesh's record — a person's push — leaves the condition open")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The review of #273: a gate failed the newer controller, put the record back, and the put-back's own send was
|
||||||
|
// refused. The next automatic send — a healer's, which judges nothing — completes the put-back: a build that
|
||||||
|
// failed its gate is never kept.
|
||||||
|
func TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend(t *testing.T) {
|
||||||
|
open := core201(t)
|
||||||
|
ctx := t.Context()
|
||||||
|
k, _ := withConditionsInMemory(t)
|
||||||
|
if err := open.inventory.RecordGate(ctx, inventory.GateVerdict{Build: "build-mesh-controller-c2bbbbbbbbbb",
|
||||||
|
Module: "mesh-controller", Commit: "c2bbbbbbbbbb", Previous: "c1aaaaaaaaaa", Machines: []string{"anchor"},
|
||||||
|
Verdict: inventory.GateFailed, Rollback: inventory.NotRolledBack}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
healer := withScope(ctx, sendScope{})
|
||||||
|
if kept, err := ungatedIn(healer, open, []string{"anchor"}, ""); err != nil || len(kept) != 1 {
|
||||||
|
t.Fatalf("the healer's send is made: %v %v", kept, err)
|
||||||
|
}
|
||||||
|
if keeps, err := recordedKept(keepingRecorded(healer), open, "anchor"); err != nil || len(keeps) != 0 {
|
||||||
|
t.Fatalf("the failed controller was kept over the build put back: %v %v", keeps, err)
|
||||||
|
}
|
||||||
|
if open, _ := k.Open(ctx); len(open) != 0 {
|
||||||
|
t.Fatalf("a completed put-back raised a condition: %+v", open)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -182,7 +182,7 @@ var plainWordings = map[string]func(conditions.Observation) words{
|
|||||||
"register it, so it was not made.", module, m),
|
"register it, so it was not made.", module, m),
|
||||||
Resolved: "Resolved: " + module + " on " + m + " is settled"}
|
Resolved: "Resolved: " + module + " on " + m + " is settled"}
|
||||||
}),
|
}),
|
||||||
// A send that would have put the controller or the node-engine back behind what a machine runs (novox/hq
|
// The controller or the node-engine kept at the build a machine was last sent, over an older record (novox/hq
|
||||||
// issue 201).
|
// issue 201).
|
||||||
kindCoreDowngrade: worded(func(o conditions.Observation) words {
|
kindCoreDowngrade: worded(func(o conditions.Observation) words {
|
||||||
m := machineOr(o, "a machine")
|
m := machineOr(o, "a machine")
|
||||||
@@ -190,11 +190,11 @@ var plainWordings = map[string]func(conditions.Observation) words{
|
|||||||
if o.Scope != conditions.ScopeCore || !guarded(component) {
|
if o.Scope != conditions.ScopeCore || !guarded(component) {
|
||||||
component = "core component"
|
component = "core component"
|
||||||
}
|
}
|
||||||
return words{Headline: "An older " + component + " was not sent to " + m,
|
return words{Headline: m + " keeps a newer " + component + " than the mesh holds",
|
||||||
Needs: "decide which build of the " + component + " the mesh should hold; the details say which.",
|
Needs: fmt.Sprintf("push %s to take the older build, or a newer merge or rebuild moves the record forward.", m),
|
||||||
Explanation: fmt.Sprintf("A send would have put the mesh's own %s on %s back to an older build than "+
|
Explanation: fmt.Sprintf("The mesh's record of its own %s is older than the build %s was last sent, so "+
|
||||||
"the one it runs, so it was not sent and %s keeps its build.", component, m, m),
|
"the mesh's sends keep the newer one there rather than put it back.", component, m),
|
||||||
Resolved: "Resolved: sends to " + m + " carry no older " + component}
|
Resolved: "Resolved: " + m + " runs the " + component + " the mesh holds"}
|
||||||
}),
|
}),
|
||||||
kindBindingKept: worded(func(o conditions.Observation) words {
|
kindBindingKept: worded(func(o conditions.Observation) words {
|
||||||
m := machineOr(o, "a machine")
|
m := machineOr(o, "a machine")
|
||||||
|
|||||||
@@ -1027,6 +1027,9 @@ func (b overTheBus) declare(ctx context.Context, s readyNode, body []byte) (stri
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
|
// A send that carried the mesh's record of the controller and the node-engine — a person's push included —
|
||||||
|
// ends the condition of one kept over it (novox/hq issue 201).
|
||||||
|
settleCoreDowngrades(ctx, b.open.inventory, s.node, s.declared.Builds)
|
||||||
if len(s.declared.Bindings) > 0 {
|
if len(s.declared.Bindings) > 0 {
|
||||||
// And where it bound each consumer of a provision that keeps its data (novox/hq ADR 0232):
|
// And where it bound each consumer of a provision that keeps its data (novox/hq ADR 0232):
|
||||||
// what the next resolution keeps it at. On the same outliving context as the send's record.
|
// what the next resolution keeps it at. On the same outliving context as the send's record.
|
||||||
|
|||||||
@@ -135,6 +135,17 @@ func recordedKept(ctx context.Context, open *stores, node string) (map[string]st
|
|||||||
}
|
}
|
||||||
out[m] = was
|
out[m] = was
|
||||||
}
|
}
|
||||||
|
// **And the controller and the node-engine at the build the machine was last sent, over an older record**
|
||||||
|
// (novox/hq issue 201, core_downgrade.go). Under keepingEveryBuild they are kept already.
|
||||||
|
if !every {
|
||||||
|
core, err := coreKept(ctx, inv, node, sent, current)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
for m, was := range core {
|
||||||
|
out[m] = was
|
||||||
|
}
|
||||||
|
}
|
||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -159,6 +170,12 @@ func keepRecorded(ctx context.Context, open *stores, node string, shelf map[stri
|
|||||||
return nil, fmt.Errorf("%s runs %s's build %s, which the build records no longer hold: the bus's user "+
|
return nil, fmt.Errorf("%s runs %s's build %s, which the build records no longer hold: the bus's user "+
|
||||||
"list cannot be sent there alone with it kept (novox/hq issue 490)", node, m, short(kept[m]))
|
"list cannot be sent there alone with it kept (novox/hq issue 490)", node, m, short(kept[m]))
|
||||||
}
|
}
|
||||||
|
if !found && guarded(coreComponent(m)) {
|
||||||
|
return nil, fmt.Errorf("%s was last sent %s's build %s, newer than the one the mesh holds, and the build records "+
|
||||||
|
"no longer hold it: this send cannot keep it and does not move it back — `push %s` sends the older build on "+
|
||||||
|
"a person's word, and a newer merge or `rebuild %s` moves the record forward (novox/hq issue 201)",
|
||||||
|
node, m, short(kept[m]), node, m)
|
||||||
|
}
|
||||||
if !found {
|
if !found {
|
||||||
return nil, fmt.Errorf("%s records rather than rolls out, and %s runs its build %s, which the build "+
|
return nil, fmt.Errorf("%s records rather than rolls out, and %s runs its build %s, which the build "+
|
||||||
"records no longer hold: this send cannot keep it and does not move it — `push %s` sends the new "+
|
"records no longer hold: this send cannot keep it and does not move it — `push %s` sends the new "+
|
||||||
|
|||||||
@@ -251,47 +251,26 @@ func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder st
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
var kept, refused, older []string
|
var kept, refused []string
|
||||||
for _, n := range names {
|
for _, n := range names {
|
||||||
// Every move, gated or not: a move to a build that passed a gate elsewhere is no gate's concern here, and
|
moves, err := machineMoves(ctx, open, f, n, false)
|
||||||
// may still be a core module's downgrade (novox/hq issue 201).
|
|
||||||
every, err := machineMoves(ctx, open, f, n, true)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
// The grants step composes every module at the build its machine was last sent (novox/hq issue 490):
|
// The grants step composes every module at the build its machine was last sent (novox/hq issue 490),
|
||||||
// a move it keeps is not made. Read, not assumed, so a move it could not keep is still refused here.
|
// and every send keeps the controller and the node-engine over an older record (issue 201): a move it
|
||||||
var keeps map[string]string
|
// keeps is not made. Read, not assumed, so a move it could not keep is still refused here.
|
||||||
if everyBuildKept(ctx, n) {
|
keeping := ctx
|
||||||
if keeps, err = recordedKept(ctx, open, n); err != nil {
|
if _, on := keptExcept(ctx); !on {
|
||||||
return nil, err
|
keeping = keepingRecorded(ctx) // as the send composes it (sendKeeps), which is set after this check
|
||||||
}
|
}
|
||||||
}
|
keeps, err := recordedKept(keeping, open, n)
|
||||||
var made []inventory.CarriedMove
|
|
||||||
for _, mv := range every {
|
|
||||||
if was, kept := keeps[mv.Module]; kept && sameCommit(was, mv.From) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
made = append(made, mv)
|
|
||||||
}
|
|
||||||
// **No send but a gate's own put-back moves the controller or the node-engine to an older build than
|
|
||||||
// the one its machine runs** (novox/hq issue 201).
|
|
||||||
downgrades, err := refuseCoreDowngrades(ctx, open.inventory, scope, n, made)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if len(downgrades) > 0 {
|
|
||||||
if n == addedHolder {
|
|
||||||
fmt.Printf("%s holds the bus and its user list changed, and it is not sent now: %s\n", n,
|
|
||||||
strings.Join(downgrades, "; "))
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
older = append(older, downgrades...)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
var waiting []string
|
var waiting []string
|
||||||
for _, mv := range made {
|
for _, mv := range moves {
|
||||||
if f.gated(mv.Module, mv.To) {
|
if was, kept := keeps[mv.Module]; kept && sameCommit(was, mv.From) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if !scope.judged[n] && !scope.modules[mv.Module] {
|
if !scope.judged[n] && !scope.modules[mv.Module] {
|
||||||
@@ -308,9 +287,6 @@ func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder st
|
|||||||
refused = append(refused, fmt.Sprintf("%s (%s)", n, strings.Join(waiting, ", ")))
|
refused = append(refused, fmt.Sprintf("%s (%s)", n, strings.Join(waiting, ", ")))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if len(older) > 0 {
|
|
||||||
return nil, fmt.Errorf("%w: %s", errCoreDowngrade, strings.Join(older, "; "))
|
|
||||||
}
|
|
||||||
if len(refused) > 0 {
|
if len(refused) > 0 {
|
||||||
return nil, fmt.Errorf("%w: %s — a walk sends them, one machine at a time, each judged (novox/hq ADR "+
|
return nil, fmt.Errorf("%w: %s — a walk sends them, one machine at a time, each judged (novox/hq ADR "+
|
||||||
"0236); `upgrade backlog` lists them", errUngated, strings.Join(refused, "; "))
|
"0236); `upgrade backlog` lists them", errUngated, strings.Join(refused, "; "))
|
||||||
|
|||||||
Reference in New Issue
Block a user