Compare commits

..
Author SHA1 Message Date
jschoubben b2af3d654f Trust a trunk whose whitelist adds the operator's own login, never an agent's (ADR 0534 §2, §6)
The gitea module keeps the operator's login beside mesh-tools on every main so
the operator can merge by hand; an agent's account there is still refused.
2026-10-23 12:10:11 +02:00
jschoubben 7676e67c5e Trust a module's trunk for who can merge into it, not for a required status (ADR 0534 §6)
The forge requires no status to merge any more (ADR 0534 §1); a trunk is
trusted when it takes no push, its merge whitelist names mesh-tools alone and
no administrator merges past its rule, so only mesh-delivery lands a commit
there, on a passing mesh/check and the operator's Approve. The old rule failed
the 11:51 delivery plan once mesh-catalog's status was unticked.
2026-10-23 12:02:00 +02:00
mesh-tools 0173d9c70e Merge pull request 'Keep the admin address's query in a test database's address, so store tests reach the checker's socket store (issue 1889)' (#445) from fix/1889-identity-store-ready into main 2026-10-23 07:37:06 +00:00
mesh-tools 89abce5df9 Merge pull request 'Judge each machine on a plan's last send and resume a plan lost to silence with plans retry, so a resumed plan is not failed by its own old sends (issue 1886, ADR 0531)' (#444) from fix/1886-resume-times into main 2026-10-23 07:26:31 +00:00
jschoubben 185d0294eb Keep the store address's query in every test database's address, so store tests reach a socket-addressed store (issue 1889)
mesh/merge-gate head 185d0294ebd4 merged into trunk 89abce5df955 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; the ch…
mesh/repo-check head 185d0294ebd4 merged into trunk 89abce5df955 — pass: its merge-check.sh passed
mesh/delivery covered after its stop: a later merge that contains it was delivered: novox/mesh-controller@0b312218f90f (merged as 791786d5 into main, wal…
mesh/check pass 185d0294ebd4 on 89abce5df955 | unit 1756 run, 0 kept, 4 skipped | integration 3034 run, 0 kept, 16 skipped | over bound: integration (…
The checker's integration store answers on a unix socket named in the query of MESH_TEST_POSTGRES
(?host=<socket dir>&sslmode=disable). The identity, inventory and licences fixtures cut the address at
its last slash and appended ?sslmode=disable, dropping host=, so pgx dialled its default socket where no
server is: every store test waited out its 20 s readiness bound, each package ran into go test's
10-minute timeout, and the verdict named only the first failure,
identity.TestACertificateChainsToTheMeshsOwnAuthority.

store.DatabaseForTest now builds that address for every fixture, keeping the query as the template raise
already did; identity's fresh is ForTest rather than a second copy. The readiness bound and its error,
which wraps the driver's cause, are unchanged.
2026-10-23 08:55:50 +02:00
jschoubben 4d4d51eb01 Pin the resume of a lost walk's refusals beside a newer or a related plan, and cite ADR 0360 §7 and ADR 0531 (issue 1886)
mesh/merge-gate head 4d4d51eb017f merged into trunk 219fdcd7f00c — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; the ch…
mesh/repo-check head 4d4d51eb017f merged into trunk 219fdcd7f00c — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-23 08:10:08 +02:00
jschoubben b7d996878b Judge a walk's machine on its last send, enter a resumed tier anew, and resume a walk lost to silence (issue 1886)
R4e, resumed at tier 1 after a stop of five hours, named ace and shanks silent within seconds of tier 1's send: the
walk judged each machine on every send it had made it, tier 0's of 00:23 too, held against tier 1's templates, and
measured the bound from it. A send carries the whole declaration, so the last one is what a machine is judged on.
The resume kept the tier's entry of before the stop, so the plan read waiting for 5h48m. And plans retry refused a
walk failed only by the gate's silence, though its machines had since applied.
2026-10-23 08:02:11 +02:00
18 changed files with 479 additions and 422 deletions
+36 -66
View File
@@ -30,9 +30,7 @@ import (
// closes at once and the process exits, so its service manager restarts it as a candidate (serve,
// in push.go). A renewal the bus does not answer in time is not a loss (novox/hq issue 477): the
// lease is kept, no new act starts once half the key's age passed unrenewed, and the next renewal
// made opens the gate again. A candidate backs off before it stands (novox/hq issue 906, lease_backoff.go):
// after an epoch lost lately, and while its bus does not answer it, in its own process rather than in a
// restart every five seconds.
// made opens the gate again.
// - **A command run at a shell** — `push` in the installer, the lab, a person repairing a mesh whose
// controller is down (issue 201) — acts **under the holder's epoch** when a controller holds the
// lease: it is the same mesh's word, composed and sent under the store's hold of each machine like
@@ -74,9 +72,6 @@ type actor struct {
// lease over in a second rather than in a renewal of five (novox/hq issue 591). A renewal is given
// the lease's own RenewEvery to answer however often it is made, so a slow bus does not lose it.
renewEvery, poll time.Duration
// lossBase, when set, is the backoff after a lost epoch in place of the key's age: a test's (novox/hq
// issue 906).
lossBase time.Duration
}
// theLease is this process's standing to the lease.
@@ -216,73 +211,48 @@ func (a *actor) serveUnderTheLease(ctx context.Context, inv *inventory.Inventory
a.mu.Lock()
a.serving = true
a.mu.Unlock()
say := func(format string, args ...any) { fmt.Printf(format+"\n", args...) }
// **Backing off before standing** (novox/hq issue 906, lease_backoff.go): after an epoch lost lately, before
// this process touches the bus; and while the bus does not answer it, here rather than in a restart.
if wait := a.backoffAfterLosses(ctx, inv, say); wait > 0 {
if err := sleepOrDone(ctx, wait); err != nil {
return nil, err
}
}
var (
js *broker.JetStream
l *lease.Lease
epoch uint64
unwritable error
)
err = standUp(ctx, sleepOrDone, say, func(ctx context.Context) error {
unwritable = nil
if js == nil {
// Its own connection, held as long as the lease, and named so (novox/hq issue 327).
dialled, err := broker.Dial(address, nats.Name(broker.ConnectionName+" lease"))
if err != nil {
return fmt.Errorf("the mesh is on the bus at %s and this control plane cannot reach it to take the "+
"lease: %w", broker.BareAddress(address), err)
}
js = dialled
}
api, err := jetstream.New(js.Conn())
if err != nil {
return err
}
asserting, cancel := context.WithTimeout(ctx, 10*time.Second)
err = broker.EnsureLeaseBucket(asserting, api)
cancel()
if err != nil {
return err
}
l, err = lease.Open(ctx, api, broker.LeaseBucket, lease.Options{Holder: holderOf(instance),
RenewEvery: a.renewEvery, RenewWithin: lease.RenewEvery, Poll: a.poll, Floor: inv.HighestEpoch, Say: say, Health: controllerHealth, Moved: func(was, floor uint64) {
a.mu.Lock()
defer a.mu.Unlock()
a.reset = time.Now()
a.resetSaid = fmt.Sprintf("the lease bucket was at revision %d with epoch %d already issued: it was "+
"raised again from nothing (a bus whose data was replaced), and its revisions were moved past %d so "+
"no machine refuses the next epoch", was, floor, floor)
}})
if err != nil {
return err
}
epoch, err = l.Take(ctx)
if errors.Is(err, lease.ErrUnwritable) {
unwritable = err
return nil
}
// Any other error: whether another controller acts cannot be told, so this one does not act, and tries
// again.
return err
})
// Its own connection, held as long as the lease, and named so (novox/hq issue 327).
js, err := broker.Dial(address, nats.Name(broker.ConnectionName+" lease"))
if err != nil {
if js != nil {
js.Close()
}
return nil, fmt.Errorf("the mesh is on the bus at %s and this control plane cannot reach it to take the "+
"lease: %w", broker.BareAddress(address), err)
}
api, err := jetstream.New(js.Conn())
if err != nil {
js.Close()
return nil, err
}
asserting, cancel := context.WithTimeout(ctx, 10*time.Second)
err = broker.EnsureLeaseBucket(asserting, api)
cancel()
if err != nil {
js.Close()
return nil, err
}
say := func(format string, args ...any) { fmt.Printf(format+"\n", args...) }
l, err := lease.Open(ctx, api, broker.LeaseBucket, lease.Options{Holder: holderOf(instance),
RenewEvery: a.renewEvery, RenewWithin: lease.RenewEvery, Poll: a.poll, Floor: inv.HighestEpoch, Say: say, Health: controllerHealth, Moved: func(was, floor uint64) {
a.mu.Lock()
defer a.mu.Unlock()
a.reset = time.Now()
a.resetSaid = fmt.Sprintf("the lease bucket was at revision %d with epoch %d already issued: it was "+
"raised again from nothing (a bus whose data was replaced), and its revisions were moved past %d so "+
"no machine refuses the next epoch", was, floor, floor)
}})
if err != nil {
js.Close()
return nil, err
}
gone := make(chan struct{})
switch err := unwritable; {
epoch, err := l.Take(ctx)
switch {
case ctx.Err() != nil:
js.Close()
return nil, ctx.Err()
case err != nil && !errors.Is(err, lease.ErrUnwritable):
// Whether another controller acts cannot be told: this one does not act, and exits to try again.
js.Close()
return nil, err
case err != nil:
// Nobody holds it and the bus will not let it be written: unleased, said, tried again (see above).
a.mu.Lock()
+29 -6
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"os"
"regexp"
"slices"
"strings"
"time"
@@ -205,8 +206,9 @@ func (f *sourceForms) buildsFrom(entries []inventory.Entry, repository, seat str
type forgeFacts struct {
// ID is the forge's own id of the repository: what tells it from one deleted and made again by its name.
ID int64
// Guarded is whether the branch is protected as a module's trunk must be: no direct push, at least one
// required status, and no administrator merging past one. Why says what is missing when it is not.
// Guarded is whether the branch is protected as a module's trunk must be: no direct push, a merge whitelist naming
// the tools' account and no agent's account, and no administrator merging past the rule (novox/hq ADR 0534 §6; no
// required status, which the forge no longer asks, §1). Why says what is missing when it is not.
Guarded bool
Why string
}
@@ -361,27 +363,48 @@ func askTheForgeOnce(ctx context.Context, owner, repo, branch string) (forgeFact
return facts, nil
}
// forgeToolsAccount is the forge tools' account, which main's merge whitelist names for mesh-delivery (novox/hq ADR 0398
// §2, the gitea module's tools-account.ts TOOLS_ACCOUNT); beside it the whitelist names only people's own logins, the
// operator's (ADR 0534 §2), never an agent's account.
const forgeToolsAccount = "mesh-tools"
// isAgentAccount is whether a forge login is a node's agent's account, agent-<node> (novox/hq ADR 0398 §1): one that
// may never merge into a module's trunk by itself.
func isAgentAccount(login string) bool { return strings.HasPrefix(login, "agent-") }
// protectionRule is a branch protection rule as the forge's tool summarises it.
type protectionRule struct {
Rule string `json:"rule"`
Push bool `json:"push"`
RequiredStatuses []string `json:"required_statuses"`
AdminMayOverride bool `json:"admin_may_override"`
// MergeWhitelist is who may merge into the branch; nil when the rule keeps no merge whitelist (anyone who writes).
MergeWhitelist []string `json:"merge_whitelist"`
}
// judgedRule judges the one rule the forge applies to a branch, as the forge picks it: the rule named for the
// branch, else the first glob rule, in the forge's order, that covers it (gitea's first matching rule). Never a
// later rule that happens to be stronger: the forge does not read it.
// later rule that happens to be stronger: the forge does not read it. A trunk is trusted for who can land a commit on
// it, not for a status it requires (novox/hq ADR 0534 §6): no push, a merge whitelist naming the tools' account and
// beside it only people's own logins, never an agent's, and no administrator merging past the rule; then a commit lands
// there through mesh-delivery, on a passing mesh/check and the operator's Approve (ADR 0534 §4), or by the operator's
// own hand at the forge (§2).
func judgedRule(rules []protectionRule, branch string) forgeFacts {
switch r := appliedRule(rules, branch); {
case r == nil:
return forgeFacts{Why: fmt.Sprintf("no protection rule covers %s", branch)}
case r.Push:
return forgeFacts{Why: fmt.Sprintf("the rule %s lets a person push to %s directly", r.Rule, branch)}
case len(r.RequiredStatuses) == 0:
return forgeFacts{Why: fmt.Sprintf("the rule %s requires no status before a merge into %s", r.Rule, branch)}
case r.MergeWhitelist == nil || len(r.MergeWhitelist) == 0:
return forgeFacts{Why: fmt.Sprintf("the rule %s keeps no merge whitelist naming %s alone for %s", r.Rule, forgeToolsAccount, branch)}
case !slices.Contains(r.MergeWhitelist, forgeToolsAccount):
return forgeFacts{Why: fmt.Sprintf("the rule %s lets %s merge into %s, and not %s", r.Rule,
strings.Join(r.MergeWhitelist, ", "), branch, forgeToolsAccount)}
case slices.ContainsFunc(r.MergeWhitelist, isAgentAccount):
return forgeFacts{Why: fmt.Sprintf("the rule %s lets an agent's account merge into %s: %s", r.Rule, branch,
strings.Join(r.MergeWhitelist, ", "))}
case r.AdminMayOverride:
return forgeFacts{Why: fmt.Sprintf("the rule %s lets an administrator merge into %s past a status", r.Rule, branch)}
return forgeFacts{Why: fmt.Sprintf("the rule %s lets an administrator merge into %s past it", r.Rule, branch)}
}
return forgeFacts{Guarded: true}
}
+38 -4
View File
@@ -102,16 +102,17 @@ func init() {
// The forge applies one rule to a branch: the rule named for it, else the first glob that covers it. A stronger
// rule later in the list is not what guards the branch, and is not read as if it were (the confirmation review).
func TestTheRuleJudgedIsTheOneTheForgeApplies(t *testing.T) {
guarded := protectionRule{Rule: "*", RequiredStatuses: []string{"mesh/merge-gate"}}
open := protectionRule{Rule: "main", Push: true, RequiredStatuses: []string{"mesh/merge-gate"}}
tools := []string{forgeToolsAccount}
guarded := protectionRule{Rule: "*", MergeWhitelist: tools}
open := protectionRule{Rule: "main", Push: true, MergeWhitelist: tools}
if f := judgedRule([]protectionRule{guarded, open}, "main"); f.Guarded {
t.Error("an exact rule letting pushes was passed over for a glob that guards")
}
if f := judgedRule([]protectionRule{{Rule: "ma*", RequiredStatuses: nil}, {Rule: "*", RequiredStatuses: []string{"x"}}},
if f := judgedRule([]protectionRule{{Rule: "ma*", MergeWhitelist: nil}, {Rule: "*", MergeWhitelist: tools}},
"main"); f.Guarded || !strings.Contains(f.Why, "ma*") {
t.Errorf("the first glob covering the branch was not the one judged: %+v", f)
}
if f := judgedRule([]protectionRule{{Rule: "release/*"}, {Rule: "main", RequiredStatuses: []string{"x"}}}, "main"); !f.Guarded {
if f := judgedRule([]protectionRule{{Rule: "release/*"}, {Rule: "main", MergeWhitelist: tools}}, "main"); !f.Guarded {
t.Errorf("the rule named for the branch was not judged: %+v", f)
}
if f := judgedRule(nil, "main"); f.Guarded {
@@ -119,6 +120,39 @@ func TestTheRuleJudgedIsTheOneTheForgeApplies(t *testing.T) {
}
}
// **A module's trunk is trusted when it takes no push, only the tools' account and people's own logins merge into it,
// and no administrator merges past its rule** (novox/hq ADR 0534 §2, §6): the forge requires no status to merge (§1), so
// a rule requiring none is trusted; one with no merge whitelist, no tools' account in it, an agent's account in it, a
// push allowed or an administrator's override is not.
func TestATrunkIsTrustedForItsMergersNotForAStatus(t *testing.T) {
tools := []string{forgeToolsAccount}
for _, c := range []struct {
name string
rule protectionRule
want bool
why string
}{
{"no status, no push, the tools alone merge", protectionRule{Rule: "main", MergeWhitelist: tools}, true, ""},
{"a status required as well", protectionRule{Rule: "main", MergeWhitelist: tools, RequiredStatuses: []string{"mesh/check"}}, true, ""},
{"no merge whitelist", protectionRule{Rule: "main"}, false, "merge whitelist"},
{"an empty merge whitelist", protectionRule{Rule: "main", MergeWhitelist: []string{}}, false, "merge whitelist"},
{"the operator's own login beside the tools", protectionRule{Rule: "main", MergeWhitelist: []string{forgeToolsAccount, "jschoubben"}}, true, ""},
{"an agent's account in the whitelist", protectionRule{Rule: "main", MergeWhitelist: []string{forgeToolsAccount, "agent-ace"}}, false, "agent-ace"},
{"an agent's account alone", protectionRule{Rule: "main", MergeWhitelist: []string{"agent-ace"}}, false, "agent-ace"},
{"a person's login without the tools", protectionRule{Rule: "main", MergeWhitelist: []string{"jschoubben"}}, false, forgeToolsAccount},
{"a push allowed", protectionRule{Rule: "main", Push: true, MergeWhitelist: tools}, false, "push"},
{"an administrator's override", protectionRule{Rule: "main", MergeWhitelist: tools, AdminMayOverride: true}, false, "administrator"},
} {
f := judgedRule([]protectionRule{c.rule}, "main")
if f.Guarded != c.want || !strings.Contains(f.Why, c.why) {
t.Errorf("%s: %+v, want guarded %v saying %q", c.name, f, c.want, c.why)
}
if strings.Contains(f.Why, "requires no status") {
t.Errorf("%s: a trunk is still judged on a required status: %s", c.name, f.Why)
}
}
}
// An id the forge could not give keeps the id already recorded.
// **The operator's Approve of a reviewed group registers a moved module from its new repository** (novox/hq ADR 0420 §7,
-117
View File
@@ -1,117 +0,0 @@
package main
import (
"context"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// Backing off before standing (novox/hq issue 906).
//
// A controller loses its lease when its bus stops answering for longer than the key's age: on the control-node,
// a bus whose disks are saturated has taken 30 to 42 seconds to answer (2026-10-23 05:52). The process then
// exits, as ADR 0229 rule 4 says, and its service manager starts it again five seconds later. **A start is the
// heaviest thing a controller asks of the bus** — every bucket, stream and consumer asserted again — and a start
// that cannot assert the lease's bucket used to exit at once and be started again five seconds later, on the
// same overloaded bus, nine times in twelve minutes (2026-10-16). So a candidate backs off twice:
//
// - **After a loss**, before it touches the bus: from the newest epoch lost in the last ten minutes (the
// controller's store says it, rule 7), it waits one key's age, doubled for every further loss in that
// window, at most two minutes. A single loss costs a key's age; a controller losing it again and again
// stands less and less often.
// - **While the bus does not answer it**, in its own process: asserting the lease's bucket, opening it or
// reading who holds it is tried again after 5 s, doubling to at most 60 s, instead of exiting into a restart
// every five seconds. Nothing acts meanwhile: the process holds no lease, and the gate refuses every act.
const (
// lossWindow is how far back lost epochs count towards the backoff.
lossWindow = 10 * time.Minute
// lossMost is the longest a candidate waits after losses.
lossMost = 2 * time.Minute
// standFirst and standMost are the waits between tries of a candidate the bus does not answer.
standFirst = 5 * time.Second
standMost = time.Minute
)
// lossBackoff is how long a candidate waits now before it stands, after the epochs it read: one key's age from
// the newest loss in the window, doubled for every further loss in it, at most lossMost.
func lossBackoff(epochs []inventory.Epoch, now time.Time) time.Duration {
return lossBackoffFrom(epochs, now, broker.LeaseTTL)
}
func lossBackoffFrom(epochs []inventory.Epoch, now time.Time, base time.Duration) time.Duration {
lost, newest := lostSince(epochs, now)
if lost == 0 {
return 0
}
delay := base
for i := 1; i < lost && delay < lossMost; i++ {
delay *= 2
}
delay = min(delay, lossMost)
return max(newest.Add(delay).Sub(now), 0)
}
// standUp runs try until it answers nil, waiting standFirst between tries and doubling to standMost; it answers
// ctx's error once ctx ends.
func standUp(ctx context.Context, sleep func(context.Context, time.Duration) error, say func(string, ...any),
try func(context.Context) error) error {
wait := standFirst
for {
err := try(ctx)
if err == nil {
return nil
}
if ctx.Err() != nil {
return ctx.Err()
}
say("this controller cannot stand for the lease yet: %v — it tries again in %s, in this process, rather "+
"than start again on a bus that does not answer (novox/hq issue 906)", err, wait)
if err := sleep(ctx, wait); err != nil {
return err
}
wait = min(wait*2, standMost)
}
}
// backoffAfterLosses is how long this candidate waits before it stands, read from the epochs its store kept,
// and said. A store that cannot be read makes it wait nothing: the backoff spares the bus, and never keeps a
// controller from standing.
func (a *actor) backoffAfterLosses(ctx context.Context, inv *inventory.Inventory, say func(string, ...any)) time.Duration {
now := time.Now()
reading, cancel := context.WithTimeout(ctx, 10*time.Second)
epochs, err := inv.EpochsSince(reading, now.Add(-lossWindow))
cancel()
if err != nil {
say("whether the lease was lost lately cannot be read, so this controller stands without backing off: %v", err)
return 0
}
base := a.lossBase
if base <= 0 {
base = broker.LeaseTTL
}
wait := lossBackoffFrom(epochs, now, base)
if wait > 0 {
lost, _ := lostSince(epochs, now)
say("the controller lease was lost lately (%d time(s) in the last %s): this controller backs off %s "+
"before it stands, so its start does not add to the load that lost it (novox/hq issue 906)",
lost, lossWindow, wait.Round(time.Second))
}
return wait
}
// lostSince is how many epochs were lost in the window before now, and when the newest of them was.
func lostSince(epochs []inventory.Epoch, now time.Time) (lost int, newest time.Time) {
for _, e := range epochs {
if e.Ended == nil || e.How != inventory.EpochLost || now.Sub(*e.Ended) > lossWindow {
continue
}
lost++
if e.Ended.After(newest) {
newest = *e.Ended
}
}
return lost, newest
}
-97
View File
@@ -1,97 +0,0 @@
package main
import (
"context"
"errors"
"slices"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A controller that lost its lease to a slow bus backs off before it stands again (novox/hq issue 906): it
// does not add its start — every bucket and consumer asserted again — to the load that lost it the lease.
func lostAt(epoch uint64, ended time.Time, how string) inventory.Epoch {
return inventory.Epoch{Epoch: epoch, Instance: "controller@anchor", Taken: ended.Add(-time.Hour), Ended: &ended,
How: how}
}
// 906/AC1: given epochs lost in the last ten minutes, when a candidate starts, then it waits from the newest
// loss one key's age, doubled for each further loss, at most two minutes; nothing else makes it wait.
func TestACandidateAfterALostLeaseBacksOff(t *testing.T) {
now := time.Date(2026, 10, 23, 5, 52, 40, 0, time.UTC)
for _, c := range []struct {
name string
epochs []inventory.Epoch
want time.Duration
}{
{"nothing lost", nil, 0},
{"one held, one given back", []inventory.Epoch{
{Epoch: 3, Taken: now.Add(-time.Minute)},
lostAt(2, now.Add(-2*time.Second), inventory.EpochReleased),
}, 0},
{"one lost five seconds ago", []inventory.Epoch{lostAt(7, now.Add(-5*time.Second), inventory.EpochLost)},
10 * time.Second},
{"one lost long ago enough", []inventory.Epoch{lostAt(7, now.Add(-20*time.Second), inventory.EpochLost)}, 0},
{"three lost, the newest a second ago", []inventory.Epoch{
lostAt(9, now.Add(-time.Second), inventory.EpochLost),
lostAt(8, now.Add(-3*time.Minute), inventory.EpochLost),
lostAt(7, now.Add(-6*time.Minute), inventory.EpochLost),
}, 59 * time.Second},
{"six lost: at most two minutes", []inventory.Epoch{
lostAt(12, now, inventory.EpochLost), lostAt(11, now.Add(-time.Minute), inventory.EpochLost),
lostAt(10, now.Add(-2*time.Minute), inventory.EpochLost), lostAt(9, now.Add(-3*time.Minute), inventory.EpochLost),
lostAt(8, now.Add(-4*time.Minute), inventory.EpochLost), lostAt(7, now.Add(-5*time.Minute), inventory.EpochLost),
}, 2 * time.Minute},
{"a loss older than ten minutes does not count", []inventory.Epoch{
lostAt(9, now.Add(-time.Second), inventory.EpochLost),
lostAt(8, now.Add(-11*time.Minute), inventory.EpochLost),
}, 14 * time.Second},
{"expired is not lost", []inventory.Epoch{lostAt(9, now.Add(-time.Second), inventory.EpochExpired)}, 0},
} {
t.Run(c.name, func(t *testing.T) {
if got := lossBackoff(c.epochs, now); got != c.want {
t.Fatalf("a candidate after %d epoch(s) waits %s, want %s", len(c.epochs), got, c.want)
}
})
}
}
// 906/AC2: given a bus that does not answer a candidate, when it asserts the lease's bucket or reads who holds
// it, then the candidate tries again in its own process after 5 s, doubling to at most 60 s, instead of exiting
// to be restarted by its service manager every five seconds.
func TestACandidateTheBusDoesNotAnswerTriesAgainWithBackoff(t *testing.T) {
var waited []time.Duration
sleep := func(ctx context.Context, d time.Duration) error { waited = append(waited, d); return nil }
tries := 0
slow := errors.New("asserting bucket mesh-controller_lease: context deadline exceeded")
err := standUp(t.Context(), sleep, func(string, ...any) {}, func(context.Context) error {
tries++
if tries < 7 {
return slow
}
return nil
})
if err != nil {
t.Fatalf("a candidate whose bus answered at the seventh try gave up: %v", err)
}
want := []time.Duration{5 * time.Second, 10 * time.Second, 20 * time.Second, 40 * time.Second, time.Minute,
time.Minute}
if tries != 7 || !slices.Equal(waited, want) {
t.Fatalf("a candidate tried %d time(s), waiting %v between; want 7, waiting %v", tries, waited, want)
}
}
// 906/AC2: and a candidate stopping while it waits stops, with why.
func TestACandidateStoppingWhileItBacksOffStops(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
cancel()
err := standUp(ctx, sleepOrDone, func(string, ...any) {}, func(context.Context) error {
return errors.New("who holds the lease cannot be read: context deadline exceeded")
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("a candidate stopped while backing off answered %v", err)
}
}
@@ -152,33 +152,3 @@ func TestTwoControllersOneActs(t *testing.T) {
t.Fatalf("B's epoch does not say it was lost: %+v", ended)
}
}
// 906/AC1, on a real bus and store: a controller starting right after an epoch was lost backs off before it
// stands — it takes the lease no sooner than the backoff after the loss, and then holds it as usual.
func TestAControllerAfterALostLeaseBacksOffBeforeItStands(t *testing.T) {
url, _ := aBusForTheLease(t)
inv := inventory.ForTest(t)
ctx := t.Context()
// The controller before it lost epoch 5, and said so in the store, as `took` does on a loss.
if _, err := inv.TookEpoch(ctx, inventory.Epoch{Epoch: 5, Instance: "controller@anchor pid 1", Host: "anchor",
Taken: time.Now()}); err != nil {
t.Fatal(err)
}
if err := inv.EndEpoch(ctx, 5, inventory.EpochLost); err != nil {
t.Fatal(err)
}
// The key's age stands in as a backoff of two seconds, so the case runs in seconds (issue 591).
const backoff = 2 * time.Second
a := &actor{renewEvery: lease.RenewEvery / 5, poll: lease.Poll / 10, lossBase: backoff}
start := time.Now()
if _, err := a.serveUnderTheLease(ctx, inv, url); err != nil {
t.Fatal(err)
}
defer a.release()
if waited := time.Since(start); waited < backoff-300*time.Millisecond {
t.Fatalf("a controller took the lease %s after an epoch was lost, without backing off %s", waited, backoff)
}
if epoch, err := a.epoch(ctx); err != nil || epoch <= 5 {
t.Fatalf("after backing off the controller holds epoch %d (%v), want one above the lost 5", epoch, err)
}
}
+71
View File
@@ -218,6 +218,15 @@ func retryRefusal(p inventory.Plan, plans []inventory.Plan, onPath map[string]st
}
return oneWalkAtATime(p, plans, onPath)
}
// **A walk lost to silence is resumed** (novox/hq ADR 0531, issue 1886): its tier was sent whole and only the gate after it failed it, every
// machine it waited on silent; once they reported, it goes on from that gate.
if len(stopped) == 0 && lostToSilence(p) {
if q, found := newerOpenPlan(p, plans); found {
return fmt.Errorf("%s supersedes it: a newer merge of %s (%s at %s) is open, and resuming %s would send "+
"what that one replaced", q.ID, q.Repository, q.ID, short(q.Commit), p.ID)
}
return oneWalkAtATime(p, plans, onPath)
}
if len(stopped) == 0 {
return fmt.Errorf("nothing in tier %d of %s failed to build or stopped rolling out — it stopped at: %s",
p.Tier, p.ID, p.Note)
@@ -412,6 +421,20 @@ func retryPlan(ctx context.Context, open *stores, id string) (string, error) {
}
return said, nil
}
if len(failed) == 0 && lostToSilence(p) {
reports, err := inv.LastReports(ctx)
if err != nil {
return "", err
}
said, err := resumeLost(&p, silentNodes(p, reports, time.Now().UTC(), provLookup(ctx)))
if err != nil {
return "", err
}
if err := inv.SavePlan(ctx, &p); err != nil {
return "", err
}
return said, nil
}
if len(failed) == 0 {
return retryRollouts(ctx, open, &p)
}
@@ -693,3 +716,51 @@ func resumeStopped(p *inventory.Plan) string {
return fmt.Sprintf("%s resumed at tier %d of %d, stopped by %s: %s; this verb asks nothing — the next look carries it "+
"on as any walk", p.ID, p.Tier+1, len(p.Tiers), by, waits)
}
// silentGateWords say, in a plan's note, that the gate between two tiers found every machine it waits on silent past
// ApplySilentAfter, and failed the plan as lost (issue 1160, ADR 0531): one text for the writer and the reader (lostToSilence).
const silentGateWords = "every machine it waits on said nothing on its send within"
// lostToSilence is whether a plan failed only at the gate after its tier, every machine it waited on silent (ADR 0531,
// issue 1886): nothing in its tier failed to build, stopped rolling out or is still to ask — the look had sent the whole tier.
func lostToSilence(p inventory.Plan) bool {
if p.State != inventory.PlanFailed || p.Tier >= len(p.Tiers) || !strings.Contains(p.Note, silentGateWords) ||
len(failedIn(p)) > 0 || len(stoppedRollouts(p)) > 0 {
return false
}
asked, _, unasked := waitingIn(p)
return asked == 0 && unasked == 0
}
// resumeLost sets a walk lost to silence building again at the tier it failed after (ADR 0531, issue 1886), for the next look to
// judge its gate again on the machines' reports, unless every machine its tier was sent to is still silent: then
// nothing it needs is delivered anywhere yet, and the retry is refused as the gate would fail it again.
func resumeLost(p *inventory.Plan, silent map[string]bool) (string, error) {
var sent, still []string
for _, m := range p.Tiers[p.Tier] {
if s := p.Modules[m]; s != nil {
for n := range s.Rest {
if !slices.Contains(sent, n) {
sent = append(sent, n)
}
}
}
}
for _, n := range sent {
if silent[n] {
still = append(still, n)
}
}
sort.Strings(still)
if len(sent) > 0 && len(still) == len(sent) {
return "", fmt.Errorf("%s failed because every machine its tier %d waits on said nothing on its send, and %s "+
"still said nothing: retry once it reported", p.ID, p.Tier+1, strings.Join(still, ", "))
}
was := p.Note
p.State = inventory.PlanBuilding
goesOnAgain(p)
p.Note = fmt.Sprintf("tier %d resumed by hand after it failed as lost (%s): its gate is judged again on the "+
"machines' reports", p.Tier, was)
return fmt.Sprintf("%s resumed at tier %d of %d, failed as lost to silence (%s); this verb asks and sends nothing — "+
"the next look judges its gate again and carries it on as any walk", p.ID, p.Tier+1, len(p.Tiers), was), nil
}
+43 -32
View File
@@ -260,7 +260,7 @@ type gateNeed struct {
}
// silentNodes is the machines a walk sent and heard nothing from past ApplySilentAfter, on whichever tier it sent
// them (novox/hq ADR 0360 §5, issue 1160): the one rule both the gates between tiers and the walk's end read, through
// them (novox/hq ADR 0360 §7, issue 1160): the one rule both the gates between tiers and the walk's end read, through
// walkReports. A machine that reports on its send is not silent from then on.
func silentNodes(p inventory.Plan, reports []inventory.Reported, now time.Time, prov inventory.ProvLookup) map[string]bool {
_, _, silent := walkReports(p, reports, now, prov)
@@ -272,7 +272,7 @@ func silentNodes(p inventory.Plan, reports []inventory.Reported, now time.Time,
}
// withoutSilent is the machines of running that are not silent: a later tier is not sent to a machine the walk heard
// nothing from, which keeps its want of the plan's builds for the send it gets when it is heard again (ADR 0360 §5).
// nothing from, which keeps its want of the plan's builds for the send it gets when it is heard again (ADR 0360 §7).
func withoutSilent(running []string, silent map[string]bool) []string {
return slices.DeleteFunc(slices.Clone(running), func(n string) bool { return silent[n] })
}
@@ -1129,7 +1129,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
// being behind. Now every module of the tier whose first machine is the same is sent there in one
// send, judged there by one gate — kept on the first of them, the others gated by it — and the rest
// of the tier's machines are sent once, together, when the gates have passed.
// **A machine silent past the bound is left as it was, on every tier** (novox/hq ADR 0360 §5, issue 1160): the
// **A machine silent past the bound is left as it was, on every tier** (novox/hq ADR 0360 §7, issue 1160): the
// walk's end names it, and the gates between tiers and the sends of the tiers after do not wait for it or go to it.
allReports, err := inv.LastReports(ctx)
if err != nil {
@@ -1457,8 +1457,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
// Every machine the next tier waits on is silent: nothing it needs was delivered anywhere, so the plan does
// not go on as if it had been (issue 1160).
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("tier %d built; every machine it waits on said nothing on its send within %s: %s",
p.Tier, applySilentAfter, strings.Join(quiet, ", "))
p.Note = fmt.Sprintf("tier %d built; %s %s: %s", p.Tier, silentGateWords, applySilentAfter, strings.Join(quiet, ", "))
fmt.Printf("%s: %s\n", p.ID, p.Note)
return true, nil
}
@@ -1476,7 +1475,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
return changed, nil
}
}
// **A walk ends on its machines' reports** (novox/hq ADR 0360 §4, §5): after its last tier, every machine it sent
// **A walk ends on its machines' reports** (novox/hq ADR 0360 §4, §7): after its last tier, every machine it sent
// has reported on that send, or is silent past ApplySilentAfter; never a health reading.
end := ""
if p.Tier == len(p.Tiers)-1 {
@@ -1543,50 +1542,62 @@ func walkReports(p inventory.Plan, reports []inventory.Reported, now time.Time,
carried[node] = append(carried[node], m)
}
}
seen := map[string]bool{}
// **Each machine on the walk's last send to it** (novox/hq ADR 0531, issue 1886): a send carries the machine's whole declaration (ADR
// 0221), so a report on the last one answers every earlier one, and the bound runs from it. Judged on every send
// it had made, a walk whose tiers were sent hours apart — R4e, stopped at tier 1 and resumed — held tier 0's send of
// 00:23 against tier 1's templates and named its machines silent 35 seconds after tier 1 was sent.
last := map[string]inventory.SentDeclaration{}
lastAt := map[string]time.Time{}
for _, s := range p.Modules {
if s == nil {
continue
}
for node, sent := range s.Rest {
sent = sent.Only(carried[node]...)
if seen[node] {
continue
}
r := byNode[node]
at := sent.At
if at.IsZero() && s.SentAt != nil {
at = *s.SentAt
}
// A module handed to the provisioner is applied only once its client said so too (ADR 0362 §1-5).
outcome, reported := provisionedOutcome(node, sent, r, r.At != nil && sent.ReportsOn(r), prov)
switch {
case reported:
seen[node] = true
switch {
case outcome == inventory.OutcomeFailed || outcome == inventory.OutcomeRefused:
failed = append(failed, node)
case strings.HasPrefix(outcome, inventory.OutcomeProvisioningFailed):
failed = append(failed, node+" ("+outcome+")")
}
case !at.IsZero() && now.Sub(at) > applySilentAfter:
seen[node] = true
silent = append(silent, node)
default:
if !slices.Contains(waiting, node) {
waiting = append(waiting, node)
}
was, ok := last[node]
if !ok || laterSend(sent, at, was, lastAt[node]) {
last[node], lastAt[node] = sent, at
}
}
}
// A machine one module's send was reported on and another's not yet is the same send: reported.
waiting = slices.DeleteFunc(waiting, func(n string) bool { return seen[n] })
for node, sent := range last {
at := lastAt[node]
sent = sent.Only(carried[node]...)
r := byNode[node]
// A module handed to the provisioner is applied only once its client said so too (ADR 0362 §1-5).
outcome, reported := provisionedOutcome(node, sent, r, r.At != nil && sent.ReportsOn(r), prov)
switch {
case reported:
switch {
case outcome == inventory.OutcomeFailed || outcome == inventory.OutcomeRefused:
failed = append(failed, node)
case strings.HasPrefix(outcome, inventory.OutcomeProvisioningFailed):
failed = append(failed, node+" ("+outcome+")")
}
case !at.IsZero() && now.Sub(at) > applySilentAfter:
silent = append(silent, node)
default:
waiting = append(waiting, node)
}
}
sort.Strings(waiting)
sort.Strings(failed)
sort.Strings(silent)
return waiting, failed, silent
}
// laterSend is whether a, sent at aAt, is a later send to a machine than b, sent at bAt: by its sequence when both carry
// one, else by when it went.
func laterSend(a inventory.SentDeclaration, aAt time.Time, b inventory.SentDeclaration, bAt time.Time) bool {
if a.Sequence > 0 && b.Sequence > 0 && a.Sequence != b.Sequence {
return a.Sequence > b.Sequence
}
return aAt.After(bAt)
}
// restOf is, for one module, every machine of its rest that the send reached and what it carried there.
func restOf(to, sent []string, what map[string]inventory.SentDeclaration) map[string]inventory.SentDeclaration {
out := map[string]inventory.SentDeclaration{}
@@ -0,0 +1,173 @@
package main
import (
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// Issue 1886, the resume of R4e (plan-1792707174627079192): tier 0 was sent at 00:23:28, the walk was stopped at
// 02:01 and resumed at 07:43, and tier 1 was sent at 07:49:54. 35 seconds later the gate named ace silent "within
// 15m0s", and at 07:51:30 shanks, in the same second shanks reported it applied: the plan failed as lost. The walk
// judged each machine on every send it had made it, tier 0's of 00:23 too, narrowed to every module the walk carried
// there — so tier 0's send was held against the templates of tier 1's modules it never carried, and silence was
// measured from 00:23. A machine is judged on the walk's last send to it: that send carries its whole declaration.
func r4eResumed() (inventory.Plan, []inventory.Reported, inventory.ProvLookup, time.Time) {
tier0 := time.Date(2026, 10, 22, 22, 23, 28, 0, time.UTC)
tier1 := time.Date(2026, 10, 23, 5, 49, 54, 0, time.UTC)
now := time.Date(2026, 10, 23, 5, 51, 30, 0, time.UTC)
t0, t1 := tier0, tier1
p := inventory.Plan{ID: "plan-1792707174627079192", State: inventory.PlanRolling, Tier: 1,
Tiers: [][]string{{"pacman"}, {"mesh-build-image"}, {"gitea"}},
Modules: map[string]*inventory.PlanModule{
"pacman": {State: "built", SentAt: &t0, Rest: map[string]inventory.SentDeclaration{
"shanks": {Digest: "d0-shanks", Sequence: 900, At: tier0,
Provisioned: map[string]string{"pacman": "pac-1", "mesh-build-image": "img-old"}},
"ace": {Digest: "d0-ace", Sequence: 1100, At: tier0,
Provisioned: map[string]string{"pacman": "pac-1", "mesh-build-image": "img-old"}}}},
"mesh-build-image": {State: "built", SentAt: &t1, Rest: map[string]inventory.SentDeclaration{
"shanks": {Digest: "d1-shanks", Sequence: 971, At: tier1,
Provisioned: map[string]string{"pacman": "pac-1", "mesh-build-image": "img-new"}},
"ace": {Digest: "d1-ace", Sequence: 1212, At: tier1,
Provisioned: map[string]string{"pacman": "pac-1", "mesh-build-image": "img-new"}}}},
"gitea": {}}}
shanksAt := now
aceAt := time.Date(2026, 10, 23, 0, 57, 50, 0, time.UTC)
reports := []inventory.Reported{
// shanks reported on tier 1's send in the second the gate looked.
{Node: "shanks", Outcome: inventory.OutcomeApplied, At: &shanksAt, Declared: "d1-shanks", Current: true,
ReportedSequence: 971},
// ace's last report is from before tier 1's send, on a declaration sequenced after tier 0's.
{Node: "ace", Outcome: inventory.OutcomeApplied, At: &aceAt, Declared: "d-ace-0257", ReportedSequence: 1150},
}
says := map[string]inventory.ProvReport{
"shanks/pacman": {Status: inventory.ProvApplied, Templates: "pac-1", At: now},
"shanks/mesh-build-image": {Status: inventory.ProvApplied, Templates: "img-new", At: now},
"ace/pacman": {Status: inventory.ProvApplied, Templates: "pac-1", At: aceAt},
// ace's client had not placed the old image's files (issue 1884), and has not yet the new one's.
"ace/mesh-build-image": {Status: inventory.ProvWaiting, Templates: "img-old", At: aceAt},
}
prov := func(node, module string) (inventory.ProvReport, bool) { r, ok := says[node+"/"+module]; return r, ok }
return p, reports, prov, now
}
func TestAResumedWalkJudgesEachMachineOnItsLastSend(t *testing.T) {
p, reports, prov, now := r4eResumed()
waiting, failed, silent := walkReports(p, reports, now, prov)
if len(silent) != 0 {
t.Fatalf("1886: %v named silent within %s of tier 1's send, from tier 0's send of 00:23", silent, applySilentAfter)
}
if len(failed) != 0 || !slices.Equal(waiting, []string{"ace"}) {
t.Fatalf("1886: waiting %v, failed %v; want ace waited for (sent 96 s ago) and shanks reported", waiting, failed)
}
// The gate the plan failed at: mesh-build-image runs on ace and shanks, which tier 2 needs.
needs := []gateNeed{{module: "mesh-build-image", running: []string{"ace", "shanks"},
since: func(string) time.Time { return *p.Modules["mesh-build-image"].SentAt }}}
w, quiet, lost := settleTierGate(needs, reports, silentNodes(p, reports, now, prov))
if lost || len(quiet) != 0 || !slices.Equal(w, []string{"mesh-build-image on ace"}) {
t.Fatalf("1886: the gate says waiting %v, silent %v, lost %v; want mesh-build-image on ace waited for", w, quiet, lost)
}
}
// The machine silent on the walk's last send to it is still silent once the bound has passed since that send.
func TestAMachineSilentOnItsLastSendIsStillSilent(t *testing.T) {
p, reports, prov, now := r4eResumed()
later := now.Add(applySilentAfter + time.Minute)
_, _, silent := walkReports(p, reports, later, prov)
if !slices.Equal(silent, []string{"ace"}) {
t.Fatalf("1886: silent %v past the bound from tier 1's send; want ace", silent)
}
}
// A walk failed only because every machine its gate waited on was silent, whose tiers are all sent, is carried on by
// `plans retry` once those machines reported (issue 1886): it failed on no build and no rollout, so the retry of failed
// builds and of stopped rollouts refused it with "nothing in tier 1 … failed to build or stopped rolling out".
func r4eLost() inventory.Plan {
p, _, _, _ := r4eResumed()
p.State = inventory.PlanFailed
p.Note = "tier 1 built; " + silentGateWords + " 15m0s: shanks"
// mesh-tools is built and recorded, never sent, as R4e's was.
p.Tiers[1] = append(p.Tiers[1], "mesh-tools")
p.Modules["mesh-tools"] = &inventory.PlanModule{State: "built"}
return p
}
func TestRetryCarriesOnAWalkLostToSilenceOnceItsMachinesReported(t *testing.T) {
p, reports, prov, now := r4eResumed()
p = r4eLost()
if !lostToSilence(p) {
t.Fatalf("1886: the plan R4e failed as is not read as lost to silence: %q", p.Note)
}
if err := retryRefusal(p, []inventory.Plan{p}, ownPathOf(nil)); err != nil {
t.Fatalf("1886: a walk lost to silence was refused a retry: %v", err)
}
reports[1].Current, reports[1].Declared, reports[1].ReportedSequence = true, "d1-ace", 1212
reports[1].At = &now
says := map[string]inventory.ProvReport{}
for _, n := range []string{"ace", "shanks"} {
says[n+"/pacman"] = inventory.ProvReport{Status: inventory.ProvApplied, Templates: "pac-1", At: now}
says[n+"/mesh-build-image"] = inventory.ProvReport{Status: inventory.ProvApplied, Templates: "img-new", At: now}
}
prov = func(node, module string) (inventory.ProvReport, bool) { r, ok := says[node+"/"+module]; return r, ok }
said, err := resumeLost(&p, silentNodes(p, reports, now.Add(time.Hour), prov))
if err != nil {
t.Fatalf("1886: the resume of a walk whose machines reported refused: %v", err)
}
if p.State != inventory.PlanBuilding || p.Tier != 1 {
t.Fatalf("1886: resumed walk is %s at tier %d; want building at tier 1, for the next look to pass its gate", p.State, p.Tier)
}
if !strings.Contains(p.Note, "resumed by hand") || !strings.Contains(said, "resumed at tier 2 of 3") {
t.Fatalf("1886: note %q, said %q", p.Note, said)
}
}
// Every machine its tier was sent to still silent: the gate would fail it again, so the retry says so instead.
func TestRetryRefusesAWalkLostToSilenceWhileEveryMachineIsStillSilent(t *testing.T) {
p := r4eLost()
if _, err := resumeLost(&p, map[string]bool{"ace": true, "shanks": true}); err == nil || !strings.Contains(err.Error(), "ace, shanks") {
t.Fatalf("1886: a walk whose machine is still silent was resumed: %v", err)
}
if p.State != inventory.PlanFailed {
t.Fatalf("1886: a refused resume changed the plan to %s", p.State)
}
}
// A walk lost to silence with something of its tier never sent is not this: its send is the look's to make.
func TestALostWalkWithATierUnsentIsNotLostToSilence(t *testing.T) {
p := r4eLost()
p.Modules["mesh-build-image"].State = "asked"
if lostToSilence(p) {
t.Fatal("1886: a tier with a module still asked was read as lost to silence")
}
p = r4eLost()
p.Note = "tier 1 built; mesh-build-image failed to build"
if lostToSilence(p) {
t.Fatal("1886: a plan failed by a build was read as lost to silence")
}
}
// A walk lost to silence is refused its resume beside a newer open plan of its repository and branch, which supersedes
// it, and beside a related open plan: one plan at a time (ADR 0531).
func TestRetryRefusesAWalkLostToSilenceBesideANewerOrARelatedOpenPlan(t *testing.T) {
p := r4eLost()
created := time.Date(2026, 10, 22, 22, 15, 9, 0, time.UTC)
p.Repository, p.Branch, p.Commit, p.Created = "novox/mesh-sdk", "main", "7aacbc16", created
newerOpen := inventory.Plan{ID: "plan-newer", Repository: "novox/mesh-sdk", Branch: "main", Commit: "d00dfeed",
Created: created.Add(time.Hour), State: inventory.PlanBuilding, Tiers: [][]string{{"zsh"}},
Modules: map[string]*inventory.PlanModule{"zsh": {State: "asked"}}}
if err := retryRefusal(p, []inventory.Plan{p, newerOpen}, ownPathOf(nil)); err == nil ||
!strings.Contains(err.Error(), "plan-newer supersedes it") {
t.Errorf("1886: a walk lost to silence was resumed under a newer open plan of its branch: %v", err)
}
beside := inventory.Plan{ID: "plan-beside", Repository: "novox/mesh-sdk", Branch: "release", Commit: "bbadbb2a",
Created: created.Add(time.Hour), State: inventory.PlanBuilding, Tiers: [][]string{{"mesh-build-image"}},
Modules: map[string]*inventory.PlanModule{"mesh-build-image": {State: "asked"}}}
if err := retryRefusal(p, []inventory.Plan{p, beside}, ownPathOf(nil)); err == nil ||
!strings.Contains(err.Error(), "one walk at a time") {
t.Errorf("1886: a walk lost to silence was resumed beside a related open plan: %v", err)
}
}
+1 -2
View File
@@ -37,8 +37,7 @@ func ForTest(t *testing.T) *Identity {
}
store.CreateMigratedForTest(t, admin, name, migrations)
cut := strings.LastIndex(admin, "/")
t.Setenv(store.Variable(Name), admin[:cut]+"/"+name+"?sslmode=disable")
t.Setenv(store.Variable(Name), store.DatabaseForTest(admin, name))
ident, err := Open(t.Context())
if err != nil {
+4 -46
View File
@@ -7,61 +7,19 @@ import (
"context"
"crypto/ed25519"
"errors"
"fmt"
"os"
"strings"
"sync"
"testing"
"time"
"github.com/jackc/pgx/v5"
"github.com/novox/mesh-controller/internal/store"
)
// fresh is the package's own name for ForTest: one fixture for every store test of identity, so the way
// a test's database is made and addressed is kept true in one place (novox/hq issue 1889: this copy
// dropped the store address's query and reached no server on the checker's socket-addressed store).
func fresh(t *testing.T) *Identity {
t.Helper()
admin := os.Getenv("MESH_TEST_POSTGRES")
if admin == "" {
t.Skip("no MESH_TEST_POSTGRES; run `make check` to raise one")
}
name := fmt.Sprintf("ident_%d", time.Now().UnixNano()%10_000_000)
conn, err := pgx.Connect(t.Context(), admin)
if err != nil {
t.Fatalf("cannot reach the test PostgreSQL: %v", err)
}
if _, err := conn.Exec(t.Context(), "create database "+name); err != nil {
t.Fatalf("cannot create %s: %v", name, err)
}
conn.Close(t.Context())
cut := strings.LastIndex(admin, "/")
t.Setenv(store.Variable(Name), admin[:cut]+"/"+name+"?sslmode=disable")
ident, err := Open(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
ident.Close()
c, err := pgx.Connect(context.Background(), admin)
if err != nil {
return
}
defer c.Close(context.Background())
_, _ = c.Exec(context.Background(), "drop database if exists "+name+" with (force)")
})
if err := ident.Ready(t.Context(), 20*time.Second); err != nil {
t.Fatal(err)
}
migrations, err := Migrations()
if err != nil {
t.Fatal(err)
}
if _, err := ident.store.Migrate(t.Context(), migrations); err != nil {
t.Fatal(err)
}
return ident
return ForTest(t)
}
func TestNoKeyIsAnErrorRatherThanAnEmptyKey(t *testing.T) {
+6
View File
@@ -675,6 +675,12 @@ func tierLeftOf(old *Plan, p Plan, now time.Time) (time.Time, *Duration) {
since = old.Created
}
wasOpen := old.State == PlanBuilding || old.State == PlanRolling
if !wasOpen && p.Open() {
// **A walk set open again enters its tier anew** (issue 1886): a resume or a retry at the tier it ended at. Kept,
// the entry of before its stop measured the tier from hours before the resume — R4e read "waiting … for 5h48m53s"
// seven minutes after it was resumed — and the watchdog of its progress (S3) found it late at once.
return now, nil
}
if !wasOpen || (old.Tier == p.Tier && p.Open()) {
return since, nil // still in the tier, or already ended
}
+2 -4
View File
@@ -42,8 +42,7 @@ func ForTest(t *testing.T) *Inventory {
store.CreateMigratedForTest(t, admin, name, migrations)
busPasswordCostForTest.Store(int32(bcrypt.MinCost))
cut := strings.LastIndex(admin, "/")
t.Setenv(store.Variable(Name), admin[:cut]+"/"+name+"?sslmode=disable")
t.Setenv(store.Variable(Name), store.DatabaseForTest(admin, name))
inv, err := Open(t.Context())
if err != nil {
@@ -113,8 +112,7 @@ func ForTestBefore(t *testing.T, number int) (*Inventory, func()) {
}
store.CreateMigratedForTest(t, admin, name, before)
busPasswordCostForTest.Store(int32(bcrypt.MinCost))
cut := strings.LastIndex(admin, "/")
t.Setenv(store.Variable(Name), admin[:cut]+"/"+name+"?sslmode=disable")
t.Setenv(store.Variable(Name), store.DatabaseForTest(admin, name))
inv, err := Open(t.Context())
if err != nil {
t.Fatal(err)
@@ -0,0 +1,32 @@
package inventory
import (
"testing"
"time"
)
// Issue 1886: R4e entered tier 1 at 02:01, was stopped, and was resumed at 07:43 at the same tier. Its save kept the
// tier's entry of 02:01, so at 07:50 the plan read "waiting for mesh-build-image on shanks to be applied for 5h48m53s"
// and its progress watchdog measured the tier from before the stop. A walk set open again at the tier it ended at
// enters that tier anew: its time in the tier runs from the resume.
func TestAWalkResumedAtItsTierEntersItAnew(t *testing.T) {
entered := time.Date(2026, 10, 23, 0, 1, 7, 0, time.UTC)
resumed := time.Date(2026, 10, 23, 5, 43, 30, 0, time.UTC)
old := &Plan{ID: "plan-r4e", State: PlanFailed, Tier: 1, Tiers: [][]string{{"a"}, {"b"}, {"c"}},
Created: entered.Add(-2 * time.Hour), TierEntered: entered}
p := *old
p.State = PlanBuilding
got, d := tierLeftOf(old, p, resumed)
if !got.Equal(resumed) || d != nil {
t.Fatalf("1886: a walk resumed at tier 1 entered it at %s (duration %v); want %s, no tier left", got, d, resumed)
}
// A walk still in its tier keeps its entry; one that ended keeps it too.
open := *old
open.State = PlanRolling
if got, _ := tierLeftOf(&open, open, resumed); !got.Equal(entered) {
t.Fatalf("an open walk still in its tier was stamped anew: %s", got)
}
if got, _ := tierLeftOf(old, *old, resumed); !got.Equal(entered) {
t.Fatalf("a failed walk saved again was stamped anew: %s", got)
}
}
+1 -2
View File
@@ -40,8 +40,7 @@ func ForTest(t *testing.T) *Licences {
}
store.CreateMigratedForTest(t, admin, name, migrations)
cut := strings.LastIndex(admin, "/")
t.Setenv(store.Variable(Name), admin[:cut]+"/"+name+"?sslmode=disable")
t.Setenv(store.Variable(Name), store.DatabaseForTest(admin, name))
held, err := Open(t.Context())
if err != nil {
+17 -6
View File
@@ -110,12 +110,7 @@ func raiseTemplate(ctx context.Context, admin, name string, key int64, migration
if _, err := conn.Exec(ctx, "create database "+name); err != nil {
return err
}
cut := strings.LastIndex(admin, "/")
rest := ""
if q := strings.Index(admin[cut:], "?"); q >= 0 {
rest = admin[cut+q:]
}
pool, err := pgxpool.New(ctx, admin[:cut]+"/"+name+rest)
pool, err := pgxpool.New(ctx, DatabaseForTest(admin, name))
if err != nil {
return err
}
@@ -136,3 +131,19 @@ func raiseTemplate(ctx context.Context, admin, name string, key int64, migration
}
return nil
}
// DatabaseForTest is the address of the database name on the same test PostgreSQL that admin reaches: admin
// with its database replaced and its query kept.
//
// **The query is kept, never replaced** (novox/hq issue 1889): the checker's integration store answers on a
// unix socket named in the query (`?host=<socket directory>&sslmode=disable`), and an address that drops it
// reaches pgx's default socket, where no server is, so every store test waited out its readiness bound and
// failed. Every store fixture builds its database's address here, so none can drop it again.
func DatabaseForTest(admin, name string) string {
cut := strings.LastIndex(admin, "/")
query := ""
if q := strings.Index(admin[cut:], "?"); q >= 0 {
query = admin[cut+q:]
}
return admin[:cut] + "/" + name + query
}
+25
View File
@@ -0,0 +1,25 @@
package store
import "testing"
// 1889/AC1: the checker's integration store answers on a unix socket named in the address's query
// (mesh-build-go's store). A test database's address that drops the query reaches pgx's default socket,
// where no server is, and every store test waits out its 20 s and fails.
func TestATestDatabaseIsOnTheSocketItsAdminAddressNames(t *testing.T) {
admin := "postgres://postgres@/postgres?host=%2Fwork%2Frun-1%2Fpg%2Fsocket&sslmode=disable"
got := DatabaseForTest(admin, "ident_1_x")
want := "postgres://postgres@/ident_1_x?host=%2Fwork%2Frun-1%2Fpg%2Fsocket&sslmode=disable"
if got != want {
t.Fatalf("the test database's address is %q, want %q", got, want)
}
}
// 1889/AC2: the address `make check` gives (TCP, with sslmode) builds what the fixtures always built.
func TestATestDatabaseOnTCPIsAddressedAsBefore(t *testing.T) {
admin := "postgres://postgres:check@127.0.0.1:55566/postgres?sslmode=disable"
got := DatabaseForTest(admin, "inv_2_y")
want := "postgres://postgres:check@127.0.0.1:55566/inv_2_y?sslmode=disable"
if got != want {
t.Fatalf("the test database's address is %q, want %q", got, want)
}
}
+1 -10
View File
@@ -55,7 +55,7 @@ func freshStore(t *testing.T) *Store {
}
conn.Close(t.Context())
t.Setenv(Variable("testing"), replaceDatabase(dsn, name))
t.Setenv(Variable("testing"), DatabaseForTest(dsn, name))
s, err := Open(t.Context(), "testing")
if err != nil {
t.Fatal(err)
@@ -75,15 +75,6 @@ func freshStore(t *testing.T) *Store {
return s
}
func replaceDatabase(dsn, name string) string {
cut := strings.LastIndex(dsn, "/")
rest := ""
if q := strings.Index(dsn[cut:], "?"); q >= 0 {
rest = dsn[cut+q:]
}
return dsn[:cut] + "/" + name + rest
}
func TestTheSchemaIsAppliedAndRecorded(t *testing.T) {
s := freshStore(t)
migrations := []Migration{{Number: 1, Name: "people", SQL: "create table person (id int)", Checksum: "a"}}