Compare commits
42
Commits
f45b4cb213
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7d80155d16 | ||
|
|
bafc31bf85 | ||
|
|
fbf5a3f909 | ||
|
|
20e14872ef | ||
|
|
5702187695 | ||
|
|
a79182e53b | ||
|
|
ff16938096 | ||
|
|
e3f22c737f | ||
|
|
2b99d455fc | ||
|
|
9e99f31c23 | ||
|
|
df39a499ef | ||
|
|
11725908ec | ||
|
|
379a52c2e8 | ||
|
|
b0d489a86c | ||
|
|
6355176fda | ||
|
|
300d3c1bf4 | ||
|
|
baf1501d81 | ||
|
|
a5ba47e23c | ||
|
|
6a62a7e552 | ||
|
|
5890eb104d | ||
|
|
262b39ae09 | ||
|
|
2f2a8bb57e | ||
|
|
1cdf12df00 | ||
|
|
f614eeb445 | ||
|
|
53a3c91f53 | ||
|
|
df5f01bfa1 | ||
|
|
32014ac6de | ||
|
|
c7cb9f0a2c | ||
|
|
87410d79ed | ||
|
|
1c3c3edc7d | ||
|
|
b2e2bf0638 | ||
|
|
e231185572 | ||
|
|
b12755c163 | ||
|
|
abf37c7212 | ||
|
|
ead062569e | ||
|
|
f4ddddd59d | ||
|
|
210f7cf5e6 | ||
|
|
9cb08299b3 | ||
|
|
5fa523f608 | ||
|
|
9a5c02bd60 | ||
|
|
9ce77e4e7b | ||
|
|
470c97654b |
+135
-68
@@ -27,9 +27,10 @@ import (
|
|||||||
// commit in it, with at most one commit per repository, the latest of its branch, which contains the earlier
|
// commit in it, with at most one commit per repository, the latest of its branch, which contains the earlier
|
||||||
// ones. The walk names every merge it answers.
|
// ones. The walk names every merge it answers.
|
||||||
//
|
//
|
||||||
// - One walk at a time. Merges that come while a walk runs join the next batch. A walk that started is never
|
// - One walk at a time beside a related one (ADR 0407 decision 4, issue 973). Merges that come while a walk
|
||||||
// superseded; one still waiting for its delivery's word is folded into the next batch's walk, which
|
// runs join the next batch; a closed batch that shares nothing with an open walk is cut beside it. A walk
|
||||||
// answers every merge it answered — the only supersession left.
|
// that started is never superseded; one still waiting for its delivery's word is folded into the walk of a
|
||||||
|
// batch related to it, which answers every merge it answered — the only supersession left.
|
||||||
// - A merge heard after a later merge of its branch was put in a walk is answered by that walk: a batch
|
// - A merge heard after a later merge of its branch was put in a walk is answered by that walk: a batch
|
||||||
// still assembling, an open walk that builds everything it moves, or a walk done that did, which then
|
// still assembling, an open walk that builds everything it moves, or a walk done that did, which then
|
||||||
// names it at once. Otherwise it joins the next batch.
|
// names it at once. Otherwise it joins the next batch.
|
||||||
@@ -39,8 +40,9 @@ import (
|
|||||||
//
|
//
|
||||||
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, ADR 0306 decisions
|
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, ADR 0306 decisions
|
||||||
// 2 and 3, issue 525): it joins a batch it is related to, and one that is related to none and whose delivery
|
// 2 and 3, issue 525): it joins a batch it is related to, and one that is related to none and whose delivery
|
||||||
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Walks stay one at
|
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Such a batch is cut
|
||||||
// a time: such a batch is cut when no walk is open, and queued behind the one that is otherwise.
|
// at once, beside any open walk it shares nothing with, and queued behind the first one it is related to
|
||||||
|
// (cutUnrelated).
|
||||||
//
|
//
|
||||||
// A merge on the controller's own path (a module whose walk waits for nobody's word) never shares a batch with
|
// A merge on the controller's own path (a module whose walk waits for nobody's word) never shares a batch with
|
||||||
// one that waits for mesh-delivery's: each kind has a batch of its own, an own-path batch is cut first and folds
|
// one that waits for mesh-delivery's: each kind has a batch of its own, an own-path batch is cut first and folds
|
||||||
@@ -61,13 +63,14 @@ var (
|
|||||||
// mergeWindowFileVar names the file the controller's window settings are written to.
|
// mergeWindowFileVar names the file the controller's window settings are written to.
|
||||||
const mergeWindowFileVar = "MESH_MERGE_WINDOW_FILE"
|
const mergeWindowFileVar = "MESH_MERGE_WINDOW_FILE"
|
||||||
|
|
||||||
// batchCutEvery is how often the controller looks whether a batch's window closed.
|
|
||||||
const batchCutEvery = 5 * time.Second
|
|
||||||
|
|
||||||
// batchLateAfter is how long past its maximum a batch may still be assembling, with no walk open, before
|
// batchLateAfter is how long past its maximum a batch may still be assembling, with no walk open, before
|
||||||
// the controller is said to have failed to cut it (ADR 0276 decision 9).
|
// the controller is said to have failed to cut it (ADR 0276 decision 9).
|
||||||
const batchLateAfter = time.Minute
|
const batchLateAfter = time.Minute
|
||||||
|
|
||||||
|
// mergeGrace is how long before a batch's first merge a merge may have been made and still be told apart from one
|
||||||
|
// made before the controller began keeping merges (owedLate).
|
||||||
|
const mergeGrace = 10 * time.Minute
|
||||||
|
|
||||||
// edgeGroupOrder is a delivery group's order made an edge of a walk's tiers (ADR 0276 decision 5): the
|
// edgeGroupOrder is a delivery group's order made an edge of a walk's tiers (ADR 0276 decision 5): the
|
||||||
// modules of the member after depend on the modules of the member before, and are gated on them.
|
// modules of the member after depend on the modules of the member before, and are gated on them.
|
||||||
const edgeGroupOrder = "group-order"
|
const edgeGroupOrder = "group-order"
|
||||||
@@ -125,8 +128,7 @@ func durationSetting(v any) (time.Duration, bool) {
|
|||||||
// hearMerge puts a merge the forge announced into the batch that answers it, at the moment it was heard,
|
// hearMerge puts a merge the forge announced into the batch that answers it, at the moment it was heard,
|
||||||
// and cuts what is due.
|
// and cuts what is due.
|
||||||
func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.Time) error {
|
func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.Time) error {
|
||||||
// One merge acted on at a time, whoever hands it over: the bus, or the catch-up that reads back
|
// One merge acted on at a time, whoever hands it over (novox/hq issue 266).
|
||||||
// what the bus did not hand over (novox/hq issue 266).
|
|
||||||
actingOnMerges.Lock()
|
actingOnMerges.Lock()
|
||||||
defer actingOnMerges.Unlock()
|
defer actingOnMerges.Unlock()
|
||||||
|
|
||||||
@@ -245,8 +247,8 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
|
|||||||
}
|
}
|
||||||
|
|
||||||
// owedLate says a merge that reads as history is still the record's to answer (review of this change): a cut
|
// owedLate says a merge that reads as history is still the record's to answer (review of this change): a cut
|
||||||
// marks every module it moved seen at that moment, so a merge of that branch the bus lost and the catch-up
|
// marks every module it moved seen at that moment, so a merge of that branch
|
||||||
// hands over later — made before the cut, heard after it — reads as history and was dropped. One that touches
|
// handed over late — made before the cut, heard after it — reads as history and was dropped. One that touches
|
||||||
// a module the mesh holds, of a branch whose merges the controller keeps, made since it began keeping them,
|
// a module the mesh holds, of a branch whose merges the controller keeps, made since it began keeping them,
|
||||||
// is kept; one made before it began is old news, as before.
|
// is kept; one made before it began is old news, as before.
|
||||||
func owedLate(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, merged time.Time,
|
func owedLate(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, merged time.Time,
|
||||||
@@ -340,8 +342,8 @@ func ownPath(touches []inventory.Entry) bool {
|
|||||||
// names why): a new batch, which waits for the window, beside the ones it shares nothing with; its words say
|
// names why): a new batch, which waits for the window, beside the ones it shares nothing with; its words say
|
||||||
// why.
|
// why.
|
||||||
// - none, and no member of its group is still to merge: a new batch whose window is closed at once. Cut at
|
// - none, and no member of its group is still to merge: a new batch whose window is closed at once. Cut at
|
||||||
// once means only that: walks stay one at a time (ADR 0276 decision 4), so with a walk open the batch is
|
// once means only that: with a walk open that it is related to the batch is queued behind it as any closed
|
||||||
// queued behind it as any closed batch is.
|
// batch is (ADR 0276 decision 4, ADR 0407 decision 4); beside one it shares nothing with it is cut.
|
||||||
func batchToJoin(ctx context.Context, inv *inventory.Inventory, heard inventory.BatchedMerge, entries []inventory.Entry,
|
func batchToJoin(ctx context.Context, inv *inventory.Inventory, heard inventory.BatchedMerge, entries []inventory.Entry,
|
||||||
read map[string][]inventory.ReadRepository, now time.Time, own bool, waits string) (inventory.Plan, string, error) {
|
read map[string][]inventory.ReadRepository, now time.Time, own bool, waits string) (inventory.Plan, string, error) {
|
||||||
edges, err := inv.Dependencies(ctx)
|
edges, err := inv.Dependencies(ctx)
|
||||||
@@ -406,7 +408,9 @@ func relatedBatches(ctx context.Context, inv *inventory.Inventory, merges []inve
|
|||||||
}
|
}
|
||||||
carry := newFootprint()
|
carry := newFootprint()
|
||||||
if !own {
|
if !own {
|
||||||
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch may fold every one.
|
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch folds those it is related to
|
||||||
|
// (cutUnrelated). Every one is counted here, which only makes a merge join a batch more readily: it waits
|
||||||
|
// for the window, never walks beside what it shares something with (issue 973).
|
||||||
plans, err := inv.OpenPlans(ctx)
|
plans, err := inv.OpenPlans(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, nil, err
|
return nil, nil, nil, err
|
||||||
@@ -544,6 +548,11 @@ func newBatch(ctx context.Context, inv *inventory.Inventory, first inventory.Bat
|
|||||||
// keepBatch writes a batch as its merges and the clock say it now: the commits it holds, the merges they
|
// keepBatch writes a batch as its merges and the clock say it now: the commits it holds, the merges they
|
||||||
// answer, its window, and assembling or queued behind a walk. Saved only when that changed.
|
// answer, its window, and assembling or queued behind a walk. Saved only when that changed.
|
||||||
func keepBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, now time.Time, behind string) error {
|
func keepBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, now time.Time, behind string) error {
|
||||||
|
return keepBatchBecause(ctx, inv, b, now, behind, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
// keepBatchBecause is keepBatch for a batch queued behind a walk it is related to, which its words say (issue 973).
|
||||||
|
func keepBatchBecause(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, now time.Time, behind, why string) error {
|
||||||
merges, err := inv.MergesOf(ctx, b.ID)
|
merges, err := inv.MergesOf(ctx, b.ID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -574,19 +583,25 @@ func keepBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan,
|
|||||||
cutAtOnce = was.CutAtOnce // the merge's, until another joins it (ADR 0304 decision 1)
|
cutAtOnce = was.CutAtOnce // the merge's, until another joins it (ADR 0304 decision 1)
|
||||||
waits = was.Waits // its group's, as said when it was made (ADR 0306 decision 2)
|
waits = was.Waits // its group's, as said when it was made (ADR 0306 decision 2)
|
||||||
}
|
}
|
||||||
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, Own: b.OwnPath(),
|
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, BehindWhy: why, Own: b.OwnPath(),
|
||||||
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits}
|
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits}
|
||||||
b.State = inventory.PlanAssembling
|
b.State = inventory.PlanAssembling
|
||||||
if behind != "" && windowClosed(*b, now) {
|
if behind != "" && windowClosed(*b, now) {
|
||||||
b.State = inventory.PlanQueued
|
b.State = inventory.PlanQueued
|
||||||
} else {
|
} else {
|
||||||
b.Delivery.Batch.Behind = ""
|
b.Delivery.Batch.Behind, b.Delivery.Batch.BehindWhy = "", ""
|
||||||
}
|
}
|
||||||
b.Note = batchWords(*b, now)
|
b.Note = batchWords(*b, now)
|
||||||
if unNoted(*b) == before {
|
if unNoted(*b) == before {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return inv.SavePlan(ctx, b)
|
if err := inv.SavePlan(ctx, b); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// The window's clock follows the record, only when the record moved: a merge made the quiet key again, a
|
||||||
|
// batch that opened made its most (ADR 0407 decision 1).
|
||||||
|
armWindow(ctx, *b, now)
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// windowOf is when a batch's window closes, unless another merge comes, and when at the latest: the last
|
// windowOf is when a batch's window closes, unless another merge comes, and when at the latest: the last
|
||||||
@@ -739,31 +754,10 @@ func nameMerges(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan
|
|||||||
return inv.SavePlan(ctx, p)
|
return inv.SavePlan(ctx, p)
|
||||||
}
|
}
|
||||||
|
|
||||||
// batchCutter cuts what is due on a timer, until the context ends: a window closes with no merge to say so.
|
|
||||||
func batchCutter(ctx context.Context, open *stores) {
|
|
||||||
tick := time.NewTicker(batchCutEvery)
|
|
||||||
defer tick.Stop()
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-tick.C:
|
|
||||||
}
|
|
||||||
release, err := open.inventory.HoldPlans(ctx, false)
|
|
||||||
if err != nil {
|
|
||||||
continue // whoever holds the plans moves them, and cuts at the end
|
|
||||||
}
|
|
||||||
if err := cutBatchesHeld(ctx, open, time.Now().UTC()); err != nil {
|
|
||||||
fmt.Printf("batches: %v\n", err)
|
|
||||||
}
|
|
||||||
release()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// cutBatchesHeld takes the step that is due, for a caller holding the plans: after a failed walk, the search
|
// cutBatchesHeld takes the step that is due, for a caller holding the plans: after a failed walk, the search
|
||||||
// for the merge that brought the failure; with no walk open, the next merge of that search walked alone, or a
|
// for the merge that brought the failure; with no walk open, the next merge of that search walked alone; else
|
||||||
// closed batch cut into one walk with the waiting walks folded in; with a walk open, a closed batch said
|
// each closed batch cut into a walk with the waiting walks it is related to folded in, unless it is related to a
|
||||||
// queued behind it.
|
// walk open, behind which it is said queued (cutUnrelated). While the search runs everything waits.
|
||||||
func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
|
func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
|
||||||
inv := open.inventory
|
inv := open.inventory
|
||||||
if err := searchAfterFailures(ctx, inv); err != nil {
|
if err := searchAfterFailures(ctx, inv); err != nil {
|
||||||
@@ -780,8 +774,7 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
var started *inventory.Plan
|
var started, waiting []inventory.Plan
|
||||||
var waiting []inventory.Plan
|
|
||||||
for i := range plans {
|
for i := range plans {
|
||||||
p := plans[i]
|
p := plans[i]
|
||||||
if p.Release != nil {
|
if p.Release != nil {
|
||||||
@@ -789,8 +782,8 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
|
|||||||
}
|
}
|
||||||
if p.Waiting() {
|
if p.Waiting() {
|
||||||
waiting = append(waiting, p)
|
waiting = append(waiting, p)
|
||||||
} else if started == nil {
|
} else {
|
||||||
started = &plans[i]
|
started = append(started, p)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
batches, err := inv.Batches(ctx)
|
batches, err := inv.Batches(ctx)
|
||||||
@@ -805,46 +798,116 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
if started != nil {
|
|
||||||
// **One walk at a time** (ADR 0276 decision 4): every batch waits behind it, whatever its window says.
|
|
||||||
return keepAll(started.ID)
|
|
||||||
}
|
|
||||||
if len(alone) > 0 {
|
if len(alone) > 0 {
|
||||||
|
// A failed walk's search walks its merges alone, one at a time: everything waits for it (kept as it was,
|
||||||
|
// issue 973).
|
||||||
|
if len(started) > 0 {
|
||||||
|
return keepAll(started[0].ID)
|
||||||
|
}
|
||||||
if len(waiting) > 0 {
|
if len(waiting) > 0 {
|
||||||
// The waiting walk is the one open; the search walks after it, and the batches wait behind both.
|
// The waiting walk is the one open; the search walks after it, and the batches wait behind both.
|
||||||
return keepAll(waiting[0].ID)
|
return keepAll(waiting[0].ID)
|
||||||
}
|
}
|
||||||
return walkAlone(ctx, open, alone[0], now)
|
return walkAlone(ctx, open, alone[0], now)
|
||||||
}
|
}
|
||||||
if err := keepAll(""); err != nil {
|
return cutUnrelated(ctx, open, batches, started, waiting, now)
|
||||||
|
}
|
||||||
|
|
||||||
|
// cutUnrelated cuts the closed batches that share nothing with an open walk, each into a walk beside the open
|
||||||
|
// ones, and says the others queued behind the first walk they are related to (novox/hq ADR 0407 decision 4,
|
||||||
|
// issue 973). A batch cut in this look counts as open for the batches after it.
|
||||||
|
//
|
||||||
|
// - An own-path batch goes first and folds no waiting walk; a catalogue batch folds the waiting walks it is
|
||||||
|
// related to, and those it shares nothing with keep waiting.
|
||||||
|
// - **The controller's own path is related to everything** (decided for issue 973): a walk or batch that
|
||||||
|
// replaces the controller, the node-engine, the tool runner, the bus or mesh-delivery never runs beside
|
||||||
|
// another, and nothing runs beside it.
|
||||||
|
// - A batch whose window is still open is assembling; one related to an open walk is queued behind it, and
|
||||||
|
// ADR 0276 decision 4 stands for it: nothing supersedes a walk that started.
|
||||||
|
func cutUnrelated(ctx context.Context, open *stores, batches, started, waiting []inventory.Plan, now time.Time) error {
|
||||||
|
inv := open.inventory
|
||||||
|
sort.SliceStable(batches, func(i, j int) bool { return batches[i].OwnPath() && !batches[j].OwnPath() })
|
||||||
|
var entries []inventory.Entry
|
||||||
|
var read map[string][]inventory.ReadRepository
|
||||||
|
var edges []inventory.Edge
|
||||||
|
judged := false
|
||||||
|
openWalks, err := openWalksOf(ctx, inv, started)
|
||||||
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
// The oldest batch whose window closed is cut, an own-path one before a catalogue one (it goes first, and
|
|
||||||
// the catalogue batch queues behind it rather than being cut and overtaken). Several of a kind may be open
|
|
||||||
// since a merge that shares nothing is cut at once (ADR 0304 decision 1): walks stay one at a time, so one
|
|
||||||
// is cut per look, and once its walk started the others are queued behind it.
|
|
||||||
sort.SliceStable(batches, func(i, j int) bool { return batches[i].OwnPath() && !batches[j].OwnPath() })
|
|
||||||
for i := range batches {
|
for i := range batches {
|
||||||
b := &batches[i]
|
b := &batches[i]
|
||||||
if !windowClosed(*b, now) {
|
if !windowClosed(*b, now) {
|
||||||
|
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if b.OwnPath() {
|
if !judged {
|
||||||
// A walk on the controller's own path folds no walk waiting for its word (decided during the build,
|
var err error
|
||||||
// 2026-10-10): that walk keeps waiting beside it, and starts once this one ended (advanceOnce).
|
if entries, err = inv.Catalogued(ctx); err != nil {
|
||||||
waiting = nil
|
|
||||||
}
|
|
||||||
if err := cutBatch(ctx, open, b, waiting, now); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
// The batches left wait behind the walk just cut when it started; the next look does the same.
|
if read, err = readForPlanning(ctx, inv); err != nil {
|
||||||
if b.Open() && !b.Waiting() {
|
|
||||||
if batches, err = inv.Batches(ctx); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return keepAll(b.ID)
|
if edges, err = inv.Dependencies(ctx); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
judged = true
|
||||||
|
}
|
||||||
|
held, err := inv.MergesOf(ctx, b.ID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fp := footprintAsCut(held, entries, read, edges)
|
||||||
|
var fold, keep []inventory.Plan
|
||||||
|
if !b.OwnPath() {
|
||||||
|
for _, w := range waiting {
|
||||||
|
ww, err := openWalkOf(ctx, inv, w)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if ok, _ := related(fp, ww.fp, edges); ok {
|
||||||
|
fold = append(fold, w)
|
||||||
|
fp = unionFootprint(fp, ww.fp)
|
||||||
|
} else {
|
||||||
|
keep = append(keep, w)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if behind, why := relatedOpen(fp, b.OwnPath(), openWalks, edges); behind != "" {
|
||||||
|
if err := keepBatchBecause(ctx, inv, b, now, behind, why); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := cutBatch(ctx, open, b, fold, now); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
waiting = keep
|
||||||
|
if c := windowClockOf(); c != nil {
|
||||||
|
c.Forget(ctx, b.ID) // a batch cut has no window left to close
|
||||||
|
}
|
||||||
|
cut, err := inv.PlanByID(ctx, b.ID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !cut.Open() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
w, err := openWalkOf(ctx, inv, cut)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if cut.Waiting() {
|
||||||
|
waiting = append(waiting, cut)
|
||||||
|
} else {
|
||||||
|
openWalks = append(openWalks, w)
|
||||||
}
|
}
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -1396,7 +1459,11 @@ func batchWords(b inventory.Plan, now time.Time) string {
|
|||||||
grouped += " (on the controller's own path)"
|
grouped += " (on the controller's own path)"
|
||||||
}
|
}
|
||||||
if b.State == inventory.PlanQueued {
|
if b.State == inventory.PlanQueued {
|
||||||
return fmt.Sprintf("queued behind %s; grouped: %s; plan not yet calculated", w.Behind, grouped)
|
because := ""
|
||||||
|
if w.BehindWhy != "" {
|
||||||
|
because = " (" + w.BehindWhy + ")"
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("queued behind %s%s; grouped: %s; plan not yet calculated", w.Behind, because, grouped)
|
||||||
}
|
}
|
||||||
if w.CutAtOnce {
|
if w.CutAtOnce {
|
||||||
return fmt.Sprintf("cut at once, sharing nothing with a batch assembling; grouped: %s; plan not yet calculated",
|
return fmt.Sprintf("cut at once, sharing nothing with a batch assembling; grouped: %s; plan not yet calculated",
|
||||||
|
|||||||
@@ -334,8 +334,8 @@ func captured(t *testing.T, run func() error) string {
|
|||||||
return b.String()
|
return b.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// One walk at a time: a merge heard while a walk runs joins the next batch, which is cut when the walk ends;
|
// A merge heard during a walk it is related to (the same repository, issue 973) joins the next batch, which is
|
||||||
// the walk that started is never superseded.
|
// cut when the walk ends; the walk that started is never superseded (ADR 0276 decision 4, ADR 0407 decision 4).
|
||||||
func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
|
func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
|
||||||
open := windowed(t)
|
open := windowed(t)
|
||||||
asksWithPaths(t)
|
asksWithPaths(t)
|
||||||
@@ -346,7 +346,7 @@ func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
|
|||||||
t.Fatalf("the first batch was not cut: %+v", ws)
|
t.Fatalf("the first batch was not cut: %+v", ws)
|
||||||
}
|
}
|
||||||
first := ws[0]
|
first := ws[0]
|
||||||
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
|
hear(t, open, repoMerge("one", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
|
||||||
cutAt(t, open, t0.Add(5*time.Minute))
|
cutAt(t, open, t0.Add(5*time.Minute))
|
||||||
ws, bs := walks(t, open)
|
ws, bs := walks(t, open)
|
||||||
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID {
|
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID {
|
||||||
@@ -391,7 +391,7 @@ func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
|
|||||||
t.Fatalf("the walk does not wait for its word: %+v", ws)
|
t.Fatalf("the walk does not wait for its word: %+v", ws)
|
||||||
}
|
}
|
||||||
waiting := ws[0]
|
waiting := ws[0]
|
||||||
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
|
hear(t, open, repoMerge("one", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
|
||||||
cutAt(t, open, t0.Add(2*time.Minute+90*time.Second))
|
cutAt(t, open, t0.Add(2*time.Minute+90*time.Second))
|
||||||
ws, bs := walks(t, open)
|
ws, bs := walks(t, open)
|
||||||
if len(bs) != 0 || len(ws) != 2 {
|
if len(bs) != 0 || len(ws) != 2 {
|
||||||
@@ -405,7 +405,7 @@ func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
|
|||||||
if folded.State != inventory.PlanSuperseded || folded.Delivery.TakenOverBy != newer.ID {
|
if folded.State != inventory.PlanSuperseded || folded.Delivery.TakenOverBy != newer.ID {
|
||||||
t.Fatalf("the waiting walk is %s, taken over by %q", folded.State, folded.Delivery.TakenOverBy)
|
t.Fatalf("the waiting walk is %s, taken over by %q", folded.State, folded.Delivery.TakenOverBy)
|
||||||
}
|
}
|
||||||
if len(newer.Delivery.Merges) != 2 || newer.CommitOf("novox/one") != "c1" || newer.CommitOf("novox/two") != "c2" {
|
if len(newer.Delivery.Merges) != 2 || newer.CommitOf("novox/one") != "c2" {
|
||||||
t.Fatalf("the newer walk answers %+v", newer.Delivery.Merges)
|
t.Fatalf("the newer walk answers %+v", newer.Delivery.Merges)
|
||||||
}
|
}
|
||||||
if _, in := newer.Modules["one"]; !in {
|
if _, in := newer.Modules["one"]; !in {
|
||||||
@@ -662,8 +662,8 @@ func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// One walk at a time holds against the delivery's word too: a waiting walk let go beside a started one takes the
|
// A related walk waits for the delivery's word too: a waiting walk let go beside a started one it is related to
|
||||||
// word and starts once that one ended, asking nothing before.
|
// (the same repository, issue 973) takes the word and starts once that one ended, asking nothing before.
|
||||||
func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
|
func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
|
||||||
open := windowed(t)
|
open := windowed(t)
|
||||||
asked := asksWithPaths(t)
|
asked := asksWithPaths(t)
|
||||||
@@ -671,8 +671,8 @@ func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
|
|||||||
hear(t, open, repoMerge("one", "c1", t0), t0)
|
hear(t, open, repoMerge("one", "c1", t0), t0)
|
||||||
cutAt(t, open, t0.Add(2*time.Minute))
|
cutAt(t, open, t0.Add(2*time.Minute))
|
||||||
started, _ := walks(t, open)
|
started, _ := walks(t, open)
|
||||||
waiting := inventory.Plan{ID: "plan-waiting", Repository: "novox/two", Branch: "main", Commit: "d1", Created: t0,
|
waiting := inventory.Plan{ID: "plan-waiting", Repository: "novox/one", Branch: "main", Commit: "d1", Created: t0,
|
||||||
State: inventory.PlanBuilding, Tiers: [][]string{{"two"}}, Modules: map[string]*inventory.PlanModule{"two": {}},
|
State: inventory.PlanBuilding, Tiers: [][]string{{"one"}}, Modules: map[string]*inventory.PlanModule{"one": {}},
|
||||||
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
|
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
|
||||||
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
|
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -729,49 +729,6 @@ func TestARemovedFileIsTheLastMergesWord(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The catch-up hands over what the bus lost after its branch's later merges were cut, and the record keeps it:
|
|
||||||
// an earlier merge is named by the walk that carried it; a merge made before a cut and heard after it, which
|
|
||||||
// no later merge carries, joins the next batch — neither reads as history and is dropped (review of this change).
|
|
||||||
func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
|
|
||||||
open := windowed(t)
|
|
||||||
asksWithPaths(t)
|
|
||||||
ctx := t.Context()
|
|
||||||
base := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
|
|
||||||
hear(t, open, repoMerge("one", "c2", base.Add(10*time.Second)), base.Add(11*time.Second))
|
|
||||||
cutAt(t, open, base.Add(2*time.Minute))
|
|
||||||
ws, _ := walks(t, open)
|
|
||||||
w := ws[0]
|
|
||||||
w.State = inventory.PlanDone
|
|
||||||
if err := open.inventory.SavePlan(ctx, &w); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
lost := []link.AnnouncedMerge{
|
|
||||||
{SourceMoved: repoMerge("one", "c1", base), At: base.Add(time.Second)},
|
|
||||||
{SourceMoved: repoMerge("one", "c3lost", base.Add(20*time.Second)), At: base.Add(21 * time.Second)},
|
|
||||||
}
|
|
||||||
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
|
|
||||||
entries, err := open.inventory.Catalogued(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
read, err := readForPlanning(ctx, open.inventory)
|
|
||||||
return entries, read, err
|
|
||||||
}
|
|
||||||
if err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announcedList(lost), open.inventory}, catalogued,
|
|
||||||
(following{open: open}).SourceMoved, t.Logf); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
got, _ := open.inventory.PlanByID(ctx, w.ID)
|
|
||||||
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
|
|
||||||
Commit: "c1", Carried: "c2"}) {
|
|
||||||
t.Fatalf("the earlier merge the catch-up handed over is not named by the walk that carried it: %+v",
|
|
||||||
got.Delivery.Merges)
|
|
||||||
}
|
|
||||||
if _, kept, err := open.inventory.MergeOf(ctx, "novox/one", "c3lost"); err != nil || !kept {
|
|
||||||
t.Fatalf("the merge made before the cut and heard after it was dropped: %v %v", kept, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// A merge on the controller's own path never shares a batch with one that waits for mesh-delivery's word
|
// A merge on the controller's own path never shares a batch with one that waits for mesh-delivery's word
|
||||||
// (decided during the build, 2026-10-10): the two are batches of their own, cut one after the other; a walk on
|
// (decided during the build, 2026-10-10): the two are batches of their own, cut one after the other; a walk on
|
||||||
// the own path folds no waiting catalogue walk but batches its merges again behind it; the catalogue's walk
|
// the own path folds no waiting catalogue walk but batches its merges again behind it; the catalogue's walk
|
||||||
|
|||||||
@@ -0,0 +1,106 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/novox/mesh-controller/internal/inventory"
|
||||||
|
)
|
||||||
|
|
||||||
|
// An unrelated closed batch is cut into a walk beside an open one, and that walk asks its first tier at once
|
||||||
|
// (issue 973, novox/hq ADR 0407 decision 4): a hotfix is never held by an unrelated delivery.
|
||||||
|
func TestAnUnrelatedBatchWalksBesideAnOpenWalk(t *testing.T) {
|
||||||
|
open := windowed(t)
|
||||||
|
asksWithPaths(t)
|
||||||
|
ctx := t.Context()
|
||||||
|
hear(t, open, repoMerge("one", "c1", t0), t0)
|
||||||
|
cutAt(t, open, t0.Add(90*time.Second))
|
||||||
|
if ws, _ := walks(t, open); len(ws) != 1 || !ws[0].Open() {
|
||||||
|
t.Fatalf("the first walk is not open: %+v", ws)
|
||||||
|
}
|
||||||
|
heard := t0.Add(100 * time.Second)
|
||||||
|
hear(t, open, lone(repoMerge("two", "c2", heard)), heard)
|
||||||
|
advanceHeld(ctx, open)
|
||||||
|
ws, bs := walks(t, open)
|
||||||
|
if len(ws) != 2 || len(bs) != 0 {
|
||||||
|
t.Fatalf("%d walk(s) and %d batch(es); want the unrelated batch cut beside the open walk", len(ws), len(bs))
|
||||||
|
}
|
||||||
|
for _, w := range ws {
|
||||||
|
if !w.Open() || w.Waiting() || deferred(w) || !askedAny(w) {
|
||||||
|
t.Errorf("%s (%s) is not walking: open %v, deferred %v, asked %v, note %q", w.ID, w.Named(), w.Open(),
|
||||||
|
deferred(w), askedAny(w), w.Note)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A closed batch related to an open walk is queued behind it with the reason, while one that shares nothing is
|
||||||
|
// cut beside it; the walk that started is never superseded (ADR 0276 decision 4, issue 973).
|
||||||
|
func TestARelatedBatchWaitsBehindTheOpenWalk(t *testing.T) {
|
||||||
|
open := windowed(t)
|
||||||
|
asksWithPaths(t)
|
||||||
|
ctx := t.Context()
|
||||||
|
hear(t, open, repoMerge("one", "c1", t0), t0)
|
||||||
|
cutAt(t, open, t0.Add(90*time.Second))
|
||||||
|
first := func() inventory.Plan { ws, _ := walks(t, open); return ws[len(ws)-1] }()
|
||||||
|
heard := t0.Add(100 * time.Second)
|
||||||
|
hear(t, open, lone(repoMerge("one", "c2", heard)), heard)
|
||||||
|
hear(t, open, lone(repoMerge("three", "c3", heard.Add(time.Second))), heard.Add(time.Second))
|
||||||
|
ws, bs := walks(t, open)
|
||||||
|
if len(ws) != 2 || len(bs) != 1 {
|
||||||
|
t.Fatalf("%d walk(s) and %d batch(es); want the unrelated merge walking and the related one queued", len(ws), len(bs))
|
||||||
|
}
|
||||||
|
b := bs[0]
|
||||||
|
words := batchWords(b, heard)
|
||||||
|
if b.State != inventory.PlanQueued || b.Delivery.Batch.Behind != first.ID || b.CommitOf("novox/one") != "c2" ||
|
||||||
|
!strings.Contains(words, "queued behind "+first.ID) || !strings.Contains(words, "both change novox/one") {
|
||||||
|
t.Fatalf("the related batch is %s, behind %q, reading %q", b.State, b.Delivery.Batch.Behind, words)
|
||||||
|
}
|
||||||
|
if got, _ := open.inventory.PlanByID(ctx, first.ID); !got.Open() {
|
||||||
|
t.Fatalf("the started walk was %s by the related merge", got.State)
|
||||||
|
}
|
||||||
|
first.State = inventory.PlanDone
|
||||||
|
if err := open.inventory.SavePlan(ctx, &first); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
cutAt(t, open, heard.Add(2*time.Second))
|
||||||
|
if _, bs := walks(t, open); len(bs) != 0 {
|
||||||
|
t.Fatalf("the queued batch was not cut once the walk it waited for ended: %+v", bs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A batch on the controller's own path never walks beside another walk, and nothing walks beside it, whatever
|
||||||
|
// the two share (issue 973): it replaces the controller, so it goes alone.
|
||||||
|
func TestAnOwnPathBatchNeverWalksBeside(t *testing.T) {
|
||||||
|
open := windowed(t)
|
||||||
|
asksWithPaths(t)
|
||||||
|
ctx := t.Context()
|
||||||
|
hear(t, open, repoMerge("one", "c1", t0), t0)
|
||||||
|
cutAt(t, open, t0.Add(90*time.Second))
|
||||||
|
ws, _ := walks(t, open)
|
||||||
|
first := ws[0]
|
||||||
|
heard := t0.Add(100 * time.Second)
|
||||||
|
hear(t, open, lone(repoMerge("mesh-controller", "k1", heard)), heard)
|
||||||
|
ws, bs := walks(t, open)
|
||||||
|
if len(ws) != 1 || len(bs) != 1 || !bs[0].OwnPath() || bs[0].State != inventory.PlanQueued ||
|
||||||
|
bs[0].Delivery.Batch.Behind != first.ID || !strings.Contains(batchWords(bs[0], heard), "own path") {
|
||||||
|
t.Fatalf("an own-path batch beside an open walk: %d walk(s), batches %+v", len(ws), bs)
|
||||||
|
}
|
||||||
|
first.State = inventory.PlanDone
|
||||||
|
if err := open.inventory.SavePlan(ctx, &first); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
cutAt(t, open, heard.Add(time.Second))
|
||||||
|
ws, _ = walks(t, open)
|
||||||
|
if len(ws) != 2 || !ws[0].Open() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
|
||||||
|
t.Fatalf("the own-path batch was not cut once the walk ended: %+v", ws)
|
||||||
|
}
|
||||||
|
own := ws[0]
|
||||||
|
// Unrelated to the own-path walk by every footprint, a merge still waits for it.
|
||||||
|
later := heard.Add(time.Minute)
|
||||||
|
hear(t, open, lone(repoMerge("two", "c2", later)), later)
|
||||||
|
ws, bs = walks(t, open)
|
||||||
|
if len(ws) != 2 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != own.ID {
|
||||||
|
t.Fatalf("a merge beside the own-path walk: %d walk(s), batches %+v", len(ws), bs)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -136,8 +136,9 @@ func letGo(ctx context.Context, inv *inventory.Inventory, id, by, why string) (i
|
|||||||
return p, fmt.Errorf("%s was let go by %s at %s already", p.ID, p.Delivery.By,
|
return p, fmt.Errorf("%s was let go by %s at %s already", p.ID, p.Delivery.By,
|
||||||
p.Delivery.Go.Local().Format("15:04:05"))
|
p.Delivery.Go.Local().Format("15:04:05"))
|
||||||
}
|
}
|
||||||
// **One walk at a time** (novox/hq ADR 0276): the word is taken, and the walk starts once no other walk is
|
// **One walk at a time beside a related one** (novox/hq ADR 0276, ADR 0407 decision 4, issue 973): the word is
|
||||||
// started (advanceOnce), so the delivery's owner says it once and is not refused.
|
// taken, and the walk starts once no started walk it is related to is open (advanceOnce), so the delivery's
|
||||||
|
// owner says it once and is not refused.
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
p.Delivery.Go, p.Delivery.By, p.Delivery.Why = &now, by, why
|
p.Delivery.Go, p.Delivery.By, p.Delivery.Why = &now, by, why
|
||||||
p.Note = "let go by " + by + "; its first tier is asked next"
|
p.Note = "let go by " + by + "; its first tier is asked next"
|
||||||
@@ -268,7 +269,7 @@ func stopWalkWithout(ctx context.Context, inv *inventory.Inventory, id, by, why
|
|||||||
append(moves, foldMoves...)); err != nil {
|
append(moves, foldMoves...)); err != nil {
|
||||||
return p, said, fmt.Errorf("%s is not stopped, and none of its merges moved: %w", p.ID, err)
|
return p, said, fmt.Errorf("%s is not stopped, and none of its merges moved: %w", p.ID, err)
|
||||||
}
|
}
|
||||||
// What the batch holds and names now: written again by the cutter's next look should this fail.
|
// What the batch holds and names now: written again by the next look should this fail.
|
||||||
if err := keepBatch(ctx, inv, &target, now, ""); err != nil {
|
if err := keepBatch(ctx, inv, &target, now, ""); err != nil {
|
||||||
fmt.Printf(" %s holds its merges; its record is written again at the next look: %v\n", target.ID, err)
|
fmt.Printf(" %s holds its merges; its record is written again at the next look: %v\n", target.ID, err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -287,6 +287,12 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
|
|||||||
return snapshot.Facts{}, err
|
return snapshot.Facts{}, err
|
||||||
}
|
}
|
||||||
m.Architecture, m.Kernel = reported.Architecture, reported.Kernel
|
m.Architecture, m.Kernel = reported.Architecture, reported.Kernel
|
||||||
|
// Whether its node-engine takes handed files is its contract (issue 969).
|
||||||
|
if stated, had, err := inv.HealthOf(ctx, n.Name); err != nil {
|
||||||
|
return snapshot.Facts{}, err
|
||||||
|
} else if had {
|
||||||
|
m.Contract = stated.Contract
|
||||||
|
}
|
||||||
outward, err := inv.OutwardLinksOf(ctx, n.Name)
|
outward, err := inv.OutwardLinksOf(ctx, n.Name)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return snapshot.Facts{}, err
|
return snapshot.Facts{}, err
|
||||||
|
|||||||
@@ -227,8 +227,9 @@ func TestAMergeRelatedToTwoBatchesFoldsThem(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// With a walk open, a batch cut at once still waits for it, said queued behind it (walks stay one at a time, ADR
|
// With a walk open that it is related to (the same repository, issue 973), a batch cut at once still waits for
|
||||||
// 0276 decision 4); it is cut once that walk ended, with no window to wait for.
|
// it, said queued behind it (ADR 0276 decision 4, ADR 0407 decision 4); it is cut once that walk ended, with no
|
||||||
|
// window to wait for.
|
||||||
func TestACutAtOnceBatchQueuesBehindAnOpenWalk(t *testing.T) {
|
func TestACutAtOnceBatchQueuesBehindAnOpenWalk(t *testing.T) {
|
||||||
open := windowed(t)
|
open := windowed(t)
|
||||||
groupsWaiting(t, "pr/c1")
|
groupsWaiting(t, "pr/c1")
|
||||||
@@ -241,7 +242,7 @@ func TestACutAtOnceBatchQueuesBehindAnOpenWalk(t *testing.T) {
|
|||||||
}
|
}
|
||||||
first := ws[0]
|
first := ws[0]
|
||||||
heard := t0.Add(100 * time.Second)
|
heard := t0.Add(100 * time.Second)
|
||||||
hear(t, open, lone(repoMerge("two", "c2", heard)), heard)
|
hear(t, open, lone(repoMerge("one", "c2", heard)), heard)
|
||||||
ws, bs := walks(t, open)
|
ws, bs := walks(t, open)
|
||||||
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID ||
|
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID ||
|
||||||
!bs[0].Delivery.Batch.CutAtOnce {
|
!bs[0].Delivery.Batch.CutAtOnce {
|
||||||
@@ -280,7 +281,8 @@ func TestAStopWalksTheOthersAgainInABatchOfTheirOwnWhenTheyShareNothing(t *testi
|
|||||||
cutAt(t, open, t0.Add(2*time.Minute))
|
cutAt(t, open, t0.Add(2*time.Minute))
|
||||||
ws, _ := walks(t, open)
|
ws, _ := walks(t, open)
|
||||||
walk := ws[0]
|
walk := ws[0]
|
||||||
hear(t, open, repoMerge("two", "two1cccccccc", t0.Add(3*time.Minute)), t0.Add(3*time.Minute))
|
// Related to the walk by its repository (queued behind it), and to nothing the stop walks again.
|
||||||
|
hear(t, open, catalogueMerge("cat2cccccccc", "notes", t0.Add(3*time.Minute)), t0.Add(3*time.Minute))
|
||||||
_, bs := walks(t, open)
|
_, bs := walks(t, open)
|
||||||
if len(bs) != 1 {
|
if len(bs) != 1 {
|
||||||
t.Fatalf("the merge heard while the walk ran is in %d batches", len(bs))
|
t.Fatalf("the merge heard while the walk ran is in %d batches", len(bs))
|
||||||
|
|||||||
@@ -1017,6 +1017,13 @@ func raiseFromFacts(ctx context.Context, open *stores, f snapshot.Facts, shelf m
|
|||||||
"kernel": m.Kernel, "capabilities": caps}); err != nil {
|
"kernel": m.Kernel, "capabilities": caps}); err != nil {
|
||||||
return notes, err
|
return notes, err
|
||||||
}
|
}
|
||||||
|
// What its node-engine takes, as the mesh composes it today (issue 969): without the statement the gate
|
||||||
|
// composes a moved module the old way on a machine the mesh hands it to.
|
||||||
|
if stated, ok := standInHealth(m, f.Taken); ok {
|
||||||
|
if _, err := inv.RecordHealth(ctx, stated); err != nil {
|
||||||
|
return notes, err
|
||||||
|
}
|
||||||
|
}
|
||||||
for _, record := range []func(context.Context, string, string) error{inv.RecordSealingKey, inv.RecordOverlayKey} {
|
for _, record := range []func(context.Context, string, string) error{inv.RecordSealingKey, inv.RecordOverlayKey} {
|
||||||
key, err := standInKey()
|
key, err := standInKey()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1355,3 +1362,12 @@ func firstOr(items, otherwise []string) string {
|
|||||||
}
|
}
|
||||||
return "it says nothing more"
|
return "it says nothing more"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// standInHealth is the statement the gate gives a machine of the snapshot so that it composes as the mesh does:
|
||||||
|
// only its contract, which says whether it takes handed files (issue 969). False for a snapshot that carries none.
|
||||||
|
func standInHealth(m snapshot.Machine, taken time.Time) (inventory.NodeHealth, bool) {
|
||||||
|
if m.Contract == 0 {
|
||||||
|
return inventory.NodeHealth{}, false
|
||||||
|
}
|
||||||
|
return inventory.NodeHealth{Node: m.Name, Contract: m.Contract, SaidAt: taken, HeardAt: taken}, true
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,85 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/novox/mesh-controller/internal/catalogue"
|
||||||
|
snapshot "github.com/novox/mesh-controller/internal/facts"
|
||||||
|
"github.com/novox/mesh-controller/internal/inventory"
|
||||||
|
"github.com/novox/mesh-controller/internal/link"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Issue 969: the gate composes a machine as the mesh does, so a moved module is handed where the machine takes
|
||||||
|
// handed files and composed the old way where it does not.
|
||||||
|
func TestIssue969TheGateGivesAMachineItsContractSoItTakesHandedFiles(t *testing.T) {
|
||||||
|
taken := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
|
||||||
|
|
||||||
|
stated, ok := standInHealth(snapshot.Machine{Name: "f35", Contract: link.HandedContract}, taken)
|
||||||
|
if !ok || !takesHanded(stated, ok) {
|
||||||
|
t.Fatalf("a machine whose engine takes handed files must be given that in the gate: %+v, %v", stated, ok)
|
||||||
|
}
|
||||||
|
old, ok := standInHealth(snapshot.Machine{Name: "e12", Contract: link.HandedContract - 1}, taken)
|
||||||
|
if !ok || takesHanded(old, ok) {
|
||||||
|
t.Fatalf("a machine whose engine is older must not take handed files: %+v, %v", old, ok)
|
||||||
|
}
|
||||||
|
none, ok := standInHealth(snapshot.Machine{Name: "d01"}, taken)
|
||||||
|
if ok || takesHanded(none, ok) {
|
||||||
|
t.Fatalf("a snapshot that carries no contract states nothing: %+v, %v", none, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Issue 969, through the store: a machine whose engine stated the handed contract is in the snapshot with it, the
|
||||||
|
// gate raises it as taking handed files, and one whose statement is below it is raised as not taking them. Run in
|
||||||
|
// the merge check with the other tests that need a store; the composition itself, handed here and refused there, is
|
||||||
|
// the catalogue's TestAHandedModuleNamingARunTimeBoundKeyComposesOnlyWhereItIsHanded.
|
||||||
|
func TestIssue969TheGateRaisesEachMachineAsItsEngineStatedItself(t *testing.T) {
|
||||||
|
open := aMesh(t)
|
||||||
|
ctx := t.Context()
|
||||||
|
said := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
|
||||||
|
for node, contract := range map[string]int{"anchor": link.HandedContract, "laptop": link.HandedContract - 1} {
|
||||||
|
if _, err := open.inventory.RecordHealth(ctx, inventory.NodeHealth{Node: node, Contract: contract,
|
||||||
|
SaidAt: said, HeardAt: said}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
f, err := gatherFacts(ctx, open, "2.11.17")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(f.Machines) != 2 {
|
||||||
|
t.Fatalf("the snapshot holds %d machines", len(f.Machines))
|
||||||
|
}
|
||||||
|
want := map[string]bool{}
|
||||||
|
for _, m := range f.Machines {
|
||||||
|
if m.Contract == 0 {
|
||||||
|
t.Fatalf("%s: the snapshot carries no contract", m.Name)
|
||||||
|
}
|
||||||
|
want[m.Name] = m.Contract >= link.HandedContract
|
||||||
|
}
|
||||||
|
handing := 0
|
||||||
|
for _, hands := range want {
|
||||||
|
if hands {
|
||||||
|
handing++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if handing != 1 {
|
||||||
|
t.Fatalf("one machine takes handed files in the snapshot, got %v", want)
|
||||||
|
}
|
||||||
|
|
||||||
|
raised, drop, err := throwawayStores(ctx, os.Getenv("MESH_TEST_POSTGRES"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer drop()
|
||||||
|
if _, err := raiseFromFacts(ctx, raised, f, map[string]catalogue.Manifest{}, map[string]inventory.Source{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for name, hands := range want {
|
||||||
|
got, err := engineTakesHanded(ctx, raised.inventory, name)
|
||||||
|
if err != nil || got != hands {
|
||||||
|
t.Errorf("%s: the gate raises it as taking handed files = %v (%v), the mesh says %v", name, got, err, hands)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/novox/mesh-controller/internal/broker"
|
||||||
|
"github.com/novox/mesh-controller/internal/inventory"
|
||||||
|
"github.com/novox/mesh-controller/internal/link"
|
||||||
|
"github.com/novox/mesh-controller/internal/testbus"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The merge path whole, against a real bus and the store (issue 973 and issue 921, novox/hq ADR 0407 decisions 1
|
||||||
|
// and 3; ADR 0391's proof): the forge's merged event is published on the subject the controller consumes, the
|
||||||
|
// controller's server hands it over and the window keys are on the bus. Store-backed: skips without one.
|
||||||
|
|
||||||
|
// noListener hears no node: these tests send merges only.
|
||||||
|
type noListener struct{ link.Listener }
|
||||||
|
|
||||||
|
// theMergePathOn serves the controller's consume loop and its merge window on a test bus, and returns a way to
|
||||||
|
// publish the forge's merged event.
|
||||||
|
func theMergePathOn(t *testing.T, open *stores) func(link.SourceMoved) {
|
||||||
|
t.Helper()
|
||||||
|
js, err := broker.Dial(testbus.URL(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(js.Close)
|
||||||
|
if err := broker.AssertMeshStreams(js); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := js.EnsureControllerBuckets(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
server := link.ConnectNats(js, nil, noListener{})
|
||||||
|
if err := server.Follows(following{open: open}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx, stop := context.WithCancel(context.Background())
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() { defer close(done); _ = server.Serve(ctx) }()
|
||||||
|
if err := startMergeWindow(ctx, open, js); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
stop()
|
||||||
|
<-done
|
||||||
|
setWindowClock(nil)
|
||||||
|
windowLook.Store(nil)
|
||||||
|
})
|
||||||
|
return func(m link.SourceMoved) {
|
||||||
|
t.Helper()
|
||||||
|
body, err := json.Marshal(m)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := js.Context().Publish(broker.ControllerFollows[3], body); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func becomes(t *testing.T, within time.Duration, what string, is func() bool) {
|
||||||
|
t.Helper()
|
||||||
|
deadline := time.Now().Add(within)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if is() {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
time.Sleep(50 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatalf("%s did not happen within %s", what, within)
|
||||||
|
}
|
||||||
|
|
||||||
|
func mergeKept(open *stores, repo, commit string) bool {
|
||||||
|
_, known, err := open.inventory.MergeOf(context.Background(), repo, commit)
|
||||||
|
return err == nil && known
|
||||||
|
}
|
||||||
|
|
||||||
|
// Two merges of one repository heard while another controller holds the plans, and a rollout of that repository
|
||||||
|
// open, are both recorded once the plans are free (handed over again by the bus, never dropped), join one batch,
|
||||||
|
// and the next rollout carries both: no person released anything (issue 921).
|
||||||
|
func TestMergesHeardWhileAnotherHolderHasThePlansAreRecordedAndCarried(t *testing.T) {
|
||||||
|
open := windowed(t)
|
||||||
|
asksWithPaths(t)
|
||||||
|
was, wasPoll := inventory.HoldWaitFor, inventory.HoldPoll
|
||||||
|
inventory.HoldWaitFor, inventory.HoldPoll = time.Second, 20*time.Millisecond
|
||||||
|
t.Cleanup(func() { inventory.HoldWaitFor, inventory.HoldPoll = was, wasPoll })
|
||||||
|
// A window of a second: a merge joining a batch opens its window again (ADR 0407 decision 1), so the batch the
|
||||||
|
// two held merges make is cut when that window has closed and the rollout before it has ended, not before.
|
||||||
|
settings := t.TempDir() + "/merge-window.json"
|
||||||
|
if err := os.WriteFile(settings, []byte(`{"merge-window": "1s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv(mergeWindowFileVar, settings)
|
||||||
|
publish := theMergePathOn(t, open)
|
||||||
|
ctx := t.Context()
|
||||||
|
|
||||||
|
publish(repoMerge("one", "c1", time.Now()))
|
||||||
|
becomes(t, 15*time.Second, "the first rollout opening", func() bool {
|
||||||
|
ws, _ := walks(t, open)
|
||||||
|
return len(ws) == 1 && ws[0].Open()
|
||||||
|
})
|
||||||
|
ws, _ := walks(t, open)
|
||||||
|
first := ws[0]
|
||||||
|
|
||||||
|
release, err := open.inventory.HoldPlans(ctx, true) // the other controller works the plans
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
publish(repoMerge("one", "c2", time.Now()))
|
||||||
|
publish(repoMerge("one", "c3", time.Now()))
|
||||||
|
time.Sleep(2500 * time.Millisecond) // both met the hold and were given back to the bus
|
||||||
|
if mergeKept(open, "novox/one", "c2") || mergeKept(open, "novox/one", "c3") {
|
||||||
|
t.Fatal("a merge was recorded while another holder had the plans")
|
||||||
|
}
|
||||||
|
release()
|
||||||
|
becomes(t, 30*time.Second, "both merges recorded after the hold", func() bool {
|
||||||
|
return mergeKept(open, "novox/one", "c2") && mergeKept(open, "novox/one", "c3")
|
||||||
|
})
|
||||||
|
|
||||||
|
first.State = inventory.PlanDone
|
||||||
|
if err := open.inventory.SavePlan(ctx, &first); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// The rollout ended: the look that cuts what waited behind it, once the batch's window is closed too.
|
||||||
|
var bs []inventory.Plan
|
||||||
|
becomes(t, 15*time.Second, "the next rollout being cut with both merges", func() bool {
|
||||||
|
advanceHeld(ctx, open)
|
||||||
|
ws, bs = walks(t, open)
|
||||||
|
return len(bs) == 0 && len(ws) == 2
|
||||||
|
})
|
||||||
|
if ws[0].CommitOf("novox/one") != "c3" || len(ws[0].Delivery.Merges) != 2 {
|
||||||
|
t.Fatalf("the next rollout does not carry both merges: commit %q, merges %+v", ws[0].CommitOf("novox/one"),
|
||||||
|
ws[0].Delivery.Merges)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A batch's window closes by the expiry of its key on the bus, heard by the watcher, and the batch is cut with
|
||||||
|
// no cutter and no call to cut (ADR 0407 decision 1): a window of a second, a merge that waits for it.
|
||||||
|
func TestABatchWindowClosesByTheExpiryOfItsKey(t *testing.T) {
|
||||||
|
open := windowed(t)
|
||||||
|
asksWithPaths(t)
|
||||||
|
everyHeadWaits(t)
|
||||||
|
settings := t.TempDir() + "/merge-window.json"
|
||||||
|
if err := os.WriteFile(settings, []byte(`{"merge-window": "1s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv(mergeWindowFileVar, settings)
|
||||||
|
publish := theMergePathOn(t, open)
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
publish(repoMerge("one", "c1", start))
|
||||||
|
becomes(t, 10*time.Second, "the merge waiting in a batch", func() bool {
|
||||||
|
_, bs := walks(t, open)
|
||||||
|
return len(bs) == 1 && bs[0].State == inventory.PlanAssembling
|
||||||
|
})
|
||||||
|
if ws, _ := walks(t, open); len(ws) != 0 {
|
||||||
|
t.Fatalf("the batch was cut before its window closed: %+v", ws)
|
||||||
|
}
|
||||||
|
becomes(t, 15*time.Second, "the batch being cut by its key's expiry", func() bool {
|
||||||
|
ws, bs := walks(t, open)
|
||||||
|
return len(ws) == 1 && len(bs) == 0
|
||||||
|
})
|
||||||
|
if took := time.Since(start); took < time.Second {
|
||||||
|
t.Fatalf("the batch was cut after %s, before its window of a second was up", took)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,198 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/nats-io/nats.go/jetstream"
|
||||||
|
|
||||||
|
"github.com/novox/mesh-controller/internal/broker"
|
||||||
|
"github.com/novox/mesh-controller/internal/inventory"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The merge window's clock is on the bus (novox/hq ADR 0407 decision 1, issue 973): an open batch has a key
|
||||||
|
// `<batch>.quiet` that lives the window and is made again by each merge into the batch, and a key `<batch>.most`
|
||||||
|
// made once when the batch opens that lives the window's most. The first of the two to expire closes the batch,
|
||||||
|
// and the watcher hears it: no timer polls the batches. The store still keeps the times (ClosesAt, AtMost) as the
|
||||||
|
// record of when a window closed, which `plans` reads; the keys are what wakes the controller.
|
||||||
|
|
||||||
|
// windowClock is the bus's side of the merge window.
|
||||||
|
type windowClock interface {
|
||||||
|
// Arm makes a batch's keys: its quiet key again, to live quiet; its most key if it has none, to live most.
|
||||||
|
Arm(ctx context.Context, batch string, quiet, most time.Duration) error
|
||||||
|
// Forget takes a batch's keys away: it was cut, or closed, and nothing is to wake the controller for it.
|
||||||
|
Forget(ctx context.Context, batch string)
|
||||||
|
}
|
||||||
|
|
||||||
|
// windowLook is the look a window's expiry takes: the watcher's, and the fallback timer's.
|
||||||
|
var windowLook atomic.Pointer[func(batch string)]
|
||||||
|
|
||||||
|
var (
|
||||||
|
windowMu sync.RWMutex
|
||||||
|
windowKeyed windowClock // nil in a process that is not the serving controller
|
||||||
|
)
|
||||||
|
|
||||||
|
// setWindowClock makes clock the merge window's; nil takes it away.
|
||||||
|
func setWindowClock(c windowClock) {
|
||||||
|
windowMu.Lock()
|
||||||
|
defer windowMu.Unlock()
|
||||||
|
windowKeyed = c
|
||||||
|
}
|
||||||
|
|
||||||
|
func windowClockOf() windowClock {
|
||||||
|
windowMu.RLock()
|
||||||
|
defer windowMu.RUnlock()
|
||||||
|
return windowKeyed
|
||||||
|
}
|
||||||
|
|
||||||
|
// windowDeadlines are how long a batch's two keys are to live from now: what is left of its window and of its
|
||||||
|
// most, each at least a second (the least a key can live). Not armed for a batch whose window is closed already
|
||||||
|
// (by a stop, by its time, or because it is cut at once) or that is no batch.
|
||||||
|
func windowDeadlines(b inventory.Plan, now time.Time) (quiet, most time.Duration, armed bool) {
|
||||||
|
if b.Delivery == nil || b.Delivery.Batch == nil || windowClosed(b, now) {
|
||||||
|
return 0, 0, false
|
||||||
|
}
|
||||||
|
// Rounded up to a whole second, at least one: the server rounds a key's time to live down to whole seconds, so
|
||||||
|
// a fraction would close the key before the record's time and the look would cut nothing.
|
||||||
|
least := func(d time.Duration) time.Duration {
|
||||||
|
if d < time.Second {
|
||||||
|
return time.Second
|
||||||
|
}
|
||||||
|
return (d + time.Second - 1) / time.Second * time.Second
|
||||||
|
}
|
||||||
|
w := b.Delivery.Batch
|
||||||
|
return least(w.ClosesAt.Sub(now)), least(w.AtMost.Sub(now)), true
|
||||||
|
}
|
||||||
|
|
||||||
|
// armWindow puts a batch's clock on the bus after its record was kept. Where the bus will not take it the window
|
||||||
|
// closes by a timer of this batch alone, once, as long as is left: never silently never.
|
||||||
|
func armWindow(ctx context.Context, b inventory.Plan, now time.Time) {
|
||||||
|
c := windowClockOf()
|
||||||
|
quiet, most, armed := windowDeadlines(b, now)
|
||||||
|
if c == nil || !armed {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := c.Arm(ctx, b.ID, quiet, most); err != nil {
|
||||||
|
fmt.Printf("batches: the window of %s could not be put on the bus (%v); it closes by a timer of its own\n", b.ID, err)
|
||||||
|
if look := windowLook.Load(); look != nil {
|
||||||
|
time.AfterFunc(quiet, func() { (*look)(b.ID) })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// busWindow is the clock on the controller's bucket.
|
||||||
|
type busWindow struct {
|
||||||
|
kv jetstream.KeyValue
|
||||||
|
}
|
||||||
|
|
||||||
|
// newBusWindow opens the bucket and starts the watcher, which names each batch whose key expired to look until
|
||||||
|
// the context ends. Only expiry wakes it: a key taken away by Forget or made again by Arm leaves a delete.
|
||||||
|
func newBusWindow(ctx context.Context, api jetstream.JetStream, look func(batch string)) (*busWindow, error) {
|
||||||
|
kv, err := api.KeyValue(ctx, broker.MergeWindowBucket)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("the merge window's bucket: %w", err)
|
||||||
|
}
|
||||||
|
w, err := kv.WatchAll(ctx, jetstream.UpdatesOnly())
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("watching the merge window: %w", err)
|
||||||
|
}
|
||||||
|
go func() {
|
||||||
|
defer w.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case e, ok := <-w.Updates():
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if e == nil || e.Operation() != jetstream.KeyValuePurge {
|
||||||
|
continue // a put, or a delete this controller made
|
||||||
|
}
|
||||||
|
if batch, _, found := strings.Cut(e.Key(), "."); found && look != nil {
|
||||||
|
look(batch)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
return &busWindow{kv: kv}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w *busWindow) Arm(ctx context.Context, batch string, quiet, most time.Duration) error {
|
||||||
|
// The most is made once, when the batch opens: it is there already for every merge after the first.
|
||||||
|
if _, err := w.kv.Create(ctx, batch+".most", []byte("most"), jetstream.KeyTTL(most)); err != nil &&
|
||||||
|
!errors.Is(err, jetstream.ErrKeyExists) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// The quiet key is made again, not changed: a key's time to live is set when it is made.
|
||||||
|
if err := w.kv.Delete(ctx, batch+".quiet"); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if _, err := w.kv.Create(ctx, batch+".quiet", []byte("quiet"), jetstream.KeyTTL(quiet)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w *busWindow) Forget(ctx context.Context, batch string) {
|
||||||
|
for _, key := range []string{batch + ".quiet", batch + ".most"} {
|
||||||
|
_ = w.kv.Delete(ctx, key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// lookAtBatches is the look an expired window asks for: the plans held, as a merge heard holds them, and the batches
|
||||||
|
// whose window closed cut (cutBatchesHeld). Waiting for the plans rather than giving up: an expiry is heard once.
|
||||||
|
func lookAtBatches(ctx context.Context, open *stores) {
|
||||||
|
release, err := open.inventory.HoldPlans(ctx, true)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Printf("batches: a window closed and the plans could not be held for the look: %v\n", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer release()
|
||||||
|
if err := cutBatchesHeld(ctx, open, time.Now().UTC()); err != nil {
|
||||||
|
fmt.Printf("batches: %v\n", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// startMergeWindow puts the merge window's clock on the bus for as long as the context lives: the controller is
|
||||||
|
// the lease holder here, the only one that writes it. The watcher looks at the batches when a key expires; the
|
||||||
|
// batches open now get their keys, and one look is taken, so a window that closed while no controller served is
|
||||||
|
// cut (ADR 0407 decision 1). No timer polls them.
|
||||||
|
func startMergeWindow(ctx context.Context, open *stores, js *broker.JetStream) error {
|
||||||
|
api, err := jetstream.New(js.Conn())
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
look := func(string) { lookAtBatches(ctx, open) }
|
||||||
|
windowLook.Store(&look)
|
||||||
|
clock, err := newBusWindow(ctx, api, look)
|
||||||
|
if err != nil {
|
||||||
|
windowLook.Store(nil)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
setWindowClock(clock)
|
||||||
|
go func() {
|
||||||
|
<-ctx.Done()
|
||||||
|
setWindowClock(nil)
|
||||||
|
windowLook.Store(nil)
|
||||||
|
}()
|
||||||
|
release, err := open.inventory.HoldPlans(ctx, true)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer release()
|
||||||
|
batches, err := open.inventory.Batches(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
now := time.Now().UTC()
|
||||||
|
for _, b := range batches {
|
||||||
|
armWindow(ctx, b, now)
|
||||||
|
}
|
||||||
|
return cutBatchesHeld(ctx, open, now)
|
||||||
|
}
|
||||||
@@ -0,0 +1,151 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/nats-io/nats.go"
|
||||||
|
"github.com/nats-io/nats.go/jetstream"
|
||||||
|
|
||||||
|
"github.com/novox/mesh-controller/internal/broker"
|
||||||
|
"github.com/novox/mesh-controller/internal/inventory"
|
||||||
|
"github.com/novox/mesh-controller/internal/testbus"
|
||||||
|
)
|
||||||
|
|
||||||
|
// windowOnABus is the merge window's clock on a real bus, and the batches its watcher heard expire.
|
||||||
|
func windowOnABus(t *testing.T) (*busWindow, <-chan string) {
|
||||||
|
t.Helper()
|
||||||
|
js, err := broker.Dial(testbus.URL(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(js.Close)
|
||||||
|
if err := js.EnsureControllerBuckets(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
conn, err := nats.Connect(testbus.URL(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(conn.Close)
|
||||||
|
api, err := jetstream.New(conn)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
heard := make(chan string, 16)
|
||||||
|
ctx, stop := context.WithCancel(context.Background())
|
||||||
|
t.Cleanup(stop)
|
||||||
|
w, err := newBusWindow(ctx, api, func(batch string) { heard <- batch })
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return w, heard
|
||||||
|
}
|
||||||
|
|
||||||
|
func hears(t *testing.T, heard <-chan string, within time.Duration) (string, bool) {
|
||||||
|
t.Helper()
|
||||||
|
select {
|
||||||
|
case b := <-heard:
|
||||||
|
return b, true
|
||||||
|
case <-time.After(within):
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A batch's quiet key expires when no merge came for the window, and the watcher names the batch (issue 973,
|
||||||
|
// novox/hq ADR 0407 decision 1).
|
||||||
|
func TestTheQuietKeyExpiresAndTheWatcherNamesTheBatch(t *testing.T) {
|
||||||
|
w, heard := windowOnABus(t)
|
||||||
|
if err := w.Arm(t.Context(), "plan-1", time.Second, time.Minute); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if b, ok := hears(t, heard, 10*time.Second); !ok || b != "plan-1" {
|
||||||
|
t.Fatalf("heard %q (%v); want plan-1", b, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A merge into the batch makes the quiet key again, so the window restarts: it does not expire at the first
|
||||||
|
// deadline, and the batch's most closes it all the same.
|
||||||
|
func TestAMergeRestartsTheQuietKeyAndTheMostKeyClosesAllTheSame(t *testing.T) {
|
||||||
|
w, heard := windowOnABus(t)
|
||||||
|
ctx := t.Context()
|
||||||
|
if err := w.Arm(ctx, "plan-2", 2*time.Second, 5*time.Second); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
time.Sleep(1200 * time.Millisecond)
|
||||||
|
if err := w.Arm(ctx, "plan-2", 2*time.Second, 5*time.Second); err != nil { // a merge: quiet again, most as it was
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if b, ok := hears(t, heard, 1500*time.Millisecond); ok {
|
||||||
|
t.Fatalf("heard %q before the restarted window was up", b)
|
||||||
|
}
|
||||||
|
// Another merge keeps the window open; the most (5 s from the first) closes it at about 5 s.
|
||||||
|
start := time.Now()
|
||||||
|
for time.Since(start) < 2500*time.Millisecond {
|
||||||
|
if err := w.Arm(ctx, "plan-2", 2*time.Second, 5*time.Second); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
time.Sleep(700 * time.Millisecond)
|
||||||
|
}
|
||||||
|
if b, ok := hears(t, heard, 8*time.Second); !ok || b != "plan-2" {
|
||||||
|
t.Fatalf("the most did not close the batch: heard %q (%v)", b, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A batch cut is forgotten: its keys are taken away by a delete, which is no expiry and wakes nobody.
|
||||||
|
func TestForgettingABatchWakesNobody(t *testing.T) {
|
||||||
|
w, heard := windowOnABus(t)
|
||||||
|
if err := w.Arm(t.Context(), "plan-3", time.Second, time.Minute); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
w.Forget(t.Context(), "plan-3")
|
||||||
|
if b, ok := hears(t, heard, 3*time.Second); ok {
|
||||||
|
t.Fatalf("a forgotten batch was heard expire: %q", b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The deadlines a batch's keys are made with: what is left of its window and of its most, at least a second;
|
||||||
|
// none for a batch whose window is closed or that is no batch.
|
||||||
|
func TestTheDeadlinesOfABatchsKeys(t *testing.T) {
|
||||||
|
now := time.Date(2026, 10, 17, 12, 0, 0, 0, time.UTC)
|
||||||
|
batch := func(closes, most time.Duration, closed bool) inventory.Plan {
|
||||||
|
return inventory.Plan{State: inventory.PlanAssembling, Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{
|
||||||
|
ClosesAt: now.Add(closes), AtMost: now.Add(most), Closed: closed}}}
|
||||||
|
}
|
||||||
|
for _, c := range []struct {
|
||||||
|
what string
|
||||||
|
b inventory.Plan
|
||||||
|
quiet, most time.Duration
|
||||||
|
armed bool
|
||||||
|
}{
|
||||||
|
{"open", batch(90*time.Second, 10*time.Minute, false), 90 * time.Second, 10 * time.Minute, true},
|
||||||
|
{"almost up", batch(100*time.Millisecond, 5*time.Second, false), time.Second, 5 * time.Second, true},
|
||||||
|
{"closed by a stop", batch(90*time.Second, 10*time.Minute, true), 0, 0, false},
|
||||||
|
{"up", batch(-time.Second, 10*time.Minute, false), 0, 0, false},
|
||||||
|
{"cut at once", func() inventory.Plan {
|
||||||
|
b := batch(90*time.Second, 10*time.Minute, false)
|
||||||
|
b.Delivery.Batch.CutAtOnce = true
|
||||||
|
return b
|
||||||
|
}(), 0, 0, false},
|
||||||
|
{"no batch", inventory.Plan{}, 0, 0, false},
|
||||||
|
} {
|
||||||
|
quiet, most, armed := windowDeadlines(c.b, now)
|
||||||
|
if armed != c.armed || quiet != c.quiet || most != c.most {
|
||||||
|
t.Errorf("%s: %v, %v, %v; want %v, %v, %v", c.what, quiet, most, armed, c.quiet, c.most, c.armed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A deadline that is not a whole number of seconds is rounded up, never down: the server rounds a key's time to live
|
||||||
|
// down to whole seconds, so 89.999999 s would expire at 89 s, before the window is closed by the record's times, and
|
||||||
|
// the look would cut nothing (review of issue 973).
|
||||||
|
func TestAFractionalDeadlineIsRoundedUpToTheNextWholeSecond(t *testing.T) {
|
||||||
|
now := time.Date(2026, 10, 17, 12, 0, 0, 0, time.UTC)
|
||||||
|
b := inventory.Plan{State: inventory.PlanAssembling, Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{
|
||||||
|
ClosesAt: now.Add(89*time.Second + 999999*time.Microsecond), AtMost: now.Add(599*time.Second + time.Nanosecond)}}}
|
||||||
|
quiet, most, armed := windowDeadlines(b, now)
|
||||||
|
if !armed || quiet != 90*time.Second || most != 600*time.Second {
|
||||||
|
t.Errorf("deadlines %v, %v (armed %v); want 1m30s, 10m0s", quiet, most, armed)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,243 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
"sync"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/novox/mesh-controller/internal/inventory"
|
|
||||||
"github.com/novox/mesh-controller/internal/link"
|
|
||||||
)
|
|
||||||
|
|
||||||
// **A merge the bus announced and the controller never acted on is caught up** (novox/hq issue 266).
|
|
||||||
//
|
|
||||||
// The forge's poll announces every merge on the events stream, and the controller acts on what its
|
|
||||||
// consumer there hands it. On 2026-10-06 one merge was on the stream and never handed over: the bus
|
|
||||||
// server moved the consumer past it — a fault of consumers with several filters in the server the
|
|
||||||
// mesh ran — and the controller, which only acts on what it is handed, said nothing. The modules
|
|
||||||
// built from that repository stayed behind and were built by hand.
|
|
||||||
//
|
|
||||||
// So the stream is read back on a timer, on a consumer of its own filtered on merges alone, and every
|
|
||||||
// announcement older than mergeGrace is judged as SourceMoved would judge it. **No record of what
|
|
||||||
// was handled is kept, because none is needed**: acting on a merge marks every module it moved as
|
|
||||||
// looked at since, so an announcement already acted on reads as history and moves nothing. One that
|
|
||||||
// would still move something was never acted on — it is said, and acted on now.
|
|
||||||
const (
|
|
||||||
// mergeGrace is how long an announcement is left to the controller's own consumer before it is
|
|
||||||
// judged missed. That consumer hands over one event at a time, and a merge waits behind a build
|
|
||||||
// outcome that is being acted on; acting on a merge itself asks builds and does not wait for them.
|
|
||||||
mergeGrace = 10 * time.Minute
|
|
||||||
// mergeLookBack is how far back a pass reads. A merge missed longer ago than this was missed by a
|
|
||||||
// controller that was not running this, and is the operator's to look at, not a surprise rebuild.
|
|
||||||
mergeLookBack = 24 * time.Hour
|
|
||||||
// mergeCatchUpEvery is how often the stream is read back.
|
|
||||||
mergeCatchUpEvery = 5 * time.Minute
|
|
||||||
)
|
|
||||||
|
|
||||||
// merges is what reads back the forge's announcements; the link server, or a test's list.
|
|
||||||
type merges interface {
|
|
||||||
AnnouncedMerges(ctx context.Context, since time.Time) ([]link.AnnouncedMerge, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// unheardMerges is the announcements of merges the controller has not heard (novox/hq ADR 0276): a merge kept
|
|
||||||
// in a batch is not acted on until its batch is cut, which can be past mergeGrace behind a long walk, and is
|
|
||||||
// not missed for that.
|
|
||||||
type unheardMerges struct {
|
|
||||||
merges
|
|
||||||
inv *inventory.Inventory
|
|
||||||
}
|
|
||||||
|
|
||||||
func (u unheardMerges) AnnouncedMerges(ctx context.Context, since time.Time) ([]link.AnnouncedMerge, error) {
|
|
||||||
all, err := u.merges.AnnouncedMerges(ctx, since)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
var out []link.AnnouncedMerge
|
|
||||||
for _, a := range all {
|
|
||||||
_, kept, err := u.inv.MergeOf(ctx, a.Owner+"/"+a.Repo, a.Commit)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
if !kept {
|
|
||||||
out = append(out, a)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return out, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// owedToTheRecord says a merge read as history is still owed to the record (novox/hq ADR 0276): the merges
|
|
||||||
// reader knows, when it keeps them (unheardMerges).
|
|
||||||
func owedToTheRecord(ctx context.Context, announced merges, m link.SourceMoved, entries []inventory.Entry,
|
|
||||||
read map[string][]inventory.ReadRepository) bool {
|
|
||||||
u, ok := announced.(unheardMerges)
|
|
||||||
if !ok {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
merged, err := time.Parse(time.RFC3339Nano, m.MergedAt)
|
|
||||||
if err != nil {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
raw := m
|
|
||||||
raw.MergedAt = ""
|
|
||||||
owed, err := owedLate(ctx, u.inv, m, merged, wouldMove(raw, entries, read))
|
|
||||||
return err == nil && owed
|
|
||||||
}
|
|
||||||
|
|
||||||
// catchingUpOnMerges reads back the forge's announcements on a timer, until the context ends.
|
|
||||||
func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) {
|
|
||||||
f := following{open: open}
|
|
||||||
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
|
|
||||||
entries, err := open.inventory.Catalogued(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
read, err := readForPlanning(ctx, open.inventory)
|
|
||||||
return entries, read, err
|
|
||||||
}
|
|
||||||
failing := ""
|
|
||||||
tick := time.NewTicker(mergeCatchUpEvery)
|
|
||||||
defer tick.Stop()
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-tick.C:
|
|
||||||
}
|
|
||||||
watchedMerges.begin()
|
|
||||||
err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announced, open.inventory}, catalogued, f.SourceMoved, func(format string, args ...any) {
|
|
||||||
fmt.Printf(format+"\n", args...)
|
|
||||||
})
|
|
||||||
// What the pass found is what S5 says (novox/hq to-be 45 §3); a pass that could not read
|
|
||||||
// says that instead, and leaves what the last one found standing.
|
|
||||||
watchedMerges.end(time.Now(), err)
|
|
||||||
// A pass that cannot read says so once, not every five minutes, and says when it reads again.
|
|
||||||
why := ""
|
|
||||||
if err != nil {
|
|
||||||
why = err.Error()
|
|
||||||
}
|
|
||||||
if why != failing {
|
|
||||||
if why != "" {
|
|
||||||
fmt.Printf("merges the bus may not have handed over cannot be looked for: %s\n", why)
|
|
||||||
} else {
|
|
||||||
fmt.Println("merges the bus may not have handed over are looked for again")
|
|
||||||
}
|
|
||||||
failing = why
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// catchUpOnMerges is one pass: every announcement older than mergeGrace that acting on would still
|
|
||||||
// move something is said and acted on, oldest first.
|
|
||||||
//
|
|
||||||
// Judged twice: once against the catalogue as the pass found it, and again just before acting,
|
|
||||||
// because acting on an earlier missed merge of the same repository may have moved what a later one
|
|
||||||
// would have.
|
|
||||||
func catchUpOnMerges(ctx context.Context, now time.Time, announced merges,
|
|
||||||
catalogued func(context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error),
|
|
||||||
act func(context.Context, link.SourceMoved) error, say func(string, ...any)) error {
|
|
||||||
|
|
||||||
all, err := announced.AnnouncedMerges(ctx, now.Add(-mergeLookBack))
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
entries, read, err := catalogued(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
for _, a := range all {
|
|
||||||
// A merge the forge said no time of is dated by its announcement, so a packaging module's look can
|
|
||||||
// make it history once acted on, and the catch-up does not act on it again every pass (ADR 0267).
|
|
||||||
if a.SourceMoved.MergedAt == "" && !a.At.IsZero() {
|
|
||||||
a.SourceMoved.MergedAt = a.At.UTC().Format(time.RFC3339)
|
|
||||||
}
|
|
||||||
if now.Sub(a.At) < mergeGrace {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if len(wouldMove(a.SourceMoved, entries, read)) == 0 && !owedToTheRecord(ctx, announced, a.SourceMoved, entries, read) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if entries, read, err = catalogued(ctx); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
moves := wouldMove(a.SourceMoved, entries, read)
|
|
||||||
if len(moves) == 0 {
|
|
||||||
if !owedToTheRecord(ctx, announced, a.SourceMoved, entries, read) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
raw := a.SourceMoved
|
|
||||||
raw.MergedAt = ""
|
|
||||||
moves = wouldMove(raw, entries, read)
|
|
||||||
}
|
|
||||||
var names []string
|
|
||||||
for _, e := range moves {
|
|
||||||
names = append(names, e.Manifest.Module)
|
|
||||||
}
|
|
||||||
watchedMerges.found(missedMerge{Owner: a.Owner, Repo: a.Repo, Base: a.Base, Commit: a.Commit,
|
|
||||||
At: a.At, Modules: names})
|
|
||||||
say("%s/%s merged into %s (%.8s), announced %s ago, and the controller never acted on it: the bus "+
|
|
||||||
"did not hand the announcement over (novox/hq issue 266). %s %s behind it; acting on it now",
|
|
||||||
a.Owner, a.Repo, a.Base, a.Commit, now.Sub(a.At).Round(time.Minute), readableList(names),
|
|
||||||
isAre(len(names)))
|
|
||||||
if err := act(ctx, a.SourceMoved); err != nil {
|
|
||||||
say("%s/%s moved to %.8s and the mesh could not act on it: %v; the next pass tries again",
|
|
||||||
a.Owner, a.Repo, a.Commit, err)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if entries, read, err = catalogued(ctx); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// missedMerge is one merge the bus announced and never handed over, as a pass found it.
|
|
||||||
type missedMerge struct {
|
|
||||||
Owner, Repo, Base, Commit string
|
|
||||||
// At is when the bus took the announcement.
|
|
||||||
At time.Time
|
|
||||||
Modules []string
|
|
||||||
}
|
|
||||||
|
|
||||||
// mergeWatch is what the passes found, for S5 (novox/hq to-be 45 §3): **a merge nothing read is
|
|
||||||
// said**, urgent, even though the pass acts on it at once — the bus skipping a message is a fault of
|
|
||||||
// the transport the mesh's every change rides on, and acting late is the repair, not the absence of
|
|
||||||
// the fault. The next pass, finding it acted on, clears it.
|
|
||||||
type mergeWatch struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
passed time.Time
|
|
||||||
err error
|
|
||||||
finding []missedMerge
|
|
||||||
missed []missedMerge
|
|
||||||
}
|
|
||||||
|
|
||||||
var watchedMerges = &mergeWatch{}
|
|
||||||
|
|
||||||
func (w *mergeWatch) begin() {
|
|
||||||
w.mu.Lock()
|
|
||||||
defer w.mu.Unlock()
|
|
||||||
w.finding = nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (w *mergeWatch) found(m missedMerge) {
|
|
||||||
w.mu.Lock()
|
|
||||||
defer w.mu.Unlock()
|
|
||||||
w.finding = append(w.finding, m)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (w *mergeWatch) end(at time.Time, err error) {
|
|
||||||
w.mu.Lock()
|
|
||||||
defer w.mu.Unlock()
|
|
||||||
w.passed, w.err = at, err
|
|
||||||
if err == nil {
|
|
||||||
w.missed = w.finding
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// last is when the last pass ended, what the last pass that read found, and what the last pass
|
|
||||||
// could not read.
|
|
||||||
func (w *mergeWatch) last() (time.Time, []missedMerge, error) {
|
|
||||||
w.mu.Lock()
|
|
||||||
defer w.mu.Unlock()
|
|
||||||
return w.passed, append([]missedMerge(nil), w.missed...), w.err
|
|
||||||
}
|
|
||||||
@@ -1,180 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/novox/mesh-controller/internal/inventory"
|
|
||||||
"github.com/novox/mesh-controller/internal/link"
|
|
||||||
)
|
|
||||||
|
|
||||||
// announcedList is the events stream's merges as a test gives them.
|
|
||||||
type announcedList []link.AnnouncedMerge
|
|
||||||
|
|
||||||
func (a announcedList) AnnouncedMerges(_ context.Context, since time.Time) ([]link.AnnouncedMerge, error) {
|
|
||||||
var out []link.AnnouncedMerge
|
|
||||||
for _, m := range a {
|
|
||||||
if !m.At.Before(since) {
|
|
||||||
out = append(out, m)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return out, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// aCatalogue is what the inventory holds, changed the way acting on a merge changes it: every module
|
|
||||||
// the merge moved is marked as looked at (inventory.SourceMoved writes source_seen = now()).
|
|
||||||
type aCatalogue struct {
|
|
||||||
entries []inventory.Entry
|
|
||||||
acted []string
|
|
||||||
fail error
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *aCatalogue) read(context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
|
|
||||||
return append([]inventory.Entry(nil), c.entries...), nil, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *aCatalogue) act(now func() time.Time) func(context.Context, link.SourceMoved) error {
|
|
||||||
return func(_ context.Context, m link.SourceMoved) error {
|
|
||||||
if c.fail != nil {
|
|
||||||
return c.fail
|
|
||||||
}
|
|
||||||
c.acted = append(c.acted, m.Repo+"@"+m.Commit[:8])
|
|
||||||
for _, moved := range wouldMove(m, c.entries, nil) {
|
|
||||||
for i := range c.entries {
|
|
||||||
if c.entries[i].Manifest.Module == moved.Manifest.Module {
|
|
||||||
c.entries[i].Source.Head = m.Commit
|
|
||||||
c.entries[i].Source.Seen = now()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func at(s string) time.Time {
|
|
||||||
t, err := time.Parse(time.RFC3339, s)
|
|
||||||
if err != nil {
|
|
||||||
panic(err)
|
|
||||||
}
|
|
||||||
return t
|
|
||||||
}
|
|
||||||
|
|
||||||
func announced(repo, commit, mergedAt, onTheBus string, paths ...string) link.AnnouncedMerge {
|
|
||||||
return link.AnnouncedMerge{
|
|
||||||
SourceMoved: link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Commit: commit,
|
|
||||||
MergedAt: mergedAt, Paths: paths,
|
|
||||||
CloneURL: "http://forge.internal:20000/novox/" + repo + ".git"},
|
|
||||||
At: at(onTheBus),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func built(module, repo, path, commit, seen string) inventory.Entry {
|
|
||||||
e := fromRepo(module, "http://forge.internal:20000/novox/"+repo+".git", path)
|
|
||||||
e.Source.BuiltFrom, e.Source.Head, e.Source.Seen = commit, commit, at(seen)
|
|
||||||
return e
|
|
||||||
}
|
|
||||||
|
|
||||||
// **novox/hq issue 266, as it happened.** The forge announced a merge of the tools repository on the
|
|
||||||
// events stream; the bus never handed it to the controller, which acted on the merges around it and
|
|
||||||
// not on this one, and said nothing. Read back from the stream, it is the one merge that would still
|
|
||||||
// move something — so it is said and acted on, once, and only after the controller's own consumer
|
|
||||||
// has had its time with it.
|
|
||||||
func TestAMergeTheBusNeverHandedOverIsActedOnLate(t *testing.T) {
|
|
||||||
cat := &aCatalogue{entries: []inventory.Entry{
|
|
||||||
built("mesh-tools", "mesh-tools", "", "8b789578aaaaaaaa", "2026-10-04T15:24:32Z"),
|
|
||||||
built("node-tools", "mesh-tools", "node-tools", "8b789578aaaaaaaa", "2026-10-04T15:24:32Z"),
|
|
||||||
// Acted on when it was announced: looked at after it was merged.
|
|
||||||
built("gitea", "mesh-catalog", "modules/gitea", "5c2157b8bbbbbbbb", "2026-10-05T22:39:21Z"),
|
|
||||||
}}
|
|
||||||
stream := announcedList{
|
|
||||||
// Nothing the mesh holds is built from the records repository.
|
|
||||||
announced("hq", "88f7f79fcccccccc", "2026-10-05T22:43:00Z", "2026-10-05T22:43:04Z", "04-ISSUES/x.md"),
|
|
||||||
// Acted on: its module was looked at since.
|
|
||||||
announced("mesh-catalog", "78328d4adddddddd", "2026-10-05T22:39:00Z", "2026-10-05T22:39:21Z", "modules/gitea/x.ts"),
|
|
||||||
// Never handed over.
|
|
||||||
announced("mesh-tools", "9730bd89c3e48d0e", "2026-10-05T22:46:47Z", "2026-10-05T22:47:06Z",
|
|
||||||
"node-tools/internal/console/console.go"),
|
|
||||||
}
|
|
||||||
var said []string
|
|
||||||
say := func(format string, args ...any) { said = append(said, fmt.Sprintf(format, args...)) }
|
|
||||||
clock := at("2026-10-05T22:50:00Z")
|
|
||||||
now := func() time.Time { return clock }
|
|
||||||
pass := func() {
|
|
||||||
t.Helper()
|
|
||||||
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(now), say); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
pass()
|
|
||||||
if len(cat.acted) != 0 {
|
|
||||||
t.Fatalf("a merge three minutes old was taken from the controller's own consumer: %v", cat.acted)
|
|
||||||
}
|
|
||||||
|
|
||||||
clock = at("2026-10-05T22:58:00Z")
|
|
||||||
pass()
|
|
||||||
if strings.Join(cat.acted, ",") != "mesh-tools@9730bd89" {
|
|
||||||
t.Fatalf("acted on %v, wanted the one merge never handed over", cat.acted)
|
|
||||||
}
|
|
||||||
if len(said) != 1 || !strings.Contains(said[0], "novox/mesh-tools merged into main (9730bd89)") ||
|
|
||||||
!strings.Contains(said[0], "mesh-tools and node-tools are behind it") {
|
|
||||||
t.Fatalf("the missed merge was not said as one: %q", said)
|
|
||||||
}
|
|
||||||
|
|
||||||
clock = at("2026-10-05T23:03:00Z")
|
|
||||||
pass()
|
|
||||||
if len(cat.acted) != 1 || len(said) != 1 {
|
|
||||||
t.Fatalf("a merge acted on was acted on again: %v %q", cat.acted, said)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// A merge that changed none of the held modules' files moves nothing, so it is never "missed"; one
|
|
||||||
// that could not be acted on is said and tried again on the next pass.
|
|
||||||
func TestAMissedMergeThatCouldNotBeActedOnIsTriedAgain(t *testing.T) {
|
|
||||||
cat := &aCatalogue{
|
|
||||||
entries: []inventory.Entry{built("gitea", "mesh-catalog", "modules/gitea", "5c2157b8bbbbbbbb", "2026-10-05T20:00:00Z")},
|
|
||||||
fail: errors.New("the store is restarting"),
|
|
||||||
}
|
|
||||||
stream := announcedList{
|
|
||||||
announced("mesh-catalog", "aaaaaaaa11111111", "2026-10-05T21:00:00Z", "2026-10-05T21:00:10Z",
|
|
||||||
"modules/plex/module.json", "modules/plex/x.ts"),
|
|
||||||
announced("mesh-catalog", "bbbbbbbb22222222", "2026-10-05T21:10:00Z", "2026-10-05T21:10:10Z", "modules/gitea/x.ts"),
|
|
||||||
}
|
|
||||||
var said []string
|
|
||||||
say := func(format string, args ...any) { said = append(said, fmt.Sprintf(format, args...)) }
|
|
||||||
clock := at("2026-10-05T22:00:00Z")
|
|
||||||
now := func() time.Time { return clock }
|
|
||||||
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(now), say); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if len(said) != 2 || !strings.Contains(said[1], "could not act on it") {
|
|
||||||
t.Fatalf("a failed catch-up was not said: %q", said)
|
|
||||||
}
|
|
||||||
cat.fail = nil
|
|
||||||
clock = at("2026-10-05T22:05:00Z")
|
|
||||||
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(now), say); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if strings.Join(cat.acted, ",") != "mesh-catalog@bbbbbbbb" {
|
|
||||||
t.Fatalf("acted on %v, wanted only the merge that changed a held module", cat.acted)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// A merge older than the look-back is left to the operator: a controller that did not run this
|
|
||||||
// missed it, and acting on it days later would be a surprise rebuild.
|
|
||||||
func TestAMergeOlderThanTheLookBackIsLeftAlone(t *testing.T) {
|
|
||||||
cat := &aCatalogue{entries: []inventory.Entry{built("gitea", "mesh-catalog", "modules/gitea", "5c2157b8bbbbbbbb", "2026-10-01T00:00:00Z")}}
|
|
||||||
stream := announcedList{announced("mesh-catalog", "cccccccc33333333", "2026-10-03T00:00:00Z", "2026-10-03T00:00:05Z", "modules/gitea/x.ts")}
|
|
||||||
clock := at("2026-10-05T22:00:00Z")
|
|
||||||
if err := catchUpOnMerges(context.Background(), clock, stream, cat.read, cat.act(func() time.Time { return clock }),
|
|
||||||
func(string, ...any) {}); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if len(cat.acted) != 0 {
|
|
||||||
t.Fatalf("a merge of three days ago was acted on: %v", cat.acted)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -429,12 +429,6 @@ var plainWordings = map[string]func(conditions.Observation) words{
|
|||||||
"out) has not let it start yet.",
|
"out) has not let it start yet.",
|
||||||
Resolved: "Resolved: the delivery is no longer waiting"}
|
Resolved: "Resolved: the delivery is no longer waiting"}
|
||||||
}),
|
}),
|
||||||
"merge-not-acted": worded(func(o conditions.Observation) words {
|
|
||||||
return words{Headline: "A merge was never picked up",
|
|
||||||
Explanation: "A pull request was merged, and the controller never heard of it, so nothing was built or " +
|
|
||||||
"sent. The mesh catches up on missed merges by itself.",
|
|
||||||
Resolved: "Resolved: the merge was picked up"}
|
|
||||||
}),
|
|
||||||
"ask-lost": worded(func(o conditions.Observation) words {
|
"ask-lost": worded(func(o conditions.Observation) words {
|
||||||
return words{Headline: "A build has no result",
|
return words{Headline: "A build has no result",
|
||||||
Explanation: "A build was asked for and never answered. It is asked again with the next delivery.",
|
Explanation: "A build was asked for and never answered. It is asked again with the next delivery.",
|
||||||
|
|||||||
@@ -1126,7 +1126,13 @@ func engineTakesHanded(ctx context.Context, inv *inventory.Inventory, node strin
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return false, err
|
return false, err
|
||||||
}
|
}
|
||||||
return had && stated.Contract >= link.HandedContract, nil
|
return takesHanded(stated, had), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// takesHanded is whether a machine's newest statement (had: it ever stated one) is of an engine that writes a
|
||||||
|
// filled file on the provisioner client's word alone.
|
||||||
|
func takesHanded(stated inventory.NodeHealth, had bool) bool {
|
||||||
|
return had && stated.Contract >= link.HandedContract
|
||||||
}
|
}
|
||||||
|
|
||||||
// provisionerAccount is the account the provisioner's client runs as on this machine (novox/hq ADR 0349 §4): the user
|
// provisionerAccount is the account the provisioner's client runs as on this machine (novox/hq ADR 0349 §4): the user
|
||||||
|
|||||||
@@ -197,12 +197,23 @@ func retryRefusal(p inventory.Plan, plans []inventory.Plan) error {
|
|||||||
return oneWalkAtATime(p, plans)
|
return oneWalkAtATime(p, plans)
|
||||||
}
|
}
|
||||||
|
|
||||||
// oneWalkAtATime refuses a retry while another walk is open, started or waiting for its word (novox/hq ADR
|
// oneWalkAtATime refuses a retry while an open walk, started or waiting for its word, is related to it (novox/hq
|
||||||
// 0276): a walk retried beside it would be two walks at once.
|
// ADR 0276, ADR 0407 decision 4, issue 973): a walk retried beside a related one would be two walks on the same
|
||||||
|
// things at once. A walk related to none may be retried beside them. The retried walk is judged by the modules it
|
||||||
|
// plans, which already hold their dependents, and the repositories it carries.
|
||||||
func oneWalkAtATime(p inventory.Plan, plans []inventory.Plan) error {
|
func oneWalkAtATime(p inventory.Plan, plans []inventory.Plan) error {
|
||||||
|
mine := footprintOfWalk(p, nil)
|
||||||
for _, q := range plans {
|
for _, q := range plans {
|
||||||
if q.ID != p.ID && q.Open() && q.Release == nil {
|
if q.ID == p.ID || !q.Open() || q.Release != nil {
|
||||||
return fmt.Errorf("%s is open (%s): one walk at a time — retry %s once it ended", q.ID, q.Named(), p.ID)
|
continue
|
||||||
|
}
|
||||||
|
if ownWalk(p) || ownWalk(q) {
|
||||||
|
return fmt.Errorf("%s is open (%s) and one of them is on the controller's own path: one walk at a time — "+
|
||||||
|
"retry %s once it ended", q.ID, q.Named(), p.ID)
|
||||||
|
}
|
||||||
|
if ok, why := related(mine, footprintOfWalk(q, nil), nil); ok {
|
||||||
|
return fmt.Errorf("%s is open (%s) and %s: one walk at a time beside a related one — retry %s once it ended",
|
||||||
|
q.ID, q.Named(), why, p.ID)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -320,7 +320,11 @@ func probeAgentRoot(ctx context.Context, d *doctor) ([]conditions.Observation, e
|
|||||||
quiet, err := searchStillRunning(ctx, inv, node, n.AgentAccount, h, had, now)
|
quiet, err := searchStillRunning(ctx, inv, node, n.AgentAccount, h, had, now)
|
||||||
return err == nil && quiet
|
return err == nil && quiet
|
||||||
}
|
}
|
||||||
return rootObservations(ctx, r, trustedMachines(r.entries), time.Now()), nil
|
obs := rootObservations(ctx, r, trustedMachines(r.entries), time.Now())
|
||||||
|
// A membership that lost its verified sender to a judgement that failed at a push gets it back here, once
|
||||||
|
// the machines pass (issue 930); the bound is one self-check.
|
||||||
|
giveVerifiedSendersBack(ctx, d)
|
||||||
|
return obs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// rootObservations judges the machines and says each that fails.
|
// rootObservations judges the machines and says each that fails.
|
||||||
@@ -365,14 +369,3 @@ func rootFreeAnswer(ctx context.Context, machines string, now time.Time) (any, e
|
|||||||
}
|
}
|
||||||
return map[string]any{"machines": judgeRootFree(ctx, newRootReader(ctx, d.open.inventory, conn), names, now)}, nil
|
return map[string]any{"machines": judgeRootFree(ctx, newRootReader(ctx, d.open.inventory, conn), names, now)}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// rootFreeNow is the machines judged root-free, for composing a push's memberships: only those that pass.
|
|
||||||
func rootFreeNow(ctx context.Context, r *rootReader, machines []string, now time.Time) map[string]bool {
|
|
||||||
free := map[string]bool{}
|
|
||||||
for _, v := range judgeRootFree(ctx, r, machines, now) {
|
|
||||||
if v.Free {
|
|
||||||
free[v.Machine] = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return free
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -183,9 +183,12 @@ func serve(ctx context.Context) (err error) {
|
|||||||
// Open plans move on a timer as well as on outcomes (novox/hq ADR 0162): a tier waiting for
|
// Open plans move on a timer as well as on outcomes (novox/hq ADR 0162): a tier waiting for
|
||||||
// machines to report moves when they have, and a plan left by a replaced controller resumes.
|
// machines to report moves when they have, and a plan left by a replaced controller resumes.
|
||||||
go planTicker(ctx, open)
|
go planTicker(ctx, open)
|
||||||
// And the merge window (novox/hq ADR 0276): a batch is cut into its walk when the window closes, which no
|
// And the merge window (novox/hq ADR 0276, ADR 0407): a batch is cut into its walk when the window closes,
|
||||||
// merge or outcome says.
|
// which no merge or outcome says. The window is two keys on the bus that expire and a watcher that hears it;
|
||||||
go batchCutter(ctx, open)
|
// nothing polls (issue 973).
|
||||||
|
if err := startMergeWindow(ctx, open, server.JetStream()); err != nil {
|
||||||
|
return fmt.Errorf("the merge window could not be put on the bus: %w", err)
|
||||||
|
}
|
||||||
// And the pending assignments settled on a tick of their own (novox/hq ADR 0261): made once their module
|
// And the pending assignments settled on a tick of their own (novox/hq ADR 0261): made once their module
|
||||||
// is registered, ended with why when its build will not register it, raised and cleared as conditions.
|
// is registered, ended with why when its build will not register it, raised and cleared as conditions.
|
||||||
// Never by a read.
|
// Never by a read.
|
||||||
@@ -198,9 +201,8 @@ func serve(ctx context.Context) (err error) {
|
|||||||
if err := server.Follows(following{open: open}); err != nil {
|
if err := server.Follows(following{open: open}); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
// And the merges the bus announced and never handed over, read back on a timer and acted on late
|
// No timer reads the merges back (novox/hq ADR 0407 decision 3, issue 973): the forge's event and the durable
|
||||||
// rather than never (novox/hq issue 266).
|
// consumer on the bus keep every merge, and one the controller cannot record yet is held and handed over again.
|
||||||
go catchingUpOnMerges(ctx, open, server)
|
|
||||||
// And a catalogue that has just started, asking for what it missed. The same type answers
|
// And a catalogue that has just started, asking for what it missed. The same type answers
|
||||||
// both: what a build meant and what the builds were are two questions about one record.
|
// both: what a build meant and what the builds were are two questions about one record.
|
||||||
if err := server.Answers(following{open: open}); err != nil {
|
if err := server.Answers(following{open: open}); err != nil {
|
||||||
@@ -1333,8 +1335,14 @@ func issueMemberships(ctx context.Context, open *stores, server *link.Server, se
|
|||||||
}
|
}
|
||||||
// Which machines are root-free now (novox/hq ADR 0259 §8): a channel's verified sender is composed for the
|
// Which machines are root-free now (novox/hq ADR 0259 §8): a channel's verified sender is composed for the
|
||||||
// router only from one, beside a router on one. Judged once per push, by the root-free verb's judgement.
|
// router only from one, beside a router on one. Judged once per push, by the root-free verb's judgement.
|
||||||
records.RootFree = rootFreeNow(ctx, newRootReader(ctx, open.inventory, bus.Conn), records.Nodes, time.Now())
|
var whys map[string]string
|
||||||
|
records.RootFree, whys = rootVerdictsOf(judgeRootFree(ctx, newRootReader(ctx, open.inventory, bus.Conn),
|
||||||
|
records.Nodes, time.Now()))
|
||||||
where := broker.PlacementsOf(records, records.Interchangeable)
|
where := broker.PlacementsOf(records, records.Interchangeable)
|
||||||
|
// What the judgement kept off is said, with its reason (novox/hq issue 930); the self-check gives it back.
|
||||||
|
for _, line := range droppedVerifiedSenders(records, where, whys) {
|
||||||
|
fmt.Println(line)
|
||||||
|
}
|
||||||
// **Every declared state's bucket, before the memberships that name it** (novox/hq ADR 0201). The
|
// **Every declared state's bucket, before the memberships that name it** (novox/hq ADR 0201). The
|
||||||
// raise at start asserts them too, but a module registered and assigned since would otherwise have
|
// raise at start asserts them too, but a module registered and assigned since would otherwise have
|
||||||
// its bucket only after the control plane next restarts — found the first time a module declared
|
// its bucket only after the control plane next restarts — found the first time a module declared
|
||||||
|
|||||||
@@ -0,0 +1,224 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"slices"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/nats-io/nats.go"
|
||||||
|
|
||||||
|
"github.com/novox/mesh-controller/internal/broker"
|
||||||
|
"github.com/novox/mesh-controller/internal/link"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A channel's verified sender is composed into a membership only while the machines are judged root-free at
|
||||||
|
// that moment (novox/hq ADR 0259 §8), and the judgement fails closed when a read fails (novox/hq issue 930):
|
||||||
|
// a push composed during a failed read issued the router a membership without it, and nothing composed it
|
||||||
|
// again, so every approval was refused until somebody pushed. Two things follow.
|
||||||
|
//
|
||||||
|
// What a composition drops it says (droppedVerifiedSenders), with the reason the judgement gave.
|
||||||
|
//
|
||||||
|
// And the self-check issues it again (reissueVerifiedSenders): each time it judges the machines, a membership
|
||||||
|
// that lacks a verified sender the composition would now give it is issued again with only its kinds changed.
|
||||||
|
// **Bound: one self-check, doctorEvery (five minutes), after the judgement passes.** It only ever adds: a
|
||||||
|
// machine that stops passing keeps what it was issued, as at a push, because the router asks the root-free
|
||||||
|
// verb again before it honours an approval, and that is the check that holds.
|
||||||
|
|
||||||
|
// droppedVerifiedSenders says, one line each, every kind that declares `verified-sender` and was composed
|
||||||
|
// without it, naming the kind, its machine and why. whys is each machine's verdict reason (rootVerdict.Why).
|
||||||
|
func droppedVerifiedSenders(r broker.Records, where broker.Placements, whys map[string]string) []string {
|
||||||
|
var routerNodes []string
|
||||||
|
for node, declared := range r.Assigned {
|
||||||
|
for _, d := range declared {
|
||||||
|
for _, s := range d.Holds {
|
||||||
|
if s.Name == routerSeat && !slices.Contains(routerNodes, node) {
|
||||||
|
routerNodes = append(routerNodes, node)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(routerNodes)
|
||||||
|
var out []string
|
||||||
|
for _, k := range where.Kinds {
|
||||||
|
declared := false
|
||||||
|
for _, d := range r.Assigned[k.Node] {
|
||||||
|
for _, s := range d.Holds {
|
||||||
|
if d.Module == k.Module && s.Name == k.Seat && s.Kind == k.Kind && slices.Contains(s.Capabilities, "verified-sender") {
|
||||||
|
declared = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !declared || slices.Contains(k.Capabilities, "verified-sender") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
var why string
|
||||||
|
switch {
|
||||||
|
case !r.RootFree[k.Node]:
|
||||||
|
why = k.Node + " is not judged root-free: " + whys[k.Node]
|
||||||
|
case len(routerNodes) != 1:
|
||||||
|
why = fmt.Sprintf("the router is placed on %d machines, not one", len(routerNodes))
|
||||||
|
case !r.RootFree[routerNodes[0]]:
|
||||||
|
why = "the router's machine " + routerNodes[0] + " is not judged root-free: " + whys[routerNodes[0]]
|
||||||
|
default:
|
||||||
|
why = k.Module + " does not run as an account of its own"
|
||||||
|
}
|
||||||
|
out = append(out, fmt.Sprintf(" %s kind %s of %s on %s is composed without its verified sender: %s",
|
||||||
|
k.Seat, k.Kind, k.Module, k.Node, why))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// withVerifiedSenders is a membership as issued with its seat traffic's kinds replaced by the composed ones,
|
||||||
|
// and whether that changed it: only when a composed kind carries a verified sender the issued one lacks. Pure.
|
||||||
|
func withVerifiedSenders(issued []byte, composed *broker.SeatTraffic) ([]byte, bool, error) {
|
||||||
|
if composed == nil {
|
||||||
|
return issued, false, nil
|
||||||
|
}
|
||||||
|
var fields map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(issued, &fields); err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
var traffic map[string]json.RawMessage
|
||||||
|
var had []broker.KindHeld
|
||||||
|
if raw, ok := fields["seat-traffic"]; ok {
|
||||||
|
if err := json.Unmarshal(raw, &traffic); err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
if raw, ok := traffic["kinds"]; ok {
|
||||||
|
if err := json.Unmarshal(raw, &had); err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
gains := false
|
||||||
|
for _, k := range composed.Kinds {
|
||||||
|
if !slices.Contains(k.Capabilities, "verified-sender") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
held := false
|
||||||
|
for _, h := range had {
|
||||||
|
if h.Seat == k.Seat && h.Kind == k.Kind && h.Module == k.Module && h.Node == k.Node &&
|
||||||
|
slices.Contains(h.Capabilities, "verified-sender") {
|
||||||
|
held = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
gains = gains || !held
|
||||||
|
}
|
||||||
|
if !gains {
|
||||||
|
return issued, false, nil
|
||||||
|
}
|
||||||
|
if traffic == nil {
|
||||||
|
traffic = map[string]json.RawMessage{}
|
||||||
|
}
|
||||||
|
raw, err := json.Marshal(composed.Kinds)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
traffic["kinds"] = raw
|
||||||
|
if fields["seat-traffic"], err = json.Marshal(traffic); err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
body, err := json.Marshal(fields)
|
||||||
|
return body, true, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// reissueVerifiedSenders issues again each membership that lacks a verified sender the records now give it:
|
||||||
|
// read as last issued, its kinds replaced, published only if no newer one was issued since (a push runs beside
|
||||||
|
// the self-check, and its newer membership must not be overwritten by the older fields read here). One never issued is left to its own send. Failures are
|
||||||
|
// said and passed over; the next self-check tries again.
|
||||||
|
func reissueVerifiedSenders(r broker.Records, where broker.Placements,
|
||||||
|
last func(node, module string) ([]byte, uint64, error), publish func(node, module string, body []byte, seq uint64) error,
|
||||||
|
say func(format string, args ...any)) int {
|
||||||
|
issued := 0
|
||||||
|
for node, declared := range r.Assigned {
|
||||||
|
for _, d := range declared {
|
||||||
|
composed := broker.MembershipFor(node, d, where).SeatTraffic
|
||||||
|
if composed == nil || !slices.ContainsFunc(composed.Kinds, func(k broker.KindHeld) bool {
|
||||||
|
return slices.Contains(k.Capabilities, "verified-sender")
|
||||||
|
}) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
was, seq, err := last(node, d.Module)
|
||||||
|
if err != nil {
|
||||||
|
if !errors.Is(err, nats.ErrMsgNotFound) {
|
||||||
|
say(" %s on %s: its membership could not be read to give its verified sender back: %v", d.Module, node, err)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
body, changed, err := withVerifiedSenders(was, composed)
|
||||||
|
if err != nil {
|
||||||
|
say(" %s on %s: its membership could not be read to give its verified sender back: %v", d.Module, node, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !changed {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := publish(node, d.Module, body, seq); err != nil {
|
||||||
|
// A push issued another since it was read: that one is newer, and the next self-check looks again.
|
||||||
|
if strings.Contains(err.Error(), "wrong last sequence") {
|
||||||
|
say(" %s on %s: a newer membership was issued meanwhile; its verified sender is looked at again next time",
|
||||||
|
d.Module, node)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
say(" %s on %s: its verified sender could not be issued again: %v", d.Module, node, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
say(" %s on %s: issued its membership again, with the verified sender its machines now pass for", d.Module, node)
|
||||||
|
issued++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return issued
|
||||||
|
}
|
||||||
|
|
||||||
|
// rootVerdictsOf is each machine's verdict reason, for saying why one was not free.
|
||||||
|
func rootVerdictsOf(verdicts []rootVerdict) (free map[string]bool, whys map[string]string) {
|
||||||
|
free, whys = map[string]bool{}, map[string]string{}
|
||||||
|
for _, v := range verdicts {
|
||||||
|
if v.Free {
|
||||||
|
free[v.Machine] = true
|
||||||
|
}
|
||||||
|
whys[v.Machine] = strings.TrimSpace(v.Why)
|
||||||
|
}
|
||||||
|
return free, whys
|
||||||
|
}
|
||||||
|
|
||||||
|
// giveVerifiedSendersBack is the self-check's step: every machine judged, the memberships that lost a verified
|
||||||
|
// sender to a failed judgement issued again. Never fails the probe: what could not be done is said.
|
||||||
|
func giveVerifiedSendersBack(ctx context.Context, d *doctor) {
|
||||||
|
if d.js == nil || d.open == nil || d.open.inventory == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
records, err := d.open.inventory.BusRecords(ctx)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Printf(" the memberships could not be read to give a verified sender back: %v\n", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Judged for every machine the records name, as a push does, not only those the self-check judged.
|
||||||
|
r := newRootReader(ctx, d.open.inventory, d.js.Conn())
|
||||||
|
all := judgeRootFree(ctx, r, records.Nodes, time.Now())
|
||||||
|
records.RootFree, _ = rootVerdictsOf(all)
|
||||||
|
where := broker.PlacementsOf(records, records.Interchangeable)
|
||||||
|
reissueVerifiedSendersOn(ctx, link.OverNATS{Conn: d.js.Conn(), JS: d.js.Context()}, records, where)
|
||||||
|
}
|
||||||
|
|
||||||
|
// reissueVerifiedSendersOn is reissueVerifiedSenders on the bus: memberships read from the AssignmentsStream and
|
||||||
|
// published to it.
|
||||||
|
func reissueVerifiedSendersOn(ctx context.Context, bus link.OverNATS, records broker.Records, where broker.Placements) int {
|
||||||
|
return reissueVerifiedSenders(records, where,
|
||||||
|
func(node, module string) ([]byte, uint64, error) {
|
||||||
|
m, err := bus.JS.GetLastMsg(broker.AssignmentsStream, broker.MembershipSubject(node, module))
|
||||||
|
if err != nil {
|
||||||
|
return nil, 0, err
|
||||||
|
}
|
||||||
|
return m.Data, m.Sequence, nil
|
||||||
|
},
|
||||||
|
func(node, module string, body []byte, seq uint64) error {
|
||||||
|
return bus.PublishMembershipAfter(ctx, node, module, body, seq)
|
||||||
|
},
|
||||||
|
func(format string, args ...any) { fmt.Printf(format+"\n", args...) })
|
||||||
|
}
|
||||||
@@ -0,0 +1,187 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/nats-io/nats.go"
|
||||||
|
|
||||||
|
"github.com/novox/mesh-controller/internal/broker"
|
||||||
|
"github.com/novox/mesh-controller/internal/link"
|
||||||
|
"github.com/novox/mesh-controller/internal/testbus"
|
||||||
|
)
|
||||||
|
|
||||||
|
func verifiedSenderRecords(free map[string]bool) (broker.Records, broker.Declared) {
|
||||||
|
tg := broker.Seat{Name: "channel", Scope: "mesh", Accepts: []string{"show"}, Kinded: true, Kind: "telegram",
|
||||||
|
DeclaredBy: "messenger", Capabilities: []string{"choice", "verified-sender"}}
|
||||||
|
router := broker.Declared{Module: "messenger", RunsAs: "messenger", Holds: []broker.Seat{{Name: "operator-channel",
|
||||||
|
Scope: "mesh", Accepts: []string{"ask"}}}, Uses: []broker.Seat{{Name: "channel", Scope: "mesh", Kinded: true,
|
||||||
|
DeclaredBy: "messenger"}}}
|
||||||
|
return broker.Records{Nodes: []string{"novox"}, RootFree: free, Assigned: map[string][]broker.Declared{
|
||||||
|
"novox": {router, {Module: "telegram", RunsAs: "telegram", Holds: []broker.Seat{tg}}}}}, router
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeBus struct {
|
||||||
|
last map[string][]byte
|
||||||
|
published map[string][]byte
|
||||||
|
seq map[string]uint64
|
||||||
|
// between runs once, after a read and before the write: a push issuing meanwhile.
|
||||||
|
between func()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeBus) get(node, module string) ([]byte, uint64, error) {
|
||||||
|
if b, ok := f.last[node+"/"+module]; ok {
|
||||||
|
return b, f.seq[node+"/"+module], nil
|
||||||
|
}
|
||||||
|
return nil, 0, nats.ErrMsgNotFound
|
||||||
|
}
|
||||||
|
func (f *fakeBus) put(node, module string, body []byte, seq uint64) error {
|
||||||
|
if f.between != nil {
|
||||||
|
f.between()
|
||||||
|
f.between = nil
|
||||||
|
}
|
||||||
|
if seq != f.seq[node+"/"+module] {
|
||||||
|
return errors.New("nats: wrong last sequence")
|
||||||
|
}
|
||||||
|
f.published[node+"/"+module] = body
|
||||||
|
f.last[node+"/"+module] = body
|
||||||
|
f.seq[node+"/"+module]++
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasVerified(t *testing.T, membership []byte) bool {
|
||||||
|
t.Helper()
|
||||||
|
var m broker.Membership
|
||||||
|
if err := json.Unmarshal(membership, &m); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, k := range m.SeatTraffic.Kinds {
|
||||||
|
if k.Kind == "telegram" {
|
||||||
|
return strings.Contains(strings.Join(k.Capabilities, ","), "verified-sender")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.Fatal("telegram not in the router's membership")
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// A membership composed while the judgement failed is issued again once it passes, with the verified sender
|
||||||
|
// (novox/hq issue 930); while it fails, nothing is given back (ADR 0259 §8).
|
||||||
|
func TestAMembershipThatLostItsVerifiedSenderGetsItBackWhenTheJudgementPasses(t *testing.T) {
|
||||||
|
failed, router := verifiedSenderRecords(nil)
|
||||||
|
whereFailed := broker.PlacementsOf(failed, nil)
|
||||||
|
bus := &fakeBus{last: map[string][]byte{}, published: map[string][]byte{}, seq: map[string]uint64{}}
|
||||||
|
body, _ := json.Marshal(broker.MembershipFor("novox", router, whereFailed))
|
||||||
|
bus.last["novox/messenger"] = body
|
||||||
|
if hasVerified(t, body) {
|
||||||
|
t.Fatal("composed with a failed judgement, and the verified sender is kept")
|
||||||
|
}
|
||||||
|
say := func(string, ...any) {}
|
||||||
|
|
||||||
|
if n := reissueVerifiedSenders(failed, whereFailed, bus.get, bus.put, say); n != 0 || len(bus.published) != 0 {
|
||||||
|
t.Fatalf("issued %d while the judgement still fails", n)
|
||||||
|
}
|
||||||
|
passing, _ := verifiedSenderRecords(map[string]bool{"novox": true})
|
||||||
|
wherePassing := broker.PlacementsOf(passing, nil)
|
||||||
|
if n := reissueVerifiedSenders(passing, wherePassing, bus.get, bus.put, say); n != 1 {
|
||||||
|
t.Fatalf("issued %d memberships after the judgement passed, want 1", n)
|
||||||
|
}
|
||||||
|
if !hasVerified(t, bus.published["novox/messenger"]) {
|
||||||
|
t.Fatal("the membership issued again lacks the verified sender")
|
||||||
|
}
|
||||||
|
if n := reissueVerifiedSenders(passing, wherePassing, bus.get, bus.put, say); n != 0 {
|
||||||
|
t.Fatalf("issued %d again with nothing changed", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// What a composition drops it says, naming the kind, the machine and the judgement's reason.
|
||||||
|
func TestADroppedVerifiedSenderIsSaidWithItsReason(t *testing.T) {
|
||||||
|
records, _ := verifiedSenderRecords(nil)
|
||||||
|
where := broker.PlacementsOf(records, nil)
|
||||||
|
lines := droppedVerifiedSenders(records, where, map[string]string{"novox": "not measured: no verb heard"})
|
||||||
|
if len(lines) != 1 || !strings.Contains(lines[0], "telegram") || !strings.Contains(lines[0], "novox") ||
|
||||||
|
!strings.Contains(lines[0], "not measured: no verb heard") {
|
||||||
|
t.Fatalf("said %q", lines)
|
||||||
|
}
|
||||||
|
records, _ = verifiedSenderRecords(map[string]bool{"novox": true})
|
||||||
|
if lines := droppedVerifiedSenders(records, broker.PlacementsOf(records, nil), nil); len(lines) != 0 {
|
||||||
|
t.Fatalf("said %q of a kind that kept its verified sender", lines)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAMembershipNeverIssuedIsLeftToItsOwnSend(t *testing.T) {
|
||||||
|
records, _ := verifiedSenderRecords(map[string]bool{"novox": true})
|
||||||
|
bus := &fakeBus{last: map[string][]byte{}, published: map[string][]byte{}, seq: map[string]uint64{}}
|
||||||
|
var said []string
|
||||||
|
n := reissueVerifiedSenders(records, broker.PlacementsOf(records, nil), bus.get, bus.put,
|
||||||
|
func(f string, a ...any) { said = append(said, f) })
|
||||||
|
if n != 0 || len(said) != 0 {
|
||||||
|
t.Fatalf("issued %d, said %v", n, said)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// **On a real bus** (novox/hq issue 930): a router's membership issued without the verified sender because the
|
||||||
|
// judgement failed is, once the judgement passes, published again to the AssignmentsStream with it in its seat
|
||||||
|
// traffic's kinds, and a second pass publishes nothing.
|
||||||
|
func TestNatsAMembershipIssuedWithoutItsVerifiedSenderIsIssuedAgainWhenTheJudgementPasses(t *testing.T) {
|
||||||
|
js, err := broker.Dial(testbus.URL(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(js.Close)
|
||||||
|
if err := broker.AssertMeshStreams(js); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
bus := link.OverNATS{Conn: js.Conn(), JS: js.Context()}
|
||||||
|
|
||||||
|
failed, router := verifiedSenderRecords(nil)
|
||||||
|
body, err := json.Marshal(broker.MembershipFor("novox", router, broker.PlacementsOf(failed, nil)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := bus.PublishMembership(t.Context(), "novox", "messenger", body); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
issued := func() []byte {
|
||||||
|
m, err := js.Context().GetLastMsg(broker.AssignmentsStream, broker.MembershipSubject("novox", "messenger"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return m.Data
|
||||||
|
}
|
||||||
|
if hasVerified(t, issued()) {
|
||||||
|
t.Fatal("issued with a failed judgement, and the verified sender is kept: the test proves nothing")
|
||||||
|
}
|
||||||
|
if n := reissueVerifiedSendersOn(t.Context(), bus, failed, broker.PlacementsOf(failed, nil)); n != 0 {
|
||||||
|
t.Fatalf("issued %d memberships while the judgement still fails", n)
|
||||||
|
}
|
||||||
|
passing, _ := verifiedSenderRecords(map[string]bool{"novox": true})
|
||||||
|
where := broker.PlacementsOf(passing, nil)
|
||||||
|
if n := reissueVerifiedSendersOn(t.Context(), bus, passing, where); n != 1 {
|
||||||
|
t.Fatalf("issued %d memberships after the judgement passed, want 1", n)
|
||||||
|
}
|
||||||
|
if !hasVerified(t, issued()) {
|
||||||
|
t.Fatal("the membership on the bus lacks the verified sender after the judgement passed")
|
||||||
|
}
|
||||||
|
if n := reissueVerifiedSendersOn(t.Context(), bus, passing, where); n != 0 {
|
||||||
|
t.Fatalf("a second pass issued %d memberships", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A push issuing a newer membership between the self-check's read and its write is not overwritten (issue 930).
|
||||||
|
func TestANewerMembershipIssuedMeanwhileIsNotOverwritten(t *testing.T) {
|
||||||
|
failed, router := verifiedSenderRecords(nil)
|
||||||
|
bus := &fakeBus{last: map[string][]byte{}, published: map[string][]byte{}, seq: map[string]uint64{}}
|
||||||
|
old, _ := json.Marshal(broker.MembershipFor("novox", router, broker.PlacementsOf(failed, nil)))
|
||||||
|
bus.last["novox/messenger"], bus.seq["novox/messenger"] = old, 1
|
||||||
|
newer := []byte(`{"node":"novox","module":"messenger","newer":true}`)
|
||||||
|
bus.between = func() { bus.last["novox/messenger"], bus.seq["novox/messenger"] = newer, 2 }
|
||||||
|
passing, _ := verifiedSenderRecords(map[string]bool{"novox": true})
|
||||||
|
if n := reissueVerifiedSenders(passing, broker.PlacementsOf(passing, nil), bus.get, bus.put, func(string, ...any) {}); n != 0 {
|
||||||
|
t.Fatalf("issued %d over a newer membership", n)
|
||||||
|
}
|
||||||
|
if string(bus.last["novox/messenger"]) != string(newer) || len(bus.published) != 0 {
|
||||||
|
t.Fatalf("the newer membership was overwritten: %s", bus.last["novox/messenger"])
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,8 +1,10 @@
|
|||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/novox/mesh-controller/internal/inventory"
|
"github.com/novox/mesh-controller/internal/inventory"
|
||||||
)
|
)
|
||||||
@@ -11,7 +13,9 @@ import (
|
|||||||
// related when they share a repository, a module, a delivery group's head, or an edge of the module graph
|
// related when they share a repository, a module, a delivery group's head, or an edge of the module graph
|
||||||
// between their modules. A module is every one the walk builds or sends: the modules its merges moved and the
|
// between their modules. A module is every one the walk builds or sends: the modules its merges moved and the
|
||||||
// modules that depend on them, not only the ones the merges touched. Unrelated changes may walk beside each
|
// modules that depend on them, not only the ones the merges touched. Unrelated changes may walk beside each
|
||||||
// other (ADR 0304 decision 2, ADR 0282 decision 3); nothing walks so yet, and nothing but tests reads this.
|
// other (ADR 0304 decision 2, ADR 0282 decision 3); a closed batch is cut beside an open walk it is
|
||||||
|
// not related to, and a walk about to ask its first tier waits only for a started walk it is related to (issue 973,
|
||||||
|
// ADR 0407 decision 4).
|
||||||
|
|
||||||
// footprint is what one change reaches: its repositories (lower case), its modules, and its delivery group
|
// footprint is what one change reaches: its repositories (lower case), its modules, and its delivery group
|
||||||
// heads (the head branch names of its merges). unknown says a merge of it was kept without the modules it
|
// heads (the head branch names of its merges). unknown says a merge of it was kept without the modules it
|
||||||
@@ -109,3 +113,83 @@ func firstShared(a, b map[string]bool) string {
|
|||||||
sort.Strings(both)
|
sort.Strings(both)
|
||||||
return both[0]
|
return both[0]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// openWalk is a walk open beside a change: its id, what it reaches, and whether it is on the controller's own path.
|
||||||
|
type openWalk struct {
|
||||||
|
id string
|
||||||
|
fp footprint
|
||||||
|
own bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// relatedOpen is the first open walk a change is related to, with the reason; empty when it is related to none
|
||||||
|
// and may walk beside them all (issue 973, ADR 0407 decision 4, ADR 0306 decision 3).
|
||||||
|
//
|
||||||
|
// **The controller's own path is related to everything** (decided for issue 973): a walk or batch that replaces
|
||||||
|
// the controller, the node-engine, the tool runner, the bus or mesh-delivery never runs beside another, and
|
||||||
|
// nothing runs beside it, whatever the footprints say.
|
||||||
|
func relatedOpen(cand footprint, candOwn bool, open []openWalk, edges []inventory.Edge) (behind, why string) {
|
||||||
|
for _, w := range open {
|
||||||
|
if candOwn {
|
||||||
|
return w.id, "it is on the controller's own path"
|
||||||
|
}
|
||||||
|
if w.own {
|
||||||
|
return w.id, w.id + " is on the controller's own path"
|
||||||
|
}
|
||||||
|
if ok, why := related(cand, w.fp, edges); ok {
|
||||||
|
return w.id, why
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// ownWalk says a walk is on the controller's own path: its batch says so, or it plans a module of that path
|
||||||
|
// (issue 973).
|
||||||
|
func ownWalk(p inventory.Plan) bool {
|
||||||
|
if p.OwnPath() {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
for name := range p.Modules {
|
||||||
|
if _, own := onTheControllersPath[name]; own {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// startedWalks are the open walks started beside the walk id, made at created: a merge's walk past its wait, not
|
||||||
|
// the backlog's, with a tier asked, or on its way (let go, or waiting for nobody) before this one.
|
||||||
|
func startedWalks(plans []inventory.Plan, id string, created time.Time) []inventory.Plan {
|
||||||
|
var out []inventory.Plan
|
||||||
|
for _, q := range plans {
|
||||||
|
if q.ID == id || q.Release != nil || q.Waiting() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if q.Tier > 0 || askedAny(q) || q.Created.Before(created) {
|
||||||
|
out = append(out, q)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// openWalkOf is an open walk as a change is judged against it: what it reaches, from its planned modules and the
|
||||||
|
// merges the record holds for it.
|
||||||
|
func openWalkOf(ctx context.Context, inv *inventory.Inventory, p inventory.Plan) (openWalk, error) {
|
||||||
|
kept, err := inv.MergesOf(ctx, p.ID)
|
||||||
|
if err != nil {
|
||||||
|
return openWalk{}, err
|
||||||
|
}
|
||||||
|
return openWalk{id: p.ID, fp: footprintOfWalk(p, kept), own: ownWalk(p)}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// openWalksOf are openWalkOf for each of plans.
|
||||||
|
func openWalksOf(ctx context.Context, inv *inventory.Inventory, plans []inventory.Plan) ([]openWalk, error) {
|
||||||
|
var out []openWalk
|
||||||
|
for _, p := range plans {
|
||||||
|
w, err := openWalkOf(ctx, inv, p)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, w)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,8 +2,11 @@ package main
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/novox/mesh-controller/internal/inventory"
|
"github.com/novox/mesh-controller/internal/inventory"
|
||||||
"github.com/novox/mesh-controller/internal/link"
|
"github.com/novox/mesh-controller/internal/link"
|
||||||
@@ -108,3 +111,125 @@ func TestABatchWhoseMovesAreNotKnownIsRelatedToEverything(t *testing.T) {
|
|||||||
t.Fatalf("a known, unrelated batch read as related: %q", why)
|
t.Fatalf("a known, unrelated batch read as related: %q", why)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRelatedOpenNamesTheFirstOpenWalkAChangeIsRelatedTo: a change that shares nothing with any open walk is
|
||||||
|
// held behind none (issue 973, ADR 0407 decision 4); one that shares a repository, a module, a graph edge or a
|
||||||
|
// group head is held behind the first walk it shares it with, with the reason; one whose moves are not known
|
||||||
|
// is held behind the first; a walk or change on the controller's own path is related to every other.
|
||||||
|
func TestRelatedOpenNamesTheFirstOpenWalkAChangeIsRelatedTo(t *testing.T) {
|
||||||
|
edges := []inventory.Edge{{From: "nzbget", To: "base-image", Kind: inventory.EdgeStandsOn}}
|
||||||
|
walk := func(id, repo string, own bool, modules ...string) openWalk {
|
||||||
|
return openWalk{id: id, fp: footprintOfWalk(walkOf(repo, modules...), nil), own: own}
|
||||||
|
}
|
||||||
|
one, two := walk("plan-1", "novox/one", false, "notes"), walk("plan-2", "novox/two", false, "nzbget")
|
||||||
|
cand := func(repo, head string, moves ...string) footprint {
|
||||||
|
return footprintOfBatch([]inventory.BatchedMerge{heardMerge(repo, head, moves...)}, edges)
|
||||||
|
}
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
fp footprint
|
||||||
|
own bool
|
||||||
|
open []openWalk
|
||||||
|
behind string
|
||||||
|
why string
|
||||||
|
}{
|
||||||
|
{"nothing open", cand("novox/x", "fix/a", "app"), false, nil, "", ""},
|
||||||
|
{"unrelated to every open walk", cand("novox/x", "fix/a", "app"), false, []openWalk{one, two}, "", ""},
|
||||||
|
{"a repository", cand("novox/ONE", "fix/a", "app"), false, []openWalk{two, one}, "plan-1", "both change novox/one"},
|
||||||
|
{"a module through its dependents", cand("novox/x", "fix/a", "base-image"), false, []openWalk{one, two},
|
||||||
|
"plan-2", "both move nzbget"},
|
||||||
|
{"a graph edge", cand("novox/x", "fix/a", "nzbget"),
|
||||||
|
false, []openWalk{one, walk("plan-3", "novox/y", false, "base-image")}, "plan-3",
|
||||||
|
"nzbget depends on base-image (stands-on)"},
|
||||||
|
{"moves not known", cand("novox/x", "fix/a"), false, []openWalk{one, two}, "plan-1",
|
||||||
|
"the modules a merge of novox/x moves are not known until its batch is cut"},
|
||||||
|
{"the candidate on the own path", cand("novox/x", "fix/a", "app"), true, []openWalk{one}, "plan-1",
|
||||||
|
"it is on the controller's own path"},
|
||||||
|
{"an open walk on the own path", cand("novox/x", "fix/a", "app"), false,
|
||||||
|
[]openWalk{one, walk("plan-4", "novox/mesh-controller", true, "mesh-controller")}, "plan-4",
|
||||||
|
"plan-4 is on the controller's own path"},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
behind, why := relatedOpen(c.fp, c.own, c.open, edges)
|
||||||
|
if behind != c.behind || why != c.why {
|
||||||
|
t.Errorf("relatedOpen = %q, %q; want %q, %q", behind, why, c.behind, c.why)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A walk is on the controller's own path when its batch says so or it plans a module of that path (issue 973).
|
||||||
|
func TestAWalkOnTheControllersPathIsKnownByItsFlagOrItsModules(t *testing.T) {
|
||||||
|
if ownWalk(walkOf("novox/one", "notes")) {
|
||||||
|
t.Error("an ordinary walk read as on the controller's own path")
|
||||||
|
}
|
||||||
|
flagged := walkOf("novox/one", "notes")
|
||||||
|
flagged.Delivery = &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: true}}
|
||||||
|
if !ownWalk(flagged) {
|
||||||
|
t.Error("a walk whose batch is own-path read as ordinary")
|
||||||
|
}
|
||||||
|
var module string
|
||||||
|
for m := range onTheControllersPath {
|
||||||
|
module = m
|
||||||
|
}
|
||||||
|
if !ownWalk(walkOf("novox/one", "notes", module)) {
|
||||||
|
t.Errorf("a walk planning %s read as ordinary", module)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A walk about to ask its first tier waits only for a started walk it is related to (issue 973): started is a
|
||||||
|
// tier asked, or a walk made before it; a waiting walk, the backlog's and itself are not.
|
||||||
|
func TestWalksStartedBesideAreTheOpenOnesBeforeIt(t *testing.T) {
|
||||||
|
at := time.Date(2026, 10, 17, 12, 0, 0, 0, time.UTC)
|
||||||
|
mk := func(id string, created time.Time, tier int) inventory.Plan {
|
||||||
|
p := walkOf("novox/"+id, id)
|
||||||
|
p.ID, p.Created, p.Tier, p.State = id, created, tier, inventory.PlanBuilding
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
older, newer, ticking := mk("older", at, 0), mk("newer", at.Add(time.Minute), 0), mk("ticking", at.Add(2*time.Minute), 1)
|
||||||
|
backlog := mk("backlog", at, 1)
|
||||||
|
backlog.Release = &inventory.PlanRelease{}
|
||||||
|
got := startedWalks([]inventory.Plan{older, newer, ticking, backlog, mk("me", at.Add(time.Minute), 0)}, "me", at.Add(time.Minute))
|
||||||
|
var ids []string
|
||||||
|
for _, p := range got {
|
||||||
|
ids = append(ids, p.ID)
|
||||||
|
}
|
||||||
|
if strings.Join(ids, ",") != "older,ticking" {
|
||||||
|
t.Errorf("started beside me = %v; want older, ticking", ids)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A retry is refused beside an open walk it is related to and allowed beside one it shares nothing with
|
||||||
|
// (issue 973); a retry on the controller's own path is refused beside any open walk.
|
||||||
|
func TestARetryWaitsOnlyForAnOpenWalkItIsRelatedTo(t *testing.T) {
|
||||||
|
failed := walkOf("novox/one", "notes")
|
||||||
|
failed.ID, failed.State = "plan-1", inventory.PlanFailed
|
||||||
|
open := func(id, repo string, modules ...string) inventory.Plan {
|
||||||
|
p := walkOf(repo, modules...)
|
||||||
|
p.ID, p.State = id, inventory.PlanBuilding
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
if err := oneWalkAtATime(failed, []inventory.Plan{open("plan-2", "novox/two", "app")}); err != nil {
|
||||||
|
t.Errorf("refused beside an unrelated walk: %v", err)
|
||||||
|
}
|
||||||
|
err := oneWalkAtATime(failed, []inventory.Plan{open("plan-3", "novox/three", "notes")})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "plan-3") || !strings.Contains(err.Error(), "both move notes") {
|
||||||
|
t.Errorf("a retry beside a walk moving the same module: %v", err)
|
||||||
|
}
|
||||||
|
failed.Delivery = &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: true}}
|
||||||
|
if err := oneWalkAtATime(failed, []inventory.Plan{open("plan-2", "novox/two", "app")}); err == nil {
|
||||||
|
t.Error("an own-path retry went beside an open walk")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Plans another controller is working are "not now", not a failure: the merge is held and handed over again
|
||||||
|
// (issue 921).
|
||||||
|
func TestPlansHeldByAnotherControllerAreNotNow(t *testing.T) {
|
||||||
|
if err := notNow(fmt.Errorf("x: %w", inventory.ErrPlansBusy)); !errors.Is(err, link.ErrTryAgain) {
|
||||||
|
t.Errorf("busy plans read as %v; want ErrTryAgain", err)
|
||||||
|
}
|
||||||
|
if err := notNow(errors.New("no such module")); errors.Is(err, link.ErrTryAgain) {
|
||||||
|
t.Errorf("another failure read as ErrTryAgain: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -618,23 +618,32 @@ func planBuiltSaying(ctx context.Context, open *stores, module, commit, failed s
|
|||||||
advanceHeld(ctx, open)
|
advanceHeld(ctx, open)
|
||||||
}
|
}
|
||||||
|
|
||||||
// startedBeside is the id of a started walk open beside this one — a merge's walk past its wait, not the
|
// startedBeside is the id of a started walk open beside this one that it is related to — a merge's walk past
|
||||||
// backlog's — or empty: one walk at a time (novox/hq ADR 0276).
|
// its wait, not the backlog's — with why, or empty: a walk waits only for a started walk it shares something with
|
||||||
func startedBeside(ctx context.Context, inv *inventory.Inventory, id string, created time.Time) (string, error) {
|
// (novox/hq ADR 0276 decision 4, ADR 0407 decision 4, issue 973).
|
||||||
|
func startedBeside(ctx context.Context, inv *inventory.Inventory, p inventory.Plan) (behind, why string, err error) {
|
||||||
plans, err := inv.OpenPlans(ctx)
|
plans, err := inv.OpenPlans(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", "", err
|
||||||
}
|
}
|
||||||
for _, q := range plans {
|
started := startedWalks(plans, p.ID, p.Created)
|
||||||
if q.ID == id || q.Release != nil || q.Waiting() {
|
if len(started) == 0 {
|
||||||
continue
|
return "", "", nil
|
||||||
}
|
}
|
||||||
// Started: a tier asked, or on its way (let go, or waiting for nobody) before this one.
|
edges, err := inv.Dependencies(ctx)
|
||||||
if q.Tier > 0 || askedAny(q) || q.Created.Before(created) {
|
if err != nil {
|
||||||
return q.ID, nil
|
return "", "", err
|
||||||
}
|
}
|
||||||
|
open, err := openWalksOf(ctx, inv, started)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", err
|
||||||
}
|
}
|
||||||
return "", nil
|
mine, err := openWalkOf(ctx, inv, p)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
behind, why = relatedOpen(mine.fp, mine.own, open, edges)
|
||||||
|
return behind, why, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// deferredNote begins the note of a walk deferred behind another.
|
// deferredNote begins the note of a walk deferred behind another.
|
||||||
@@ -693,7 +702,7 @@ func keepWalkPhases(ctx context.Context, open *stores) {
|
|||||||
// advanceHeld is advancePlans for a caller already holding the plans.
|
// advanceHeld is advancePlans for a caller already holding the plans.
|
||||||
func advanceHeld(ctx context.Context, open *stores) {
|
func advanceHeld(ctx context.Context, open *stores) {
|
||||||
inv := open.inventory
|
inv := open.inventory
|
||||||
// A walk that ended lets the next batch be cut at once, not at the cutter's next look (novox/hq ADR 0276).
|
// A walk that ended lets the next batch be cut at once, not at a window's next expiry (novox/hq ADR 0276, ADR 0407).
|
||||||
defer func() {
|
defer func() {
|
||||||
if err := cutBatchesHeld(ctx, open, time.Now().UTC()); err != nil {
|
if err := cutBatchesHeld(ctx, open, time.Now().UTC()); err != nil {
|
||||||
fmt.Printf("batches: %v\n", err)
|
fmt.Printf("batches: %v\n", err)
|
||||||
@@ -793,13 +802,14 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
|
|||||||
tier := p.Tiers[p.Tier]
|
tier := p.Tiers[p.Tier]
|
||||||
// Not yet asked: ask — the whole tier, or what of it is still unasked (novox/hq issue 754).
|
// Not yet asked: ask — the whole tier, or what of it is still unasked (novox/hq issue 754).
|
||||||
if unasked := unaskedOf(*p); len(unasked) > 0 {
|
if unasked := unaskedOf(*p); len(unasked) > 0 {
|
||||||
// **One walk at a time** (novox/hq ADR 0276): a walk about to ask its first tier while another started
|
// **One walk at a time beside a related one** (novox/hq ADR 0276, ADR 0407 decision 4, issue 973): a walk
|
||||||
// walk is open waits for that one to end, let go or not, and says so.
|
// about to ask its first tier while a started walk it is related to is open waits for that one to end,
|
||||||
|
// let go or not, and says so and why. A started walk it shares nothing with does not hold it.
|
||||||
if p.Tier == 0 && !askedInTier(*p) {
|
if p.Tier == 0 && !askedInTier(*p) {
|
||||||
if behind, err := startedBeside(ctx, inv, p.ID, p.Created); err != nil {
|
if behind, why, err := startedBeside(ctx, inv, *p); err != nil {
|
||||||
return false, err
|
return false, err
|
||||||
} else if behind != "" {
|
} else if behind != "" {
|
||||||
note := fmt.Sprintf("%s%s ended — one walk at a time", deferredNote, behind)
|
note := fmt.Sprintf("%s%s ended — %s", deferredNote, behind, why)
|
||||||
changed := p.Note != note
|
changed := p.Note != note
|
||||||
p.Note = note
|
p.Note = note
|
||||||
return changed, nil
|
return changed, nil
|
||||||
@@ -1722,7 +1732,7 @@ func planLineSaid(p inventory.Plan, now time.Time, pause pauseView, bound time.D
|
|||||||
return fmt.Sprintf("%s · %s, %s, for %s", name, where, waitingSaid(p, waits[p.ID]), since)
|
return fmt.Sprintf("%s · %s, %s, for %s", name, where, waitingSaid(p, waits[p.ID]), since)
|
||||||
}
|
}
|
||||||
if deferred(p) {
|
if deferred(p) {
|
||||||
// Nor is waiting for the walk before it to end (one walk at a time, novox/hq ADR 0276).
|
// Nor is waiting for the walk before it to end (a related walk, novox/hq ADR 0276, ADR 0407).
|
||||||
return fmt.Sprintf("%s · %s, %s, for %s", name, where, p.Note, since)
|
return fmt.Sprintf("%s · %s, %s, for %s", name, where, p.Note, since)
|
||||||
}
|
}
|
||||||
if waiting, paused := pausedWaiting(p, pause, now); paused {
|
if waiting, paused := pausedWaiting(p, pause, now); paused {
|
||||||
|
|||||||
@@ -126,11 +126,6 @@ var signalsTable = []signalRow{
|
|||||||
Kind: "controller-deaf", Severity: conditions.Urgent, Phase: 1,
|
Kind: "controller-deaf", Severity: conditions.Urgent, Phase: 1,
|
||||||
needs: func(f *signalFacts) error { return f.loopErr }, watch: watchLoop,
|
needs: func(f *signalFacts) error { return f.loopErr }, watch: watchLoop,
|
||||||
newest: func(f *signalFacts) time.Time { return f.loop.took }},
|
newest: func(f *signalFacts) time.Time { return f.loop.took }},
|
||||||
{Row: "S5", Signal: "a merge announced becomes a plan, or nothing reads it", Emitter: "announcer → controller",
|
|
||||||
Trigger: "each merge", Bound: "10 min (the catch-up pass of issue 266)",
|
|
||||||
Kind: "merge-not-acted", Severity: conditions.Urgent, Phase: 1,
|
|
||||||
needs: func(f *signalFacts) error { return f.mergesErr }, watch: watchMerges,
|
|
||||||
newest: func(f *signalFacts) time.Time { return f.mergesPassed }},
|
|
||||||
{Row: "S6", Signal: "a build asked → its outcome", Emitter: "build seat", Trigger: "each ask",
|
{Row: "S6", Signal: "a build asked → its outcome", Emitter: "build seat", Trigger: "each ask",
|
||||||
Bound: "max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, provisional — the " +
|
Bound: "max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, provisional — the " +
|
||||||
"build seat declares no timeout; and an ask the queue gave up on (dead) at once",
|
"build seat declares no timeout; and an ask the queue gave up on (dead) at once",
|
||||||
@@ -459,18 +454,6 @@ func watchLoop(f *signalFacts) []conditions.Observation {
|
|||||||
Said: fmt.Sprintf("%d pending (%s), last taken %s", f.loop.pending, f.loop.where, since.UTC().Format(time.RFC3339))}}
|
Said: fmt.Sprintf("%d pending (%s), last taken %s", f.loop.pending, f.loop.where, since.UTC().Format(time.RFC3339))}}
|
||||||
}
|
}
|
||||||
|
|
||||||
func watchMerges(f *signalFacts) []conditions.Observation {
|
|
||||||
var out []conditions.Observation
|
|
||||||
for _, m := range f.merges {
|
|
||||||
out = append(out, conditions.Observation{Scope: conditions.ScopeMerge, ID: m.Repo + "." + short(m.Commit),
|
|
||||||
Kind: "merge-not-acted", Token: "not-acted", Severity: conditions.Urgent,
|
|
||||||
Summary: fmt.Sprintf("%s/%s merged into %s (%s) was never handed to the controller by the bus; "+
|
|
||||||
"acted on late by the catch-up", m.Owner, m.Repo, m.Base, short(m.Commit)),
|
|
||||||
Said: fmt.Sprintf("announced %s, %s behind it", m.At.UTC().Format(time.RFC3339), readableList(m.Modules))})
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
func watchAsks(f *signalFacts) []conditions.Observation {
|
func watchAsks(f *signalFacts) []conditions.Observation {
|
||||||
var out []conditions.Observation
|
var out []conditions.Observation
|
||||||
for _, a := range f.asks {
|
for _, a := range f.asks {
|
||||||
@@ -686,7 +669,7 @@ func watchDeadLetters(f *signalFacts) []conditions.Observation {
|
|||||||
// controller, or a seat's holders.
|
// controller, or a seat's holders.
|
||||||
func consumerWho(stream, consumer string) string {
|
func consumerWho(stream, consumer string) string {
|
||||||
switch {
|
switch {
|
||||||
case consumer == broker.ControllerName:
|
case consumer == broker.ControllerName || consumer == broker.MergesConsumerName:
|
||||||
return "the controller"
|
return "the controller"
|
||||||
case strings.HasPrefix(stream, "SEAT_") && strings.HasSuffix(consumer, "_worker"):
|
case strings.HasPrefix(stream, "SEAT_") && strings.HasSuffix(consumer, "_worker"):
|
||||||
seat := strings.ToLower(strings.ReplaceAll(strings.TrimSuffix(strings.TrimPrefix(consumer, "SEAT_"), "_worker"), "_", "-"))
|
seat := strings.ToLower(strings.ReplaceAll(strings.TrimSuffix(strings.TrimPrefix(consumer, "SEAT_"), "_worker"), "_", "-"))
|
||||||
|
|||||||
@@ -33,7 +33,6 @@ func calm(now time.Time) *signalFacts {
|
|||||||
plans: []planFacts{{id: "plan-1", repository: "novox/app", commit: "c0ffee00", tier: 0, tiers: 2,
|
plans: []planFacts{{id: "plan-1", repository: "novox/app", commit: "c0ffee00", tier: 0, tiers: 2,
|
||||||
entered: now.Add(-time.Minute), bound: 30 * time.Minute, waiting: "building"}},
|
entered: now.Add(-time.Minute), bound: 30 * time.Minute, waiting: "building"}},
|
||||||
loop: loopFacts{took: now.Add(-time.Second), pending: 1},
|
loop: loopFacts{took: now.Add(-time.Second), pending: 1},
|
||||||
mergesPassed: now.Add(-time.Minute),
|
|
||||||
selfCheck: selfCheckFacts{last: now.Add(-time.Minute), every: 5 * time.Minute},
|
selfCheck: selfCheckFacts{last: now.Add(-time.Minute), every: 5 * time.Minute},
|
||||||
lostConsumers: map[string]bool{}, epochs: map[int64]inventory.Epoch{},
|
lostConsumers: map[string]bool{}, epochs: map[int64]inventory.Epoch{},
|
||||||
lease: leaseFacts{held: true, epoch: 57, renewed: now.Add(-2 * time.Second)},
|
lease: leaseFacts{held: true, epoch: 57, renewed: now.Add(-2 * time.Second)},
|
||||||
@@ -72,12 +71,6 @@ var suppressions = map[string]suppression{
|
|||||||
inside: func(f *signalFacts) { f.loop.took = f.now.Add(-119 * time.Second) },
|
inside: func(f *signalFacts) { f.loop.took = f.now.Add(-119 * time.Second) },
|
||||||
past: func(f *signalFacts) { f.loop.took = f.now.Add(-121 * time.Second) },
|
past: func(f *signalFacts) { f.loop.took = f.now.Add(-121 * time.Second) },
|
||||||
},
|
},
|
||||||
"S5": {
|
|
||||||
inside: func(f *signalFacts) {},
|
|
||||||
past: func(f *signalFacts) {
|
|
||||||
f.merges = []missedMerge{{Owner: "novox", Repo: "app", Base: "main", Commit: "c0ffee0011", At: f.now.Add(-11 * time.Minute)}}
|
|
||||||
},
|
|
||||||
},
|
|
||||||
"S6": {
|
"S6": {
|
||||||
inside: func(f *signalFacts) {
|
inside: func(f *signalFacts) {
|
||||||
f.asks = []askFacts{{id: "build-1", seat: "node-build-agent", what: "novox/app", state: link.AskInFlight,
|
f.asks = []askFacts{{id: "build-1", seat: "node-build-agent", what: "novox/app", state: link.AskInFlight,
|
||||||
|
|||||||
@@ -294,10 +294,10 @@ func (f following) Announceable(ctx context.Context) ([]link.Announcement, error
|
|||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// notNow marks a store that could not be read right now, so the announcement is held rather than
|
// notNow marks a store that could not be read right now, or plans another controller is working (issue 921), so
|
||||||
// lost; anything else is returned as it was.
|
// the announcement is held rather than lost; anything else is returned as it was.
|
||||||
func notNow(err error) error {
|
func notNow(err error) error {
|
||||||
if inventory.Unreachable(err) {
|
if inventory.Unreachable(err) || errors.Is(err, inventory.ErrPlansBusy) {
|
||||||
return fmt.Errorf("%w: %w", link.ErrTryAgain, err)
|
return fmt.Errorf("%w: %w", link.ErrTryAgain, err)
|
||||||
}
|
}
|
||||||
return err
|
return err
|
||||||
|
|||||||
@@ -70,10 +70,6 @@ type signalFacts struct {
|
|||||||
loop loopFacts
|
loop loopFacts
|
||||||
loopErr error
|
loopErr error
|
||||||
|
|
||||||
merges []missedMerge
|
|
||||||
mergesPassed time.Time
|
|
||||||
mergesErr error
|
|
||||||
|
|
||||||
asks []askFacts
|
asks []askFacts
|
||||||
asksErr error
|
asksErr error
|
||||||
|
|
||||||
@@ -358,10 +354,6 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
|
|||||||
f.batches, f.plansErr = gatherBatches(ctx, inv, now)
|
f.batches, f.plansErr = gatherBatches(ctx, inv, now)
|
||||||
}
|
}
|
||||||
f.loop, f.loopErr = w.gatherLoop()
|
f.loop, f.loopErr = w.gatherLoop()
|
||||||
f.mergesPassed, f.merges, f.mergesErr = watchedMerges.last()
|
|
||||||
if f.mergesErr == nil && !f.mergesPassed.IsZero() && now.Sub(f.mergesPassed) > 3*mergeCatchUpEvery {
|
|
||||||
f.mergesErr = fmt.Errorf("the catch-up of merges has not passed since %s", f.mergesPassed.UTC().Format(time.RFC3339))
|
|
||||||
}
|
|
||||||
f.asks, f.asksErr = w.gatherAsks(ctx, inv)
|
f.asks, f.asksErr = w.gatherAsks(ctx, inv)
|
||||||
if open, err := w.keeper.Open(ctx); err != nil {
|
if open, err := w.keeper.Open(ctx); err != nil {
|
||||||
f.standingsErr = err
|
f.standingsErr = err
|
||||||
|
|||||||
@@ -160,3 +160,87 @@ func TestNatsOnlyMeshConfigsAccountPutsToItsBuckets(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// **The controller asks mesh-config's `config` in its own name** (issue 925): on a real server whose users are
|
||||||
|
// composed from records that hold mesh-config on novox serving `config`, the controller's request on
|
||||||
|
// `mesh.seat.mesh-config.call.config.controller` reaches mesh-config's account and its reply reaches the
|
||||||
|
// controller's inbox; the same request under another caller's name is refused by the server.
|
||||||
|
func TestNatsTheControllerAsksMeshConfigInItsOwnName(t *testing.T) {
|
||||||
|
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
seat := Seat{Name: "mesh-config", Scope: "mesh", Serves: []string{"config"}}
|
||||||
|
users, err := Users(Records{Nodes: []string{"novox"}, Assigned: map[string][]Declared{
|
||||||
|
"novox": {{Module: RuntimeModule}, {Module: "mesh-config", RunsAs: "mesh-config", Holds: []Seat{seat}}},
|
||||||
|
}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for i := range users {
|
||||||
|
users[i].PasswordHash = string(hash)
|
||||||
|
}
|
||||||
|
accounts, err := ComposeAccounts(users)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
conf := filepath.Join(t.TempDir(), "bus.conf")
|
||||||
|
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\n"+accounts), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
opts, err := server.ProcessConfigFile(conf)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("the composed accounts do not parse: %v", err)
|
||||||
|
}
|
||||||
|
opts.NoLog, opts.NoSigs = true, true
|
||||||
|
s, err := server.NewServer(opts)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
go s.Start()
|
||||||
|
if !s.ReadyForConnections(10 * time.Second) {
|
||||||
|
t.Fatal("the bus did not come up")
|
||||||
|
}
|
||||||
|
t.Cleanup(s.Shutdown)
|
||||||
|
connect := func(user string) (*nats.Conn, chan error) {
|
||||||
|
t.Helper()
|
||||||
|
refused := make(chan error, 8)
|
||||||
|
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw"), nats.CustomInboxPrefix("_INBOX."+user),
|
||||||
|
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refused <- err }))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s cannot connect: %v", user, err)
|
||||||
|
}
|
||||||
|
t.Cleanup(nc.Close)
|
||||||
|
return nc, refused
|
||||||
|
}
|
||||||
|
owner, _ := connect("novox.mesh-config")
|
||||||
|
subject := CalledSubject("mesh.seat.mesh-config.tool.config", ControllerName)
|
||||||
|
if subject != "mesh.seat.mesh-config.call.config.controller" {
|
||||||
|
t.Fatalf("the subject is %s", subject)
|
||||||
|
}
|
||||||
|
if _, err := owner.Subscribe("mesh.seat.mesh-config.call.config.*", func(m *nats.Msg) {
|
||||||
|
_ = m.Respond([]byte(`{"ok":true}`))
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := owner.Flush(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
controller, refused := connect(ControllerName)
|
||||||
|
reply, err := controller.Request(subject, []byte(`{"action":"set"}`), 5*time.Second)
|
||||||
|
if err != nil || string(reply.Data) != `{"ok":true}` {
|
||||||
|
t.Fatalf("the controller's own-name request got %v (%v), want mesh-config's reply", reply, err)
|
||||||
|
}
|
||||||
|
if err := controller.Publish(CalledSubject("mesh.seat.mesh-config.tool.config", "anchor.node-tools"), nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_ = controller.Flush()
|
||||||
|
select {
|
||||||
|
case err := <-refused:
|
||||||
|
if !errors.Is(err, nats.ErrPermissionViolation) {
|
||||||
|
t.Errorf("the server said %v, want a permissions violation", err)
|
||||||
|
}
|
||||||
|
case <-time.After(3 * time.Second):
|
||||||
|
t.Error("the server did not refuse the controller a request in another's name")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -38,6 +38,11 @@ var (
|
|||||||
// each ask by its id, its options and the actions they stand for, how it ended and whether the
|
// each ask by its id, its options and the actions they stand for, how it ended and whether the
|
||||||
// controller acted on its warrant — so a restart neither asks twice nor acts twice.
|
// controller acted on its warrant — so a restart neither asks twice nor acts twice.
|
||||||
AskedBucket = BucketName(ControllerSeat, "asked")
|
AskedBucket = BucketName(ControllerSeat, "asked")
|
||||||
|
// MergeWindowBucket holds the clock of every open batch's merge window (novox/hq ADR 0407 decision 1): per
|
||||||
|
// batch a key `<batch>.quiet`, made again by each merge into it and living the window, and a key
|
||||||
|
// `<batch>.most`, made once when it opens and living the window's most. The first to expire closes the
|
||||||
|
// batch, and the controller's watcher hears the delete marker. Written by the lease holder alone.
|
||||||
|
MergeWindowBucket = BucketName(ControllerSeat, "merge-window")
|
||||||
)
|
)
|
||||||
|
|
||||||
// The controller's record mirrors (novox/hq ADR 0289, design 54, design 57 package F3): a bucket per kind of
|
// The controller's record mirrors (novox/hq ADR 0289, design 54, design 57 package F3): a bucket per kind of
|
||||||
@@ -79,6 +84,10 @@ const (
|
|||||||
MirrorMarkerTTL = 24 * time.Hour
|
MirrorMarkerTTL = 24 * time.Hour
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// MergeWindowMarkerTTL is how long the marker an expired window key leaves is kept: long enough for a watcher
|
||||||
|
// that is up to hear it, short enough to leave nothing behind.
|
||||||
|
const MergeWindowMarkerTTL = time.Minute
|
||||||
|
|
||||||
// AskedKeptFor is how long an ask is kept after it was made: a month, as the router keeps its own.
|
// AskedKeptFor is how long an ask is kept after it was made: a month, as the router keeps its own.
|
||||||
const AskedKeptFor = 30 * 24 * time.Hour
|
const AskedKeptFor = 30 * 24 * time.Hour
|
||||||
|
|
||||||
@@ -106,13 +115,14 @@ const (
|
|||||||
func IsControllerBucket(bucket string) bool {
|
func IsControllerBucket(bucket string) bool {
|
||||||
return bucket == CallsBucket || bucket == HandActsBucket || bucket == ConditionsBucket ||
|
return bucket == CallsBucket || bucket == HandActsBucket || bucket == ConditionsBucket ||
|
||||||
bucket == ConditionHistoryBucket || bucket == LeaseBucket || bucket == AskedBucket ||
|
bucket == ConditionHistoryBucket || bucket == LeaseBucket || bucket == AskedBucket ||
|
||||||
bucket == AssignmentsBucket || bucket == DeclaredConfigBucket || bucket == PredefinedBucket
|
bucket == AssignmentsBucket || bucket == DeclaredConfigBucket || bucket == PredefinedBucket ||
|
||||||
|
bucket == MergeWindowBucket
|
||||||
}
|
}
|
||||||
|
|
||||||
// ControllerBuckets are the controller's own buckets, in the order they are asserted.
|
// ControllerBuckets are the controller's own buckets, in the order they are asserted.
|
||||||
func ControllerBuckets() []string {
|
func ControllerBuckets() []string {
|
||||||
return []string{LeaseBucket, CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket, AskedBucket,
|
return []string{LeaseBucket, CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket, AskedBucket,
|
||||||
AssignmentsBucket, DeclaredConfigBucket, PredefinedBucket}
|
AssignmentsBucket, DeclaredConfigBucket, PredefinedBucket, MergeWindowBucket}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ControllerBucketsAsserter is what raising the controller's buckets needs of a connection.
|
// ControllerBucketsAsserter is what raising the controller's buckets needs of a connection.
|
||||||
@@ -209,6 +219,21 @@ func (j *JetStream) EnsureControllerBuckets() error {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
return fmt.Errorf("asserting bucket %s: %w", AskedBucket, err)
|
return fmt.Errorf("asserting bucket %s: %w", AskedBucket, err)
|
||||||
}
|
}
|
||||||
|
// **The merge window: keys that expire.** A key's own time to live is the window; the marker an expired key
|
||||||
|
// leaves is what a watcher hears, kept a minute (ADR 0407 decision 1).
|
||||||
|
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
|
||||||
|
Bucket: MergeWindowBucket,
|
||||||
|
Description: "the clock of every open batch's merge window (novox/hq ADR 0407): `<batch>.quiet` lives the " +
|
||||||
|
"window and is made again by each merge, `<batch>.most` lives the most; the first to expire closes the " +
|
||||||
|
"batch; written by the controller alone",
|
||||||
|
History: 1,
|
||||||
|
MaxValueSize: 4 << 10,
|
||||||
|
MaxBytes: 8 << 20,
|
||||||
|
Storage: jetstream.FileStorage,
|
||||||
|
LimitMarkerTTL: MergeWindowMarkerTTL,
|
||||||
|
}); err != nil {
|
||||||
|
return fmt.Errorf("asserting bucket %s: %w", MergeWindowBucket, err)
|
||||||
|
}
|
||||||
// **The record mirrors: no age** (design 54 §5). A key is removed when its record is, by a purge whose
|
// **The record mirrors: no age** (design 54 §5). A key is removed when its record is, by a purge whose
|
||||||
// marker lives a day; a key that expired would be a record the bus forgot while the store still holds it.
|
// marker lives a day; a key that expired would be a record the bus forgot while the store still holds it.
|
||||||
for _, m := range []struct{ bucket, why string }{
|
for _, m := range []struct{ bucket, why string }{
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package broker
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"github.com/nats-io/nats.go"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -226,3 +227,175 @@ func TestTheControllerMayAskTheVaultInItsOwnName(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// **The controller may ask mesh-config's `config` in its own name** (issue 925): move-config and the settings verb
|
||||||
|
// set a trusted value through mesh-config, which takes it as the controller carries it only from the subject that
|
||||||
|
// names the controller as caller. Granted when the records hold a holder serving the verb, and not otherwise;
|
||||||
|
// the installer's first user list carries none.
|
||||||
|
func TestTheControllerMayAskMeshConfigInItsOwnName(t *testing.T) {
|
||||||
|
config := Declared{Module: "mesh-config", Holds: []Seat{{Name: "mesh-config", Serves: []string{"config"}}}}
|
||||||
|
users, err := Users(Records{Nodes: []string{"novox"}, Assigned: map[string][]Declared{"novox": {config}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if strings.Join(users[0].AsksTheConfig, ",") != "config" {
|
||||||
|
t.Fatalf("composed %v", users[0].AsksTheConfig)
|
||||||
|
}
|
||||||
|
p, err := PermissionsFor(users[0])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !slices.Contains(p.Publish, "mesh.seat.mesh-config.call.config.controller") {
|
||||||
|
t.Errorf("the controller may not publish mesh.seat.mesh-config.call.config.controller: %v", p.Publish)
|
||||||
|
}
|
||||||
|
if !MayPublish(p, CalledSubject("mesh.seat.mesh-config.tool.config", ControllerName)) {
|
||||||
|
t.Error("the controller may not ask mesh-config's config in its own name")
|
||||||
|
}
|
||||||
|
if MayPublish(p, CalledSubject("mesh.seat.mesh-config.tool.config", "anchor.node-tools")) {
|
||||||
|
t.Error("the controller may ask mesh-config's config in another's name")
|
||||||
|
}
|
||||||
|
// No holder serving it composes no grant, nor does genesis.
|
||||||
|
other := Declared{Module: "mesh-config", Holds: []Seat{{Name: "mesh-config", Serves: []string{"list"}}}}
|
||||||
|
for _, r := range []Records{{Nodes: []string{"novox"}}, {Nodes: []string{"novox"}, Assigned: map[string][]Declared{"novox": {other}}}} {
|
||||||
|
users, _ = Users(r)
|
||||||
|
if len(users[0].AsksTheConfig) != 0 {
|
||||||
|
t.Errorf("composed %v without a holder serving config", users[0].AsksTheConfig)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
genesis, _ := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
|
||||||
|
for _, s := range genesis.Publish {
|
||||||
|
if strings.Contains(s, "mesh-config") {
|
||||||
|
t.Errorf("the installer's controller may publish %s", s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// **The merge window is a bucket whose keys expire** (novox/hq ADR 0407 decision 1, issue 973): a key made with a
|
||||||
|
// time to live goes when it is up, and a watcher on the bucket hears it go. The controller may write it, and it is
|
||||||
|
// no undeclared state.
|
||||||
|
func TestTheMergeWindowsKeysExpireAndAWatcherHearsIt(t *testing.T) {
|
||||||
|
controller, err := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !slices.Contains(controller.Publish, "$KV."+MergeWindowBucket+".>") {
|
||||||
|
t.Error("the controller may not write the merge window's bucket")
|
||||||
|
}
|
||||||
|
if !IsControllerBucket(MergeWindowBucket) || !slices.Contains(ControllerBuckets(), MergeWindowBucket) {
|
||||||
|
t.Error("the merge window's bucket is not asserted as the controller's own")
|
||||||
|
}
|
||||||
|
js, err := Dial(testbus.URL(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer js.Close()
|
||||||
|
if err := js.EnsureControllerBuckets(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
api, err := jetstream.New(js.Conn())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
kv, err := api.KeyValue(ctx, MergeWindowBucket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
w, err := kv.WatchAll(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer w.Stop()
|
||||||
|
if _, err := kv.Create(ctx, "plan-1.quiet", []byte("x"), jetstream.KeyTTL(time.Second)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case e := <-w.Updates():
|
||||||
|
if e != nil && e.Key() == "plan-1.quiet" && e.Operation() != jetstream.KeyValuePut {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
case <-ctx.Done():
|
||||||
|
t.Fatal("no watcher heard the key expire")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// **The merges have a consumer of their own, which a reset of the events consumer leaves alone** (issue 973, novox/hq
|
||||||
|
// ADR 0407 decision 3): healer H4 re-makes the events consumer from now, dropping what it holds pending; a merge
|
||||||
|
// waiting in the merges consumer is not that consumer's, and nothing would catch it up.
|
||||||
|
func TestAResetOfTheEventsConsumerLeavesTheMergesAndTheirConsumerAlone(t *testing.T) {
|
||||||
|
js, err := Dial(testbus.URL(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer js.Close()
|
||||||
|
if err := AssertMeshStreams(js); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := AssertMeshConsumers(js); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, c := range MeshConsumers() {
|
||||||
|
if c.Stream == EventsStream && c.Name == MergesConsumerName {
|
||||||
|
if c.Resettable != "" || c.MaxDeliver != 0 || len(c.Filters) != 1 || c.Filters[0] != ControllerFollows[3] {
|
||||||
|
t.Errorf("the merges consumer is %+v; want one filter, no max-deliver, not resettable", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if c.Stream == EventsStream && c.Name == ControllerName {
|
||||||
|
for _, f := range c.Filters {
|
||||||
|
if f == ControllerFollows[3] {
|
||||||
|
t.Errorf("the events consumer still follows the merges, which the merges consumer takes")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, err := js.Context().Publish(ControllerFollows[3], []byte(`{"commit":"c1"}`)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := js.Context().Publish(ControllerFollows[0], []byte(`{}`)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
merges, err := js.Context().ConsumerInfo(EventsStream, MergesConsumerName)
|
||||||
|
if err != nil || merges.NumPending != 1 {
|
||||||
|
t.Fatalf("the merges consumer holds %v (%v); want the merge", merges, err)
|
||||||
|
}
|
||||||
|
if _, _, err := js.ResetConsumer(EventsStream, ControllerName); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
merges, err = js.Context().ConsumerInfo(EventsStream, MergesConsumerName)
|
||||||
|
if err != nil || merges.NumPending != 1 || merges.Delivered.Stream != 0 {
|
||||||
|
t.Fatalf("after the reset the merges consumer holds %+v (%v); want its merge left alone", merges, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// **Delivering this change loses no merge** (review of issue 973): a merge published under the old filter of the
|
||||||
|
// `controller` consumer and never acknowledged by it is delivered by the merges consumer once the assertion has
|
||||||
|
// made it, because it starts where `controller` has acknowledged up to, not from now.
|
||||||
|
func TestAMergeTheOldConsumerHadNotAcknowledgedIsTheMergesConsumers(t *testing.T) {
|
||||||
|
js, err := Dial(testbus.URL(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer js.Close()
|
||||||
|
if err := AssertMeshStreams(js); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
old := &nats.ConsumerConfig{Durable: ControllerName, AckPolicy: nats.AckExplicitPolicy, AckWait: 30 * time.Second,
|
||||||
|
MaxDeliver: 5, MaxAckPending: 1, DeliverSubject: "_DELIVER.controller.EVENTS", DeliverPolicy: nats.DeliverNewPolicy,
|
||||||
|
FilterSubjects: append(append([]string(nil), ControllerFollows...), AgainFilter(ControllerName))}
|
||||||
|
if _, err := js.Context().AddConsumer(EventsStream, old); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := js.Context().Publish(ControllerFollows[3], []byte(`{"commit":"c1"}`)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := AssertMeshConsumers(js); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
merges, err := js.Context().ConsumerInfo(EventsStream, MergesConsumerName)
|
||||||
|
if err != nil || merges.NumPending != 1 {
|
||||||
|
t.Fatalf("the merges consumer holds %v (%v); want the merge the old consumer never acknowledged", merges, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -36,6 +36,12 @@ type Consumer struct {
|
|||||||
// them over, and a loop that acts on one message at a time is the property the store window
|
// them over, and a loop that acts on one message at a time is the property the store window
|
||||||
// depends on. A queue group implies this, because a group has nothing to pull from.
|
// depends on. A queue group implies this, because a group has nothing to pull from.
|
||||||
Push bool
|
Push bool
|
||||||
|
// TakesOverFrom names a consumer of the same stream whose filter this one takes over (issue 973): made while
|
||||||
|
// that consumer still filters this one's subject, it starts after what that consumer has acknowledged, so a
|
||||||
|
// message the old one held unacknowledged is not lost to the change. Otherwise it starts as FromNow says.
|
||||||
|
TakesOverFrom string
|
||||||
|
// DeliverSubject, when set, is where a push consumer delivers instead of DeliverSubjectFor's.
|
||||||
|
DeliverSubject string
|
||||||
// AckWaitSeconds before an unacknowledged delivery is redelivered.
|
// AckWaitSeconds before an unacknowledged delivery is redelivered.
|
||||||
AckWaitSeconds int
|
AckWaitSeconds int
|
||||||
// MaxDeliver is how often a message is handed over before the consumer gives it up; zero for no
|
// MaxDeliver is how often a message is handed over before the consumer gives it up; zero for no
|
||||||
|
|||||||
@@ -343,6 +343,9 @@ func (j *JetStream) EnsureConsumer(c Consumer) error {
|
|||||||
if c.FromNow {
|
if c.FromNow {
|
||||||
want.DeliverPolicy = nats.DeliverNewPolicy
|
want.DeliverPolicy = nats.DeliverNewPolicy
|
||||||
}
|
}
|
||||||
|
if from := j.ackFloorOfTheConsumerItTakesOver(c); from > 0 {
|
||||||
|
want.DeliverPolicy, want.OptStartSeq = nats.DeliverByStartSequencePolicy, from
|
||||||
|
}
|
||||||
if _, err := j.js.AddConsumer(c.Stream, want); err != nil {
|
if _, err := j.js.AddConsumer(c.Stream, want); err != nil {
|
||||||
return fmt.Errorf("creating consumer %s on %s: %w", c.Name, c.Stream, err)
|
return fmt.Errorf("creating consumer %s on %s: %w", c.Name, c.Stream, err)
|
||||||
}
|
}
|
||||||
@@ -352,6 +355,26 @@ func (j *JetStream) EnsureConsumer(c Consumer) error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ackFloorOfTheConsumerItTakesOver is the stream sequence a consumer made now starts at so as to lose nothing the
|
||||||
|
// consumer it takes over from (Consumer.TakesOverFrom) has not acknowledged: that consumer's ack floor plus one, when
|
||||||
|
// it exists and still filters this one's subject; zero otherwise.
|
||||||
|
func (j *JetStream) ackFloorOfTheConsumerItTakesOver(c Consumer) uint64 {
|
||||||
|
if c.TakesOverFrom == "" || len(c.Filters) != 1 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
have, err := j.js.ConsumerInfo(c.Stream, c.TakesOverFrom)
|
||||||
|
if err != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
filters := append([]string{have.Config.FilterSubject}, have.Config.FilterSubjects...)
|
||||||
|
for _, f := range filters {
|
||||||
|
if f == c.Filters[0] {
|
||||||
|
return have.AckFloor.Stream + 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
// ConsumerState is what a reset says about a consumer, before and after.
|
// ConsumerState is what a reset says about a consumer, before and after.
|
||||||
type ConsumerState struct {
|
type ConsumerState struct {
|
||||||
DeliverPolicy string
|
DeliverPolicy string
|
||||||
|
|||||||
+30
-1
@@ -141,6 +141,12 @@ type Principal struct {
|
|||||||
// controller asks. Composed from the records like AsksIssued, so the installer's first user list carries none.
|
// controller asks. Composed from the records like AsksIssued, so the installer's first user list carries none.
|
||||||
AsksTheVault []string
|
AsksTheVault []string
|
||||||
|
|
||||||
|
// AsksTheConfig are the mesh-config verbs the controller asks in its own name (VerbsTheControllerAsksTheConfig,
|
||||||
|
// issue 925), those mesh-config's holder serves: on the subject that names its caller, so mesh-config takes a
|
||||||
|
// value as the controller carries it (ADR 0348 §8). Composed from the records like AsksTheVault, so the
|
||||||
|
// installer's first user list carries none.
|
||||||
|
AsksTheConfig []string
|
||||||
|
|
||||||
// AsksTheDesk says this principal is the vault's holder, which asks the desks' secret prompt in its own name
|
// AsksTheDesk says this principal is the vault's holder, which asks the desks' secret prompt in its own name
|
||||||
// (novox/hq ADR 0349 clause 9, issue 723): `node-launcher.secret` on any machine, on the subject that names
|
// (novox/hq ADR 0349 clause 9, issue 723): `node-launcher.secret` on any machine, on the subject that names
|
||||||
// its caller, and nothing else of that seat. With AsksIssued on the same principal, the vault's whole reach
|
// its caller, and nothing else of that seat. With AsksIssued on the same principal, the vault's whole reach
|
||||||
@@ -316,6 +322,12 @@ var VerbsTheControllerAsksTheVault = []SeatVerb{{Seat: "mesh-vault", Verb: "acce
|
|||||||
{Seat: "mesh-vault", Verb: "secret-ask"}, {Seat: "mesh-vault", Verb: "issued"}, {Seat: "mesh-vault", Verb: "rotate"},
|
{Seat: "mesh-vault", Verb: "secret-ask"}, {Seat: "mesh-vault", Verb: "issued"}, {Seat: "mesh-vault", Verb: "rotate"},
|
||||||
{Seat: "mesh-vault", Verb: "move-asks"}, {Seat: "mesh-vault", Verb: "get"}}
|
{Seat: "mesh-vault", Verb: "move-asks"}, {Seat: "mesh-vault", Verb: "get"}}
|
||||||
|
|
||||||
|
// VerbsTheControllerAsksTheConfig are the mesh-config seat's verbs the controller calls in its own name (issue 925):
|
||||||
|
// `config`, the one verb mesh-config serves, which move-config uses to set each value and the settings verb to set,
|
||||||
|
// clear and propose through mesh-config. mesh-config takes a value as the controller carries it only from the
|
||||||
|
// subject that names the controller as caller (ADR 0348 §8); granted as Principal.AsksTheConfig says.
|
||||||
|
var VerbsTheControllerAsksTheConfig = []SeatVerb{{Seat: "mesh-config", Verb: "config"}}
|
||||||
|
|
||||||
// VaultMoveIn is the vault's verb a machine's node-engine asks in its own name to hand in a value sealed to its
|
// VaultMoveIn is the vault's verb a machine's node-engine asks in its own name to hand in a value sealed to its
|
||||||
// machine (design 57's V3, issue 676); granted as Principal.MovesIntoTheVault says.
|
// machine (design 57's V3, issue 676); granted as Principal.MovesIntoTheVault says.
|
||||||
var VaultMoveIn = SeatVerb{Seat: "mesh-vault", Verb: "move-in"}
|
var VaultMoveIn = SeatVerb{Seat: "mesh-vault", Verb: "move-in"}
|
||||||
@@ -356,6 +368,16 @@ var DegradableGrants = []DegradableGrant{
|
|||||||
Subject: "mesh.seat." + VerbsTheControllerAsksTheOperatorChannel[0].Seat + ".tool." + VerbsTheControllerAsksTheOperatorChannel[0].Verb,
|
Subject: "mesh.seat." + VerbsTheControllerAsksTheOperatorChannel[0].Seat + ".tool." + VerbsTheControllerAsksTheOperatorChannel[0].Verb,
|
||||||
Reason: "refused: the controller raises the secret-given condition as before (ADR 0380 §9)",
|
Reason: "refused: the controller raises the secret-given condition as before (ADR 0380 §9)",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
Subject: "$JS.ACK.EVENTS." + MergesConsumerName + ".>",
|
||||||
|
Reason: "carried only at genesis: the controller composes its own permissions and pushes them to the bus at " +
|
||||||
|
"its first push, which is when it acknowledges a merge (ADR 0407 decision 3, issue 973)",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Subject: "$KV." + MergeWindowBucket + ".>",
|
||||||
|
Reason: "refused: the controller closes a batch's window by a timer of its own for that batch, as the " +
|
||||||
|
"window setting says (ADR 0407 decision 1, issue 973)",
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
// perMachineEvents are a node-scoped seat's events about the holder itself, whose last token is the
|
// perMachineEvents are a node-scoped seat's events about the holder itself, whose last token is the
|
||||||
@@ -623,7 +645,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
|
|||||||
// delivery from EVENTS — a consumer that cannot ack has every message redelivered for
|
// delivery from EVENTS — a consumer that cannot ack has every message redelivered for
|
||||||
// ever, refused by the list it already has.
|
// ever, refused by the list it already has.
|
||||||
sub = append(sub, ControllerFollows...)
|
sub = append(sub, ControllerFollows...)
|
||||||
pub = append(pub, "$JS.ACK.EVENTS."+ControllerName+".>")
|
pub = append(pub, "$JS.ACK.EVENTS."+ControllerName+".>", "$JS.ACK.EVENTS."+MergesConsumerName+".>")
|
||||||
|
|
||||||
// **Where an enrolment's answer goes**, and `allow_responses` does not cover it. That
|
// **Where an enrolment's answer goes**, and `allow_responses` does not cover it. That
|
||||||
// permits one reply to the reply subject of a message the user received — and a message a
|
// permits one reply to the reply subject of a message the user received — and a message a
|
||||||
@@ -1089,6 +1111,13 @@ func PermissionsFor(p Principal) (Permissions, error) {
|
|||||||
pub = append(pub, "mesh.seat."+v.Seat+"."+CallKind+"."+v.Verb+"."+CallerToken(p.Username()))
|
pub = append(pub, "mesh.seat."+v.Seat+"."+CallKind+"."+v.Verb+"."+CallerToken(p.Username()))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// **And mesh-config's verb** (issue 925): it takes a trusted value from the controller alone, known from the
|
||||||
|
// subject's last token.
|
||||||
|
for _, v := range VerbsTheControllerAsksTheConfig {
|
||||||
|
if slices.Contains(p.AsksTheConfig, v.Verb) {
|
||||||
|
pub = append(pub, "mesh.seat."+v.Seat+"."+CallKind+"."+v.Verb+"."+CallerToken(p.Username()))
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
sort.Strings(pub)
|
sort.Strings(pub)
|
||||||
|
|||||||
@@ -233,6 +233,18 @@ const ProvAppliedStream = "PROV_APPLIED"
|
|||||||
// all (ProvisionerAppliedSubject).
|
// all (ProvisionerAppliedSubject).
|
||||||
const ProvAppliedSubjects = "prov.applied.>"
|
const ProvAppliedSubjects = "prov.applied.>"
|
||||||
|
|
||||||
|
// eventsTheControllerFollows are the events the controller's events consumer takes: all it follows but the forge's
|
||||||
|
// merges, which the merges consumer takes (MergesConsumerName).
|
||||||
|
func eventsTheControllerFollows() []string {
|
||||||
|
var out []string
|
||||||
|
for _, subject := range ControllerFollows {
|
||||||
|
if subject != ControllerFollows[3] {
|
||||||
|
out = append(out, subject)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
// DeliverSubjectFor is where a push consumer's messages land.
|
// DeliverSubjectFor is where a push consumer's messages land.
|
||||||
//
|
//
|
||||||
// **Per consumer, which means per stream as well as per name** (novox/hq 04-ISSUES/146). A push
|
// **Per consumer, which means per stream as well as per name** (novox/hq 04-ISSUES/146). A push
|
||||||
@@ -248,6 +260,9 @@ const ProvAppliedSubjects = "prov.applied.>"
|
|||||||
// durable's name to its stream, and this subject was the one place that scoping was dropped. It
|
// durable's name to its stream, and this subject was the one place that scoping was dropped. It
|
||||||
// stays inside what a controller may already subscribe (`_DELIVER.controller.>`).
|
// stays inside what a controller may already subscribe (`_DELIVER.controller.>`).
|
||||||
func DeliverSubjectFor(c Consumer) string {
|
func DeliverSubjectFor(c Consumer) string {
|
||||||
|
if c.DeliverSubject != "" {
|
||||||
|
return c.DeliverSubject
|
||||||
|
}
|
||||||
return "_DELIVER." + c.Name + "." + c.Stream
|
return "_DELIVER." + c.Name + "." + c.Stream
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -317,6 +332,13 @@ func Overlaps() []string {
|
|||||||
// ack subject is derived from (nats.go: `$JS.ACK.<stream>.controller.>`).
|
// ack subject is derived from (nats.go: `$JS.ACK.<stream>.controller.>`).
|
||||||
const ControllerName = "controller"
|
const ControllerName = "controller"
|
||||||
|
|
||||||
|
// MergesConsumerName is the controller's durable consumer of the forge's merged event on EVENTS, apart from the
|
||||||
|
// events consumer (novox/hq ADR 0407 decision 3, issue 973). A merge is never dropped, so this consumer has no
|
||||||
|
// max-deliver, which would send a merge held five times to DEAD_LETTERS to wait for a person, and no healer
|
||||||
|
// resets it, which would drop the merges it holds. A consumer of its own because both of those are properties of
|
||||||
|
// a consumer, not of a subject.
|
||||||
|
const MergesConsumerName = "controller-merges"
|
||||||
|
|
||||||
// ControllerSeat is the role the control plane holds, and ControllerStates are the facts it states
|
// ControllerSeat is the role the control plane holds, and ControllerStates are the facts it states
|
||||||
// under it (novox/hq ADR 0134).
|
// under it (novox/hq ADR 0134).
|
||||||
//
|
//
|
||||||
@@ -450,10 +472,27 @@ func MeshConsumers() []Consumer {
|
|||||||
Why: "the controller is the single consumer of what nodes say; explicit ack and no " +
|
Why: "the controller is the single consumer of what nodes say; explicit ack and no " +
|
||||||
"max-deliver, because the store window's bound is the controller's own",
|
"max-deliver, because the store window's bound is the controller's own",
|
||||||
},
|
},
|
||||||
|
// The forge's merges (novox/hq ADR 0407 decision 3, issue 973): never dropped, so no max-deliver and no
|
||||||
|
// reset; a merge the controller cannot record yet is held with a growing delay until it can. **Asserted before
|
||||||
|
// the events consumer**, which stops filtering merges: the order of this list is the order of assertion.
|
||||||
|
{
|
||||||
|
Name: MergesConsumerName,
|
||||||
|
Stream: "EVENTS",
|
||||||
|
// Within what the controller may subscribe already (`_DELIVER.controller.>`).
|
||||||
|
DeliverSubject: "_DELIVER." + ControllerName + ".merges",
|
||||||
|
Filters: []string{ControllerFollows[3]},
|
||||||
|
Push: true,
|
||||||
|
AckWaitSeconds: 30,
|
||||||
|
MaxAckPending: 1,
|
||||||
|
FromNow: true,
|
||||||
|
TakesOverFrom: ControllerName,
|
||||||
|
Why: "the forge's merges, one at a time; no max-deliver and never reset, because a merge is never " +
|
||||||
|
"dropped and no person releases one",
|
||||||
|
},
|
||||||
{
|
{
|
||||||
Name: ControllerName,
|
Name: ControllerName,
|
||||||
Stream: "EVENTS",
|
Stream: "EVENTS",
|
||||||
Filters: append(append([]string(nil), ControllerFollows...), AgainFilter(ControllerName)),
|
Filters: append(eventsTheControllerFollows(), AgainFilter(ControllerName)),
|
||||||
Push: true,
|
Push: true,
|
||||||
AckWaitSeconds: 30,
|
AckWaitSeconds: 30,
|
||||||
MaxDeliver: 5,
|
MaxDeliver: 5,
|
||||||
@@ -463,12 +502,13 @@ func MeshConsumers() []Consumer {
|
|||||||
MaxAckPending: 1,
|
MaxAckPending: 1,
|
||||||
FromNow: true,
|
FromNow: true,
|
||||||
// **The one consumer the mesh resets by itself** (healer H4, issue 248): it fell a week
|
// **The one consumer the mesh resets by itself** (healer H4, issue 248): it fell a week
|
||||||
// behind once and held every merge after it. What a reset drops is caught up elsewhere —
|
// behind once and held every merge after it. What a reset drops is caught up elsewhere — a
|
||||||
// a merge by the catch-up pass that reads the forge (issue 266), a build's outcome from the
|
// build's outcome from the build records a plan settles from (issue 214), a provider's failing
|
||||||
// build records a plan settles from (issue 214), a provider's failing word by the provider
|
// word by the provider saying it again every quarter of an hour (ADR 0224). **Not the merges**:
|
||||||
// saying it again every quarter of an hour (ADR 0224).
|
// they are the merges consumer's below, which a reset leaves alone (ADR 0407 decision 3).
|
||||||
Resettable: "what it drops is caught up: merges by the catch-up pass (issue 266), build outcomes " +
|
Resettable: "what it drops is caught up: build outcomes from the build records (issue 214), a " +
|
||||||
"from the build records (issue 214), a provider's failing word said again (ADR 0224)",
|
"provider's failing word said again (ADR 0224); the forge's merges are not its, they have a " +
|
||||||
|
"consumer of their own that no reset touches",
|
||||||
Why: "the events the mesh's own controller reacts to, one at a time; after " +
|
Why: "the events the mesh's own controller reacts to, one at a time; after " +
|
||||||
"max-deliver it gives the event up, and the controller keeps it in DEAD_LETTERS until " +
|
"max-deliver it gives the event up, and the controller keeps it in DEAD_LETTERS until " +
|
||||||
"a person delivers it again or drops it",
|
"a person delivers it again or drops it",
|
||||||
|
|||||||
+1
-1
@@ -24,7 +24,7 @@ accounts {
|
|||||||
jetstream: enabled
|
jetstream: enabled
|
||||||
users = [
|
users = [
|
||||||
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
|
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
|
||||||
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_predefined.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
|
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_merge-window.>", "$KV.mesh-controller_predefined.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
|
||||||
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
|
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
|
||||||
allow_responses: { max: 1, ttl: "1m" }
|
allow_responses: { max: 1, ttl: "1m" }
|
||||||
} }
|
} }
|
||||||
|
|||||||
@@ -102,7 +102,8 @@ type Records struct {
|
|||||||
// is a mesh that cannot be told anything, and there is no state of the records in which that is
|
// is a mesh that cannot be told anything, and there is no state of the records in which that is
|
||||||
// correct.
|
// correct.
|
||||||
func Users(r Records) ([]Principal, error) {
|
func Users(r Records) ([]Principal, error) {
|
||||||
out := []Principal{{Kind: KindController, Uses: asksSeatOf(r), AsksIssued: issuedOf(r), AsksTheVault: vaultVerbsOf(r)}}
|
out := []Principal{{Kind: KindController, Uses: asksSeatOf(r), AsksIssued: issuedOf(r), AsksTheVault: vaultVerbsOf(r),
|
||||||
|
AsksTheConfig: servedOf(r, VerbsTheControllerAsksTheConfig)}}
|
||||||
|
|
||||||
for _, node := range sortedCopy(r.Nodes) {
|
for _, node := range sortedCopy(r.Nodes) {
|
||||||
witness := false
|
witness := false
|
||||||
@@ -290,13 +291,16 @@ func holdsSeat(d Declared, seat string) bool {
|
|||||||
|
|
||||||
// vaultVerbsOf are the vault's intake verbs its holder serves, as the records say (issue 675): what the controller
|
// vaultVerbsOf are the vault's intake verbs its holder serves, as the records say (issue 675): what the controller
|
||||||
// is granted to ask it in its own name. None while no holder serves them.
|
// is granted to ask it in its own name. None while no holder serves them.
|
||||||
func vaultVerbsOf(r Records) []string {
|
func vaultVerbsOf(r Records) []string { return servedOf(r, VerbsTheControllerAsksTheVault) }
|
||||||
|
|
||||||
|
// servedOf are the verbs of asked that a seat's holder serves, as the records say, by name.
|
||||||
|
func servedOf(r Records, asked []SeatVerb) []string {
|
||||||
seen := map[string]bool{}
|
seen := map[string]bool{}
|
||||||
var out []string
|
var out []string
|
||||||
for _, node := range sortedCopy(r.Nodes) {
|
for _, node := range sortedCopy(r.Nodes) {
|
||||||
for _, d := range r.Assigned[node] {
|
for _, d := range r.Assigned[node] {
|
||||||
for _, s := range d.Holds {
|
for _, s := range d.Holds {
|
||||||
for _, v := range VerbsTheControllerAsksTheVault {
|
for _, v := range asked {
|
||||||
if s.Name == v.Seat && slices.Contains(s.Serves, v.Verb) && !seen[v.Verb] {
|
if s.Name == v.Seat && slices.Contains(s.Serves, v.Verb) && !seen[v.Verb] {
|
||||||
seen[v.Verb] = true
|
seen[v.Verb] = true
|
||||||
out = append(out, v.Verb)
|
out = append(out, v.Verb)
|
||||||
|
|||||||
@@ -117,8 +117,11 @@ var WritersTable = []WriterRow{
|
|||||||
Subjects: []string{"mesh.control.*.report", "mesh.control.*.health", "mesh.control.*.cli"}, Writes: ownMachine},
|
Subjects: []string{"mesh.control.*.report", "mesh.control.*.health", "mesh.control.*.cli"}, Writes: ownMachine},
|
||||||
{State: "the controller lease", Writer: "the controller instance holding it", KeptIn: "key-value " + LeaseBucket,
|
{State: "the controller lease", Writer: "the controller instance holding it", KeptIn: "key-value " + LeaseBucket,
|
||||||
Others: "a candidate waits", Subjects: kvOf(LeaseBucket), Writes: isController},
|
Others: "a candidate waits", Subjects: kvOf(LeaseBucket), Writes: isController},
|
||||||
|
// And the clock of an open batch's merge window, a plan before it is one (novox/hq ADR 0407 decision 1): keys that
|
||||||
|
// expire in a bucket of their own, written by the lease holder.
|
||||||
{State: "plans and their tiers", Writer: "controller (lease holder), compare-and-set on the plan's revision",
|
{State: "plans and their tiers", Writer: "controller (lease holder), compare-and-set on the plan's revision",
|
||||||
KeptIn: "the controller's store", Others: "read through plans"},
|
KeptIn: "the controller's store; a batch's merge window in key-value " + MergeWindowBucket, Others: "read through plans",
|
||||||
|
Subjects: kvOf(MergeWindowBucket), Writes: isController},
|
||||||
{State: "conditions", Writer: "controller", KeptIn: "key-value " + ConditionsBucket + " (and its history, " +
|
{State: "conditions", Writer: "controller", KeptIn: "key-value " + ConditionsBucket + " (and its history, " +
|
||||||
ConditionHistoryBucket + "); how often each was observed, when last and its newest evidence in the " +
|
ConditionHistoryBucket + "); how often each was observed, when last and its newest evidence in the " +
|
||||||
"controller's store, so a look that changes nothing writes no bucket", Others: "raise or clear only through observations the controller reads",
|
"controller's store, so a look that changes nothing writes no bucket", Others: "raise or clear only through observations the controller reads",
|
||||||
|
|||||||
@@ -111,7 +111,7 @@ func boundNeed(n Needed, bound Chosen, catalogue map[string]Manifest, node Node,
|
|||||||
}
|
}
|
||||||
held.From, held.At, held.Module = node.Name, at, m.Module
|
held.From, held.At, held.Module = node.Name, at, m.Module
|
||||||
held.Serves, held.SharedOwn, held.Identity = servedByOne(m, n.Name), sharedByOne(m, n.Name), m.IdentityBoundOf(n.Name)
|
held.Serves, held.SharedOwn, held.Identity = servedByOne(m, n.Name), sharedByOne(m, n.Name), m.IdentityBoundOf(n.Name)
|
||||||
held.CredentialForm = m.CredentialFormOf(n.Name)
|
held.CredentialForm, held.CredentialTool = m.CredentialFormOf(n.Name), m.CredentialToolOf(n.Name)
|
||||||
return held, true, ""
|
return held, true, ""
|
||||||
}
|
}
|
||||||
matching := bound.among(world.Offered[n.Name])
|
matching := bound.among(world.Offered[n.Name])
|
||||||
@@ -125,7 +125,7 @@ func boundNeed(n Needed, bound Chosen, catalogue map[string]Manifest, node Node,
|
|||||||
identity := DefaultIdentityBound
|
identity := DefaultIdentityBound
|
||||||
if pm, known := catalogue[p.Module]; known {
|
if pm, known := catalogue[p.Module]; known {
|
||||||
held.SharedOwn, _ = pm.SharedCredentialOf(n.Name)
|
held.SharedOwn, _ = pm.SharedCredentialOf(n.Name)
|
||||||
held.CredentialForm = pm.CredentialFormOf(n.Name)
|
held.CredentialForm, held.CredentialTool = pm.CredentialFormOf(n.Name), pm.CredentialToolOf(n.Name)
|
||||||
identity = pm.IdentityBoundOf(n.Name)
|
identity = pm.IdentityBoundOf(n.Name)
|
||||||
}
|
}
|
||||||
held.From, held.At, held.Module, held.Serves, held.Identity = p.Node, p.At, p.Module, p.Serves, identity
|
held.From, held.At, held.Module, held.Serves, held.Identity = p.Node, p.At, p.Module, p.Serves, identity
|
||||||
|
|||||||
@@ -0,0 +1,147 @@
|
|||||||
|
package catalogue
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"path"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A node's files, read and managed (novox/hq ADR 0394, issue 900; superseding ADR 0310; hq to-be 41 WP9).
|
||||||
|
//
|
||||||
|
// **One seat for every file on a node.** The holder runs as the tool runner's account, the node's operator
|
||||||
|
// account (ADR 0394 §2), and reaches further through `sudo -n` where that account's rights do not. It refuses by
|
||||||
|
// name the places a secret is kept (reads only, ADR 0394 §7) and the paths a module owns, unless a write names a reason (`override`).
|
||||||
|
// node-reads stays beside it until mesh-files is assigned on every node (ADR 0394 §9); it goes in the next
|
||||||
|
// delivery.
|
||||||
|
|
||||||
|
// FilesSeat is the role that reads and manages a node's files and accounts (novox/hq ADR 0394).
|
||||||
|
const FilesSeat = "node-files"
|
||||||
|
|
||||||
|
// FilesOwnedPaths is the word the holder of node-files is given beside the secret places: a JSON object from
|
||||||
|
// each absolute path a module owns on its node to "module (tool to use)" (novox/hq ADR 0394 §5).
|
||||||
|
const FilesOwnedPaths = "MESH_OWNED_PATHS"
|
||||||
|
|
||||||
|
const (
|
||||||
|
filesAsAccount = "as the node's operator account, through sudo -n where its rights do not reach; a place a " +
|
||||||
|
"secret is kept, or a known credential file, is refused by name"
|
||||||
|
overrideWord = "a reason, to change a path a module owns anyway (optional)"
|
||||||
|
)
|
||||||
|
|
||||||
|
// filesVerbs is the contract every holder of node-files serves (novox/hq ADR 0394 §1): the reads of node-reads,
|
||||||
|
// with their descriptions restated for the operator's account, and the writes.
|
||||||
|
func filesVerbs() []Verb {
|
||||||
|
var verbs []Verb
|
||||||
|
for _, v := range readsVerbs() {
|
||||||
|
v.Description = strings.Replace(v.Description,
|
||||||
|
"Read as the holder's own account, which holds no secret: a component it may not reach says so.",
|
||||||
|
"", 1)
|
||||||
|
v.Description = strings.TrimSpace(v.Description) + " Read " + filesAsAccount + "."
|
||||||
|
verbs = append(verbs, v)
|
||||||
|
}
|
||||||
|
write := func(name, description string, in map[string]string, required []string, optional []string, replaces ...string) Verb {
|
||||||
|
in["reason"] = "why the file is changed, for the audit trail"
|
||||||
|
in["override"] = overrideWord
|
||||||
|
return Verb{Name: name, Description: description + " Refuses a path a module owns " +
|
||||||
|
"unless override names a reason; done as the node's operator account, through sudo -n where its rights do not reach.",
|
||||||
|
Input: schema(in, append(required, "reason"), append(optional, "override")...), Replaces: replaces}
|
||||||
|
}
|
||||||
|
return append(verbs,
|
||||||
|
write("write", "Write one text file whole, replacing what it holds; the old content goes to the node's trash.",
|
||||||
|
map[string]string{"path": "an absolute path", "content": "the text to write",
|
||||||
|
"mode": "the file's mode, such as 0644 (optional)"},
|
||||||
|
[]string{"path", "content"}, []string{"mode"}, "tee", ">"),
|
||||||
|
write("copy", "Copy a file or directory to a path on this node. A target on another node (<node>:/path) is "+
|
||||||
|
"refused: copying between nodes comes in step 2 (issue 903). An existing target is overwritten, the old one "+
|
||||||
|
"kept in the node's trash for 7 days.",
|
||||||
|
map[string]string{"from": "an absolute path", "to": "an absolute path on this node"},
|
||||||
|
[]string{"from", "to"}, nil, "cp", "cp -a", "ssh <node> cp"),
|
||||||
|
write("move", "Move or rename a file or directory. A target on another node is refused: copying between nodes "+
|
||||||
|
"comes in step 2 (issue 903). An existing target is overwritten, the old one kept in the node's trash for 7 days.",
|
||||||
|
map[string]string{"from": "an absolute path", "to": "an absolute path"},
|
||||||
|
[]string{"from", "to"}, nil, "mv"),
|
||||||
|
write("delete", "Delete a file or directory into the node's trash, where restore can bring it back.",
|
||||||
|
map[string]string{"path": "an absolute path", "recursive": "\"true\": delete a directory with what it holds"},
|
||||||
|
[]string{"path"}, []string{"recursive"}, "rm", "rm -r"),
|
||||||
|
write("mkdir", "Make a directory.",
|
||||||
|
map[string]string{"path": "an absolute path", "parents": "\"true\": make the missing parents too"},
|
||||||
|
[]string{"path"}, []string{"parents"}, "mkdir", "mkdir -p"),
|
||||||
|
write("chmod", "Change the mode of a file or directory.",
|
||||||
|
map[string]string{"path": "an absolute path", "mode": "the mode, such as 0644 or u+x",
|
||||||
|
"recursive": "\"true\": also everything under a directory"},
|
||||||
|
[]string{"path", "mode"}, []string{"recursive"}, "chmod"),
|
||||||
|
write("chown", "Change the owner and group of a file or directory.",
|
||||||
|
map[string]string{"path": "an absolute path", "owner": "an account's name or number",
|
||||||
|
"group": "a group's name or number (optional)", "recursive": "\"true\": also everything under a directory"},
|
||||||
|
[]string{"path", "owner"}, []string{"group", "recursive"}, "chown", "chgrp"),
|
||||||
|
Verb{Name: "trash", Description: "What the node's trash holds: each deleted or replaced item with its id, " +
|
||||||
|
"original path and when it went.",
|
||||||
|
Input: schema(map[string]string{}, nil), Replaces: []string{"ls ~/.local/share/Trash"}},
|
||||||
|
write("restore", "Put an item of the node's trash back where it was; refuses when something is there now.",
|
||||||
|
map[string]string{"id": "an id from trash"}, []string{"id"}, nil, "trash-restore"),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// FilesHolderProblems is what a holder of node-files is held to (novox/hq ADR 0394 §2): it runs as the tool
|
||||||
|
// runner's account, the node's operator account, so it declares no runs-as.
|
||||||
|
func FilesHolderProblems(m Manifest) []string {
|
||||||
|
if !m.ClaimsSeat(FilesSeat) || m.RunsAs == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return []string{fmt.Sprintf("%s claims %s and declares runs-as %q: its holder runs as the tool runner's account, "+
|
||||||
|
"the node's operator account (novox/hq ADR 0394 §2)", m.Module, FilesSeat, m.RunsAs)}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ownedPaths is every path a module owns on this node, as the holder is told (novox/hq ADR 0394 §5): the
|
||||||
|
// paths every seat declares (ADR 0287) with the module holding the seat here, and the paths and files each
|
||||||
|
// module's manifest places (declaredPaths). A seat's path wins where both name one.
|
||||||
|
func (r Resolution) ownedPaths() map[string]string {
|
||||||
|
owned := map[string]string{}
|
||||||
|
for _, m := range r.Modules {
|
||||||
|
for _, p := range sortedKeys(declaredPaths(m)) {
|
||||||
|
owned[p] = m.Module + " (its own tools)"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, s := range Seats() {
|
||||||
|
for _, sp := range s.Paths {
|
||||||
|
for _, m := range r.Modules {
|
||||||
|
if m.ClaimsSeat(s.Name) && declaresPath(m, path.Clean(sp.Path)) {
|
||||||
|
owned[sp.Path] = m.Module + " (" + s.Name + ")"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return owned
|
||||||
|
}
|
||||||
|
|
||||||
|
// filesWords is what the holder of node-files is given beside its own words: the node's secret places and the
|
||||||
|
// paths modules own here.
|
||||||
|
func (r Resolution) filesWords(m Manifest, words map[string]string, with Rendering) (map[string]string, error) {
|
||||||
|
if !m.ClaimsSeat(FilesSeat) {
|
||||||
|
return words, nil
|
||||||
|
}
|
||||||
|
places, err := r.secretPlaces(with)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
places2, err := json.Marshal(places)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
owned, err := json.Marshal(r.ownedPaths())
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if words == nil {
|
||||||
|
words = map[string]string{}
|
||||||
|
}
|
||||||
|
for _, w := range []string{ReadsSecretPlaces, FilesOwnedPaths} {
|
||||||
|
if _, has := words[w]; has {
|
||||||
|
return nil, fmt.Errorf("%s gives its own %s; the node's secret places and owned paths are the mesh's to "+
|
||||||
|
"say (novox/hq ADR 0394 §5)", m.Module, w)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
words[ReadsSecretPlaces] = string(places2)
|
||||||
|
words[FilesOwnedPaths] = string(owned)
|
||||||
|
return words, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
package catalogue
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// issue 900, novox/hq ADR 0394 §1: node-files is a node seat whose verbs say the commands they replace.
|
||||||
|
func TestNodeFilesIsANodeSeatWithEveryVerbOfADR0394(t *testing.T) {
|
||||||
|
s, ok := SeatNamed(FilesSeat)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("the mesh defines no seat %s", FilesSeat)
|
||||||
|
}
|
||||||
|
if s.Scope != ScopeNode || s.Delivers != "" || !strings.Contains(s.Decision, "ADR 0394") {
|
||||||
|
t.Errorf("%s: scope %q, delivers %q, decision %q", FilesSeat, s.Scope, s.Delivers, s.Decision)
|
||||||
|
}
|
||||||
|
have := map[string]bool{}
|
||||||
|
replaced := map[string]string{}
|
||||||
|
for _, v := range s.Serves {
|
||||||
|
have[v.Name] = true
|
||||||
|
if v.Optional {
|
||||||
|
t.Errorf("%s.%s is optional", FilesSeat, v.Name)
|
||||||
|
}
|
||||||
|
if len(v.Replaces) == 0 {
|
||||||
|
t.Errorf("%s.%s says nothing it replaces", FilesSeat, v.Name)
|
||||||
|
}
|
||||||
|
for _, c := range v.Replaces {
|
||||||
|
replaced[c] = v.Name
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, name := range []string{"stat", "readlink", "list", "read", "search", "account", "write", "copy", "move",
|
||||||
|
"delete", "mkdir", "chmod", "chown", "trash", "restore"} {
|
||||||
|
if !have[name] {
|
||||||
|
t.Errorf("%s serves no verb %s", FilesSeat, name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for command, verb := range map[string]string{"cat": "read", "tee": "write", "cp": "copy", "mv": "move",
|
||||||
|
"rm": "delete", "mkdir -p": "mkdir", "chmod": "chmod", "chgrp": "chown"} {
|
||||||
|
if replaced[command] != verb {
|
||||||
|
t.Errorf("%q is replaced by %q, want %s.%s", command, replaced[command], FilesSeat, verb)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// issue 900, ADR 0394 §2: a holder of node-files runs as the tool runner's account, so it declares no runs-as.
|
||||||
|
func TestAHolderOfNodeFilesDeclaringRunsAsIsRefused(t *testing.T) {
|
||||||
|
p := FilesHolderProblems(Manifest{Module: "files", RunsAs: "mesh-files", Claims: []Claim{{Name: FilesSeat}}})
|
||||||
|
if len(p) != 1 || !strings.Contains(p[0], "0394") || !strings.Contains(p[0], "runs-as") {
|
||||||
|
t.Errorf("a holder declaring runs-as was not refused: %v", p)
|
||||||
|
}
|
||||||
|
if p := FilesHolderProblems(Manifest{Module: "files", Claims: []Claim{{Name: FilesSeat}}}); len(p) != 0 {
|
||||||
|
t.Errorf("a holder declaring none was refused: %v", p)
|
||||||
|
}
|
||||||
|
if p := FilesHolderProblems(Manifest{Module: "other", RunsAs: "x"}); len(p) != 0 {
|
||||||
|
t.Errorf("a non-holder was judged: %v", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// issue 900, ADR 0394 §4: the holder is told the secret places and the paths modules own on its node.
|
||||||
|
func TestTheHolderOfNodeFilesIsToldSecretPlacesAndOwnedPaths(t *testing.T) {
|
||||||
|
files := Manifest{Module: "files", Version: "1", Claims: []Claim{{Name: FilesSeat}},
|
||||||
|
OwnSecrets: OwnSecrets{"broker": {Path: "/var/lib/files/broker"}}}
|
||||||
|
host := Manifest{Module: "hostnames", Version: "1", Claims: []Claim{{Name: "node-hostname"}},
|
||||||
|
Resources: []map[string]any{{"id": "hosts", "type": "file", "path": "/etc/hosts"}}}
|
||||||
|
app := Manifest{Module: "app", Version: "1",
|
||||||
|
Resources: []map[string]any{{"id": "conf", "type": "file", "path": "/etc/app/app.conf"}}}
|
||||||
|
r := Resolution{Node: "anchor", Modules: []Manifest{files, host, app}}
|
||||||
|
words, err := r.filesWords(files, nil, Rendering{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var places []string
|
||||||
|
if err := json.Unmarshal([]byte(words[ReadsSecretPlaces]), &places); err != nil || len(places) == 0 {
|
||||||
|
t.Errorf("no secret places: %v %v", err, words)
|
||||||
|
}
|
||||||
|
var owned map[string]string
|
||||||
|
if err := json.Unmarshal([]byte(words[FilesOwnedPaths]), &owned); err != nil {
|
||||||
|
t.Fatalf("no owned paths: %v (%v)", err, words)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(owned["/etc/hosts"], "hostnames") {
|
||||||
|
t.Errorf("/etc/hosts is owned by %q, want hostnames", owned["/etc/hosts"])
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(owned["/etc/app/app.conf"], "app") {
|
||||||
|
t.Errorf("a placed file is owned by %q, want app", owned["/etc/app/app.conf"])
|
||||||
|
}
|
||||||
|
if w, _ := r.filesWords(app, nil, Rendering{}); w[FilesOwnedPaths] != "" {
|
||||||
|
t.Error("a module not holding node-files is told the owned paths")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// issue 900: restore, like every write verb, takes a reason and an override.
|
||||||
|
func TestNodeFilesRestoreTakesIdAndReason(t *testing.T) {
|
||||||
|
s, _ := SeatNamed(FilesSeat)
|
||||||
|
for _, v := range s.Serves {
|
||||||
|
if v.Name != "restore" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
props, _ := v.Input["properties"].(map[string]any)
|
||||||
|
for _, p := range []string{"id", "reason", "override"} {
|
||||||
|
if _, has := props[p]; !has {
|
||||||
|
t.Errorf("restore has no input %s", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req, _ := v.Input["required"].([]string); strings.Join(req, ",") != "id,reason" {
|
||||||
|
t.Errorf("restore requires %v, want id and reason", req)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.Error("no restore verb")
|
||||||
|
}
|
||||||
@@ -104,6 +104,9 @@ type PlanProvision struct {
|
|||||||
Provider string `json:"provider"`
|
Provider string `json:"provider"`
|
||||||
ProviderModule string `json:"provider-module"`
|
ProviderModule string `json:"provider-module"`
|
||||||
CredentialForm string `json:"credential-form,omitempty"`
|
CredentialForm string `json:"credential-form,omitempty"`
|
||||||
|
// CredentialTool is, for an offer whose credential the provider's software issues (issue 928), the provider's tool
|
||||||
|
// the vault asks for each holder's value; empty for any other form.
|
||||||
|
CredentialTool string `json:"credential-tool,omitempty"`
|
||||||
Path string `json:"path,omitempty"`
|
Path string `json:"path,omitempty"`
|
||||||
// Contributes is what the consumer's manifest tells its provider for this provision (Manifest.Contributes: a
|
// Contributes is what the consumer's manifest tells its provider for this provision (Manifest.Contributes: a
|
||||||
// database's name, its extensions), so the provider's helper learns it from the bus (issue 776). Nil where the
|
// database's name, its extensions), so the provider's helper learns it from the bus (issue 776). Nil where the
|
||||||
@@ -134,7 +137,7 @@ func provisionsOf(needs []Needed, module, as string, contributes map[string]map[
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
p := PlanProvision{Provision: n.Name, Local: n.Local, Provider: n.From, ProviderModule: n.Module,
|
p := PlanProvision{Provision: n.Name, Local: n.Local, Provider: n.From, ProviderModule: n.Module,
|
||||||
CredentialForm: n.CredentialForm, Contributes: maps.Clone(contributes[n.Name]),
|
CredentialForm: n.CredentialForm, CredentialTool: n.CredentialTool, Contributes: maps.Clone(contributes[n.Name]),
|
||||||
At: n.At, From: n.From, As: as}
|
At: n.At, From: n.From, As: as}
|
||||||
// **A credential only a template names** (issue 788): named like a file's, so the client fills it from the
|
// **A credential only a template names** (issue 788): named like a file's, so the client fills it from the
|
||||||
// vault's row for this holder.
|
// vault's row for this holder.
|
||||||
|
|||||||
@@ -123,3 +123,37 @@ func TestAHandedModulesLocalCredentialsOfTheVaultAreNamedByLocalInItsPlan(t *tes
|
|||||||
t.Fatalf("the plan names %v, want admin and internal-token", names)
|
t.Fatalf("the plan names %v, want admin and internal-token", names)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// **Issue 969: a key its provider serves only at run time.** A moved module's file may name `${bound:…}` of a
|
||||||
|
// provision whose provider tells it at run time (ssh-client's known_host of the forge). Where the machine takes
|
||||||
|
// handed files the client fills it and the module composes; where it does not the controller must fill it from what
|
||||||
|
// the provider serves statically, cannot, and refuses it with the very words an unmarked module is refused with
|
||||||
|
// (ADR 0349 §17: the old way, byte for byte).
|
||||||
|
func TestAHandedModuleNamingARunTimeBoundKeyComposesOnlyWhereItIsHanded(t *testing.T) {
|
||||||
|
text := "Host ${bound:postgres-database:at}\n KnownHost ${bound:postgres-database:known_host}\n"
|
||||||
|
moved := strings.Replace(botMoved(), `"own-secrets":{"token":"${dir:state}/token"}`,
|
||||||
|
`"requires":["postgres-database"],"own-secrets":{"token":"${dir:state}/token"}`, 1)
|
||||||
|
unmoved := strings.Replace(moved, `"moved":true,`, ``, 1)
|
||||||
|
if unmoved == moved {
|
||||||
|
t.Fatal("the twin is still marked moved")
|
||||||
|
}
|
||||||
|
|
||||||
|
c, got := composeWith(t, withDatabase(t, moved, text, true), handingTo("store:5000"))
|
||||||
|
if strings.Contains(got, "known_host") || len(c.Filled) == 0 {
|
||||||
|
t.Fatalf("handed, the module is filled by the client and the controller names nothing: %v %s", c.Filled, got)
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, with := range map[string]Rendering{
|
||||||
|
"an engine that does not take a filled file alone": {Provisioner: "mesh-provisioner-client", ArtifactStore: "store:5000"},
|
||||||
|
"no provisioner client on the machine": {Hands: true, ArtifactStore: "store:5000"},
|
||||||
|
} {
|
||||||
|
_, err := withDatabase(t, moved, text, true).Compose(sealedForBot(with))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), `asks its postgres-database binding for "known_host"`) {
|
||||||
|
t.Fatalf("%s: the key is not refused: %v", name, err)
|
||||||
|
}
|
||||||
|
_, before := withDatabase(t, unmoved, text, false).Compose(sealedForBot(with))
|
||||||
|
if before == nil || before.Error() != err.Error() {
|
||||||
|
t.Fatalf("%s: refused otherwise than the unmarked module: %v / %v", name, err, before)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -242,3 +242,33 @@ func TestAnOfferAnsweredByTheHelperIsReadAndHandsNoFile(t *testing.T) {
|
|||||||
t.Fatalf("the flag changed what is composed or handed:\nbefore: %s\nnow: %s\n%v %v", before, got, c2.Filled, c2.Plans)
|
t.Fatalf("the flag changed what is composed or handed:\nbefore: %s\nnow: %s\n%v %v", before, got, c2.Filled, c2.Plans)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A plan entry names the tool of an offer whose credential the provider's software issues (issue 928), so the vault
|
||||||
|
// asks that tool for each holder; an offer of any other form names none.
|
||||||
|
func TestAHandedModulesPlanNamesTheCredentialToolOfASoftwareIssuedOffer(t *testing.T) {
|
||||||
|
for credential, want := range map[string]string{
|
||||||
|
``: "",
|
||||||
|
`,"credential":{"own":"admin"}`: "",
|
||||||
|
`,"credential":{"issued-by":"outside"}`: "",
|
||||||
|
`,"credential":{"issued-by":"software","tool":"gitea_agent_token"}`: "gitea_agent_token",
|
||||||
|
} {
|
||||||
|
raw := `{"module":"store","version":"1","provides":[{"name":"postgres-database","scope":"mesh"` +
|
||||||
|
credential + `}],"serves":{"postgres-database":{"port":5432}}`
|
||||||
|
if strings.Contains(credential, `"own"`) {
|
||||||
|
raw += `,"own-secrets":{"admin":{"path":"/var/lib/store/admin","taken":"at-start"}}`
|
||||||
|
}
|
||||||
|
node := workstation()
|
||||||
|
node.At = "workstation.internal"
|
||||||
|
consumer := notifierConsumer(t, true, false)
|
||||||
|
r, err := Resolve(shelf(consumer, mustParse(t, raw+`}`)), []string{consumer.Module}, node, twoStores("anchor"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r.Account = "op"
|
||||||
|
c, _ := composeWith(t, r, handingTo("store:5000"))
|
||||||
|
got := c.Plans["notify"].Provisions
|
||||||
|
if len(got) != 1 || got[0].CredentialTool != want {
|
||||||
|
t.Errorf("an offer with %q: the plan names %+v, want the tool %q", credential, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
package catalogue
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A jail whose pattern names <HOST> twice is refused by name: fail2ban would read it as a duplicate
|
||||||
|
// capture group and refuse its whole configuration, leaving the machine with no bans at all — which
|
||||||
|
// is what happened on the control node the day this was built (novox/hq ADR 0179).
|
||||||
|
func TestAJailsPatternNamesTheHostOncePerShape(t *testing.T) {
|
||||||
|
twice := Manifest{Module: "route-proxy", Jails: []Jail{{
|
||||||
|
Name: "route-proxy",
|
||||||
|
Failregex: `^.*(?:refused for <HOST>:\d+|asked from <HOST>:\d+)$`,
|
||||||
|
Jail: "backend = systemd",
|
||||||
|
}}}
|
||||||
|
problems := twice.jailProblems()
|
||||||
|
if len(problems) != 1 || !strings.Contains(problems[0], "names <HOST> 2 times") {
|
||||||
|
t.Fatalf("a pattern naming <HOST> twice was not refused by name: %v", problems)
|
||||||
|
}
|
||||||
|
if !strings.Contains(problems[0], "no bans at all") {
|
||||||
|
t.Errorf("the refusal does not say what it costs: %q", problems[0])
|
||||||
|
}
|
||||||
|
|
||||||
|
// The same two shapes, one pattern each, is how it is written — and is accepted.
|
||||||
|
split := Manifest{Module: "route-proxy", Jails: []Jail{{
|
||||||
|
Name: "route-proxy",
|
||||||
|
Failregex: "^.*refused for <HOST>:\\d+$\n ^.*asked from <HOST>:\\d+$",
|
||||||
|
Jail: "backend = systemd",
|
||||||
|
}}}
|
||||||
|
if problems := split.jailProblems(); len(problems) != 0 {
|
||||||
|
t.Fatalf("one <HOST> per pattern was refused: %v", problems)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A jail with no name, no pattern, or a name another jail of the same module already took is refused.
|
||||||
|
func TestAJailIsNamedOnceAndSaysWhatAFailureLooksLike(t *testing.T) {
|
||||||
|
m := Manifest{Module: "mailu", Jails: []Jail{
|
||||||
|
{Name: "", Failregex: "^.*rip=<HOST>$"},
|
||||||
|
{Name: "mailu-front", Failregex: " "},
|
||||||
|
{Name: "mailu-front", Failregex: "^.*rip=<HOST>$"},
|
||||||
|
}}
|
||||||
|
problems := strings.Join(m.jailProblems(), "\n")
|
||||||
|
for _, want := range []string{"a jail with no name", "says nothing a failed attempt looks like", "two jails called"} {
|
||||||
|
if !strings.Contains(problems, want) {
|
||||||
|
t.Errorf("missing %q in:\n%s", want, problems)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -303,6 +303,13 @@ func (m Manifest) IssuedCredentialOf(provision string) (string, bool) {
|
|||||||
return "", false
|
return "", false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// CredentialToolOf is the tool a plan names for a provision, set only where the offer says the provider's software
|
||||||
|
// issues its credential (issue 928); empty otherwise.
|
||||||
|
func (m Manifest) CredentialToolOf(provision string) string {
|
||||||
|
tool, _ := m.IssuedCredentialOf(provision)
|
||||||
|
return tool
|
||||||
|
}
|
||||||
|
|
||||||
// IssuedOutsideOf is whether this module's offer of a provision says only a person outside the mesh can give its
|
// IssuedOutsideOf is whether this module's offer of a provision says only a person outside the mesh can give its
|
||||||
// credential (novox/hq ADR 0337): the mesh never makes a value for it, and a consumer holds one only once a person
|
// credential (novox/hq ADR 0337): the mesh never makes a value for it, and a consumer holds one only once a person
|
||||||
// gave it with `secret accept`.
|
// gave it with `secret accept`.
|
||||||
@@ -1984,6 +1991,7 @@ func parseManifest(raw []byte, predefined PredefinedList) (Manifest, error) {
|
|||||||
problems = append(problems, declaredSeatProblems(m)...)
|
problems = append(problems, declaredSeatProblems(m)...)
|
||||||
problems = append(problems, RunsAsProblems(m)...)
|
problems = append(problems, RunsAsProblems(m)...)
|
||||||
problems = append(problems, ReadsHolderProblems(m)...)
|
problems = append(problems, ReadsHolderProblems(m)...)
|
||||||
|
problems = append(problems, FilesHolderProblems(m)...)
|
||||||
if m.Computed != "" && len(m.Resources) > 0 {
|
if m.Computed != "" && len(m.Resources) > 0 {
|
||||||
// One or the other. A module that both ships files and has them computed would leave
|
// One or the other. A module that both ships files and has them computed would leave
|
||||||
// nobody able to say where a given file came from.
|
// nobody able to say where a given file came from.
|
||||||
|
|||||||
@@ -230,6 +230,8 @@ type Needed struct {
|
|||||||
// CredentialForm is the credential form the provider's offer declares (Manifest.CredentialFormOf, issue 768),
|
// CredentialForm is the credential form the provider's offer declares (Manifest.CredentialFormOf, issue 768),
|
||||||
// named in a handed consumer's plan for the vault; empty where the provider's manifest is not in hand.
|
// named in a handed consumer's plan for the vault; empty where the provider's manifest is not in hand.
|
||||||
CredentialForm string
|
CredentialForm string
|
||||||
|
// CredentialTool is the tool of the offer that the provider's software issues the credential through (issue 928).
|
||||||
|
CredentialTool string
|
||||||
// Manager is set when this holder is a refreshable-grant licence's MANAGER, delivered the refresh
|
// Manager is set when this holder is a refreshable-grant licence's MANAGER, delivered the refresh
|
||||||
// token rather than an access token (novox/hq ADR 0050). It changes one thing downstream: an empty
|
// token rather than an access token (novox/hq ADR 0050). It changes one thing downstream: an empty
|
||||||
// Sealed is tolerated — the manager has not adopted a refresh token yet, which is a real waiting
|
// Sealed is tolerated — the manager has not adopted a refresh token yet, which is a real waiting
|
||||||
@@ -543,7 +545,7 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
|
|||||||
needs = append(needs, Needed{
|
needs = append(needs, Needed{
|
||||||
Name: want, From: node.Name, At: at, Module: by.Module,
|
Name: want, From: node.Name, At: at, Module: by.Module,
|
||||||
Serves: servedByOne(by, want), For: because[want],
|
Serves: servedByOne(by, want), For: because[want],
|
||||||
SharedOwn: sharedByOne(by, want), CredentialForm: by.CredentialFormOf(want),
|
SharedOwn: sharedByOne(by, want), CredentialForm: by.CredentialFormOf(want), CredentialTool: by.CredentialToolOf(want),
|
||||||
Identity: by.IdentityBoundOf(want)})
|
Identity: by.IdentityBoundOf(want)})
|
||||||
} else if served := servedByOne(by, want); len(served) > 0 {
|
} else if served := servedByOne(by, want); len(served) > 0 {
|
||||||
// Answered here with no credential to mint, but the provider serves facts the
|
// Answered here with no credential to mint, but the provider serves facts the
|
||||||
@@ -565,7 +567,7 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
|
|||||||
}
|
}
|
||||||
needs = append(needs, Needed{
|
needs = append(needs, Needed{
|
||||||
Name: want, From: node.Name, At: at, Serves: served, For: because[want], Module: by.Module,
|
Name: want, From: node.Name, At: at, Serves: served, For: because[want], Module: by.Module,
|
||||||
CredentialForm: by.CredentialFormOf(want)})
|
CredentialForm: by.CredentialFormOf(want), CredentialTool: by.CredentialToolOf(want)})
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -594,17 +596,17 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
|
|||||||
node.Name, want, p.Node, meshNetwork))
|
node.Name, want, p.Node, meshNetwork))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
shared, form := "", ""
|
shared, form, tool := "", "", ""
|
||||||
// A provider whose definition is not in hand is held to the tightest bound the
|
// A provider whose definition is not in hand is held to the tightest bound the
|
||||||
// mesh knows, not to none: what it keeps is not known here (ADR 0225).
|
// mesh knows, not to none: what it keeps is not known here (ADR 0225).
|
||||||
bound := DefaultIdentityBound
|
bound := DefaultIdentityBound
|
||||||
if pm, known := catalogue[p.Module]; known {
|
if pm, known := catalogue[p.Module]; known {
|
||||||
shared, _ = pm.SharedCredentialOf(want)
|
shared, _ = pm.SharedCredentialOf(want)
|
||||||
form = pm.CredentialFormOf(want)
|
form, tool = pm.CredentialFormOf(want), pm.CredentialToolOf(want)
|
||||||
bound = pm.IdentityBoundOf(want)
|
bound = pm.IdentityBoundOf(want)
|
||||||
}
|
}
|
||||||
needs = append(needs, Needed{Name: want, From: p.Node, At: p.At, Module: p.Module,
|
needs = append(needs, Needed{Name: want, From: p.Node, At: p.At, Module: p.Module,
|
||||||
Serves: p.Serves, For: because[want], SharedOwn: shared, CredentialForm: form, Identity: bound,
|
Serves: p.Serves, For: because[want], SharedOwn: shared, CredentialForm: form, CredentialTool: tool, Identity: bound,
|
||||||
LeftOut: p.LeftOut})
|
LeftOut: p.LeftOut})
|
||||||
}
|
}
|
||||||
switch {
|
switch {
|
||||||
|
|||||||
@@ -178,6 +178,10 @@ func (r Resolution) runtimeProcess(with Rendering) (map[string]any, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
// The holder of node-files is told the same, and the paths modules own here (novox/hq ADR 0394 §4).
|
||||||
|
if words, err = r.filesWords(m, words, with); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
if len(words) > 0 {
|
if len(words) > 0 {
|
||||||
given[m.Module] = words
|
given[m.Module] = words
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -421,6 +421,9 @@ var defaultSeats = append([]Seat{
|
|||||||
// A node's files and accounts, read (novox/hq ADR 0310, reads.go): by an account of the holder's own that holds
|
// A node's files and accounts, read (novox/hq ADR 0310, reads.go): by an account of the holder's own that holds
|
||||||
// no secret, on every node, so the reads ADR 0268 §5 takes over from the login shell's execute have a tool.
|
// no secret, on every node, so the reads ADR 0268 §5 takes over from the login shell's execute have a tool.
|
||||||
{Name: ReadsSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0310", Serves: readsVerbs()},
|
{Name: ReadsSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0310", Serves: readsVerbs()},
|
||||||
|
// A node's files, read and managed (novox/hq ADR 0394, files.go): beside node-reads until mesh-files is assigned
|
||||||
|
// on every node, then node-reads goes (ADR 0394 §9).
|
||||||
|
{Name: FilesSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0394", Serves: filesVerbs()},
|
||||||
},
|
},
|
||||||
// The graphical session's roles (novox/hq ADR 0208), last because they are a workstation's.
|
// The graphical session's roles (novox/hq ADR 0208), last because they are a workstation's.
|
||||||
graphicalSessionSeats()...)
|
graphicalSessionSeats()...)
|
||||||
|
|||||||
@@ -48,6 +48,7 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
|
|||||||
delivered[s.Delivers] = s.Name
|
delivered[s.Delivers] = s.Name
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// Forty-five with node-files (novox/hq ADR 0394); forty-four again once node-reads goes.
|
||||||
// Forty-four with node-provisioner (novox/hq ADR 0349 clause 3); forty-three with mesh-provisioner (novox/hq ADR 0349 clause 3); forty-two with mesh-config (novox/hq ADR 0348 §9); forty-one with node-reads (novox/hq ADR 0310); forty with node-nfs-server and node-mounts (novox/hq ADR 0263); thirty-eight with node-resolver (novox/hq ADR 0247); thirty-seven with mesh-delivery (novox/hq ADR 0239); thirty-six since node-resolver-config retired
|
// Forty-four with node-provisioner (novox/hq ADR 0349 clause 3); forty-three with mesh-provisioner (novox/hq ADR 0349 clause 3); forty-two with mesh-config (novox/hq ADR 0348 §9); forty-one with node-reads (novox/hq ADR 0310); forty with node-nfs-server and node-mounts (novox/hq ADR 0263); thirty-eight with node-resolver (novox/hq ADR 0247); thirty-seven with mesh-delivery (novox/hq ADR 0239); thirty-six since node-resolver-config retired
|
||||||
// into node-uplink (novox/hq ADR 0223); thirty-seven
|
// into node-uplink (novox/hq ADR 0223); thirty-seven
|
||||||
// since the retired node-dns-resolver went (novox/hq ADR 0220); thirty-eight with
|
// since the retired node-dns-resolver went (novox/hq ADR 0220); thirty-eight with
|
||||||
@@ -59,8 +60,8 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
|
|||||||
// node-container-runtime (ADR 0207); nineteen with node-environment and node-login-shell (ADR 0203,
|
// node-container-runtime (ADR 0207); nineteen with node-environment and node-login-shell (ADR 0203,
|
||||||
// ADR 0204); seventeen with node-build-agent (ADR 0190). One fewer once the retired
|
// ADR 0204); seventeen with node-build-agent (ADR 0190). One fewer once the retired
|
||||||
// mesh-build-machine row goes, when no registered manifest claims it.
|
// mesh-build-machine row goes, when no registered manifest claims it.
|
||||||
if len(Seats()) != 44 {
|
if len(Seats()) != 45 {
|
||||||
t.Errorf("the mesh defines %d seats rather than 44; the set is closed, so a change here is "+
|
t.Errorf("the mesh defines %d seats rather than 45; the set is closed, so a change here is "+
|
||||||
"a decision (novox/hq ADR 0110): %s", len(Seats()), seatNames())
|
"a decision (novox/hq ADR 0110): %s", len(Seats()), seatNames())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,19 +1,66 @@
|
|||||||
package catalogue
|
package catalogue
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"io/fs"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/novox/mesh-controller/internal/beside"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// boundKeys are every key the files of a module ask of ${bound:forge-account:<key>}, read from the module's own
|
||||||
|
// tree on the shelf, so the test offers what ssh-client asks however often it asks one more (issue 985).
|
||||||
|
func boundKeys(t *testing.T, module, provision string) map[string]bool {
|
||||||
|
t.Helper()
|
||||||
|
ask := regexp.MustCompile(`\$\{bound:` + regexp.QuoteMeta(provision) + `:([A-Za-z0-9_.-]+)\}`)
|
||||||
|
keys := map[string]bool{}
|
||||||
|
root := filepath.Join(beside.Catalogue(t), module)
|
||||||
|
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
|
||||||
|
if err != nil || d.IsDir() || strings.HasSuffix(path, "_test.go") {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
body, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, m := range ask.FindAllStringSubmatch(string(body), -1) {
|
||||||
|
keys[m[1]] = true
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("reading what %s asks of %s: %v", module, provision, err)
|
||||||
|
}
|
||||||
|
return keys
|
||||||
|
}
|
||||||
|
|
||||||
// The ssh-client module, composed as a machine receives it (novox/hq to-be 29, research 027/03): a
|
// The ssh-client module, composed as a machine receives it (novox/hq to-be 29, research 027/03): a
|
||||||
// region at the START of the operator's ~/.ssh/config that includes ~/.ssh/config.d/* — first,
|
// region at the START of the operator's ~/.ssh/config that includes ~/.ssh/config.d/* — first,
|
||||||
// because ssh takes the first value it finds for each option — and the mesh's Host blocks as the
|
// because ssh takes the first value it finds for each option — and the mesh's Host blocks as the
|
||||||
// whole of ~/.ssh/config.d/00-mesh, every other node with its account. ~/.ssh and ~/.ssh/config.d
|
// whole of ~/.ssh/config.d/00-mesh, every other node with its account. ~/.ssh and ~/.ssh/config.d
|
||||||
// are created 0700 and owned by the account; the operator's own config below the region is kept.
|
// are created 0700 and owned by the account; the operator's own config below the region is kept.
|
||||||
func TestSSHClientOwnsTheOperatorsSSHConfig(t *testing.T) {
|
func TestSSHClientOwnsTheOperatorsSSHConfig(t *testing.T) {
|
||||||
shelf := shelf(catalogueManifest(t, "ssh-client"))
|
shelf := shelf(catalogueManifest(t, "ssh-client"), catalogueManifest(t, "gitea"))
|
||||||
|
// ssh-client requires forge-account since mesh-catalog #340 (novox/hq issue 955), answered by the forge's
|
||||||
|
// holder on another node; this test is about the operator's config, so the forge is offered from there
|
||||||
|
// (issue 985).
|
||||||
|
// Every key ssh-client's files ask is offered, read from them (boundKeys); a few have a realistic value.
|
||||||
|
serves := map[string]any{}
|
||||||
|
for key := range boundKeys(t, "ssh-client", "forge-account") {
|
||||||
|
serves[key] = "sample-" + key
|
||||||
|
}
|
||||||
|
for key, value := range map[string]any{"ssh_host": "marge.internal", "ssh_port": 222,
|
||||||
|
"known_host": "[marge.internal]:222 ssh-ed25519 AAAA", "ssh_base": "ssh://git@marge.internal:222/",
|
||||||
|
"https_base": "https://git.example.org/"} {
|
||||||
|
serves[key] = value
|
||||||
|
}
|
||||||
|
forge := World{Offered: map[string][]Provider{
|
||||||
|
"forge-account": {{Node: "marge", At: "marge.internal", Module: "gitea", Serves: serves}}}}
|
||||||
got, err := Resolve(shelf, []string{"ssh-client"},
|
got, err := Resolve(shelf, []string{"ssh-client"},
|
||||||
Node{Name: "homer", At: "homer.internal", Account: "jo", AccountHome: "/home/jo"}, World{})
|
Node{Name: "homer", At: "homer.internal", Account: "jo", AccountHome: "/home/jo"}, forge)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -112,6 +112,10 @@ type Machine struct {
|
|||||||
// NodeEngine and NodeTools are the builds it runs.
|
// NodeEngine and NodeTools are the builds it runs.
|
||||||
NodeEngine string `json:"node-engine,omitempty"`
|
NodeEngine string `json:"node-engine,omitempty"`
|
||||||
NodeTools string `json:"node-tools,omitempty"`
|
NodeTools string `json:"node-tools,omitempty"`
|
||||||
|
// Contract is the contract of the node-engine's newest health statement (0 when it never stated one): what
|
||||||
|
// says whether the machine takes handed files (novox/hq issue 969). A check that composed it without would
|
||||||
|
// judge a moved module the old way where the mesh hands it.
|
||||||
|
Contract int `json:"contract,omitempty"`
|
||||||
// Capabilities are what it reported it can do; the detail kept only where it is a version.
|
// Capabilities are what it reported it can do; the detail kept only where it is a version.
|
||||||
Capabilities []Capability `json:"capabilities,omitempty"`
|
Capabilities []Capability `json:"capabilities,omitempty"`
|
||||||
// Site, Hub, Public: where it is on the private network — its site (a pseudonym), whether it is the
|
// Site, Hub, Public: where it is on the private network — its site (a pseudonym), whether it is the
|
||||||
|
|||||||
@@ -119,6 +119,8 @@ type PlanBatch struct {
|
|||||||
ClosesAt time.Time `json:"closes_at"`
|
ClosesAt time.Time `json:"closes_at"`
|
||||||
AtMost time.Time `json:"at_most"`
|
AtMost time.Time `json:"at_most"`
|
||||||
Behind string `json:"behind,omitempty"`
|
Behind string `json:"behind,omitempty"`
|
||||||
|
// BehindWhy says why the batch waits behind that walk: what the two share (issue 973).
|
||||||
|
BehindWhy string `json:"behind_why,omitempty"`
|
||||||
// Own says the batch holds merges on the controller's own path, whose walk waits for nobody's word: such
|
// Own says the batch holds merges on the controller's own path, whose walk waits for nobody's word: such
|
||||||
// a merge never shares a batch with one that waits for mesh-delivery's (ADR 0276, decided during the
|
// a merge never shares a batch with one that waits for mesh-delivery's (ADR 0276, decided during the
|
||||||
// build, 2026-10-10).
|
// build, 2026-10-10).
|
||||||
|
|||||||
@@ -81,18 +81,10 @@ type ProvisionedCollection struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ProvisionedCredential is one provision the module requires, under one local name, and its provider (issue 760);
|
// ProvisionedCredential is one provision the module requires, under one local name, and its provider (issue 760);
|
||||||
// with a name, a credential issued for this holder alone (issue 757).
|
// with a name, a credential issued for this holder alone (issue 757). It is the composition's own type, not a copy:
|
||||||
type ProvisionedCredential struct {
|
// the plan is kept and read back as the one it was sent as, so no field of it (contributes, at, from, as, names) can
|
||||||
Name string `json:"name,omitempty"`
|
// be lost between the two (issue 923).
|
||||||
Provision string `json:"provision"`
|
type ProvisionedCredential = catalogue.PlanProvision
|
||||||
Local string `json:"local,omitempty"`
|
|
||||||
Provider string `json:"provider"`
|
|
||||||
ProviderModule string `json:"provider-module"`
|
|
||||||
// CredentialForm is the form the provider's offer declares for its credential (issue 768): shared, pair,
|
|
||||||
// issued or given; empty when the provider's manifest was not in hand.
|
|
||||||
CredentialForm string `json:"credential-form,omitempty"`
|
|
||||||
Path string `json:"path,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// ProvisionedSecret is one of the module's own secrets, by its name in the manifest. Path, when set, is the file
|
// ProvisionedSecret is one of the module's own secrets, by its name in the manifest. Path, when set, is the file
|
||||||
// holding the secret alone (the manifest's `own-secrets`), which the client fills with its value and nothing else.
|
// holding the secret alone (the manifest's `own-secrets`), which the client fills with its value and nothing else.
|
||||||
|
|||||||
@@ -352,3 +352,42 @@ func TestAPlanMayHoldSeveralNamedItemsOfOneCollectionAndRefusesABadOne(t *testin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// **A kept plan's provisions keep every field the composition sets** (issue 923): the names, the place, the machine
|
||||||
|
// and the identity the provider's helper writes into the consumer's entry, and what the manifest contributes, are
|
||||||
|
// read back as they were sent, so the assignments mirror carries them to the provider.
|
||||||
|
func TestAKeptPlansProvisionsKeepContributesAtFromAsAndNames(t *testing.T) {
|
||||||
|
sent := catalogue.PlanProvision{Provision: "route", Provider: "novox", ProviderModule: "route-proxy",
|
||||||
|
Contributes: map[string]any{"database": "gitea"}, At: "10.0.0.1", From: "novox", As: "gitea@novox",
|
||||||
|
Names: map[string]string{"name-web": "git.example.org", "internal-name": "gitea.mesh"}}
|
||||||
|
raw, err := json.Marshal(map[string]any{"gitea": map[string]any{
|
||||||
|
"provisions": []catalogue.PlanProvision{sent}, "required-by": "forge"}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var kept map[string]keptPlan
|
||||||
|
if err := json.Unmarshal(raw, &kept); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got := kept["gitea"].Provisions
|
||||||
|
if len(got) != 1 || !reflect.DeepEqual(got[0], sent) {
|
||||||
|
t.Fatalf("the kept plan's provisions are %+v, want %+v", got, sent)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A kept plan's provision keeps the tool the vault asks for a software-issued credential (issue 928).
|
||||||
|
func TestAKeptPlansProvisionKeepsItsCredentialTool(t *testing.T) {
|
||||||
|
sent := catalogue.PlanProvision{Provision: "gitea-read", Provider: "novox", ProviderModule: "gitea",
|
||||||
|
CredentialForm: catalogue.CredentialFormIssued, CredentialTool: "gitea_agent_token"}
|
||||||
|
raw, err := json.Marshal(map[string]any{"claude-code": map[string]any{"provisions": []catalogue.PlanProvision{sent}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var kept map[string]keptPlan
|
||||||
|
if err := json.Unmarshal(raw, &kept); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := kept["claude-code"].Provisions; len(got) != 1 || got[0].CredentialTool != "gitea_agent_token" {
|
||||||
|
t.Fatalf("the kept plan's provisions are %+v, want the tool kept", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -145,7 +145,7 @@ func MeshNamed(stream, name string) bool {
|
|||||||
return true
|
return true
|
||||||
case strings.HasPrefix(stream, "SEAT_"):
|
case strings.HasPrefix(stream, "SEAT_"):
|
||||||
return strings.HasSuffix(name, "_worker")
|
return strings.HasSuffix(name, "_worker")
|
||||||
case name == broker.ControllerName:
|
case name == broker.ControllerName || name == broker.MergesConsumerName:
|
||||||
return true
|
return true
|
||||||
case stream == broker.EventsStream:
|
case stream == broker.EventsStream:
|
||||||
return strings.Contains(name, "_")
|
return strings.Contains(name, "_")
|
||||||
@@ -156,7 +156,7 @@ func MeshNamed(stream, name string) bool {
|
|||||||
// ConsumerInWords is a durable consumer as the mesh says it: whose, and for what.
|
// ConsumerInWords is a durable consumer as the mesh says it: whose, and for what.
|
||||||
func ConsumerInWords(stream, name string) string {
|
func ConsumerInWords(stream, name string) string {
|
||||||
switch {
|
switch {
|
||||||
case name == broker.ControllerName:
|
case name == broker.ControllerName || name == broker.MergesConsumerName:
|
||||||
return "the controller's consumer on " + stream
|
return "the controller's consumer on " + stream
|
||||||
case stream == "NODES":
|
case stream == "NODES":
|
||||||
return "how " + name + " hears what it should be (its declaration consumer)"
|
return "how " + name + " hears what it should be (its declaration consumer)"
|
||||||
|
|||||||
@@ -179,6 +179,19 @@ func (b OverNATS) PublishMembership(ctx context.Context, node, module string, bo
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// PublishMembershipAfter issues a membership only if the one read at sequence seq is still the newest of its
|
||||||
|
// subject: a push that issued another since is not overwritten, and the refusal comes back (novox/hq issue 930).
|
||||||
|
func (b OverNATS) PublishMembershipAfter(ctx context.Context, node, module string, body []byte, seq uint64) error {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, MembershipWait)
|
||||||
|
defer cancel()
|
||||||
|
_, err := b.JS.Publish(broker.MembershipSubject(node, module), body, nats.Context(ctx),
|
||||||
|
nats.ExpectLastSequencePerSubject(seq))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("issuing %s on %s its membership after sequence %d: %w", module, node, seq, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (b OverNATS) PublishDeclaration(ctx context.Context, node string, body []byte) error {
|
func (b OverNATS) PublishDeclaration(ctx context.Context, node string, body []byte) error {
|
||||||
_, err := b.JS.Publish(DeclareSubject(node), body, nats.Context(ctx))
|
_, err := b.JS.Publish(DeclareSubject(node), body, nats.Context(ctx))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -68,6 +68,7 @@ type fakeControl struct {
|
|||||||
on *fakeInbound
|
on *fakeInbound
|
||||||
redelivered bool
|
redelivered bool
|
||||||
since time.Time
|
since time.Time
|
||||||
|
holdFails error
|
||||||
}
|
}
|
||||||
|
|
||||||
func (m *fakeControl) Kind() string { return m.kind }
|
func (m *fakeControl) Kind() string { return m.kind }
|
||||||
@@ -88,6 +89,9 @@ func (m *fakeControl) HeldFor() time.Duration {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (m *fakeControl) Hold(time.Duration) error {
|
func (m *fakeControl) Hold(time.Duration) error {
|
||||||
|
if m.holdFails != nil {
|
||||||
|
return m.holdFails
|
||||||
|
}
|
||||||
if m.since.IsZero() {
|
if m.since.IsZero() {
|
||||||
m.since = time.Now()
|
m.since = time.Now()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -148,6 +148,24 @@ func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Con
|
|||||||
defer func() { _ = followed.Unsubscribe() }()
|
defer func() { _ = followed.Unsubscribe() }()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// And the forge's merges, off a consumer of their own: no max-deliver and no reset, so a merge held for the
|
||||||
|
// plans waits for them however long (novox/hq ADR 0407 decision 3). Bound to the same channel: one loop.
|
||||||
|
if n.follows[KindSourceMoved] {
|
||||||
|
if events == nil {
|
||||||
|
events = make(chan *nats.Msg, Prefetch)
|
||||||
|
}
|
||||||
|
merges, err := standingBy(ctx, log.Default(), "EVENTS", func() (*nats.Subscription, error) {
|
||||||
|
return js.ChanSubscribe(MergedSubject, events, nats.Bind("EVENTS", broker.MergesConsumerName))
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("subscribing to the forge's merges: %w", err)
|
||||||
|
}
|
||||||
|
if merges == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
defer func() { _ = merges.Unsubscribe() }()
|
||||||
|
}
|
||||||
|
|
||||||
// A connection that dropped is said, not discovered. A controller whose bus connection is gone
|
// A connection that dropped is said, not discovered. A controller whose bus connection is gone
|
||||||
// is a mesh where nothing can be told anything.
|
// is a mesh where nothing can be told anything.
|
||||||
gone := make(chan error, 1)
|
gone := make(chan error, 1)
|
||||||
|
|||||||
@@ -590,3 +590,77 @@ func TestNatsAReportThatIsNotNewsIsNotStated(t *testing.T) {
|
|||||||
type sameAgain struct{}
|
type sameAgain struct{}
|
||||||
|
|
||||||
func (sameAgain) Heard(context.Context, Report) (bool, error) { return false, nil }
|
func (sameAgain) Heard(context.Context, Report) (bool, error) { return false, nil }
|
||||||
|
|
||||||
|
// manyTimesBusy is an upgrader whose plans are busy for its first merges, as another controller's hold is.
|
||||||
|
type manyTimesBusy struct {
|
||||||
|
toldAbout
|
||||||
|
busy int
|
||||||
|
calls int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *manyTimesBusy) SourceMoved(context.Context, SourceMoved) error {
|
||||||
|
u.mu.Lock()
|
||||||
|
defer u.mu.Unlock()
|
||||||
|
u.calls++
|
||||||
|
if u.calls <= u.busy {
|
||||||
|
return errors.Join(ErrTryAgain, errors.New("another controller is working the plans"))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *manyTimesBusy) called() int {
|
||||||
|
u.mu.Lock()
|
||||||
|
defer u.mu.Unlock()
|
||||||
|
return u.calls
|
||||||
|
}
|
||||||
|
|
||||||
|
// **A merge held more than five times is still handed over and recorded** (issue 973, novox/hq ADR 0407 decision
|
||||||
|
// 3): the merges have a durable consumer of their own with no max-deliver, so they never go to DEAD_LETTERS to wait
|
||||||
|
// for a person; the delay grows, and is capped.
|
||||||
|
func TestNatsAMergeHeldMoreThanFiveTimesIsStillHandedOverAndRecorded(t *testing.T) {
|
||||||
|
first, most := MergeHeldFirst, MergeHeldMost
|
||||||
|
MergeHeldFirst, MergeHeldMost = 5*time.Millisecond, 40*time.Millisecond
|
||||||
|
t.Cleanup(func() { MergeHeldFirst, MergeHeldMost = first, most })
|
||||||
|
js := aBus(t)
|
||||||
|
up := &manyTimesBusy{busy: 8}
|
||||||
|
s := &Server{inbound: Nats(js), bus: OverNATS{Conn: js.Conn(), JS: js.Context()}, log: quiet()}
|
||||||
|
if err := s.Follows(up); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx, stop := context.WithCancel(context.Background())
|
||||||
|
defer stop()
|
||||||
|
go func() { _ = s.Serve(ctx) }()
|
||||||
|
eventually(t, "the merges consumer being made", func() bool {
|
||||||
|
_, err := js.Context().ConsumerInfo("EVENTS", broker.MergesConsumerName)
|
||||||
|
return err == nil
|
||||||
|
})
|
||||||
|
|
||||||
|
body, _ := json.Marshal(SourceMoved{Owner: "novox", Repo: "one", Base: "main", Commit: "abcdef0123"})
|
||||||
|
if _, err := js.Context().Publish(MergedSubject, body); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
eventually(t, "the ninth hand-over recording the merge", func() bool { return up.called() == 9 })
|
||||||
|
eventually(t, "the merge being acknowledged", func() bool {
|
||||||
|
info, err := js.Context().ConsumerInfo("EVENTS", broker.MergesConsumerName)
|
||||||
|
return err == nil && info.NumAckPending == 0 && info.NumPending == 0
|
||||||
|
})
|
||||||
|
if info, err := js.Context().StreamInfo(broker.DeadLettersStream); err == nil && info.State.Msgs != 0 {
|
||||||
|
t.Fatalf("%d message(s) in dead letters: a merge was given up on", info.State.Msgs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The delay a held merge waits grows from the first to the most and stays there.
|
||||||
|
func TestAHeldMergesDelayGrowsAndIsCapped(t *testing.T) {
|
||||||
|
var got []time.Duration
|
||||||
|
for _, held := range []time.Duration{0, 4 * time.Second, 10 * time.Second, 25 * time.Second, 70 * time.Second,
|
||||||
|
200 * time.Second, 20 * time.Minute} {
|
||||||
|
got = append(got, MergeRedeliverAfter(held))
|
||||||
|
}
|
||||||
|
want := []time.Duration{5 * time.Second, 5 * time.Second, 10 * time.Second, 25 * time.Second, 70 * time.Second,
|
||||||
|
200 * time.Second, 5 * time.Minute}
|
||||||
|
for i := range want {
|
||||||
|
if got[i] != want[i] {
|
||||||
|
t.Errorf("held %d: waits %v; want %v", i, got[i], want[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+38
-3
@@ -636,8 +636,25 @@ func short(commit string) string {
|
|||||||
return commit
|
return commit
|
||||||
}
|
}
|
||||||
|
|
||||||
// sourceMoved acts on the forge's announcement of a merge. Taken whatever happens: a build that
|
// MergeHeldFirst is how long a merge the controller could not record yet waits before the bus hands it over
|
||||||
// fails is reported by the build itself, and re-delivering the merge would only re-fail it.
|
// again, and MergeHeldMost the longest it ever waits: the delay grows with how long the merge has been held, from
|
||||||
|
// the first to the most (novox/hq ADR 0407 decision 3). Variables so a test need not wait.
|
||||||
|
var (
|
||||||
|
MergeHeldFirst = 5 * time.Second
|
||||||
|
MergeHeldMost = 5 * time.Minute
|
||||||
|
)
|
||||||
|
|
||||||
|
// MergeRedeliverAfter is the delay for a merge held for the time given: that time, so the delay about doubles
|
||||||
|
// with each hold, between MergeHeldFirst and MergeHeldMost.
|
||||||
|
func MergeRedeliverAfter(heldFor time.Duration) time.Duration {
|
||||||
|
return min(max(heldFor, MergeHeldFirst), MergeHeldMost)
|
||||||
|
}
|
||||||
|
|
||||||
|
// sourceMoved acts on the forge's announcement of a merge. **A merge is never dropped** (issue 921, novox/hq ADR
|
||||||
|
// 0407 decision 3): one the controller could not record for now (the plans held by another controller, the store
|
||||||
|
// away: ErrTryAgain) is held and handed over again until it is recorded; the controller giving up on it would
|
||||||
|
// lose a change for good. Anything else is an answer and is taken: a build that fails is reported by the build
|
||||||
|
// itself, and re-delivering the merge would only re-fail it.
|
||||||
func (s *Server) sourceMoved(ctx context.Context, m Control) {
|
func (s *Server) sourceMoved(ctx context.Context, m Control) {
|
||||||
var moved SourceMoved
|
var moved SourceMoved
|
||||||
if err := json.Unmarshal(m.Body(), &moved); err != nil {
|
if err := json.Unmarshal(m.Body(), &moved); err != nil {
|
||||||
@@ -650,7 +667,25 @@ func (s *Server) sourceMoved(ctx context.Context, m Control) {
|
|||||||
_ = m.Took()
|
_ = m.Took()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if err := s.upgrader.SourceMoved(ctx, moved); err != nil {
|
err := s.upgrader.SourceMoved(ctx, moved)
|
||||||
|
// Left unsettled when the controller is ending, whatever error that gave (the plans' hold returns a bare
|
||||||
|
// context.Canceled): the bus hands it to whatever consumes next, and taking it would lose the merge.
|
||||||
|
if err != nil && ctx.Err() != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil && errors.Is(err, ErrTryAgain) {
|
||||||
|
// Also left unsettled when the bus will not hold it.
|
||||||
|
delay := MergeRedeliverAfter(m.HeldFor())
|
||||||
|
if held := m.Hold(delay); held != nil {
|
||||||
|
s.log.Printf("%s/%s moved to %.8s and could not be held (%v): %v; left for the bus to hand over again",
|
||||||
|
moved.Owner, moved.Repo, moved.Commit, held, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.log.Printf("%s/%s moved to %.8s and is held: %v; handed over again in %s", moved.Owner, moved.Repo,
|
||||||
|
moved.Commit, err, delay)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
s.log.Printf("%s/%s moved to %.8s and the mesh could not act on it: %v", moved.Owner, moved.Repo, moved.Commit, err)
|
s.log.Printf("%s/%s moved to %.8s and the mesh could not act on it: %v", moved.Owner, moved.Repo, moved.Commit, err)
|
||||||
}
|
}
|
||||||
_ = m.Took()
|
_ = m.Took()
|
||||||
|
|||||||
@@ -106,3 +106,61 @@ func TestAMessageHandledDuringShutdownIsLeftForTheBus(t *testing.T) {
|
|||||||
t.Fatalf("a build result handled during shutdown was settled, and so lost: %+v", *to)
|
t.Fatalf("a build result handled during shutdown was settled, and so lost: %+v", *to)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A merge the controller could not record for now (the plans held by another controller, the store away) is held
|
||||||
|
// and handed over again, never taken: the bus keeps it until it is recorded (issue 921). Any other failure is an
|
||||||
|
// answer and is taken, as before.
|
||||||
|
func TestAMergeThatCouldNotBeRecordedNowIsHeldNotDropped(t *testing.T) {
|
||||||
|
merged := SourceMoved{Owner: "novox", Repo: "one", Base: "main", Commit: "abcdef0123"}
|
||||||
|
for _, c := range []struct {
|
||||||
|
what string
|
||||||
|
err error
|
||||||
|
want func(*settled) bool
|
||||||
|
}{
|
||||||
|
{"the plans held by another controller", errors.Join(ErrTryAgain, errors.New("another controller is working the plans")),
|
||||||
|
func(s *settled) bool { return s.unsettled() }},
|
||||||
|
{"the store away", errors.Join(ErrTryAgain, restarting), func(s *settled) bool { return s.unsettled() }},
|
||||||
|
{"cannot act", errors.New("no such module"), func(s *settled) bool { return s.acked && !s.nacked }},
|
||||||
|
{"kept", nil, func(s *settled) bool { return s.acked && !s.nacked }},
|
||||||
|
} {
|
||||||
|
s, in := serving()
|
||||||
|
s.upgrader = upgradesWith{err: c.err}
|
||||||
|
to := &settled{}
|
||||||
|
held := in.sends(t, to, KindSourceMoved, merged).(*fakeControl)
|
||||||
|
s.act(context.Background(), held)
|
||||||
|
if !c.want(to) {
|
||||||
|
t.Errorf("%s: a merge was settled as %+v", c.what, *to)
|
||||||
|
}
|
||||||
|
if c.err != nil && errors.Is(c.err, ErrTryAgain) && len(in.held) != 1 {
|
||||||
|
t.Errorf("%s: the merge was not held to be handed over again", c.what)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A merge being held when the controller is replaced (its context ended) or when the bus will not hold it is left
|
||||||
|
// unsettled, never taken: the bus hands it to whatever consumes next (review of issue 921).
|
||||||
|
func TestAMergeIsLeftUnsettledWhenTheControllerEndsOrTheHoldFails(t *testing.T) {
|
||||||
|
merged := SourceMoved{Owner: "novox", Repo: "one", Base: "main", Commit: "abcdef0123"}
|
||||||
|
try := errors.Join(ErrTryAgain, errors.New("another controller is working the plans"))
|
||||||
|
|
||||||
|
// The plans' hold returns a bare context.Canceled when the controller is replaced, not ErrTryAgain.
|
||||||
|
s, in := serving()
|
||||||
|
s.upgrader = upgradesWith{err: context.Canceled}
|
||||||
|
to := &settled{}
|
||||||
|
ended, end := context.WithCancel(context.Background())
|
||||||
|
end()
|
||||||
|
s.act(ended, in.sends(t, to, KindSourceMoved, merged))
|
||||||
|
if !to.unsettled() {
|
||||||
|
t.Errorf("a merge was settled as %+v when the controller's context ended", *to)
|
||||||
|
}
|
||||||
|
|
||||||
|
s, in = serving()
|
||||||
|
s.upgrader = upgradesWith{err: try}
|
||||||
|
to = &settled{}
|
||||||
|
m := in.sends(t, to, KindSourceMoved, merged).(*fakeControl)
|
||||||
|
m.holdFails = errors.New("the bus will not hold it")
|
||||||
|
s.act(context.Background(), m)
|
||||||
|
if !to.unsettled() {
|
||||||
|
t.Errorf("a merge was settled as %+v when it could not be held", *to)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
Vendored
+6
@@ -8,6 +8,7 @@
|
|||||||
./cmd/mesh-controller TestAStopWithoutAMergeLeavesItOutOfABatch
|
./cmd/mesh-controller TestAStopWithoutAMergeLeavesItOutOfABatch
|
||||||
./cmd/mesh-controller TestSavingPlansAndMovingMergesIsOneAct
|
./cmd/mesh-controller TestSavingPlansAndMovingMergesIsOneAct
|
||||||
./cmd/mesh-controller TestAPlanFollowedFromTheStoreAnswersWhenItMoves
|
./cmd/mesh-controller TestAPlanFollowedFromTheStoreAnswersWhenItMoves
|
||||||
|
./cmd/mesh-controller TestIssue969TheGateRaisesEachMachineAsItsEngineStatedItself
|
||||||
./internal/inventory TestTheLastSendSaysWhatTheMachineAnswered
|
./internal/inventory TestTheLastSendSaysWhatTheMachineAnswered
|
||||||
./internal/inventory TestAReportIsTheLastSendsAnswerOnlyWhenItIsAboutThatSend
|
./internal/inventory TestAReportIsTheLastSendsAnswerOnlyWhenItIsAboutThatSend
|
||||||
./cmd/mesh-controller TestNodeShowAndStatusReadTheLastSendFromTheStore
|
./cmd/mesh-controller TestNodeShowAndStatusReadTheLastSendFromTheStore
|
||||||
@@ -127,3 +128,8 @@
|
|||||||
./cmd/mesh-controller TestAGroupsCompositionIntoAReleaseBranchIsJudgedByItsOwnController
|
./cmd/mesh-controller TestAGroupsCompositionIntoAReleaseBranchIsJudgedByItsOwnController
|
||||||
./cmd/mesh-controller TestAGroupsCompositionJudgedWithTheValidatorIntoAReleaseBranchIsJudgedByItsOwnController
|
./cmd/mesh-controller TestAGroupsCompositionJudgedWithTheValidatorIntoAReleaseBranchIsJudgedByItsOwnController
|
||||||
./cmd/mesh-controller TestAssigningTheClientIssuesItsAccount
|
./cmd/mesh-controller TestAssigningTheClientIssuesItsAccount
|
||||||
|
./cmd/mesh-controller TestAnUnrelatedBatchWalksBesideAnOpenWalk
|
||||||
|
./cmd/mesh-controller TestARelatedBatchWaitsBehindTheOpenWalk
|
||||||
|
./cmd/mesh-controller TestAnOwnPathBatchNeverWalksBeside
|
||||||
|
./cmd/mesh-controller TestMergesHeardWhileAnotherHolderHasThePlansAreRecordedAndCarried
|
||||||
|
./cmd/mesh-controller TestABatchWindowClosesByTheExpiryOfItsKey
|
||||||
|
|||||||
Reference in New Issue
Block a user