Compare commits

...
10 Commits
Author SHA1 Message Date
jochen 3be6e86f77 Say an act's answer from one resolution after its write (issue 237)
mesh/merge-gate pass: builds mesh-controller, route-proxy → ace, novox; no bus step; every machine composes with the change as it did without (4 of 4 compo…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
assign read the whole mesh before its write, to record the seats held by
derivation, and that read printed on standard error that the node did not
resolve for lack of the seat the act was about to give it. The seat verb
answers with both streams, so the stale line came last and named the
assignment just made as its remedy.

The reads inside an act no longer print 'left out' lines; the node's
unmet seat dependencies and whether it resolves come from one planFor
taken after the write, and every other machine from blockedElsewhere.
unassign now says when its node does not resolve, from the same
resolution.
2026-10-12 13:53:19 +02:00
mesh-admin 4b1dee6b79 Merge pull request 'A merge that shares nothing with the assembling batch, and whose group waits for no one, is cut at once (hq issue 525, ADR 0304 §1, ADR 0306 §2-3)' (#250) from feat/525-a-lone-merge-skips-the-window into main 2026-10-12 11:46:59 +00:00
mesh-admin 4ab441ce16 Merge pull request 'Ask only once the bus enforces the controller's own grant, judged for real and failing closed (review of hq issue 353)' (#188) from fix/353-the-asker-judges-the-controllers-own-grant into main 2026-10-12 11:34:14 +00:00
jochen f996cf7b6b Take an older bus module's allowed as held, and name only remedies that work (review 31 of PR 188, hq issue 353)
mesh/merge-gate pass: builds mesh-controller → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
After mesh-catalog #166 merges and before the bus upgrade, the bus's
machine cannot be pushed, so any change to the composed list made the
list check read behind for good: every ask was withheld and the condition
named a push that is refused. An older module's allowed now counts as
held, the unconfirmed reload said once; the worst case is one ask refused
in the seconds before a reload, which is asked again (issue 369).

The list check stays for a module that says not allowed or cannot answer,
and its remedy is the plan's own reason when the list cannot be composed,
the bus upgrade while sends there wait on it, and a push otherwise. An
in_force false is a wait only when the module gives both times; otherwise
what is not known is named.
2026-10-12 13:21:32 +02:00
jochen bcfcfaaaf3 Keep asking on an older bus module and name the real remedy for each grant not held (review of hq issue 353)
A bus module older than its in_force answer read as not held, so a merge
before mesh-catalog #166 is live would silence every ask until a person's
bus upgrade. Such an answer now falls back to the check main makes (is the
list sent to the bus's machine the one composed now), says the reload is
not confirmed, and each case names its own remedy: push <node> for a list
behind or not allowing it, a ~5 s wait for a reload, the bus upgrade when
an older module's list cannot be judged.
2026-10-12 13:21:32 +02:00
jschoubben a3181e356b Ask only once the bus enforces the controller's own grant, judged for real and failing closed (review of hq issue 353)
change held asking back, a list sent was taken for a list the bus enforces, a plan that could not be
worked out read as not behind, and no test held the real judgement. Now the asker asks only when the
composition grants the controller itself to publish its ask, and the bus's own module says that grant
is in force on the running server (nats_user_can's in_force, mesh-catalog); each fact not known is not
held, said with why. brokerBehind takes a list nobody could compose as behind.
2026-10-12 13:21:32 +02:00
jschoubben 71b3aeaf2c Keep a group member of the other kind's batch from being overtaken (hq issue 525)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A merge's group member counted as merged when its merge waited in any batch,
but kinds never share a batch: the later member got a batch of its own, cut
at once, and walked before the earlier member's batch was cut, breaking the
group's order (ADR 0276 §5). Only a member kept in a batch of the merge's own
kind counts as merged now; one kept in the other kind's keeps the merge
waiting, and its batch names that member, so the two are cut in the order
their windows close (review of #250).
2026-10-12 12:54:11 +02:00
jschoubben 6e5a1df575 Judge the installer's user list against mesh-host with the groups grant (hq issue 525)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose (0.78s)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
The controller now asks mesh-delivery.groups, and mesh-host a76baa4b carries
that grant in the installer's first user list (mesh-host #85). The captured
copy is moved to that commit so the genesis test judges the list that ships.
2026-10-12 12:31:50 +02:00
jschoubben 4d203b5a5d Ask mesh-delivery whether a merge's group still assembles (hq issue 525)
A merge from any head waited for the window because the controller could not
see a group's members, so the window skip changed nothing in production and
unrelated merges walked one after another. The controller now asks
mesh-delivery's groups, within 3 s and before the plans are held: a merge
waits only when its head's group has another member whose pull request is
open and whose merge it does not keep yet (ADR 0306 §2). No answer means it
waits, and its batch says why. A headed merge in no waiting group and related
to no batch is cut at once, as one from no head is (ADR 0304 §1, ADR 0306 §3).

The controller's grant names mesh-delivery.groups. The installer's first user
list in mesh-host must carry it too before the genesis test can pass.
2026-10-12 12:14:46 +02:00
jschoubben 6196a4815e Let a merge that shares nothing skip the merge window (hq issue 525)
A merge waited the whole window even when nothing assembling shared anything
with it. It now joins only a batch it is related to (ADR 0304 §1, read by ADR
0306 §3: repository, a module the cut would move, a graph edge, a group head),
folding the newer into the oldest when it relates to two. A merge related to
none starts a batch whose window is closed at once (PlanBatch.CutAtOnce),
unless its delivery group may still have a member to merge (ADR 0306 §2).

The controller cannot see a group's unmerged members (that is mesh-delivery's
record), so a merge from any head still waits, in a batch of its own: the safe
error is to wait. Walks stay one at a time; a batch cut at once queues behind
an open walk. A stop's merges walked again join a related batch, or their own.
2026-10-12 12:14:46 +02:00
24 changed files with 1838 additions and 127 deletions
+67 -15
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/broker"
"io"
"slices"
"sort"
"strings"
@@ -54,6 +55,9 @@ func assignWith(ctx context.Context, open *stores, node string, opts assignOptio
if err := refusedMovingTheController(modules); err != nil {
return "", err
}
// Every line of the answer comes from the one resolution after the write (novox/hq issue 237):
// what the reads before it would say about a machine left out is not the answer's.
ctx = leftOutSaidBy(ctx, io.Discard)
// Held while it is recorded, so it cannot land between a converge's preview and its flip and
// be taken without ever having been previewed (novox/hq ADR 0100).
ctx, release, err := holdNodes(ctx, open, []string{node})
@@ -125,22 +129,28 @@ func assignWith(ctx context.Context, open *stores, node string, opts assignOptio
for _, line := range settled {
answer += "\n " + line
}
// What this act changed about this node's unmet seat dependencies, and nothing else (novox/hq
// ADR 0207): a dependency of a module just assigned, or one this assignment met. The rest of the
// node's list, and every other node's, is `status`'s.
for _, line := range unheldChange(shelf, node, before, append(append([]string(nil), before...), added...)) {
answer += "\n " + line
}
// Its bus credential, in the same act (novox/hq issue 203): an assignment pushed before its
// credential exists delivers a process that cannot authenticate and crash-loops until somebody
// runs a second verb and a second push. Issued here when the module speaks on the bus and has
// no credential yet; kept when it has one, so re-assigning rotates nothing.
var issued []string
for _, module := range added {
if line := issueOnAssign(ctx, open, node, module); line != "" {
answer += "\n " + line
issued = append(issued, line)
}
}
plan, _, err := planFor(ctx, open, node)
// **The one resolution the rest of the answer is said from** (novox/hq issue 237), taken after
// the write: what this act changed about this node's unmet seat dependencies, and nothing else
// (novox/hq ADR 0207) — a dependency of a module just assigned, or one this assignment met; the
// rest of the node's list, and every other node's, is `status`'s — and whether the node resolves.
after := resolvedAfter(ctx, open, shelf, node)
for _, line := range unheldChangeTo(shelf, node, before, after.modules, after.unheld) {
answer += "\n " + line
}
for _, line := range issued {
answer += "\n " + line
}
plan, err := after.plan, after.err
if err != nil {
// Kept, and still refused. Both halves are the answer, and the rest of the mesh is still
// worth reporting: this machine's refusal is rarely the only consequence.
@@ -218,6 +228,8 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
if len(modules) == 0 {
return "", fmt.Errorf("unassign %s names no module", node)
}
// One resolution after the write, as assign's answer (novox/hq issue 237).
ctx = leftOutSaidBy(ctx, io.Discard)
ctx, release, err := holdNodes(ctx, open, []string{node})
if err != nil {
return "", err
@@ -262,15 +274,22 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
}
answer := fmt.Sprintf("%s no longer runs %s — run `push %s` to make it so",
node, strings.Join(modules, ", "), node)
var left []string
for _, a := range assigned {
if !slices.Contains(modules, a) {
left = append(left, a)
}
}
for _, line := range unheldChange(shelf, node, assigned, left) {
after := resolvedAfter(ctx, open, shelf, node)
for _, line := range unheldChangeTo(shelf, node, assigned, after.modules, after.unheld) {
answer += "\n " + line
}
// Whether the node itself still resolves, from the same resolution: before, only the rest of the
// mesh's reads said it, on standard error and once per other machine.
if after.err != nil {
if unresolvable(after.err) {
answer += fmt.Sprintf("\n but %s does not resolve now, so nothing will be sent to it:", node)
} else {
answer += fmt.Sprintf("\n but whether %s still resolves could not be read:", node)
}
for _, line := range strings.Split(strings.TrimRight(after.err.Error(), "\n"), "\n") {
answer += "\n " + strings.TrimSpace(line)
}
}
// What it leaves behind that is irreplaceable is kept and retired, never removed (novox/hq ADR 0233).
for _, line := range keptOnUnassign(ctx, open.inventory, node, modules) {
answer += "\n " + line
@@ -278,6 +297,39 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
return answer + blockedElsewhere(ctx, open, node), nil
}
// afterAct is the one resolution of a node an act's answer is said from, taken after its write
// (novox/hq issue 237): the plan or why there is none, the modules it has on the node, and its unmet
// seat dependencies.
type afterAct struct {
plan catalogue.Resolution
err error
modules []string
unheld []catalogue.Unheld
}
// resolvedAfter resolves the node as `plan` does, once, after the act's write. When the node resolves,
// its modules and unmet seat dependencies are the resolution's — the closure, so a holder pulled in by
// a requirement counts as `plan` counts it. When it does not, they are judged over the assignments read
// after the write, which can only find more unmet than the closure would: a seat said held there is
// held in the closure too.
func resolvedAfter(ctx context.Context, open *stores, shelf map[string]catalogue.Manifest, node string) afterAct {
plan, _, err := planFor(ctx, open, node)
if err == nil {
out := afterAct{plan: plan, unheld: plan.Unheld}
for _, m := range plan.Modules {
out.modules = append(out.modules, m.Module)
}
return out
}
out := afterAct{err: err}
assigned, readErr := open.inventory.Assigned(ctx, node)
if readErr != nil {
return out
}
out.modules, out.unheld = assigned, unheldOver(shelf, node, assigned)
return out
}
// splitModules is a surface's one `module` field as the modules it names: several, comma-separated,
// are one act (novox/hq ADR 0207), so the holders that depend on each other go on together from the
// command API and the controller seat's verbs as they do from the command line.
+9
View File
@@ -213,6 +213,8 @@ type asker struct {
saidNoRouter bool
saidNoGrant bool
// saidGrantCaveat is whether a grant held with a caveat was said, so it is said once.
saidGrantCaveat bool
mu sync.Mutex
nudged chan struct{}
@@ -317,6 +319,13 @@ func (a *asker) reconcile(ctx context.Context) error {
}
return a.sayUnasked(ctx, unasked, "the bus does not hold the controller's grant to ask yet: "+why)
}
if why != "" && !a.saidGrantCaveat {
// Held, with a caveat worth one line (an older bus module cannot confirm the reload).
a.logf("the controller asks: %s", why)
a.saidGrantCaveat = true
} else if why == "" {
a.saidGrantCaveat = false
}
a.saidNoGrant = false
}
channels := ""
+205
View File
@@ -12,6 +12,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
@@ -808,3 +809,207 @@ func TestARefusalIsAskedAgainAndTheUndeliveredConditionClearsOnceTaken(t *testin
t.Errorf("the explanation says its verdict twice: %q", e)
}
}
// hq issue 353, the review: the bus holds the controller's grant to ask only when both facts hold — the mesh
// composes the controller's own grant, and the bus's module says it is in force on the running server — and
// each fact not known is not held, said with why.
func TestTheGrantIsHeldOnlyWhenComposedAndInForceOnTheBus(t *testing.T) {
yes := func(context.Context) (bool, string, error) { return true, "", nil }
no := func(why string) grantJudge {
return func(context.Context) (bool, string, error) { return false, why, nil }
}
broken := func(context.Context) (bool, string, error) { return false, "", errors.New("the store is away") }
for _, c := range []struct {
name string
composed, inForce grantJudge
held bool
says string
}{
{"composed and in force", yes, yes, true, ""},
{"not composed", no("no router is assigned"), yes, false, "no router is assigned"},
{"composed, written, not reloaded", yes, no("the server has not reloaded"), false, "has not reloaded"},
{"the composition unknown", broken, yes, false, "could not be worked out: the store is away"},
{"the bus unknown", yes, broken, false, "could not be worked out: the store is away"},
{"nothing judges the bus", yes, nil, false, "not judged"},
} {
held, why, err := grantHeldBy(c.composed, c.inForce, time.Now)(context.Background())
if err != nil || held != c.held || !strings.Contains(why, c.says) {
t.Errorf("%s: held %v (%q, %v)", c.name, held, why, err)
}
}
// Remembered for grantLookEvery, then judged again.
now := time.Date(2026, 10, 9, 18, 0, 0, 0, time.UTC)
inForce := false
judge := grantHeldBy(yes, func(context.Context) (bool, string, error) { return inForce, "not yet", nil },
func() time.Time { return now })
if held, _, _ := judge(context.Background()); held {
t.Fatal("held before the bus enforces it")
}
inForce = true
if held, _, _ := judge(context.Background()); held {
t.Error("judged again inside grantLookEvery")
}
now = now.Add(grantLookEvery)
if held, _, _ := judge(context.Background()); !held {
t.Error("not judged again after grantLookEvery")
}
}
// The controller's own grant, as composed: present while a router that takes asks is assigned, absent
// otherwise — whatever else in the user list changes.
func TestTheControllersOwnGrantToAskIsReadFromTheComposition(t *testing.T) {
router := broker.Declared{Module: "messenger", Holds: []broker.Seat{{Name: broker.AsksSeat, Scope: "mesh",
Accepts: []string{"ask", "cancel"}, Emits: []string{"decided"}, ByCaller: []string{"ask", "cancel", "decided"}}}}
other := broker.Declared{Module: "dunst"}
with := broker.Records{Nodes: []string{"anchor"}, Assigned: map[string][]broker.Declared{"anchor": {router, other}}}
without := broker.Records{Nodes: []string{"anchor"}, Assigned: map[string][]broker.Declared{"anchor": {other}}}
if may, err := controllerMayAsk(with); err != nil || !may {
t.Errorf("with a router assigned: %v %v", may, err)
}
if may, err := controllerMayAsk(without); err != nil || may {
t.Errorf("with no router assigned: %v %v", may, err)
}
}
// The bus module's answer, with the record's word on the list sent as the check only where the answer does
// not settle it (reviews 22 and 31 of PR 188); each way it is not held names its own remedy.
func TestTheBusModulesAnswerNamesTheRemedyForEachCase(t *testing.T) {
const (
reloaded = `{"allowed":true,"in_force":true,"file_written":"2026-10-09T15:55:56Z","server_loaded":"2026-10-09T15:56:05Z"}`
notLoaded = `{"allowed":true,"in_force":false,"why":"the user list on the machine allows it, and the server has not reloaded since it was written","file_written":"2026-10-09T15:55:56Z","server_loaded":"2026-10-09T15:50:00Z"}`
noVarz = `{"allowed":true,"in_force":false,"why":"the server's last configuration load could not be read: refused"}`
noStat = `{"allowed":true,"in_force":false,"why":"when the bus's user list was written could not be read inside mesh-broker-nats: exit 1"}`
notAllowed = `{"allowed":false,"in_force":false,"why":"the user list on the machine does not allow it"}`
)
current := busListState{Machine: "anchor"}
behind := busListState{Machine: "anchor", Behind: true}
stepped := busListState{Machine: "anchor", Behind: true, BusStep: "sending anchor would replace the bus; `bus upgrade --why …`"}
uncomposed := busListState{Machine: "anchor", Behind: true, Uncomposed: errors.New("module foo requires a provision nobody provides")}
for _, c := range []struct {
name, answer string
askErr error
list busListState
listErr error
held bool
says, not string
}{
{"in force", reloaded, nil, behind, nil, true, "", "push"},
{"written, not reloaded", notLoaded, nil, current, nil, false, "within about 5 s", "push"},
{"in force not known: varz", noVarz, nil, current, nil, false, "is not known: the server's last configuration load", "wait"},
{"in force not known: stat", noStat, nil, current, nil, false, "is not known: when the bus's user list was written", "wait"},
{"not allowed, list behind", notAllowed, nil, behind, nil, false, "`push anchor` carries it", "5 s"},
{"not allowed, bus step pending", notAllowed, nil, stepped, nil, false, "`bus upgrade", "`push anchor`"},
{"not allowed, list uncomposed", notAllowed, nil, uncomposed, nil, false, "`plan anchor`", "`push anchor`"},
{"older module, allowed, list current", `{"allowed":true}`, nil, current, nil, true, "not confirmed", "push"},
{"older module, allowed, list behind", `{"allowed":true}`, nil, behind, nil, true, "not confirmed", "push"},
{"older module, allowed, list unknown", `{"allowed":true}`, nil, current, errors.New("the store is away"), true, "`bus upgrade`", "push"},
{"older module, not allowed, list behind", `{"allowed":false}`, nil, behind, nil, false, "`push anchor` carries it", "5 s"},
{"module cannot answer, list current", "", errors.New("no responders"), current, nil, true, "could not answer", "push"},
{"module cannot answer, list behind", "", errors.New("no responders"), behind, nil, false, "`push anchor` carries it", ""},
{"module cannot answer, list unknown", "", errors.New("no responders"), current, errors.New("the store is away"), false, "could not be worked out", ""},
{"answer unreadable, list behind", `not json`, nil, behind, nil, false, "could not be read", ""},
} {
judge := busGrantBy(
func(context.Context) (string, json.RawMessage, error) {
return "anchor", json.RawMessage(c.answer), c.askErr
},
func(context.Context) (busListState, error) { return c.list, c.listErr })
held, why, _ := judgeGrantOnce(judge)
if held != c.held || !strings.Contains(why, c.says) || (c.not != "" && strings.Contains(why, c.not)) {
t.Errorf("%s: held %v (%q)", c.name, held, why)
}
}
none := busGrantBy(func(context.Context) (string, json.RawMessage, error) { return "", nil, nil }, nil)
if held, why, _ := judgeGrantOnce(none); held || !strings.Contains(why, "no module holds the bus") {
t.Errorf("no bus: held %v (%q)", held, why)
}
}
// judgeGrantOnce is the whole judgement with the composed grant held, as the asker makes it.
func judgeGrantOnce(bus grantJudge) (bool, string, error) {
yes := func(context.Context) (bool, string, error) { return true, "", nil }
return grantHeldBy(yes, bus, time.Now)(context.Background())
}
// The orders reviews 22 and 31 of PR 188 found: this controller live before the bus module that answers
// in_force, with the bus's machine behind on its list — and, after mesh-catalog #166 merged, unpushable until a
// person's bus upgrade. An older module that allows the ask asks (once, with the caveat said); a new module that
// has not reloaded names the wait, not a push; one that cannot say whether it reloaded names what is not known.
func TestAnOlderBusModuleNeitherSilencesTheAsksNorNamesTheWrongRemedy(t *testing.T) {
for _, c := range []struct {
name, answer string
list busListState
asks bool
says, not string
}{
{"older module, list current", `{"allowed":true}`, busListState{Machine: "anchor"}, true, "not confirmed", ""},
{"older module, list behind, bus step pending", `{"allowed":true}`,
busListState{Machine: "anchor", Behind: true, BusStep: "a planned step"}, true, "not confirmed", ""},
{"new module, not reloaded",
`{"allowed":true,"in_force":false,"why":"not reloaded","file_written":"2026-10-09T15:55:56Z","server_loaded":"2026-10-09T15:50:00Z"}`,
busListState{Machine: "anchor"}, false, "within about 5 s", "push"},
{"new module, reload not known", `{"allowed":true,"in_force":false,"why":"the server's last configuration load could not be read: refused"}`,
busListState{Machine: "anchor"}, false, "is not known", "wait"},
} {
t.Run(c.name, func(t *testing.T) {
r := newAskerRig(t)
var said []string
r.a.logf = func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }
var raised [][]conditions.Observation
r.a.raise = func(_ context.Context, obs []conditions.Observation) error {
raised = append(raised, obs)
return nil
}
yes := func(context.Context) (bool, string, error) { return true, "", nil }
r.a.grantHeld = grantHeldBy(yes, busGrantBy(
func(context.Context) (string, json.RawMessage, error) {
return "anchor", json.RawMessage(c.answer), nil
},
func(context.Context) (busListState, error) { return c.list, nil }),
func() time.Time { return r.now })
r.open = []conditions.Condition{heldCondition()}
_ = r.a.reconcile(context.Background())
_ = r.a.reconcile(context.Background())
sent := r.asksSent(t)
if c.asks {
if len(sent) != 1 || sent[0].About != heldCondition().Key {
t.Fatalf("not asked: %+v", sent)
}
n := 0
for _, s := range said {
if strings.Contains(s, c.says) {
n++
}
}
if n != 1 {
t.Errorf("the caveat said %d times: %q", n, said)
}
return
}
if len(sent) != 0 {
t.Fatalf("asked while the bus does not hold the grant: %+v", sent)
}
if len(raised) == 0 || len(raised[len(raised)-1]) != 1 ||
raised[len(raised)-1][0].Kind != "asks-undelivered" {
t.Fatalf("not said as undelivered: %+v", raised)
}
summary := raised[len(raised)-1][0].Summary
if !strings.Contains(summary, c.says) || (c.not != "" && strings.Contains(summary, c.not)) {
t.Errorf("the condition names the wrong remedy: %q", summary)
}
})
}
}
// The serving asker is wired to the real judgement: with records and a bus it cannot read, the grant is not
// held, and so nothing is asked (a stub saying yes, or no judgement at all, would ask).
func TestTheServingAskerJudgesTheGrantForReal(t *testing.T) {
a := servingAsker(&stores{}, nil, nil, nil)
if a.grantHeld == nil {
t.Fatal("the serving asker does not judge the bus's grant")
}
held, why, err := a.grantHeld(context.Background())
if err != nil || held || !strings.Contains(why, "could not be worked out") {
t.Errorf("held %v (%q, %v) with nothing to read", held, why, err)
}
}
+271 -21
View File
@@ -235,12 +235,27 @@ func routerHereIn(inv *inventory.Inventory) func(ctx context.Context) (bool, err
}
}
// grantHeldIn says whether the bus holds the controller's grant to ask (novox/hq issue 353): the user list the
// machine holding the bus was last sent is the one the mesh composes now (brokerBehind, the same judgement a
// push makes to send that machine first). While it is behind, the controller's ask is refused by the bus,
// whatever the record says of the router, so nothing is asked and the operator is told to push that machine.
// Judged at most every grantLookEvery: composing the list resolves the bus's machine whole.
func grantHeldIn(open *stores) func(ctx context.Context) (bool, string, error) {
// grantHeldIn says whether the bus holds the controller's grant to ask (novox/hq issue 353), judged on the two
// facts that make it so, and failing closed on each:
//
// 1. **composed**: the user list the mesh composes now grants the controller itself to publish its ask
// (controllerMayAsk) — the controller's own grant, not whether any user of the list changed;
// 2. **in force**: the bus's own module, on the machine holding the bus, says the user list there allows it
// and the server reloaded after that list was written (the nats module's `nats_user_can`, `in_force`) — a
// list sent is not a list the bus enforces until it reloads.
//
// Anything that cannot be worked out is not held, with why, so the conditions that need the operator are said
// as undelivered rather than asked into a refusal.
func grantHeldIn(open *stores, conn *nats.Conn) func(ctx context.Context) (bool, string, error) {
return grantHeldBy(composedGrantIn(open), busGrantIn(open, conn), time.Now)
}
// grantJudge is one of the two facts: whether it holds, and why not.
type grantJudge func(ctx context.Context) (bool, string, error)
// grantHeldBy is the judgement over the two facts, remembered for grantLookEvery: composing the list resolves
// the bus's machine whole, and asking the bus's module is a call.
func grantHeldBy(composed, inForce grantJudge, now func() time.Time) func(ctx context.Context) (bool, string, error) {
var mu sync.Mutex
var at time.Time
var held bool
@@ -248,22 +263,251 @@ func grantHeldIn(open *stores) func(ctx context.Context) (bool, string, error) {
return func(ctx context.Context) (bool, string, error) {
mu.Lock()
defer mu.Unlock()
if !at.IsZero() && time.Since(at) < grantLookEvery {
if !at.IsZero() && now().Sub(at) < grantLookEvery {
return held, why, nil
}
machine, behind, err := brokerBehind(ctx, open, nil)
held, why = judgeGrant(ctx, composed, inForce)
at = now()
return held, why, nil
}
}
// judgeGrant is held only when both facts hold; an error is a fact not known, never a yes. A fact that holds
// with a caveat (a reload not confirmed) gives that caveat as the why of a yes.
func judgeGrant(ctx context.Context, composed, inForce grantJudge) (bool, string) {
var caveat []string
for _, fact := range []struct {
judge grantJudge
what string
}{{composed, "whether the mesh composes the controller a grant to ask"}, {inForce, "whether the bus enforces it"}} {
if fact.judge == nil {
return false, fact.what + " is not judged here"
}
ok, why, err := fact.judge(ctx)
if err != nil {
return false, fact.what + " could not be worked out: " + err.Error()
}
if !ok {
return false, why
}
if why != "" {
caveat = append(caveat, why)
}
}
return true, strings.Join(caveat, "; ")
}
// controllerMayAsk says whether the controller's own grant, as composed from these records, lets it publish its
// ask (ADR 0259 §3: composed while a router that takes asks under its asker's name is assigned).
func controllerMayAsk(records broker.Records) (bool, error) {
users, err := broker.Users(records)
if err != nil {
return false, err
}
for _, u := range users {
if u.Kind != broker.KindController {
continue
}
perms, err := broker.PermissionsFor(u)
if err != nil {
return false, err
}
return broker.MayPublish(perms, asks.AskSubject(askerName)), nil
}
return false, errors.New("the composition holds no controller")
}
// composedGrantIn is the first fact, from the mesh's records.
func composedGrantIn(open *stores) grantJudge {
return func(ctx context.Context) (bool, string, error) {
if open == nil || open.inventory == nil {
return false, "", errors.New("the mesh's records are not open")
}
records, err := open.inventory.BusRecords(ctx)
if err != nil {
return false, "", err
}
at, held, why = time.Now(), !behind, ""
if behind {
why = fmt.Sprintf("the bus's user list on %s is behind what the mesh composes, so the bus has not been "+
"given the controller's grant to ask; `push %s` carries it", machine, machine)
may, err := controllerMayAsk(records)
if err != nil || may {
return may, "", err
}
return held, why, nil
return false, "the mesh composes the controller no grant to ask: no router that takes asks under its " +
"asker's name is assigned", nil
}
}
// busGrantIn is the second fact, asked of the bus's own module on the machine holding the bus, with the check
// main made before it (novox/hq issue 353, PR 186: is the user list sent to the bus's machine the one composed
// now) kept for the cases the module's word does not settle.
func busGrantIn(open *stores, conn *nats.Conn) grantJudge {
ask := func(ctx context.Context) (string, json.RawMessage, error) {
if open == nil || open.inventory == nil || conn == nil {
return "", nil, errors.New("the mesh's records or the bus are not open")
}
holders, err := seatHolders(ctx, open.inventory)
if err != nil {
return "", nil, err
}
h, held := holders[theBrokerSeat]
if !held || h.Node == "" || h.Module == "" {
return "", nil, nil
}
a, err := link.AskModuleToolOn(ctx, conn, h.Module, busUserCanTool, h.Node, map[string]any{
"user": broker.ControllerName, "action": "publish", "subject": asks.AskSubject(askerName)}, 15*time.Second)
if err != nil {
return h.Node, nil, err
}
if a.Error != "" {
return h.Node, nil, errors.New(a.Error)
}
return h.Node, a.Result, nil
}
list := func(ctx context.Context) (busListState, error) {
if open == nil || open.inventory == nil {
return busListState{}, errors.New("the mesh's records are not open")
}
s, err := brokerListState(ctx, open, nil)
if err != nil || s.Machine == "" || !s.Behind {
return s, err
}
step, err := busHeld(ctx, open.inventory, []string{s.Machine})
if err != nil {
return busListState{}, err
}
s.BusStep = step[s.Machine]
return s, nil
}
return busGrantBy(ask, list)
}
// busListState is the user list on the bus's machine as the controller's record knows it: whether the list
// sent there is behind the one composed now, why it could not be composed, and the bus step a send there waits
// on (that machine cannot be pushed until a person's `bus upgrade`).
type busListState struct {
Machine string
Behind bool
Uncomposed error
BusStep string
}
// remedy is the one thing that sends the bus's machine the list composed now: the plan's own reason when it
// cannot be composed, the bus upgrade while sends there wait on it, and otherwise a push of that machine.
func (s busListState) remedy(node string) string {
if s.Machine != "" {
node = s.Machine
}
switch {
case s.Uncomposed != nil:
return fmt.Sprintf("the user list for %s cannot be composed (%v); `plan %s` says what to fix first",
node, s.Uncomposed, node)
case s.BusStep != "":
return fmt.Sprintf("sends to %s wait on the bus's planned step, so a push will not carry it: %s", node, s.BusStep)
}
return fmt.Sprintf("`push %s` carries it", node)
}
// busGrantBy judges the bus's word, and names for each way it is not held the one thing that makes it held:
//
// - **in force**: held;
// - **allowed, from a module older than `in_force`**: held, with the caveat that the reload is not confirmed.
// The file allows the publish, and the bus reloads within about 5 s of a change to it, so the worst case is
// one ask refused in those seconds, which is asked again (issue 369). The list check is not consulted:
// while a bus step is pending the bus's machine cannot be pushed, so any change to the composed list would
// read as behind for good and silence every ask until a person's bus upgrade (the review of PR 188);
// - **allowed, written, not reloaded** (the module gives both times): wait about 5 s;
// - **allowed, in force not known** (the module could not read a time): what is not known, said;
// - **not allowed, or the module cannot answer**: the list check decides — the remedy that sends the list
// when it is behind (busListState.remedy), and, for a module that cannot answer, held while the list sent
// is current, as main judged it.
//
// ask answers the machine holding the bus and its module's answer ("" when nothing holds the bus); list is the
// record's word on the list sent to that machine.
func busGrantBy(ask func(ctx context.Context) (string, json.RawMessage, error),
list func(ctx context.Context) (busListState, error)) grantJudge {
return func(ctx context.Context) (bool, string, error) {
node, raw, askErr := ask(ctx)
if askErr == nil && node == "" {
return false, "no module holds the bus, so nothing says what it enforces", nil
}
var answer busAnswer
if askErr == nil {
if answer, askErr = readBusAnswer(raw); askErr != nil {
askErr = fmt.Errorf("its answer could not be read: %w", askErr)
}
}
if askErr != nil {
if node == "" {
node = "the bus's machine"
}
s, err := list(ctx)
if err != nil {
return false, "", fmt.Errorf("the bus module on %s could not answer (%v), and whether the list sent "+
"there is the one composed now could not be worked out: %w", node, askErr, err)
}
if s.Behind {
return false, fmt.Sprintf("the bus module on %s could not answer (%v), and the user list sent there is "+
"behind what the mesh composes; %s", node, askErr, s.remedy(node)), nil
}
return true, fmt.Sprintf("the bus module on %s could not answer (%v); judged by the user list sent there, "+
"which is the one composed now", node, askErr), nil
}
notAllowed := func(older string) (bool, string, error) {
s, err := list(ctx)
if err != nil {
return false, "", fmt.Errorf("the user list on %s does not allow the controller to ask%s, and whether "+
"the list sent there is the one composed now could not be worked out: %w", node, older, err)
}
if s.Behind {
return false, fmt.Sprintf("the user list on %s does not allow the controller to ask%s: it is behind "+
"what the mesh composes; %s", node, older, s.remedy(node)), nil
}
return false, fmt.Sprintf("the user list on %s does not allow the controller to ask%s, though the list "+
"sent there is the one composed now; %s again", node, older, s.remedy(node)), nil
}
switch {
case answer.InForce == nil && answer.Allowed:
return true, fmt.Sprintf("the user list on %s allows the controller to ask, but its reload is not "+
"confirmed: the bus module there is older than the answer whether a grant is in force, and the bus "+
"upgrade (`bus upgrade`) carries one that confirms it", node), nil
case answer.InForce == nil:
return notAllowed(" (the bus module there is older than the answer whether a grant is in force)")
case !answer.Allowed:
return notAllowed("")
case *answer.InForce:
return true, "", nil
case answer.FileWritten != "" && answer.ServerLoaded != "":
return false, fmt.Sprintf("the bus on %s has the controller's grant to ask written (%s) but has not "+
"reloaded since (last load %s); it reloads in place within about 5 s, so nothing is to be done but wait",
node, answer.FileWritten, answer.ServerLoaded), nil
}
why := answer.Why
if why == "" {
why = "it gives no reason"
}
return false, fmt.Sprintf("the user list on %s allows the controller to ask, but whether the bus enforces it "+
"is not known: %s", node, why), nil
}
}
// busUserCanTool is the bus module's answer to whether a user may do something, and whether it is in force.
const busUserCanTool = "nats_user_can"
// busAnswer is the bus module's answer to busUserCanTool; InForce is nil from a module older than that field,
// and FileWritten and ServerLoaded are given only when it could read both times (mesh-catalog's nats InForce).
type busAnswer struct {
Allowed bool `json:"allowed"`
InForce *bool `json:"in_force"`
Why string `json:"why"`
FileWritten string `json:"file_written"`
ServerLoaded string `json:"server_loaded"`
}
func readBusAnswer(raw json.RawMessage) (busAnswer, error) {
var answer busAnswer
err := json.Unmarshal(raw, &answer)
return answer, err
}
// grantLookEvery is how often the bus's user list is judged against the one its machine was last sent.
const grantLookEvery = 30 * time.Second
@@ -302,7 +546,19 @@ func startAsking(ctx context.Context, open *stores, server *link.Server, conn *n
fmt.Printf("the operator cannot be asked: %v\n", err)
return
}
a := &asker{
a := servingAsker(open, conn, js, keeper)
if err := server.Decides(a); err != nil {
fmt.Printf("the operator's answers cannot be heard, so nothing is asked: %v\n", err)
return
}
askerFrom = a
go a.keep(ctx)
}
// servingAsker is the serving controller's asker, wired to the mesh: what it reads, publishes, performs and
// records, and the judgements it asks by — the bus's grant to ask among them (novox/hq issue 353).
func servingAsker(open *stores, conn *nats.Conn, js jetstream.JetStream, keeper *conditions.Keeper) *asker {
return &asker{
open: keeper.Open,
silence: func(ctx context.Context, key string, d time.Duration, by, why string) error {
_, err := keeper.Silence(ctx, key, d, by, why)
@@ -322,7 +578,7 @@ func startAsking(ctx context.Context, open *stores, server *link.Server, conn *n
},
routerRecord: routerRecordOf(conn, open.inventory),
routerHere: routerHereIn(open.inventory),
grantHeld: grantHeldIn(open),
grantHeld: grantHeldIn(open, conn),
channels: channelsIn(open.inventory),
raise: func(ctx context.Context, obs []conditions.Observation) error {
return keeper.Reconcile(ctx, sourceAsker, obs)
@@ -330,10 +586,4 @@ func startAsking(ctx context.Context, open *stores, server *link.Server, conn *n
now: time.Now,
logf: func(format string, args ...any) { fmt.Printf(format+"\n", args...) },
}
if err := server.Decides(a); err != nil {
fmt.Printf("the operator's answers cannot be heard, so nothing is asked: %v\n", err)
return
}
askerFrom = a
go a.keep(ctx)
}
+232 -17
View File
@@ -37,6 +37,11 @@ import (
// own commit, the newest first, one walk each, until one is delivered; the older ones are then answered
// by that one.
//
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, ADR 0306 decisions
// 2 and 3, issue 525): it joins a batch it is related to, and one that is related to none and whose delivery
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Walks stay one at
// a time: such a batch is cut when no walk is open, and queued behind the one that is otherwise.
//
// A merge on the controller's own path (a module whose walk waits for nobody's word) never shares a batch with
// one that waits for mesh-delivery's: each kind has a batch of its own, an own-path batch is cut first and folds
// no waiting catalogue walk, and a catalogue walk let go while another walk runs starts once it ended — so no
@@ -173,6 +178,12 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
fmt.Printf("%s merged into %s (%.8s); nothing the mesh holds reads it\n", repository, m.Base, m.Commit)
return nil
}
// **Its delivery group's word, before the plans are held** (ADR 0306 decision 2): a slow answer holds up no walk.
own := ownPath(touches)
waits, err := groupWait(ctx, inv, m, deliverySeatHeld(entries), own)
if err != nil {
return notNow(err)
}
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return notNow(err)
@@ -198,7 +209,6 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
// **A merge on the controller's own path never shares a batch with one that waits for the delivery's
// word** (decided during the build, 2026-10-10): batched together, the catalogue's deliveries would start
// with the controller's and skip their turn. Each kind has a batch of its own.
own := ownPath(touches)
if p, ok, err := answeredByALaterMerge(ctx, inv, heard, touches, own, deliverySeatHeld(entries)); err != nil {
return notNow(err)
} else if ok {
@@ -218,7 +228,7 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
return cutBatchesHeld(ctx, f.open, now)
}
batch, err := theOpenBatch(ctx, inv, heard, now, own)
batch, why, err := batchToJoin(ctx, inv, heard, entries, read, now, own, waits)
if err != nil {
return notNow(err)
}
@@ -229,7 +239,7 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
if err := keepBatch(ctx, inv, &batch, now, ""); err != nil {
return notNow(err)
}
fmt.Printf("%s merged into %s (%.8s); it joins %s — %s\n", repository, m.Base, m.Commit, batch.ID,
fmt.Printf("%s merged into %s (%.8s); %s, it joins %s — %s\n", repository, m.Base, m.Commit, why, batch.ID,
batchWords(batch, now))
return cutBatchesHeld(ctx, f.open, now)
}
@@ -320,22 +330,211 @@ func ownPath(touches []inventory.Entry) bool {
return false
}
// theOpenBatch is the batch a merge heard now joins: the one of its kind not yet cut, or a new one.
func theOpenBatch(ctx context.Context, inv *inventory.Inventory, first inventory.BatchedMerge, now time.Time, own bool) (inventory.Plan, error) {
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, as ADR 0306
// decisions 2 and 3 read it). batchToJoin is the batch a merge heard now joins, and why in words:
//
// - an assembling or queued batch of its kind it is related to (related.go): it joins it and waits for the
// window, which its coming restarts. Related to two or more, the newer are folded into the oldest, which
// answers all their merges: a merge never waits in two windows, and what it shares with each walks in one.
// - none, and its delivery group has a member still to merge, or mesh-delivery could not say (groupWait, waits
// names why): a new batch, which waits for the window, beside the ones it shares nothing with; its words say
// why.
// - none, and no member of its group is still to merge: a new batch whose window is closed at once. Cut at
// once means only that: walks stay one at a time (ADR 0276 decision 4), so with a walk open the batch is
// queued behind it as any closed batch is.
func batchToJoin(ctx context.Context, inv *inventory.Inventory, heard inventory.BatchedMerge, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, now time.Time, own bool, waits string) (inventory.Plan, string, error) {
edges, err := inv.Dependencies(ctx)
if err != nil {
return inventory.Plan{}, "", err
}
joins, whys, _, err := relatedBatches(ctx, inv, []inventory.BatchedMerge{heard}, own, "", entries, read, edges)
if err != nil {
return inventory.Plan{}, "", err
}
if len(joins) > 0 {
into := joins[0]
why := "related to " + into.ID + " (" + whys[0] + ")"
if len(joins) > 1 {
plans, moves, err := foldInto(ctx, inv, &into, joins[1:])
if err != nil {
return inventory.Plan{}, "", err
}
if err := inv.SavePlansMoving(ctx, plans, moves); err != nil {
return inventory.Plan{}, "", err
}
var folded []string
for i, b := range joins[1:] {
folded = append(folded, b.ID+" ("+whys[i+1]+")")
}
why += ", and to " + strings.Join(folded, ", ") + ", folded into it"
}
into.Delivery.Batch.CutAtOnce = false
return into, why, nil
}
if waits != "" {
// Its own window, beside the batches it shares nothing with: waiting is for its group's members, which
// are related to it by their head and join it (ADR 0306 decision 2).
b, err := newBatch(ctx, inv, heard, now, own, false, waits)
return b, waits + ", it waits for the window in a batch of its own", err
}
b, err := newBatch(ctx, inv, heard, now, own, true, "")
return b, "it shares nothing with a batch assembling and no member of a group is still to merge: it is cut at once", err
}
// relatedBatches are the batches not yet cut of a kind, oldest first, parted into those related to merges (with
// the first reason of each) and the others. Each batch is judged by what its cut would move (ADR 0306 decision
// 3): its merges' moves as they were kept and as they read now — the catalogue may have moved since a merge was
// heard — widened to their dependents, and, for a catalogue batch, every walk waiting for its word, which the
// cut folds in (cutBatch's carry). A merge kept with no moves leaves its batch unknown, related to everything.
// except is a walk not to count as folded: the one being stopped.
func relatedBatches(ctx context.Context, inv *inventory.Inventory, merges []inventory.BatchedMerge, own bool,
except string, entries []inventory.Entry, read map[string][]inventory.ReadRepository,
edges []inventory.Edge) (joins []inventory.Plan, whys []string, others []inventory.Plan, err error) {
batches, err := inv.Batches(ctx)
if err != nil {
return inventory.Plan{}, err
return nil, nil, nil, err
}
var kind []inventory.Plan
for _, b := range batches {
if b.OwnPath() == own {
return b, nil
kind = append(kind, b)
}
}
if len(kind) == 0 {
return nil, nil, nil, nil
}
carry := newFootprint()
if !own {
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch may fold every one.
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, nil, nil, err
}
for _, p := range plans {
if p.Release != nil || !p.Waiting() || p.ID == except {
continue
}
kept, err := inv.MergesOf(ctx, p.ID)
if err != nil {
return nil, nil, nil, err
}
carry = unionFootprint(carry, footprintOfWalk(p, kept))
}
}
mine := footprintAsCut(merges, entries, read, edges)
for _, b := range kind {
held, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return nil, nil, nil, err
}
if b.Delivery == nil {
b.Delivery = &inventory.PlanDelivery{}
}
if b.Delivery.Batch == nil {
b.Delivery.Batch = &inventory.PlanBatch{Own: own}
}
if ok, why := related(mine, unionFootprint(footprintAsCut(held, entries, read, edges), carry), edges); ok {
joins, whys = append(joins, b), append(whys, why)
} else {
others = append(others, b)
}
}
return joins, whys, others, nil
}
// footprintAsCut is a batch's footprint as a cut now would read it: footprintOfBatch, widened by the modules each
// merge moves as the catalogue reads now (planBatch judges each merge again at the cut, so what it moves may
// have changed since it was heard). Both are kept: when in doubt, related.
func footprintAsCut(merges []inventory.BatchedMerge, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, edges []inventory.Edge) footprint {
f := footprintOfBatch(merges, edges)
var moved []string
for _, m := range merges {
s := announcedOf(m).SourceMoved
if s.Owner == "" {
continue
}
s.MergedAt = "" // news, whatever a cut marked seen since (as planBatch reads it)
for _, e := range wouldMove(s, entries, read) {
moved = append(moved, e.Manifest.Module)
}
}
for _, name := range reachableFrom(moved, edges) {
f.modules[name] = true
}
return f
}
// unionFootprint is what two changes reach together.
func unionFootprint(a, b footprint) footprint {
f := newFootprint()
for _, from := range []footprint{a, b} {
for k := range from.repos {
f.repos[k] = true
}
for k := range from.modules {
f.modules[k] = true
}
for k := range from.heads {
f.heads[k] = true
}
}
f.unknown = a.unknown
if f.unknown == "" {
f.unknown = b.unknown
}
return f
}
// foldInto folds batches into the oldest one related to a merge: the plans to write (into first, so no merge
// names a plan the store does not hold) and the merges to move, for one SavePlansMoving. The folded batches name
// the batch that took them over; a stop's closed window and its withheld modules (ADR 0299) go with their merges.
func foldInto(ctx context.Context, inv *inventory.Inventory, into *inventory.Plan, from []inventory.Plan) ([]*inventory.Plan,
[]inventory.MergeMove, error) {
if into.Delivery == nil {
into.Delivery = &inventory.PlanDelivery{}
}
if into.Delivery.Batch == nil {
into.Delivery.Batch = &inventory.PlanBatch{Own: into.OwnPath()}
}
plans := []*inventory.Plan{into}
var moves []inventory.MergeMove
for i := range from {
f := from[i]
merges, err := inv.MergesOf(ctx, f.ID)
if err != nil {
return nil, nil, err
}
for _, m := range merges {
moves = append(moves, inventory.MergeMove{Repository: m.Repository, Commit: m.Commit, Plan: into.ID, Alone: m.Alone})
}
if f.Delivery == nil {
f.Delivery = &inventory.PlanDelivery{}
}
if w := f.Delivery.Batch; w != nil {
into.Delivery.Batch.Closed = into.Delivery.Batch.Closed || w.Closed
into.Delivery.Batch.Withheld = union(into.Delivery.Batch.Withheld, w.Withheld)
sort.Strings(into.Delivery.Batch.Withheld)
}
f.State = inventory.PlanSuperseded
f.Delivery.Batch = nil
f.Delivery.TakenOverBy = into.ID
f.Note = fmt.Sprintf("folded into %s before it was cut: a merge related to both joined it, and %s answers "+
"its merges", into.ID, into.ID)
plans = append(plans, &f)
}
return plans, moves, nil
}
// newBatch is a new batch of a kind, kept before its first merge names it: a merge never names a plan the store
// does not hold. cutAtOnce closes its window at once; waits says why it waits for its group, when it does.
func newBatch(ctx context.Context, inv *inventory.Inventory, first inventory.BatchedMerge, now time.Time, own,
cutAtOnce bool, waits string) (inventory.Plan, error) {
b := inventory.Plan{ID: fmt.Sprintf("plan-%d", now.UnixNano()), Repository: first.Repository, Branch: first.Branch,
Commit: first.Commit, Merged: first.Merged, Created: now, State: inventory.PlanAssembling,
Tiers: [][]string{}, Modules: map[string]*inventory.PlanModule{},
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own}}}
// Kept before its first merge names it: a merge never names a plan the store does not hold.
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own, CutAtOnce: cutAtOnce, Waits: waits}}}
if err := inv.SavePlan(ctx, &b); err != nil {
return inventory.Plan{}, err
}
@@ -367,13 +566,16 @@ func keepBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan,
}
closes, latest := windowOf(merges, b.Created, window, atMost)
b.Delivery.Merges = named
var closed bool
var closed, cutAtOnce bool
var withheld []string
var waits string
if was := b.Delivery.Batch; was != nil {
closed, withheld = was.Closed, was.Withheld // a stop's, which no look takes back (novox/hq ADR 0299)
cutAtOnce = was.CutAtOnce // the merge's, until another joins it (ADR 0304 decision 1)
waits = was.Waits // its group's, as said when it was made (ADR 0306 decision 2)
}
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, Own: b.OwnPath(),
Closed: closed, Withheld: withheld}
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits}
b.State = inventory.PlanAssembling
if behind != "" && windowClosed(*b, now) {
b.State = inventory.PlanQueued
@@ -402,13 +604,14 @@ func windowOf(merges []inventory.BatchedMerge, created time.Time, window, atMost
return last.Add(window), created.Add(atMost)
}
// windowClosed says a batch's window is closed: no merge for the window's length, or its maximum reached.
// windowClosed says a batch's window is closed: closed by a stop, closed at once for a merge that shares
// nothing (ADR 0304 decision 1), no merge for the window's length, or its maximum reached.
func windowClosed(b inventory.Plan, now time.Time) bool {
if b.Delivery == nil || b.Delivery.Batch == nil {
return true
}
w := b.Delivery.Batch
return w.Closed || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
return w.Closed || w.CutAtOnce || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
}
// carriedOf is a set of merges as a walk carries them: the latest merge of each repository's branch, and
@@ -617,8 +820,9 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
return err
}
// The oldest batch whose window closed is cut, an own-path one before a catalogue one (it goes first, and
// the catalogue batch queues behind it rather than being cut and overtaken); one of each kind at most is
// open (theOpenBatch).
// the catalogue batch queues behind it rather than being cut and overtaken). Several of a kind may be open
// since a merge that shares nothing is cut at once (ADR 0304 decision 1): walks stay one at a time, so one
// is cut per look, and once its walk started the others are queued behind it.
sort.SliceStable(batches, func(i, j int) bool { return batches[i].OwnPath() && !batches[j].OwnPath() })
for i := range batches {
b := &batches[i]
@@ -886,6 +1090,9 @@ func walkTimesAtCut(batch, walk inventory.Plan, entries []inventory.Entry, now t
if !w.AtMost.IsZero() && w.AtMost.Before(closed) {
closed = w.AtMost
}
if w.CutAtOnce && !batch.Created.IsZero() {
closed = batch.Created // closed when it was made (ADR 0304 decision 1)
}
if !closed.IsZero() {
if closed.After(now) {
closed = now
@@ -1142,6 +1349,10 @@ func batchWords(b inventory.Plan, now time.Time) string {
if b.State == inventory.PlanQueued {
return fmt.Sprintf("queued behind %s; grouped: %s; plan not yet calculated", w.Behind, grouped)
}
if w.CutAtOnce {
return fmt.Sprintf("cut at once, sharing nothing with a batch assembling; grouped: %s; plan not yet calculated",
grouped)
}
left := w.ClosesAt.Sub(now)
if w.AtMost.Before(w.ClosesAt) {
left = w.AtMost.Sub(now)
@@ -1149,8 +1360,12 @@ func batchWords(b inventory.Plan, now time.Time) string {
if left < 0 {
left = 0
}
return fmt.Sprintf("assembling: %s left (at the latest %s); grouped: %s; plan not yet calculated",
secondsWords(left), w.AtMost.Local().Format("15:04:05"), grouped)
waits := ""
if w.Waits != "" {
waits = " — " + w.Waits
}
return fmt.Sprintf("assembling: %s left (at the latest %s)%s; grouped: %s; plan not yet calculated",
secondsWords(left), w.AtMost.Local().Format("15:04:05"), waits, grouped)
}
// groupedWords is a batch's commits, each with the earlier merges it answers.
+76 -12
View File
@@ -34,6 +34,7 @@ func windowed(t *testing.T) *stores {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
noGroups(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
@@ -58,19 +59,70 @@ func windowed(t *testing.T) *stores {
// t0 is the moment a test's clock starts: merges are dated by it, and heard after it.
var t0 = time.Now().UTC().Truncate(time.Second)
// catalogueMerge is a merge of the catalogue changing one module's directory, made at a moment.
// catalogueMerge is a merge of the catalogue changing one module's directory, made at a moment, from a pull
// request's head branch of its own, as the forge announces every merge. Whether its head names a delivery group
// with a member still to merge is mesh-delivery's word, which windowed stubs as no group at all (noGroups); a test
// whose merges wait for the window as a group's says so (groupsWaiting).
func catalogueMerge(commit, module string, made time.Time) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit,
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Head: "pr/" + commit, Commit: commit,
MergedAt: made.Format(time.RFC3339Nano), Paths: []string{"modules/" + module + "/module.json"},
ModuleDirs: []string{"modules/" + module}, ModuleDirsSaid: true}
}
// repoMerge is a merge of a repository of one module's own.
// repoMerge is a merge of a repository of one module's own, from a head branch of its own (catalogueMerge).
func repoMerge(repo, commit string, made time.Time) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Commit: commit,
return link.SourceMoved{Owner: "novox", Repo: repo, Base: "main", Head: "pr/" + commit, Commit: commit,
MergedAt: made.Format(time.RFC3339Nano), Paths: []string{"main.go"}}
}
// noGroups stubs mesh-delivery's word on groups for a test: no head names a group.
func noGroups(t *testing.T) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) { return nil, false, nil }
t.Cleanup(func() { deliveryGroupOf = was })
}
// groupsWaiting stubs mesh-delivery's word on groups for a test: each head named is a group with a member of another
// repository whose pull request is open (novox/elsewhere, ready), so a merge from it waits for the window (ADR 0306
// decision 2); every other head names no group.
func groupsWaiting(t *testing.T, heads ...string) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(_ context.Context, _ bool, head string) ([]groupMember, bool, error) {
if slices.Contains(heads, head) {
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
return nil, false, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
}
// everyHeadWaits stubs mesh-delivery's word on groups for a test whose merges each wait for the window as they did
// before novox/hq ADR 0304: every head is a group with a member of another repository still to merge.
func everyHeadWaits(t *testing.T) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
}
// fromHead is a merge from a head branch a test names: merges from one head are members of one delivery group,
// related by it (novox/hq ADR 0304 decision 1), so they share a batch whatever else they share.
func fromHead(m link.SourceMoved, head string) link.SourceMoved {
m.Head = head
return m
}
// lone is a merge that names no head: known to be no delivery group's, it waits for no window unless it is
// related to a batch assembling (ADR 0304 decision 1).
func lone(m link.SourceMoved) link.SourceMoved {
m.Head = ""
return m
}
// hear is a merge heard at a moment.
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
@@ -108,6 +160,7 @@ func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
// later commit, which names both merges — the earlier carried by the later.
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
@@ -152,9 +205,10 @@ func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
// Merges of three repositories in one window are one walk, one commit for each.
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/three")
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)),
hear(t, open, fromHead(repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)), "feat/three"),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
@@ -187,9 +241,10 @@ func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
// closes at most ten minutes after its first merge however busy.
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
hear(t, open, repoMerge("two", "c2", t0.Add(80*time.Second)), t0.Add(80*time.Second))
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(80*time.Second)), "feat/both"), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
@@ -200,12 +255,13 @@ func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
}
busy := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, repoMerge(repo, "c"+string(rune('a'+i)), last), last)
hear(t, busy, fromHead(repoMerge(repo, "c"+string(rune('a'+i)), last), "feat/busy"), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
@@ -223,6 +279,7 @@ func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
// what is grouped, and that the plan is not yet calculated. It says so on the bus as plan-moved, too.
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/553b7191claude")
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
@@ -235,7 +292,7 @@ func TestTheAssemblingBatchIsShown(t *testing.T) {
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, repoMerge("one", "a6bc0931one", now), now)
hear(t, open, fromHead(repoMerge("one", "a6bc0931one", now), dunst.Head), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
@@ -360,6 +417,7 @@ func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
// moves — named at once on a walk done — and joins the next batch when it does not.
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
@@ -391,6 +449,7 @@ func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
// first; the first delivered answers the older ones, and the search stops.
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
@@ -459,6 +518,7 @@ func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
@@ -496,9 +556,10 @@ func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
// store, a merge handed over again is not doubled, and the batch is cut when its window closes.
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both")
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
hear(t, open, repoMerge("two", "c2", t0.Add(30*time.Second)), t0.Add(30*time.Second))
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(30*time.Second)), "feat/both"), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
@@ -506,7 +567,7 @@ func TestABatchSurvivesARestart(t *testing.T) {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, repoMerge("one", "c1", t0), t0.Add(60*time.Second)) // the bus hands it over again
hear(t, again, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
@@ -563,6 +624,7 @@ func TestAWaitingWalkNamesTheMergesItAnswers(t *testing.T) {
// nothing, and the merge joins the next batch.
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
@@ -716,6 +778,7 @@ func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
// still waits for the word, so no catalogue delivery skips its turn.
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
@@ -860,6 +923,7 @@ func sameMerge(a, b inventory.PlanMerge) bool {
// its delivery would skip its turn. It joins the next catalogue batch.
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
+4
View File
@@ -48,6 +48,10 @@ func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time)
if f.atMost.Before(f.closed) {
f.closed = f.atMost
}
if w.Batch.CutAtOnce && !b.Created.IsZero() {
// Its window closed when it was made, which is its latest too (ADR 0304 decision 1).
f.atMost, f.closed = b.Created, b.Created
}
}
if f.behind != "" {
if walk, err := inv.PlanByID(ctx, f.behind); err == nil {
+27 -19
View File
@@ -241,7 +241,7 @@ func stopWalkWithout(ctx context.Context, inv *inventory.Inventory, id, by, why
}
return p, said, inv.SavePlan(ctx, &p)
}
target, err := batchToWalkAgain(ctx, inv, again, now)
target, folded, foldMoves, err := batchToWalkAgain(ctx, inv, p.ID, again, now)
if err != nil {
return p, said, fmt.Errorf("%s is not stopped: %w", p.ID, err)
}
@@ -262,8 +262,10 @@ func stopWalkWithout(ctx context.Context, inv *inventory.Inventory, id, by, why
if len(said.Kept) > 0 {
p.Note += "; " + keptWords(said.Kept)
}
// The batch first, so no merge names a plan the store does not hold; all of it, or nothing (decision 7).
if err := inv.SavePlansMoving(ctx, []*inventory.Plan{&target, &p}, moves); err != nil {
// The batch first, so no merge names a plan the store does not hold; all of it, or nothing (decision 7) — the
// batches folded into it too.
if err := inv.SavePlansMoving(ctx, append([]*inventory.Plan{&target, &p}, folded...),
append(moves, foldMoves...)); err != nil {
return p, said, fmt.Errorf("%s is not stopped, and none of its merges moved: %w", p.ID, err)
}
// What the batch holds and names now: written again by the cutter's next look should this fail.
@@ -318,36 +320,42 @@ func stopInBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Pla
return nil
}
// batchToWalkAgain is the batch a stopped walk's merges join: the open batch of their kind, or a new one, not yet
// written. The kind is read from what the merges moved.
func batchToWalkAgain(ctx context.Context, inv *inventory.Inventory, again []inventory.BatchedMerge,
now time.Time) (inventory.Plan, error) {
// batchToWalkAgain is the batch a stopped walk's merges join (novox/hq ADR 0299): a batch of their kind not yet cut
// that they are related to (ADR 0304 decision 1, ADR 0306 decision 3), or a new one, not yet written. Related to
// two or more, the newer are folded into the oldest: the plans and merge moves that fold them, to be written in
// the stop's one act. The kind is read from what the merges moved; the stopped walk itself is no walk a cut folds.
func batchToWalkAgain(ctx context.Context, inv *inventory.Inventory, stopping string, again []inventory.BatchedMerge,
now time.Time) (inventory.Plan, []*inventory.Plan, []inventory.MergeMove, error) {
own := false
for _, n := range movesOf(again) {
if _, on := onTheControllersPath[n]; on {
own = true
}
}
batches, err := inv.Batches(ctx)
entries, read, edges, err := theGraph(ctx, inv)
if err != nil {
return inventory.Plan{}, err
return inventory.Plan{}, nil, nil, err
}
for _, b := range batches {
if b.OwnPath() == own {
if b.Delivery == nil {
b.Delivery = &inventory.PlanDelivery{}
}
if b.Delivery.Batch == nil {
b.Delivery.Batch = &inventory.PlanBatch{Own: own}
}
return b, nil
joins, _, _, err := relatedBatches(ctx, inv, again, own, stopping, entries, read, edges)
if err != nil {
return inventory.Plan{}, nil, nil, err
}
if len(joins) > 0 {
into := joins[0]
if len(joins) == 1 {
return into, nil, nil, nil
}
plans, moves, err := foldInto(ctx, inv, &into, joins[1:])
if err != nil {
return inventory.Plan{}, nil, nil, err
}
return into, plans[1:], moves, nil
}
first := again[0]
return inventory.Plan{ID: fmt.Sprintf("plan-%d", now.UnixNano()), Repository: first.Repository, Branch: first.Branch,
Commit: first.Commit, Merged: first.Merged, Created: now, State: inventory.PlanAssembling,
Tiers: [][]string{}, Modules: map[string]*inventory.PlanModule{},
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own}}}, nil
Delivery: &inventory.PlanDelivery{Batch: &inventory.PlanBatch{Own: own}}}, nil, nil, nil
}
// leaveOut splits a walk's merges by the merges named to leave out, each `<repository>@<commit>` (a commit's
@@ -119,6 +119,13 @@ func TestTheBusMachineIsBehindByItsUserListAlone(t *testing.T) {
if userListBehind("", "") {
t.Fatal("a machine sent no list reads as behind")
}
// A list that could not be composed is behind: not known is never "not behind" (the review of hq issue 353).
if !listBehind("", errors.New("the plan cannot be worked out"), digestOf([]byte(list))) {
t.Fatal("a list nobody could compose reads as current")
}
if listBehind(list, nil, digestOf([]byte(list))) || !listBehind(list, nil, "") {
t.Fatal("listBehind does not read the digest as userListBehind does")
}
}
// The machine holding the bus goes first: its declaration carries the user list the new grants are
+158
View File
@@ -0,0 +1,158 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group still assembles** (novox/hq ADR 0306 decision 2): a merge whose delivery group has a member
// not yet merged waits for the merge window, whether or not a batch is assembling, so the group's members walk in
// one batch, in their order (ADR 0276 decision 5). Which groups exist and where each member stands is
// mesh-delivery's record (ADR 0239): a group is the pull requests of two repositories or more sharing a head
// branch name, which is its id. The controller asks it, through its `groups` verb, when a merge from a head is
// heard; a member whose pull request is open (proposed, checked, ready or rejected) is still to merge. When
// mesh-delivery does not answer within groupAskWithin, or answers something unreadable, the merge waits: the safe
// error is to wait, and its batch says why.
// groupAskWithin is how long mesh-delivery is given to say a merge's group, while the merge is being heard.
var groupAskWithin = 3 * time.Second
// openPullStates are the states of a delivery whose pull request is open and not merged (mesh-delivery's table):
// a member in one of them is still to merge. Closed unmerged is stopped, final, and waits for nothing.
var openPullStates = map[string]bool{"proposed": true, "checked": true, "ready": true, "rejected": true}
// groupMember is one member of a delivery group as mesh-delivery's `groups` says it: `<owner/repo>@<commit> (state)`.
type groupMember struct {
Repository, State string
}
// deliveryGroupOf is what mesh-delivery says of the group a head names: its members, and whether the head names a
// group at all; an error when it cannot be asked or did not answer. A seam a test replaces.
var deliveryGroupOf = func(ctx context.Context, held bool, head string) ([]groupMember, bool, error) {
if !held {
return nil, false, fmt.Errorf("no holder of %s is on record to say it", catalogue.DeliverySeat)
}
serving := servingBus.Load()
if serving == nil {
return nil, false, errors.New("this controller is not on the bus")
}
asking, cancel := context.WithTimeout(ctx, groupAskWithin)
defer cancel()
raw, err := askDeliveryOwner(asking, serving.Conn(), "groups", map[string]any{})
if err != nil {
if errors.Is(err, link.ErrNothingServes) {
return nil, false, fmt.Errorf("nothing serves %s.groups", catalogue.DeliverySeat)
}
return nil, false, err
}
var lines []struct {
ID string `json:"id"`
Members []string `json:"members"`
}
if err := json.Unmarshal(raw, &lines); err != nil {
return nil, false, fmt.Errorf("%s.groups answered something unreadable: %w", catalogue.DeliverySeat, err)
}
for _, l := range lines {
if l.ID != head {
continue
}
var members []groupMember
for _, s := range l.Members {
members = append(members, memberOf(s))
}
return members, true, nil
}
return nil, false, nil
}
// memberOf reads one member as `groups` says it: `novox/mesh-sdk@d41d255f28ad (held)`.
func memberOf(s string) groupMember {
id, state := strings.TrimSpace(s), ""
if i := strings.LastIndex(id, " ("); i >= 0 && strings.HasSuffix(id, ")") {
id, state = id[:i], id[i+2:len(id)-1]
}
repository, _, _ := strings.Cut(id, "@")
return groupMember{Repository: repository, State: state}
}
// groupWait says whether a merge heard now waits for the window for its delivery group, and why in words; nothing
// when it names no head, its head names no group, or every other member is merged or closed. own is the merge's
// kind (ownPath).
//
// A member of another repository whose merge this controller already keeps in a batch not yet cut of the same
// kind is merged, whatever mesh-delivery says yet (it marks a member merged once its walk opened): the merge joins
// that batch by their head. One kept in a batch of the other kind keeps the merge waiting, whatever mesh-delivery
// says (review of #250): kinds never share a batch, so the merge's batch is a new one, and cut at once it would
// walk before that member's batch is cut and break the group's order (ADR 0276 decision 5). Waiting, the two
// batches are cut in the order their windows close, an own-path one first at one look, as cutBatchesHeld cuts
// every pair of kinds.
//
// Asked before the plans are held: a slow answer holds up no walk.
func groupWait(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, held, own bool) (string, error) {
if m.Head == "" {
return "", nil
}
same, other, err := keptFromHead(ctx, inv, m.Head, own)
if err != nil {
return "", err
}
if len(other) > 0 {
return "group " + m.Head + " has a member waiting in a batch of the other kind: " + strings.Join(other, ", "), nil
}
members, isGroup, err := deliveryGroupOf(ctx, held, m.Head)
if err != nil {
return fmt.Sprintf("%s did not say whether group %s has a member still to merge (%v): it waits for the "+
"window, the safe side", catalogue.DeliverySeat, m.Head, err), nil
}
if !isGroup {
return "", nil
}
var open []string
for _, g := range members {
if strings.EqualFold(g.Repository, m.Owner+"/"+m.Repo) || same[strings.ToLower(g.Repository)] ||
!openPullStates[g.State] {
continue
}
open = append(open, g.Repository+" ("+g.State+")")
}
if len(open) == 0 {
return "", nil
}
return "group " + m.Head + " has a member still to merge: " + strings.Join(open, ", "), nil
}
// keptFromHead are the merges from a head that batches not yet cut hold: the repositories (lower case) of those in
// a batch of the kind own says, and, in words, each of the other kind with its batch.
func keptFromHead(ctx context.Context, inv *inventory.Inventory, head string, own bool) (map[string]bool, []string, error) {
batches, err := inv.Batches(ctx)
if err != nil {
return nil, nil, err
}
same := map[string]bool{}
var other []string
for _, b := range batches {
merges, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return nil, nil, err
}
for _, k := range merges {
if announcedOf(k).Head != head {
continue
}
if b.OwnPath() == own {
same[strings.ToLower(k.Repository)] = true
} else {
other = append(other, fmt.Sprintf("%s@%s in %s", k.Repository, short(k.Commit), b.ID))
}
}
}
return same, other, nil
}
+11 -5
View File
@@ -123,11 +123,12 @@ func TestDeliveryStopTakesTheMergesToLeaveOut(t *testing.T) {
// that builds the other repository's module alone.
func TestAStopWithoutAMergeWalksTheOthersAgain(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
@@ -181,12 +182,13 @@ func TestAStopWithoutAMergeWalksTheOthersAgain(t *testing.T) {
// ended. A merge the walk does not answer is refused, and nothing ends.
func TestAStopWithoutAMergeEndsTheMergesThatContainIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
first := catalogueMerge("cat1aaaaaaaa", "app", t0)
later := catalogueMerge("cat2cccccccc", "notes", t0.Add(10*time.Second))
one := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), first.Head)
hear(t, open, first, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
hear(t, open, later, t0.Add(11*time.Second))
@@ -238,11 +240,12 @@ func TestAStopWithoutAMergeEndsTheMergesThatContainIt(t *testing.T) {
// (review of #232): its window closes, and its walk is at the newer commit, answering both.
func TestAStopWalksTheOthersAgainInTheOpenBatch(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one1 := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one1 := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one1, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
@@ -290,11 +293,12 @@ func TestAStopWalksTheOthersAgainInTheOpenBatch(t *testing.T) {
// that walked its merges again is refused a retry, saying why.
func TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one1 := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one1 := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one1, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
@@ -335,11 +339,12 @@ func TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt(t *testing.T) {
// with the others and withholds the stopped merge's modules; with every merge left out, the batch is stopped.
func TestAStopWithoutAMergeLeavesItOutOfABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one := repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second))
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
_, bs := walks(t, open)
@@ -382,6 +387,7 @@ func TestAStopWithoutAMergeLeavesItOutOfABatch(t *testing.T) {
// else since it was read refuses all of it, and no merge moves, no plan is written.
func TestSavingPlansAndMovingMergesIsOneAct(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
endsItsLine(t)
ctx := t.Context()
hear(t, open, catalogueMerge("cat1aaaaaaaa", "app", t0), t0.Add(time.Second))
+398
View File
@@ -0,0 +1,398 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// The tests of novox/hq issue 525's window skip: a merge waits for the merge window only when a batch assembling
// holds a merge related to it, or its delivery group may have a member still to merge (ADR 0304 decision 1, ADR
// 0306 decisions 2 and 3). Walks stay one at a time.
// A merge from no head that shares nothing, with no walk open, is cut into its walk when it is heard: no window.
func TestALoneMergeSharingNothingIsCutAtOnce(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, lone(repoMerge("one", "c1", t0)), t0)
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 || !ws[0].Open() || ws[0].Commit != "c1" {
t.Fatalf("a lone merge sharing nothing: %d walk(s) %+v, %d batch(es)", len(ws), ws, len(bs))
}
if ws[0].Times == nil || ws[0].Times.WindowClosed == nil || !ws[0].Times.WindowClosed.Equal(t0) {
t.Fatalf("its window closed at %+v; want when it was heard, %s", ws[0].Times, t0)
}
}
// A merge related to the batch assembling joins it and waits for the window, which its coming restarts: by its
// repository, by a module in common (a dependent of what the batch moves), by an edge of the module graph between
// their modules (one that widens nothing), or by its delivery group's head.
func TestARelatedMergeJoinsTheBatchAndWaitsForTheWindow(t *testing.T) {
ctx := t.Context()
for _, c := range []string{"same repository", "a module in common", "an edge of the graph", "the same group head"} {
t.Run(c, func(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c1")
asksWithPaths(t)
inv := open.inventory
first := repoMerge("one", "c1", t0)
second := lone(repoMerge("two", "c2", t0.Add(10*time.Second)))
why := map[string]string{"same repository": "both change novox/one", "a module in common": "both move two",
"an edge of the graph": "two depends on builder (built-by)",
"the same group head": "both are members of group " + first.Head}[c]
switch c {
case "same repository":
second = lone(repoMerge("one", "c2", t0.Add(10*time.Second)))
case "a module in common":
// two is built on one: a merge of one moves two too.
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "two", Version: "1",
Build: &catalogue.Build{On: []catalogue.BuildsOn{{Arg: "BASE", Module: "one"}}}},
inventory.Source{Repository: "novox/two", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
case "an edge of the graph":
// two is built by the build machine: an edge that orders and widens nothing.
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "builder", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-build-machine", Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/builder", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
first = repoMerge("builder", "c1", t0)
case "the same group head":
second = fromHead(repoMerge("two", "c2", t0.Add(10*time.Second)), first.Head)
}
hear(t, open, first, t0)
said := captured(t, func() error {
return (following{open: open}).hearMerge(ctx, second, t0.Add(10*time.Second))
})
if !strings.Contains(said, "("+why+")") {
t.Fatalf("the merge was not said related by %q: %s", why, said)
}
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the related merge: %d walk(s), batches %+v", len(ws), bs)
}
if bs[0].Delivery.Batch.CutAtOnce {
t.Fatal("a batch a related merge joined is cut at once")
}
cutAt(t, open, t0.Add(99*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window the related merge restarted closed: %+v", ws)
}
cutAt(t, open, t0.Add(100*time.Second))
ws, bs = walks(t, open)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
})
}
}
// A merge from a head whose delivery group has a member still to merge waits for the window, even with no batch
// assembling, and says why; its sibling joins the same batch by their head (ADR 0306 decision 2). A merge from a
// head that names no group shares nothing and is cut at once, as one from no head is.
func TestAMergeOfAGroupWaitsEvenAlone(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
groupsWaiting(t, "feat/x")
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/x"), t0)
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 1 || bs[0].Delivery.Batch.CutAtOnce || windowClosed(bs[0], t0) {
t.Fatalf("a group's merge alone: %d walk(s), batches %+v", len(ws), bs)
}
if words := batchWords(bs[0], t0); !strings.HasPrefix(words, "assembling: 90 s left") ||
!strings.Contains(words, "group feat/x has a member still to merge: novox/elsewhere (ready)") {
t.Fatalf("it reads %q", words)
}
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(5*time.Second)), "feat/x"), t0.Add(5*time.Second))
if _, bs := walks(t, open); len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the group's second member did not join the first: %+v", bs)
}
cutAt(t, open, t0.Add(94*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before its window closed: %+v", ws)
}
cutAt(t, open, t0.Add(95*time.Second))
ws, bs = walks(t, open)
if len(ws) != 1 || len(bs) != 0 || ws[0].CommitOf("novox/one") != "c1" || ws[0].CommitOf("novox/two") != "c2" {
t.Fatalf("at its window: %d walk(s) %+v, %d batch(es)", len(ws), ws, len(bs))
}
}
// A merge from a head that names no group, sharing nothing, with no walk open, is cut at once.
func TestAHeadedMergeInNoGroupIsCutAtOnce(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "fix/alone"), t0)
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 || !ws[0].Open() {
t.Fatalf("a headed merge in no group: %d walk(s), %d batch(es)", len(ws), len(bs))
}
}
// When mesh-delivery cannot say a head's group, the merge waits for the window, the safe side, and its batch says
// why.
func TestAGroupNobodyCanSayWaits(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return nil, false, errors.New("nothing serves mesh-delivery.groups")
}
t.Cleanup(func() { deliveryGroupOf = was })
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/x"), t0)
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 1 || bs[0].Delivery.Batch.CutAtOnce {
t.Fatalf("a merge whose group nobody could say: %d walk(s), batches %+v", len(ws), bs)
}
if words := batchWords(bs[0], t0); !strings.Contains(words, "mesh-delivery did not say whether group feat/x has a "+
"member still to merge (nothing serves mesh-delivery.groups): it waits for the window, the safe side") {
t.Fatalf("it reads %q", words)
}
}
// A member counts as still to merge only while its pull request is open, and not once this controller keeps its
// merge: a group whose other members are merged or closed makes nobody wait.
func TestAGroupsMergedOrClosedMembersMakeNobodyWait(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
members := []groupMember{{Repository: "novox/one", State: "ready"}, {Repository: "novox/two", State: "published"},
{Repository: "novox/three", State: "stopped"}}
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) { return members, true, nil }
t.Cleanup(func() { deliveryGroupOf = was })
m := fromHead(repoMerge("one", "c1", t0), "feat/x")
if why, err := groupWait(ctx, open.inventory, m, true, false); err != nil || why != "" {
t.Fatalf("the group's other members are merged or closed, yet: %q %v", why, err)
}
members = append(members, groupMember{Repository: "novox/four", State: "checked"})
if why, _ := groupWait(ctx, open.inventory, m, true, false); !strings.Contains(why, "novox/four (checked)") {
t.Fatalf("an open member: %q", why)
}
// novox/four's merge is kept in a batch already: merged, whatever mesh-delivery says yet.
hear(t, open, fromHead(repoMerge("two", "c4", t0), "feat/x"), t0)
members[3].Repository = "novox/two"
if why, _ := groupWait(ctx, open.inventory, m, true, false); why != "" {
t.Fatalf("a member this controller keeps the merge of still waits: %q", why)
}
if got := memberOf("novox/mesh-sdk@d41d255f28ad (held)"); got != (groupMember{Repository: "novox/mesh-sdk", State: "held"}) {
t.Fatalf("a member reads %+v", got)
}
}
// A merge related to two batches assembling folds the newer into the older, which answers every merge of both
// and names nothing twice; the folded one names the batch that took it over.
func TestAMergeRelatedToTwoBatchesFoldsThem(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/a", "feat/b")
asksWithPaths(t)
ctx := t.Context()
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/a"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(5*time.Second)), "feat/b"), t0.Add(5*time.Second))
_, bs := walks(t, open)
if len(bs) != 2 {
t.Fatalf("two merges sharing nothing: %d batch(es)", len(bs))
}
older, newer := bs[1], bs[0] // newest first
// Of one's repository and from two's head: related to both.
hear(t, open, fromHead(repoMerge("one", "c3", t0.Add(10*time.Second)), "feat/b"), t0.Add(10*time.Second))
_, bs = walks(t, open)
if len(bs) != 1 || bs[0].ID != older.ID || len(bs[0].Delivery.Merges) != 3 {
t.Fatalf("after a merge related to both: batches %+v; want %s answering three merges", bs, older.ID)
}
folded, err := open.inventory.PlanByID(ctx, newer.ID)
if err != nil || folded.State != inventory.PlanSuperseded || folded.Delivery == nil ||
folded.Delivery.TakenOverBy != older.ID {
t.Fatalf("the newer batch reads %s %+v (%v)", folded.State, folded.Delivery, err)
}
if left, _ := open.inventory.MergesOf(ctx, newer.ID); len(left) != 0 {
t.Fatalf("the folded batch still answers %+v", left)
}
cutAt(t, open, t0.Add(100*time.Second))
ws, bs := walks(t, open)
var cut []inventory.Plan
for _, w := range ws {
if w.State != inventory.PlanSuperseded {
cut = append(cut, w)
}
}
if len(cut) != 1 || len(bs) != 0 || cut[0].ID != older.ID || cut[0].CommitOf("novox/one") != "c3" ||
cut[0].CommitOf("novox/two") != "c2" {
t.Fatalf("the folded batch's walk: %+v, %d batch(es)", cut, len(bs))
}
}
// With a walk open, a batch cut at once still waits for it, said queued behind it (walks stay one at a time, ADR
// 0276 decision 4); it is cut once that walk ended, with no window to wait for.
func TestACutAtOnceBatchQueuesBehindAnOpenWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c1")
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("no walk open: %+v", ws)
}
first := ws[0]
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("two", "c2", heard)), heard)
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID ||
!bs[0].Delivery.Batch.CutAtOnce {
t.Fatalf("a lone merge during a walk: %d walk(s), batches %+v", len(ws), bs)
}
if !strings.HasPrefix(batchWords(bs[0], heard), "queued behind "+first.ID) {
t.Fatalf("it reads %q", batchWords(bs[0], heard))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, heard.Add(time.Second))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut once the walk ended: %+v %+v", ws, bs)
}
if ws[0].Times == nil || ws[0].Times.WindowClosed == nil || !ws[0].Times.WindowClosed.Equal(heard) {
t.Fatalf("its window closed at %+v; want when its merge was heard, %s", ws[0].Times, heard)
}
}
// A stopped walk's other merges are walked again in a batch they are related to, and a batch of their own when
// they share nothing with the one assembling (ADR 0299, ADR 0304 decision 1): never in a batch of another
// change, whose window the stop would close.
func TestAStopWalksTheOthersAgainInABatchOfTheirOwnWhenTheyShareNothing(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/cat1aaaaaaaa")
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
cat := catalogueMerge("cat1aaaaaaaa", "app", t0)
one := fromHead(repoMerge("one", "one1bbbbbbbb", t0.Add(5*time.Second)), cat.Head)
hear(t, open, cat, t0.Add(time.Second))
hear(t, open, one, t0.Add(6*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
walk := ws[0]
hear(t, open, repoMerge("two", "two1cccccccc", t0.Add(3*time.Minute)), t0.Add(3*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 {
t.Fatalf("the merge heard while the walk ran is in %d batches", len(bs))
}
other := bs[0]
_, said, err := stopWalkWithout(ctx, open.inventory, walk.ID, "mesh-delivery for jochen", "its group stopped",
[]string{"novox/mesh-catalog@" + cat.Commit}, t0.Add(3*time.Minute+10*time.Second))
if err != nil {
t.Fatal(err)
}
if said.WalkedAgainBy == "" || said.WalkedAgainBy == other.ID {
t.Fatalf("walked again by %q; want a batch of its own, not %s", said.WalkedAgainBy, other.ID)
}
b, _ := open.inventory.PlanByID(ctx, other.ID)
if b.Delivery.Batch == nil || b.Delivery.Batch.Closed || len(b.Delivery.Merges) != 1 {
t.Fatalf("the other change's batch reads %+v; want it untouched by the stop", b.Delivery)
}
}
// A catalogue batch's cut folds in every walk waiting for its delivery's word (cutBatch's carry), so what that
// walk moves is the batch's too: a lone merge related to the waiting walk joins the batch assembling and waits for
// its window, rather than being cut at once as sharing nothing with what the batch's own merges move.
func TestALoneMergeRelatedToAWalkTheCutFoldsJoinsTheBatch(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c3")
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
hear(t, open, repoMerge("three", "c3", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
hear(t, open, lone(repoMerge("one", "c2", t0.Add(2*time.Minute+5*time.Second))), t0.Add(2*time.Minute+5*time.Second))
_, bs := walks(t, open)
if len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 || bs[0].Delivery.Batch.CutAtOnce {
t.Fatalf("a lone merge related to the waiting walk: batches %+v", bs)
}
}
// A delivery group with a member of each kind (review of #250): a merge on the controller's own path and a
// catalogue merge never share a batch, so the later member cannot join the earlier one's. Kept in a batch of the
// other kind, the earlier member keeps the later one waiting, whatever mesh-delivery says yet — never cut at once
// to walk before it. The two batches are then cut in the order their windows close (cutBatchesHeld: the oldest
// closed batch first, an own-path one first at one look, and the other queued behind a walk that started): the
// earlier member's window closes first, so it walks first, in both orders.
func TestAGroupAcrossBothKindsKeepsItsOrder(t *testing.T) {
for _, ownFirst := range []bool{true, false} {
name := "catalogue member first"
if ownFirst {
name = "controller member first"
}
t.Run(name, func(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
was := deliveryGroupOf
// mesh-delivery still says both members ready: it marks one merged only once its walk opened.
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return []groupMember{{Repository: "novox/mesh-controller", State: "ready"},
{Repository: "novox/mesh-catalog", State: "ready"}}, true, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
own := fromHead(repoMerge("mesh-controller", "k1", t0), "feat/both-kinds")
cat := fromHead(catalogueMerge("cat1aaaaaaaa", "app", t0), "feat/both-kinds")
first, second := cat, own
if ownFirst {
first, second = own, cat
}
second.MergedAt = t0.Add(10 * time.Second).Format(time.RFC3339Nano)
hear(t, open, first, t0)
hear(t, open, second, t0.Add(10*time.Second))
ws, bs := walks(t, open)
if len(ws) != 0 || len(bs) != 2 {
t.Fatalf("both members heard: %d walk(s), %d batch(es); want two batches, nothing cut", len(ws), len(bs))
}
later := bs[0] // newest first
if later.Delivery.Batch.CutAtOnce || !strings.Contains(later.Delivery.Batch.Waits,
"has a member waiting in a batch of the other kind: novox/"+first.Repo+"@") {
t.Fatalf("the later member's batch reads %+v; want it waiting for the earlier member", later.Delivery.Batch)
}
// The earlier member's window closes first: its batch alone is cut.
cutAt(t, open, t0.Add(90*time.Second))
ws, bs = walks(t, open)
if len(ws) != 1 || !ws[0].Open() || ws[0].CommitOf("novox/"+first.Repo) != first.Commit || len(bs) != 1 ||
bs[0].ID != later.ID {
t.Fatalf("at the earlier window: walks %+v, batches %+v; want the earlier member alone cut", ws, bs)
}
earlier := ws[0]
// The later member's window closes while the earlier walk runs: it queues behind it.
cutAt(t, open, t0.Add(100*time.Second))
if _, bs := walks(t, open); len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != earlier.ID {
t.Fatalf("the later member's batch does not queue behind the earlier walk: %+v", bs)
}
earlier.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &earlier); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(101*time.Second))
got, err := open.inventory.PlanByID(ctx, later.ID)
if err != nil || got.Batch() || got.CommitOf("novox/"+second.Repo) != second.Commit {
t.Fatalf("the later member was not walked after the earlier: %s %v", got.State, err)
}
})
}
}
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"bytes"
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq issue 237: every line of an act's answer comes from one resolution of the node taken after
// its write, so the answer agrees with `plan` for the node right after.
func aProviderAndItsConsumer(t *testing.T, open *stores) {
t.Helper()
register(t, open, catalogue.Manifest{Module: "db", Version: "1",
Provides: []catalogue.Offer{{Name: "rows", Scope: catalogue.ScopeNode}}})
register(t, open, catalogue.Manifest{Module: "app", Version: "1", Requires: []string{"rows"}})
}
// twoRivalsAndANote is a node that does not resolve — two modules claim one node seat there — and a
// module beside them that changes nothing about it.
func twoRivalsAndANote(t *testing.T, open *stores) {
t.Helper()
one, two := rivals()
register(t, open, one)
register(t, open, two)
register(t, open, catalogue.Manifest{Module: "notes", Version: "1"})
}
func TestAnUnassignThatLeavesItsNodeUnresolvedSaysSoOnceInItsAnswer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
twoRivalsAndANote(t, open)
// Straight into the store, as a node that came to this before the act.
for _, m := range []string{"rival-one", "rival-two", "notes"} {
if _, err := open.inventory.Assign(ctx, "laptop", m); err != nil {
t.Fatal(err)
}
}
said := answered(t, func() error { return assignCommand(ctx, "unassign", []string{"laptop", "notes"}) })
if _, _, err := planFor(ctx, open, "laptop"); err == nil || !unresolvable(err) {
t.Fatalf("laptop resolves with two claimants of one seat: %v", err)
}
if n := strings.Count(said, "laptop does not resolve now"); n != 1 {
t.Errorf("the answer says laptop does not resolve %d times, want once:\n%s", n, said)
}
if strings.Contains(said, "is left out of the rest of the mesh") {
t.Errorf("the answer carries a line from a read that is not its resolution:\n%s", said)
}
}
func TestAnUnassignThatLeavesItsNodeResolvedSaysNothingOfIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
aProviderAndItsConsumer(t, open)
if _, err := assign(ctx, open, "laptop", "db", "app"); err != nil {
t.Fatal(err)
}
said, err := unassign(ctx, open, "laptop", "app")
if err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
if strings.Contains(said, "does not resolve") || strings.Contains(said, "could not be read") {
t.Errorf("a node that resolves is said not to:\n%s", said)
}
}
func TestAnAssignThatLeavesItsNodeUnresolvedSaysOnlyThePlansRefusal(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
twoRivalsAndANote(t, open)
if _, err := assign(ctx, open, "laptop", "rival-one"); err != nil {
t.Fatal(err)
}
// Kept and not resolvable: a second claimant of the seat, which assignment keeps.
said, err := assign(ctx, open, "laptop", "rival-two")
if err == nil {
t.Fatalf("two claimants of one node seat resolved:\n%s", said)
}
_, _, planErr := planFor(ctx, open, "laptop")
if planErr == nil || planErr.Error() != err.Error() {
t.Errorf("the act's refusal is not the plan's right after:\nact: %v\nplan: %v", err, planErr)
}
if strings.Contains(said, "now has") {
t.Errorf("the answer says a seat is now held on a node that does not resolve:\n%s", said)
}
}
// The other readers of the whole mesh still say a machine they leave out, on standard error (novox/hq
// issue 188); an act sends it elsewhere.
func TestAMachineLeftOutIsSaidWhereTheContextSays(t *testing.T) {
var buf bytes.Buffer
sayLeftOut(leftOutSaidBy(context.Background(), &buf), "laptop", errors.New("why"))
if buf.String() != "laptop is left out of the rest of the mesh: it does not resolve: why\n" {
t.Errorf("said %q", buf.String())
}
got := answered(t, func() error {
sayLeftOut(context.Background(), "laptop", errors.New("why"))
return nil
})
if !strings.Contains(got, "laptop is left out") {
t.Errorf("without an act's writer the line did not reach standard error: %q", got)
}
}
+61 -17
View File
@@ -292,7 +292,7 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
if err != nil {
// Said, not skipped: a machine dropped here offers nothing and holds nothing as far
// as every other machine's plan can tell (novox/hq issue 188).
fmt.Fprintf(os.Stderr, "%s is left out of the rest of the mesh: it does not resolve: %v\n", o.node.Name, err)
sayLeftOut(ctx, o.node.Name, err)
continue
}
firstHeld = append(firstHeld, got.Claims...)
@@ -337,7 +337,7 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
// resolved without its offers, and one that consumes them cannot resolve here by
// design — that is not the machine being dropped, it is the view being partial.
if exclude == "" {
fmt.Fprintf(os.Stderr, "%s is left out of the rest of the mesh: it does not resolve: %v\n", o.node.Name, err)
sayLeftOut(ctx, o.node.Name, err)
}
continue
}
@@ -347,6 +347,35 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
return world, nil
}
// leftOutKey carries where theRestOfTheMesh says a machine it leaves out, for an act that answers with
// its own resolution instead (novox/hq issue 237).
type leftOutKey struct{}
// leftOutSaidBy is ctx with theRestOfTheMesh's "left out" lines written to w rather than to standard
// error.
//
// **An act's answer is one resolution taken after its write** (novox/hq issue 237, decided 2026-10-11).
// `assign` read the whole mesh once before its write — to record the seats held by derivation — and that
// read said, on standard error, that the node did not resolve for lack of the seat the act was about to
// give it. The controller seat's verb answers with both streams, so the stale line came last, after the
// act's own "now has it held", and named as its remedy the assignment just made. An act gives io.Discard
// here: what the node resolves to after the write is said by its own planFor, and every other machine
// that does not resolve by blockedElsewhere, both read after the write.
func leftOutSaidBy(ctx context.Context, w io.Writer) context.Context {
return context.WithValue(ctx, leftOutKey{}, w)
}
// sayLeftOut says that a machine is left out of the rest of the mesh, where ctx says to: standard
// error unless an act said otherwise. Said, not skipped: a machine dropped there offers nothing and
// holds nothing as far as every other machine's plan can tell (novox/hq issue 188).
func sayLeftOut(ctx context.Context, node string, err error) {
w, ok := ctx.Value(leftOutKey{}).(io.Writer)
if !ok {
w = os.Stderr
}
fmt.Fprintf(w, "%s is left out of the rest of the mesh: it does not resolve: %v\n", node, err)
}
// declarationFor is everything a node would be sent.
//
// One place, because there were three and one of them was written before credentials existed and
@@ -1751,28 +1780,43 @@ func (l *unheldLogger) log(node string, unheld []catalogue.Unheld) {
// which includes every one of a module just assigned — and each now met that was not. Nothing about
// any other node, and nothing that was already true before the act.
func unheldChange(shelf map[string]catalogue.Manifest, node string, before, after []string) []string {
judge := func(names []string) map[string]catalogue.Unheld {
var set []catalogue.Manifest
for _, n := range names {
if m, known := shelf[n]; known {
set = append(set, m)
}
return unheldChangeTo(shelf, node, before, after, unheldOver(shelf, node, after))
}
// unheldOver is the node's unmet seat dependencies judged over a set of module names alone.
func unheldOver(shelf map[string]catalogue.Manifest, node string, names []string) []catalogue.Unheld {
var set []catalogue.Manifest
for _, n := range names {
if m, known := shelf[n]; known {
set = append(set, m)
}
out := map[string]catalogue.Unheld{}
for _, u := range catalogue.UnheldDependencies(shelf, node, set, nil) {
out[u.Module+" "+u.Seat] = u
}
return out
}
was, now := judge(before), judge(after)
return catalogue.UnheldDependencies(shelf, node, set, nil)
}
// unheldChangeTo is unheldChange with the node's dependencies as they now stand given: `now` is what
// the act's one resolution after its write says is unmet, and `after` the modules that resolution has
// on the node (novox/hq issue 237). A line about how things stand after the act is never judged from
// anything else, so it cannot disagree with `plan` for the node.
func unheldChangeTo(shelf map[string]catalogue.Manifest, node string, before, after []string,
now []catalogue.Unheld) []string {
key := func(u catalogue.Unheld) string { return u.Module + " " + u.Seat }
was := map[string]catalogue.Unheld{}
for _, u := range unheldOver(shelf, node, before) {
was[key(u)] = u
}
is := map[string]catalogue.Unheld{}
for _, u := range now {
is[key(u)] = u
}
var lines []string
for _, k := range sortedNames(now) {
for _, k := range sortedNames(is) {
if _, already := was[k]; !already {
lines = append(lines, "but "+now[k].String())
lines = append(lines, "but "+is[k].String())
}
}
for _, k := range sortedNames(was) {
if _, still := now[k]; still {
if _, still := is[k]; still {
continue
}
u := was[k]
+1 -1
View File
@@ -591,7 +591,7 @@ func proposeCommand(ctx context.Context, module, node, valuesArg string, clear,
return err
},
routerHere: routerHereIn(open.inventory),
grantHeld: grantHeldIn(open),
grantHeld: grantHeldIn(open, conn),
routerRecord: func(ctx context.Context, id string) (string, error) {
bucket, err := asksRecords(ctx, open.inventory)
if err != nil || bucket == "" {
+29 -13
View File
@@ -1084,42 +1084,58 @@ func brokerFirst(names []string, holder string, behind bool) []string {
// send anywhere, and a module running there always put it in its first wave. A digest of the list
// last sent is kept for this (ADR 0043: the list is composed on each push, never kept itself).
func brokerBehind(ctx context.Context, open *stores, names []string) (string, bool, error) {
s, err := brokerListState(ctx, open, names)
return s.Machine, s.Behind, err
}
// brokerListState is brokerBehind with why the list could not be composed, when it could not: the plan's own
// reason, which a remedy names rather than a push that would fail on it (the review of hq issue 353).
func brokerListState(ctx context.Context, open *stores, names []string) (busListState, error) {
inv := open.inventory
holders, err := seatHolders(ctx, inv)
if err != nil {
return "", false, err
return busListState{}, err
}
h, held := holders[theBrokerSeat]
if !held || h.Node == "" {
return "", false, nil
return busListState{}, nil
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
return "", false, err
return busListState{}, err
}
if m, known := shelf[h.Module]; !known || m.BusUsers == "" {
// A holder that is sent no user list carries no grant: nothing to send first.
return "", false, nil
return busListState{}, nil
}
for _, n := range names {
if n == h.Node {
return h.Node, false, nil
return busListState{Machine: h.Node}, nil
}
}
// **What cannot be worked out is behind** (the review of hq issue 353). The list it would be sent cannot be
// composed, so whether it carries the grants a send relies on is not known; it is taken as behind, so a push
// sends that machine first and says why it cannot (`plan` says it too), and a rollout that needs the list
// refuses rather than sending code whose grants the bus may not hold. It said "not behind" until then: a
// send went ahead as if the bus held a list nobody could compose.
list, composeErr := "", error(nil)
plan, _, err := planFor(ctx, open, h.Node)
if err != nil {
// It cannot be worked out: sending it would refuse the whole send, and `plan` says why.
return h.Node, false, nil
}
list, _, err := busUserList(ctx, inv, plan.Modules)
if err != nil {
return h.Node, false, nil
composeErr = err
} else if list, _, err = busUserList(ctx, inv, plan.Modules); err != nil {
composeErr = err
}
sent, err := inv.SentBusUsers(ctx, h.Node)
if err != nil {
return "", false, err
return busListState{}, err
}
return h.Node, userListBehind(list, sent), nil
return busListState{Machine: h.Node, Behind: listBehind(list, composeErr, sent), Uncomposed: composeErr}, nil
}
// listBehind is whether the bus's machine is behind: the list composed now is not the one last sent, or it
// could not be composed at all — not known is behind, never "not behind".
func listBehind(list string, composeErr error, sentDigest string) bool {
return composeErr != nil || userListBehind(list, sentDigest)
}
// userListBehind is whether the user list composed now is not the one last sent, by its digest. An
+85
View File
@@ -0,0 +1,85 @@
package main
import (
"bytes"
"io"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// TestReplay237 replays novox/hq issue 237 (2026-10-04): a workstation already ran a module that
// depends on a node seat nothing there held, and the module holding that seat was assigned to it. The
// answer said the seat was now held, and then that the workstation was left out of the rest of the mesh
// for lack of it, naming as the remedy the assignment just made. `plan` for the node, right after,
// resolved. The outcome asserted: every line of the answer, as the controller seat's verb gives it
// (standard output, then standard error), agrees with `plan` for the node right after the act. Written
// with only what the controller had before its fix — register, assign's command, planFor — so it is laid
// over the commit before.
func TestReplay237(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aContainer("web"))
register(t, open, catalogue.Manifest{Module: "docker", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.ContainerRuntimeSeat, Scope: catalogue.ScopeNode}}})
// Already on the node, from before the rule refused it: the node does not resolve until the
// holder comes.
if _, err := open.inventory.Assign(ctx, "laptop", "web"); err != nil {
t.Fatal(err)
}
said := answered(t, func() error { return assignCommand(ctx, "assign", []string{"laptop", "docker"}) })
if _, _, err := planFor(ctx, open, "laptop"); err != nil {
t.Fatalf("laptop does not resolve right after the act: %v\nthe answer was:\n%s", err, said)
}
if !strings.Contains(said, "web on laptop now has "+catalogue.ContainerRuntimeSeat+" held") {
t.Errorf("the answer does not say the seat is now held:\n%s", said)
}
for _, not := range []string{"laptop is left out", "does not resolve", "which nothing on laptop holds"} {
if strings.Contains(said, not) {
t.Errorf("the answer says %q, which `plan` right after contradicts:\n%s", not, said)
}
}
}
// answered is what a command says as the controller seat's verb answers it: standard output, then
// standard error (answerOf).
func answered(t *testing.T, f func() error) string {
t.Helper()
read := func(w **os.File) (func() string, error) {
old := *w
r, pw, err := os.Pipe()
if err != nil {
return nil, err
}
*w = pw
got := make(chan string)
go func() {
var buf bytes.Buffer
_, _ = io.Copy(&buf, r)
got <- buf.String()
}()
return func() string {
_ = pw.Close()
*w = old
return <-got
}, nil
}
stdout, err := read(&os.Stdout)
if err != nil {
t.Fatal(err)
}
stderr, err := read(&os.Stderr)
if err != nil {
t.Fatal(err)
}
runErr := f()
said := stdout() + stderr()
if runErr != nil {
t.Fatalf("%v\n%s", runErr, said)
}
return said
}
+4 -1
View File
@@ -262,9 +262,12 @@ func MaySubscribe(perms Permissions, subject string) bool {
// delivery held past its bound, and calls them on the operator's warrant, with its why.
//
// And, since novox/hq ADR 0282, `times`: the self-check reads the delivery times to say a delivery over its budget.
//
// And, since novox/hq ADR 0306, `groups`: a merge heard from a head waits for the window while its delivery group
// has a member still to merge.
var VerbsTheControllerAsksTheDeliveryOwner = []SeatVerb{{Seat: "mesh-delivery", Verb: "stalled"},
{Seat: "mesh-delivery", Verb: "close"}, {Seat: "mesh-delivery", Verb: "release"}, {Seat: "mesh-delivery", Verb: "stop"},
{Seat: "mesh-delivery", Verb: "times"}}
{Seat: "mesh-delivery", Verb: "times"}, {Seat: "mesh-delivery", Verb: "groups"}}
// VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant
// chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the
+1 -1
View File
@@ -24,7 +24,7 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
allow_responses: { max: 1, ttl: "1m" }
} }
+8
View File
@@ -125,6 +125,14 @@ type PlanBatch struct {
// the batch's walk does not build them.
Closed bool `json:"closed,omitempty"`
Withheld []string `json:"withheld,omitempty"`
// CutAtOnce says the batch's window closed when it was made: its merge shares nothing with a batch
// assembling and has no delivery group member still to merge, so it waits for no window (novox/hq ADR 0304
// decision 1, ADR 0306 decisions 2 and 3). A merge that joins it later opens its window again. It is not
// Closed, which is a stop's.
CutAtOnce bool `json:"cut_at_once,omitempty"`
// Waits says why a batch made for one merge waits for the window: its delivery group has a member still to
// merge, or mesh-delivery could not say whether it has (novox/hq ADR 0306 decision 2).
Waits string `json:"waits,omitempty"`
}
// OwnPath says the record is a batch of merges on the controller's own path.
+1 -1
View File
@@ -8,7 +8,7 @@ private addresses. That is acceptable while mesh-controller is private; if it ev
a second place to clean besides the catalogue itself.
mesh-catalog b9de001833b1b61b297182b8e3bcdae1cbdeace9 modules/*/module.json, modules/nats/Dockerfile
mesh-host bd5cc6980419c1bd4824be6d58dfebd2381a9de3 examples/foundation-first-node-nats.lock
mesh-host a76baa4b47f07a8b2bab065603414530bd02b53a examples/foundation-first-node-nats.lock
mesh-sdk 33917f91ac3cc213b879b32f3c70e2aa70395f98 conformance/events/module-event.json
To move them, from this repository's root, with the two repositories checked out beside it:
@@ -161,7 +161,7 @@
"type": "file",
"path": "/var/lib/mesh-bus-conf/accounts.conf",
"mode": "0600",
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
{
"id": "broker",
@@ -31,6 +31,13 @@ package apply
// 5. **A mount source that is a refused directory or a refused access is refused with it**: the container
// would otherwise bind the very path the engine would not place, and the runtime creates a missing one
// as root.
// 6. **A file is judged as a directory is** (novox/hq issue 495): never one of the machine's own roots, nor
// in a tree nothing is placed in (rule 2), and never below a directory or an access refused here. A file
// or an archive below a refused path is refused with it, whichever module declares it and in whatever
// order. Rules 3 and 4 are a directory's owner and do not reach a file: a file below /etc is the mesh's
// ordinary configuration, and one below a home is judged by home_links.go and agent_home_guard.go.
// A refused directory also gates the rest of its module, as a step that did not complete does
// (apply.go's `gated`): what a module declares after its directory was written for that directory.
//
// Each is a failed resource with its reason in the node's report; nothing is touched. Paths are judged as
// declared and again with every link in them resolved, so a link at /srv/x pointing at /etc places nothing.
@@ -400,6 +407,7 @@ const (
placingDirectory placing = iota
placingAccess
placingMount
placingFile
)
// refusePath says why a path is not placed or mounted; nil when it may be. module is the resource's module,
@@ -466,7 +474,7 @@ func refuseOne(path string, kind placing, module, owner string) *PlacementRefuse
for _, part := range strings.Split(path, "/") {
ssh = ssh || part == ".ssh"
}
if ssh && kind != placingDirectory {
if ssh && (kind == placingAccess || kind == placingMount) {
return &PlacementRefusedError{Path: path, Why: "it is an account's .ssh, which holds its keys and who may " +
"log in as it, and is never an access or a mount"}
}
@@ -502,8 +510,12 @@ func moduleOfID(id string) string {
return module
}
// refusedPlaces judges every directory and access of a declaration before anything is applied, and answers
// each refusal by path: what is refused is refused again as a container's mount source.
// refusedPlaces judges every directory, access, file and archive of a declaration before anything is applied,
// and answers each refusal by path: what is refused is refused again as a container's mount source.
//
// Directories and accesses first, so a file or an archive below one of them is refused with it whatever order
// they are declared in (novox/hq issue 495): the engine refused Docker's data root as a directory and then wrote
// a file into it, because nothing judged the file's path.
func refusedPlaces(resources []declaration.Resource) map[string]error {
out := map[string]error{}
for _, r := range resources {
@@ -520,9 +532,60 @@ func refusedPlaces(resources []declaration.Resource) map[string]error {
out[filepath.Clean(r.Target())] = err
}
}
trees := make(map[string]error, len(out))
for path, err := range out {
trees[path] = err
}
for _, r := range resources {
var err error
switch res := r.(type) {
case *declaration.File:
if err = belowRefused(res.Path, trees); err == nil {
err = refusePath(res.Path, placingFile, moduleOfID(res.ID), "")
}
case *declaration.Archive:
// Only below a refused path: where an archive may unpack is otherwise the archive's own judgement
// (archive.go), and the engine's own module unpacks its versions into its own tree.
err = belowRefused(res.Path, trees)
default:
continue
}
if err != nil {
out[filepath.Clean(r.Target())] = err
}
}
return out
}
// belowRefused says why a path below a refused directory or access is refused with it; nil when it is below
// none. Judged as declared and with its links resolved, as refusePath judges.
func belowRefused(path string, refused map[string]error) error {
clean := filepath.Clean(path)
if !filepath.IsAbs(clean) {
return nil
}
for _, p := range []string{clean, resolved(clean)} {
// The deepest refused tree it is below, so the same declaration always gives the same reason.
var tree string
for t := range refused {
if below(p, t) && len(t) > len(tree) {
tree = t
}
}
if tree == "" {
continue
}
why := refused[tree]
reason := why.Error()
var refusal *PlacementRefusedError
if errors.As(why, &refusal) {
reason = refusal.Why
}
return &PlacementRefusedError{Path: clean, Why: "it is below " + tree + ", which is refused: " + reason}
}
return nil
}
// refuseMounts says why a container's mounts are not made; nil when they may be.
func refuseMounts(c *declaration.Container, refused map[string]error) error {
for _, v := range c.Volumes {
+4
View File
@@ -22,3 +22,7 @@
./cmd/mesh-controller TestAWalkEndedByCancelOrClearIsSaidEnded
./cmd/mesh-controller TestAWalkRestartedByARebuildIsSaidStarted
./cmd/mesh-controller TestAWalkStoppedByHandAndRetriedEndsFailedWhenItFailsAgain
./cmd/mesh-controller TestReplay237
./cmd/mesh-controller TestAnUnassignThatLeavesItsNodeUnresolvedSaysSoOnceInItsAnswer
./cmd/mesh-controller TestAnUnassignThatLeavesItsNodeResolvedSaysNothingOfIt
./cmd/mesh-controller TestAnAssignThatLeavesItsNodeUnresolvedSaysOnlyThePlansRefusal