Compare commits

..
11 Commits
Author SHA1 Message Date
mesh-tools b691428ffc Merge pull request 'Builder: the publish dispatch sends live first at one processor; judge test row; done-key test bound (H-A-465, H-A-469, H-A-473)' (#329) from fix/main-tests into main 2026-11-22 13:44:24 +00:00
mesh-tools a41a7c2862 Merge pull request 'Orphaned durables: removable by the bus module, left out of the behind check, not composed again (H-A-472)' (#328) from fix/orphan-durables into main 2026-11-22 13:42:31 +00:00
jschoubben 7e103f3221 Builder: with again items held, fetch live once more before the gap read
A live item put after the turn's live fetch and before the again items were held
was not yet delivered at the gap read, so no gap showed and the later again item
went first (review round 3 on c37a7317, publishitem.go:792-809). With again items
held the dispatch now fetches live without waiting once more: what it returns runs
and the again items stay held; only an empty fetch leads to the gap read. The test's
idle-dispatch case is a subtest of its own, so it runs alone.

Builds: design 47 H2 §793, H-A-473 (third addendum)
2026-11-22 14:40:17 +01:00
jschoubben c37a7317b7 Builder: every again fetch and gap read follows a live fetch
On the turn after an again item ran, the dispatch fetched again and read the live gap
without fetching live first, so a live item the stream held but had not yet given to
the waiting live pull showed no gap and a later again item went first (review round 2
on b77b70a6, publishitem.go:791). The live consumer is now fetched without waiting on
every turn no live item is held. The test row adds that case and bounds its rounds by
the level's set-up share.

Builds: design 47 H2 §793, H-A-473 (third addendum)
2026-11-22 14:35:41 +01:00
jschoubben b2c9e60ac3 Grants: the bound of a current twin is the setting twin-current-hours
Review round 1: the day after which a twin is no longer current was a
constant here and again in the bus module. It is now the setting
twin-current-hours of the manifest (24 by default), written into a file
the part reads at its start, the field name the bus module's manifest
names too, so the operator sets it once per module alike; the
durable-orphaned condition names the bound. hasOwnProcess no longer sits
under ownServesOf's doc comment.

Builds: design 47 H2 H-A-472 (2) and its third addendum (2); review round 1 findings (2), (3)
2026-11-22 14:34:06 +01:00
jschoubben b77b70a66c Builder: an again item waits for a live one in flight to the idle dispatch's waiting pull
The idle dispatch keeps a waiting pull on each consumer, the bus's only wake for an
item put or delivered again. A live item the waiting pull was given but the dispatch
had not yet received let an again item go first (review finding on 1f33a4c0,
publishitem.go:700, 761, 797). An again item taken while the live pull waits now runs
only once the live consumer's delivered sequence is no further than the live items
received; a gap holds it until the live item arrives and runs.

Builds: design 47 H2 §793, H-A-473 (third addendum)
2026-11-22 14:30:22 +01:00
jschoubben 7f5f175b05 Builder whole test: done builds under the level's set-up bound; the hang at its own
The test ran the builder at a 1 s step bound throughout, so a loaded build
node could fail a build that must end done. Those builds now run under
testbus.SetupBound; the hung step's part starts the builder again at 1 s,
and the build again after it under the set-up bound.

Builds: design 47 H2 §793; H-A-473 (addendum)
2026-11-22 14:16:37 +01:00
jschoubben 1f33a4c04f Builder: the publish dispatch pulls the live consumer itself, then the again
The dispatch read the live consumer's pending count against items the
pull's callback had not yet handed it, so at one processor an again item
was sent before a live one already delivered. It now fetches the live
consumer without waiting and pulls the again consumer only when that
returns nothing; an item is received when its fetch returns it.

Builds: design 47 H2 §793; H-A-473
2026-11-22 14:14:15 +01:00
jschoubben 49c656a4ce Grants: compose a module's program consumer only where it names a program
The pass made <node>_<module> beside <node>_<module>_tools for every
consuming module, so a module whose events only the tool runner reads
got a durable nobody pulls, which piled up and came back after each
removal. The program's consumer is now composed only for a module with
an account of its own and a process resource with no bus of its own;
the tool runner's twin stays for every consuming module.

Builds: design 47 H2 H-A-472 addendum and second addendum, G A-397
2026-11-22 14:12:45 +01:00
jschoubben 9005a2ee8e Grants: the behind check leaves out an orphaned durable and names it apart
About 110 durables left by modules that moved into or out of the tool
runner pile up behind their stream's head and raised 'Messages pile up
for a listener' for no real backlog. A durable with no pulls waiting
whose twin on the stream (<node>_<module> and <node>_<module>_tools) has
pulls waiting and delivered within the day is now left out of that check
and named once in mesh-grants.durable-orphaned, cleared when gone.

Builds: design 47 H2 H-A-472 (2), G A-340
2026-11-22 14:03:55 +01:00
jschoubben b914974b1d Judge test: the unknown top-level word is one no SDK knows
The row proving that the judgement's strict read refuses an unknown
top-level word used class, which the H2 SDK knows, so it failed at main.
It now uses a word no SDK knows and proves the same refusal.

Builds: design 47 H2 H-A-465, H-A-469
2026-11-22 13:57:22 +01:00
17 changed files with 789 additions and 328 deletions
@@ -40,6 +40,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/seatdef"
"git.novox.be/novox/mesh-sdk/go/testbus"
"git.novox.be/novox/mesh-sdk/go/views/viewstest"
"mesh-delivery-builder/internal/build"
@@ -578,7 +579,10 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
first := "src2:" + strings.Repeat("1", 64)
b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, commit, first, 1)
digest := b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, commit, fp, 2)
b.start(t, "n1", map[string]string{"MESH_BUILDER_STEP_BOUND": "1s"})
// H-A-473 (addendum): the builds that must end done run under the level bound's set-up share, never a bound a loaded
// node cannot meet; the hung step's part below starts the builder again at a short bound of its own.
doneBound := map[string]string{"MESH_BUILDER_STEP_BOUND": testbus.SetupBound(t).String()}
b.start(t, "n1", doneBound)
v := b.await(t, "netcheck", isState(builds.BuildDone))
for _, a := range []string{"tools-go", "tools-typescript"} {
es := reg.of(t, "netcheck/"+a)
@@ -693,6 +697,8 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
}
// DLV-B10: a build module's bound running out fails the trunk build, which is the builder's finding build-failed; the
// fingerprint built again clears it with its done build, numbering nothing of the builder's own (H-A-159).
b.stop()
b.start(t, "n1", map[string]string{"MESH_BUILDER_STEP_BOUND": "1s"}) // a step that never ends runs out of any bound
release := make(chan struct{})
b.fake.hang.Store(&release)
third := "src2:" + strings.Repeat("4", 64)
@@ -713,6 +719,8 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
reg.awaitFinding(t, failed, true)
b.fake.hang.Store(nil)
close(release)
b.stop() // the build again must end done
b.start(t, "n1", doneBound)
b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, "89abcdef0123456789abcdef0123456789abcdef", third, 3) // the entry moves to a later commit, same fingerprint
b.awaitFP(t, "netcheck", third, isState(builds.BuildDone))
reg.awaitFinding(t, failed, false)
@@ -22,6 +22,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/testbus"
"git.novox.be/novox/mesh-sdk/go/views"
"mesh-delivery-builder/internal/versions"
@@ -31,7 +32,8 @@ import (
// A live publish enqueued behind 100 again items while the builder was away is sent first; a dropped item is never sent, and
// put again within the duplicates window is not stored; the move of the old state run twice puts each item once, and the
// builder removes the old state's bucket itself (H-A-461 (1)). The seat mesh-builds' verb publish-queue reads the stream: counts
// per holder and class, the oldest and newest, the live item listed first (H-A-454). Then §796's tag at another commit, its
// per holder and class, the oldest and newest, the live item listed first (H-A-454). An idle dispatch sends a live item put
// before an again one first, 50 rounds (H-A-473, third addendum). Then §796's tag at another commit, its
// reservation's version-reserved waited on by name (§821), and
// §794's backlog held six minutes, as steps (H-A-453 (3)).
func TestThePublishQueueSendsLiveFirstAndNeverADroppedItem(t *testing.T) {
@@ -43,281 +45,347 @@ func TestThePublishQueueSendsLiveFirstAndNeverADroppedItem(t *testing.T) {
Retention: jetstream.WorkQueuePolicy, Discard: jetstream.DiscardNew, Duplicates: 24 * time.Hour, MaxAge: 7 * 24 * time.Hour}); err != nil {
t.Fatal(err)
}
kv := &itemKV{vals: map[string][]byte{}, revs: map[string]uint64{}}
var mu sync.Mutex
var sent []string
published := map[string]bool{} // the modules whose publish is heard
sentCh := make(chan string, 200)
items := &publishItems{reasks: func() jetstream.KeyValue { return kv }, set: publishSettings{Pause: time.Second, PauseMax: time.Minute, Tries: 5},
now: time.Now, logf: t.Logf,
reserved: func(_ context.Context, m, v string) (versions.ReservedVersion, bool, error) {
mu.Lock()
defer mu.Unlock()
return versions.ReservedVersion{Module: m, Version: v, Artifacts: map[string]string{m + ":a": "sha256:a"}, Published: map[string]bool{m + ":a": published[m]}, BuiltAt: time.Now()}, true, nil
},
send: func(_ context.Context, q versions.QueuedPublish) versions.SendFate {
mu.Lock()
sent = append(sent, q.Module)
mu.Unlock()
sentCh <- q.Module
return versions.SendFate{Sent: true}
// An idle dispatch (H-A-473, its third addendum): its two pulls wait; a live item put, then an again one, is sent live first;
// and an again item put after a live one, while the dispatch runs an again item put before both, is sent after the live one.
// Every time, 50 rounds, each a holder of its own.
sentOf := make(chan string, 8)
idle := &publishQueue{js: b.js, ackWait: time.Minute, logf: func(string, ...any) {}, binding: publishesOf("mesh-delivery-builder"),
take: func(_ context.Context, m itemMsg) {
var q versions.QueuedPublish
_ = json.Unmarshal(m.Data(), &q)
_ = m.Ack()
sentOf <- q.Module
}}
pq := &publishQueue{js: b.js, ackWait: time.Minute, logf: t.Logf, take: items.take, binding: publishesOf("mesh-delivery-builder")}
item := func(module, class string) versions.QueuedPublish {
return versions.QueuedPublish{V: 1, Holder: "oci", Module: module, Version: "0.1.0", Artifacts: map[string]string{module + ":a": "sha256:a"}, Class: class, Queued: time.Now()}
}
away := func(q versions.QueuedPublish) { // put while the builder was away: no holder learnt, nothing pulled
data, _ := json.Marshal(q)
if _, err := b.js.Publish(ctx, pq.subject(bind, q), data, jetstream.WithMsgID(versions.PublishMsgID(q.Holder, q.Class, q.Module, q.Version))); err != nil {
t.Run("idle dispatch", func(t *testing.T) {
roundBound := testbus.SetupBound(t) // the level's set-up share (MESH_TEST_LEVEL_BOUND), as the other whole tests
for r := range 50 {
holder := fmt.Sprintf("idle%d", r)
idle.learn(ctx, holder)
for _, class := range []string{versions.PublishClassLive, versions.PublishClassStart} { // both pulls stand waiting
c, err := b.js.Consumer(ctx, bind.Stream, publishesConsumer(holder, class))
if err != nil {
t.Fatal(err)
}
for {
info, err := c.Info(ctx)
if err != nil {
t.Fatal(err)
}
if info.NumWaiting > 0 {
break
}
}
}
put := func(module, class string) {
q := versions.QueuedPublish{V: 1, Holder: holder, Module: module, Version: "0.1.0", Class: class, Queued: time.Now()}
data, _ := json.Marshal(q)
if _, err := b.js.Publish(ctx, idle.subject(bind, q), data); err != nil {
t.Fatal(err)
}
}
sent := func(n int) []string {
var order []string
for range n {
select {
case m := <-sentOf:
order = append(order, m)
case <-time.After(roundBound):
t.Fatalf("round %d: sent %v of %d items", r, order, n)
}
}
return order
}
put("live", versions.PublishClassLive)
put("again", versions.PublishClassStart)
if order := sent(2); order[0] != "live" {
t.Errorf("round %d: an idle dispatch sent %v, want the live item put first sent first", r, order)
}
put("before", versions.PublishClassStart)
put("live2", versions.PublishClassLive)
put("after", versions.PublishClassStart)
if order := sent(3); slices.Index(order, "after") < slices.Index(order, "live2") {
t.Errorf("round %d: the dispatch sent %v, want the again item put after the live one sent after it", r, order)
}
}
})
t.Run("the queue, the move, tags and backlog", func(t *testing.T) {
kv := &itemKV{vals: map[string][]byte{}, revs: map[string]uint64{}}
var mu sync.Mutex
var sent []string
published := map[string]bool{} // the modules whose publish is heard
sentCh := make(chan string, 200)
items := &publishItems{reasks: func() jetstream.KeyValue { return kv }, set: publishSettings{Pause: time.Second, PauseMax: time.Minute, Tries: 5},
now: time.Now, logf: t.Logf,
reserved: func(_ context.Context, m, v string) (versions.ReservedVersion, bool, error) {
mu.Lock()
defer mu.Unlock()
return versions.ReservedVersion{Module: m, Version: v, Artifacts: map[string]string{m + ":a": "sha256:a"}, Published: map[string]bool{m + ":a": published[m]}, BuiltAt: time.Now()}, true, nil
},
send: func(_ context.Context, q versions.QueuedPublish) versions.SendFate {
mu.Lock()
sent = append(sent, q.Module)
mu.Unlock()
sentCh <- q.Module
return versions.SendFate{Sent: true}
}}
pq := &publishQueue{js: b.js, ackWait: time.Minute, logf: t.Logf, take: items.take, binding: publishesOf("mesh-delivery-builder")}
item := func(module, class string) versions.QueuedPublish {
return versions.QueuedPublish{V: 1, Holder: "oci", Module: module, Version: "0.1.0", Artifacts: map[string]string{module + ":a": "sha256:a"}, Class: class, Queued: time.Now()}
}
away := func(q versions.QueuedPublish) { // put while the builder was away: no holder learnt, nothing pulled
data, _ := json.Marshal(q)
if _, err := b.js.Publish(ctx, pq.subject(bind, q), data, jetstream.WithMsgID(versions.PublishMsgID(q.Holder, q.Class, q.Module, q.Version))); err != nil {
t.Fatal(err)
}
}
for i := range 100 {
away(item(fmt.Sprintf("a%03d", i), versions.PublishClassStart))
}
away(item("dropme", versions.PublishClassStart))
away(item("live", versions.PublishClassLive))
if answer, err := pq.drop(ctx, items, "dropme", "0.1.0", "op@node", "a test drops it"); err != nil {
t.Fatalf("drop: %v", err)
} else {
t.Log(answer)
}
if dup, err := pq.put(ctx, item("dropme", versions.PublishClassStart)); err != nil || !dup {
t.Errorf("a dropped item put again within the window: duplicate %v (%v), want it not stored", dup, err)
}
raw, err := publishQueueVerb(ctx, pq)("op@node", json.RawMessage(`{}`))
if a, ok := raw.(meshbuilds.PublishQueueAnswer); err != nil || !ok || a.Total != 101 || len(a.Counts) != 2 || a.Counts[0].Class != versions.PublishClassStart ||
a.Counts[0].Count != 100 || a.Counts[1].Count != 1 || a.Counts[0].Oldest.After(a.Counts[0].Newest) || len(a.Queue) != versions.PublishQueueListCap ||
a.Queue[0].Module != "live" || a.Capped {
t.Errorf("publish-queue answered %+v (%v), want 101 items, oci's 100 again and 1 live, fifty listed, the live first", raw, err)
}
if err := pq.learnAll(ctx); err != nil {
t.Fatal(err)
}
}
for i := range 100 {
away(item(fmt.Sprintf("a%03d", i), versions.PublishClassStart))
}
away(item("dropme", versions.PublishClassStart))
away(item("live", versions.PublishClassLive))
if answer, err := pq.drop(ctx, items, "dropme", "0.1.0", "op@node", "a test drops it"); err != nil {
t.Fatalf("drop: %v", err)
} else {
t.Log(answer)
}
if dup, err := pq.put(ctx, item("dropme", versions.PublishClassStart)); err != nil || !dup {
t.Errorf("a dropped item put again within the window: duplicate %v (%v), want it not stored", dup, err)
}
raw, err := publishQueueVerb(ctx, pq)("op@node", json.RawMessage(`{}`))
if a, ok := raw.(meshbuilds.PublishQueueAnswer); err != nil || !ok || a.Total != 101 || len(a.Counts) != 2 || a.Counts[0].Class != versions.PublishClassStart ||
a.Counts[0].Count != 100 || a.Counts[1].Count != 1 || a.Counts[0].Oldest.After(a.Counts[0].Newest) || len(a.Queue) != versions.PublishQueueListCap ||
a.Queue[0].Module != "live" || a.Capped {
t.Errorf("publish-queue answered %+v (%v), want 101 items, oci's 100 again and 1 live, fifty listed, the live first", raw, err)
}
if err := pq.learnAll(ctx); err != nil {
t.Fatal(err)
}
select {
case first := <-sentCh:
if first != "live" {
t.Errorf("the first item sent is %s, want the live one, enqueued behind 100 again items", first)
}
case <-time.After(30 * time.Second):
t.Fatal("nothing was sent")
}
for range 4 { // the again items in flight: at most four held unacknowledged, their publish unheard
select {
case <-sentCh:
case first := <-sentCh:
if first != "live" {
t.Errorf("the first item sent is %s, want the live one, enqueued behind 100 again items", first)
}
case <-time.After(30 * time.Second):
t.Fatal("the again items were not sent")
t.Fatal("nothing was sent")
}
for range 4 { // the again items in flight: at most four held unacknowledged, their publish unheard
select {
case <-sentCh:
case <-time.After(30 * time.Second):
t.Fatal("the again items were not sent")
}
}
deadline := time.After(60 * time.Second)
for { // each publish sent is heard: its item acknowledged, the next sent
mu.Lock()
n, done := len(sent), slices.Clone(sent)
for _, m := range done {
published[m] = true
}
mu.Unlock()
if n >= 101 {
break
}
for _, m := range done {
items.heard(ctx, m, "0.1.0")
}
select {
case <-sentCh:
case <-deadline:
t.Fatalf("sent %d of 101", n)
}
}
}
deadline := time.After(60 * time.Second)
for { // each publish sent is heard: its item acknowledged, the next sent
mu.Lock()
n, done := len(sent), slices.Clone(sent)
for _, m := range done {
published[m] = true
if slices.Contains(sent, "dropme") || len(sent) != 101 {
t.Errorf("sent %d items (dropme among them: %v), want the 101 and never the dropped one", len(sent), slices.Contains(sent, "dropme"))
}
mu.Unlock()
if n >= 101 {
break
// The move of the old state, run twice, puts each item once (at another holder, so the pull of oci is not asked).
old, err := b.js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: builds.StateBucket("mesh-delivery-builder", versions.PublishQueueState)})
if err != nil {
t.Fatal(err)
}
for _, m := range done {
items.heard(ctx, m, "0.1.0")
var tagsWritten []string
call := func(tool string, body any) (json.RawMessage, error) {
switch tool {
case "seat:" + gitseat.Seat + "." + gitseat.VerbTags: // m4's 0.2.0 is tagged at another commit (§796)
return json.Marshal(gitseat.TagsAnswer{Form: 2, Tags: []gitseat.Tag{{Tag: "modules/m4/v0.2.0", Commit: "c-other"}}})
case "seat:" + gitseat.Seat + ".tag":
tagsWritten = append(tagsWritten, body.(gitseat.TagInput).Tag+"@"+body.(gitseat.TagInput).Commit)
return json.Marshal(gitseat.TagAnswer{Performed: true})
}
return json.Marshal(meshcatalog.VersionsAnswer{Module: "m"})
}
found := &memFindings{vals: map[string][]byte{}}
rv := newReservations(&memReserve{vals: map[string][]byte{}, revs: map[string]uint64{}}, call, func() jetstream.KeyValue { return &boundsKV{keys: map[string]bool{}} },
&findings{module: builds.BuilderModule, w: found, logf: t.Logf, now: time.Now}, nil, t.Logf)
rv.filled.Store(true)
type oldItem struct {
key string
raw []byte
}
var olds []oldItem
for _, m := range []string{"m1", "m2", "m3"} {
v, err := rv.Reserve(ctx, m, "0.1", []string{m + ":a"}, versions.Source{Fingerprint: "src2:" + m})
if err != nil {
t.Fatal(err)
}
if err := rv.Built(ctx, m, v.Version, map[string]string{m + ":a": "sha256:a"}, nil); err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(versions.QueuedPublish{V: 1, Holder: "go", Module: m, Version: v.Version, Artifacts: map[string]string{m + ":a": "sha256:a"}, Class: "asked-again-at-start"})
if _, err := old.Put(ctx, versions.QueueKey("go", m, v.Version), raw); err != nil {
t.Fatal(err)
}
olds = append(olds, oldItem{versions.QueueKey("go", m, v.Version), raw})
}
rp := &versions.Republisher{Reserved: rv.r, Put: func(ctx context.Context, q versions.QueuedPublish) (bool, error) {
data, _ := json.Marshal(q)
ack, err := b.js.Publish(ctx, pq.subject(bind, q), data, jetstream.WithMsgID(versions.PublishMsgID(q.Holder, q.Class, q.Module, q.Version)))
if err != nil {
return false, err
}
return ack.Duplicate, nil
}, Started: time.Now(), Logf: t.Logf}
oldBucket := builds.StateBucket("mesh-delivery-builder", versions.PublishQueueState)
moveOld(ctx, b.js, oldBucket, rp, t.Logf)
if _, err := b.js.KeyValue(ctx, oldBucket); !errors.Is(err, jetstream.ErrBucketNotFound) {
t.Errorf("after the move the old state's bucket answers %v, want it removed by the builder (H-A-461 (1))", err)
}
// A start that stopped before the removal moves the same items again: the duplicates window keeps each once.
again, err := b.js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: oldBucket})
if err != nil {
t.Fatal(err)
}
for _, e := range olds {
if _, err := again.Put(ctx, e.key, e.raw); err != nil {
t.Fatal(err)
}
}
moveOld(ctx, b.js, oldBucket, rp, t.Logf)
moveOld(ctx, b.js, oldBucket, rp, t.Logf) // no bucket left: nothing moves
if _, err := b.js.KeyValue(ctx, oldBucket); !errors.Is(err, jetstream.ErrBucketNotFound) {
t.Errorf("after the second move the old state's bucket answers %v, want it removed", err)
}
st, err := b.js.Stream(ctx, bind.Stream)
if err != nil {
t.Fatal(err)
}
info, err := st.Info(ctx, jetstream.WithSubjectFilter(bind.Subject+".publish.go.>"))
if err != nil {
t.Fatal(err)
}
if len(info.State.Subjects) != 3 || info.State.Subjects[bind.Subject+".publish.go.again.m1.0=2E1=2E0"] != 1 {
t.Errorf("the move run twice left %v on the work queue: want each of the three once, as again", info.State.Subjects)
}
// Design 47 H2 §796 (whole, H-A-453 (3)): a module's tags are read before it reserves; a number whose tag stands at another
// commit is skipped, the next number taken and version-tagged-elsewhere raised; the release tags <directory>/v<version> at
// the build's commit, and the module's version-released clears the finding.
rv.r.Names = func(context.Context, string, string) (string, bool, error) {
return "git.example/o/r/modules/m4", true, nil
}
// Design 47 H2 §821, §805: the reservation is said as the seat mesh-builds' version-reserved, said as the process says it
// (sayReservedOn), and waited on by the seat and the event's name.
rv.names, rv.sayReserved = rv.r.Names, sayReservedOn(b.js, process.Account{Module: builds.BuilderModule, Node: "n1"})
said, err := nats.Connect(b.srvURL, nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
defer said.Close()
reservedCh := make(chan *nats.Msg, 4)
rs, err := said.ChanSubscribe(meshbuilds.SubjectVersionReserved, reservedCh)
if err != nil {
t.Fatal(err)
}
defer rs.Unsubscribe()
if err := said.Flush(); err != nil {
t.Fatal(err)
}
v4, err := rv.Reserve(ctx, "m4", "0.2", []string{"m4:lib"}, versions.Source{Fingerprint: "src2:m4", GoPackage: true, GoArtifact: "lib",
Repository: "o/r", Commit: "c-mine", Branch: "main", CatalogRevision: 1})
elsewhere := views.FindingKey(meshconditionsclient.ScopeModule, "m4", versions.VersionTaggedElsewhere)
if _, raised := found.vals[elsewhere]; err != nil || v4.Version != "0.2.1" || v4.Tag != "modules/m4/v0.2.1" || !raised {
t.Fatalf("m4 reserved %+v (%v), %s raised %v: want 0.2.1 past 0.2.0's tag at another commit, its tag modules/m4/v0.2.1, and the finding", v4, err, elsewhere, raised)
}
select {
case <-sentCh:
case <-deadline:
t.Fatalf("sent %d of 101", n)
case m := <-reservedCh:
ev, _, err := meshbuilds.DecodeVersionReservedEvent(m.Data)
if err != nil || ev.Module != "m4" || ev.Artifact != "m4:lib" || ev.Path != "git.example/o/r/modules/m4" || ev.Version != "0.2.1" ||
ev.Fingerprint != "src2:m4" || ev.Revision != "main.1" || m.Header.Get("x-source") != builds.BuilderModule {
t.Errorf("version-reserved = %+v (%v) from %q, want m4:lib at 0.2.1, its path, fingerprint src2:m4 and revision main.1, said by the builder", ev, err, m.Header.Get("x-source"))
}
case <-time.After(10 * time.Second):
t.Fatal("no version-reserved of m4's reservation within 10 s on " + meshbuilds.SubjectVersionReserved)
}
catalogEvents, err := b.js.StreamNameBySubject(ctx, meshcatalog.SubjectVersionReleased) // the bed's stream of the catalog's events
if err != nil {
t.Fatal(err)
}
evs, err := b.js.CreateOrUpdateConsumer(ctx, catalogEvents, jetstream.ConsumerConfig{Durable: "builder-events", AckPolicy: jetstream.AckExplicitPolicy,
FilterSubject: meshcatalog.SubjectVersionReleased, DeliverPolicy: jetstream.DeliverNewPolicy})
if err != nil {
t.Fatal(err)
}
next := func(module, version string) jetstream.Msg {
t.Helper()
data, _ := json.Marshal(meshcatalog.VersionReleasedEvent{Module: module, Version: version})
if _, err := b.js.Publish(ctx, meshcatalog.SubjectVersionReleased, data); err != nil {
t.Fatal(err)
}
m, err := evs.Next(jetstream.FetchMaxWait(10 * time.Second))
if err != nil {
t.Fatal(err)
}
return m
}
if !rv.heard(ctx, next("m4", "0.2.1")) {
t.Fatal("the reservations did not take version-released")
}
if _, raised := found.vals[elsewhere]; raised || !slices.Equal(tagsWritten, []string{"modules/m4/v0.2.1@c-mine"}) {
t.Errorf("after version-released %s raised %v, tags written %v: want it cleared and modules/m4/v0.2.1 written at c-mine, no tag moved", elsewhere, raised, tagsWritten)
}
}
mu.Lock()
if slices.Contains(sent, "dropme") || len(sent) != 101 {
t.Errorf("sent %d items (dropme among them: %v), want the 101 and never the dropped one", len(sent), slices.Contains(sent, "dropme"))
}
mu.Unlock()
// The move of the old state, run twice, puts each item once (at another holder, so the pull of oci is not asked).
old, err := b.js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: builds.StateBucket("mesh-delivery-builder", versions.PublishQueueState)})
if err != nil {
t.Fatal(err)
}
var tagsWritten []string
call := func(tool string, body any) (json.RawMessage, error) {
switch tool {
case "seat:" + gitseat.Seat + "." + gitseat.VerbTags: // m4's 0.2.0 is tagged at another commit (§796)
return json.Marshal(gitseat.TagsAnswer{Form: 2, Tags: []gitseat.Tag{{Tag: "modules/m4/v0.2.0", Commit: "c-other"}}})
case "seat:" + gitseat.Seat + ".tag":
tagsWritten = append(tagsWritten, body.(gitseat.TagInput).Tag+"@"+body.(gitseat.TagInput).Commit)
return json.Marshal(gitseat.TagAnswer{Performed: true})
}
return json.Marshal(meshcatalog.VersionsAnswer{Module: "m"})
}
found := &memFindings{vals: map[string][]byte{}}
rv := newReservations(&memReserve{vals: map[string][]byte{}, revs: map[string]uint64{}}, call, func() jetstream.KeyValue { return &boundsKV{keys: map[string]bool{}} },
&findings{module: builds.BuilderModule, w: found, logf: t.Logf, now: time.Now}, nil, t.Logf)
rv.filled.Store(true)
type oldItem struct {
key string
raw []byte
}
var olds []oldItem
for _, m := range []string{"m1", "m2", "m3"} {
v, err := rv.Reserve(ctx, m, "0.1", []string{m + ":a"}, versions.Source{Fingerprint: "src2:" + m})
// Design 47 H2 §794 (whole, H-A-453 (3), H-A-462): a message the builder's handler leaves unacknowledged is kept on its
// consumer's ledger in the state consumers on the bus; held six minutes, the health names that backlog, and within five
// minutes it does not.
consumers, err := b.js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: builds.StateBucket(builds.BuilderModule, process.ConsumersState)})
if err != nil {
t.Fatal(err)
}
if err := rv.Built(ctx, m, v.Version, map[string]string{m + ":a": "sha256:a"}, nil); err != nil {
t.Fatal(err)
held := time.Now()
hub := &oneHandler{}
aged := agedHub{hub: hub, ledgers: &ledgers{logf: t.Logf, now: func() time.Time { return held },
sink: consumersSink(process.AwaitBucket(ctx, b.js, nil, builds.StateBucket(builds.BuilderModule, process.ConsumersState), "", t.Logf))}}
_ = aged.Handle(meshcatalog.SubjectVersionReleased, func(m jetstream.Msg) { _ = m.Nak() }) // a handler standing still: asked again
hub.handle(next("m5", "0.1.0"))
state := versions.Reader{Keys: func() ([]string, error) {
l, err := consumers.ListKeys(ctx)
if err != nil {
return nil, err
}
var keys []string
for k := range l.Keys() {
keys = append(keys, k)
}
return keys, nil
}, Get: func(k string) (json.RawMessage, bool, error) {
e, err := consumers.Get(ctx, k)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
return e.Value(), true, nil
}}
if a := versions.BuilderHealth(state, held.Add(6*time.Minute), 5*time.Minute); a.Healthy || !strings.Contains(a.Why, "builder-events (its oldest unacknowledged message waits 6m0s") {
t.Errorf("the health with a backlog held six minutes = %+v, want unhealthy naming builder-events", a)
}
raw, _ := json.Marshal(versions.QueuedPublish{V: 1, Holder: "go", Module: m, Version: v.Version, Artifacts: map[string]string{m + ":a": "sha256:a"}, Class: "asked-again-at-start"})
if _, err := old.Put(ctx, versions.QueueKey("go", m, v.Version), raw); err != nil {
t.Fatal(err)
if a := versions.BuilderHealth(state, held.Add(4*time.Minute), 5*time.Minute); !a.Healthy {
t.Errorf("the health four minutes in = %+v, want healthy", a)
}
olds = append(olds, oldItem{versions.QueueKey("go", m, v.Version), raw})
}
rp := &versions.Republisher{Reserved: rv.r, Put: func(ctx context.Context, q versions.QueuedPublish) (bool, error) {
data, _ := json.Marshal(q)
ack, err := b.js.Publish(ctx, pq.subject(bind, q), data, jetstream.WithMsgID(versions.PublishMsgID(q.Holder, q.Class, q.Module, q.Version)))
if err != nil {
return false, err
}
return ack.Duplicate, nil
}, Started: time.Now(), Logf: t.Logf}
oldBucket := builds.StateBucket("mesh-delivery-builder", versions.PublishQueueState)
moveOld(ctx, b.js, oldBucket, rp, t.Logf)
if _, err := b.js.KeyValue(ctx, oldBucket); !errors.Is(err, jetstream.ErrBucketNotFound) {
t.Errorf("after the move the old state's bucket answers %v, want it removed by the builder (H-A-461 (1))", err)
}
// A start that stopped before the removal moves the same items again: the duplicates window keeps each once.
again, err := b.js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: oldBucket})
if err != nil {
t.Fatal(err)
}
for _, e := range olds {
if _, err := again.Put(ctx, e.key, e.raw); err != nil {
t.Fatal(err)
}
}
moveOld(ctx, b.js, oldBucket, rp, t.Logf)
moveOld(ctx, b.js, oldBucket, rp, t.Logf) // no bucket left: nothing moves
if _, err := b.js.KeyValue(ctx, oldBucket); !errors.Is(err, jetstream.ErrBucketNotFound) {
t.Errorf("after the second move the old state's bucket answers %v, want it removed", err)
}
st, err := b.js.Stream(ctx, bind.Stream)
if err != nil {
t.Fatal(err)
}
info, err := st.Info(ctx, jetstream.WithSubjectFilter(bind.Subject+".publish.go.>"))
if err != nil {
t.Fatal(err)
}
if len(info.State.Subjects) != 3 || info.State.Subjects[bind.Subject+".publish.go.again.m1.0=2E1=2E0"] != 1 {
t.Errorf("the move run twice left %v on the work queue: want each of the three once, as again", info.State.Subjects)
}
// Design 47 H2 §796 (whole, H-A-453 (3)): a module's tags are read before it reserves; a number whose tag stands at another
// commit is skipped, the next number taken and version-tagged-elsewhere raised; the release tags <directory>/v<version> at
// the build's commit, and the module's version-released clears the finding.
rv.r.Names = func(context.Context, string, string) (string, bool, error) {
return "git.example/o/r/modules/m4", true, nil
}
// Design 47 H2 §821, §805: the reservation is said as the seat mesh-builds' version-reserved, said as the process says it
// (sayReservedOn), and waited on by the seat and the event's name.
rv.names, rv.sayReserved = rv.r.Names, sayReservedOn(b.js, process.Account{Module: builds.BuilderModule, Node: "n1"})
said, err := nats.Connect(b.srvURL, nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
defer said.Close()
reservedCh := make(chan *nats.Msg, 4)
rs, err := said.ChanSubscribe(meshbuilds.SubjectVersionReserved, reservedCh)
if err != nil {
t.Fatal(err)
}
defer rs.Unsubscribe()
if err := said.Flush(); err != nil {
t.Fatal(err)
}
v4, err := rv.Reserve(ctx, "m4", "0.2", []string{"m4:lib"}, versions.Source{Fingerprint: "src2:m4", GoPackage: true, GoArtifact: "lib",
Repository: "o/r", Commit: "c-mine", Branch: "main", CatalogRevision: 1})
elsewhere := views.FindingKey(meshconditionsclient.ScopeModule, "m4", versions.VersionTaggedElsewhere)
if _, raised := found.vals[elsewhere]; err != nil || v4.Version != "0.2.1" || v4.Tag != "modules/m4/v0.2.1" || !raised {
t.Fatalf("m4 reserved %+v (%v), %s raised %v: want 0.2.1 past 0.2.0's tag at another commit, its tag modules/m4/v0.2.1, and the finding", v4, err, elsewhere, raised)
}
select {
case m := <-reservedCh:
ev, _, err := meshbuilds.DecodeVersionReservedEvent(m.Data)
if err != nil || ev.Module != "m4" || ev.Artifact != "m4:lib" || ev.Path != "git.example/o/r/modules/m4" || ev.Version != "0.2.1" ||
ev.Fingerprint != "src2:m4" || ev.Revision != "main.1" || m.Header.Get("x-source") != builds.BuilderModule {
t.Errorf("version-reserved = %+v (%v) from %q, want m4:lib at 0.2.1, its path, fingerprint src2:m4 and revision main.1, said by the builder", ev, err, m.Header.Get("x-source"))
}
case <-time.After(10 * time.Second):
t.Fatal("no version-reserved of m4's reservation within 10 s on " + meshbuilds.SubjectVersionReserved)
}
catalogEvents, err := b.js.StreamNameBySubject(ctx, meshcatalog.SubjectVersionReleased) // the bed's stream of the catalog's events
if err != nil {
t.Fatal(err)
}
evs, err := b.js.CreateOrUpdateConsumer(ctx, catalogEvents, jetstream.ConsumerConfig{Durable: "builder-events", AckPolicy: jetstream.AckExplicitPolicy,
FilterSubject: meshcatalog.SubjectVersionReleased, DeliverPolicy: jetstream.DeliverNewPolicy})
if err != nil {
t.Fatal(err)
}
next := func(module, version string) jetstream.Msg {
t.Helper()
data, _ := json.Marshal(meshcatalog.VersionReleasedEvent{Module: module, Version: version})
if _, err := b.js.Publish(ctx, meshcatalog.SubjectVersionReleased, data); err != nil {
t.Fatal(err)
}
m, err := evs.Next(jetstream.FetchMaxWait(10 * time.Second))
if err != nil {
t.Fatal(err)
}
return m
}
if !rv.heard(ctx, next("m4", "0.2.1")) {
t.Fatal("the reservations did not take version-released")
}
if _, raised := found.vals[elsewhere]; raised || !slices.Equal(tagsWritten, []string{"modules/m4/v0.2.1@c-mine"}) {
t.Errorf("after version-released %s raised %v, tags written %v: want it cleared and modules/m4/v0.2.1 written at c-mine, no tag moved", elsewhere, raised, tagsWritten)
}
// Design 47 H2 §794 (whole, H-A-453 (3), H-A-462): a message the builder's handler leaves unacknowledged is kept on its
// consumer's ledger in the state consumers on the bus; held six minutes, the health names that backlog, and within five
// minutes it does not.
consumers, err := b.js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: builds.StateBucket(builds.BuilderModule, process.ConsumersState)})
if err != nil {
t.Fatal(err)
}
held := time.Now()
hub := &oneHandler{}
aged := agedHub{hub: hub, ledgers: &ledgers{logf: t.Logf, now: func() time.Time { return held },
sink: consumersSink(process.AwaitBucket(ctx, b.js, nil, builds.StateBucket(builds.BuilderModule, process.ConsumersState), "", t.Logf))}}
_ = aged.Handle(meshcatalog.SubjectVersionReleased, func(m jetstream.Msg) { _ = m.Nak() }) // a handler standing still: asked again
hub.handle(next("m5", "0.1.0"))
state := versions.Reader{Keys: func() ([]string, error) {
l, err := consumers.ListKeys(ctx)
if err != nil {
return nil, err
}
var keys []string
for k := range l.Keys() {
keys = append(keys, k)
}
return keys, nil
}, Get: func(k string) (json.RawMessage, bool, error) {
e, err := consumers.Get(ctx, k)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
return e.Value(), true, nil
}}
if a := versions.BuilderHealth(state, held.Add(6*time.Minute), 5*time.Minute); a.Healthy || !strings.Contains(a.Why, "builder-events (its oldest unacknowledged message waits 6m0s") {
t.Errorf("the health with a backlog held six minutes = %+v, want unhealthy naming builder-events", a)
}
if a := versions.BuilderHealth(state, held.Add(4*time.Minute), 5*time.Minute); !a.Healthy {
t.Errorf("the health four minutes in = %+v, want healthy", a)
}
})
}
@@ -637,44 +637,35 @@ func (pq *publishQueue) learn(ctx context.Context, holder string) {
}
// pull makes holder's two consumers and dispatches their items, the live class first, at most four sends at once: an again item
// waits while a live one waits on the queue.
// is run only when no live one waits on the queue or is in flight to the dispatch (dispatch, H-A-473).
func (pq *publishQueue) pull(ctx context.Context, holder string) error {
b, err := pq.binding()
if err != nil {
return err
}
chans := map[string]chan jetstream.Msg{}
var live jetstream.Consumer
for _, class := range []string{versions.PublishClassStart, versions.PublishClassLive} { // the again first: the live is first by the dispatch, not by luck
cons := map[string]jetstream.Consumer{}
for _, class := range []string{versions.PublishClassStart, versions.PublishClassLive} {
c, err := pq.js.CreateOrUpdateConsumer(ctx, b.Stream, jetstream.ConsumerConfig{Durable: publishesConsumer(holder, class),
FilterSubject: b.Subject + ".publish." + versions.EscapeToken(holder) + "." + class + ".>", AckPolicy: jetstream.AckExplicitPolicy,
AckWait: pq.ackWait, MaxAckPending: publishesMaxAckPending, MaxDeliver: publishesMaxDeliver})
if err != nil {
return fmt.Errorf("the consumer %s: %w", publishesConsumer(holder, class), err)
}
ch := make(chan jetstream.Msg, publishesMaxAckPending)
cc, err := c.Consume(func(m jetstream.Msg) { ch <- m }, jetstream.PullMaxMessages(publishesMaxAckPending))
if err != nil {
return fmt.Errorf("the consumer %s is not pulled: %w", publishesConsumer(holder, class), err)
}
go func() { <-ctx.Done(); cc.Stop() }()
chans[class] = ch
if class == versions.PublishClassLive {
live = c
}
}
liveWaits := func() bool {
info, err := live.Info(ctx)
return err == nil && info.NumPending > 0
cons[class] = c
}
pq.logf("pulling the publishes of %s, the live first", holder)
go pq.dispatch(ctx, chans[versions.PublishClassLive], chans[versions.PublishClassStart], liveWaits)
go pq.dispatch(ctx, holder, cons[versions.PublishClassLive], cons[versions.PublishClassStart])
return nil
}
// dispatch runs the items of one holder, at most four at once, every live item before an again one: an again item pulled while
// a live one waits on the queue (liveWaits) runs after it.
func (pq *publishQueue) dispatch(ctx context.Context, live, again <-chan jetstream.Msg, liveWaits func() bool) {
// dispatch runs the items of one holder, at most four at once, every live item before an again one (design 47 H2 §793,
// H-A-473 and its third addendum): on every turn no live item is held it fetches live without waiting, up to its free slots,
// and takes again items only when that fetch returned nothing. With again items held it fetches live once more; only when
// that returns nothing are they run, and while the live pull waits only once the live consumer's delivered sequence is no
// further than the live items received: a gap is a live item in flight to the waiting pull, and the again items wait until it
// arrives and runs. When neither holds an item, a pull of one item waits on each, the bus's only wake for an item put or
// delivered again. A pull the bus refuses ends the dispatch, said; the holder is pulled again when an item of it is put.
func (pq *publishQueue) dispatch(ctx context.Context, holder string, live, again jetstream.Consumer) {
sem := make(chan struct{}, versions.PublishSendsAtOnce)
run := func(m jetstream.Msg) {
go func() {
@@ -682,48 +673,186 @@ func (pq *publishQueue) dispatch(ctx context.Context, live, again <-chan jetstre
pq.take(ctx, m)
}()
}
var spare jetstream.Msg // an again item pulled while a live one waited
failed := func(err error) {
pq.logf("the publishes of %s are not pulled: %v; they are pulled when an item of it is put", holder, err)
pq.mu.Lock()
delete(pq.pulling, holder)
pq.mu.Unlock()
}
refused := func(err error) bool {
return err != nil && !errors.Is(err, jetstream.ErrNoMessages) && !errors.Is(err, nats.ErrTimeout) && ctx.Err() == nil
}
// received is the live consumer's highest delivery sequence the dispatch holds; it starts at what the consumer delivered
// before this dispatch, to pulls no longer there.
var received uint64
if info, err := live.Info(ctx); err != nil {
if refused(err) {
failed(err)
return
}
} else {
received = info.Delivered.Consumer
}
gotLive := func(ms ...jetstream.Msg) {
for _, m := range ms {
if md, err := m.Metadata(); err == nil && md.Sequence.Consumer > received {
received = md.Sequence.Consumer
}
}
}
inFlight := func() bool { // a live item delivered to the waiting pull and not yet received
info, err := live.Info(ctx)
if err != nil {
pq.logf("the publishes of %s: the live consumer cannot be read (%v); the again items wait for the live pull's answer", holder, err)
return true
}
return info.Delivered.Consumer > received
}
now := func(c jetstream.Consumer, n int) ([]jetstream.Msg, error) {
b, err := c.FetchNoWait(n)
if err != nil {
return nil, err
}
var got []jetstream.Msg
for m := range b.Messages() {
got = append(got, m)
}
return got, b.Error()
}
type waited struct {
m jetstream.Msg
err error
}
liveDone, againDone := make(chan waited, 1), make(chan waited, 1)
wait := func(c jetstream.Consumer, done chan<- waited) {
go func() {
var w waited
b, err := c.Fetch(1)
if err == nil {
for m := range b.Messages() {
w.m = m
}
err = b.Error()
}
w.err = err
done <- w
}()
}
var heldLive jetstream.Msg // a live item the waiting pull received
var heldAgain []jetstream.Msg // again items received, not yet run
var liveWaiting, againWaiting bool
heard := func(w waited, waiting *bool, keep func(jetstream.Msg)) bool {
*waiting = false
if refused(w.err) {
failed(w.err)
return false
}
if w.m != nil {
keep(w.m)
}
return true
}
keepLive := func(m jetstream.Msg) { gotLive(m); heldLive = m }
keepAgain := func(m jetstream.Msg) { heldAgain = append(heldAgain, m) }
for {
select {
case sem <- struct{}{}:
case <-ctx.Done():
return
}
select {
case m := <-live:
run(m)
continue
default:
}
m := spare
spare = nil
if m == nil {
n := 1
slots:
for n < cap(sem) {
select {
case lm := <-live:
run(lm)
continue
case m = <-again:
case <-ctx.Done():
return
case sem <- struct{}{}:
n++
default:
break slots
}
}
if liveWaits() {
spare = m
select {
case lm := <-live:
run(lm)
case <-ctx.Done():
select {
case w := <-liveDone:
if !heard(w, &liveWaiting, keepLive) {
return
}
continue
default:
}
select {
case lm := <-live: // came while the queue was asked
spare = m
run(lm)
case w := <-againDone:
if !heard(w, &againWaiting, keepAgain) {
return
}
default:
}
var got []jetstream.Msg
held := false // again items held for a live item in flight
if heldLive != nil {
got, heldLive = []jetstream.Msg{heldLive}, nil
} else {
ms, err := now(live, n)
if refused(err) {
failed(err)
return
}
gotLive(ms...)
got = ms
}
if len(got) == 0 {
if len(heldAgain) == 0 && !againWaiting {
ms, err := now(again, n)
if refused(err) {
failed(err)
return
}
heldAgain = ms
}
if len(heldAgain) > 0 { // with the again items held, live once more: one put since the first fetch goes first
ms, err := now(live, n)
if refused(err) {
failed(err)
return
}
gotLive(ms...)
got = ms
}
switch {
case len(got) > 0 || len(heldAgain) == 0:
case liveWaiting && inFlight():
held = true
default:
k := min(n, len(heldAgain))
got, heldAgain = heldAgain[:k], heldAgain[k:]
}
}
for _, m := range got {
run(m)
}
for range n - len(got) {
<-sem
}
if len(got) > 0 {
continue
}
if !liveWaiting {
liveWaiting = true
wait(live, liveDone)
}
if !againWaiting && !held {
againWaiting = true
wait(again, againDone)
}
select {
case w := <-liveDone:
if !heard(w, &liveWaiting, keepLive) {
return
}
case w := <-againDone:
if !heard(w, &againWaiting, keepAgain) {
return
}
case <-ctx.Done():
return
}
}
}
@@ -27,9 +27,10 @@ func TestTheJudge(t *testing.T) {
want: map[string]string{"a": `secret-`}},
{name: "an unknown field", raws: map[string]string{"a": `{"module":"a","version":"1.0","a-field-from-later":1}`},
want: map[string]string{"a": `"a-field-from-later"`}},
// H-A-465: the take-in sets a top-level word it does not know aside (ForInputs); the judgement still refuses it.
{name: "a word a newer SDK adds", raws: map[string]string{"a": `{"module":"a","version":"1.0","class":"core"}`},
want: map[string]string{"a": `"class"`}},
// H-A-465: the take-in sets a top-level word it does not know aside (ForInputs); the judgement still refuses it. The
// word is one no SDK knows: class, the word of H-A-465's night, is known since H2.
{name: "a word a newer SDK adds", raws: map[string]string{"a": `{"module":"a","version":"1.0","zzz-unknown-word":"core"}`},
want: map[string]string{"a": `"zzz-unknown-word"`}},
// Design 47 G §23, §28: a step's status is kept in the registration and never in a manifest.
{name: "a manifest naming a step's status", raws: map[string]string{"a": `{"module":"a","version":"1.0","policy":{"lint":"bypass"}}`},
want: map[string]string{"a": `"policy"`}},
@@ -110,7 +110,10 @@ func TestTheGrantsPartWhole(t *testing.T) {
}
manifest("nats", `{"module":"nats","version":"1","claims":[{"name":"mesh-broker"}],"own-secrets":{"broker":"/var/lib/nats/broker"},`+
`"emits":["created","done","dropped","recreated"],"streams":[{"name":"broker-events","events":["created","dropped","recreated"],"max-age-days":30,"max-mib":256}]}`)
manifest("mod-c", `{"module":"mod-c","version":"1","own-secrets":{"broker":"/var/lib/c/broker"},"consumes":["nats.done"]}`)
// mod-c names a program of its own, so its program's consumer a_mod-c is composed beside the tool runner's twin (design 47
// H2, H-A-472 second addendum).
manifest("mod-c", `{"module":"mod-c","version":"1","own-secrets":{"broker":"/var/lib/c/broker"},"consumes":["nats.done"],`+
`"resources":[{"id":"program","type":"process","run":["./mod-c"]}]}`)
// The part itself, placed as on the mesh, so its own consumer on EVENTS hears the seat mesh-broker's events (DLV-G1).
own, err := os.ReadFile("../../module.json")
if err != nil {
@@ -188,6 +188,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err
}
f.Part.RevisionsPerBranch, f.Part.Followed = perBranch, branches.Followed
// The setting twin-current-hours, read at the start: the bound of a current twin for the orphaned durables (design 47 H2,
// H-A-472 third addendum).
if f.Part.TwinCurrentHours, err = grants.TwinCurrentHoursSetting(strings.TrimSpace(getenv(grants.EnvTwinCurrentHours))); err != nil {
return err
}
f.Watch = fv
f.Part.ReadingRunning(nodeengine.StreamRunning)
// The catalog-revision event names each branch's newest revision, so grants-for asks the seat mesh-catalog's revision only
+74 -7
View File
@@ -547,6 +547,10 @@ func (b *fakeBus) setAssignments(pl meshmoduleassigner.Assignments) {
maps.Copy(b.states[grants.ReadAssignments], values)
}
// programResource names a program of the module's own, a process resource with no bus of its own: its consumer
// <node>_<module> is composed beside the tool runner's twin (design 47 H2, H-A-472 second addendum).
const programResource = `,"resources":[{"id":"program","type":"process","run":["./program"]}]`
const (
natsManifest = `{"module":"nats","version":"1","claims":[{"name":"mesh-broker"}],"own-secrets":{"broker":"/var/lib/nats/broker"}}`
xManifest = `{"module":"mod-x","version":"1","own-secrets":{"broker":"/var/lib/x/broker"},"state":[{"name":"kept","history":3}],` +
@@ -973,7 +977,8 @@ func TestGrants(t *testing.T) {
if again := p.Compose(); again.Withheld == nil || len(again.Withheld.Written) != 0 || again.Withheld.Withheld != 1 {
t.Errorf("a second pass wrote %+v, want nothing (on change only)", again.Withheld)
}
b.manifestOf("reader", `{"module":"reader","version":"1","runs-as":"reader","resources":[{"id":"account","type":"user","name":"reader","shell":"/usr/bin/nologin","home":"/var/lib/reader"}],`+
b.manifestOf("reader", `{"module":"reader","version":"1","runs-as":"reader","resources":[{"id":"account","type":"user","name":"reader","shell":"/usr/bin/nologin","home":"/var/lib/reader"},`+
`{"id":"program","type":"process","run":["./reader"]}],`+
`"own-secrets":{"broker":"/var/lib/r/broker"},"secrets-owner":"reader","consumes":["mesh-mail.mail.box.message.added"]}`)
if heard := p.Compose(); heard.Withheld == nil || !slices.Equal(heard.Withheld.Deleted, []string{key}) || b.own[grants.StateWithheld][key] != nil {
t.Errorf("once reader runs as its own account the pass did %+v (errors %v), want %s deleted", heard.Withheld, heard.Errors, key)
@@ -2244,7 +2249,7 @@ func TestGrants(t *testing.T) {
// A seat's event the seat list says is kept in a stream of its own (design 47): its stream is made once, and the module that
// consumes it has its durable on that stream, none on the shared one.
kb := fixture()
kb.manifestOf("mod-y", `{"module":"mod-y","version":"1","own-secrets":{"broker":"/var/lib/y/broker"},"consumes":["mesh-conditions.condition-raised"]}`)
kb.manifestOf("mod-y", `{"module":"mod-y","version":"1","own-secrets":{"broker":"/var/lib/y/broker"},"consumes":["mesh-conditions.condition-raised"]`+programResource+`}`)
kb.loaded = shadowOf(t, kb)
kp := partOn(kb)
kp.Create, kp.Consumers = kb, kb
@@ -2401,7 +2406,7 @@ func TestGrants(t *testing.T) {
})
t.Run("account consumers: a held consumer gains the filters of a new event, never loses one", func(t *testing.T) {
const cManifest = `{"module":"mod-c","version":"1","own-secrets":{"broker":"/var/lib/c/broker"},"consumes":["mod-y.done","mod-y.more"]}`
const cManifest = `{"module":"mod-c","version":"1","own-secrets":{"broker":"/var/lib/c/broker"},"consumes":["mod-y.done","mod-y.more"]` + programResource + `}`
b := fixture()
b.manifestOf("mod-c", cManifest)
b.assign("b", "mod-c")
@@ -3305,7 +3310,7 @@ func TestGrants(t *testing.T) {
// The part's own consumer on the bus module's stream, and every module's on a module stream it hears, is created by
// the part since the controller ended (ADR 0363 §5, ADR 0590): here mod-y's on mod-x's THINGS.
heard := fixture()
heard.manifestOf("mod-y", strings.Replace(yManifest, `"reads"`, `"consumes":["mod-x.thing.added"],"reads"`, 1))
heard.manifestOf("mod-y", strings.Replace(yManifest, `"reads"`, `"consumes":["mod-x.thing.added"],"resources":[{"id":"program","type":"process","run":["./y"]}],"reads"`, 1))
hp := partOn(heard)
hp.Consumers = heard
hp.Compose()
@@ -3769,7 +3774,8 @@ func TestGrants(t *testing.T) {
p := partOn(b)
p.Consumers = b
p.Compose()
for _, c := range []string{"INDEXER_VIEWS.a_node-tools", "ASSIGNER_VIEWS.a_node-tools"} {
// The runner names no process of its own, so its consumers are its twins alone (H-A-472 second addendum).
for _, c := range []string{"INDEXER_VIEWS.a_node-tools_tools", "ASSIGNER_VIEWS.a_node-tools_tools"} {
if !slices.Contains(b.consumerCreates, c) {
t.Errorf("the tool runner's consumer %s was not made: %v", c, b.consumerCreates)
}
@@ -4140,7 +4146,7 @@ func TestGrants(t *testing.T) {
// configuration, are created where the bus lacks them with no lease key at all (the controller ended by its
// unassignment, never by the lease write); one the bus holds is never changed.
t.Run("account consumers: created where the bus lacks them, with no lease key", func(t *testing.T) {
const cManifest = `{"module":"mod-c","version":"1","own-secrets":{"broker":"/var/lib/c/broker"},"consumes":["mod-y.done"]}`
const cManifest = `{"module":"mod-c","version":"1","own-secrets":{"broker":"/var/lib/c/broker"},"consumes":["mod-y.done"]` + programResource + `}`
b := fixture()
b.manifestOf("mod-c", cManifest)
b.assign("b", "mod-c")
@@ -4167,12 +4173,14 @@ func TestGrants(t *testing.T) {
}
// A held consumer that differs is never changed by a pass: it is raised as a condition naming it, and brought to
// the composed configuration only by hand (grants_consumer_match), with no lease key too.
var said, behind []oc.Condition
var said, behind, orphaned []oc.Condition
p.Say = func(c oc.Condition) error {
switch c.Key {
case grants.GrantMissingCondition, grants.RefusedCondition, grants.HeldModuleCondition, grants.FullCondition: // their own rows
case grants.ConsumerBehindCondition:
behind = append(behind, c)
case grants.DurableOrphanedCondition:
orphaned = append(orphaned, c)
default:
said = append(said, c)
}
@@ -4222,6 +4230,65 @@ func TestGrants(t *testing.T) {
if len(res.Consumers.Behind) != 0 || behind[len(behind)-1].State != occ.StateCleared {
t.Errorf("a consumer back within the bound: %+v, said %+v", res.Consumers, behind)
}
// Design 47 H2, H-A-472 (2): a durable far behind with no pulls waiting, whose twin on the stream (b_mod-c_tools) has
// pulls waiting and delivered within the day, is an orphan: left out of the behind check, named once in
// mesh-grants.durable-orphaned, and cleared when it is gone; with its twin not current, it is behind again.
var twin map[string]any
for _, c := range b.consumers["EVENTS"] {
if c["name"] == "b_mod-c_tools" {
twin = c
}
}
twin["waiting_pulls"], twin["last_delivered"] = 1, "2026-10-25T21:00:00.5Z"
made["pending"] = grants.BehindBound + 1
p.Compose()
o := len(orphaned)
if res = p.Compose(); len(res.Consumers.Behind) != 0 || len(orphaned) != o+1 || orphaned[o].State != occ.StateRaised ||
!strings.Contains(orphaned[o].Explanation, "EVENTS.b_mod-c") || strings.Contains(orphaned[o].Explanation, "EVENTS.b_mod-c_tools") {
t.Fatalf("an orphaned durable: %+v, behind said %+v, orphaned said %+v", res.Consumers, behind[len(behind)-1], orphaned)
}
if p.Compose(); len(orphaned) != o+1 {
t.Fatalf("an orphan was said again: %+v", orphaned)
}
twin["last_delivered"] = "2026-10-23T21:00:00Z"
if res = p.Compose(); len(res.Consumers.Behind) != 1 || len(orphaned) != o+2 || orphaned[o+1].State != occ.StateCleared {
t.Fatalf("a durable whose twin delivered two days ago: %+v, orphaned said %+v", res.Consumers, orphaned)
}
// The third addendum (2): the bound is the setting twin-current-hours; at 72 hours, a twin that delivered two days
// ago is current, so the durable is an orphan again; the setting's file gives its hours, its default when missing.
p.TwinCurrentHours = 72
if res = p.Compose(); len(res.Consumers.Behind) != 0 || len(orphaned) != o+3 || orphaned[o+2].State != occ.StateRaised {
t.Fatalf("with twin-current-hours 72, a durable whose twin delivered two days ago: %+v, orphaned said %+v", res.Consumers, orphaned)
}
p.TwinCurrentHours = 0
if res = p.Compose(); len(res.Consumers.Behind) != 1 || len(orphaned) != o+4 || orphaned[o+3].State != occ.StateCleared {
t.Fatalf("back at the default bound: %+v, orphaned said %+v", res.Consumers, orphaned)
}
for _, row := range []struct {
text string
hours int
fails bool
}{{"72\n", 72, false}, {"", grants.DefaultTwinCurrentHours, false}, {"0", 0, true}, {"x", 0, true}} {
path := filepath.Join(t.TempDir(), "twin-current-hours")
if err := os.WriteFile(path, []byte(row.text), 0o600); err != nil {
t.Fatal(err)
}
if n, err := grants.TwinCurrentHoursSetting(path); n != row.hours || (err != nil) != row.fails {
t.Errorf("twin-current-hours %q: %d, %v", row.text, n, err)
}
}
if n, err := grants.TwinCurrentHoursSetting(filepath.Join(t.TempDir(), "none")); n != 24 || err != nil {
t.Errorf("with no file twin-current-hours is %d, %v; want 24", n, err)
}
made["pending"] = grants.BehindBound
p.Compose()
// H-A-472 second addendum: a module with an account of its own but no process resource with no bus of its own has
// no program reading b_<module>, so only the tool runner's twin is made.
b.manifestOf("mod-t", `{"module":"mod-t","version":"1","own-secrets":{"broker":"/var/lib/t/broker"},"consumes":["mod-y.done"]}`)
b.assign("b", "mod-t")
if res = p.Compose(); !slices.Contains(res.Consumers.Created, "EVENTS.b_mod-t_tools") || slices.Contains(res.Consumers.Created, "EVENTS.b_mod-t") {
t.Errorf("a module with no program of its own: created %v", res.Consumers.Created)
}
})
// 774/AC13 (novox/hq ADR 0561 §7): after take-over a list in which more than one loaded user's hash changes, with no
@@ -618,7 +618,8 @@ func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
}
// The grants part is granted the create of each account's consumer by its stream and name, and no other.
users, err := Users(Records{Nodes: []string{"a"}, Assigned: map[string][]Declared{"a": {
{Module: GrantsModule}, {Module: "m", Consumes: []string{"x.happened"}}, {Module: "quiet"}}}})
{Module: GrantsModule}, {Module: "m", Consumes: []string{"x.happened"}, HasProgram: true}, {Module: "quiet"},
{Module: "tooled", Consumes: []string{"x.happened"}}}}})
if err != nil {
t.Fatal(err)
}
@@ -639,6 +640,13 @@ func TestTheCopiedComposerComposesAsTheControllers(t *testing.T) {
MayPublish(perms, "$JS.API.CONSUMER.DELETE.EVENTS.a_m") || MayPublish(perms, "$JS.API.CONSUMER.CREATE.EVENTS.a_m.mesh.mod.x.event.happened") {
t.Errorf("the grants part creates consumers as %v", u.CreatesConsumers)
}
// Design 47 H2, H-A-472 second addendum: the program's consumer only for a module with a program of its own; the
// tool runner's twin for every consuming module.
for durable, want := range map[string]bool{"a_m": true, "a_m_tools": true, "a_tooled": false, "a_tooled_tools": true} {
if MayPublish(perms, "$JS.API.CONSUMER.CREATE.EVENTS."+durable) != want {
t.Errorf("the grants part creates EVENTS.%s: %v, want %v (%v)", durable, !want, want, u.CreatesConsumers)
}
}
}
// 1952 S5 (novox/hq ADR 0564 clauses 2, 5, 8; Limits 2, 4, 8): the health holders' credentials.
healthChecks(t)
@@ -191,8 +191,9 @@ func ConsumersOf(users []Principal) []ModuleConsumer {
}
seen[key] = true
if needed {
// Each with the tool runner's twin beside it (A-397).
for _, tc := range withToolsTwin(c) {
// Each with the tool runner's twin beside it (A-397), the program's only for a module with a program of its
// own (H-A-472 second addendum).
for _, tc := range twinsOf(c, p.HasProgram) {
out = append(out, ModuleConsumer{Node: p.Node, Module: p.Module, Consumer: tc})
}
// And one on each module stream it hears (novox/hq ADR 0363 §5).
@@ -200,7 +201,7 @@ func ConsumersOf(users []Principal) []ModuleConsumer {
if len(h.Filters) == 0 {
continue
}
for _, tc := range withToolsTwin(heardConsumer(p, h)) {
for _, tc := range twinsOf(heardConsumer(p, h), p.HasProgram) {
out = append(out, ModuleConsumer{Node: p.Node, Module: p.Module, Consumer: tc})
}
}
@@ -243,7 +244,7 @@ func ConsumersOf(users []Principal) []ModuleConsumer {
// 0436 §1, §2, §6), the same reason its ordinary consumer is (ADR 0198): mesh-mail has no
// account of its own (NoAccount) and is carried here, but its process does, and the consumer
// it reads through must exist and survive PruneHeard whether or not the module itself speaks.
add(Principal{Kind: KindModule, Node: p.Node, Module: d.Module, Emits: d.Emits,
add(Principal{Kind: KindModule, Node: p.Node, Module: d.Module, HasProgram: d.HasProgram, Emits: d.Emits,
Consumes: d.Consumes, Serves: d.Serves, Holds: d.Holds, Uses: d.Uses, Watches: d.Watches,
Streams: d.Streams, Hears: ComposedHears(d), Queues: d.Queues})
}
@@ -278,6 +278,17 @@ func withToolsTwin(c Consumer) []Consumer {
return []Consumer{c, twin}
}
// twinsOf are the consumers composed of a module's consumer c (design 47 H2, H-A-472 addendum and second addendum): the
// tool runner's twin <name>_tools for every consuming module, and c itself, the program's, only for a module with a
// program of its own (Declared.HasProgram), so no pass makes a durable nobody reads.
func twinsOf(c Consumer, hasProgram bool) []Consumer {
both := withToolsTwin(c)
if hasProgram {
return both
}
return both[1:]
}
// heardGrants are what a principal publishes to read and acknowledge the consumers it hears module streams
// through: asking about each, pulling from it, acknowledging it — its own name on each stream, no other's.
func heardGrants(p Principal, hears []Heard) []string {
@@ -156,6 +156,9 @@ type Principal struct {
// Process is the process resource's id, for a KindProcess principal alone (novox/hq ADR 0436 §2):
// Username and inbox are <node>.<module>.<process id>.
Process string
// HasProgram is Declared.HasProgram, for a KindModule principal: its program's consumer <node>_<module> is composed
// beside the tool runner's twin (H-A-472 second addendum).
HasProgram bool
// Objects are the bus names of its module's own object buckets, and ReadsObjects those of others it reads (novox/hq
// ADR 0592 §5, §9).
File diff suppressed because one or more lines are too long
@@ -88,6 +88,10 @@ type Declared struct {
// delivered to it and nothing can connect as it (novox/hq issue 195). Said in the negative so a
// record that does not say is composed as it always was.
NoAccount bool
// HasProgram says the module runs a program of its own that reads its consumer <node>_<module>: an account of its own
// (an own broker secret) and a process resource with no `bus` of its own (design 47 H2, H-A-472 second addendum).
// Without it, only the tool runner's twin <node>_<module>_tools is composed, so the pass makes no orphan.
HasProgram bool
// SnapshotsTheBus says the module holds mesh-broker — it is the bus — and so is the one module
// granted the snapshot API, to copy the bus's streams for the night's backup (novox/hq ADR 0235).
SnapshotsTheBus bool
@@ -349,7 +353,7 @@ func Users(r Records) ([]Principal, error) {
continue
}
out = append(out, Principal{
Kind: KindModule, Node: node, Module: d.Module,
Kind: KindModule, Node: node, Module: d.Module, HasProgram: d.HasProgram,
Emits: d.Emits, Consumes: d.Consumes, Serves: d.Serves,
Holds: d.Holds, Uses: d.Uses, Watches: d.Watches, Invokes: d.Invokes,
State: stateNames(d.State), Reads: d.Reads, ReadLogs: d.ReadLogs, SnapshotsTheBus: d.SnapshotsTheBus,
@@ -4,9 +4,13 @@ import (
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"slices"
"sort"
"strconv"
"strings"
"time"
"github.com/nats-io/nats.go/jetstream"
@@ -60,6 +64,74 @@ type heldConsumer struct {
Filters []string `json:"filters"`
MaxDeliver int `json:"max_deliver"`
Pending int64 `json:"pending"`
// WaitingPulls and LastDelivered say whether it is read now: pulls waiting on it, and when it last delivered (H-A-472).
WaitingPulls int64 `json:"waiting_pulls"`
LastDelivered string `json:"last_delivered"`
}
// DefaultTwinCurrentHours is the setting twin-current-hours's default (design 47 H2, H-A-472 third addendum): how many
// hours since a twin's last delivery it is current, within the day; the bus module's manifest names the same setting for
// nats_consumer_remove, so "current" and "orphaned" share one bound.
const DefaultTwinCurrentHours = 24
// EnvTwinCurrentHours names the file the manifest writes the setting twin-current-hours into.
const EnvTwinCurrentHours = "MESH_GRANTS_TWIN_CURRENT_HOURS_FILE"
// TwinCurrentHoursSetting reads the setting twin-current-hours from its file, read at the start: a missing or empty file
// is its default, one that holds no positive whole number an error naming the file.
func TwinCurrentHoursSetting(path string) (int, error) {
if path == "" {
return DefaultTwinCurrentHours, nil
}
raw, err := os.ReadFile(path)
if errors.Is(err, fs.ErrNotExist) {
return DefaultTwinCurrentHours, nil
}
if err != nil {
return 0, err
}
v := strings.TrimSpace(string(raw))
if v == "" {
return DefaultTwinCurrentHours, nil
}
n, err := strconv.Atoi(v)
if err != nil || n < 1 {
return 0, fmt.Errorf("%s holds %q, which is not a positive whole number", path, v)
}
return n, nil
}
// twinCurrentWithin is the part's bound of a current twin: its setting twin-current-hours, else the default.
func (p *Part) twinCurrentWithin() time.Duration {
if p.TwinCurrentHours > 0 {
return time.Duration(p.TwinCurrentHours) * time.Hour
}
return DefaultTwinCurrentHours * time.Hour
}
// durableTwin names the twin of a module's durable: its program's <node>_<module> and its tool runner's
// <node>_<module>_tools (design 47 G, A-340), each the other's.
func durableTwin(name string) string {
if t, ok := strings.CutSuffix(name, "_tools"); ok {
return t
}
return name + "_tools"
}
// orphaned says whether a held durable is an orphan (design 47 H2, H-A-472 (2)): no pulls waiting on it, while its twin
// on the same stream is current, with pulls waiting and a delivery within the bound (the setting twin-current-hours,
// the third addendum). A module that moved into the tool runner left its program's durable so, and one that runs its own
// program again its tool runner's.
func orphaned(h heldConsumer, held map[string]heldConsumer, now time.Time, within time.Duration) bool {
if h.WaitingPulls > 0 {
return false
}
t, ok := held[durableTwin(h.Name)]
if !ok || t.WaitingPulls == 0 {
return false
}
last, err := time.Parse(time.RFC3339Nano, t.LastDelivered)
return err == nil && now.Sub(last) < within
}
// heldConsumers reads the consumers the bus holds on stream, by name.
@@ -123,6 +195,7 @@ func (p *Part) consumers(res *Result, recs buscompose.Records, match string) {
st.Wanted = len(wanted)
seen := map[string]bool{}
defer func() { p.behindSeen = seen }()
var orphans []string
byStream := map[string][]buscompose.Consumer{}
for _, mc := range wanted {
byStream[mc.Consumer.Stream] = append(byStream[mc.Consumer.Stream], mc.Consumer)
@@ -141,7 +214,13 @@ func (p *Part) consumers(res *Result, recs buscompose.Records, match string) {
}
if ok {
if h.Pending > BehindBound {
if p.behindSeen[key] {
// An orphan is left out of the behind check and named in a finding of its own (H-A-472 (2)), so the
// alarm says a real backlog only.
switch {
case !p.behindSeen[key]:
case orphaned(h, held, p.now(), p.twinCurrentWithin()):
orphans = append(orphans, key)
default:
st.Behind = append(st.Behind, fmt.Sprintf("%s: %d pending", key, h.Pending))
}
seen[key] = true
@@ -216,6 +295,7 @@ func (p *Part) consumers(res *Result, recs buscompose.Records, match string) {
}
p.sayDiffer(st)
p.sayBehind(st)
p.sayOrphaned(orphans)
}
// sayDiffer raises the warning condition mesh-grants.consumer-differs while an account consumer the part owns (from the
@@ -347,6 +427,41 @@ func (p *Part) sayBehind(st *ConsumersState) {
}
}
// DurableOrphanedCondition is the key of the warning raised while an account consumer is an orphaned durable: far
// behind, no pulls waiting, its twin on the stream current (design 47 H2, H-A-472 (2)).
const DurableOrphanedCondition = "mesh-grants.durable-orphaned"
// sayOrphaned raises the warning condition mesh-grants.durable-orphaned naming the orphaned durables the behind check
// left out, once per set, and clears it when none is left (they were removed, by nats_consumer_remove).
func (p *Part) sayOrphaned(orphans []string) {
if p.Say == nil {
return
}
why := strings.Join(orphans, ", ")
p.holdMu.Lock()
defer p.holdMu.Unlock()
if p.orphanedKnown && why == p.orphanedSaid {
return
}
c := oc.Condition{Key: DurableOrphanedCondition, Subject: oc.ConditionSubject{Scope: "module", ID: "mesh-grants"}}
if why == "" {
c.State = occ.StateCleared
} else {
c.Severity, c.State = occ.SeverityWarning, occ.StateRaised
c.Summary = fmt.Sprintf("%d durable consumers are orphans: nothing reads them, while their twin on the stream does", len(orphans))
c.Headline = "A listener's old consumer is left over"
needsYou(&c, "their removal, with the bus module's nats_consumer_remove", "These durables are more than the bound behind "+
"their stream's head with no pulls waiting, while their module's twin on the same stream (<node>_<module> and "+
"<node>_<module>_tools) has pulls waiting and delivered within "+p.twinCurrentWithin().String()+" (the setting "+
"twin-current-hours): a module that moved into or out of the tool runner left them, and they are not said as "+
"messages piling up: "+why)
c.Resolved = "no orphaned durable is left"
}
if p.Say(c) == nil {
p.orphanedSaid, p.orphanedKnown = why, true
}
}
// sayWorker logs one named seat worker's outcome (created, held already) once per change, so the journal names each worker the
// pass considers (A-413).
func (p *Part) sayWorker(line string) {
@@ -197,6 +197,9 @@ type Part struct {
// RevisionsPerBranch is the setting grants-for-revisions-per-branch, read at the start (grantsfor.go, design 47 H H-A-195);
// 0 is its default.
RevisionsPerBranch int
// TwinCurrentHours is the setting twin-current-hours, read at the start (consumers.go, design 47 H2, H-A-472 third
// addendum): the bound of a current twin, by which a durable is orphaned; 0 is its default.
TwinCurrentHours int
// Followed is the watched branches the part follows, as named (the branch follower's): grants-for keeps the revisions
// of these alone, and frees a branch's once it leaves them; nil in the tools, where every branch is kept.
Followed func() []string
@@ -230,6 +233,9 @@ type Part struct {
behindSaid string
behindKnown bool
behindSeen map[string]bool
// orphanedSaid is the orphaned durables last said as a condition, once orphanedKnown (consumers.go, H-A-472).
orphanedSaid string
orphanedKnown bool
// fullCleared is whether the old history-full key was cleared since the start (full.go, ADR 0595 §6).
fullCleared bool
// filterRefused is, by consumer, the refusal of the last addition of filters, so a standing refusal is said once.
@@ -206,6 +206,7 @@ func declaredFor(m catalogue.Manifest, seats map[string]catalogue.SeatDeclaratio
if _, reads := m.OwnSecrets["broker"]; !reads {
d.NoAccount = true
}
d.HasProgram = !d.NoAccount && hasOwnProcess(m)
if _, reads := m.OwnSecrets[buscompose.ProvisionerSecret]; reads && m.ClaimsSeat(buscompose.NodeProvisionerSeat) {
d.ProvisionsNode = true
}
@@ -651,6 +652,21 @@ func queueOwnersOf(m catalogue.Manifest) []buscompose.QueueOwner {
return out
}
// hasOwnProcess says the manifest names a process resource with no `bus` of its own: one that runs on the module's own
// account and reads its consumer <node>_<module> (H-A-472 second addendum).
func hasOwnProcess(m catalogue.Manifest) bool {
buses := m.ProcessBuses()
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "process" {
continue
}
if _, bus := buses[fmt.Sprint(r["id"])]; !bus {
return true
}
}
return false
}
// ownServesOf is the seat verbs of the module's processes that declare no bus, which run on the module's own account (design
// 47 G A-115, A-127); a process with a bus carries its own serves (processesOf).
func ownServesOf(m catalogue.Manifest) []string {
+17 -1
View File
@@ -73,6 +73,13 @@
"maximum": 32,
"trusted": false,
"default": 4
},
"twin-current-hours": {
"type": "number",
"description": "how many hours since a module's durable's twin last delivered it is current (design 47 H2 H-A-472): the consumer-behind check leaves out a durable with no pulls waiting whose twin has pulls waiting and a delivery within these hours, and names it in durable-orphaned. The bus module's manifest names the same setting for nats_consumer_remove.",
"minimum": 1,
"trusted": false,
"default": 24
}
},
"emits": [
@@ -175,6 +182,14 @@
"mode": "0600",
"content": "${config:grants-for-revisions-per-branch}"
},
{
"id": "twin-current-hours",
"type": "file",
"path": "${dir:state}/twin-current-hours",
"owner": "mesh-grants",
"mode": "0600",
"content": "${config:twin-current-hours}"
},
{
"id": "grants",
"type": "process",
@@ -185,7 +200,8 @@
],
"env": {
"MESH_BROKER_FILE": "${dir:state}/broker",
"MESH_GRANTS_REVISIONS_PER_BRANCH_FILE": "${dir:state}/grants-for-revisions-per-branch"
"MESH_GRANTS_REVISIONS_PER_BRANCH_FILE": "${dir:state}/grants-for-revisions-per-branch",
"MESH_GRANTS_TWIN_CURRENT_HOURS_FILE": "${dir:state}/twin-current-hours"
},
"user": "mesh-grants",
"health": {