Compare commits

..
Author SHA1 Message Date
mesh-tools 28726acaaf Merge pull request 'The indexer's packages integration test follows H-A-495: a pin moves on a release, never on a reservation (design 47 H2 H-A-495, §805)' (#340) from fix/stale-integration-tests into main 2026-11-22 22:24:16 +00:00
jschoubben 7bacccf562 H-A-495 indexer test: the release re-pins the importer, a reservation moves no pin
TestASourceRevisionMovingMeshSDKNamesItAndItsImporters still expected the
twenty reservations' pipeline-started to keep a re-pin revision pinning
importer to the reserved version, which H-A-495 ended (repin_test.go was
brought in line, this bed was not), so it failed on every pipeline with
"main.5 is not kept after 20s". The bed now waits for the indexer's
acknowledgement of each pipeline-started (its settle moves nothing), then
releases the version: the Go package registry seat's published and every
artifact's package-published. The next revision must be the release's
re-pin, importer its mover at that version. The harness's events durable
now also hears the registry's published, as the manifest's consumes names it,
and this bed keeps package-published on its stream and the state versions.
Fails with the pre-H-A-495 pin rule (main.5 is the reservations' re-pin).

Builds: design 47 H2 H-A-495 and its addendum, §805
2026-11-22 23:18:17 +01:00
mesh-tools 29c4673ad1 Merge pull request 'The runner hears a take-in on its own durable and acknowledges a late event of an ended ask at once; a held key-ttl state gets per-message TTL (design 47 H2 H-A-500, H-A-501)' (#339) from fix/runner-consumer-ttl into main 2026-11-22 22:07:31 +00:00
jschoubben a8240f5549 Runner: a takeins bind that failed is bound again on registration-changed, not on a timer
A takeins bind that failed was tried again every five seconds. Nothing in the
mesh waits on a timer for another part: the runner binds its takeins durable
at its start and again on each registration-changed, as the indexer binds a
branch's consumer, and says each failed bind; until bound, the main durable
works each take-in.

Builds: design 47 H2 H-A-500 fifth addendum
2026-11-22 23:03:41 +01:00
jschoubben 5792e83bc6 Runner: the Ager deletes an aged test run's ask keys with the run
The ask key ask.<ask> is created with the run and deleted with it, but the
Ager deleted an aged run and left its ask keys in test-runs for good; only a
run whose last reader left lost them. The Ager now holds the run clock and
unmarks the aged run's ask keys as the Hearer does.

Builds: design 47 H2 H-A-500 addenda (1)
2026-11-22 23:02:05 +01:00
jschoubben 678d8fa5b3 Runner: a take-in said before the takeins durable is bound is still worked
The takeins durable delivers from the moment it is made, so a take-in said
while the runner was down reached only the main durable, which acknowledged
it with no work: that take-in started no pipeline. Until the takeins durable
is bound, the main durable now works each take-in.taken; from the bind, it
acknowledges with no work those the takeins durable delivers, told apart by
the stream sequence the durable delivers after, so a take-in still waiting on
the main durable is worked whatever the moment it is heard.

Builds: design 47 H2 H-A-500 fourth addendum
2026-11-22 22:59:21 +01:00
jschoubben b2d9d1ac90 Runner: take-in.taken is heard on a durable of its own
The runner heard every event on one serial durable, so a take-in waited behind
hundreds of late step events and its pipeline started ten minutes late. The
runner now makes its own durable <node>_<module>_takeins on EVENTS, filtered
to take-in.taken, and acts on a take-in there alone; the main durable
acknowledges a take-in at once with no work. The test-runs integration test
keeps the runner's findings on their own stream, as the manifest declares:
on EVENTS each reached the main durable unhandled and wrote event-unhandled
again, a flood the take-in now overtakes.

Builds: design 47 H2 H-A-500 (b) and its addenda (2), third addendum (Q2)
2026-11-22 22:53:04 +01:00
jschoubben 6ffddedfbf Grants: every module is granted its own takeins durable on the events stream
A module that hears take-in.taken behind a long run of other events hears it
late; it now hears it on a durable of its own, <node>_<module>_takeins, which
it makes itself. Every module principal, on its own account or carried by a
tool runner, is granted the create, information, pull and acknowledgement of
that one name on EVENTS, none special-cased.

Builds: design 47 H2 H-A-500 (b) and its addenda (2), third addendum (Q1)
2026-11-22 22:27:39 +01:00
jschoubben 146b9b4ff8 Grants: a held bucket of a key-ttl state gets per-message TTL
The indexer's state heard is declared key-ttl, but its bucket was made before
that declaration, and grants' handling of a held stream only added subjects,
raised caps and moved a source filter: every version-reserved the indexer
kept with a time to live was refused (per-message TTL is disabled, 10166) and
given up after five tries. A held bucket whose state is declared key-ttl now
has per-message time to live turned on, add-only like the cap raise, once per
manifest, and said as asserted; the indexer's TTL stays.

Builds: design 47 H2 H-A-501 (corrected)
2026-11-22 22:22:31 +01:00
jschoubben 1d0a40592b Runner: a late event of an ended test run's ask is acknowledged at once from its ask key
Every ask-taken and step-progress listed and read every pipeline to find its
test run, so after a cancel hundreds of late events of ended runs cost one to
two seconds each and held the runner's one event consumer for minutes. The
state test-runs now keeps an ask key ask.<ask> holding the run's key, written
with each ask a run is given and deleted with the run; an event of an ended
run's ask is judged from that key and the run alone and acknowledged at once,
one journal line per hundred.

Builds: design 47 H2 H-A-500 (a) and its addenda (1)
2026-11-22 22:19:34 +01:00
28 changed files with 900 additions and 39 deletions
@@ -31,6 +31,7 @@ import (
"git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/gopackageregistry"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
@@ -800,8 +801,10 @@ func serveAssignerAssignments(t *testing.T, ctx context.Context, js jetstream.Je
}
if _, err := js.CreateOrUpdateConsumer(ctx, "EVENTS", jetstream.ConsumerConfig{Durable: "n1_mesh-delivery-indexer", AckPolicy: jetstream.AckExplicitPolicy,
// and the build seat's step-ended the manifest's consumes names (design 47 G §713: the end of a validation ask), and the
// builder's version-reserved, which no stream of its own keeps (design 47 H2 §805).
FilterSubjects: []string{"mesh.mod." + assignerHolderModule + ".event.views.>", meshcatalog.SubjectRoom, nodebuildgo.SubjectStepEndedAll, meshbuilds.SubjectVersionReserved}}); err != nil {
// builder's version-reserved, which no stream of its own keeps (design 47 H2 §805), and the Go package registry seat's
// published, a series' newest a pin names (H-A-495 addendum).
FilterSubjects: []string{"mesh.mod." + assignerHolderModule + ".event.views.>", meshcatalog.SubjectRoom, nodebuildgo.SubjectStepEndedAll, meshbuilds.SubjectVersionReserved,
gopackageregistry.SubjectPublished}}); err != nil {
t.Fatal(err)
}
em := views.Emitter{Owner: assignerHolderModule, Subject: process.Account{Node: "n1", Module: assignerHolderModule}.EventSubject,
@@ -19,9 +19,11 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/gopackageregistry"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
"mesh-delivery-indexer/internal/index"
@@ -76,10 +78,11 @@ func untilModule(t *testing.T, ctx context.Context, js jetstream.JetStream, name
// of lib-a and one of lib-b, and so their trees alone, moves lib-a, lib-b and their importers alone, never the other 32
// modules whose repository's commit moved too.
//
// Design 47 H2 §805 (H-A-453 (3)): the builder's twenty version-reserved of mesh-sdk in that revision's pipeline (twenty
// artifacts of its Go package, as repin_test.go reads them) are kept in heard and gathered, and its pipeline-started keeps one
// re-pin revision of main, importer its mover pinned to the reserved version; the next pipeline's reservation keeps the
// revision after it, so none came between.
// Design 47 H2 §805 (H-A-453 (3)) and H-A-495 with its addendum: the builder's twenty version-reserved of mesh-sdk in that
// revision's pipeline (twenty artifacts of its Go package, as repin_test.go reads them) are kept in heard and gathered, and
// they and its pipeline-started move no pin: a pin names only a published version. The release of the reserved version (the
// Go package registry seat's published and every artifact's package-published) keeps one re-pin revision of main, importer
// its mover pinned to it; the next pipeline's reservation keeps none, so the revision after it is the next release's.
func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
const (
sdkGo = "git.novox.be/novox/mesh-sdk/go"
@@ -143,6 +146,24 @@ func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
if heardKV, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, index.HeardState), History: 64, LimitMarkerTTL: time.Minute}); err != nil {
t.Fatal(err)
}
// package-published on its kept stream with the indexer's durable, as the grants part makes them, and the state versions
// a release is recorded in (H-A-495: the release re-pins).
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, index.VersionsState), History: 1}); err != nil {
t.Fatal(err)
}
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: meshbuilds.StreamPackagePublished, Subjects: []string{meshbuilds.SubjectPackagePublished}}); err != nil {
t.Fatal(err)
}
if _, err := js.CreateOrUpdateConsumer(ctx, meshbuilds.StreamPackagePublished, jetstream.ConsumerConfig{Durable: "n1_mesh-delivery-indexer", AckPolicy: jetstream.AckExplicitPolicy,
MaxDeliver: EventDeliveries}); err != nil {
t.Fatal(err)
}
// The indexer's durable on pipeline-started samples every acknowledgement, so the bed hears that the settle a
// pipeline-started makes (H-A-424 (1)) has ended before it releases, never by sleeping.
if _, err := js.CreateOrUpdateConsumer(ctx, meshpipelinerunner.StreamPipelineStarted, jetstream.ConsumerConfig{Durable: "n1_mesh-delivery-indexer",
AckPolicy: jetstream.AckExplicitPolicy, FilterSubjects: []string{meshpipelinerunner.SubjectPipelineStartedAll}, SampleFrequency: "100%"}); err != nil {
t.Fatal(err)
}
// A dry run's key left by an indexer that stopped mid-run (§804).
if dryRuns, err = js.KeyValue(ctx, index.Bucket(testIndexer, index.DryRunsState)); err != nil {
t.Fatal(err)
@@ -260,8 +281,9 @@ func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
t.Errorf("the catalogue's commit changing lib-a's and lib-b's tests moves %v of its 36 modules, want lib-a, lib-b and their importers alone", movers)
}
// §805: twenty reservations of mesh-sdk's packages in main's revision of source 12, kept and gathered; its pipeline-started
// keeps one re-pin revision of main, and the next pipeline's reservation the one after it.
// §805, H-A-495: twenty reservations of mesh-sdk's packages in main's revision of source 12, kept and gathered, and its
// pipeline-started move no pin; the release of the reserved version keeps one re-pin revision of main, and the next
// pipeline's reservation none, so the revision after it is the next release's.
objects := newBusObjects(js, testAccount, index.CatalogRevisionsObjects)
untilRevision := func(n int64) meshcatalog.CatalogRevision {
t.Helper()
@@ -299,19 +321,61 @@ func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
return v != nil && json.Unmarshal(v, &r) == nil && len(r.Reservations) == artifacts
})
}
// release publishes mesh-sdk's Go package at version, as the registry seat says it (the series' newest a pin names,
// H-A-495 addendum), and every artifact of mesh-sdk's entry in main's revision n, as the builder says them.
release := func(n int64, version string) {
t.Helper()
raw, _ := json.Marshal(gopackageregistry.PublishedEvent{Kind: "package", Name: sdkGo, Module: sdkGo, Version: version, Commit: m2, Digest: "sha256:" + version})
if _, err := js.Publish(ctx, gopackageregistry.SubjectPublished, raw); err != nil {
t.Fatal(err)
}
_, entries, err := index.ReadRevision(ctx, objects, index.BranchObject("main", n))
if err != nil {
t.Fatal(err)
}
i := slices.IndexFunc(entries, func(e meshcatalog.CatalogEntry) bool { return e.Module == "mesh-sdk" })
if i < 0 {
t.Fatalf("main.%d holds no entry of mesh-sdk", n)
}
for _, a := range entries[i].Artifacts {
raw, _ := json.Marshal(meshbuilds.PackagePublished{Module: "mesh-sdk", Artifact: a.Ref, Digest: "sha256:" + a.Ref + version, Version: version, Source: 12, Branch: "main"})
if _, err := js.Publish(ctx, meshbuilds.SubjectPackagePublished, raw); err != nil {
t.Fatal(err)
}
}
}
// started says pipeline-started of main's revision n and waits for the indexer's acknowledgement of it, which follows its settle.
started := func(n int64) {
t.Helper()
acks, err := js.Conn().SubscribeSync("$JS.EVENT.METRIC.CONSUMER.ACK." + meshpipelinerunner.StreamPipelineStarted + ".n1_mesh-delivery-indexer")
if err != nil {
t.Fatal(err)
}
defer func() { _ = acks.Unsubscribe() }()
if err := js.Conn().Flush(); err != nil {
t.Fatal(err)
}
sayPipeline(t, ctx, js, false, "main", n)
if _, err := acks.NextMsg(waitBound); err != nil {
t.Fatalf("the pipeline-started of main.%d is not acknowledged after %s: %v", n, waitBound, err)
}
}
r := taken.Revision
reserve(r, "0.1.28", 20)
sayPipeline(t, ctx, js, false, "main", r)
if rev := untilRevision(r + 1); rev.CallKind != meshcatalog.CatalogRevisionCallKindRelease || !strings.Contains(rev.Call, index.BranchObject("main", r)+"'s") ||
started(r)
release(r, "0.1.28")
if rev := untilRevision(r + 1); rev.CallKind != meshcatalog.CatalogRevisionCallKindRelease || rev.Call != index.ReleaseCall("mesh-sdk", "0.1.28") ||
len(rev.Movers) != 1 || rev.Movers[0].Module != "importer" {
t.Errorf("main.%d = %+v, want one re-pin revision of the twenty reservations, importer its mover", r+1, rev)
t.Errorf("main.%d = %+v, want the re-pin revision of the release of mesh-sdk 0.1.28, importer its mover: twenty reservations move no pin", r+1, rev)
}
untilModule(t, ctx, js, "importer", func(m meshcatalog.ModulesByModuleValue) bool { return m.Packages[sdkGo] == "v0.1.28" })
reserve(r+1, "0.1.29", 1)
sayPipeline(t, ctx, js, false, "main", r+1)
if rev := untilRevision(r + 2); !strings.Contains(rev.Call, index.BranchObject("main", r+1)+"'s") {
t.Errorf("main.%d = %+v, want the next pipeline's re-pin: the twenty reservations kept more than one revision", r+2, rev)
started(r + 1)
release(r+1, "0.1.29")
if rev := untilRevision(r + 2); rev.Call != index.ReleaseCall("mesh-sdk", "0.1.29") {
t.Errorf("main.%d = %+v, want the re-pin of the release of mesh-sdk 0.1.29: the reservation of 0.1.29 keeps no revision", r+2, rev)
}
untilModule(t, ctx, js, "importer", func(m meshcatalog.ModulesByModuleValue) bool { return m.Packages[sdkGo] == "v0.1.29" })
// A branch's take-in moving mesh-sdk to m3 is a dry run: its packages stay in dry-runs while it runs, the trunk's are
// untouched, and its keys go once it answered (§804).
@@ -60,3 +60,37 @@ func TestAgedHub(t *testing.T) {
t.Errorf("a message asked again is noted applied: %+v", last)
}
}
// TestTheTakeInsSwitch: until the takeins durable is bound, the main durable works every take-in; from the bind, it works
// only those said at or before the sequence the durable delivers after, and acknowledges the rest with no work
// (design 47 H2 H-A-500 fourth addendum).
func TestTheTakeInsSwitch(t *testing.T) {
for _, row := range []struct {
name string
boundFrom uint64
bound bool
seq uint64
delivers bool
}{
{"not bound, any sequence is worked", 0, false, 7, false},
{"bound after 5, the take-in at 5 is worked", 5, true, 5, false},
{"bound after 5, the take-in at 6 is the durable's", 5, true, 6, true},
{"bound on an empty stream, the first take-in is the durable's", 0, true, 1, true},
} {
t.Run(row.name, func(t *testing.T) {
sw := &takeInsSwitch{}
if row.bound {
sw.boundFrom(row.boundFrom)
}
hub := &oneHub{}
worked := false
if err := dropTakenIn(hub, sw, func(jetstream.Msg) { worked = true }); err != nil {
t.Fatal(err)
}
hub.handle(ackedMsg{seq: row.seq})
if worked == row.delivers {
t.Errorf("worked by the main durable: %v, want %v", worked, !row.delivers)
}
})
}
}
@@ -237,8 +237,13 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
}
advancer.Store(&advance)
// A registration-changed places the steps of its module's pipelines started before it, from it (A-327).
// A registration-changed places the steps of its module's pipelines started before it, from it (A-327), and binds the
// takeins durable when its bind failed at the start (H-A-500 fifth addendum).
var bindTakeInsAgain atomic.Pointer[func()]
register := func(reg catalog.Registration) {
if f := bindTakeInsAgain.Load(); f != nil {
(*f)()
}
if n, err := starter.Registered(store, reg, rv.Catalog.View); err != nil || n > 0 {
logger.Printf("the pipelines of %s started before its registration: %d placed (%v)", reg.Module, n, err)
}
@@ -321,6 +326,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err
}
if err := hearStepProgress(heard, func(onSeat, ask string) error {
if hearer.LateRunAsk("step-progress", ask) {
return nil // an ended run's ask: acknowledged at once, nothing read but its ask key and run (H-A-500 (a))
}
progressed(clock, onSeat, ask, logger.Printf) // the ask's bound is armed again from now (H-A-395)
return hearer.RunProgress(ask)
}, logger.Printf); err != nil {
@@ -333,6 +341,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err
}
taken := func(ask, node string, at time.Time) (*time.Time, error) {
if hearer.LateRunAsk("ask-taken", ask) {
return nil, nil // an ended run's ask: acknowledged at once, nothing read but its ask key and run (H-A-500 (a))
}
takenAt, err := hearer.Taken(ask, node, at)
if err != nil {
return nil, err
@@ -370,7 +381,17 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err := hearRecovery(heard, kvRecoveries{ctx, recovered}, logger.Printf); err != nil {
return err
}
if err := hearTakenIn(heard, starter, rv.Catalog, logger.Printf); err != nil {
// take-in.taken is acted on on the runner's own takeins durable once that is bound, so no step traffic holds a take-in;
// until then the main durable works it, and from the bind acknowledges it at once (design 47 H2 H-A-500 (b) and its
// addenda, the fourth: no take-in said while the runner was down is lost).
takeInsBound := &takeInsSwitch{}
if err := dropTakenIn(heard, takeInsBound, takenIn(starter, rv.Catalog, logger.Printf)); err != nil {
return err
}
takeIns := &oneHandler{}
takeInsName := cred.EventConsumerName() + "_" + takeInsSuffix
takeInsHeard := agedHub{hub: takeIns, ledger: process.NewConsumerLedger(process.EventsStream, takeInsName, consumersSink(consumers), nil, logger.Printf)}
if err := hearTakenIn(takeInsHeard, starter, rv.Catalog, logger.Printf); err != nil {
return err
}
if err := hearReleased(heard, hearer, starter.Registration, logger.Printf); err != nil {
@@ -387,6 +408,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// their callers against (A-186): registered before the hub starts, read again once its events are live.
seats := seatlist.NewPart("runner")
seats.Follow(seatlist.ProcessBus(calls.Call, hub), nil, logger.Printf)
again := bindTakeIns(ctx, js, cred, takeIns.handle, takeInsBound, logger.Printf)
bindTakeInsAgain.Store(&again)
if err := hub.StartUntilBound(ctx, js, cred, nil, seats.EventsLive); err != nil {
logger.Printf("the consumer %s is not bound yet (%v): bound once the bus makes it", cred.EventConsumerName(), err)
}
@@ -172,6 +172,7 @@ func testTheQueueRunsPipelinesInTurn(t *testing.T) {
return ""
}, logs)
}()
takeInsHeard(t, ctx, logs)
return func() { stop(); <-done }
}
stopRunner := start()
@@ -52,6 +52,9 @@ func (s *syncBuffer) String() string { s.mu.Lock(); defer s.mu.Unlock(); return
// bus proved; the pipeline then ends passed.
func TestTheRunnerPlacesAPipelineAsksItsBuildTakesItsEndAndAnswersAReview(t *testing.T) {
t.Run("the queue runs pipelines in turn", testTheQueueRunsPipelinesInTurn)
t.Run("a take-in is heard while late events queue on the main durable", testATakeInIsHeardWhileLateEventsQueue)
t.Run("a take-in said before the bind is still worked", testATakeInSaidBeforeTheBindIsStillWorked)
t.Run("a takeins bind that failed is bound again on registration-changed", testATakeInsBindThatFailedIsBoundAgainOnRegistrationChanged)
t.Run("step log read does not wait for a batch it cannot fill", testStepLogReadDoesNotWaitForABatchItCannotFill)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
@@ -384,6 +387,7 @@ func startWhole(t *testing.T, ctx context.Context, regs []catalog.Registration,
}, logs)
}()
t.Cleanup(func() { stop(); <-done })
takeInsHeard(t, ctx, logs)
pipelines, err := js.KeyValue(ctx, module+"_pipelines")
if err != nil {
t.Fatal(err)
@@ -500,6 +504,14 @@ func findingsStream(t *testing.T, ctx context.Context, js jetstream.JetStream, m
}
}
// takeInsHeard waits until the runner's takeins durable is bound, which the runner makes at its first start from the
// moment it is made (H-A-500 (b)); a take-in said before is worked by the main durable (the fourth addendum), but the tests
// publish after the bind so that each take-in is heard where the running runner hears it.
func takeInsHeard(t *testing.T, ctx context.Context, logs *syncBuffer) {
t.Helper()
waitFor(t, ctx, logs, func() bool { return strings.Contains(logs.String(), "take-ins: heard on the consumer") })
}
func waitFor(t *testing.T, ctx context.Context, logs *syncBuffer, cond func() bool) {
t.Helper()
for ctx.Err() == nil {
@@ -3,12 +3,19 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sync"
"sync/atomic"
"time"
"github.com/nats-io/nats.go/jetstream"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-pipeline/internal/pipeline"
)
@@ -57,10 +64,140 @@ func revisedHear(revised func(ev catalog.CatalogRevisionEvent) (int, error), log
}
}
// takeInsSuffix names the runner's own second durable on the shared events stream, `<node>_<module>_takeins`, on which alone
// take-in.taken is acted on, so a pipeline starts within seconds of its take-in whatever the step traffic on the runner's
// main durable (design 47 H2 H-A-500 (b) and its addenda).
const takeInsSuffix = "takeins"
// The ack wait and the deliveries of the takeins durable, as the grants part gives a module's consumers (H-A-500 (b)).
const (
takeInsAckWait = 30 * time.Second
takeInsMaxDeliver = 5
)
// takeInsConsumerConfig is the takeins durable as the runner makes it: take-in.taken alone, acknowledged one by one, from
// the moment it is made (H-A-500 (b)).
func takeInsConsumerConfig(cred process.Account) jetstream.ConsumerConfig {
cfg := cred.SeatEventConsumerConfig([]string{pipeline.TakenInSubject}, false)
cfg.Durable = cred.EventConsumerName() + "_" + takeInsSuffix
cfg.AckWait, cfg.MaxDeliver = takeInsAckWait, takeInsMaxDeliver
cfg.DeliverPolicy = jetstream.DeliverNewPolicy
return cfg
}
// oneHandler is the handlers of one durable that hears one pattern: the takeins durable's (H-A-500 (b)).
type oneHandler struct {
pattern string
handle func(jetstream.Msg)
}
func (o *oneHandler) Handle(pattern string, handle func(jetstream.Msg)) error {
if o.handle != nil {
return fmt.Errorf("the durable hears %q already, so %q has no taker", o.pattern, pattern)
}
o.pattern, o.handle = pattern, handle
return nil
}
// takeInsSwitch says which durable a take-in.taken is worked on (H-A-500 fourth addendum): until the runner's takeins
// durable is bound, the main durable does the work, so a take-in said while the runner was down is lost to no one; from
// the bind, the takeins durable alone works the take-ins said after it was made, and the main durable acknowledges those
// at once with no work. The boundary is the stream sequence the takeins durable delivers from, so a take-in said before
// the durable was made and still waiting on the main durable is worked there whatever the moment it is heard.
type takeInsSwitch struct {
// from holds 1 + the stream sequence the takeins durable delivers after; 0 while it is not bound.
from atomic.Uint64
}
// boundFrom says the takeins durable is bound and delivers every take-in said after stream sequence seq.
func (s *takeInsSwitch) boundFrom(seq uint64) { s.from.Store(seq + 1) }
// delivers says whether the takeins durable delivers m, so the main durable has no work on it.
func (s *takeInsSwitch) delivers(m jetstream.Msg) bool {
from := s.from.Load()
if from == 0 {
return false
}
meta, err := m.Metadata()
if err != nil {
// A message whose sequence is unknown is worked rather than dropped: the work is done once per key, never twice.
return false
}
return meta.Sequence.Stream >= from
}
// dropTakenIn has the main durable work each take-in.taken the takeins durable does not deliver, and acknowledge the rest at
// once with no work, so the finding event-unhandled stays clear (H-A-500 third and fourth addenda).
func dropTakenIn(hub handlers, sw *takeInsSwitch, work func(jetstream.Msg)) error {
return hub.Handle(pipeline.TakenInSubject, func(m jetstream.Msg) {
if sw.delivers(m) {
_ = m.Ack()
return
}
work(m)
})
}
// bindTakeIns binds the runner's takeins durable and hears it with handle, the durable made when the bus does not hold it
// (H-A-500 (b)): bound at once when it can be, so no take-in said after the runner's start is missed. A bind that fails
// (the bus has not made the runner's grant yet) is said, and the answered func binds again, called on each
// registration-changed as the indexer binds a branch's consumer on each (H-A-500 fifth addendum: nothing waits on a timer
// for another part); once bound it does nothing. Once bound, sw says so with the stream sequence the durable delivers
// after, and the main durable stops working the take-ins it delivers (the fourth addendum).
func bindTakeIns(ctx context.Context, js jetstream.JetStream, cred process.Account, handle func(jetstream.Msg), sw *takeInsSwitch,
logf func(string, ...any)) (again func()) {
bind := func() error {
cons, err := cred.NamedEventConsumer(ctx, js, process.EventsStream, takeInsSuffix)
if errors.Is(err, jetstream.ErrConsumerNotFound) {
cfg := takeInsConsumerConfig(cred)
if cons, err = js.CreateConsumer(ctx, process.EventsStream, cfg); err != nil {
return fmt.Errorf("the consumer %s on %s is not there and is not made: %w", cfg.Durable, process.EventsStream, err)
}
logf("take-ins: the consumer %s on %s was not there: the runner made it (H-A-500 (b))", cfg.Durable, process.EventsStream)
}
if err != nil {
return err
}
// A durable just made delivers after the stream's last sequence; one found, after what it delivered already.
info, err := cons.Info(ctx)
if err != nil {
return fmt.Errorf("the consumer %s_%s is not read: %w", cred.EventConsumerName(), takeInsSuffix, err)
}
if _, err = process.HearEvents(cons, handle, logf); err != nil {
return err
}
sw.boundFrom(info.Delivered.Stream)
return nil
}
var mu sync.Mutex
bound := false
try := func() {
mu.Lock()
defer mu.Unlock()
if bound || ctx.Err() != nil {
return
}
if err := bind(); err != nil {
logf("take-ins: the consumer %s_%s is not bound, so the main durable works each take-in; bound again at the next registration-changed: %v",
cred.EventConsumerName(), takeInsSuffix, err)
return
}
bound = true
logf("take-ins: heard on the consumer %s_%s (H-A-500 (b))", cred.EventConsumerName(), takeInsSuffix)
}
try()
return try
}
// hearTakenIn starts the pipeline of each take-in.taken heard (design 47 H §746): one per take-in, the message acknowledged
// once it is written, and heard again when a write failed, so the key already written starts nothing twice.
func hearTakenIn(hub handlers, starter *pipeline.Starter, v *pipeline.Views, logf func(string, ...any)) error {
return hub.Handle(pipeline.TakenInSubject, func(m jetstream.Msg) {
return hub.Handle(pipeline.TakenInSubject, takenIn(starter, v, logf))
}
// takenIn is the work of one take-in.taken, on whichever durable delivers it (H-A-500 fourth addendum).
func takenIn(starter *pipeline.Starter, v *pipeline.Views, logf func(string, ...any)) func(jetstream.Msg) {
return func(m jetstream.Msg) {
// An event of the old form is refused by the contract's strict decode, said in one journal line and ended (§765, H-A-98).
ev, err := pipeline.DecodeTakenIn(m.Data())
if err != nil {
@@ -76,7 +213,7 @@ func hearTakenIn(hub handlers, starter *pipeline.Starter, v *pipeline.Views, log
}
logf("the take-in of %s source %d started %d pipelines", ev.Branch, ev.Source, n)
_ = m.Ack()
})
}
}
// hearReleased writes each version-released's version into its mover (design 47 H §746, H-A-52), so the pipeline's end names
@@ -0,0 +1,221 @@
//go:build integration
package main
import (
"context"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/testbus"
"mesh-delivery-pipeline/internal/pipeline"
)
// Design 47 H2 H-A-500 (b) and its addenda: a take-in.taken said while 500 late step-progress events wait on the runner's
// main durable, its handler held, is heard at once on the runner's own takeins durable, which the runner makes itself; the
// main durable acknowledges the take-in with no work. testATakeInIsHeardWhileLateEventsQueue is a row of
// TestTheRunnerPlacesAPipelineAsksItsBuildTakesItsEndAndAnswersAReview (the module's tests are folded under its budget).
func testATakeInIsHeardWhileLateEventsQueue(t *testing.T) {
url := testbus.URL(t)
nc, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cred := process.Account{URL: url, Node: "n1", Module: "mesh-delivery-pipeline"}
events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}})
if err != nil {
t.Fatal(err)
}
if _, err := events.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{Durable: cred.EventConsumerName(), AckPolicy: jetstream.AckExplicitPolicy}); err != nil {
t.Fatal(err)
}
const progress = "mesh.seat.node-build-go.event.step-progress.n1"
for i := 0; i < 500; i++ {
if _, err := js.Publish(ctx, progress, []byte(`{"ask":"late","phase":"run"}`)); err != nil {
t.Fatal(err)
}
}
// The main durable's step-progress handler is held until the test ends: every late event waits behind the first.
held := make(chan struct{})
defer close(held)
hub := process.NewEventHub(t.Logf)
if err := hub.Handle(progress, func(m jetstream.Msg) { <-held; _ = m.Ack() }); err != nil {
t.Fatal(err)
}
sw := &takeInsSwitch{}
if err := dropTakenIn(hub, sw, func(m jetstream.Msg) { t.Errorf("the main durable worked %s after the bind", m.Subject()); _ = m.Ack() }); err != nil {
t.Fatal(err)
}
if err := hub.Start(ctx, js, cred); err != nil {
t.Fatal(err)
}
defer hub.Stop()
heard := make(chan string, 1)
takeIns := &oneHandler{}
if err := takeIns.Handle(pipeline.TakenInSubject, func(m jetstream.Msg) { heard <- m.Subject(); _ = m.Ack() }); err != nil {
t.Fatal(err)
}
bindTakeIns(ctx, js, cred, takeIns.handle, sw, t.Logf)
if _, err := js.Publish(ctx, "mesh.mod.mesh-delivery-indexer.event.take-in.taken", []byte(`{}`)); err != nil {
t.Fatal(err)
}
select {
case s := <-heard:
if s != "mesh.mod.mesh-delivery-indexer.event.take-in.taken" {
t.Errorf("the takeins durable heard %s", s)
}
case <-ctx.Done():
t.Fatal("the take-in is not heard while late events wait on the main durable")
}
info, err := events.Consumer(ctx, cred.EventConsumerName()+"_"+takeInsSuffix)
if err != nil {
t.Fatalf("the takeins durable is not made: %v", err)
}
if f := info.CachedInfo().Config.FilterSubjects; len(f) != 1 || f[0] != pipeline.TakenInSubject {
t.Errorf("the takeins durable filters %v, want take-in.taken alone", f)
}
}
// Design 47 H2 H-A-500 fourth addendum: a take-in.taken said before the runner's takeins durable is bound (the runner was
// down, or the bind failed) is worked by the main durable, so it starts its pipeline; one said after the bind is worked by
// the takeins durable alone and the main durable acknowledges it with no work. A row of
// TestTheRunnerPlacesAPipelineAsksItsBuildTakesItsEndAndAnswersAReview.
func testATakeInSaidBeforeTheBindIsStillWorked(t *testing.T) {
url := testbus.URL(t)
nc, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cred := process.Account{URL: url, Node: "n1", Module: "mesh-delivery-pipeline"}
events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}})
if err != nil {
t.Fatal(err)
}
if _, err := events.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{Durable: cred.EventConsumerName(), AckPolicy: jetstream.AckExplicitPolicy}); err != nil {
t.Fatal(err)
}
const subject = "mesh.mod.mesh-delivery-indexer.event.take-in.taken"
// Said while the runner was down: the main durable holds it, the takeins durable does not exist yet.
if _, err := js.Publish(ctx, subject, []byte(`{"before":true}`)); err != nil {
t.Fatal(err)
}
// The main durable's take-in handler is held until the takeins durable is bound, as a main durable busy with step
// events would be: the take-in said before the bind is still worked, by its sequence, not by the moment it is heard.
held := make(chan struct{})
worked, heard := make(chan string, 2), make(chan string, 2)
hub := process.NewEventHub(t.Logf)
sw := &takeInsSwitch{}
if err := dropTakenIn(hub, sw, func(m jetstream.Msg) { <-held; worked <- string(m.Data()); _ = m.Ack() }); err != nil {
t.Fatal(err)
}
if err := hub.Start(ctx, js, cred); err != nil {
t.Fatal(err)
}
defer hub.Stop()
takeIns := &oneHandler{}
if err := takeIns.Handle(pipeline.TakenInSubject, func(m jetstream.Msg) { heard <- string(m.Data()); _ = m.Ack() }); err != nil {
t.Fatal(err)
}
bindTakeIns(ctx, js, cred, takeIns.handle, sw, t.Logf)
if sw.from.Load() == 0 {
t.Fatal("the takeins durable is not bound")
}
// Said after the bind: the takeins durable's alone.
if _, err := js.Publish(ctx, subject, []byte(`{"after":true}`)); err != nil {
t.Fatal(err)
}
close(held)
select {
case d := <-worked:
if d != `{"before":true}` {
t.Errorf("the main durable worked %s, want the take-in said before the bind", d)
}
case <-ctx.Done():
t.Fatal("the take-in said before the bind is worked by no durable")
}
select {
case d := <-heard:
if d != `{"after":true}` {
t.Errorf("the takeins durable heard %s, want the take-in said after the bind", d)
}
case <-ctx.Done():
t.Fatal("the take-in said after the bind is not heard on the takeins durable")
}
// The main durable acknowledged the second with no work: nothing more is worked once its queue is drained.
select {
case d := <-worked:
t.Errorf("the main durable worked %s after the bind", d)
case <-time.After(2 * time.Second):
}
info, err := events.Consumer(ctx, cred.EventConsumerName())
if err != nil {
t.Fatal(err)
}
ci, err := info.Info(ctx)
if err != nil {
t.Fatal(err)
}
if ci.NumAckPending != 0 || ci.NumPending != 0 {
t.Errorf("the main durable holds %d pending and %d unacknowledged, want none", ci.NumPending, ci.NumAckPending)
}
}
// Design 47 H2 H-A-500 fifth addendum: a takeins bind that failed at the start (here the events stream is not there yet, as
// the runner's grant is not there before the bus composes its account) is said and bound again on the next
// registration-changed, never on a timer. A row of TestTheRunnerPlacesAPipelineAsksItsBuildTakesItsEndAndAnswersAReview.
func testATakeInsBindThatFailedIsBoundAgainOnRegistrationChanged(t *testing.T) {
url := testbus.URL(t)
nc, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cred := process.Account{URL: url, Node: "n1", Module: "mesh-delivery-pipeline"}
sw := &takeInsSwitch{}
heard := make(chan string, 1)
again := bindTakeIns(ctx, js, cred, func(m jetstream.Msg) { heard <- m.Subject(); _ = m.Ack() }, sw, t.Logf)
if sw.from.Load() != 0 {
t.Fatal("the takeins durable is bound with no events stream")
}
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}}); err != nil {
t.Fatal(err)
}
again() // the registration-changed heard once the bus holds the stream and the grant
if sw.from.Load() == 0 {
t.Fatal("the takeins durable is not bound again on registration-changed")
}
if _, err := js.Publish(ctx, "mesh.mod.mesh-delivery-indexer.event.take-in.taken", []byte(`{}`)); err != nil {
t.Fatal(err)
}
select {
case <-heard:
case <-ctx.Done():
t.Fatal("the take-in is not heard on the durable bound again")
}
again() // bound already: nothing is bound twice
}
@@ -166,6 +166,39 @@ func (t testRunsKV) Has(key string) (bool, error) {
return err == nil, err
}
// askValue is an ask key's value: the key of the run the ask is of (H-A-500 addendum (1)).
type askValue struct {
Run string `json:"run"`
}
// MarkAsk creates the ask key of ask holding run; one written already is kept.
func (t testRunsKV) MarkAsk(ask, run string) error {
raw, err := json.Marshal(askValue{Run: run})
if err != nil {
return err
}
if _, err := t.kv.Create(t.ctx, pipeline.AskKey(ask), raw); err != nil && !errors.Is(err, jetstream.ErrKeyExists) {
return err
}
return nil
}
// RunOfAsk reads the ask key of ask: the run it holds.
func (t testRunsKV) RunOfAsk(ask string) (string, bool, error) {
e, err := t.kv.Get(t.ctx, pipeline.AskKey(ask))
if errors.Is(err, jetstream.ErrKeyNotFound) {
return "", false, nil
}
if err != nil {
return "", false, err
}
var v askValue
if err := json.Unmarshal(e.Value(), &v); err != nil || v.Run == "" {
return "", false, fmt.Errorf("the ask key of ask %s holds no run: %v", ask, err)
}
return v.Run, true, nil
}
func (t testRunsKV) Keys() ([]string, error) {
all, err := t.kv.Keys(t.ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
@@ -416,7 +449,7 @@ func openTestRuns(ctx context.Context, js jetstream.JetStream, cred process.Acco
hearer.TestRuns, hearer.Clock, hearer.Runs, hearer.DropRun = runs, runs, settings, drop
hearer.RunOutputs = objects
if hearer.Ager != nil {
hearer.Ager.Runs, hearer.Ager.RunOutputs, hearer.Ager.RunsKept = runs, objects.Delete, settings.Kept
hearer.Ager.Runs, hearer.Ager.RunOutputs, hearer.Ager.RunsKept, hearer.Ager.Clock = runs, objects.Delete, settings.Kept, runs
}
logf("test runs: cap %d bytes, kept %s, step bound %s, untaken bound %s", settings.CapBytes, settings.Kept, settings.StepBound, settings.Untaken)
return kv, objects, nil
@@ -75,7 +75,10 @@ func TestTestRunsAreAskedOncePerLevelAndMoveEveryReader(t *testing.T) {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(module, seat.StateStepLog), Subjects: []string{buscompose.LogSubject(module, seat.StateStepLog) + ".>"}}); err != nil {
t.Fatal(err)
}
events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>", "mesh.stream.*.event.>"}})
// The runner's findings ride its own stream, as its manifest declares, never EVENTS: on EVENTS each one would reach its
// main durable unhandled and write event-unhandled again, a flood a take-in on its own durable overtakes (H-A-500 (b)).
findingsStream(t, ctx, js, module)
events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}})
if err != nil {
t.Fatal(err)
}
@@ -224,6 +227,7 @@ func TestTestRunsAreAskedOncePerLevelAndMoveEveryReader(t *testing.T) {
runCtx, stop := context.WithCancel(ctx)
done := make(chan error, 1)
go func() { done <- run(runCtx, func(k string) string { return env[k] }, logs) }()
takeInsHeard(t, ctx, logs)
return func() { stop(); <-done }
}
stop := start()
@@ -57,7 +57,9 @@ type Ager struct {
Runs TestRuns
RunOutputs func(name string) error
RunsKept time.Duration
Now func() time.Time
// Clock is test-runs' bound and ask keys: a run's ask keys are deleted with the run (H-A-500 addendum (1)); nil keeps them.
Clock RunClock
Now func() time.Time
// Logf is the runner's journal: each delete writes a line naming the key or object and its retention.
Logf func(string, ...any)
}
@@ -159,6 +161,7 @@ func (a *Ager) ageRun(r Retention, e seat.LogEntry) {
return
}
a.logf("aged: deleted the test run %s and its output (%s after its end)", run, r.Age)
unmarkAsks(a.Clock, run, a.logf)
return
}
}
@@ -96,6 +96,7 @@ func (h *Hearer) putDeclined(key, old string, s seat.StepResult, a Asked, ev Ste
} else if err != nil {
return fmt.Errorf("the test run %s is not written: %w", a.Run, err)
}
markAsk(h.Clock, a.ID, a.Run, h.logf) // H-A-500 addendum (1)
break
}
}
@@ -4,6 +4,7 @@ import (
"errors"
"fmt"
"strings"
"sync/atomic"
"time"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
@@ -54,6 +55,81 @@ type RunClock interface {
Has(key string) (bool, error)
// Keys are test-runs' keys.
Keys() ([]string, error)
// MarkAsk writes the ask key of ask (AskKey) holding run, create-only: one written already is kept (H-A-500 addendum (1)).
MarkAsk(ask, run string) error
// RunOfAsk reads the ask key of ask: the key of the run it holds; ok false when it is not written.
RunOfAsk(ask string) (run string, ok bool, err error)
}
// AskPrefix begins the ask key of a test run's ask in test-runs, `ask.<ask>`, holding the run's key: written with each ask
// a run is given and deleted with the run, so an ask-taken or step-progress finds its run in one read and never lists the
// pipelines (design 47 H2 H-A-500 addendum (1)).
const AskPrefix = "ask."
// AskKey is the ask key of ask.
func AskKey(ask string) string { return AskPrefix + ask }
// IsRunAsk says whether a key of test-runs is an ask key rather than a run.
func IsRunAsk(key string) bool { return strings.HasPrefix(key, AskPrefix) }
// markAsk writes the ask key of a run's ask; one not written is said, and the ask's events are then found by the pipelines.
func markAsk(clock RunClock, ask, run string, logf func(string, ...any)) {
if clock == nil || ask == "" || run == "" {
return
}
if err := clock.MarkAsk(ask, run); err != nil && logf != nil {
logf("the ask key of ask %s of the test run %s is not written, so its events are found by the pipelines: %v", ask, run, err)
}
}
// unmarkAsks deletes, with the run under run, every ask key holding it (H-A-500 addendum (1)): the ask it names now and each
// it was given before (a rerun, a declined step asked again). A key not read or not deleted is said. The Hearer calls it
// when a run's last reader leaves, the Ager when a run is aged, so no ask key outlives its run.
func unmarkAsks(clock RunClock, run string, logf func(string, ...any)) {
if clock == nil {
return
}
all, err := clock.Keys()
if err != nil {
logf("the ask keys of the deleted test run %s are not listed, so they are kept: %v", run, err)
return
}
for _, k := range all {
if !IsRunAsk(k) {
continue
}
if held, ok, err := clock.RunOfAsk(strings.TrimPrefix(k, AskPrefix)); err != nil || !ok || held != run {
continue
}
if err := clock.Unbound(k); err != nil {
logf("the ask key %s of the deleted test run %s is not deleted: %v", k, run, err)
}
}
}
// LateEveryLine is how many late events of ended runs' asks one journal line says (H-A-500 (a)).
const LateEveryLine = 100
// LateRunAsk says whether ask is the ask of a test run that has ended, or whose run was asked again under another ask
// since, read from its ask key and its run, never from the pipelines (design 47 H2 H-A-500 (a) and its addenda): such an
// event is acknowledged at once and moves nothing, which replaces H-A-311 (2)'s placing and H-A-238 (2)'s clearing for
// it. One journal line says every LateEveryLine of them. An ask with no ask key, or whose run is gone, is not judged here.
func (h *Hearer) LateRunAsk(event, ask string) bool {
if h.TestRuns == nil || h.Clock == nil || ask == "" {
return false
}
run, ok, err := h.Clock.RunOfAsk(ask)
if err != nil || !ok {
return false
}
r, _, err := h.TestRuns.Get(run)
if err != nil || (r.Ended == nil && r.Ask == ask) {
return false
}
if n := atomic.AddInt64(&h.lateRuns, 1); n%LateEveryLine == 0 {
h.logf("%d late events of ended test runs' asks acknowledged at once, moving nothing; the last the %s of ask %s of the test run %s (H-A-500)", n, event, ask, run)
}
return true
}
// ParseRunBound reads a key of test-runs as a run's bound: the run's key and whether it is the untaken bound.
@@ -84,6 +160,13 @@ func (h *Hearer) runAndPipelineOfAsk(ask string) (string, string, bool) {
if h.TestRuns == nil || ask == "" {
return "", "", false
}
// The ask key names the run in one read (H-A-500 addendum (1)); the pipeline is then left unnamed, the run's readers
// naming every pipeline that reads it, the asking one among them.
if h.Clock != nil {
if run, ok, err := h.Clock.RunOfAsk(ask); err == nil && ok {
return run, "", true
}
}
pkeys, err := allKeys(h.Store)
if err != nil {
return "", "", false
@@ -254,7 +337,7 @@ func (h *Hearer) TakeUp() (int, error) {
ended := 0
var first error
for _, run := range all {
if IsRunBound(run) {
if IsRunBound(run) || IsRunAsk(run) {
continue
}
r, _, err := h.TestRuns.Get(run)
@@ -194,6 +194,9 @@ type Hearer struct {
// queue (Starter.PutAgain); with either nil a declined step ends could not run.
AskDeclined func(p seat.Pipeline, s seat.StepResult, now time.Time) (*Asked, error)
PutAgain func(key string, a Asked) error
// lateRuns counts the late events of ended runs' asks acknowledged at once (LateRunAsk, H-A-500 (a)).
lateRuns int64
}
// EndReaders ends each of readers' steps of run with r's result (§725, §726): a step that read an ended run, or the steps
@@ -250,6 +250,8 @@ func (s *Starter) started(a Asked, write func() error) error {
if err := write(); err != nil {
return err
}
// The run's ask gets its ask key, so its events find the run in one read (H-A-500 addendum (1)).
markAsk(s.Clock, a.ID, a.Run, s.logf)
if s.Clock != nil {
untaken := s.Runs.Untaken
if untaken <= 0 {
@@ -38,7 +38,7 @@ func (r RunsRead) TestRuns(in seat.TestRunsInput) (seat.TestRunsAnswer, error) {
sort.Strings(all)
a := seat.TestRunsAnswer{Runs: []seat.TestRun{}}
for _, key := range all {
if IsRunBound(key) || (in.Candidate != "" && !strings.HasPrefix(key, in.Candidate+".")) {
if IsRunBound(key) || IsRunAsk(key) || (in.Candidate != "" && !strings.HasPrefix(key, in.Candidate+".")) {
continue
}
raw, err := r.Runs.Value(key)
@@ -165,6 +165,7 @@ func (v Verbs) TestRunsDrop(caller string, in seat.TestRunsDropInput) (seat.Test
for _, k := range []string{BoundPrefix + in.Key, UntakenPrefix + in.Key} {
_ = h.Clock.Unbound(k)
}
unmarkAsks(h.Clock, in.Key, h.logf)
}
gone := r.Answer()
gone.Key = in.Key
@@ -20,6 +20,8 @@ type clockMem struct {
asks map[string]string
revs map[string]uint64
rev uint64
// askRuns are the ask keys, each holding its run (H-A-500 addendum (1)).
askRuns map[string]string
// before is called once at the next Bound, a write that comes between its caller's read and its write.
before func()
}
@@ -40,7 +42,11 @@ func (c *clockMem) Bounded(key string) (string, uint64, bool, error) {
_, ok := c.bound[key]
return c.asks[key], c.revs[key], ok, nil
}
func (c *clockMem) Unbound(key string) error { delete(c.bound, key); return nil }
func (c *clockMem) Unbound(key string) error {
delete(c.bound, key)
delete(c.askRuns, key)
return nil
}
func (c *clockMem) UnboundAt(key string, rev uint64) error {
if c.revs[key] == rev {
delete(c.bound, key)
@@ -48,6 +54,20 @@ func (c *clockMem) UnboundAt(key string, rev uint64) error {
return nil
}
func (c *clockMem) Has(key string) (bool, error) { _, ok := c.bound[key]; return ok, nil }
func (c *clockMem) MarkAsk(ask, run string) error {
if _, ok := c.askRuns[AskKey(ask)]; ok {
return nil
}
if c.askRuns == nil {
c.askRuns = map[string]string{}
}
c.askRuns[AskKey(ask)] = run
return nil
}
func (c *clockMem) RunOfAsk(ask string) (string, bool, error) {
run, ok := c.askRuns[AskKey(ask)]
return run, ok, nil
}
func (c *clockMem) Keys() ([]string, error) {
var out []string
for k := range c.runs {
@@ -56,6 +76,9 @@ func (c *clockMem) Keys() ([]string, error) {
for k := range c.bound {
out = append(out, k)
}
for k := range c.askRuns {
out = append(out, k)
}
return out, nil
}
@@ -245,7 +268,7 @@ func TestATestRunHasOneClockACapAndAnAge(t *testing.T) {
f.objects["out-1"] = []byte("ok")
now := agedNow
a := &Ager{Runs: f.runs, RunOutputs: f.objects.Delete, RunsKept: 7 * 24 * time.Hour, Pipelines: agedDeleter{f.store},
Aged: agedState{}, Now: func() time.Time { return now },
Clock: f.clock, Aged: agedState{}, Now: func() time.Time { return now },
Log: func(from uint64, limit int) ([]LoggedEntry, error) {
if from > 1 {
return nil, nil
@@ -257,11 +280,18 @@ func TestATestRunHasOneClockACapAndAnAge(t *testing.T) {
if _, _, err := f.runs.Get(run); err != nil {
t.Fatal("a run is aged before test-runs-kept passed")
}
if held, ok, _ := f.clock.RunOfAsk(unit.Ask); !ok || held != run {
t.Fatalf("the ask key of %s holds %q before the run is aged, want %s", unit.Ask, held, run)
}
now = now.Add(2 * 24 * time.Hour)
a.Age()
if _, _, err := f.runs.Get(run); err != ErrNoRun || len(f.objects) != 0 {
t.Errorf("the run past test-runs-kept is %v, objects %v; want both deleted", err, f.objects)
}
// The ask key is deleted with the aged run (H-A-500 addendum (1)), as it is when the run's last reader leaves.
if _, ok, _ := f.clock.RunOfAsk(unit.Ask); ok {
t.Errorf("the ask key of %s outlives its aged run", unit.Ask)
}
})
}
@@ -631,6 +661,44 @@ func TestNoWriteOfATestRunActsOnAReadBeforeItsLatestChange(t *testing.T) {
t.Errorf("the late step-progress placed in %v, want %s's waiting movers placed", released, p.Pipeline)
}
})
// H-A-500 (a) and its addenda: a late ask-taken or step-progress of an ended run's ask is judged from its ask key and
// its run alone, never the pipelines, one journal line per hundred; the ask key is deleted with the run.
t.Run("a late event of an ended run's ask reads no pipeline", func(t *testing.T) {
f := newRunFixture()
p := f.start(t, 4, 9)
unit := stepNamed(p, "app:program", "unit")
run := unit.Shared[0].Run
if got, ok, _ := f.clock.RunOfAsk(unit.Ask); !ok || got != run {
t.Fatalf("the ask key of %s holds %q, %v; want the run %s", unit.Ask, got, ok, run)
}
if f.h.LateRunAsk("step-progress", unit.Ask) {
t.Fatal("the ask of a running run is judged late")
}
if err := f.h.EndRun(run, seat.TestRun{Ask: unit.Ask, Outcome: seat.StepResultOutcomePassed}, nil); err != nil {
t.Fatal(err)
}
store := f.h.Store
f.h.Store, f.logged = nil, nil // a read of the pipelines would panic
for i := 0; i < 2*LateEveryLine; i++ {
if !f.h.LateRunAsk("ask-taken", unit.Ask) {
t.Fatalf("event %d of the ended run's ask is not judged late", i)
}
}
if f.h.LateRunAsk("ask-taken", "anaskofnorun") {
t.Error("an ask with no ask key is judged late")
}
if len(f.logged) != 2 || !strings.Contains(f.logged[1], "200 late events") {
t.Errorf("journal %q; want one line per hundred late events", f.logged)
}
f.h.Store = store
v := Verbs{Hearer: f.h, Starter: f.s, Acts: &actLog{}}
if _, err := v.TestRunsDrop("op@shanks", seat.TestRunsDropInput{Key: run, Why: "gone"}); err != nil {
t.Fatal(err)
}
if _, ok, _ := f.clock.RunOfAsk(unit.Ask); ok {
t.Error("the ask key is kept after its run is deleted")
}
})
// A run's end is written as its step-ended is heard, before the finding the step says (A-470o).
t.Run("the run ends before the step's finding is written", func(t *testing.T) {
f := newRunFixture()
@@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"slices"
"time"
"github.com/nats-io/nats.go/jetstream"
@@ -107,6 +108,30 @@ func (c creating) RaiseMaxBytes(stream string, to int64) (int64, bool, error) {
return from, true, nil
}
// AllowMsgTTL turns on a held stream's per-message time to live, its delete markers living markerTTL, when it is off, and
// changes nothing else (design 47 H2 H-A-501, corrected): it reads the stream's configuration and updates with the rest as
// it was. A stream with it on already is left as it is.
func (c creating) AllowMsgTTL(stream string, markerTTL time.Duration) (bool, error) {
ctx, cancel := context.WithTimeout(c.ctx, requestWithin)
defer cancel()
s, err := c.js.Stream(ctx, stream)
if err != nil {
return false, err
}
cfg := s.CachedInfo().Config
if cfg.AllowMsgTTL {
return false, nil
}
cfg.AllowMsgTTL = true
if cfg.SubjectDeleteMarkerTTL == 0 {
cfg.SubjectDeleteMarkerTTL = markerTTL
}
if _, err := c.js.UpdateStream(ctx, cfg); err != nil {
return false, err
}
return true, nil
}
// SourceFilter brings the filter subject of a held stream's source from `source` to `filter` and changes nothing else
// (design 47 H H-A-145): it reads the stream's configuration and updates with the rest as it was, so its messages stay. A
// stream with no source from `source`, or one already at `filter`, is left as it is.
+18 -4
View File
@@ -202,6 +202,19 @@ func (b *fakeBus) AddSubjects(stream string, add []string) error {
// SourceFilter is the bus's update of a held stream's source filter: the fake holds no sources, so nothing changes.
func (b *fakeBus) SourceFilter(string, string, string) (string, bool, error) { return "", false, nil }
// AllowMsgTTL is the bus's turning on of a held stream's per-message time to live, every other setting kept.
func (b *fakeBus) AllowMsgTTL(stream string, _ time.Duration) (bool, error) {
b.mu.Lock()
defer b.mu.Unlock()
for _, s := range b.streams {
if s["name"] == stream && s["allow_msg_ttl"] != true {
s["allow_msg_ttl"] = true
return true, nil
}
}
return false, nil
}
// RaiseMaxBytes is the bus's raise of a held stream's size cap: only to a larger cap, an unbounded stream and every other
// setting kept.
func (b *fakeBus) RaiseMaxBytes(stream string, to int64) (int64, bool, error) {
@@ -719,10 +732,11 @@ func TestGrants(t *testing.T) {
if n := strings.Count(string(src), "CreateStream"); (path == createOnly && n != 1) || (path != createOnly && n != 0) {
t.Errorf("%s holds CreateStream %d times: only create.go creates, once", path, n)
}
// The three updates of a stream: adding subjects to a held one (the operator, 2026-11-05), raising its size cap
// (2026-11-09) and bringing a kept stream's source filter to its definition's (design 47 H H-A-145), all in create.go.
if n := strings.Count(string(src), "UpdateStream("); (path == createOnly && n != 3) || (path != createOnly && n != 0) {
t.Errorf("%s holds UpdateStream %d times: only create.go updates a stream, three times, to add subjects, to raise a size cap and to bring a kept stream's source filter", path, n)
// The four updates of a stream: adding subjects to a held one (the operator, 2026-11-05), raising its size cap
// (2026-11-09), bringing a kept stream's source filter to its definition's (design 47 H H-A-145) and turning on a
// key-ttl state's per-message time to live (design 47 H2 H-A-501, corrected), all in create.go.
if n := strings.Count(string(src), "UpdateStream("); (path == createOnly && n != 4) || (path != createOnly && n != 0) {
t.Errorf("%s holds UpdateStream %d times: only create.go updates a stream, four times, to add subjects, to raise a size cap, to bring a kept stream's source filter and to turn on per-message time to live", path, n)
}
// The one object bucket: the part's own, of the user list sealed to the bus node, in sealed.go (ADR 0547 clause 7).
if path == sealedOnly {
@@ -1546,7 +1546,9 @@ func theToolRunnerHearsEveryEvent(t *testing.T) {
}
}
for _, s := range p.Publish {
if strings.Contains(s, "LEDGER") || strings.HasPrefix(s, "$JS.ACK.") {
// A module's own takeins durable on the shared events stream is no restricted stream (H-A-500 (b)).
takeIns := strings.HasPrefix(s, "$JS.ACK."+EventsStream+".") && strings.HasSuffix(s, TakeInsSuffix+".>")
if strings.Contains(s, "LEDGER") || (strings.HasPrefix(s, "$JS.ACK.") && !takeIns) {
t.Errorf("the tool runner's runtime may %s on the restricted stream", s)
}
}
@@ -254,6 +254,19 @@ func heardConsumer(p Principal, h Heard) Consumer {
// under a name of its own, since only a work queue shares one (lead answer A-397; mesh-tools bus.ToolsConsumerOf).
const ToolsSuffix = "_tools"
// TakeInsSuffix ends the name of a module's own second durable on the shared events stream, `<node>_<module>_takeins`,
// which the module makes itself and hears take-in.taken on alone, so a take-in never waits behind its other events (design
// 47 H2 H-A-500 (b) and its addenda): every module principal is granted it, none special-cased.
const TakeInsSuffix = "_takeins"
// takeInsGrants are what a module publishes to make, ask about, pull from and acknowledge its takeins durable on the shared
// events stream (TakeInsSuffix): the create by its name alone, every filter in FilterSubjects.
func takeInsGrants(durable string) []string {
d := durable + TakeInsSuffix
return []string{"$JS.API.CONSUMER.CREATE." + EventsStream + "." + d, "$JS.API.CONSUMER.INFO." + EventsStream + "." + d,
"$JS.API.CONSUMER.MSG.NEXT." + EventsStream + "." + d, "$JS.ACK." + EventsStream + "." + d + ".>"}
}
// readerGrants are what a reader publishes to ask about, pull from and acknowledge a durable consumer on a stream: for the
// module's program's name and for its tool runner's twin (A-397); and the create and delete of the twin by its name alone,
// which the tool runner makes from the membership's spec and removes when the module leaves the node (design 47 H2, H-A-472
@@ -1159,6 +1159,8 @@ func permissionsFor(p Principal) (Permissions, error) {
// which asks for these permissions to build the consumer and would ask forever. A
// subject for a consumer that turns out not to exist grants nothing anybody can use.
pub = append(pub, readerGrants(consumerStream(p), consumerDurable(p))...)
// 2d. Its own takeins durable on the shared events stream, made and heard by the module (H-A-500 (b)).
pub = append(pub, takeInsGrants(consumerDurable(p))...)
// 3. Seats it holds: full participation. But mesh-broker's acting verbs, which nobody serves on the bus (busstate.go).
for _, s := range p.Holds {
@@ -1326,6 +1328,8 @@ func permissionsFor(p Principal) (Permissions, error) {
// And the module streams it hears, each through its own consumer there (novox/hq ADR 0363 §8) — never
// a restricted one, which only a module on an account of its own hears (ADR 0376 §1).
pub = append(pub, heardGrants(own, own.Hears)...)
// Its takeins durable on the shared events stream, as a module on an account of its own has (H-A-500 (b)).
pub = append(pub, takeInsGrants(consumerDurable(own))...)
if _, consumes := ConsumerFor(own); !consumes {
continue
}
@@ -38,7 +38,7 @@ accounts {
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "one.mesh-delivery-watchman", password: "$2a$11$wwwwwwwwwwwwwwwwwwwwww", permissions: {
publish: { allow: ["$JS.ACK.DEAD_LETTER_NOTICES.mesh-delivery-watchman.>", "$JS.ACK.EVENTS.one_mesh-delivery-watchman.>", "$JS.ACK.EVENTS.one_mesh-delivery-watchman_tools.>", "$JS.API.CONSUMER.CREATE.DEAD_LETTER_NOTICES.mesh-delivery-watchman", "$JS.API.CONSUMER.CREATE.EVENTS.one_mesh-delivery-watchman_tools", "$JS.API.CONSUMER.CREATE.KV_mesh-grants_memberships.*.$KV.mesh-grants_memberships.one.mesh-delivery-watchman", "$JS.API.CONSUMER.DELETE.EVENTS.one_mesh-delivery-watchman_tools", "$JS.API.CONSUMER.DELETE.KV_mesh-grants_memberships.>", "$JS.API.CONSUMER.INFO.DEAD_LETTER_NOTICES.controller", "$JS.API.CONSUMER.INFO.DEAD_LETTER_NOTICES.mesh-delivery-watchman", "$JS.API.CONSUMER.INFO.EVENTS.one_mesh-delivery-watchman", "$JS.API.CONSUMER.INFO.EVENTS.one_mesh-delivery-watchman_tools", "$JS.API.CONSUMER.MSG.NEXT.DEAD_LETTER_NOTICES.mesh-delivery-watchman", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_mesh-delivery-watchman", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_mesh-delivery-watchman_tools", "$JS.API.DIRECT.GET.KV_mesh-grants_memberships.$KV.mesh-grants_memberships.one.mesh-delivery-watchman", "$JS.API.DIRECT.GET.PROV_APPLIED.>", "$JS.API.STREAM.INFO.DEAD_LETTERS", "$JS.API.STREAM.INFO.DEAD_LETTER_NOTICES", "$JS.API.STREAM.INFO.KV_mesh-grants_memberships", "$JS.API.STREAM.INFO.PROV_APPLIED", "$JS.API.STREAM.MSG.GET.DEAD_LETTERS", "$JS.API.STREAM.MSG.GET.PROV_APPLIED", "$JS.API.STREAM.NAMES", "$JS.FC.KV_mesh-grants_memberships.>", "mesh.bus.connz", "mesh.events.dead.>", "mesh.seat.*.accept.>", "mesh.seat.*.call.*.*.one~mesh-delivery-watchman", "mesh.seat.*.call.*.one~mesh-delivery-watchman", "mesh.seat.*.tool.>"], deny: ["mesh.seat.node-launcher.tool.secret", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-launcher.call.secret.>", "mesh.seat.mesh-broker.call.apply.>", "mesh.seat.mesh-broker.call.create.>", "mesh.seat.mesh-broker.call.drop.>", "mesh.seat.mesh-broker.call.recreate.>", "mesh.seat.mesh-broker.tool.apply", "mesh.seat.mesh-broker.tool.create", "mesh.seat.mesh-broker.tool.drop", "mesh.seat.mesh-broker.tool.recreate"] }
publish: { allow: ["$JS.ACK.DEAD_LETTER_NOTICES.mesh-delivery-watchman.>", "$JS.ACK.EVENTS.one_mesh-delivery-watchman.>", "$JS.ACK.EVENTS.one_mesh-delivery-watchman_takeins.>", "$JS.ACK.EVENTS.one_mesh-delivery-watchman_tools.>", "$JS.API.CONSUMER.CREATE.DEAD_LETTER_NOTICES.mesh-delivery-watchman", "$JS.API.CONSUMER.CREATE.EVENTS.one_mesh-delivery-watchman_takeins", "$JS.API.CONSUMER.CREATE.EVENTS.one_mesh-delivery-watchman_tools", "$JS.API.CONSUMER.CREATE.KV_mesh-grants_memberships.*.$KV.mesh-grants_memberships.one.mesh-delivery-watchman", "$JS.API.CONSUMER.DELETE.EVENTS.one_mesh-delivery-watchman_tools", "$JS.API.CONSUMER.DELETE.KV_mesh-grants_memberships.>", "$JS.API.CONSUMER.INFO.DEAD_LETTER_NOTICES.controller", "$JS.API.CONSUMER.INFO.DEAD_LETTER_NOTICES.mesh-delivery-watchman", "$JS.API.CONSUMER.INFO.EVENTS.one_mesh-delivery-watchman", "$JS.API.CONSUMER.INFO.EVENTS.one_mesh-delivery-watchman_takeins", "$JS.API.CONSUMER.INFO.EVENTS.one_mesh-delivery-watchman_tools", "$JS.API.CONSUMER.MSG.NEXT.DEAD_LETTER_NOTICES.mesh-delivery-watchman", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_mesh-delivery-watchman", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_mesh-delivery-watchman_takeins", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_mesh-delivery-watchman_tools", "$JS.API.DIRECT.GET.KV_mesh-grants_memberships.$KV.mesh-grants_memberships.one.mesh-delivery-watchman", "$JS.API.DIRECT.GET.PROV_APPLIED.>", "$JS.API.STREAM.INFO.DEAD_LETTERS", "$JS.API.STREAM.INFO.DEAD_LETTER_NOTICES", "$JS.API.STREAM.INFO.KV_mesh-grants_memberships", "$JS.API.STREAM.INFO.PROV_APPLIED", "$JS.API.STREAM.MSG.GET.DEAD_LETTERS", "$JS.API.STREAM.MSG.GET.PROV_APPLIED", "$JS.API.STREAM.NAMES", "$JS.FC.KV_mesh-grants_memberships.>", "mesh.bus.connz", "mesh.events.dead.>", "mesh.seat.*.accept.>", "mesh.seat.*.call.*.*.one~mesh-delivery-watchman", "mesh.seat.*.call.*.one~mesh-delivery-watchman", "mesh.seat.*.tool.>"], deny: ["mesh.seat.node-launcher.tool.secret", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-launcher.call.secret.>", "mesh.seat.mesh-broker.call.apply.>", "mesh.seat.mesh-broker.call.create.>", "mesh.seat.mesh-broker.call.drop.>", "mesh.seat.mesh-broker.call.recreate.>", "mesh.seat.mesh-broker.tool.apply", "mesh.seat.mesh-broker.tool.create", "mesh.seat.mesh-broker.tool.drop", "mesh.seat.mesh-broker.tool.recreate"] }
subscribe: { allow: ["$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$SRV.INFO", "$SRV.INFO.mesh-delivery-watchman", "$SRV.INFO.mesh-delivery-watchman.>", "$SRV.PING", "$SRV.PING.mesh-delivery-watchman", "$SRV.PING.mesh-delivery-watchman.>", "$SRV.STATS", "$SRV.STATS.mesh-delivery-watchman", "$SRV.STATS.mesh-delivery-watchman.>", "_INBOX.one.mesh-delivery-watchman.>", "mesh.bus.connect", "mesh.bus.disconnect", "mesh.control.*.tools-alive", "mesh.events.dead.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.mesh-delivery-watchman.call.>", "mesh.mod.mesh-delivery-watchman.tool.>", "mesh.mod.nats.event.bus.connection-closed", "mesh.mod.nats.event.bus.slow-consumer", "mesh.mod.power.event.*", "mesh.seat.mesh-self-check.event.checked", "prov.applied.>"] }
allow_responses: { max: 1, ttl: "1m" }
} }
@@ -47,17 +47,17 @@ accounts {
subscribe: { allow: ["_INBOX.one.nats.>"] }
} }
{ user: "one.telegram", password: "$2a$11$tttttttttttttttttttttt", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.one_telegram.>", "$JS.ACK.EVENTS.one_telegram_tools.>", "$JS.ACK.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker.>", "$JS.API.CONSUMER.CREATE.EVENTS.one_telegram_tools", "$JS.API.CONSUMER.CREATE.KV_mesh-grants_memberships.*.$KV.mesh-grants_memberships.one.telegram", "$JS.API.CONSUMER.DELETE.EVENTS.one_telegram_tools", "$JS.API.CONSUMER.DELETE.KV_mesh-grants_memberships.>", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram_tools", "$JS.API.CONSUMER.INFO.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram_tools", "$JS.API.CONSUMER.MSG.NEXT.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.DIRECT.GET.KV_mesh-grants_memberships.$KV.mesh-grants_memberships.one.telegram", "$JS.API.STREAM.INFO.KV_mesh-grants_memberships", "$JS.API.STREAM.NAMES", "$JS.FC.KV_mesh-grants_memberships.>", "mesh.seat.*.accept.>", "mesh.seat.*.call.*.*.one~telegram", "mesh.seat.*.call.*.one~telegram", "mesh.seat.*.tool.>", "mesh.seat.telegram-sender.event.delivered", "mesh.seat.telegram-sender.event.failed"], deny: ["mesh.seat.node-launcher.tool.secret", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-launcher.call.secret.>", "mesh.seat.mesh-broker.call.apply.>", "mesh.seat.mesh-broker.call.create.>", "mesh.seat.mesh-broker.call.drop.>", "mesh.seat.mesh-broker.call.recreate.>", "mesh.seat.mesh-broker.tool.apply", "mesh.seat.mesh-broker.tool.create", "mesh.seat.mesh-broker.tool.drop", "mesh.seat.mesh-broker.tool.recreate"] }
publish: { allow: ["$JS.ACK.EVENTS.one_telegram.>", "$JS.ACK.EVENTS.one_telegram_takeins.>", "$JS.ACK.EVENTS.one_telegram_tools.>", "$JS.ACK.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker.>", "$JS.API.CONSUMER.CREATE.EVENTS.one_telegram_takeins", "$JS.API.CONSUMER.CREATE.EVENTS.one_telegram_tools", "$JS.API.CONSUMER.CREATE.KV_mesh-grants_memberships.*.$KV.mesh-grants_memberships.one.telegram", "$JS.API.CONSUMER.DELETE.EVENTS.one_telegram_tools", "$JS.API.CONSUMER.DELETE.KV_mesh-grants_memberships.>", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram_takeins", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram_tools", "$JS.API.CONSUMER.INFO.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram_takeins", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram_tools", "$JS.API.CONSUMER.MSG.NEXT.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.DIRECT.GET.KV_mesh-grants_memberships.$KV.mesh-grants_memberships.one.telegram", "$JS.API.STREAM.INFO.KV_mesh-grants_memberships", "$JS.API.STREAM.NAMES", "$JS.FC.KV_mesh-grants_memberships.>", "mesh.seat.*.accept.>", "mesh.seat.*.call.*.*.one~telegram", "mesh.seat.*.call.*.one~telegram", "mesh.seat.*.tool.>", "mesh.seat.telegram-sender.event.delivered", "mesh.seat.telegram-sender.event.failed"], deny: ["mesh.seat.node-launcher.tool.secret", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-launcher.call.secret.>", "mesh.seat.mesh-broker.call.apply.>", "mesh.seat.mesh-broker.call.create.>", "mesh.seat.mesh-broker.call.drop.>", "mesh.seat.mesh-broker.call.recreate.>", "mesh.seat.mesh-broker.tool.apply", "mesh.seat.mesh-broker.tool.create", "mesh.seat.mesh-broker.tool.drop", "mesh.seat.mesh-broker.tool.recreate"] }
subscribe: { allow: ["$JS.EVENT.ADVISORY.CONSUMER.CREATED.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$SRV.INFO", "$SRV.INFO.telegram", "$SRV.INFO.telegram.>", "$SRV.PING", "$SRV.PING.telegram", "$SRV.PING.telegram.>", "$SRV.STATS", "$SRV.STATS.telegram", "$SRV.STATS.telegram.>", "_INBOX.one.telegram.>", "mesh.mod.telegram.call.>", "mesh.mod.telegram.tool.>", "mesh.seat.telegram-sender.accept.send"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "two.audit", password: "$2a$11$aaaaaaaaaaaaaaaaaaaaaa", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.two_audit.>", "$JS.ACK.EVENTS.two_audit_tools.>", "$JS.API.CONSUMER.CREATE.EVENTS.two_audit_tools", "$JS.API.CONSUMER.CREATE.KV_mesh-grants_memberships.*.$KV.mesh-grants_memberships.two.audit", "$JS.API.CONSUMER.DELETE.EVENTS.two_audit_tools", "$JS.API.CONSUMER.DELETE.KV_mesh-grants_memberships.>", "$JS.API.CONSUMER.INFO.EVENTS.two_audit", "$JS.API.CONSUMER.INFO.EVENTS.two_audit_tools", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_audit", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_audit_tools", "$JS.API.DIRECT.GET.KV_mesh-grants_memberships.$KV.mesh-grants_memberships.two.audit", "$JS.API.STREAM.INFO.KV_mesh-grants_memberships", "$JS.API.STREAM.NAMES", "$JS.FC.KV_mesh-grants_memberships.>", "mesh.seat.*.accept.>", "mesh.seat.*.call.*.*.two~audit", "mesh.seat.*.call.*.two~audit", "mesh.seat.*.tool.>"], deny: ["mesh.seat.node-launcher.tool.secret", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-launcher.call.secret.>", "mesh.seat.mesh-broker.call.apply.>", "mesh.seat.mesh-broker.call.create.>", "mesh.seat.mesh-broker.call.drop.>", "mesh.seat.mesh-broker.call.recreate.>", "mesh.seat.mesh-broker.tool.apply", "mesh.seat.mesh-broker.tool.create", "mesh.seat.mesh-broker.tool.drop", "mesh.seat.mesh-broker.tool.recreate"] }
publish: { allow: ["$JS.ACK.EVENTS.two_audit.>", "$JS.ACK.EVENTS.two_audit_takeins.>", "$JS.ACK.EVENTS.two_audit_tools.>", "$JS.API.CONSUMER.CREATE.EVENTS.two_audit_takeins", "$JS.API.CONSUMER.CREATE.EVENTS.two_audit_tools", "$JS.API.CONSUMER.CREATE.KV_mesh-grants_memberships.*.$KV.mesh-grants_memberships.two.audit", "$JS.API.CONSUMER.DELETE.EVENTS.two_audit_tools", "$JS.API.CONSUMER.DELETE.KV_mesh-grants_memberships.>", "$JS.API.CONSUMER.INFO.EVENTS.two_audit", "$JS.API.CONSUMER.INFO.EVENTS.two_audit_takeins", "$JS.API.CONSUMER.INFO.EVENTS.two_audit_tools", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_audit", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_audit_takeins", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_audit_tools", "$JS.API.DIRECT.GET.KV_mesh-grants_memberships.$KV.mesh-grants_memberships.two.audit", "$JS.API.STREAM.INFO.KV_mesh-grants_memberships", "$JS.API.STREAM.NAMES", "$JS.FC.KV_mesh-grants_memberships.>", "mesh.seat.*.accept.>", "mesh.seat.*.call.*.*.two~audit", "mesh.seat.*.call.*.two~audit", "mesh.seat.*.tool.>"], deny: ["mesh.seat.node-launcher.tool.secret", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-launcher.call.secret.>", "mesh.seat.mesh-broker.call.apply.>", "mesh.seat.mesh-broker.call.create.>", "mesh.seat.mesh-broker.call.drop.>", "mesh.seat.mesh-broker.call.recreate.>", "mesh.seat.mesh-broker.tool.apply", "mesh.seat.mesh-broker.tool.create", "mesh.seat.mesh-broker.tool.drop", "mesh.seat.mesh-broker.tool.recreate"] }
subscribe: { allow: ["$SRV.INFO", "$SRV.INFO.audit", "$SRV.INFO.audit.>", "$SRV.PING", "$SRV.PING.audit", "$SRV.PING.audit.>", "$SRV.STATS", "$SRV.STATS.audit", "$SRV.STATS.audit.>", "_INBOX.two.audit.>", "mesh.mod.audit.call.>", "mesh.mod.audit.tool.>", "mesh.mod.shop.event.order.placed"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "two.shop", password: "$2a$11$ssssssssssssssssssssss", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.two_shop.>", "$JS.ACK.EVENTS.two_shop_tools.>", "$JS.API.CONSUMER.CREATE.EVENTS.two_shop_tools", "$JS.API.CONSUMER.CREATE.KV_mesh-grants_memberships.*.$KV.mesh-grants_memberships.two.shop", "$JS.API.CONSUMER.DELETE.EVENTS.two_shop_tools", "$JS.API.CONSUMER.DELETE.KV_mesh-grants_memberships.>", "$JS.API.CONSUMER.INFO.EVENTS.two_shop", "$JS.API.CONSUMER.INFO.EVENTS.two_shop_tools", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_shop", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_shop_tools", "$JS.API.DIRECT.GET.KV_mesh-grants_memberships.$KV.mesh-grants_memberships.two.shop", "$JS.API.STREAM.INFO.KV_mesh-grants_memberships", "$JS.API.STREAM.NAMES", "$JS.FC.KV_mesh-grants_memberships.>", "mesh.mod.shop.event.order.placed", "mesh.seat.*.accept.>", "mesh.seat.*.call.*.*.two~shop", "mesh.seat.*.call.*.two~shop", "mesh.seat.*.tool.>", "mesh.seat.telegram-sender.accept.send"], deny: ["mesh.seat.node-launcher.tool.secret", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-launcher.call.secret.>", "mesh.seat.mesh-broker.call.apply.>", "mesh.seat.mesh-broker.call.create.>", "mesh.seat.mesh-broker.call.drop.>", "mesh.seat.mesh-broker.call.recreate.>", "mesh.seat.mesh-broker.tool.apply", "mesh.seat.mesh-broker.tool.create", "mesh.seat.mesh-broker.tool.drop", "mesh.seat.mesh-broker.tool.recreate"] }
publish: { allow: ["$JS.ACK.EVENTS.two_shop.>", "$JS.ACK.EVENTS.two_shop_takeins.>", "$JS.ACK.EVENTS.two_shop_tools.>", "$JS.API.CONSUMER.CREATE.EVENTS.two_shop_takeins", "$JS.API.CONSUMER.CREATE.EVENTS.two_shop_tools", "$JS.API.CONSUMER.CREATE.KV_mesh-grants_memberships.*.$KV.mesh-grants_memberships.two.shop", "$JS.API.CONSUMER.DELETE.EVENTS.two_shop_tools", "$JS.API.CONSUMER.DELETE.KV_mesh-grants_memberships.>", "$JS.API.CONSUMER.INFO.EVENTS.two_shop", "$JS.API.CONSUMER.INFO.EVENTS.two_shop_takeins", "$JS.API.CONSUMER.INFO.EVENTS.two_shop_tools", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_shop", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_shop_takeins", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_shop_tools", "$JS.API.DIRECT.GET.KV_mesh-grants_memberships.$KV.mesh-grants_memberships.two.shop", "$JS.API.STREAM.INFO.KV_mesh-grants_memberships", "$JS.API.STREAM.NAMES", "$JS.FC.KV_mesh-grants_memberships.>", "mesh.mod.shop.event.order.placed", "mesh.seat.*.accept.>", "mesh.seat.*.call.*.*.two~shop", "mesh.seat.*.call.*.two~shop", "mesh.seat.*.tool.>", "mesh.seat.telegram-sender.accept.send"], deny: ["mesh.seat.node-launcher.tool.secret", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-launcher.call.secret.>", "mesh.seat.mesh-broker.call.apply.>", "mesh.seat.mesh-broker.call.create.>", "mesh.seat.mesh-broker.call.drop.>", "mesh.seat.mesh-broker.call.recreate.>", "mesh.seat.mesh-broker.tool.apply", "mesh.seat.mesh-broker.tool.create", "mesh.seat.mesh-broker.tool.drop", "mesh.seat.mesh-broker.tool.recreate"] }
subscribe: { allow: ["$SRV.INFO", "$SRV.INFO.shop", "$SRV.INFO.shop.>", "$SRV.PING", "$SRV.PING.shop", "$SRV.PING.shop.>", "$SRV.STATS", "$SRV.STATS.shop", "$SRV.STATS.shop.>", "_INBOX.two.shop.>", "mesh.mod.shop.call.>", "mesh.mod.shop.tool.>"] }
allow_responses: { max: 1, ttl: "1m" }
} }
File diff suppressed because one or more lines are too long
@@ -75,3 +75,31 @@ func TestTheRunnersTwinIsTheMembershipsSpec(t *testing.T) {
}
}
}
// Design 47 H2, H-A-500 (b) and its addenda: every module principal, on an account of its own or carried by a tool runner,
// is granted the create, information, pull and acknowledgement of its own takeins durable on the shared events stream.
func TestEveryModuleIsGrantedItsTakeInsDurable(t *testing.T) {
takeIns := func(node, module string) []string {
d := node + "_" + module + TakeInsSuffix
return []string{"$JS.API.CONSUMER.CREATE.EVENTS." + d, "$JS.API.CONSUMER.INFO.EVENTS." + d,
"$JS.API.CONSUMER.MSG.NEXT.EVENTS." + d, "$JS.ACK.EVENTS." + d + ".>"}
}
for _, c := range []struct {
name string
p Principal
module string
}{
{"a module's own account", Principal{Kind: KindModule, Node: "a", Module: "shop"}, "shop"},
{"a module a tool runner carries", Principal{Kind: KindNodeTools, Node: "a", Module: "node-tools", Carries: []Declared{{Module: "shop"}}}, "shop"},
} {
perms, err := PermissionsFor(c.p)
if err != nil {
t.Fatalf("%s: %v", c.name, err)
}
for _, g := range takeIns("a", c.module) {
if !slices.Contains(perms.Publish, g) {
t.Errorf("%s is not granted %s", c.name, g)
}
}
}
}
@@ -30,6 +30,7 @@ func (c *askCreator) RaiseMaxBytes(string, int64) (int64, bool, error) {
func (c *askCreator) SourceFilter(string, string, string) (string, bool, error) {
return "", false, nil
}
func (c *askCreator) AllowMsgTTL(string, time.Duration) (bool, error) { return false, nil }
// A seat whose queues came with G (mesh-builds) gets its work-queue ask stream from the seat list, as the controller made one for
// every seat before G; a seat that accepts nothing gets none, and a stream the bus holds is never made again (lead-47, 2026-11-15).
@@ -8,6 +8,7 @@ import (
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
@@ -74,6 +75,10 @@ type Creator interface {
// messages and every other setting, the stream not made again (design 47 H H-A-145). It answers the filter it held and
// whether it changed; a stream with no source from `source` is left as it is.
SourceFilter(stream, source, filter string) (from string, changed bool, err error)
// AllowMsgTTL turns on per-message time to live on a stream the bus holds, its delete markers living markerTTL, when it
// is off, and changes nothing else (design 47 H2 H-A-501, corrected): a state declared key-ttl whose bucket was made
// before the declaration. It answers whether it turned it on; one on already is left as it is. It never turns it off.
AllowMsgTTL(stream string, markerTTL time.Duration) (turned bool, err error)
}
// Raise is one held stream's size cap raised to the manifest's, said in the event and the journal.
@@ -126,6 +131,24 @@ type creations struct {
// refused is, by stream, the refusal of the last addition of subjects (an overlap with another stream, say), so a
// refusal that stands is said once and not at every pass; a success or a different refusal says again.
refused map[string]string
// ttl is, by stream, the manifest digest at which its per-message time to live was last compared with the manifest's
// key-ttl, so the bus is asked once per manifest (a hand assert asks again).
ttl map[string]string
}
func (c *creations) ttlAt(stream string) string {
c.mu.Lock()
defer c.mu.Unlock()
return c.ttl[stream]
}
func (c *creations) keepTTL(stream, digest string) {
c.mu.Lock()
defer c.mu.Unlock()
if c.ttl == nil {
c.ttl = map[string]string{}
}
c.ttl[stream] = digest
}
// refusedAgain says whether this refusal of an addition to the stream was the last one said, and keeps it.
@@ -265,6 +288,26 @@ func createMissing(c Creator, done *creations, a *Assertion, m catalogue.Manifes
made = append(made, fmt.Sprintf("%s %s size cap raised from %d to %d bytes", it.Kind, it.Stream, from, it.Composed.MaxBytes))
}
}
// A held bucket of a state declared key-ttl gets per-message time to live when it lacks it, add-only like the cap: a
// bucket made before its state was declared key-ttl refuses every key written with a time to live (design 47 H2
// H-A-501, corrected). Compared once per manifest.
for i := range a.Items {
it := &a.Items[i]
cfg, ok := cfgs[it.Stream]
if it.Held == nil || it.Kind != "bucket" || !ok || !cfg.AllowMsgTTL || (!force && done.ttlAt(it.Stream) == a.Manifest) {
continue
}
turned, err := c.AllowMsgTTL(it.Stream, cfg.SubjectDeleteMarkerTTL)
if err != nil {
it.Created = ItemNotCreated + "per-message time to live not turned on: " + err.Error() + " (tried again at the next event)"
continue
}
done.keepTTL(it.Stream, a.Manifest)
if turned {
it.Created = "per-message time to live turned on (its state is declared key-ttl)"
made = append(made, it.Kind+" "+it.Stream+" per-message time to live turned on, its state declared key-ttl")
}
}
if len(made) > 0 {
a.Asserted = true
}
@@ -1,11 +1,15 @@
package grants
import (
"fmt"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-grants/internal/buscompose"
)
// raiser is a Creator that holds one stream at a cap and records the cap it is raised to.
@@ -41,6 +45,40 @@ func TestAStatesDeclaredCapReachesItsBucket(t *testing.T) {
}
}
// ttlTurner is a Creator that records each held stream whose per-message time to live it turns on.
type ttlTurner struct {
Creator
turned []string
}
func (r *ttlTurner) AllowMsgTTL(stream string, marker time.Duration) (bool, error) {
r.turned = append(r.turned, fmt.Sprintf("%s %s", stream, marker))
return true, nil
}
func (r *ttlTurner) RaiseMaxBytes(string, int64) (int64, bool, error) { return 0, false, nil }
// A held bucket of a state declared key-ttl gets per-message time to live, once per manifest, and is said (design 47 H2
// H-A-501, corrected); a bucket of a state with no key-ttl is not touched.
func TestAHeldKeyTTLStateGetsPerMessageTTL(t *testing.T) {
m := catalogue.Manifest{Module: "mesh-delivery-indexer", State: []catalogue.StateDeclaration{{Name: "heard", KeyTTL: true}, {Name: "plain"}}}
a := Assertion{Manifest: "d", Items: Composed(m, nil)}
for i := range a.Items {
a.Items[i].Held = &Limits{}
}
r, done := &ttlTurner{}, &creations{}
made, _ := createMissing(r, done, &a, m, nil, false)
if want := []string{"KV_mesh-delivery-indexer_heard " + buscompose.KeyTTLMarkerTTL.String()}; !slices.Equal(r.turned, want) {
t.Fatalf("turned on %v, want %v", r.turned, want)
}
if !a.Asserted || !slices.ContainsFunc(made, func(s string) bool { return strings.Contains(s, "per-message time to live turned on") }) {
t.Errorf("asserted %v, said %v; want the turning on said", a.Asserted, made)
}
if createMissing(r, done, &a, m, nil, false); len(r.turned) != 1 {
t.Errorf("asked again at the same manifest: %v", r.turned)
}
}
// adder is a Creator that records the subjects it is asked to add to a held stream.
type adder struct {
Creator