Land feature 57's controller work on its release branch

The controller plans config, templates and moved providers' provisions,
hands bus accounts to the vault and judges the new manifest words, on the
feature's SDK and node-engine validator (feature 57, config and provisioning).
This commit is contained in:
2026-10-15 21:27:33 +00:00
parent fcdecb4361
commit 97a2d841c7
349 changed files with 42440 additions and 786 deletions
+9 -3
View File
@@ -24,9 +24,10 @@ import (
// which consumer of whose, which subject — for the watchdog to say as a condition, as the refused
// reply of issue 265 is said today.
//
// What the bus says about **other** principals' connections — a module's slow consumer, a module
// refused a subject — the server publishes only to a system account, which the mesh's bus does not
// have; that half is not heard yet (to-be 45 S9, recorded as deferred).
// What the bus does to **other** principals' connections the server does not publish in the mesh's
// account. A module's slow consumer it closes and records, and the bus module says that record on the
// bus (busfaults.go, novox/hq ADR 0357). A module refused a subject is told and written in the server's
// log, and nowhere else: not heard yet (novox/hq issue 702).
// The advisory kinds, as conditions name them.
const (
@@ -48,6 +49,11 @@ type Advisory struct {
Stream, Consumer string
// Token is the condition key's last part, when it is not Kind.
Token string
// User is the bus user whose connection it is about, when it is about one (novox/hq ADR 0357): the
// controller's own, or another user's the bus module said.
User string
// Reason is the server's reason for closing that connection, as it gave it.
Reason string
// Said is the newest saying, in the mesh's words.
Said string
First, Last time.Time
+50 -2
View File
@@ -114,11 +114,23 @@ type ReuseAsk struct {
// CheckedBuild is one build a check's verdict names: the pull request it checked, its ask, and the build.
type CheckedBuild struct {
Pull PullRef `json:"pull"`
Check string `json:"check"`
Pull PullRef `json:"pull"`
Check string `json:"check"`
// Head is the pull request's head the check judged, as its verdict says it (twelve characters): a build is
// taken or kept only by a check of the same head (novox/hq issue 703).
Head string `json:"head,omitempty"`
Built CheckBuilt `json:"built"`
}
// SameHead says two spellings of a commit name the same one: both said, one the other's beginning, at least
// twelve characters of it (a verdict says twelve).
func SameHead(a, b string) bool {
if len(a) > len(b) {
a, b = b, a
}
return len(a) >= 12 && strings.HasPrefix(b, a)
}
// PullRef names a pull request.
type PullRef struct {
Owner string `json:"owner"`
@@ -155,6 +167,12 @@ type CheckRequest struct {
// the controller, judged by itself; JudgeValidator for a change to the node-engine, judged by the
// running controller built with the change's validator in place of the one it vendors.
Judge string `json:"judge,omitempty"`
// Release is the release branch the pull request merges into (novox/hq ADR 0358): its base, when that is not its
// repository's trunk and mesh-controller has a branch of that name. The gate is then judged by the controller
// built from mesh-controller's release branch — a change to the controller still by itself, a change to the
// node-engine with its own validator — and each repository beside it is read at its branch of that name where it
// has one (CheckedOut.Else otherwise). Empty for every other base, checked as before it.
Release string `json:"release,omitempty"`
// Group names a delivery group whose members' heads are composed together with this one (novox/hq ADR
// 0239), and Members are those other heads, each cloned beside it and laid over the mesh in turn. The
// repository's own check is not run for a group: each member's pull request runs its own.
@@ -164,6 +182,16 @@ type CheckRequest struct {
// each, from the merge it judged, into the scratch namespace once it has judged and not failed, for the walk to
// take. A build seat from before it builds nothing, and the walk builds as before.
Builds []CheckBuild `json:"builds,omitempty"`
// Running is, keyed "<module>/<artifact>", each artifact of the build the mesh has registered for its module — the
// build its machines are sent (novox/hq issue 721): what a build for the merge is compared with, never the newest
// build, which may be one its gate failed and the mesh put back. A module absent is compared with nothing, and
// moves. A controller from before it sends none, and nothing is said unchanged.
Running map[string]string `json:"running,omitempty"`
// Named are the builds the seat's verdicts of earlier checks of this pull request at this head name, within the
// week check builds are kept (novox/hq issue 703): asked again because only the trunk moved, the check keeps a
// build kept under a module's unchanged key, instead of building it again, only when one of these names its key
// and every digest. Never another pull request's, nor an earlier head's. A build seat from before it builds.
Named []CheckedBuild `json:"named,omitempty"`
// Try marks a branch tried before its pull request (novox/hq issue 686): the repository's own check for a
// pushed branch or commit, asked by a builder through the controller's `try` verb. Its outcome is read from
// the build's log (StepTried) and said to nobody: no status on the forge, nothing recorded or registered,
@@ -212,6 +240,14 @@ type CheckBuilt struct {
Made []MadeArtifact `json:"made,omitempty"`
NotBuilt string `json:"not-built,omitempty"`
Took string `json:"took,omitempty"`
// Kept names the check whose build this check kept instead of building, its key unchanged (novox/hq issue 703):
// a walk takes it as that check's build. Empty when this check built it.
Kept string `json:"kept,omitempty"`
// Same says every artifact came out the same as the build the mesh holds of the module, and SameAs the held
// reference each was compared with (novox/hq issue 721): a walk cut while the mesh still holds those builds leaves
// the module out. Empty when it differs, or could not be compared.
Same string `json:"same,omitempty"`
SameAs map[string]string `json:"same-as,omitempty"`
}
// GroupMember is one other head of a delivery group's composed check.
@@ -237,6 +273,10 @@ const (
type CheckedOut struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
// Else is the ref read when the repository has no branch Ref (novox/hq ADR 0358 §3): beside a check of a pull
// request into a release branch, Ref is that branch and Else the repository's trunk. Empty: Ref is read, or the
// check cannot run.
Else string `json:"else,omitempty"`
}
// CheckOutcome is a merge check's verdict.
@@ -261,6 +301,14 @@ type CheckOutcome struct {
Members []CheckedMember `json:"members,omitempty"`
// Built are the builds it made for its merge (novox/hq ADR 0350 §1).
Built []CheckBuilt `json:"built,omitempty"`
// Release is the release branch the check was judged for (novox/hq ADR 0358 §4), JudgedBy the commit of the
// controller that judged its gate, as twelve characters, and BesideAt the branch each repository beside it was
// read at, by directory. Empty from a check of any other base, and from a build seat from before it.
Release string `json:"release,omitempty"`
JudgedBy string `json:"judged-by,omitempty"`
BesideAt map[string]string `json:"beside-at,omitempty"`
// Validator says which node-engine validator the judge of a release-branch check was built with (ADR 0358 §2).
Validator string `json:"validator,omitempty"`
}
// SourceOnSeat names a repository by the seat whose holder serves it and its path there.
+1 -1
View File
@@ -332,7 +332,7 @@ func (m *natsMachine) Take(ctx context.Context, do func(context.Context, Build))
// as the controller says it for its own long handlers, so the server neither hands
// the ask to a second machine nor counts the wait against its deliveries.
working := make(chan struct{})
go stillWorking(msg, working)
go stillWorking(msg, working, heartbeatWhileWorking)
do(ctx, build)
close(working)
}
+235
View File
@@ -0,0 +1,235 @@
package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
)
// What the bus did to other users' connections, as the bus module says it (novox/hq ADR 0357, issue 270).
//
// **The server says another user's slow consumer only in its record of closed connections**, which the
// bus module reads on its own machine and says as events: `bus.slow-consumer` for a connection the server
// closed as a slow consumer, `bus.connection-closed` for one it closed for breaking the bus's rules. They
// are read back from EVENTS on every watchdog tick for the hour S9 says an advisory, rather than taken by
// the controller's durable consumer: that one takes a message at a time while a merge builds for
// minutes, and read back, a controller that restarts says the last hour again without keeping anything.
//
// **Said at least once, counted once.** The module says the last hour again whenever it starts; a closure
// is its user, its connection id and when it opened, and is counted once however often it is said.
// BusModule is the module whose events say what the bus did to its users' connections.
const BusModule = "nats"
// The bus module's events, by their local names (its manifest's `emits`).
const (
BusSlowConsumerEvent = "bus.slow-consumer"
BusConnectionClosedEvent = "bus.connection-closed"
)
// AdvisoryConnectionClosed is a connection the bus closed for breaking its rules, as conditions name it.
const AdvisoryConnectionClosed = "connection-closed"
// BusClosure is one closed connection as the bus module says it: the user, never a credential.
type BusClosure struct {
User string `json:"user"`
Name string `json:"name,omitempty"`
CID uint64 `json:"cid"`
Reason string `json:"reason"`
Opened time.Time `json:"opened"`
Closed time.Time `json:"closed"`
Pending int64 `json:"pending_bytes,omitempty"`
}
func (c BusClosure) identity() string {
return fmt.Sprintf("%s/%d/%s", c.User, c.CID, c.Opened.UTC().Format(time.RFC3339Nano))
}
// BusClosures are the advisories the bus module's closures make, closed at or after since: one per user
// and kind, each closure counted once, the newest said.
func BusClosures(closures []BusClosure, kinds []string, since time.Time) []Advisory {
type entry struct {
a *Advisory
seen map[string]bool
newest BusClosure
}
by := map[string]*entry{}
var order []string
for i, c := range closures {
if c.User == "" || c.Closed.Before(since) {
continue
}
kind := kinds[i]
key := kind + "/" + c.User
e := by[key]
if e == nil {
e = &entry{a: &Advisory{Kind: kind, ID: c.User, User: c.User}, seen: map[string]bool{}}
by[key] = e
order = append(order, key)
}
if e.seen[c.identity()] {
continue
}
e.seen[c.identity()] = true
e.a.Count++
if e.a.First.IsZero() || c.Closed.Before(e.a.First) {
e.a.First = c.Closed
}
if !c.Closed.Before(e.a.Last) {
e.a.Last, e.newest = c.Closed, c
}
}
sort.Strings(order)
out := make([]Advisory, 0, len(order))
for _, key := range order {
e := by[key]
c := e.newest
name := ""
if c.Name != "" && c.Name != c.User {
name = fmt.Sprintf(" (connection %q)", c.Name)
}
pending := ""
if c.Pending > 0 {
pending = fmt.Sprintf(", %d bytes still to be written to it", c.Pending)
}
switch e.a.Kind {
case AdvisorySlowConsumer:
e.a.Said = fmt.Sprintf("the bus closed %s's connection %d%s as a slow consumer: %s%s", c.User, c.CID,
name, c.Reason, pending)
default:
e.a.Said = fmt.Sprintf("the bus closed %s's connection %d%s: %s", c.User, c.CID, name, c.Reason)
}
e.a.Reason = c.Reason
out = append(out, *e.a)
}
return out
}
// busEventKind is the advisory kind of one of the bus module's events, by its subject.
func busEventKind(subject string) (string, bool) {
switch subject {
case EventSubject(BusModule, BusSlowConsumerEvent):
return AdvisorySlowConsumer, true
case EventSubject(BusModule, BusConnectionClosedEvent):
return AdvisoryConnectionClosed, true
}
return "", false
}
// BusFaults reads back what the bus module said of its users' connections since a moment, from the events
// stream, on a consumer of its own that acknowledges nothing (as PowerStates reads), and says it as
// advisories. Nothing to read is no advisory, and costs two direct reads: the newest of each event.
func (s *Server) BusFaults(ctx context.Context, since time.Time) ([]Advisory, error) {
if s.js == nil {
return nil, errors.New("this control plane is not on the bus, so it cannot read what the bus module said")
}
js := s.js.Context()
recent := false
for _, event := range []string{BusSlowConsumerEvent, BusConnectionClosedEvent} {
last, err := js.GetLastMsg(broker.EventsStream, EventSubject(BusModule, event), nats.Context(ctx))
if errors.Is(err, nats.ErrMsgNotFound) {
continue
}
if err != nil {
return nil, fmt.Errorf("reading what the bus module said of its users' connections: %w", err)
}
if !last.Time.Before(since) {
recent = true
}
}
if !recent {
return nil, nil
}
sub, err := js.SubscribeSync(EventSubject(BusModule, "bus.>"), nats.OrderedConsumer(), nats.StartTime(since))
if err != nil {
return nil, fmt.Errorf("reading what the bus module said of its users' connections: %w", err)
}
defer func() { _ = sub.Unsubscribe() }()
var closures []BusClosure
var kinds []string
read, cut := 0, false
for {
if read >= MaxBusEventsRead {
cut = true
break
}
read++
wait, cancel := context.WithTimeout(ctx, readQuiet)
msg, err := sub.NextMsgWithContext(wait)
cancel()
if err != nil {
if ctx.Err() != nil {
return nil, ctx.Err()
}
break
}
if kind, ok := busEventKind(msg.Subject); ok {
var c BusClosure
if json.Unmarshal(msg.Data, &c) == nil && c.User != "" && !c.Closed.IsZero() {
closures, kinds = append(closures, c), append(kinds, kind)
}
}
if meta, err := msg.Metadata(); err != nil || meta.NumPending == 0 {
break
}
}
out := BusClosures(closures, kinds, since)
if cut {
// Said, never silent: a busy hour is read up to the bound, and each advisory says it is a floor.
for i := range out {
out[i].Said += fmt.Sprintf(" (at least: the read stopped at %d of the bus module's events this hour)",
MaxBusEventsRead)
}
}
return out, nil
}
// MaxBusEventsRead bounds one tick's read of the bus module's events: a watchdog tick must not spend itself
// reading a storm of closures back. A read cut there says so in every advisory it makes.
var MaxBusEventsRead = 2000
// MergeAdvisories is two lists of advisories as one: the same kind about the same thing is one advisory,
// counted once (the larger count of the two), said by the newer. The controller's own slow consumer as its client library sees it and
// as the server closed it are the same user's (`controller`).
func MergeAdvisories(a, b []Advisory) []Advisory {
out := append([]Advisory(nil), a...)
at := map[string]int{}
for i, x := range out {
at[x.Kind+"/"+x.ID+"/"+x.Token] = i
}
for _, y := range b {
key := y.Kind + "/" + y.ID + "/" + y.Token
i, had := at[key]
if !had {
at[key] = len(out)
out = append(out, y)
continue
}
x := &out[i]
// The same fault seen twice, not two faults: the controller's own slow consumer is told to its client
// library and closed by the server, and is counted once — by whichever saw more of it.
if y.Count > x.Count {
x.Count = y.Count
}
if y.First.Before(x.First) {
x.First = y.First
}
if y.Last.After(x.Last) {
x.Last, x.Said = y.Last, y.Said
if y.Reason != "" {
x.Reason = y.Reason
}
}
if x.User == "" {
x.User = y.User
}
}
return out
}
+121
View File
@@ -0,0 +1,121 @@
package link
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/testbus"
)
// What the bus module says of its users' connections, read back as advisories (novox/hq ADR 0357, issue
// 270): one per user and kind, each closure counted once however often it is said.
//
// go test ./internal/link/ -run TestNatsBusFaults (each test on a bus of its own: internal/testbus)
func aBusWithEvents(t *testing.T) *Server {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
return ConnectNats(js, nil, nil)
}
func sayClosure(t *testing.T, s *Server, event string, c BusClosure) {
t.Helper()
body, _ := json.Marshal(c)
if _, err := s.js.Context().Publish(EventSubject(BusModule, event), body); err != nil {
t.Fatal(err)
}
}
func TestNatsBusFaultsAreOnePerUserAndKindEachClosureCountedOnce(t *testing.T) {
s := aBusWithEvents(t)
now := time.Now().UTC()
slow := BusClosure{User: "laptop.claude-code", Name: "laptop.claude-code", CID: 10, Reason: "Slow Consumer (Write Deadline)",
Opened: now.Add(-20 * time.Minute), Closed: now.Add(-2 * time.Minute), Pending: 334000}
sayClosure(t, s, BusSlowConsumerEvent, slow)
sayClosure(t, s, BusSlowConsumerEvent, slow) // said again: the module started again
again := slow
again.CID, again.Opened, again.Closed = 31, now.Add(-90*time.Second), now.Add(-time.Minute)
sayClosure(t, s, BusSlowConsumerEvent, again)
sayClosure(t, s, BusConnectionClosedEvent, BusClosure{User: "anchor.desk", CID: 7,
Reason: "Maximum Message Payload Exceeded", Opened: now.Add(-time.Hour / 2), Closed: now.Add(-30 * time.Second)})
// Older than the hour: S9 no longer says it.
sayClosure(t, s, BusSlowConsumerEvent, BusClosure{User: "anchor.media", CID: 3, Reason: "Slow Consumer (Pending Bytes)",
Opened: now.Add(-3 * time.Hour), Closed: now.Add(-2 * time.Hour)})
got, err := s.BusFaults(t.Context(), now.Add(-time.Hour))
if err != nil {
t.Fatal(err)
}
if len(got) != 2 {
t.Fatalf("read %d advisories, want one for each user within the hour: %+v", len(got), got)
}
for _, a := range got {
switch a.User {
case "laptop.claude-code":
if a.Kind != AdvisorySlowConsumer || a.ID != "laptop.claude-code" || a.Count != 2 ||
!a.Last.Equal(again.Closed) || !a.First.Equal(slow.Closed) || !strings.Contains(a.Said, "connection 31") {
t.Errorf("the slow consumer read as %+v: two closures, the one said twice counted once", a)
}
case "anchor.desk":
if a.Kind != AdvisoryConnectionClosed || a.Count != 1 || a.Reason != "Maximum Message Payload Exceeded" {
t.Errorf("the closed connection read as %+v", a)
}
default:
t.Errorf("an advisory for %s", a.User)
}
}
}
func TestNatsBusFaultsReadNothingWhenTheBusModuleSaidNothing(t *testing.T) {
s := aBusWithEvents(t)
started := time.Now()
got, err := s.BusFaults(t.Context(), time.Now().Add(-time.Hour))
if err != nil || len(got) != 0 {
t.Fatalf("read %v, %v from a stream the bus module never wrote", got, err)
}
if took := time.Since(started); took > time.Second {
t.Fatalf("reading nothing took %s: a tick must not wait on an empty stream", took)
}
}
func TestTheControllersOwnSlowConsumerAndItsClosedConnectionAreOneAdvisory(t *testing.T) {
at := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
own := []Advisory{{Kind: AdvisorySlowConsumer, ID: "controller", Said: "the controller fell behind", First: at, Last: at, Count: 1}}
closed := BusClosures([]BusClosure{{User: "controller", CID: 5, Reason: "Slow Consumer (Pending Bytes)",
Opened: at.Add(-time.Hour), Closed: at.Add(time.Minute)}}, []string{AdvisorySlowConsumer}, at.Add(-time.Hour))
got := MergeAdvisories(own, closed)
if len(got) != 1 || got[0].Count != 1 || !got[0].Last.Equal(at.Add(time.Minute)) || got[0].User != "controller" {
t.Fatalf("merged as %+v: one fall behind, told to the client and closed by the server, is counted once", got)
}
}
// **A busy hour is read up to a bound, and says it was cut** (review of #326): a storm of closures must not
// spend a watchdog tick, and a count cut short is a floor, said as one.
func TestNatsBusFaultsReadAtMostTheBoundAndSaySo(t *testing.T) {
s := aBusWithEvents(t)
was := MaxBusEventsRead
MaxBusEventsRead = 2
t.Cleanup(func() { MaxBusEventsRead = was })
now := time.Now().UTC()
for i := range 3 {
sayClosure(t, s, BusSlowConsumerEvent, BusClosure{User: "laptop.claude-code", CID: uint64(10 + i),
Reason: "Slow Consumer (Pending Bytes)", Opened: now.Add(-time.Hour / 2), Closed: now.Add(-time.Duration(3-i) * time.Minute)})
}
got, err := s.BusFaults(t.Context(), now.Add(-time.Hour))
if err != nil {
t.Fatal(err)
}
if len(got) != 1 || got[0].Count != 2 || !strings.Contains(got[0].Said, "at least: the read stopped at 2") {
t.Fatalf("a read past its bound read as %+v", got)
}
}
+14 -6
View File
@@ -106,6 +106,13 @@ type CallKeeper interface {
Kept(ctx context.Context, id string) (Call, bool, error)
// Recent is the kept calls, newest first, without their answers.
Recent(ctx context.Context) ([]Call, error)
// MarkAbandoned marks the call id abandoned, unless its record already says it finished (novox/hq
// issue 711): a finish recorded before, or racing, the mark must stand — a call whose answer was
// kept never reads abandoned. It is the record's own compare-and-set that decides the race, not
// the order these two writers happen to run in, so it retries against a finish landing in
// between. It returns the record as it stands once this returns: marked abandoned, the finish
// that won instead, or a zero Call when the record is gone.
MarkAbandoned(ctx context.Context, id string) (Call, error)
}
// CallLog keeps the latest calls a holder served.
@@ -171,14 +178,15 @@ func (l *CallLog) Durably(ctx context.Context, keeper CallKeeper, holder string,
if c.State != CallRunning || c.Holder == holder {
continue
}
whole, found, err := keeper.Kept(ctx, c.ID)
if err != nil || !found {
whole = c
}
whole.State = CallAbandoned
if err := keeper.Keep(ctx, whole); err != nil {
final, err := keeper.MarkAbandoned(ctx, c.ID)
if err != nil {
return fmt.Errorf("marking %s abandoned: %w", c.ID, err)
}
if final.State != CallAbandoned {
// Its finish was recorded before this sweep reached it, or while it raced this mark
// (novox/hq issue 711): that word stands, with its kept answer, and is never overwritten.
continue
}
if logger != nil {
logger.Printf("%s (%s.%s, asked %s by %s) was running under %s (epoch %d), which stopped: marked "+
"abandoned — it may have done part of what it was asked, and nothing will finish it", c.ID, c.Seat,
+47
View File
@@ -87,6 +87,53 @@ func (b *BusCalls) Kept(ctx context.Context, id string) (Call, bool, error) {
return c, true, nil
}
// markAbandonedTries bounds the compare-and-set loop below (novox/hq issue 711 review round 1): a
// record that keeps changing on every read is not this sweep's to keep retrying forever over.
const markAbandonedTries = 10
// MarkAbandoned marks id abandoned unless its record already says it finished (novox/hq issue 711):
// a controller starting finds a call running under a holder that stopped, but that holder may have
// written the finish — with its kept answer — around the same moment. The record's own
// compare-and-set decides the race rather than the order these two writers happen to run in: this
// reads the record, and if it still says running, writes abandoned at that revision; a finish landing
// in between fails the write with a revision mismatch, and this rereads and finds the finish, which
// then stands. A record already resolved, or gone, is returned as found, so the caller never writes
// abandoned over it. Past markAbandonedTries attempts, it gives up and returns an error naming the
// call instead of looping forever (novox/hq issue 711 review round 1); Durably reports that as it
// reports any other keeper error.
func (b *BusCalls) MarkAbandoned(ctx context.Context, id string) (Call, error) {
for try := 0; try < markAbandonedTries; try++ {
entry, err := b.kv.Get(ctx, id)
if errors.Is(err, jetstream.ErrKeyNotFound) || errors.Is(err, jetstream.ErrInvalidKey) {
return Call{}, nil
}
if err != nil {
return Call{}, err
}
var c Call
if err := json.Unmarshal(entry.Value(), &c); err != nil {
return Call{}, fmt.Errorf("the record of %s on the bus is not a call: %w", id, err)
}
if c.State != CallRunning {
// A finish (or an earlier mark) is already the record's word: it stands, untouched.
return c, nil
}
c.State = CallAbandoned
record, err := json.Marshal(c)
if err != nil {
return Call{}, err
}
if _, err := b.kv.Update(ctx, id, record, entry.Revision()); err != nil {
if errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
continue // something — a finish, most likely — wrote since: reread and judge it again
}
return Call{}, err
}
return c, nil
}
return Call{}, fmt.Errorf("%s: its record kept changing; it is left as it stands", id)
}
// Recent is every call the bucket holds, newest first, without answers: read once through a watch
// of the records, which hands over the current value of each and then says it has.
func (b *BusCalls) Recent(ctx context.Context) ([]Call, error) {
+159
View File
@@ -105,6 +105,165 @@ func TestNatsACallsOutcomeOutlivesItsController(t *testing.T) {
}
}
// racingRecent wraps a CallKeeper so a finish lands right after its Recent() snapshot is taken, as
// if that finish raced the moment the snapshot was read (novox/hq issue 711): the snapshot still
// says a call is running, but by the time anything acts on that entry the record has already moved
// on.
type racingRecent struct {
CallKeeper
thenLand func()
}
func (r racingRecent) Recent(ctx context.Context) ([]Call, error) {
out, err := r.CallKeeper.Recent(ctx)
if err == nil && r.thenLand != nil {
r.thenLand()
}
return out, err
}
// **A finish recorded before the sweep reaches a call stands, with its kept answer** (novox/hq issue
// 711): a stale "running" snapshot is no excuse to overwrite a record that has since said how the
// call ended. A caller reading `abandoned` for a call that succeeded would redo work already done.
func TestNatsAFinishRacingTheSweepsSnapshotReadsFinished(t *testing.T) {
keeper := callsBucket(t)
running := Call{ID: "call-711-1", Seat: "mesh-controller", Verb: "push", State: CallRunning, Holder: "controller@one"}
if err := keeper.Keep(t.Context(), running); err != nil {
t.Fatal(err)
}
waitKept(t, keeper, running.ID, CallRunning)
finished := running
finished.State = CallAnswered
finished.Answer = json.RawMessage(`"anchor told"`)
wrapped := racingRecent{CallKeeper: keeper, thenLand: func() {
if err := keeper.Keep(context.Background(), finished); err != nil {
t.Error(err)
}
waitKept(t, keeper, running.ID, CallAnswered)
}}
var said bytes.Buffer
second := NewCallLog()
if err := second.Durably(t.Context(), wrapped, "controller@two", log.New(&said, "", 0)); err != nil {
t.Fatal(err)
}
got, found, err := second.Get(running.ID)
if err != nil || !found || got.State != CallAnswered || !strings.Contains(string(got.Answer), "anchor told") {
t.Fatalf("its finish did not stand: %+v %v %v", got, found, err)
}
if strings.Contains(said.String(), running.ID) {
t.Errorf("a finished call was said abandoned: %q", said.String())
}
}
// A call left running with no finish recorded is still marked abandoned — unchanged (novox/hq issue
// 711): only a finish actually recorded for it may stand instead.
func TestNatsARunningCallWithNoFinishIsStillAbandoned(t *testing.T) {
keeper := callsBucket(t)
running := Call{ID: "call-711-2", Seat: "mesh-controller", Verb: "plans", State: CallRunning, Holder: "controller@one"}
if err := keeper.Keep(t.Context(), running); err != nil {
t.Fatal(err)
}
waitKept(t, keeper, running.ID, CallRunning)
var said bytes.Buffer
second := NewCallLog()
if err := second.Durably(t.Context(), keeper, "controller@two", log.New(&said, "", 0)); err != nil {
t.Fatal(err)
}
got, found, err := second.Get(running.ID)
if err != nil || !found || got.State != CallAbandoned || !strings.Contains(said.String(), running.ID) {
t.Fatalf("got %+v %v %v; said %q", got, found, err, said.String())
}
}
// delayedKV wraps a jetstream.KeyValue so a hook runs just before Update, to land a write between a
// reader's Get and its own compare-and-set write.
type delayedKV struct {
jetstream.KeyValue
before func()
}
func (d delayedKV) Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error) {
d.before()
return d.KeyValue.Update(ctx, key, value, revision)
}
// **The race where a finish lands after the sweep's own read but before its write: the finish
// wins** (novox/hq issue 711). The sweep's write is a compare-and-set on the revision it read, so a
// finish landing in between invalidates it; the sweep rereads, finds the finish, and leaves it alone.
func TestNatsAFinishRacingTheSweepsWriteWins(t *testing.T) {
keeper := callsBucket(t)
running := Call{ID: "call-711-3", Seat: "mesh-controller", Verb: "push", State: CallRunning, Holder: "controller@one"}
if err := keeper.Keep(t.Context(), running); err != nil {
t.Fatal(err)
}
waitKept(t, keeper, running.ID, CallRunning)
finished := running
finished.State = CallFinishedAfter
finished.Answer = json.RawMessage(`"anchor told"`)
landed := false
racey := &BusCalls{kv: delayedKV{KeyValue: keeper.kv, before: func() {
if landed {
return
}
landed = true
if err := keeper.Keep(context.Background(), finished); err != nil {
t.Error(err)
}
waitKept(t, keeper, running.ID, CallFinishedAfter)
}}}
var said bytes.Buffer
second := NewCallLog()
if err := second.Durably(t.Context(), racey, "controller@two", log.New(&said, "", 0)); err != nil {
t.Fatal(err)
}
got, found, err := second.Get(running.ID)
if err != nil || !found || got.State != CallFinishedAfter || !strings.Contains(string(got.Answer), "anchor told") {
t.Fatalf("the finish did not win the race: %+v %v %v", got, found, err)
}
if strings.Contains(said.String(), running.ID) {
t.Errorf("a finished call was said abandoned: %q", said.String())
}
}
// alwaysMismatchKV wraps a jetstream.KeyValue whose Update always fails with a revision mismatch, as
// if some other writer kept winning the race forever (novox/hq issue 711 review round 1).
type alwaysMismatchKV struct {
jetstream.KeyValue
tries *int
}
func (a alwaysMismatchKV) Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error) {
*a.tries++
return 0, jetstream.ErrKeyRevisionMismatch
}
// **MarkAbandoned gives up after markAbandonedTries rather than loop forever** (novox/hq issue 711
// review round 1): a record whose compare-and-set keeps losing is left as it stands, said in an
// error naming the call, not retried without end.
func TestNatsAMarkAbandonedGivesUpAfterTenTries(t *testing.T) {
keeper := callsBucket(t)
running := Call{ID: "call-711-4", Seat: "mesh-controller", Verb: "push", State: CallRunning, Holder: "controller@one"}
if err := keeper.Keep(t.Context(), running); err != nil {
t.Fatal(err)
}
waitKept(t, keeper, running.ID, CallRunning)
tries := 0
stuck := &BusCalls{kv: alwaysMismatchKV{KeyValue: keeper.kv, tries: &tries}}
_, err := stuck.MarkAbandoned(t.Context(), running.ID)
if err == nil || !strings.Contains(err.Error(), running.ID) {
t.Fatalf("expected an error naming %s, got %v", running.ID, err)
}
if tries != markAbandonedTries {
t.Fatalf("tried %d times, want %d", tries, markAbandonedTries)
}
}
// The bucket holds the last thousand calls or fourteen days: a call is two keys, so the stream under
// it holds twice as many messages, and asserting it again keeps the count.
func TestNatsTheCallsBucketIsBounded(t *testing.T) {
+48
View File
@@ -37,3 +37,51 @@ func TestNatsTheBuildsAWalkMayTakeAreTheChecksOutcomesOfItsPullRequests(t *testi
t.Fatalf("read %+v", got)
}
}
// A build is read with the head its check judged, and a kept build is named as the check's that built it (novox/hq
// issue 703).
func TestNatsAKeptBuildIsTheBuildersCheckAndEveryBuildSaysItsHead(t *testing.T) {
js := aBus(t)
since := time.Now().Add(-time.Minute)
say := func(r BuildResult) {
body, _ := json.Marshal(r)
if _, err := js.Context().Publish(BuildOutcomeOf(TheBuildMachine), body); err != nil {
t.Fatal(err)
}
}
mine := &CheckRequest{Owner: "novox", Repo: "shop", Number: 7}
app := CheckBuilt{Module: "app", Key: "r1:1", Made: []MadeArtifact{{Name: "server", Reference: "s/app/server@sha256:a"}}}
say(BuildResult{ID: "build-1", Checked: mine, Check: &CheckOutcome{Verdict: "pass", Head: "aaaaaaaaaaaa",
Built: []CheckBuilt{app}}})
kept := app
kept.Kept = "build-1"
say(BuildResult{ID: "build-2", Checked: mine, Check: &CheckOutcome{Verdict: "pass", Head: "aaaaaaaaaaaa",
Built: []CheckBuilt{kept}}})
got, err := ReadCheckBuilds(t.Context(), js, TheBuildMachine, []PullRef{{Owner: "novox", Repo: "shop", Number: 7}}, since)
if err != nil {
t.Fatal(err)
}
if len(got) != 2 || got[0].Check != "build-1" || got[0].Built.Kept != "build-1" || got[0].Head != "aaaaaaaaaaaa" ||
got[1].Check != "build-1" || got[1].Head != "aaaaaaaaaaaa" {
t.Fatalf("read %+v", got)
}
}
// Two spellings of a head are the same head only when both say at least twelve characters of it.
func TestSameHeadNeedsTwelveCharactersOfBoth(t *testing.T) {
for _, c := range []struct {
a, b string
same bool
}{
{"aaaaaaaaaaaa", "aaaaaaaaaaaabbbb", true},
{"aaaaaaaaaaaabbbb", "aaaaaaaaaaaa", true},
{"aaaaaaaaaaaa", "aaaaaaaaaaab", false},
{"", "", false},
{"aaaa", "aaaaaaaaaaaa", false},
} {
if SameHead(c.a, c.b) != c.same {
t.Errorf("SameHead(%q, %q) is %v", c.a, c.b, !c.same)
}
}
}
+193 -5
View File
@@ -41,6 +41,10 @@ const (
DeadHeaderLost = "Mesh-Dead-Lost"
// DeadHeaderMsgID is the message's own de-duplication id, kept, since the copy carries its own.
DeadHeaderMsgID = "Mesh-Dead-Msg-Id"
// DeadHeaderRestricted says the message came from a restricted stream (novox/hq ADR 0376), and
// DeadHeaderAgesOut when its stream's maximum age runs out, counted from when it was first published.
DeadHeaderRestricted = "Mesh-Dead-Restricted"
DeadHeaderAgesOut = "Mesh-Dead-Ages-Out"
// AgainHeader marks a message delivered again, with the kept copy's id it came from.
AgainHeader = "Mesh-Delivered-Again"
)
@@ -65,12 +69,46 @@ type DeadLetter struct {
// Original says what became of the ask it was kept from, in its seat's work queue, when it was
// delivered again (novox/hq issue 334).
Original string `json:"original,omitempty"`
// Body and Headers are the message itself, given only for one dead letter asked by its id; a header
// with several values keeps them all.
// Restricted says it came from a restricted stream (novox/hq ADR 0376): its body and its own headers are
// never shown, said in Withheld, and it ages out at AgesOut, when its stream's maximum age runs out.
Restricted bool `json:"restricted,omitempty"`
AgesOut time.Time `json:"ages_out,omitzero"`
Withheld string `json:"withheld,omitempty"`
// Body and Headers are the message itself, given only for one dead letter asked by its id, and never for
// a restricted one; a header with several values keeps them all.
Body string `json:"body,omitempty"`
Headers map[string][]string `json:"headers,omitempty"`
}
// deadLetterNow is the clock dead letters age by: a variable, so a test can move it.
var deadLetterNow = time.Now
// leastLeftToDeliver is the least of its stream's age a restricted dead letter must have left to be delivered
// again (novox/hq ADR 0376 §5).
const leastLeftToDeliver = time.Minute
// withheldWords is what the dead-letters verb says in place of a restricted dead letter's body.
const withheldWords = "its body and its own headers are not shown: it came from a restricted stream, which only " +
"the modules its grant names may read (novox/hq ADR 0376)"
// sourceAge is whether a stream is restricted (novox/hq ADR 0363 §4), and its maximum age: read from the stream
// itself, where the controller says both. A stream that no longer exists is neither.
func sourceAge(js nats.JetStreamContext, stream string) (bool, time.Duration, error) {
info, err := js.StreamInfo(stream)
if errors.Is(err, nats.ErrStreamNotFound) {
return false, 0, nil
}
if err != nil {
return false, 0, fmt.Errorf("whether %s is restricted cannot be read: %w", stream, err)
}
return info.Config.Metadata[broker.RestrictedMetadata] == "true", info.Config.MaxAge, nil
}
// ttlOf is a time left as a message's time to live, in whole seconds and never more than what is left.
func ttlOf(left time.Duration) string {
return strconv.FormatInt(int64(left/time.Second), 10) + "s"
}
// maxDeliveries is the part of the server's notice a kept message is made from.
type maxDeliveries struct {
Stream string `json:"stream"`
@@ -94,8 +132,28 @@ func KeepDeadLetter(js nats.JetStreamContext, notice []byte) (DeadLetter, error)
gaveUp = time.Now()
}
kept := &nats.Msg{Subject: broker.DeadLetterSubject(a.Stream, a.Consumer), Header: nats.Header{}}
// **A restricted stream's message ages out with its stream** (novox/hq ADR 0376 §3): kept with a time to live
// of what is left of the stream's maximum age, counted from when it was first published; one already past it
// is kept as a record without its body. Every other dead letter keeps no age, as before (ADR 0264).
restricted, maxAge, err := sourceAge(js, a.Stream)
if err != nil {
return DeadLetter{}, err
}
if restricted {
kept.Header.Set(DeadHeaderRestricted, "true")
}
raw, err := js.GetMsg(a.Stream, a.StreamSeq)
agesOut := time.Time{}
if err == nil && restricted && maxAge > 0 {
agesOut = raw.Time.Add(maxAge)
kept.Header.Set(DeadHeaderAgesOut, agesOut.UTC().Format(time.RFC3339Nano))
}
switch {
case err == nil && !agesOut.IsZero() && agesOut.Sub(deadLetterNow()) < time.Second:
kept.Header.Set(DeadHeaderSubject, raw.Subject)
kept.Header.Set(DeadHeaderPublished, raw.Time.UTC().Format(time.RFC3339Nano))
kept.Header.Set(DeadHeaderLost, fmt.Sprintf("message %d of %s had passed its stream's maximum age of %s when "+
"the notice was taken, so it is kept without its body (novox/hq ADR 0376)", a.StreamSeq, a.Stream, maxAge))
case err == nil:
for k, v := range raw.Header {
if strings.HasPrefix(k, "Nats-") {
@@ -109,6 +167,9 @@ func KeepDeadLetter(js nats.JetStreamContext, notice []byte) (DeadLetter, error)
kept.Header.Set(DeadHeaderSubject, raw.Subject)
kept.Header.Set(DeadHeaderPublished, raw.Time.UTC().Format(time.RFC3339Nano))
kept.Data = raw.Data
if !agesOut.IsZero() {
kept.Header.Set(nats.MsgTTLHdr, ttlOf(agesOut.Sub(deadLetterNow())))
}
case errors.Is(err, nats.ErrMsgNotFound) || errors.Is(err, nats.ErrStreamNotFound):
kept.Header.Set(DeadHeaderLost, fmt.Sprintf("%s no longer held message %d when the notice was taken: %v",
a.Stream, a.StreamSeq, err))
@@ -143,6 +204,8 @@ func deadLetterOf(id uint64, subject string, h nats.Header, data []byte, whole b
d.Deliveries, _ = strconv.ParseUint(h.Get(DeadHeaderDeliveries), 10, 64)
d.GaveUp, _ = time.Parse(time.RFC3339Nano, h.Get(DeadHeaderGaveUp))
d.Published, _ = time.Parse(time.RFC3339Nano, h.Get(DeadHeaderPublished))
d.Restricted = h.Get(DeadHeaderRestricted) == "true"
d.AgesOut, _ = time.Parse(time.RFC3339Nano, h.Get(DeadHeaderAgesOut))
if whole {
d.Body = string(data)
d.Headers = map[string][]string{}
@@ -230,6 +293,9 @@ func DeadLetters(js nats.JetStreamContext, consumer string, most int) ([]DeadLet
if consumer != "" && consumer != d.Consumer && consumer != d.Stream+"."+d.Consumer {
continue
}
if d.Restricted {
d.Withheld = withheldWords
}
out = append(out, d)
}
return out, total, nil
@@ -238,8 +304,21 @@ func DeadLetters(js nats.JetStreamContext, consumer string, most int) ([]DeadLet
// ErrNoDeadLetter is a dead letter's id DEAD_LETTERS does not hold.
var ErrNoDeadLetter = errors.New("no such dead letter")
// DeadLetterNamed is one kept message whole: what it said, and the headers it said it with.
// DeadLetterNamed is one kept message whole: what it said, and the headers it said it with — **but for one from
// a restricted stream, whose body and own headers are never shown** (novox/hq ADR 0376 §4). A grant names modules,
// and this is asked by people and by tool runners' bus accounts, which a grant never names: withholding the body
// from every caller is that grant's check, said plainly.
func DeadLetterNamed(js nats.JetStreamContext, id uint64) (DeadLetter, error) {
d, err := deadLetterWhole(js, id)
if err != nil || !d.Restricted {
return d, err
}
d.Body, d.Headers, d.Withheld = "", nil, withheldWords
return d, nil
}
// deadLetterWhole is one kept message whole, a restricted one's body too: for delivering it again, never to show.
func deadLetterWhole(js nats.JetStreamContext, id uint64) (DeadLetter, error) {
raw, err := js.GetMsg(broker.DeadLettersStream, id)
if errors.Is(err, nats.ErrMsgNotFound) {
return DeadLetter{}, fmt.Errorf("%w: %s holds no message %d — delivered again or dropped already, "+
@@ -259,7 +338,8 @@ func DeadLetterNamed(js nats.JetStreamContext, id uint64) (DeadLetter, error) {
// work queue has one worker per subject, so only the worker that gave it up takes it, and DeliverAgain
// removes the original from the queue first, so there are never two (novox/hq issue 334); only while the
// worker that gave it up is on the bus. Any other stream's
// message is refused, with why: an ask to a seat the controller does not ask would need a publish it is
// message is refused, with why — but a module's own stream's, which goes back as an event while its consumer is
// still granted it (novox/hq ADR 0363 §7): an ask to a seat the controller does not ask would need a publish it is
// not granted, in its asker's name (ADR 0264's consequences, ADR 0259 §3), and a stream that is neither
// would reach every consumer of its subject.
func AgainTo(js nats.JetStreamContext, d DeadLetter) (string, error) {
@@ -280,6 +360,39 @@ func AgainTo(js nats.JetStreamContext, d DeadLetter) (string, error) {
return "", fmt.Errorf("whether %s can receive dead letter %d cannot be read: %w", d.Who, d.ID, err)
}
return d.Subject, nil
case d.Stream != broker.EventsStream && broker.OnAModuleStream(d.Subject):
// **A module stream's event goes back to its consumer only while it is still granted** (novox/hq ADR
// 0363 §7): its consumer on that stream must still be there and still filter the subject, or delivering
// it again would hand a module what its grant no longer names. Then it is delivered as an event is, on
// EVENTS under the consumer's own again-subject, which the module's consumer there filters.
//
// **A restricted one only while at least a minute of its stream's age is left** (novox/hq ADR 0376 §5):
// past that it can only be dropped, and the copy keeps what is left (DeliverAgain).
if d.Restricted {
if left := d.AgesOut.Sub(deadLetterNow()); d.AgesOut.IsZero() || left < leastLeftToDeliver {
return "", fmt.Errorf("dead letter %d has aged out: %s keeps a message only for its maximum age, and "+
"less than a minute of it is left (it ages out at %s), so it is not delivered again. Drop it "+
"(novox/hq ADR 0376)", d.ID, d.Stream, d.AgesOut.UTC().Format(time.RFC3339))
}
}
info, err := js.ConsumerInfo(d.Stream, d.Consumer)
if errors.Is(err, nats.ErrConsumerNotFound) {
return "", fmt.Errorf("dead letter %d was given up by %s, which no longer hears %s, so it would reach a "+
"module no longer granted it. Drop it", d.ID, d.Who, d.Stream)
}
if err != nil {
return "", fmt.Errorf("whether %s still hears %s cannot be read: %w", d.Who, d.Stream, err)
}
granted := false
for _, f := range append([]string{info.Config.FilterSubject}, info.Config.FilterSubjects...) {
if f != "" && broker.SubjectsOverlap(f, d.Subject) {
granted = true
}
}
if !granted {
return "", fmt.Errorf("dead letter %d is on %s, which %s is no longer granted. Drop it", d.ID, d.Subject, d.Who)
}
d.Stream = broker.EventsStream
case d.Stream != broker.EventsStream:
return "", fmt.Errorf("dead letter %d is from %s, and only an event or an ask the controller made is "+
"delivered again: publishing it again would reach every consumer of its subject, or need a grant the "+
@@ -308,7 +421,16 @@ func AgainTo(js nats.JetStreamContext, d DeadLetter) (string, error) {
// from DEAD_LETTERS; an ask's original is removed from its work queue before. The message carries its own headers and AgainHeader; its de-duplication id is the
// kept copy's, so asking twice delivers it once.
func DeliverAgain(js nats.JetStreamContext, id uint64) (DeadLetter, string, error) {
d, err := DeadLetterNamed(js, id)
d, to, err := deliverAgain(js, id)
if d.Restricted {
// What it said stays unsaid to whoever asked it delivered (novox/hq ADR 0376 §4).
d.Body, d.Headers, d.Withheld = "", nil, withheldWords
}
return d, to, err
}
func deliverAgain(js nats.JetStreamContext, id uint64) (DeadLetter, string, error) {
d, err := deadLetterWhole(js, id)
if err != nil {
return d, "", err
}
@@ -325,6 +447,10 @@ func DeliverAgain(js nats.JetStreamContext, id uint64) (DeadLetter, string, erro
}
again.Header.Set(AgainHeader, strconv.FormatUint(id, 10))
again.Header.Set(nats.MsgIdHdr, "again."+broker.DeadLettersStream+"."+strconv.FormatUint(id, 10))
if d.Restricted {
// The copy keeps what is left of its stream's age (novox/hq ADR 0376 §5).
again.Header.Set(nats.MsgTTLHdr, ttlOf(d.AgesOut.Sub(deadLetterNow())))
}
if _, err := js.PublishMsg(again); err != nil {
if d.Original != "" {
return d, to, fmt.Errorf("dead letter %d could not be delivered again on %s: %w; it is still kept, so "+
@@ -449,3 +575,65 @@ func keepGivenUp(js nats.JetStreamContext, logger *log.Logger) (*nats.Subscripti
_ = m.Ack()
}, nats.Bind(broker.DeadLetterNoticesStream, broker.ControllerName), nats.ManualAck())
}
// Aging is one restricted dead letter as it ages (novox/hq ADR 0376 §6): its id, and when its stream's maximum age
// runs out and DEAD_LETTERS lets it go.
type Aging struct {
ID uint64
AgesOut time.Time
}
// AgingDeadLetters is, for each consumer of held whose dead letters came from a restricted stream, each one that
// ages out and when: read by their headers alone, never their bodies. A consumer of any other stream is left out,
// and so is a record that keeps no age.
func AgingDeadLetters(js nats.JetStreamContext, held map[string]int) (map[string][]Aging, error) {
out := map[string][]Aging{}
restricted := map[string]bool{}
for key, n := range held {
stream, consumer, _ := strings.Cut(key, ".")
is, known := restricted[stream]
if !known {
r, _, err := sourceAge(js, stream)
if err != nil {
return nil, err
}
restricted[stream], is = r, r
}
if !is || n <= 0 {
continue
}
sub, err := js.SubscribeSync(broker.DeadLetterSubject(stream, consumer), nats.BindStream(broker.DeadLettersStream),
nats.OrderedConsumer(), nats.HeadersOnly(), nats.DeliverAll())
if err != nil {
return nil, fmt.Errorf("the dead letters of %s cannot be read: %w", key, err)
}
// held can count more than DEAD_LETTERS still holds for this consumer — one aged out,
// delivered again or dropped since it was counted (novox/hq issue 728 review round 1): read
// the ordered consumer's pending count once, from right when it was subscribed, and look for
// at most that many messages, so a look never waits out NextMsg's bound for one that is not
// there. That bound stays only as a guard against a pending count that is itself stale.
limit := n
if pending, err := sub.InitialConsumerPending(); err == nil && int(pending) < limit {
limit = int(pending)
}
for i := 0; i < limit; i++ {
m, err := sub.NextMsg(2 * time.Second)
if errors.Is(err, nats.ErrTimeout) {
break // delivered again, dropped or aged out since they were counted
}
if err != nil {
_ = sub.Unsubscribe()
return nil, fmt.Errorf("the dead letters of %s cannot be read: %w", key, err)
}
meta, err := m.Metadata()
if err != nil {
continue
}
if at, err := time.Parse(time.RFC3339Nano, m.Header.Get(DeadHeaderAgesOut)); err == nil {
out[key] = append(out[key], Aging{ID: meta.Sequence.Stream, AgesOut: at})
}
}
_ = sub.Unsubscribe()
}
return out, nil
}
@@ -0,0 +1,48 @@
package link
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
)
// A message given up on a module's own stream is delivered again on EVENTS to the consumer that gave it up,
// and only while that consumer is still granted it (novox/hq ADR 0363 §7).
func TestAModuleStreamsDeadLetterGoesBackOnlyWhileGranted(t *testing.T) {
js := aBus(t)
mail := broker.ModuleStream{Module: "mesh-mail", Name: "MAIL", Events: []string{"mail.*.message.added"},
MaxAgeDays: 7, MaxMiB: 16, Restricted: true}
if err := js.EnsureStream(mail.Stream()); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = js.Context().DeleteStream("MAIL") })
subject := broker.ModuleStreamSubject("mesh-mail", "mail.mesh.message.added")
heard := broker.Heard{Stream: "MAIL", Emitter: "mesh-mail", Filters: []string{subject}}
reader := broker.Principal{Kind: broker.KindModule, Node: "ace", Module: "reader", Hears: []broker.Heard{heard},
PasswordHash: "x"}
users := broker.ConsumersOf([]broker.Principal{reader})
if len(users) != 2 {
t.Fatalf("the reader's consumers: %+v", users)
}
for _, c := range users {
if err := js.EnsureConsumer(c.Consumer); err != nil {
t.Fatal(err)
}
}
d := DeadLetter{ID: 9, Stream: "MAIL", Consumer: "ace_reader", Subject: subject}
got, err := AgainTo(js.Context(), d)
if err != nil {
t.Fatal(err)
}
if got != broker.AgainSubject("ace_reader", subject) {
t.Fatalf("delivered again on %s", got)
}
if err := js.RemoveConsumer("MAIL", "ace_reader"); err != nil {
t.Fatal(err)
}
if _, err := AgainTo(js.Context(), d); err == nil || !strings.Contains(err.Error(), "no longer hears MAIL") {
t.Fatalf("delivered again to a consumer no longer granted: %v", err)
}
}
@@ -0,0 +1,251 @@
package link
import (
"fmt"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
)
// A restricted stream of a module, MAIL, kept a week, on the bus (novox/hq ADR 0363), and the subject an event of
// the account `mesh` is published on.
func aRestrictedStream(t *testing.T, js *broker.JetStream) string {
t.Helper()
mail := broker.ModuleStream{Module: "mesh-mail", Name: "MAIL", Events: []string{"mail.*.message.added"},
MaxAgeDays: 7, MaxMiB: 16, Restricted: true}
_ = js.Context().DeleteStream("MAIL")
if err := js.EnsureStream(mail.Stream()); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = js.Context().DeleteStream("MAIL") })
return broker.ModuleStreamSubject("mesh-mail", "mail.mesh.message.added")
}
// withClock sets the clock dead letters are kept and delivered again by, until the test ends.
func withClock(t *testing.T, now func() time.Time) {
t.Helper()
before := deadLetterNow
deadLetterNow = now
t.Cleanup(func() { deadLetterNow = before })
}
// publishMail puts one mail event on MAIL with a header of its own, and answers its sequence and when it was stored.
func publishMail(t *testing.T, js *broker.JetStream, subject string) (uint64, time.Time) {
t.Helper()
msg := &nats.Msg{Subject: subject, Data: []byte(`{"from":"someone@example.org"}`), Header: nats.Header{}}
msg.Header.Set("x-account", "mesh")
ack, err := js.Context().PublishMsg(msg)
if err != nil {
t.Fatal(err)
}
raw, err := js.Context().GetMsg("MAIL", ack.Sequence)
if err != nil {
t.Fatal(err)
}
return ack.Sequence, raw.Time
}
// notice is the server's maximum-deliveries notice for one message of MAIL.
func notice(seq uint64) []byte {
return []byte(fmt.Sprintf(`{"stream":"MAIL","consumer":"ace_mailer","stream_seq":%d,"deliveries":5}`, seq))
}
// **A dead letter from a restricted stream is listed without its body, and ages out with its stream** (ADR 0376
// §3, §4): given up on by its consumer and kept, the list and the show by id give what it was and when it ages out,
// never its body or its own headers, and the kept copy carries a time to live of what is left of MAIL's week.
func TestARestrictedDeadLetterIsListedWithoutItsBody(t *testing.T) {
js := aBus(t)
keeping(t, js)
subject := aRestrictedStream(t, js)
heard := broker.Heard{Stream: "MAIL", Emitter: "mesh-mail", Restricted: true, Filters: []string{subject}}
for _, c := range broker.ConsumersOf([]broker.Principal{{Kind: broker.KindModule, Node: "ace", Module: "mailer",
Hears: []broker.Heard{heard}, PasswordHash: "x"}}) {
if err := js.EnsureConsumer(c.Consumer); err != nil {
t.Fatal(err)
}
}
_, published := publishMail(t, js, subject)
api, err := jetstream.New(js.Conn())
if err != nil {
t.Fatal(err)
}
mailer, err := api.Consumer(t.Context(), "MAIL", "ace_mailer")
if err != nil {
t.Fatal(err)
}
d := givenUp(t, js, mailer)
weekLeft := published.Add(7 * 24 * time.Hour)
if !d.Restricted || d.Body != "" || d.Headers != nil || d.Subject != subject || d.Sequence != 1 || d.Deliveries != 5 ||
d.Published.IsZero() || d.GaveUp.IsZero() || d.Size == 0 || d.AgesOut.Sub(weekLeft).Abs() > time.Second {
t.Fatalf("listed as %+v", d)
}
shown, err := DeadLetterNamed(js.Context(), d.ID)
if err != nil {
t.Fatal(err)
}
if shown.Body != "" || shown.Headers != nil || !shown.Restricted || shown.Withheld == "" || shown.AgesOut.IsZero() {
t.Fatalf("shown by its id as %+v", shown)
}
raw, err := js.Context().GetMsg(broker.DeadLettersStream, d.ID)
if err != nil {
t.Fatal(err)
}
ttl, err := time.ParseDuration(raw.Header.Get(nats.MsgTTLHdr))
if err != nil || ttl > 7*24*time.Hour || ttl < 7*24*time.Hour-time.Minute {
t.Fatalf("kept with a time to live of %q (%v), want what is left of the week", raw.Header.Get(nats.MsgTTLHdr), err)
}
if string(raw.Data) != `{"from":"someone@example.org"}` {
t.Fatal("the kept copy does not hold the message, so it could not be delivered again")
}
}
// **Past its stream's age, a restricted dead letter is gone** (ADR 0376 §3), on a fake clock: kept with two seconds
// of MAIL's week left, DEAD_LETTERS no longer holds it once they pass; with none left, it is kept as a record
// without its body, which cannot be delivered again.
func TestARestrictedDeadLetterAgesOutWithItsStream(t *testing.T) {
js := aBus(t)
subject := aRestrictedStream(t, js)
seq, published := publishMail(t, js, subject)
withClock(t, func() time.Time { return published.Add(7*24*time.Hour - 2*time.Second) })
d, err := KeepDeadLetter(js.Context(), notice(seq))
if err != nil {
t.Fatal(err)
}
if !d.Restricted || d.Lost != "" {
t.Fatalf("kept as %+v", d)
}
eventually(t, "the dead letter aged out", func() bool {
_, err := js.Context().GetMsg(broker.DeadLettersStream, d.ID)
return err != nil
})
if held, _ := HeldDeadLetters(js.Context()); held["MAIL.ace_mailer"] != 0 {
t.Fatalf("still held after its age: %v", held)
}
seq, published = publishMail(t, js, subject)
withClock(t, func() time.Time { return published.Add(7*24*time.Hour + time.Minute) })
past, err := KeepDeadLetter(js.Context(), notice(seq))
if err != nil {
t.Fatal(err)
}
raw, err := js.Context().GetMsg(broker.DeadLettersStream, past.ID)
if err != nil {
t.Fatal(err)
}
if len(raw.Data) != 0 || raw.Header.Get("x-account") != "" || !strings.Contains(past.Lost, "maximum age") {
t.Fatalf("kept past its age as %+v with %q", past, raw.Data)
}
if _, _, err := DeliverAgain(js.Context(), past.ID); err == nil || !strings.Contains(err.Error(), "Drop it") {
t.Fatalf("a record past its age was delivered again: %v", err)
}
}
// **A restricted dead letter is delivered again only within its age, and the copy keeps that age** (ADR 0376 §5):
// with more than a minute left the copy goes on EVENTS under the consumer's again subject with a time to live of
// what is left; with less, it is refused as aged out.
func TestARestrictedDeadLetterGoesBackOnlyWithinItsAge(t *testing.T) {
js := aBus(t)
subject := aRestrictedStream(t, js)
heard := broker.Heard{Stream: "MAIL", Emitter: "mesh-mail", Restricted: true, Filters: []string{subject}}
for _, c := range broker.ConsumersOf([]broker.Principal{{Kind: broker.KindModule, Node: "ace", Module: "mailer",
Hears: []broker.Heard{heard}, PasswordHash: "x"}}) {
if err := js.EnsureConsumer(c.Consumer); err != nil {
t.Fatal(err)
}
}
seq, published := publishMail(t, js, subject)
at := published.Add(7*24*time.Hour - 10*time.Minute)
withClock(t, func() time.Time { return at })
d, err := KeepDeadLetter(js.Context(), notice(seq))
if err != nil {
t.Fatal(err)
}
at = published.Add(7*24*time.Hour - 30*time.Second)
if _, _, err := DeliverAgain(js.Context(), d.ID); err == nil || !strings.Contains(err.Error(), "aged out") {
t.Fatalf("delivered again with under a minute left: %v", err)
}
at = published.Add(7*24*time.Hour - 5*time.Minute)
_, to, err := DeliverAgain(js.Context(), d.ID)
if err != nil {
t.Fatal(err)
}
if to != broker.AgainSubject("ace_mailer", subject) {
t.Fatalf("delivered again on %s", to)
}
raw, err := js.Context().GetLastMsg(broker.EventsStream, to)
if err != nil {
t.Fatal(err)
}
ttl, err := time.ParseDuration(raw.Header.Get(nats.MsgTTLHdr))
if err != nil || ttl > 5*time.Minute || ttl < 4*time.Minute {
t.Fatalf("the copy carries a time to live of %q (%v), want what is left", raw.Header.Get(nats.MsgTTLHdr), err)
}
if string(raw.Data) != `{"from":"someone@example.org"}` || raw.Header.Get("x-account") != "mesh" {
t.Fatalf("the copy is %q %v", raw.Data, raw.Header)
}
}
// What of a consumer's dead letters ages out, and when, is read without their bodies, and only for a restricted
// stream's (ADR 0376 §6).
func TestTheAgingOfRestrictedDeadLettersIsRead(t *testing.T) {
js := aBus(t)
subject := aRestrictedStream(t, js)
seq, published := publishMail(t, js, subject)
withClock(t, func() time.Time { return published.Add(time.Hour) })
d, err := KeepDeadLetter(js.Context(), notice(seq))
if err != nil {
t.Fatal(err)
}
if _, err := KeepDeadLetter(js.Context(), []byte(`{"stream":"EVENTS","consumer":"media_sonarr","stream_seq":42,`+
`"deliveries":5}`)); err != nil {
t.Fatal(err)
}
held, err := HeldDeadLetters(js.Context())
if err != nil {
t.Fatal(err)
}
aging, err := AgingDeadLetters(js.Context(), held)
if err != nil {
t.Fatal(err)
}
if len(aging) != 1 || len(aging["MAIL.ace_mailer"]) != 1 || aging["MAIL.ace_mailer"][0].ID != d.ID ||
!aging["MAIL.ace_mailer"][0].AgesOut.Equal(d.AgesOut) {
t.Fatalf("aging %+v, kept %+v", aging, d)
}
}
// **A look never waits out NextMsg's bound for a dead letter that is not there** (novox/hq issue 728 review round
// 1): held can count more than DEAD_LETTERS still holds for a consumer, one aged out, delivered again or dropped
// since it was counted. AgingDeadLetters reads the ordered consumer's pending count once and looks for at most
// that many, so it returns in well under the 2-second NextMsg bound rather than waiting it out for messages that
// are gone.
func TestTheAgingOfRestrictedDeadLettersDoesNotWaitForWhatHeldOvercounts(t *testing.T) {
js := aBus(t)
subject := aRestrictedStream(t, js)
seq, published := publishMail(t, js, subject)
withClock(t, func() time.Time { return published.Add(time.Hour) })
d, err := KeepDeadLetter(js.Context(), notice(seq))
if err != nil {
t.Fatal(err)
}
// held says 3 for this consumer; DEAD_LETTERS holds only the one just kept.
held := map[string]int{"MAIL.ace_mailer": 3}
started := time.Now()
aging, err := AgingDeadLetters(js.Context(), held)
elapsed := time.Since(started)
if err != nil {
t.Fatal(err)
}
if elapsed >= time.Second {
t.Fatalf("took %s, well over the 2-second bound it should never wait out", elapsed)
}
if len(aging["MAIL.ace_mailer"]) != 1 || aging["MAIL.ace_mailer"][0].ID != d.ID {
t.Fatalf("aging %+v, kept %+v", aging, d)
}
}
+22 -4
View File
@@ -81,6 +81,13 @@ const (
// steps from. Said after every save of the serving controller; a delivery's holder also asks for the
// walks it follows, so a save made elsewhere is found by comparison, never lost.
KeyPlanMoved = "plan-moved"
// KeyAssignmentChanged and KeyDeclaredConfigChanged: a key of the controller's assignments or declared-config
// mirror was written or removed (novox/hq ADR 0289, design 54 §3, design 57 F3): which key, how and at which
// revision of the bucket — never the value, which the bucket holds.
KeyAssignmentChanged = "assignment-changed"
KeyDeclaredConfigChanged = "declared-config-changed"
// KeyPredefinedChanged: the predefined-keys mirror's one key was written (design 57 K3): a newer list kept.
KeyPredefinedChanged = "predefined-changed"
)
// Applied is what a machine now runs, as the mesh states it.
@@ -172,10 +179,13 @@ type Announcement struct {
// which branch, producing which commit. The mesh matches it against every module's recorded
// source and builds what moved, bases first.
type SourceMoved struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Base string `json:"base"`
Head string `json:"head"`
Owner string `json:"owner"`
Repo string `json:"repo"`
Base string `json:"base"`
Head string `json:"head"`
// HeadSHA is the head commit the forge merged (novox/hq issue 703): a walk takes only a build a check of that
// head made. Empty from a forge module from before it, and then the walk takes no check's build.
HeadSHA string `json:"head_sha,omitempty"`
Commit string `json:"merge_commit_sha"`
CloneURL string `json:"clone_url"`
HTMLURL string `json:"html_url"`
@@ -290,6 +300,14 @@ type Checked struct {
Trunk string `json:"trunk,omitempty"`
Beside map[string]string `json:"beside,omitempty"`
Took string `json:"took,omitempty"`
// Release is the release branch the check judged for (novox/hq ADR 0358 §4), JudgedBy the commit of the
// controller that judged its gate (twelve characters), and BesideAt the branch each repository beside it was
// read at, by directory: so a reader sees a release-branch judgement apart from a trunk one. Empty otherwise.
Release string `json:"release,omitempty"`
JudgedBy string `json:"judged-by,omitempty"`
BesideAt map[string]string `json:"beside-at,omitempty"`
// Validator says which node-engine validator the judge of a release-branch check was built with (ADR 0358 §2).
Validator string `json:"validator,omitempty"`
// Built are the builds the check made for its merge, each with its reuse key and digests, or why not
// (novox/hq ADR 0350 §1): what the walk of the merge takes instead of building.
Built []CheckBuilt `json:"built,omitempty"`
+129
View File
@@ -0,0 +1,129 @@
package link
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
)
// The move of one value into the vault, asked of a machine's node-engine (design 57's V3, issue 676; the operator's
// decision of 2026-10-14 on that issue).
//
// The controller cannot open a value it holds; the machine it was sealed to can. So the controller asks that
// machine's engine on its own subject, one row of its store at a time, signed as a hand-over is under a context of
// its own: the row, where its value came from, the controller's generation, when it was made, and the value as the
// store holds it, sealed to that machine. The engine opens it, hands it to the vault's move-in sealed to the call's
// key, and answers what the vault kept once the vault's read-back opens to the same value. The engine holds the
// same shapes in its own link code (mesh-host internal/link MoveSecretAsk, MoveSecretAnswer); a test on each side
// holds the field names.
// MoveSecretContext is prefixed to a move ask's bytes before signing: never another ask's, nor a declaration's.
const MoveSecretContext = "novox-mesh move-secret v1\n"
// MoveSecretAsk is a move's own fields, beside a hand-over's: module and name for a module's own secret on the
// machine asked; provider, provision, consumer, consuming module and local name for a pair credential.
type MoveSecretAsk struct {
Module string `json:"module,omitempty"`
Name string `json:"name,omitempty"`
Provider string `json:"provider,omitempty"`
Provision string `json:"provision,omitempty"`
Consumer string `json:"consumer,omitempty"`
ConsumerModule string `json:"consumer_module,omitempty"`
Local string `json:"local,omitempty"`
Origin string `json:"origin"`
Generation string `json:"generation,omitempty"`
MadeAt time.Time `json:"made_at"`
Sealed string `json:"sealed"`
SealedTo string `json:"sealed_to,omitempty"`
}
// MoveSecretAnswer is the engine's answer: what the vault kept, read back the same, or why nothing was moved.
type MoveSecretAnswer struct {
Said string `json:"said,omitempty"`
Refused string `json:"refused,omitempty"`
Secret string `json:"secret,omitempty"`
Generation int `json:"generation,omitempty"`
Unchanged bool `json:"unchanged,omitempty"`
}
// MoveSecretWithin is how long the controller waits for one move: the engine asks the vault twice.
const MoveSecretWithin = 45 * time.Second
// AskMoveSecret asks one machine's node-engine to move one row's value into the vault, and reads its answer. An
// error is the ask not reaching an engine, or an answer that is not one; a refusal is the engine's, in the answer.
func AskMoveSecret(ctx context.Context, conn *nats.Conn, signer Signer, node, by string, row MoveSecretAsk,
timeout time.Duration) (MoveSecretAnswer, error) {
if conn == nil {
return MoveSecretAnswer{}, errors.New("this controller is not on the bus")
}
if signer == nil {
return MoveSecretAnswer{}, errors.New("no signing key, so nothing can be asked of an engine")
}
nonce := make([]byte, 16)
if _, err := rand.Read(nonce); err != nil {
return MoveSecretAnswer{}, err
}
raw, err := json.Marshal(struct {
HandOverAsk
MoveSecretAsk
}{HandOverAsk{Node: node, By: by, Expires: time.Now().UTC().Add(HandOverGood), Nonce: hex.EncodeToString(nonce)},
row})
if err != nil {
return MoveSecretAnswer{}, err
}
signature, err := signer.Sign(ctx, append([]byte(MoveSecretContext), raw...))
if err != nil {
return MoveSecretAnswer{}, fmt.Errorf("cannot sign the ask: %w", err)
}
body, err := json.Marshal(SignedHandOver{Ask: raw, Signature: signature})
if err != nil {
return MoveSecretAnswer{}, err
}
asking, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
refused, stop := refusalsOf(conn, broker.AskMoveSecretSubject(node))
defer stop()
type replied struct {
msg *nats.Msg
err error
}
done := make(chan replied, 1)
go func() {
msg, err := conn.RequestWithContext(asking, broker.AskMoveSecretSubject(node), body)
done <- replied{msg, err}
}()
var reply *nats.Msg
select {
case r := <-done:
reply, err = r.msg, r.err
case why := <-refused:
cancel()
return MoveSecretAnswer{}, fmt.Errorf("the bus refused the controller asking %s to move a value: %v", node, why)
}
switch {
case errors.Is(err, nats.ErrNoResponders):
return MoveSecretAnswer{}, fmt.Errorf("nothing on %s answers a move: its node-engine is not running, is not on "+
"the bus, or is older than this ask (issue 676); nothing was moved", node)
case errors.Is(err, context.DeadlineExceeded), errors.Is(err, nats.ErrTimeout):
return MoveSecretAnswer{}, fmt.Errorf("%s did not answer a move within %s; whether the vault took it is not "+
"known — moving it again changes nothing if it did", node, timeout)
case err != nil:
return MoveSecretAnswer{}, err
}
var answer MoveSecretAnswer
if err := json.Unmarshal(reply.Data, &answer); err != nil {
return MoveSecretAnswer{}, fmt.Errorf("%s answered a move with something unreadable: %w", node, err)
}
if (answer.Refused == "") == (answer.Secret == "") {
return MoveSecretAnswer{}, fmt.Errorf("%s answered a move with neither what the vault kept nor a refusal", node)
}
return answer, nil
}
+90
View File
@@ -0,0 +1,90 @@
package link
import (
"context"
"crypto/ed25519"
"encoding/json"
"sort"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/testbus"
)
// The move's ask and answer keep the engine's field names (mesh-host internal/link,
// TestTheMovesAskKeepsItsFieldNamesAndContext): the row's fields beside a hand-over's, under a context of its own.
func TestTheMovesAskAndAnswerKeepTheirFieldNames(t *testing.T) {
keys := func(v any) string {
raw, _ := json.Marshal(v)
var m map[string]any
_ = json.Unmarshal(raw, &m)
var out []string
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return strings.Join(out, " ")
}
ask := MoveSecretAsk{Module: "m", Name: "n", Provider: "p", Provision: "q", Consumer: "c", ConsumerModule: "cm",
Local: "l", Origin: "made", Generation: "g", MadeAt: time.Unix(1, 0).UTC(), Sealed: "s", SealedTo: "k"}
if got := keys(ask); got != "consumer consumer_module generation local made_at module name origin provider "+
"provision sealed sealed_to" {
t.Fatalf("the ask's fields are %q", got)
}
answer := MoveSecretAnswer{Said: "s", Refused: "r", Secret: "own/a/b/c", Generation: 1, Unchanged: true}
if got := keys(answer); got != "generation refused said secret unchanged" {
t.Fatalf("the answer's fields are %q", got)
}
if broker.AskMoveSecretSubject("laptop") != "mesh.node.laptop.ask.move-secret" ||
MoveSecretContext != "novox-mesh move-secret v1\n" {
t.Fatalf("the subject %q, the context %q", broker.AskMoveSecretSubject("laptop"), MoveSecretContext)
}
}
// A move is asked of the machine on its own subject, signed under its own context — never a hand-over's — naming
// no path and carrying the row beside the hand-over's fields; the engine's answer comes back whole.
func TestAMoveIsAskedOfTheMachineSignedUnderItsOwnContext(t *testing.T) {
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
signer, public := testSigner(t)
var heard HandOverAsk
var row MoveSecretAsk
engine, err := conn.Subscribe(broker.AskMoveSecretSubject("laptop"), func(msg *nats.Msg) {
var signed SignedHandOver
_ = json.Unmarshal(msg.Data, &signed)
if ed25519.Verify(public, append([]byte(HandOverContext), signed.Ask...), signed.Signature) ||
!ed25519.Verify(public, append([]byte(MoveSecretContext), signed.Ask...), signed.Signature) {
_ = msg.Respond([]byte(`{"refused":"not signed as a move"}`))
return
}
_ = json.Unmarshal(signed.Ask, &heard)
_ = json.Unmarshal(signed.Ask, &row)
body, _ := json.Marshal(MoveSecretAnswer{Said: "moved", Secret: "own/laptop/" + row.Module + "/" + row.Name,
Generation: 1})
_ = msg.Respond(body)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = engine.Unsubscribe() })
a, err := AskMoveSecret(context.Background(), conn, signer, "laptop", "the move",
MoveSecretAsk{Module: "postgres", Name: "superuser-password", Origin: "made", Sealed: "c2Vh"}, 5*time.Second)
if err != nil || a.Secret != "own/laptop/postgres/superuser-password" || a.Generation != 1 {
t.Fatalf("answered %+v, %v", a, err)
}
if heard.Node != "laptop" || heard.Path != "" || heard.Nonce == "" || heard.Expires.IsZero() || heard.By != "the move" ||
row.Sealed != "c2Vh" || row.Origin != "made" {
t.Fatalf("the engine heard %+v %+v", heard, row)
}
if _, err := AskMoveSecret(context.Background(), conn, signer, "anchor", "the move", MoveSecretAsk{}, 2*time.Second); err == nil ||
!strings.Contains(err.Error(), "nothing was moved") {
t.Fatalf("a machine with no engine listening: %v", err)
}
}
+29
View File
@@ -448,6 +448,35 @@ const RootContract = 3
// mesh-host internal/link/messages.go's BusyContract.
const BusyContract = 8
// OwnershipContract is the statement of an engine that also reads a file's `owner-at-runtime` and a declaration's
// `filled_files` (novox/hq ADR 0349 §5). The same number as mesh-host internal/link/messages.go's.
const OwnershipContract = 9
// ProvisionerContract is the statement of an engine that also reads a declaration's `provisioner_account` and
// serves the local write socket a provisioner client fills through (novox/hq ADR 0349 §4). The same number as
// mesh-host internal/link/messages.go's.
const ProvisionerContract = 10
// HandedContract is the statement of an engine that writes a filled file only on the provisioner client's word
// and no longer from the declaration (novox/hq ADR 0349 §5, design 57 P7, issue 673): one writer per placed file.
// A moved module is handed to the provisioner only on a machine whose engine states this or later (design 57 P6);
// to an engine at OwnershipContract or ProvisionerContract a filled file would have two writers — the client, and
// the engine applying the declaration's file without its text — so such a machine is sent the module as before.
// mesh-host states it with P7.
const HandedContract = 11
// VaultBoxContract is the statement of an engine that also reads a file's `vault-sealed` (novox/hq ADR 0349 §16, issue
// 755): a core module's own credential the vault sealed to its machine in the mesh's sealed box, which the controller
// carries and cannot open, opened by the engine with its sealing key. Only to such an engine is the vault's box sent;
// an engine below it parses strictly and would refuse the whole declaration for the field, and is sent the
// controller's own sealed value as before. The same number as mesh-host internal/link/messages.go's.
const VaultBoxContract = 12
// NodeKeyContract is the statement of an engine that states its own sealing key to the vault, in its own name, at its
// start and whenever the key changes (novox/hq ADR 0372, issue 786). The same number as mesh-host
// internal/link/messages.go's.
const NodeKeyContract = 13
// The states of a busy container in a report (novox/hq ADR 0335 §2, §3).
const (
BusyWaiting = "waiting"
+11
View File
@@ -0,0 +1,11 @@
package link
import "testing"
// The engine that states its sealing key to the vault at start and on change (issue 786); the same number as
// mesh-host internal/link/messages.go's, after HandedContract (11) and VaultBoxContract (12).
func TestNodeKeyContractIsThirteen(t *testing.T) {
if NodeKeyContract != 13 {
t.Errorf("NodeKeyContract is %d", NodeKeyContract)
}
}
+32 -8
View File
@@ -375,6 +375,13 @@ func ReadCheckBuilds(ctx context.Context, js *broker.JetStream, seat string, pul
for _, p := range pulls {
wanted[p] = true
}
return readCheckBuilds(ctx, js, seat, since, func(p PullRef, _ CheckBuilt) bool { return wanted[p] })
}
// readCheckBuilds is every build the seat's checks said they made, or kept, that wanted takes. A kept build is named
// as the check's that built it (novox/hq issue 703): the record under its key is that check's.
func readCheckBuilds(ctx context.Context, js *broker.JetStream, seat string, since time.Time,
wanted func(PullRef, CheckBuilt) bool) ([]CheckedBuild, error) {
subject := BuildOutcomeOf(seat)
sub, err := js.Context().PullSubscribe(subject, "",
nats.BindStream(broker.EventsStream), nats.StartTime(since), nats.AckNone())
@@ -402,12 +409,13 @@ func ReadCheckBuilds(ctx context.Context, js *broker.JetStream, seat string, pul
continue
}
pull := PullRef{Owner: r.Checked.Owner, Repo: r.Checked.Repo, Number: r.Checked.Number}
if !wanted[pull] {
continue
}
for _, b := range r.Check.Built {
if b.NotBuilt == "" && b.Key != "" {
out = append(out, CheckedBuild{Pull: pull, Check: r.ID, Built: b})
if b.NotBuilt == "" && b.Key != "" && wanted(pull, b) {
check := r.ID
if b.Kept != "" {
check = b.Kept
}
out = append(out, CheckedBuild{Pull: pull, Check: check, Head: r.Check.Head, Built: b})
}
}
}
@@ -609,13 +617,29 @@ func PausedSaid(js *broker.JetStream, seat string, nodes []string) (map[string]H
// (design 33 §4: a mesh seat's verb carries no machine), and reads its answer (novox/hq ADR 0239: the
// delivery's owner's `stalled` and `close`).
func AskMeshSeatTool(ctx context.Context, conn *nats.Conn, seat, verb string, args any, timeout time.Duration) (Answer, error) {
return askMeshSeatToolAt(ctx, conn, SeatToolSubject(seat, verb), seat, verb, args, timeout)
}
// AskMeshSeatToolAs is AskMeshSeatTool in the caller's own name: on the subject that names its caller (novox/hq
// issue 365), which the bus lets only that user publish, so the holder knows who asks from the subject alone —
// how the vault takes a value from the controller and nobody else (design 57's V2, issue 675).
func AskMeshSeatToolAs(ctx context.Context, conn *nats.Conn, caller, seat, verb string, args any,
timeout time.Duration) (Answer, error) {
subject := broker.CalledSubject(SeatToolSubject(seat, verb), caller)
if subject == "" {
return Answer{}, fmt.Errorf("%q cannot ask %s.%s in its own name: it is not a bus user's name", caller, seat, verb)
}
return askMeshSeatToolAt(ctx, conn, subject, seat, verb, args, timeout)
}
func askMeshSeatToolAt(ctx context.Context, conn *nats.Conn, subject, seat, verb string, args any,
timeout time.Duration) (Answer, error) {
body, err := json.Marshal(args)
if err != nil {
return Answer{}, err
}
asking, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
subject := SeatToolSubject(seat, verb)
refused, stop := refusalsOf(conn, subject)
defer stop()
type replied struct {
@@ -633,8 +657,8 @@ func AskMeshSeatTool(ctx context.Context, conn *nats.Conn, seat, verb string, ar
reply, err = r.msg, r.err
case why := <-refused:
cancel()
return Answer{}, fmt.Errorf("the bus refused the controller asking %s.%s — its grants do not name %s: %v",
seat, verb, subject, why)
return Answer{}, fmt.Errorf("%w: the bus refused the controller asking %s.%s — its grants do not name %s: %v",
ErrGrantRefused, seat, verb, subject, why)
}
switch {
case errors.Is(err, nats.ErrNoResponders):
+6 -5
View File
@@ -220,21 +220,22 @@ func (n *natsInbound) deliver(ctx context.Context, act func(context.Context, Con
// stops saying so, and the bus redelivers exactly as it should.
working := make(chan struct{})
defer close(working)
go stillWorking(msg, working)
go stillWorking(msg, working, heartbeatWhileWorking)
}
act(ctx, m)
}
// heartbeatWhileWorking is how often a handler still running tells the bus so — comfortably inside
// the shortest acknowledgement window the mesh gives any of its consumers.
const heartbeatWhileWorking = 10 * time.Second
// the shortest acknowledgement window the mesh gives any of its consumers. A variable only so a test can
// outlive a window of a second in a few (novox/hq issue 717); nothing else sets it.
var heartbeatWhileWorking = 10 * time.Second
// stillWorking keeps one message alive until the work on it returns.
//
// An error is not worth reporting: what the bus does when it is not told is redeliver, which is
// exactly what happens if this fails, and the handler's own outcome is the thing worth logging.
func stillWorking(msg *nats.Msg, done <-chan struct{}) {
tick := time.NewTicker(heartbeatWhileWorking)
func stillWorking(msg *nats.Msg, done <-chan struct{}, every time.Duration) {
tick := time.NewTicker(every)
defer tick.Stop()
for {
select {
+40 -10
View File
@@ -405,16 +405,15 @@ func TestEverySubjectTheControllerFollowsDecodesToAKind(t *testing.T) {
// The bus waits a fixed time to be told a message was taken and then redelivers, which is right for
// a consumer that died and wrong for one that is busy. Acting on a merge builds modules — minutes
// against a thirty-second window — and the same merge was handed over five times while the first
// build was still running (2026-09-28). Here the window is two seconds and the work takes six.
// build was still running (2026-09-28). Here the window is a second and the work takes three.
func TestNatsWorkSlowerThanTheWindowIsNotHandedOverAgain(t *testing.T) {
// Work that outlives a window of a second three times over, said while it runs four times a window (novox/hq
// issue 717: the test once set its window before the serving loop asserted the controller's own of thirty
// seconds, so it waited nine seconds and passed with the heartbeat removed).
before := heartbeatWhileWorking
heartbeatWhileWorking = 250 * time.Millisecond
t.Cleanup(func() { heartbeatWhileWorking = before })
js := aBus(t)
// The controller's own consumer, with a window short enough to outlive in a test.
if err := js.EnsureConsumer(broker.Consumer{
Name: broker.ControllerName, Stream: "CONTROL", Push: true, AckWaitSeconds: 2,
Why: "a window short enough to outlive in a test",
}); err != nil {
t.Fatal(err)
}
var mu sync.Mutex
handled := 0
slow := make(chan struct{})
@@ -426,10 +425,32 @@ func TestNatsWorkSlowerThanTheWindowIsNotHandedOverAgain(t *testing.T) {
first := handled == 1
mu.Unlock()
if first {
time.Sleep(6 * time.Second)
time.Sleep(3 * time.Second)
close(slow)
}
}}
// The controller's own consumer, given a window short enough to outlive in a test once the serving loop has
// asserted it and bound to it: asserted after this, it would be the controller's window of thirty seconds,
// which is how this test once waited out a window it never had.
for deadline := time.Now().Add(10 * time.Second); ; {
info, err := js.Context().ConsumerInfo("CONTROL", broker.ControllerName)
if err == nil && info.PushBound {
break
}
if time.Now().After(deadline) {
t.Fatalf("the serving loop never bound the controller's consumer: %v", err)
}
time.Sleep(10 * time.Millisecond)
}
if err := js.EnsureConsumer(broker.Consumer{
Name: broker.ControllerName, Stream: "CONTROL", Push: true, AckWaitSeconds: 1,
Why: "a window short enough to outlive in a test",
}); err != nil {
t.Fatal(err)
}
if info, err := js.Context().ConsumerInfo("CONTROL", broker.ControllerName); err != nil || info.Config.AckWait != time.Second {
t.Fatalf("the controller's consumer does not have the test's window of a second: %v", err)
}
body, err := json.Marshal(Report{Node: "anchor", Declared: "d1", Applied: []string{"store"}})
if err != nil {
@@ -444,12 +465,21 @@ func TestNatsWorkSlowerThanTheWindowIsNotHandedOverAgain(t *testing.T) {
t.Fatal("the slow work never finished")
}
// A moment for a redelivery to arrive, if the bus were going to send one.
time.Sleep(3 * time.Second)
time.Sleep(1500 * time.Millisecond)
mu.Lock()
defer mu.Unlock()
if handled != 1 {
t.Fatalf("one report was handled %d times, so slow work is run again while it is running", handled)
}
// Not handed over again at all: the loop drops a copy of what it is already doing, so the count above alone
// would pass on a bus that redelivered the work every window; the bus's own count says it did not.
info, err := js.Context().ConsumerInfo("CONTROL", broker.ControllerName)
if err != nil {
t.Fatal(err)
}
if info.NumRedelivered != 0 {
t.Fatalf("the bus handed the report over again %d time(s) while it was being worked on", info.NumRedelivered)
}
}
// slowly is a listener that runs whatever it was given.
+5
View File
@@ -167,6 +167,11 @@ const (
// is older than the tool.
var ErrNothingServes = errors.New("nothing serves it")
// ErrGrantRefused is the bus refusing this principal's own call: its grants do not name the subject
// asked. Distinct from ErrNothingServes (nobody there to answer) — this is the asker's own grant, not
// the holder's absence (novox/hq ADR 0380 §9).
var ErrGrantRefused = errors.New("the bus refused the ask")
// ModuleToolOn is where one machine's instance of a module answers a tool: the module's tool subject
// with the machine as its last token — the subject the node tools bind beside the shared one.
func ModuleToolOn(module, tool, node string) string { return ToolSubject(module, tool) + "." + node }
+47
View File
@@ -0,0 +1,47 @@
package link
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/testbus"
)
// A mesh seat's verb asked in the caller's own name goes on the subject that names its caller (novox/hq issue
// 365), so the holder knows who asks from the subject alone: how the vault takes a value from the controller
// and nobody else (design 57's V2, issue 675).
func TestAMeshSeatsVerbIsAskedInTheCallersOwnName(t *testing.T) {
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer conn.Close()
heard := make(chan string, 1)
sub, err := conn.Subscribe(broker.CalledPattern(SeatToolSubject("mesh-vault", "accept")), func(m *nats.Msg) {
heard <- m.Subject
raw, _ := json.Marshal(Answer{Result: json.RawMessage(`{"call":"c1"}`)})
_ = m.Respond(raw)
})
if err != nil {
t.Fatal(err)
}
defer sub.Unsubscribe()
answer, err := AskMeshSeatToolAs(context.Background(), conn, broker.ControllerName, "mesh-vault", "accept",
map[string]any{"provider": "novox", "provision": "postgres"}, 5*time.Second)
if err != nil || string(answer.Result) != `{"call":"c1"}` {
t.Fatalf("%+v %v", answer, err)
}
subject := <-heard
if subject != "mesh.seat.mesh-vault.call.accept.controller" || !strings.HasSuffix(subject, "."+broker.ControllerName) {
t.Errorf("asked on %s", subject)
}
if _, err := AskMeshSeatToolAs(context.Background(), conn, "not a user!", "mesh-vault", "accept", nil, time.Second); err == nil {
t.Error("a caller that is no bus user was asked in")
}
}
+16
View File
@@ -0,0 +1,16 @@
package link
import (
"testing"
"github.com/novox/mesh-controller/internal/broker"
)
// The provisioner client is granted the vault's secret-changed at the address the vault's event lands on, so it
// hears a rotation (design 57 V4): the grant's subject and the event's are one.
func TestTheClientsGrantIsWhereTheVaultsSecretChangedLands(t *testing.T) {
if got := EventSubject("mesh-vault", "secret-changed"); got != broker.VaultSecretChangedSubject {
t.Errorf("the vault's secret-changed lands on %s, and the client is granted %s", got,
broker.VaultSecretChangedSubject)
}
}