Compare commits
179
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6be5679a0d | ||
|
|
af83854956 | ||
|
|
df2162e381 | ||
|
|
a775b42e4d | ||
|
|
481a83e11c | ||
|
|
28bece26b0 | ||
|
|
816df217cd | ||
|
|
c35c27f664 | ||
|
|
d350c98823 | ||
|
|
29757dabc3 | ||
|
|
95a265e730 | ||
|
|
d6c267d2f0 | ||
|
|
e03d8d3c7b | ||
|
|
a91386b3b5 | ||
|
|
8b980a9480 | ||
|
|
9fcb14e741 | ||
|
|
14f464e8bc | ||
|
|
1e1a1602e1 | ||
|
|
5e3639c32f | ||
|
|
ed63e5b026 | ||
|
|
2d95c0eb59 | ||
|
|
7b37e7e1d6 | ||
|
|
b5c7bf13ea | ||
|
|
e7e20e3d13 | ||
|
|
afc327392e | ||
|
|
809288da69 | ||
|
|
4116809162 | ||
|
|
ea8d9c5f56 | ||
|
|
6851ee1657 | ||
|
|
99f8c8c050 | ||
|
|
65db969f72 | ||
|
|
3640447711 | ||
|
|
0fdec2bd60 | ||
|
|
e0cfcabe60 | ||
|
|
0b0f879e86 | ||
|
|
24f257e929 | ||
|
|
9a8636a8ed | ||
|
|
662ab2a3b1 | ||
|
|
827628164a | ||
|
|
e63d707eac | ||
|
|
cb47de8517 | ||
|
|
5e373af70d | ||
|
|
3f89c96d20 | ||
|
|
64015b5952 | ||
|
|
e873cb3b86 | ||
|
|
e88e0c825c | ||
|
|
5bf7a4b656 | ||
|
|
22bff8198c | ||
|
|
3501164f4b | ||
|
|
4a614cae26 | ||
|
|
99c36d8ddf | ||
|
|
4bfae38a73 | ||
|
|
1176df0cd2 | ||
|
|
60c432838b | ||
|
|
78e5de538b | ||
|
|
a5b8ae1a72 | ||
|
|
447a8d0359 | ||
|
|
62dce9fec7 | ||
|
|
7e4f86253e | ||
|
|
20979201eb | ||
|
|
4ee3cc0c7f | ||
|
|
a25c2b4d89 | ||
|
|
312e29478e | ||
|
|
f3a9cc276e | ||
|
|
2ea4f019e5 | ||
|
|
8052fd14f4 | ||
|
|
9f61975c50 | ||
|
|
b82054d930 | ||
|
|
164547645a | ||
|
|
2135fed6cf | ||
|
|
aba2c11a27 | ||
|
|
568140f667 | ||
|
|
1a9190774d | ||
|
|
27710ccabd | ||
|
|
634c77f90f | ||
|
|
1fc006b7dd | ||
|
|
f4b4fb908f | ||
|
|
7572627aff | ||
|
|
d184862b1a | ||
|
|
6c05e42626 | ||
|
|
91a4f1b0ad | ||
|
|
b4affb3efd | ||
|
|
fb31cc4e99 | ||
|
|
bd555e6014 | ||
|
|
c3754e48c8 | ||
|
|
a00c780352 | ||
|
|
5bbe669cfb | ||
|
|
c55b1bcf38 | ||
|
|
6965119bce | ||
|
|
387f547346 | ||
|
|
fce9189579 | ||
|
|
a2402c4873 | ||
|
|
f917fc60c1 | ||
|
|
0a6fd0cb5c | ||
|
|
a918352ff1 | ||
|
|
d5e4e4e8ab | ||
|
|
07f6078050 | ||
|
|
e40cb291f8 | ||
|
|
e0d7122567 | ||
|
|
c637da2eb7 | ||
|
|
7390652f33 | ||
|
|
db3b5fe63d | ||
|
|
76e025dbbc | ||
|
|
f87c9720ed | ||
|
|
d71cd5cef6 | ||
|
|
31247dff6b | ||
|
|
8b1ab467a9 | ||
|
|
84ede95c4a | ||
|
|
7fd9d893c0 | ||
|
|
0fb4591c9d | ||
|
|
7da4c55752 | ||
|
|
b4819dc1fb | ||
|
|
40e0e8282b | ||
|
|
01ee2ef394 | ||
|
|
8b8a760047 | ||
|
|
db3dd04be7 | ||
|
|
d13890b944 | ||
|
|
b3903446f9 | ||
|
|
e550c87a29 | ||
|
|
ff7661a136 | ||
|
|
84533edf2e | ||
|
|
ea516ece5a | ||
|
|
8c1e04dd37 | ||
|
|
43d3fba501 | ||
|
|
df929b48b5 | ||
|
|
961ec0ef96 | ||
|
|
5a9e997349 | ||
|
|
fadc73f73b | ||
|
|
d206b0dde1 | ||
|
|
2bb0a6461d | ||
|
|
38a634723b | ||
|
|
8fe7ef3b82 | ||
|
|
63c1e3faa2 | ||
|
|
ad22c5e9fe | ||
|
|
d6c516ab44 | ||
|
|
b8e42cf54c | ||
|
|
24c0ec0d32 | ||
|
|
bd24ceafa9 | ||
|
|
3f538027e9 | ||
|
|
8114b45884 | ||
|
|
d7d908cf67 | ||
|
|
e475d9f849 | ||
|
|
e6f27a5f74 | ||
|
|
c678511871 | ||
|
|
4f636b7080 | ||
|
|
49f9269376 | ||
|
|
29fd452628 | ||
|
|
331fa06b54 | ||
|
|
5b2ff5e321 | ||
|
|
4fbfdfad13 | ||
|
|
75943ca7b0 | ||
|
|
fb6631aee1 | ||
|
|
d41bee158c | ||
|
|
110e55cb48 | ||
|
|
c494018074 | ||
|
|
4cdf63443d | ||
|
|
1e9995faba | ||
|
|
f6c401ecd2 | ||
|
|
c2f01bf10b | ||
|
|
be2fc2e9b6 | ||
|
|
aa05629375 | ||
|
|
35a7ff70fe | ||
|
|
747bfa71aa | ||
|
|
316a5a2fc3 | ||
|
|
0edcd2501d | ||
|
|
ac645e2cae | ||
|
|
a479a90fb2 | ||
|
|
3c9068f99a | ||
|
|
250aa73a56 | ||
|
|
e32eef2379 | ||
|
|
070ae53128 | ||
|
|
8c0618cfc5 | ||
|
|
54680238a0 | ||
|
|
200249a84f | ||
|
|
67850457ef | ||
|
|
9d67b0995f | ||
|
|
e9009b7f04 | ||
|
|
9fc97e45c9 | ||
|
|
1583fa85d9 |
@@ -23,8 +23,7 @@ marked executable — is **launched** rather than imported (novox/hq ADR 0188):
|
||||
as a child with its own environment and speaks MCP over stdio to it, `tools/list` once and
|
||||
`tools/call` per call. A tool it lists as `<seat>.<verb>` is the seat's implementation. A child that
|
||||
exits is named in the log and started again on its next call. So a tools bundle may be written in
|
||||
any language; the mesh's SDK for each is the stdio loop and nothing more (`node-tools/src/launch.ts`
|
||||
is the runtime's side of it).
|
||||
any language; the mesh's SDK for each is the stdio loop and nothing more (`node-tools/internal/launch` is the runtime's side of it).
|
||||
|
||||
Everything hard — dispatch, collection, duplicate-name safety — is the sdk's. This is the thin
|
||||
wrapper that binds the bus and loads the modules. Keeping the bus client here, out of the sdk, is
|
||||
@@ -45,7 +44,7 @@ MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME where thei
|
||||
MESH_BROKER_URL a plain URL instead of the credential, for the bootstrap case
|
||||
```
|
||||
|
||||
`node dist/main.js`. On a node the controller composes the variables and the host supervises the
|
||||
`node dist/main.js`. On a node the mesh composes the variables and the node-engine supervises the
|
||||
process like any other host-side workload (novox/hq to-be 38). As `node-tools` the same process is
|
||||
the console: MCP on `127.0.0.1:4270` (or `MESH_CONSOLE_LISTEN`).
|
||||
|
||||
@@ -66,14 +65,14 @@ answer took, its size in bytes, how many modules and tools it announced, whether
|
||||
fit the bus, when it was last heard — and who was expected and not heard).
|
||||
|
||||
`mesh_search` also finds a tool by the shell command it replaces (novox/hq ADR 0245): each seat verb and
|
||||
module tool says what it replaces in the controller's records (`replaces`, read from the controller's
|
||||
module tool says what it replaces in the indexer's records (`replaces`, read from its
|
||||
`tools` and `modules` answers), so `journalctl -u x`, `systemctl status x` or `docker ps` puts the verb
|
||||
for it first, and says the command it matched as `instead_of`. A word also matches through a short table
|
||||
of the words agents use for the mesh's (`logs` finds the journal). Best first: a replaced command, then
|
||||
words in a name or a replaced command, then words in a description; seats before modules on a tie.
|
||||
|
||||
Discovery asks the bus (ADR 0197): every runtime answers the NATS services protocol's `$SRV.PING` and
|
||||
`$SRV.INFO` with what it serves at that moment, and the console reads where the controller's records
|
||||
`$SRV.INFO` with what it serves at that moment, and the console reads where the assigner seat's records
|
||||
place each module. The console waits at least 750 ms, and up to 5 s for every runtime that answered
|
||||
PING to send what it serves — a large runtime's answer crossing the bus to a broker on another machine
|
||||
can arrive well after the rest. A runtime that said it is there and did not say what it serves, or
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
The node's tool runtime as a module (novox/hq ADR 0175, to-be 38 WP3). Assigned to a machine, it is
|
||||
one process the host runs from this bundle, as the operator's account: it serves every assigned
|
||||
module's tools and every held seat's verbs on the bus, and answers MCP on the machine's loopback —
|
||||
the console (design 34). The controller composes the process (which bundles to load, where the
|
||||
the console (design 34). The mesh composes the process (which bundles to load, where the
|
||||
credential is, whose machine it is); this manifest says only what the machine must have for it: the
|
||||
interpreter, a place for the credential, the loopback port, and leave to call every tool.
|
||||
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
handacts "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshhandacts"
|
||||
"git.novox.be/novox/mesh-sdk/go/busretry"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
|
||||
// The gate's reach through the runner's own connection, in this one place (design 46 7-approval-mark §7, §13, §14): its
|
||||
// state approvals, its publish of an ask under the marked module's name, and its read of the router's record.
|
||||
|
||||
// approvalsState is the runner's state approvals over its connection: a lost compare-and-set is stdio.ErrChanged, as
|
||||
// operator-channel's client reads it.
|
||||
type approvalsState struct {
|
||||
conn *bus.Conn
|
||||
module string
|
||||
}
|
||||
|
||||
func (s approvalsState) Get(key string) (*stdio.StateEntry, error) {
|
||||
e, err := s.conn.StateGet(s.module, approval.StateName, key)
|
||||
if err != nil || e == nil {
|
||||
return nil, err
|
||||
}
|
||||
return &stdio.StateEntry{Key: e.Key, Value: e.Value, Revision: e.Revision, At: e.At}, nil
|
||||
}
|
||||
|
||||
func (s approvalsState) Create(key string, value any) (uint64, error) {
|
||||
raw, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return changedAs(s.conn.StateCreate(s.module, approval.StateName, key, raw))
|
||||
}
|
||||
|
||||
func (s approvalsState) Update(key string, value any, revision uint64) (uint64, error) {
|
||||
raw, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return changedAs(s.conn.StateUpdate(s.module, approval.StateName, key, raw, revision))
|
||||
}
|
||||
|
||||
func (s approvalsState) Put(key string, value any) (uint64, error) {
|
||||
raw, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return s.conn.StatePut(s.module, approval.StateName, key, raw)
|
||||
}
|
||||
|
||||
func (s approvalsState) Keys() ([]string, error) {
|
||||
return s.conn.StateKeys(s.module, approval.StateName)
|
||||
}
|
||||
|
||||
func changedAs(rev uint64, err error) (uint64, error) {
|
||||
if errors.Is(err, bus.ErrStateChanged) {
|
||||
return 0, stdio.ErrChanged
|
||||
}
|
||||
return rev, err
|
||||
}
|
||||
|
||||
// Bound says the runner's membership binds its state approvals now (§13): a membership issued later binds it with no restart.
|
||||
func (s approvalsState) Bound() bool {
|
||||
m := s.conn.Membership(s.module)
|
||||
if m == nil {
|
||||
return false
|
||||
}
|
||||
for _, st := range m.State {
|
||||
if st.Name == approval.StateName && st.Writes {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// publishAsk says an accept on a seat under the module's name, on the runner's own account (§7).
|
||||
func publishAsk(conn *bus.Conn) func(module, subject string, body any, id string) error {
|
||||
return func(module, subject string, body any, id string) error {
|
||||
raw, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = conn.SeatPublish(module, subject, raw, id)
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// routerRecord reads the router's record `<module>.<id>` of the seat's state asks, on the runner's own account (§14).
|
||||
func routerRecord(conn *bus.Conn) func(module, id string) (*stdio.StateEntry, error) {
|
||||
return func(module, id string) (*stdio.StateEntry, error) {
|
||||
e, err := conn.SeatRecord(module, "", id)
|
||||
if err != nil || e == nil {
|
||||
return nil, err
|
||||
}
|
||||
return &stdio.StateEntry{Key: e.Key, Value: e.Value, Revision: e.Revision, At: e.At}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// boundState is the runner's key-ttl state bound (lead answers A-157, A-158).
|
||||
type boundState struct {
|
||||
conn *bus.Conn
|
||||
module string
|
||||
}
|
||||
|
||||
// BoundState is the runner's state its bound keys live in.
|
||||
const BoundState = "bound"
|
||||
|
||||
func (b boundState) CreateTTL(key string, value any, ttl time.Duration) error {
|
||||
raw, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = b.conn.StateCreateTTL(b.module, BoundState, key, raw, ttl)
|
||||
return err
|
||||
}
|
||||
|
||||
func (b boundState) Delete(key string) error { return b.conn.StateDelete(b.module, BoundState, key) }
|
||||
|
||||
func (b boundState) Watch(pattern string, expired func(key string)) (func(), error) {
|
||||
return b.conn.StateWatch(b.module, BoundState, pattern, func(c bus.StateChange) error {
|
||||
if c.Op == "expire" {
|
||||
expired(c.Key)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// warranted asks mesh-hand-acts.warranted for one acted ask, each try a kept call open while open says so (design 46
|
||||
// 7-approval-mark §16; lead answer A-157).
|
||||
func warranted(conn *bus.Conn) func(asker, ask string, open func() (bool, error)) error {
|
||||
return func(asker, ask string, open func() (bool, error)) error {
|
||||
_, err := busretry.DoKept("mesh-hand-acts.warranted of "+asker+"."+ask,
|
||||
busretry.Open{Key: asker + "." + ask, Open: func(string) (bool, error) { return open() }, Logf: log.Printf},
|
||||
func() (bus.Answered, error) {
|
||||
return conn.Ask("seat:"+handacts.Seat+"."+handacts.VerbWarranted, handacts.WarrantedInput{Asker: asker, Ask: ask},
|
||||
"mesh.seat."+handacts.Seat+".tool."+handacts.VerbWarranted)
|
||||
}, nil)
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// hearOne waits on the runner's connection for the first event on the subjects whose body match takes (design 34
|
||||
// every-event §36: a marked wait's ask's decided).
|
||||
func hearOne(conn *bus.Conn) func(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (bool, error) {
|
||||
return func(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (bool, error) {
|
||||
_, ok, _, err := conn.HearFirst(ctx, subjects, within, match)
|
||||
return ok, err
|
||||
}
|
||||
}
|
||||
|
||||
// gateTools is the gate as mesh_approvals reads it, with its state's fill read from the bucket (§13, §18).
|
||||
type gateTools struct {
|
||||
*approval.Gate
|
||||
state approvalsState
|
||||
}
|
||||
|
||||
func (t gateTools) Fill() (uint64, int64, error) {
|
||||
info, err := t.state.conn.StateInfo(t.state.module, approval.StateName)
|
||||
if err != nil || info == nil {
|
||||
return 0, 0, fmt.Errorf("the state %s's fill cannot be read: %v", approval.StateName, err)
|
||||
}
|
||||
return info.Bytes, info.MaxBytes, nil
|
||||
}
|
||||
@@ -0,0 +1,295 @@
|
||||
//go:build integration
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
|
||||
seat "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
|
||||
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatdef"
|
||||
"github.com/nats-io/nats.go"
|
||||
runnerapproval "github.com/novox/mesh-tools/node-tools/internal/approval"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// marked is the fixture marked module's catalog half: its build src2:1 marks its tool wipe (design 46 7-approval-mark §1).
|
||||
type marked struct{}
|
||||
|
||||
var wipeEntry = catalogseat.CatalogEntry{Module: "shell", Fp: "src2:1", Manifest: map[string]any{"needs-approval": []any{"wipe"}}}
|
||||
|
||||
func (marked) Ready() error { return nil }
|
||||
func (marked) Revision() string { return "7" }
|
||||
func (marked) Served(fp string) (approval.Entry, bool, string) {
|
||||
return runnerapproval.Entry(wipeEntry), fp == wipeEntry.Fp, "not-found"
|
||||
}
|
||||
func (marked) Newest(m string) (approval.Entry, bool) {
|
||||
return runnerapproval.Entry(wipeEntry), m == "shell"
|
||||
}
|
||||
func (marked) Definitions() map[string]seatdef.Definition { return map[string]seatdef.Definition{} }
|
||||
func (marked) Entries() []approval.Entry { return []approval.Entry{runnerapproval.Entry(wipeEntry)} }
|
||||
|
||||
// bed is the runner's gate wired as main wires it, over the runner's own connection to a real bus, with stand-ins of the
|
||||
// operator channel's router (it takes each ask into its record asks, as the router writes it) and of the hand-act seat.
|
||||
type bed struct {
|
||||
t *testing.T
|
||||
conn *bus.Conn
|
||||
asks nats.KeyValue
|
||||
taken chan struct{}
|
||||
done chan struct{}
|
||||
|
||||
mu sync.Mutex
|
||||
asked []seat.Ask
|
||||
recorded []string
|
||||
holders map[string][]byte
|
||||
}
|
||||
|
||||
func newBed(t *testing.T, bindState bool) *bed {
|
||||
t.Helper()
|
||||
mesh := mt.New(t)
|
||||
nc, err := nats.Connect(mt.URL(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(nc.Close)
|
||||
js, _ := nc.JetStream()
|
||||
if _, err := js.AddStream(&nats.StreamConfig{Name: "SEAT_OPERATOR_CHANNEL", Subjects: []string{"mesh.seat.operator-channel.accept.>"},
|
||||
Retention: nats.WorkQueuePolicy}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b := &bed{t: t, asks: mesh.Bucket(t, "messenger_asks"), taken: make(chan struct{}, 1), done: make(chan struct{}),
|
||||
holders: map[string][]byte{"operator-channel": []byte(`{"seat":"operator-channel","scope":"mesh","holders":[{"module":"messenger"}]}`)}}
|
||||
t.Cleanup(func() { close(b.done) })
|
||||
mesh.Bucket(t, "node-tools_approvals")
|
||||
mesh.BucketKeyTTL(t, "node-tools_bound")
|
||||
|
||||
// The router's stand-in: each ask accepted under the marked module's name is taken into its record, state open.
|
||||
sub, err := js.PullSubscribe("mesh.seat.operator-channel.accept.>", "router", nats.BindStream("SEAT_OPERATOR_CHANNEL"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
go func() {
|
||||
for {
|
||||
select {
|
||||
case <-b.done:
|
||||
return
|
||||
default:
|
||||
}
|
||||
msgs, _ := sub.Fetch(1, nats.MaxWait(200*time.Millisecond))
|
||||
for _, m := range msgs {
|
||||
var a seat.Ask
|
||||
if err := json.Unmarshal(m.Data, &a); err == nil {
|
||||
module := strings.TrimPrefix(m.Subject, "mesh.seat.operator-channel.accept.ask.")
|
||||
b.mu.Lock()
|
||||
b.asked = append(b.asked, a)
|
||||
b.mu.Unlock()
|
||||
b.write(seat.AsksByAskerIDValue{Asker: module, Key: module + "." + a.ID, State: "open", Headline: a.Headline,
|
||||
Expires: a.Expires, Opened: operatorchannelclient.FormatTime(time.Now()), OfferedOn: []string{"telegram"}})
|
||||
select {
|
||||
case b.taken <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
_ = m.Ack()
|
||||
}
|
||||
}
|
||||
}()
|
||||
// The hand-act seat's stand-in: warranted {asker, ask} answered, each one kept.
|
||||
if _, err := nc.Subscribe("mesh.seat.mesh-hand-acts.>", func(m *nats.Msg) {
|
||||
b.mu.Lock()
|
||||
b.recorded = append(b.recorded, string(m.Data))
|
||||
b.mu.Unlock()
|
||||
_ = m.Respond([]byte(`{"result":{"recorded":true}}`))
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
runner := mt.MembershipOf("node-tools", "n1", false, nil)
|
||||
if bindState {
|
||||
runner.State = []bus.StateIssued{{Name: approval.StateName, Bucket: "node-tools_approvals", Writes: true},
|
||||
{Name: BoundState, Bucket: "node-tools_bound", Writes: true}}
|
||||
}
|
||||
shell := mt.MembershipOf("shell", "n1", false, nil)
|
||||
shell.SeatTraffic = &bus.SeatTraffic{Publish: []string{seat.AcceptSubjectAsk("shell")},
|
||||
Records: []string{"$JS.API.DIRECT.GET.KV_messenger_asks.$KV.messenger_asks.shell.>"}}
|
||||
mesh.Issue(t, runner)
|
||||
mesh.Issue(t, shell)
|
||||
b.conn, err = bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "n1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b.conn.Logf = func(string, ...any) {}
|
||||
t.Cleanup(b.conn.Close)
|
||||
b.conn.Follow("node-tools")
|
||||
b.conn.Follow("shell")
|
||||
mt.Until(t, func() error {
|
||||
if b.conn.Membership("node-tools") == nil || b.conn.Membership("shell") == nil {
|
||||
return errors.New("not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return b
|
||||
}
|
||||
|
||||
// gate is a runner's gate as main makes it; record says how the run's recording goes beside the call (nil: at once).
|
||||
func (b *bed) gate(record func(func())) *approval.Gate {
|
||||
if record == nil {
|
||||
record = func(f func()) { f() }
|
||||
}
|
||||
g := approval.New(approval.Deps{Node: "n1", Runner: "node-tools", Catalog: marked{}, Logf: b.t.Logf,
|
||||
State: approvalsState{conn: b.conn, module: "node-tools"},
|
||||
Channel: &operatorchannelclient.GateChannel{State: approvalsState{conn: b.conn, module: "node-tools"}, KeyPrefix: "n1.",
|
||||
Publish: publishAsk(b.conn), ReadRecord: routerRecord(b.conn)},
|
||||
Warranted: warranted(b.conn), Bounds: boundState{conn: b.conn, module: "node-tools"}, Go: record,
|
||||
Holder: runnerapproval.HoldersIn(func() (map[string][]byte, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
return b.holders, nil
|
||||
}),
|
||||
// The pause while the router takes the ask ends when the stand-in has taken one: no sleeping.
|
||||
After: func(time.Duration) <-chan time.Time {
|
||||
c := make(chan time.Time, 1)
|
||||
go func() {
|
||||
select {
|
||||
case <-b.taken:
|
||||
case <-b.done:
|
||||
}
|
||||
c <- time.Now()
|
||||
}()
|
||||
return c
|
||||
}})
|
||||
_ = g.Unserved("shell", "src2:1", []string{"wipe"})
|
||||
return g
|
||||
}
|
||||
|
||||
func (b *bed) write(rec seat.AsksByAskerIDValue) {
|
||||
raw, _ := json.Marshal(rec)
|
||||
if _, err := b.asks.Put(rec.Key, raw); err != nil {
|
||||
b.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// decide writes the router's record of the newest ask as the router writes the operator's choice: chosen, with its warrant.
|
||||
func (b *bed) decide(option string) seat.Ask {
|
||||
b.mu.Lock()
|
||||
a := b.asked[len(b.asked)-1]
|
||||
b.mu.Unlock()
|
||||
e, err := b.asks.Get("shell." + a.ID)
|
||||
if err != nil {
|
||||
b.t.Fatal(err)
|
||||
}
|
||||
var rec seat.AsksByAskerIDValue
|
||||
_ = json.Unmarshal(e.Value(), &rec)
|
||||
rec.State = "chosen"
|
||||
rec.Warrant = &seat.Warrant{Ask: a.ID, Asker: "shell", Outcome: operatorchannelclient.OutcomeChosen, Option: option,
|
||||
Level: operatorchannelclient.Approve, At: operatorchannelclient.FormatTime(time.Now()), AskDigest: operatorchannelclient.Digest(a),
|
||||
Channel: "telegram", By: &seat.Person{Who: operatorchannelclient.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
|
||||
b.write(rec)
|
||||
return a
|
||||
}
|
||||
|
||||
func wipe(args string) approval.Call {
|
||||
return approval.Call{Address: "shell.wipe", Module: "shell", Name: "wipe", Fingerprint: "src2:1", Caller: "agent",
|
||||
Args: json.RawMessage(args), Description: "Wipes a scratch directory."}
|
||||
}
|
||||
|
||||
// Design 46 7-approval-mark §22: the runner's gate against the real bus with a fixture marked module: a call asked and taken
|
||||
// by the operator channel's stand-in, the operator's Approve written as the router's record, the call again run once and
|
||||
// recorded in the hand-act log, mesh_approvals' list, a restart between the run and warranted's answer recorded once; a
|
||||
// Decline runs nothing; a gate missing a part of its check runs nothing.
|
||||
func TestAMarkedCallOnTheBusIsAskedRunOnTheApprovalAndRecorded(t *testing.T) {
|
||||
t.Run("approved: asked, run once, recorded, listed", func(t *testing.T) {
|
||||
b := newBed(t, true)
|
||||
g := b.gate(nil)
|
||||
runs := 0
|
||||
run := func() (any, error) { runs++; return "wiped", nil }
|
||||
if _, err := g.Run(wipe(`{"dir":"scratch"}`), run); err == nil ||
|
||||
!strings.HasPrefix(err.Error(), "Not run: this needs the operator's approval. Asked on telegram at ") || runs != 0 {
|
||||
t.Fatalf("first call: %v, runs %d", err, runs)
|
||||
}
|
||||
if _, err := g.Run(wipe(`{"dir":"scratch"}`), run); err == nil || !strings.Contains(err.Error(), "still waiting for the operator's answer") {
|
||||
t.Fatalf("before the answer: %v", err)
|
||||
}
|
||||
a := b.decide(approval.OptionApprove)
|
||||
got, err := g.Run(wipe(`{"dir":"scratch"}`), run)
|
||||
answered, ok := got.(approval.Answered)
|
||||
if err != nil || !ok || answered.Answer != "wiped" || !strings.Contains(answered.Approval, "ask "+a.ID) || runs != 1 {
|
||||
t.Fatalf("on the approval: %+v, %v, runs %d", got, err, runs)
|
||||
}
|
||||
if _, err := g.Run(wipe(`{"dir":"scratch"}`), run); err == nil || runs != 1 {
|
||||
t.Fatalf("the warrant ran twice: %v, runs %d", err, runs)
|
||||
}
|
||||
b.mu.Lock()
|
||||
recorded := strings.Join(b.recorded, "\n")
|
||||
b.mu.Unlock()
|
||||
if !strings.Contains(recorded, a.ID) || !strings.Contains(recorded, "shell") {
|
||||
t.Fatalf("hand-act log heard %q", recorded)
|
||||
}
|
||||
list, err := g.List()
|
||||
if err != nil || len(list) == 0 || list[len(list)-1].Ask != a.ID || list[len(list)-1].State != "run, recorded" {
|
||||
t.Fatalf("mesh_approvals lists %+v, %v", list, err)
|
||||
}
|
||||
})
|
||||
t.Run("a restart between the run and warranted's answer records the run once", func(t *testing.T) {
|
||||
b := newBed(t, true)
|
||||
stopped := b.gate(func(func()) {}) // the runner stops before warranted answers
|
||||
runs := 0
|
||||
run := func() (any, error) { runs++; return "wiped", nil }
|
||||
_, _ = stopped.Run(wipe(`{"dir":"tmp"}`), run)
|
||||
a := b.decide(approval.OptionApprove)
|
||||
if _, err := stopped.Run(wipe(`{"dir":"tmp"}`), run); err != nil || runs != 1 {
|
||||
t.Fatalf("run: %v, runs %d", err, runs)
|
||||
}
|
||||
if list, _ := stopped.List(); len(list) != 1 || list[0].State != "run, not yet recorded" {
|
||||
t.Fatalf("before the restart: %+v", list)
|
||||
}
|
||||
started := b.gate(nil)
|
||||
started.Recover()
|
||||
started.Recover()
|
||||
b.mu.Lock()
|
||||
n := len(b.recorded)
|
||||
b.mu.Unlock()
|
||||
if list, _ := started.List(); n != 1 || len(list) != 1 || list[0].Ask != a.ID || list[0].State != "run, recorded" || runs != 1 {
|
||||
t.Fatalf("after the restart: recorded %d, %+v, runs %d", n, list, runs)
|
||||
}
|
||||
})
|
||||
t.Run("declined: refused, nothing runs", func(t *testing.T) {
|
||||
b := newBed(t, true)
|
||||
g := b.gate(nil)
|
||||
runs := 0
|
||||
run := func() (any, error) { runs++; return nil, nil }
|
||||
_, _ = g.Run(wipe(`{"dir":"scratch"}`), run)
|
||||
b.decide(approval.OptionDecline)
|
||||
if _, err := g.Run(wipe(`{"dir":"scratch"}`), run); err == nil || !strings.Contains(err.Error(), "the operator declined ask") || runs != 0 {
|
||||
t.Fatalf("declined: %v, runs %d", err, runs)
|
||||
}
|
||||
})
|
||||
t.Run("failing closed: no state bound, or nobody to ask, runs nothing and asks nothing", func(t *testing.T) {
|
||||
unbound := newBed(t, false)
|
||||
runs := 0
|
||||
run := func() (any, error) { runs++; return nil, nil }
|
||||
if _, err := unbound.gate(nil).Run(wipe(`{"dir":"scratch"}`), run); err == nil ||
|
||||
!strings.Contains(err.Error(), "binds no state approvals") || runs != 0 || len(unbound.asked) != 0 {
|
||||
t.Fatalf("no state: %v, runs %d", err, runs)
|
||||
}
|
||||
b := newBed(t, true)
|
||||
b.holders = map[string][]byte{}
|
||||
if _, err := b.gate(nil).Run(wipe(`{"dir":"scratch"}`), run); err == nil ||
|
||||
!strings.Contains(err.Error(), "no module holds operator-channel") || runs != 0 {
|
||||
t.Fatalf("no holder: %v, runs %d", err, runs)
|
||||
}
|
||||
if got, err := b.gate(nil).Run(approval.Call{Address: "shell.tidy", Module: "shell", Name: "tidy", Fingerprint: "src2:1",
|
||||
Args: json.RawMessage(`{}`)}, func() (any, error) { return "tidied", nil }); err != nil || got != "tidied" {
|
||||
t.Fatalf("an unmarked tool: %v, %v", got, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -5,6 +5,7 @@
|
||||
// wherever MESH_CONSOLE_LISTEN says — it is also the console, MCP over HTTP on loopback.
|
||||
//
|
||||
// MESH_BROKER_FILE the credential the mesh sealed to this machine for the runtime
|
||||
// MESH_LEDGERS_FILE its own secret ledgers, each module's ledger credential; beside the broker file when unset
|
||||
// MESH_TOOL_MODULES <module>=<entrypoint>,… the bundles to launch
|
||||
// MESH_TOOL_ENV {"<module>": {"<word>": "<value>"}} what each is given (ADR 0192)
|
||||
// MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME their home
|
||||
@@ -13,6 +14,7 @@ package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -23,9 +25,15 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
handacts "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshhandacts"
|
||||
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/alive"
|
||||
runnerapproval "github.com/novox/mesh-tools/node-tools/internal/approval"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/console"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/firstview"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/launch"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/provide"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/runtime"
|
||||
@@ -47,6 +55,8 @@ func main() {
|
||||
until := time.Now().Add(startWait)
|
||||
cred := credential(until)
|
||||
conn := connectPatiently(cred, until)
|
||||
// And the seat list it describes the seats by, read through the seat mesh-catalog (novox/hq ADR 0636 §4).
|
||||
seatList := startSeatList(conn)
|
||||
|
||||
served, err := runtime.ServedModulesFrom(os.Getenv(runtime.ToolModules), cred.Module)
|
||||
if err != nil {
|
||||
@@ -56,7 +66,39 @@ func main() {
|
||||
if err != nil {
|
||||
log.Fatalf("node-tools: %v", err)
|
||||
}
|
||||
stop, err := runtime.Run(conn, served, envs, log.Printf)
|
||||
// Its own secret ledgers: each module's ledger connection, the only one its history is published on (novox/hq ADR
|
||||
// 0597 §9). Followed, so a credential minted or retired is taken with no restart.
|
||||
var opts []runtime.Option
|
||||
if file := runtime.LedgersFileOf(os.Getenv("MESH_BROKER_FILE")); file != "" {
|
||||
ledgers := runtime.FollowLedgerCredentials(file, bus.Dial, log.Printf)
|
||||
defer ledgers.Stop()
|
||||
opts = append(opts, runtime.WithLedgerWriters(ledgers.Writer))
|
||||
}
|
||||
// The indexer's and the assigner's states the console indexes the mesh by, and the gate reads its marks from, come to it as
|
||||
// first views, asked of the owners' own tools and followed on their views families, never read from their state (novox/hq
|
||||
// ADR 0615 §1; design 46 7-approval-mark §5).
|
||||
source, stopViews := firstview.Start(context.Background(), conn, log.Printf)
|
||||
defer stopViews()
|
||||
approvals := approvalsState{conn: conn, module: cred.Module}
|
||||
gate := approval.New(approval.Deps{Node: conn.Node(), Runner: cred.Module, Catalog: source.ApprovalCatalog(definitionsOf(seatList)), Wait: source.WaitCatalog,
|
||||
Findings: runnerapproval.Findings(source.Findings(), conn.Node(), cred.Module, nil), Logf: log.Printf, State: approvals,
|
||||
Channel: &operatorchannelclient.GateChannel{State: approvals, KeyPrefix: conn.Node() + ".", Publish: publishAsk(conn), ReadRecord: routerRecord(conn)},
|
||||
Warranted: warranted(conn), Bounds: boundState{conn: conn, module: cred.Module},
|
||||
Holder: runnerapproval.HoldersIn(func() (map[string][]byte, error) { return source.State(firstview.SeatHoldersRead) }),
|
||||
// The runner's own build, the one a marked wait's act binds (design 34 every-event §35).
|
||||
Build: os.Getenv(runtime.FingerprintWord), Hear: hearOne(conn)})
|
||||
// A failed read or write of the state approvals is made again at the next membership change the runner hears (A-157).
|
||||
conn.OnMembership(func(bus.Membership) { go gate.StateAgain() })
|
||||
opts = append(opts, runtime.WithGate(gate), runtime.WithSeatDefinitions(definitionsOf(seatList)))
|
||||
// A run still unrecorded is asked again at each start and when the hand-act seat's holder is heard again (lead answer A-157).
|
||||
if stopBounds, err := gate.WatchBounds(); err != nil {
|
||||
log.Printf("[mesh-tools] the gate's bound keys are not watched (%v): an unanswered hand-act is said in the log only", err)
|
||||
} else {
|
||||
defer stopBounds()
|
||||
}
|
||||
source.OnSeatHolder(handacts.Seat, gate.Recover)
|
||||
go gate.Recover()
|
||||
stop, err := runtime.Run(conn, served, envs, log.Printf, opts...)
|
||||
if err != nil {
|
||||
log.Fatalf("node-tools: %v", err)
|
||||
}
|
||||
@@ -82,7 +124,9 @@ func main() {
|
||||
node = "?"
|
||||
}
|
||||
who := node + "." + cred.Module
|
||||
up, err = console.Serve(console.NewSurface(conn, who), listen)
|
||||
surface := console.NewSurface(conn, source, who)
|
||||
surface.Approvals = gateTools{Gate: gate, state: approvalsState{conn: conn, module: cred.Module}}
|
||||
up, err = console.Serve(surface, listen)
|
||||
if err != nil {
|
||||
log.Fatalf("node-tools: %v", err)
|
||||
}
|
||||
|
||||
@@ -13,10 +13,17 @@ import (
|
||||
"github.com/novox/mesh-tools/node-tools/internal/runtime"
|
||||
)
|
||||
|
||||
// The runner's start waits for its credential and its bus within the start's bound (issue 1496): one test, two cases, so
|
||||
// design 46 7-approval-mark §22's whole test of the gate adds no test to the module (design 47 G §419, §626).
|
||||
func TestTheRunnersStartWaitsForItsCredentialAndItsBus(t *testing.T) {
|
||||
t.Run("the credential", awaitsCredential)
|
||||
t.Run("the bus", connectsWithin)
|
||||
}
|
||||
|
||||
// A runtime started before its credential is on the machine waits for it, woken by the watch of its directory, and gives
|
||||
// up loudly at its bound; a file that is no credential ends the wait at once (issue 1496: on ace mesh-build-node's and
|
||||
// mesh-build-image's runtimes exited twice on "cannot read the broker credential ... no such file").
|
||||
func TestAwaitCredential(t *testing.T) {
|
||||
func awaitsCredential(t *testing.T) {
|
||||
const good = `{"url":"tls://bus.example:4222","module":"m"}`
|
||||
rereadEvery = time.Hour // only the watch can wake the read in time
|
||||
settle := runtime.Settle
|
||||
@@ -82,7 +89,7 @@ func TestAwaitCredential(t *testing.T) {
|
||||
|
||||
// A bus that refuses the account at a start is asked again until the start's bound, said once; a refusal past it, a pin
|
||||
// that does not match and an address that is no URL give up (issue 1496, issue 058).
|
||||
func TestConnectWithin(t *testing.T) {
|
||||
func connectsWithin(t *testing.T) {
|
||||
refused := errors.New("nats: Authorization Violation")
|
||||
unreachable := errors.New("nats: no servers available for connection")
|
||||
for _, c := range []struct {
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatdef"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatlist"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
|
||||
// The tool runner reads the seat list as every part does (novox/hq ADR 0635 §1, ADR 0636): through the seat mesh-catalog,
|
||||
// from the bootstrapped definitions until it has been read, and following the seat's events. What it holds is given to
|
||||
// catalogue at every change; the console describes the seats from it, so the descriptions are the mesh's own and not a
|
||||
// list compiled into the runner.
|
||||
|
||||
// busOfTheRunner is the reader's bus over the runner's connection: a verb of the seat asked on its subject, an event heard
|
||||
// on the seat's event subject.
|
||||
type busOfTheRunner struct{ conn *bus.Conn }
|
||||
|
||||
func (b busOfTheRunner) Ask(key string, body any) (json.RawMessage, error) {
|
||||
seat, verb, _ := strings.Cut(strings.TrimPrefix(key, "seat:"), ".")
|
||||
answered, err := b.conn.Ask(key, body, "mesh.seat."+seat+".tool."+verb)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return answered.Result, nil
|
||||
}
|
||||
|
||||
func (b busOfTheRunner) Subscribe(pattern string, handle func(string, json.RawMessage) error) error {
|
||||
seat, event, _ := strings.Cut(pattern, ".")
|
||||
_, err := b.conn.Raw(seatdef.EventSubject(seat, event), func(_ string, data []byte) []byte {
|
||||
_ = handle(pattern, data)
|
||||
return nil // an event is answered by nobody
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
// seatListWithin is how long the runner waits for the first read of the seat list before it serves; the read goes on
|
||||
// after it, and the descriptions are the bootstrapped ones until it ends (ADR 0636 §1).
|
||||
const seatListWithin = 5 * time.Second
|
||||
|
||||
// startSeatList starts the runner's reader and gives catalogue its bootstrapped seats at once, then waits for the first
|
||||
// read to end (the reader's Ready) for the bound, so the runner's first descriptions are of the list in the common case.
|
||||
func startSeatList(conn *bus.Conn) *seatlist.Reader {
|
||||
timer := time.NewTimer(seatListWithin)
|
||||
defer timer.Stop()
|
||||
return startSeatListOn(busOfTheRunner{conn}, timer.C, log.Printf)
|
||||
}
|
||||
|
||||
// startSeatListOn is startSeatList over any bus, with the bound as a channel the caller gives.
|
||||
func startSeatListOn(b seatlist.Bus, bound <-chan time.Time, logf func(string, ...any)) *seatlist.Reader {
|
||||
reader, err := seatlist.Start(seatlist.Config{Bus: b, Bootstrap: seatlist.Bootstrapped(),
|
||||
OnChange: func(seats []catalogue.Seat) { catalogue.UseSeats(seats) }})
|
||||
if err != nil {
|
||||
logf("node-tools: %v", fmt.Errorf("the tool runner cannot follow the seat list: %w", err))
|
||||
}
|
||||
if reader == nil {
|
||||
return nil
|
||||
}
|
||||
catalogue.UseSeats(reader.Seats())
|
||||
select {
|
||||
case <-reader.Ready():
|
||||
catalogue.UseSeats(reader.Seats())
|
||||
case <-bound:
|
||||
logf("node-tools: the seat list is not read yet; the runner serves with the bootstrapped seats and reads on")
|
||||
}
|
||||
return reader
|
||||
}
|
||||
|
||||
// definitionsOf is the seat definitions the reader holds, by seat, which carry the verbs' marks (design 46 7-approval-mark
|
||||
// §2, §5); none while the reader did not start.
|
||||
func definitionsOf(reader *seatlist.Reader) func() map[string]seatdef.Definition {
|
||||
return func() map[string]seatdef.Definition {
|
||||
if reader == nil {
|
||||
return nil
|
||||
}
|
||||
return reader.Definitions()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// busOfAStandIn is a seat mesh-catalog that answers after release, or never (a refusal).
|
||||
type busOfAStandIn struct {
|
||||
release chan struct{}
|
||||
asked atomic.Int32
|
||||
}
|
||||
|
||||
func (b *busOfAStandIn) Ask(string, any) (json.RawMessage, error) {
|
||||
b.asked.Add(1)
|
||||
<-b.release
|
||||
return nil, errors.New("the seat does not answer")
|
||||
}
|
||||
|
||||
func (b *busOfAStandIn) Subscribe(string, func(string, json.RawMessage) error) error { return nil }
|
||||
|
||||
// ADR 0636 §1: the runner waits for the first read of the seat list on the reader's Ready, but only to a bound that the
|
||||
// test gives as a channel; past it, it serves with the bootstrapped seats and says so, and nothing sleeps.
|
||||
func TestTheRunnerWaitsForTheFirstReadOfTheSeatListOnlyToItsBound(t *testing.T) {
|
||||
for _, row := range []struct {
|
||||
name string
|
||||
release bool // the seat ends its first read (here by refusing) before the bound
|
||||
wantSaid string
|
||||
wantReady bool
|
||||
}{
|
||||
{"the first read ends before the bound", true, "", true},
|
||||
{"the bound passes with the seat silent", false, "the seat list is not read yet", false},
|
||||
} {
|
||||
t.Run(row.name, func(t *testing.T) {
|
||||
b := &busOfAStandIn{release: make(chan struct{})}
|
||||
bound := make(chan time.Time, 1)
|
||||
var said []string
|
||||
if row.release {
|
||||
close(b.release)
|
||||
} else {
|
||||
bound <- time.Time{}
|
||||
}
|
||||
r := startSeatListOn(b, bound, func(f string, a ...any) { said = append(said, f) })
|
||||
if r == nil {
|
||||
t.Fatal("no reader")
|
||||
}
|
||||
ended := false
|
||||
select {
|
||||
case <-r.Ready():
|
||||
ended = true
|
||||
default:
|
||||
}
|
||||
if ended != row.wantReady {
|
||||
t.Errorf("first read ended = %v, want %v", ended, row.wantReady)
|
||||
}
|
||||
if got := strings.Join(said, ";"); !strings.Contains(got, row.wantSaid) || (row.wantSaid == "" && got != "") {
|
||||
t.Errorf("said %q, want %q", got, row.wantSaid)
|
||||
}
|
||||
if !row.release {
|
||||
close(b.release)
|
||||
<-r.Ready()
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+18
-5
@@ -3,20 +3,33 @@ module github.com/novox/mesh-tools/node-tools
|
||||
go 1.26.0
|
||||
|
||||
require (
|
||||
git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshhandacts v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-contracts/provisioning/go/clients/meshprovisionerclient v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient v0.0.0-00010101000000-000000000000
|
||||
git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog v0.0.0-00010101000000-000000000000
|
||||
github.com/nats-io/nats.go v1.54.0
|
||||
golang.org/x/sys v0.48.0
|
||||
)
|
||||
|
||||
require (
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468
|
||||
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
|
||||
git.novox.be/novox/mesh-sdk/go v0.0.0-00010101000000-000000000000
|
||||
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op // indirect
|
||||
github.com/google/go-tpm v0.9.8 // indirect
|
||||
github.com/klauspost/compress v1.20.0 // indirect
|
||||
github.com/klauspost/compress v1.20.1 // indirect
|
||||
github.com/minio/highwayhash v1.0.4 // indirect
|
||||
github.com/nats-io/jwt/v2 v2.8.1 // indirect
|
||||
github.com/nats-io/jwt/v2 v2.8.2 // indirect
|
||||
github.com/nats-io/nats-server/v2 v2.11.17
|
||||
github.com/nats-io/nkeys v0.4.16 // indirect
|
||||
github.com/nats-io/nuid v1.0.1 // indirect
|
||||
golang.org/x/crypto v0.57.0 // indirect
|
||||
golang.org/x/time v0.15.0 // indirect
|
||||
golang.org/x/time v0.16.0 // indirect
|
||||
)
|
||||
|
||||
require git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery v0.0.0-00010101000000-000000000000 // indirect
|
||||
|
||||
require git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk v0.0.0-00010101000000-000000000000 // indirect
|
||||
|
||||
+8
-10
@@ -1,15 +1,13 @@
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0=
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
|
||||
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
|
||||
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
|
||||
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op h1:1BOWQJweNyvZMlpAHXGLiZQn9S+QXGcz3xh94lC0w6E=
|
||||
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
|
||||
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
|
||||
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
|
||||
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
|
||||
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
|
||||
github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ=
|
||||
github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
|
||||
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
|
||||
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
|
||||
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
|
||||
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
|
||||
github.com/nats-io/jwt/v2 v2.8.2 h1:XXRgB60MSTnqsRwejQurVDs/hcv2dkt+86GjI+I/bMc=
|
||||
github.com/nats-io/jwt/v2 v2.8.2/go.mod h1:Ag/56sq9OblL4JgdYufDd16Egb17Kr/8WwwuO/forVc=
|
||||
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
|
||||
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
|
||||
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
|
||||
@@ -23,5 +21,5 @@ golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZ
|
||||
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
|
||||
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
|
||||
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
|
||||
golang.org/x/time v0.16.0 h1:vMb6ptszcQMkcwiRTAuNNU50gom6++Q/6gY2hDM6VDE=
|
||||
golang.org/x/time v0.16.0/go.mod h1:rVKOqvZeKvrDKTQiAHJ7wmwP0RzleSphoEA9RcdLA0s=
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
// **A machine heard and its runtime gone is a machine nobody can ask anything.** The node-engine says
|
||||
// every minute that the machine is there; nothing said whether the runtime serving every module's
|
||||
// tools and every held seat's verbs was. Now it says so too, on its own subject, every interval, with
|
||||
// the interval — the controller's watchdog raises `tools-silent` after three missed — and its build.
|
||||
// the interval — the mesh's watchdog raises `tools-silent` after three missed — and its build.
|
||||
// Core NATS, kept by nobody, as the host's heartbeat: a lost one is the next one.
|
||||
package alive
|
||||
|
||||
|
||||
@@ -43,6 +43,16 @@ var Patience = 5 * time.Second
|
||||
// to the laptop, and the hundred-byte PING answers queued behind it.
|
||||
const Digest = "digest"
|
||||
|
||||
// Capabilities is the service metadata key a runtime states what it can do under, a comma-separated list of capabilities
|
||||
// (runtime.CallCapabilities): its machine's node-engine asks the runtime's PING and takes them into the node's record, so a
|
||||
// module needing one is placed only where the runtime states it.
|
||||
const Capabilities = "capabilities"
|
||||
|
||||
// Fingerprint is the service metadata key the tool runner states its own fingerprint under, beside Capabilities: the build
|
||||
// its composed environment names (MESH_FINGERPRINT, design 47 G §424), so the node-engine ties the capabilities it takes
|
||||
// into the node's record to the runner that stated them (design 46 approval mark §6, lead answer A-203).
|
||||
const Fingerprint = "fingerprint"
|
||||
|
||||
// Shortened is the service metadata key a runtime sets when its announcement was cut to fit the bus.
|
||||
const Shortened = "shortened"
|
||||
|
||||
|
||||
@@ -0,0 +1,592 @@
|
||||
package approval
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
|
||||
seat "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
|
||||
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
gate "git.novox.be/novox/mesh-sdk/go/approval"
|
||||
"git.novox.be/novox/mesh-sdk/go/keys"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatdef"
|
||||
"git.novox.be/novox/mesh-sdk/go/views"
|
||||
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// mem is the runner's state approvals as a fixture (meshtest.State).
|
||||
type mem = mt.State
|
||||
|
||||
func newMem() *mem { return mt.NewState() }
|
||||
|
||||
// world is a gate on a fixture mesh: the catalog half, the router's records as the router writes them (mesh-catalog's
|
||||
// messenger, asks.go), the asks published, the clock.
|
||||
type world struct {
|
||||
g *gate.Gate
|
||||
state *mem
|
||||
cat *fakeCatalog
|
||||
records map[string]seat.AsksByAskerIDValue // by `<module>.<id>`
|
||||
takes bool // the router takes an ask as it is published
|
||||
asked []seat.Ask
|
||||
now time.Time
|
||||
runs int
|
||||
findings *findings
|
||||
bounds *bounds
|
||||
// holders is the assigner's view of the seats' holders, by key; holdersErr says it is not answered yet.
|
||||
holders map[string][]byte
|
||||
holdersErr error
|
||||
// warranted is what mesh-hand-acts.warranted answers, and recorded each ask it answered for.
|
||||
warranted error
|
||||
recorded []string
|
||||
// deps is the gate's, made anew by rebuild; publish and record are the bus's publish and the router's record read.
|
||||
deps gate.Deps
|
||||
publish func(module, subject string, body any, id string) error
|
||||
record func(module, id string) (*stdio.StateEntry, error)
|
||||
}
|
||||
|
||||
// rebuild makes the gate anew from w.deps, its module's bundle loaded.
|
||||
func (w *world) rebuild() {
|
||||
w.g = gate.New(w.deps)
|
||||
_ = w.g.Unserved("shell", "src2:1", []string{"wipe", "tidy"})
|
||||
}
|
||||
|
||||
// bounds is the runner's key-ttl state bound as a fixture: expire hands a key's expiry to the watch.
|
||||
type bounds struct {
|
||||
keys map[string]time.Duration
|
||||
expired func(string)
|
||||
}
|
||||
|
||||
func (b *bounds) CreateTTL(key string, _ any, ttl time.Duration) error { b.keys[key] = ttl; return nil }
|
||||
func (b *bounds) Delete(key string) error { delete(b.keys, key); return nil }
|
||||
func (b *bounds) Watch(_ string, expired func(string)) (func(), error) {
|
||||
b.expired = expired
|
||||
return func() {}, nil
|
||||
}
|
||||
func (b *bounds) expire(key string) {
|
||||
delete(b.keys, key)
|
||||
b.expired(key)
|
||||
}
|
||||
|
||||
func newWorld() *world {
|
||||
shell := entry("src2:1", []string{"node-login-shell.execute"}, map[string]any{"needs-approval": []any{"wipe"}})
|
||||
w := &world{state: newMem(), records: map[string]seat.AsksByAskerIDValue{}, takes: true, findings: &findings{},
|
||||
bounds: &bounds{keys: map[string]time.Duration{}},
|
||||
holders: map[string][]byte{"operator-channel": []byte(`{"seat":"operator-channel","scope":"mesh","holders":[{"module":"messenger"}]}`)},
|
||||
now: time.Date(2026, 11, 12, 10, 0, 0, 0, time.UTC),
|
||||
cat: &fakeCatalog{revision: "7", byFp: map[string]catalogseat.CatalogEntry{"src2:1": shell},
|
||||
newest: map[string]catalogseat.CatalogEntry{"shell": shell},
|
||||
defs: map[string]seatdef.Definition{"node-login-shell": {Seat: "node-login-shell",
|
||||
Verbs: []seatdef.Verb{{Name: "execute", NeedsApproval: true}}}}}}
|
||||
w.deps = gate.Deps{Node: "n1", Runner: "node-tools", Catalog: w.cat, State: w.state, Bounds: w.bounds,
|
||||
Findings: Findings(w.findings, "n1", "node-tools", func() time.Time { return w.now }),
|
||||
Channel: &operatorchannelclient.GateChannel{State: w.state, KeyPrefix: "n1.", Now: func() time.Time { return w.now },
|
||||
Publish: func(module, subject string, body any, id string) error { return w.publish(module, subject, body, id) },
|
||||
ReadRecord: func(module, id string) (*stdio.StateEntry, error) { return w.record(module, id) }},
|
||||
Go: func(f func()) { f() },
|
||||
Holder: HoldersIn(func() (map[string][]byte, error) { return w.holders, w.holdersErr }),
|
||||
Warranted: func(asker, ask string, open func() (bool, error)) error {
|
||||
if still, _ := open(); !still {
|
||||
return nil
|
||||
}
|
||||
if w.warranted == nil {
|
||||
w.recorded = append(w.recorded, asker+"."+ask)
|
||||
}
|
||||
return w.warranted
|
||||
},
|
||||
Now: func() time.Time { return w.now },
|
||||
After: func(d time.Duration) <-chan time.Time {
|
||||
w.now = w.now.Add(d)
|
||||
c := make(chan time.Time, 1)
|
||||
c <- w.now
|
||||
return c
|
||||
},
|
||||
}
|
||||
w.publish = func(module, subject string, body any, id string) error {
|
||||
if subject != seat.AcceptSubjectAsk(module) {
|
||||
return nil
|
||||
}
|
||||
a := body.(seat.Ask)
|
||||
w.asked = append(w.asked, a)
|
||||
if w.takes {
|
||||
w.records[module+"."+a.ID] = seat.AsksByAskerIDValue{Asker: module, Key: module + "." + a.ID, State: "open",
|
||||
Headline: a.Headline, Expires: a.Expires, Opened: operatorchannelclient.FormatTime(w.now), OfferedOn: []string{"telegram"}}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
w.record = func(module, id string) (*stdio.StateEntry, error) {
|
||||
rec, ok := w.records[module+"."+id]
|
||||
if !ok {
|
||||
return nil, nil
|
||||
}
|
||||
raw, _ := json.Marshal(rec)
|
||||
return &stdio.StateEntry{Key: module + "." + id, Value: raw}, nil
|
||||
}
|
||||
w.rebuild()
|
||||
return w
|
||||
}
|
||||
|
||||
// decide writes the router's record of the newest ask as the router writes a person's choice: its state and warrant.
|
||||
func (w *world) decide(option string, at time.Time) seat.Ask {
|
||||
a := w.asked[len(w.asked)-1]
|
||||
key := "shell." + a.ID
|
||||
rec := w.records[key]
|
||||
rec.State = "chosen"
|
||||
rec.Warrant = &seat.Warrant{Ask: a.ID, Asker: "shell", Outcome: operatorchannelclient.OutcomeChosen, Option: option,
|
||||
Level: operatorchannelclient.Approve, At: operatorchannelclient.FormatTime(at), AskDigest: operatorchannelclient.Digest(a),
|
||||
By: &seat.Person{Who: operatorchannelclient.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
|
||||
w.records[key] = rec
|
||||
return a
|
||||
}
|
||||
|
||||
// do makes a call through the gate, counting the runs.
|
||||
func (w *world) do(c gate.Call) (any, error) {
|
||||
return w.g.Run(c, func() (any, error) { w.runs++; return "done", nil })
|
||||
}
|
||||
|
||||
// call is a fixture call of the marked tool wipe, or of another tool of the module.
|
||||
func call(name, args string) gate.Call {
|
||||
return gate.Call{Address: "shell." + name, Module: "shell", Name: name, Fingerprint: "src2:1", Caller: "agent", Args: json.RawMessage(args),
|
||||
Description: "Wipes a scratch directory.\nMore words."}
|
||||
}
|
||||
|
||||
// Design 46 7-approval-mark §22: the refusals and answers of a marked call, on the fixture mesh, the router's record and
|
||||
// the warrant written as the router writes them.
|
||||
func TestTheRefusalsAndAnswers(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
run func(t *testing.T, w *world)
|
||||
}{
|
||||
{"§9: a secret in the arguments is refused asking nothing", func(t *testing.T, w *world) {
|
||||
_, err := w.do(call("wipe", `{"token":"ghp_0123456789abcdefghijABCDEFGHIJ012345"}`))
|
||||
if err == nil || !strings.Contains(err.Error(), "which no ask may carry") || len(w.asked) != 0 {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§9: more than 16 KiB of arguments is refused asking nothing", func(t *testing.T, w *world) {
|
||||
big, _ := json.Marshal(map[string]string{"text": strings.Repeat("a b ", 5000)})
|
||||
_, err := w.do(call("wipe", string(big)))
|
||||
if err == nil || !strings.Contains(err.Error(), "more than the 16384") || len(w.asked) != 0 {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§10, §12: the published ask's fields, and Not run with the router's record", func(t *testing.T, w *world) {
|
||||
_, err := w.do(call("wipe", `{"dir":"scratch"}`))
|
||||
if err == nil || !strings.HasPrefix(err.Error(), "Not run: this needs the operator's approval. Asked on telegram at ") ||
|
||||
!strings.Contains(err.Error(), "make the same call again within 30 minutes of the approval; the ask expires at ") {
|
||||
t.Fatalf("answer %v", err)
|
||||
}
|
||||
if len(w.asked) != 1 {
|
||||
t.Fatalf("asked %d", len(w.asked))
|
||||
}
|
||||
a := w.asked[0]
|
||||
act, _ := gate.ActOf(call("wipe", `{"dir":"scratch"}`), "n1")
|
||||
d, _ := operatorchannelclient.ActDigest(act)
|
||||
if a.About != d || a.Who != "operator" || a.OnExpiry != gate.OnExpiry || a.Headline != "Run wipe on n1?" ||
|
||||
a.Expires != operatorchannelclient.FormatTime(time.Date(2026, 11, 12, 11, 0, 0, 0, time.UTC)) || len(a.Options) != 2 ||
|
||||
a.Options[0].ID != "approve" || a.Options[1].ID != "decline" || a.Options[0].Binds != d || a.Options[1].Binds != d ||
|
||||
a.Options[1].Level != "approve" {
|
||||
t.Errorf("ask %+v", a)
|
||||
}
|
||||
if !strings.HasPrefix(a.Explanation, "Wipes a scratch directory.\nOn n1, called through the mesh's tools, which cannot tell who called.\ndir: scratch\nFingerprint ") {
|
||||
t.Errorf("explanation %q", a.Explanation)
|
||||
}
|
||||
var k operatorchannelclient.Kept
|
||||
_ = json.Unmarshal(w.state.Vals["n1.ask."+a.ID], &k)
|
||||
if k.Act["arg.dir"] != "scratch" || string(k.Arguments) != `{"dir":"scratch"}` || string(w.state.Vals["n1.act."+keys.EscapeName(d)]) != `"`+a.ID+`"` {
|
||||
t.Errorf("kept %+v, act key %s", k, w.state.Vals["n1.act."+keys.EscapeName(d)])
|
||||
}
|
||||
}},
|
||||
{"§12: with no record within 5 seconds, Not run, asked; not yet taken by the router", func(t *testing.T, w *world) {
|
||||
w.takes = false
|
||||
start := w.now
|
||||
_, err := w.do(call("wipe", `{}`))
|
||||
if err == nil || !strings.HasPrefix(err.Error(), "Not run, asked; not yet taken by the router (ask ") || w.now.Sub(start) > gate.RecordWithin {
|
||||
t.Fatalf("answer %v after %s", err, w.now.Sub(start))
|
||||
}
|
||||
}},
|
||||
{"§10, §17: a second identical call publishes nothing and is told it still waits", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
|
||||
_, err := w.do(call("wipe", `{"dir":"scratch"}`))
|
||||
if len(w.asked) != 1 || err == nil || !strings.HasPrefix(err.Error(), "Not run: still waiting for the operator's answer") || w.runs != 0 {
|
||||
t.Fatalf("asked %d, %v, runs %d", len(w.asked), err, w.runs)
|
||||
}
|
||||
}},
|
||||
{"§17: an ask the router holds back as the fourth open one still waits", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{}`))
|
||||
a := w.asked[0]
|
||||
rec := w.records["shell."+a.ID]
|
||||
rec.State, rec.HeldSince = "held-back", operatorchannelclient.FormatTime(w.now)
|
||||
w.records["shell."+a.ID] = rec
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "still waiting") || len(w.asked) != 1 {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§14, §15: approved, the same call runs once; made again it is already run and asks anew", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
|
||||
w.decide(gate.OptionApprove, w.now.Add(time.Minute))
|
||||
w.now = w.now.Add(2 * time.Minute)
|
||||
if got, err := w.do(call("wipe", `{"dir":"scratch"}`)); err != nil || got.(gate.Answered).Answer != "done" || w.runs != 1 ||
|
||||
!strings.HasPrefix(got.(gate.Answered).Approval, "run on the operator's approval via telegram at 2026-11-12T10:01:00Z, ask ") {
|
||||
t.Fatalf("%v %v, runs %d", got, err, w.runs)
|
||||
}
|
||||
_, err := w.do(call("wipe", `{"dir":"scratch"}`))
|
||||
if err == nil || !strings.Contains(err.Error(), "was already run; asked again") || w.runs != 1 || len(w.asked) != 2 {
|
||||
t.Fatalf("%v, runs %d, asked %d", err, w.runs, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§15: two calls at the same moment run it once", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{}`))
|
||||
w.decide(gate.OptionApprove, w.now)
|
||||
var wg sync.WaitGroup
|
||||
var mu sync.Mutex
|
||||
runs := 0
|
||||
for i := 0; i < 2; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
_, _ = w.g.Run(call("wipe", `{}`), func() (any, error) { mu.Lock(); runs++; mu.Unlock(); return "done", nil })
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if runs != 1 {
|
||||
t.Fatalf("ran %d times", runs)
|
||||
}
|
||||
}},
|
||||
{"§14: approved for other arguments runs nothing", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
|
||||
a := w.decide(gate.OptionApprove, w.now)
|
||||
rec := w.records["shell."+a.ID]
|
||||
other, _ := gate.ActOf(call("wipe", `{"dir":"home"}`), "n1")
|
||||
d, _ := operatorchannelclient.ActDigest(other)
|
||||
a.Options[0].Binds = d // the ask the person was shown bound another act
|
||||
rec.Warrant.AskDigest = operatorchannelclient.Digest(a)
|
||||
w.records["shell."+a.ID] = rec
|
||||
var k operatorchannelclient.Kept
|
||||
_ = json.Unmarshal(w.state.Vals["n1.ask."+a.ID], &k)
|
||||
k.Ask = a
|
||||
raw, _ := json.Marshal(k)
|
||||
w.state.Vals["n1.ask."+a.ID] = raw
|
||||
if _, err := w.do(call("wipe", `{"dir":"scratch"}`)); err == nil || w.runs != 0 || !strings.Contains(err.Error(), "nothing is done") {
|
||||
t.Fatalf("%v, runs %d", err, w.runs)
|
||||
}
|
||||
}},
|
||||
{"§14: approved 31 minutes ago asks anew", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{}`))
|
||||
w.decide(gate.OptionApprove, w.now)
|
||||
w.now = w.now.Add(31 * time.Minute)
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "approved too long ago") || w.runs != 0 || len(w.asked) != 2 {
|
||||
t.Fatalf("%v, runs %d, asked %d", err, w.runs, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§14, §17: declined, the same call is not asked again for 10 minutes", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{}`))
|
||||
w.decide(gate.OptionDecline, w.now)
|
||||
w.now = w.now.Add(9 * time.Minute)
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "declined") || len(w.asked) != 1 || w.runs != 0 {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
w.now = w.now.Add(2 * time.Minute)
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "asked again") || len(w.asked) != 2 {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§14, §17: expired asks anew", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{}`))
|
||||
a := w.asked[0]
|
||||
rec := w.records["shell."+a.ID]
|
||||
rec.State = "expired"
|
||||
w.records["shell."+a.ID] = rec
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "ended expired; asked again") || len(w.asked) != 2 {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§14: another asker's warrant runs nothing", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{}`))
|
||||
a := w.decide(gate.OptionApprove, w.now)
|
||||
rec := w.records["shell."+a.ID]
|
||||
rec.Warrant.Asker = "mail"
|
||||
w.records["shell."+a.ID] = rec
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || w.runs != 0 || !strings.Contains(err.Error(), "its warrant does not hold") {
|
||||
t.Fatalf("%v, runs %d", err, w.runs)
|
||||
}
|
||||
}},
|
||||
{"§17: the router's refusal is said with its words", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{}`))
|
||||
a := w.asked[0]
|
||||
rec := w.records["shell."+a.ID]
|
||||
rec.State, rec.Answer = "refused", "the asker has too many open asks"
|
||||
w.records["shell."+a.ID] = rec
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "the asker has too many open asks") {
|
||||
t.Fatalf("%v", err)
|
||||
}
|
||||
}},
|
||||
{"§20: a runner's own tool marked is refused unasked, never run", func(t *testing.T, w *world) {
|
||||
c := call("wipe", `{}`)
|
||||
c.Own = true
|
||||
if _, err := w.do(c); err == nil || w.runs != 0 || len(w.asked) != 0 {
|
||||
t.Fatalf("%v, runs %d, asked %d", err, w.runs, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§13: with no binding, an unmarked call runs and a marked one is refused", func(t *testing.T, w *world) {
|
||||
w.deps.State = nil
|
||||
w.rebuild()
|
||||
if got, err := w.do(call("tidy", `{}`)); err != nil || got != "done" {
|
||||
t.Fatalf("unmarked: %v %v", got, err)
|
||||
}
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "binds no state approvals") || w.runs != 1 {
|
||||
t.Fatalf("marked: %v, runs %d", err, w.runs)
|
||||
}
|
||||
}},
|
||||
{"§13: the fixture state refuses a key the bus refuses, so every key the gate writes here is one the bus holds (A-209)", func(t *testing.T, w *world) {
|
||||
if _, err := w.state.Put("n1.act.sha256:00", "x"); err == nil {
|
||||
t.Fatal("the fixture state held a key with a colon, which the bus refuses")
|
||||
}
|
||||
if _, err := w.state.Create("n1..ask", "x"); err == nil {
|
||||
t.Fatal("the fixture state held a key with an empty name")
|
||||
}
|
||||
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
|
||||
if len(w.asked) != 1 || len(w.state.Vals) == 0 {
|
||||
t.Fatalf("the gate kept nothing through the fixture state: asked %d, keys %d", len(w.asked), len(w.state.Vals))
|
||||
}
|
||||
}},
|
||||
{"§13: an ask over 256 KiB is refused naming the cause", func(t *testing.T, w *world) {
|
||||
c := call("wipe", `{}`)
|
||||
c.Description = strings.Repeat("w", 300*1024)
|
||||
if _, err := w.do(c); err == nil || !strings.Contains(err.Error(), "more than the 262144 a value of the state approvals holds") || len(w.asked) != 0 {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§3, §2: a seat verb its seat marks is asked for its holder", func(t *testing.T, w *world) {
|
||||
c := gate.Call{Address: "node-login-shell.execute", Module: "shell", Seat: "node-login-shell", Name: "execute", Fingerprint: "src2:1", Args: json.RawMessage(`{"command":"ls"}`)}
|
||||
if _, err := w.do(c); err == nil || !strings.HasPrefix(err.Error(), "Not run") || len(w.asked) != 1 || w.asked[0].Headline != "Run node-login-shell.execute on n1?" {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§16: the run is recorded once; unanswered within five minutes it is a finding, cleared when warranted answers", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{}`))
|
||||
a := w.decide(gate.OptionApprove, w.now)
|
||||
stop, _ := w.g.WatchBounds()
|
||||
defer stop()
|
||||
w.warranted = errors.New("no responders")
|
||||
if _, err := w.do(call("wipe", `{}`)); err != nil || w.runs != 1 || len(w.recorded) != 0 {
|
||||
t.Fatalf("%v, runs %d, recorded %v", err, w.runs, w.recorded)
|
||||
}
|
||||
if w.bounds.keys["n1.unanswered."+a.ID] != gate.UnansweredAfter || w.bounds.keys["n1.unrecorded."+a.ID] != gate.KeptFor-gate.SaidBefore {
|
||||
t.Fatalf("bounds %v", w.bounds.keys)
|
||||
}
|
||||
w.bounds.expire("n1.unanswered." + a.ID)
|
||||
if w.findings.raised[keys.EscapeName("n1."+a.ID)] != gate.KindCallUnanswered {
|
||||
t.Fatalf("findings %v", w.findings.raised)
|
||||
}
|
||||
w.warranted = nil
|
||||
// At a start each acted, unrecorded ask is pending in the runner's findings-stated until warranted is made again
|
||||
// (design 47 G §384).
|
||||
pendingKey := views.FindingKey("machine", keys.EscapeName("n1."+a.ID), gate.KindCallUnanswered)
|
||||
warranted := w.deps.Warranted
|
||||
var pendingAtCall bool
|
||||
w.deps.Warranted = func(asker, ask string, open func() (bool, error)) error {
|
||||
_, pendingAtCall = w.findings.pends[pendingKey]
|
||||
return warranted(asker, ask, open)
|
||||
}
|
||||
w.rebuild() // a start
|
||||
w.g.Recover() // a start, or the hand-act seat's holder heard again
|
||||
w.g.Recover()
|
||||
if !pendingAtCall || len(w.findings.pended) != 1 || len(w.findings.pends) != 0 || w.findings.pended[0] != pendingKey {
|
||||
t.Fatalf("pending at the call %v, pended %v, pending after %v", pendingAtCall, w.findings.pended, w.findings.pends)
|
||||
}
|
||||
if len(w.recorded) != 1 || w.recorded[0] != "shell."+a.ID || len(w.findings.raised) != 0 || len(w.bounds.keys) != 0 {
|
||||
t.Fatalf("recorded %v, findings %v, bounds %v", w.recorded, w.findings.raised, w.bounds.keys)
|
||||
}
|
||||
var k operatorchannelclient.Kept
|
||||
_ = json.Unmarshal(w.state.Vals["n1.ask."+a.ID], &k)
|
||||
if !k.Recorded || !k.Acted {
|
||||
t.Fatalf("kept %+v", k)
|
||||
}
|
||||
}},
|
||||
{"§19: with no holder of operator-channel in the view, a marked call is refused asking nothing", func(t *testing.T, w *world) {
|
||||
w.holders["operator-channel"] = []byte(`{"seat":"operator-channel","scope":"mesh","holders":[]}`)
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "no module holds operator-channel") || len(w.asked) != 0 {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
w.holdersErr = errors.New("the first view has not been answered yet")
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || len(w.asked) != 0 {
|
||||
t.Fatalf("an unread view: %v, asked %d", err, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§19: a publish that fails and a record that cannot be read refuse, running nothing", func(t *testing.T, w *world) {
|
||||
pub := w.publish
|
||||
w.publish = func(string, string, any, string) error { return errors.New("the bus refused it") }
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "the ask could not be raised") {
|
||||
t.Fatalf("%v", err)
|
||||
}
|
||||
w.publish = pub
|
||||
w.state.Vals = map[string]json.RawMessage{}
|
||||
_, _ = w.do(call("wipe", `{}`))
|
||||
w.record = func(string, string) (*stdio.StateEntry, error) { return nil, errors.New("timeout") }
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "the router's record of its ask") || w.runs != 0 {
|
||||
t.Fatalf("%v, runs %d", err, w.runs)
|
||||
}
|
||||
}},
|
||||
{"§5, §19: a served entry that cannot be read refuses every marked call on that runtime", func(t *testing.T, w *world) {
|
||||
c := call("wipe", `{}`)
|
||||
c.Fingerprint = "src2:unknown"
|
||||
if _, err := w.do(c); err == nil || !strings.Contains(err.Error(), "cannot be read") || len(w.asked) != 0 {
|
||||
t.Fatalf("%v, asked %d", err, len(w.asked))
|
||||
}
|
||||
}},
|
||||
{"§19: the marks unread, a call waits within 5 seconds and is refused naming mesh_first_views", func(t *testing.T, w *world) {
|
||||
w.cat.ready = errors.New("the catalog half has no first view of main yet")
|
||||
waited := time.Duration(0)
|
||||
w.deps.Wait = func(d time.Duration) error { waited = d; return w.cat.ready }
|
||||
w.rebuild()
|
||||
if _, err := w.do(call("tidy", `{}`)); err == nil || !strings.Contains(err.Error(), "mesh_first_views") || waited != gate.RecordWithin || w.runs != 0 {
|
||||
t.Fatalf("%v, waited %s", err, waited)
|
||||
}
|
||||
w.deps.Wait = func(time.Duration) error { w.cat.ready = nil; return nil }
|
||||
w.rebuild()
|
||||
if _, err := w.do(call("tidy", `{}`)); err != nil || w.runs != 1 {
|
||||
t.Fatalf("read within the wait: %v, runs %d", err, w.runs)
|
||||
}
|
||||
}},
|
||||
{"§19: the state failing refuses, raises approvals-unreadable after five minutes, and clears when read again", func(t *testing.T, w *world) {
|
||||
stop, _ := w.g.WatchBounds()
|
||||
defer stop()
|
||||
w.state.Fail = errors.New("the bus is unreachable")
|
||||
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "cannot be read") {
|
||||
t.Fatalf("%v", err)
|
||||
}
|
||||
if w.bounds.keys["n1.unreadable.approvals"] != gate.UnreadableAfter || len(w.findings.raised) != 0 {
|
||||
t.Fatalf("bounds %v, findings %v", w.bounds.keys, w.findings.raised)
|
||||
}
|
||||
w.bounds.expire("n1.unreadable.approvals")
|
||||
if w.findings.raised[keys.EscapeName("n1.approvals")] != gate.KindApprovalsUnreadable {
|
||||
t.Fatalf("findings %v", w.findings.raised)
|
||||
}
|
||||
w.state.Fail = nil
|
||||
w.g.StateAgain() // a membership change heard, no further call
|
||||
if len(w.findings.raised) != 0 || len(w.bounds.keys) != 0 {
|
||||
t.Fatalf("findings %v, bounds %v", w.findings.raised, w.bounds.keys)
|
||||
}
|
||||
}},
|
||||
{"§18: the asks of the last day, one read whole, the marks applied with their sources, a record again and a cancel", func(t *testing.T, w *world) {
|
||||
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
|
||||
list, err := w.g.List()
|
||||
if err != nil || len(list) != 1 || list[0].State != "asked" || list[0].Address != "shell.wipe" || list[0].Fingerprint == "" {
|
||||
t.Fatalf("%+v %v", list, err)
|
||||
}
|
||||
whole, err := w.g.Read(list[0].Ask)
|
||||
if err != nil || string(whole.Arguments) != `{"dir":"scratch"}` || !strings.HasPrefix(whole.Digest, "sha256:") {
|
||||
t.Fatalf("%+v %v", whole, err)
|
||||
}
|
||||
applied := w.g.Applied()
|
||||
if len(applied) != 1 || applied[0].Tools["wipe"] != "entry shell src2:1" || applied[0].Verbs["node-login-shell.execute"] != "seat node-login-shell" || applied[0].Revision != "7" {
|
||||
t.Fatalf("applied %+v", applied)
|
||||
}
|
||||
if ok, src := w.g.Describe("shell", "", "wipe"); !ok || src != "entry shell src2:1" {
|
||||
t.Errorf("describe %v %q", ok, src)
|
||||
}
|
||||
if ok, src := w.g.Describe("", "node-login-shell", "execute"); !ok || src != "seat node-login-shell" {
|
||||
t.Errorf("describe a seat verb %v %q", ok, src)
|
||||
}
|
||||
if err := w.g.Again(list[0].Ask); err == nil {
|
||||
t.Error("an ask not run was asked its hand-act again")
|
||||
}
|
||||
cancelled := ""
|
||||
w.publish = func(module, subject string, body any, id string) error { cancelled = subject + " " + id; return nil }
|
||||
if err := w.g.Cancel(list[0].Ask); err != nil || cancelled != "mesh.seat.operator-channel.accept.cancel.shell cancel."+list[0].Ask {
|
||||
t.Fatalf("cancel %v %q", err, cancelled)
|
||||
}
|
||||
}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) { tc.run(t, newWorld()) })
|
||||
}
|
||||
// Design 46 7-approval-mark §5, §3, §4 and design 34 every-event §33: the marks a module's calls are gated by, derived from
|
||||
// fixture catalog halves and served entries.
|
||||
t.Run("TheMarks", func(t *testing.T) {
|
||||
login := map[string]seatdef.Definition{"node-login-shell": {Seat: "node-login-shell",
|
||||
Verbs: []seatdef.Verb{{Name: "execute", NeedsApproval: true}, {Name: "sessions"}}}}
|
||||
v1 := entry("src2:1", []string{"node-login-shell.execute", "node-login-shell.sessions"},
|
||||
map[string]any{"needs-approval": []any{"wipe"}})
|
||||
v2 := entry("src2:2", []string{"node-login-shell.execute", "node-login-shell.sessions"},
|
||||
map[string]any{"needs-approval": []any{"erase", "node-login-shell.sessions"}})
|
||||
mail := catalogseat.CatalogEntry{Module: "mail", Fp: "src2:m", Manifest: map[string]any{
|
||||
"needs-approval": []any{"mail.received"}, "emits": []any{"mail.received"},
|
||||
"streams": []any{map[string]any{"name": "MAIL", "events": []any{"mail.**"}, "grant": map[string]any{"config": "readers", "events": "mail.{key}.**"}}}}}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
cat *fakeCatalog
|
||||
module string
|
||||
fp string
|
||||
tools []string
|
||||
verbs map[string]string
|
||||
subjects []string
|
||||
unread bool
|
||||
}{
|
||||
{name: "a seat's mark stands for every holder, and the holder's own adds up", cat: &fakeCatalog{revision: "7",
|
||||
byFp: map[string]catalogseat.CatalogEntry{"src2:1": v1}, newest: map[string]catalogseat.CatalogEntry{"shell": v1}, defs: login},
|
||||
module: "shell", fp: "src2:1", tools: []string{"wipe"}, verbs: map[string]string{"node-login-shell.execute": "seat node-login-shell"}},
|
||||
{name: "a holder's needs-approval cannot unmark its seat's verb, and marks another", cat: &fakeCatalog{revision: "8",
|
||||
byFp: map[string]catalogseat.CatalogEntry{"src2:2": v2}, newest: map[string]catalogseat.CatalogEntry{"shell": v2}, defs: login},
|
||||
module: "shell", fp: "src2:2", tools: []string{"erase"}, verbs: map[string]string{"node-login-shell.execute": "seat node-login-shell",
|
||||
"node-login-shell.sessions": "entry shell src2:2"}},
|
||||
{name: "across a version change the union stands: a dropped mark is asked while the old bundle runs", cat: &fakeCatalog{revision: "9",
|
||||
byFp: map[string]catalogseat.CatalogEntry{"src2:1": v1, "src2:2": v2}, newest: map[string]catalogseat.CatalogEntry{"shell": v2}, defs: login},
|
||||
module: "shell", fp: "src2:1", tools: []string{"erase", "wipe"}, verbs: map[string]string{"node-login-shell.execute": "seat node-login-shell",
|
||||
"node-login-shell.sessions": "entry shell src2:2"}},
|
||||
{name: "a served entry that cannot be read leaves the newest marks, every marked call refused", cat: &fakeCatalog{revision: "9",
|
||||
newest: map[string]catalogseat.CatalogEntry{"shell": v2}, defs: login},
|
||||
module: "shell", fp: "src2:1", tools: []string{"erase"}, verbs: map[string]string{"node-login-shell.execute": "seat node-login-shell",
|
||||
"node-login-shell.sessions": "entry shell src2:2"}, unread: true},
|
||||
{name: "a bundle with no fingerprint has no served entry", cat: &fakeCatalog{revision: "9", defs: login},
|
||||
module: "shell", unread: true},
|
||||
{name: "a marked event and a stream with a grant give their subjects and each .>", cat: &fakeCatalog{revision: "3",
|
||||
byFp: map[string]catalogseat.CatalogEntry{"src2:m": mail}, newest: map[string]catalogseat.CatalogEntry{"mail": mail}},
|
||||
module: "mail", fp: "src2:m", subjects: []string{"mesh.mod.mail.event.mail.received", "mesh.mod.mail.event.mail.received.>",
|
||||
"mesh.stream.mail.event.mail.>", "mesh.stream.mail.event.mail.received", "mesh.stream.mail.event.mail.received.>"}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
m := gate.Derive(tc.cat, tc.module, tc.fp)
|
||||
if got := keysOf(m.Tools); !reflect.DeepEqual(got, tc.tools) {
|
||||
t.Errorf("tools %v, want %v", got, tc.tools)
|
||||
}
|
||||
if tc.verbs == nil {
|
||||
tc.verbs = map[string]string{}
|
||||
}
|
||||
if !reflect.DeepEqual(m.Verbs, tc.verbs) {
|
||||
t.Errorf("verbs %v, want %v", m.Verbs, tc.verbs)
|
||||
}
|
||||
if got := keysOf(m.Subjects); !reflect.DeepEqual(got, tc.subjects) {
|
||||
t.Errorf("subjects %v, want %v", got, tc.subjects)
|
||||
}
|
||||
if (m.Unread != "") != tc.unread || m.Revision != tc.cat.revision {
|
||||
t.Errorf("unread %q, revision %q", m.Unread, m.Revision)
|
||||
}
|
||||
})
|
||||
}
|
||||
// §4: a bundle whose marks name a tool it does not register is not loaded, its finding mark-unserved raised; a bundle that
|
||||
// serves it clears it, and a catalog without the mark too.
|
||||
t.Run("mark-unserved raised and cleared", func(t *testing.T) {
|
||||
cat := &fakeCatalog{revision: "7", byFp: map[string]catalogseat.CatalogEntry{"src2:1": v1}, newest: map[string]catalogseat.CatalogEntry{"shell": v1}, defs: login}
|
||||
w := &findings{}
|
||||
g := gate.New(gate.Deps{Node: "n1", Runner: "node-tools", Catalog: cat, Findings: Findings(w, "n1", "node-tools", nil)})
|
||||
if got := g.Unserved("shell", "src2:1", []string{"other"}); !reflect.DeepEqual(got, []string{"wipe"}) || w.raised[keys.EscapeName("n1.shell")] != gate.KindMarkUnserved {
|
||||
t.Fatalf("unserved %v, findings %v", got, w.raised)
|
||||
}
|
||||
if got := g.Unserved("shell", "src2:1", []string{"wipe"}); got != nil || w.raised[keys.EscapeName("n1.shell")] != "" {
|
||||
t.Fatalf("a bundle serving the mark: unserved %v, findings %v", got, w.raised)
|
||||
}
|
||||
_ = g.Unserved("shell", "src2:1", nil)
|
||||
cat.byFp["src2:1"], cat.newest["shell"] = entry("src2:1", nil, nil), entry("src2:1", nil, nil)
|
||||
if got := g.Unserved("shell", "src2:1", nil); got != nil || w.raised[keys.EscapeName("n1.shell")] != "" {
|
||||
t.Fatalf("a catalog without the mark: unserved %v, findings %v", got, w.raised)
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
// Package approval is the tool runner's side of mesh-sdk's approval gate (go/approval; design 47 G lead answers A-224,
|
||||
// A-228): it translates what the gate reads beyond itself from the generated contracts, the gate's findings into the
|
||||
// contract's findings about the runner on its machine, the catalog's entries into the gate's, and the assigner's view of
|
||||
// the seats' holders into its holder function. The gate itself is mesh-sdk's; none is written here.
|
||||
package approval
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
assignerseat "git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
|
||||
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
|
||||
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
gate "git.novox.be/novox/mesh-sdk/go/approval"
|
||||
"git.novox.be/novox/mesh-sdk/go/views"
|
||||
)
|
||||
|
||||
// HeldWriter is the runner's held finding writer (views.HeldWriter over the contract's Finding).
|
||||
type HeldWriter interface {
|
||||
Write(ctx context.Context, finding condseat.Finding) error
|
||||
Clear(ctx context.Context, scope, id, kind, why string) error
|
||||
}
|
||||
|
||||
// scopeMachine is the scope of the gate's findings on the tool runner: its machine (lead answer A-154).
|
||||
const scopeMachine = "machine"
|
||||
|
||||
// Findings is the gate's finding writer over the runner's held writer: each finding about the runner's module on node, at
|
||||
// scope machine, and its pending named in the writer's `findings-stated` where the writer keeps one (design 47 G §384).
|
||||
func Findings(w HeldWriter, node, runner string, now func() time.Time) gate.FindingWriter {
|
||||
if w == nil {
|
||||
return nil
|
||||
}
|
||||
if now == nil {
|
||||
now = time.Now
|
||||
}
|
||||
return findingWriter{w: w, node: node, runner: runner, now: now}
|
||||
}
|
||||
|
||||
type findingWriter struct {
|
||||
w HeldWriter
|
||||
node, runner string
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
func (f findingWriter) Raise(ctx context.Context, kind, id, summary, headline string) error {
|
||||
return f.w.Write(ctx, condseat.Finding{Kind: kind, Scope: scopeMachine, ID: id, Severity: condseat.FindingSeverityWarning,
|
||||
Source: "approval", Observer: f.runner, Subject: condseat.Subject{Scope: "module", ID: f.runner, Machine: f.node},
|
||||
Summary: summary, Headline: headline, At: f.now().UTC()})
|
||||
}
|
||||
|
||||
func (f findingWriter) Clear(ctx context.Context, kind, id, why string) error {
|
||||
return f.w.Clear(ctx, scopeMachine, id, kind, why)
|
||||
}
|
||||
|
||||
// pender is the held writer's pending (views.HeldWriter's Pend and Judged).
|
||||
type pender interface {
|
||||
Pend(views.PendingKey)
|
||||
Judged(key string)
|
||||
}
|
||||
|
||||
func (f findingWriter) Pend(kind, id string, since time.Time, bound time.Duration) {
|
||||
if p, ok := f.w.(pender); ok {
|
||||
p.Pend(views.PendingKey{Key: views.FindingKey(scopeMachine, id, kind), Since: since, Bound: bound.String()})
|
||||
}
|
||||
}
|
||||
|
||||
func (f findingWriter) Judged(kind, id string) {
|
||||
if p, ok := f.w.(pender); ok {
|
||||
p.Judged(views.FindingKey(scopeMachine, id, kind))
|
||||
}
|
||||
}
|
||||
|
||||
// Entry is a catalog entry as the gate reads its marks.
|
||||
func Entry(e catalogseat.CatalogEntry) gate.Entry {
|
||||
manifest, _ := json.Marshal(e.Manifest)
|
||||
return gate.Entry{Module: e.Module, Fingerprint: e.Fp, Serves: e.Serves, Manifest: manifest}
|
||||
}
|
||||
|
||||
// HoldersIn is the gate's holder function over the assigner's view of the seats' holders as the runner holds it (§19):
|
||||
// view answers the view's values by key, an error while its first view is not answered; a seat is held when its key names
|
||||
// a holder.
|
||||
func HoldersIn(view func() (map[string][]byte, error)) func(seat string) (bool, error) {
|
||||
return func(seat string) (bool, error) {
|
||||
values, err := view()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
raw, ok := values[assignerseat.SeatHoldersBySeatKey(seat)]
|
||||
if !ok {
|
||||
return false, nil
|
||||
}
|
||||
var h assignerseat.SeatHolders
|
||||
if err := json.Unmarshal(raw, &h); err != nil {
|
||||
return false, fmt.Errorf("the holders of %s cannot be read: %w", seat, err)
|
||||
}
|
||||
return len(h.Holders) > 0, nil
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package approval
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
|
||||
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
|
||||
"git.novox.be/novox/mesh-sdk/go/views"
|
||||
|
||||
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
gate "git.novox.be/novox/mesh-sdk/go/approval"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatdef"
|
||||
)
|
||||
|
||||
// fakeCatalog is a catalog half as a fixture: entries by fingerprint, the newest by module, the seat definitions.
|
||||
type fakeCatalog struct {
|
||||
ready error
|
||||
revision string
|
||||
byFp map[string]catalogseat.CatalogEntry
|
||||
newest map[string]catalogseat.CatalogEntry
|
||||
defs map[string]seatdef.Definition
|
||||
}
|
||||
|
||||
func (c *fakeCatalog) Ready() error { return c.ready }
|
||||
func (c *fakeCatalog) Revision() string { return c.revision }
|
||||
func (c *fakeCatalog) Served(fp string) (gate.Entry, bool, string) {
|
||||
e, ok := c.byFp[fp]
|
||||
return Entry(e), ok, "not-found"
|
||||
}
|
||||
func (c *fakeCatalog) Newest(m string) (gate.Entry, bool) {
|
||||
e, ok := c.newest[m]
|
||||
return Entry(e), ok
|
||||
}
|
||||
func (c *fakeCatalog) Definitions() map[string]seatdef.Definition { return c.defs }
|
||||
func (c *fakeCatalog) Entries() []gate.Entry {
|
||||
var out []gate.Entry
|
||||
for _, e := range c.newest {
|
||||
out = append(out, Entry(e))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func entry(fp string, serves []string, manifest map[string]any) catalogseat.CatalogEntry {
|
||||
return catalogseat.CatalogEntry{Module: "shell", Fp: fp, Serves: serves, Manifest: manifest}
|
||||
}
|
||||
|
||||
// findings is the runner's held writer as a fixture: each finding raised now, by id.
|
||||
type findings struct {
|
||||
raised map[string]string
|
||||
// pends are the keys pending in the held writer's statement; pended every key ever pended.
|
||||
pends map[string]views.PendingKey
|
||||
pended []string
|
||||
}
|
||||
|
||||
func (f *findings) Pend(p views.PendingKey) {
|
||||
if f.pends == nil {
|
||||
f.pends = map[string]views.PendingKey{}
|
||||
}
|
||||
f.pends[p.Key] = p
|
||||
f.pended = append(f.pended, p.Key)
|
||||
}
|
||||
func (f *findings) Judged(key string) { delete(f.pends, key) }
|
||||
|
||||
func (f *findings) Write(_ context.Context, x condseat.Finding) error {
|
||||
if f.raised == nil {
|
||||
f.raised = map[string]string{}
|
||||
}
|
||||
f.raised[x.ID] = x.Kind
|
||||
return nil
|
||||
}
|
||||
func (f *findings) Clear(_ context.Context, _, id, _, _ string) error {
|
||||
delete(f.raised, id)
|
||||
return nil
|
||||
}
|
||||
|
||||
func keysOf(set map[string]string) []string {
|
||||
var out []string
|
||||
for k := range set {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,434 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// A module's bus state in the new form (novox/hq ADR 0595, design 25 § 12.5; issue 1966 T1): an object store, a stream,
|
||||
// a work queue, a ledger, each asked for in `requires` by a local name and bound in the module's membership by the bus
|
||||
// names the bus module created. **A bundle reaches only its own module's bindings**: one account per machine carries
|
||||
// every module on it, so what a module may reach is decided here, from its membership, and refused with the reason
|
||||
// before anything is sent, as its state is (state.go).
|
||||
|
||||
// Binding is the module's binding of a local name, of the storage type storage, or the reason there is none.
|
||||
func (c *Conn) Binding(module, local, storage string) (membership.BusBinding, error) {
|
||||
m := c.Membership(module)
|
||||
if m == nil {
|
||||
return membership.BusBinding{}, fmt.Errorf("%s has no membership issued on %s yet, so none of its bus state is "+
|
||||
"reachable until the mesh issues one (novox/hq ADR 0595)", module, c.node)
|
||||
}
|
||||
var names []string
|
||||
for _, b := range m.Bindings {
|
||||
if b.Local != local {
|
||||
names = append(names, b.Local+" ("+b.Storage+")")
|
||||
continue
|
||||
}
|
||||
if b.Storage != storage {
|
||||
return membership.BusBinding{}, fmt.Errorf("%s's %q is bound with the storage type %s, not %s (novox/hq ADR 0595)",
|
||||
module, local, b.Storage, storage)
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
sort.Strings(names)
|
||||
bound := "none"
|
||||
if len(names) > 0 {
|
||||
bound = strings.Join(names, ", ")
|
||||
}
|
||||
return membership.BusBinding{}, fmt.Errorf("%s was bound no %s called %q: a module reaches only the bus state its "+
|
||||
"manifest asks for in `requires`, by its local name, once the mesh created it; it was bound: %s (novox/hq ADR 0595)",
|
||||
module, storage, local, bound)
|
||||
}
|
||||
|
||||
// Bindings is every binding a module's membership carries now.
|
||||
func (c *Conn) Bindings(module string) []membership.BusBinding {
|
||||
if m := c.Membership(module); m != nil {
|
||||
return append([]membership.BusBinding(nil), m.Bindings...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// JetStream is a JetStream handle on this connection: its one accessor, for a part that hands it to a finding writer (design 47 G
|
||||
// A-16, A-27).
|
||||
func (c *Conn) JetStream() (jetstream.JetStream, error) { return jetstream.New(c.nc) }
|
||||
|
||||
func (c *Conn) objectStore(ctx context.Context, module, local string) (jetstream.ObjectStore, membership.BusBinding, error) {
|
||||
b, err := c.Binding(module, local, catalogue.StorageObjectStore)
|
||||
if err != nil {
|
||||
return nil, b, err
|
||||
}
|
||||
js, err := c.JetStream()
|
||||
if err != nil {
|
||||
return nil, b, err
|
||||
}
|
||||
obs, err := js.ObjectStore(ctx, b.ObjectStore)
|
||||
if errors.Is(err, jetstream.ErrBucketNotFound) || errors.Is(err, jetstream.ErrStreamNotFound) {
|
||||
return nil, b, fmt.Errorf("%s's objects %q are bound and their store %s is not on the bus", module, local, b.ObjectStore)
|
||||
}
|
||||
return obs, b, err
|
||||
}
|
||||
|
||||
func objectInfoOf(i *jetstream.ObjectInfo) buscalls.ObjectInfo {
|
||||
return buscalls.ObjectInfo{Name: i.Name, Size: i.Size, Chunks: i.Chunks, Digest: i.Digest, Description: i.Description,
|
||||
Modified: i.ModTime, Deleted: i.Deleted}
|
||||
}
|
||||
|
||||
// ObjectsPut stores one object whole under its name in the module's object store.
|
||||
func (c *Conn) ObjectsPut(module string, r buscalls.ObjectsPutRequest) (buscalls.ObjectInfo, error) {
|
||||
if r.Name == "" {
|
||||
return buscalls.ObjectInfo{}, errors.New("an object is put under a name")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
info, err := obs.PutBytes(ctx, r.Name, r.Data)
|
||||
if err != nil {
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
if r.Description != "" {
|
||||
if err := obs.UpdateMeta(ctx, r.Name, jetstream.ObjectMeta{Name: r.Name, Description: r.Description}); err != nil {
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
if info, err = obs.GetInfo(ctx, r.Name); err != nil {
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
}
|
||||
return objectInfoOf(info), nil
|
||||
}
|
||||
|
||||
// ObjectsGet is one object of the module's object store, its info and its bytes; nil when there is none.
|
||||
func (c *Conn) ObjectsGet(module string, r buscalls.ObjectsNameRequest) (*buscalls.ObjectData, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
res, err := obs.Get(ctx, r.Name)
|
||||
if errors.Is(err, jetstream.ErrObjectNotFound) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer res.Close()
|
||||
info, err := res.Info()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
data, err := obs.GetBytes(ctx, r.Name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &buscalls.ObjectData{Info: objectInfoOf(info), Data: data}, nil
|
||||
}
|
||||
|
||||
// ObjectsDelete deletes one object of the module's object store; one that is not there is not an error.
|
||||
func (c *Conn) ObjectsDelete(module string, r buscalls.ObjectsNameRequest) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := obs.Delete(ctx, r.Name); err != nil && !errors.Is(err, jetstream.ErrObjectNotFound) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ObjectsList is every object the module's object store holds.
|
||||
func (c *Conn) ObjectsList(module string, r buscalls.ObjectsRequest) (buscalls.ObjectsListed, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return buscalls.ObjectsListed{}, err
|
||||
}
|
||||
out := buscalls.ObjectsListed{Objects: []buscalls.ObjectInfo{}}
|
||||
infos, err := obs.List(ctx)
|
||||
if errors.Is(err, jetstream.ErrNoObjectsFound) {
|
||||
return out, nil
|
||||
}
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for _, i := range infos {
|
||||
out.Objects = append(out.Objects, objectInfoOf(i))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ObjectsInfo is how full the module's object store is: its objects, its bytes and its cap.
|
||||
func (c *Conn) ObjectsInfo(module string, r buscalls.ObjectsRequest) (buscalls.Fill, error) {
|
||||
listed, err := c.ObjectsList(module, r)
|
||||
if err != nil {
|
||||
return buscalls.Fill{}, err
|
||||
}
|
||||
b, _ := c.Binding(module, r.Objects, catalogue.StorageObjectStore)
|
||||
fill, err := c.streamFill("OBJ_" + b.ObjectStore)
|
||||
if err != nil {
|
||||
return buscalls.Fill{}, err
|
||||
}
|
||||
fill.Count = uint64(len(listed.Objects))
|
||||
return fill.Fill, nil
|
||||
}
|
||||
|
||||
// ObjectsWatch hands deliver every object of the module's object store as it is, then each stored or deleted.
|
||||
func (c *Conn) ObjectsWatch(module string, r buscalls.ObjectsWatchRequest, deliver func(buscalls.ObjectsChange) error) (func(), error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
watchCtx, stop := context.WithCancel(context.Background())
|
||||
w, err := obs.Watch(watchCtx, jetstream.IncludeHistory())
|
||||
if err != nil {
|
||||
stop()
|
||||
return nil, err
|
||||
}
|
||||
current := true
|
||||
for current {
|
||||
select {
|
||||
case i, ok := <-w.Updates():
|
||||
if !ok {
|
||||
stop()
|
||||
return nil, errors.New("the object store's watch ended before its current objects were handed over")
|
||||
}
|
||||
if i == nil {
|
||||
current = false
|
||||
continue
|
||||
}
|
||||
if err := deliver(buscalls.ObjectsChange{Watch: r.Watch, Object: objectInfoOf(i), Current: true}); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take an object of %s: %v", module, r.Objects, err)
|
||||
}
|
||||
case <-time.After(StateTimeout):
|
||||
_ = w.Stop()
|
||||
stop()
|
||||
return nil, fmt.Errorf("the object store %s did not hand its current objects in %s", r.Objects, StateTimeout)
|
||||
}
|
||||
}
|
||||
go func() {
|
||||
for i := range w.Updates() {
|
||||
if i == nil {
|
||||
continue
|
||||
}
|
||||
if err := deliver(buscalls.ObjectsChange{Watch: r.Watch, Object: objectInfoOf(i)}); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take an object of %s: %v", module, r.Objects, err)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return func() { _ = w.Stop(); stop() }, nil
|
||||
}
|
||||
|
||||
// streamFill is a stream's fill and its first and last sequence, by its name on the bus.
|
||||
func (c *Conn) streamFill(name string) (buscalls.StreamFilled, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := c.JetStream()
|
||||
if err != nil {
|
||||
return buscalls.StreamFilled{}, err
|
||||
}
|
||||
s, err := js.Stream(ctx, name)
|
||||
if errors.Is(err, jetstream.ErrStreamNotFound) {
|
||||
return buscalls.StreamFilled{}, fmt.Errorf("the stream %s is bound and not on the bus", name)
|
||||
}
|
||||
if err != nil {
|
||||
return buscalls.StreamFilled{}, err
|
||||
}
|
||||
info, err := s.Info(ctx)
|
||||
if err != nil {
|
||||
return buscalls.StreamFilled{}, err
|
||||
}
|
||||
max := uint64(0)
|
||||
if info.Config.MaxBytes > 0 {
|
||||
max = uint64(info.Config.MaxBytes)
|
||||
}
|
||||
return buscalls.StreamFilled{Fill: buscalls.Fill{Count: info.State.Msgs, Bytes: info.State.Bytes, MaxBytes: max},
|
||||
First: info.State.FirstSeq, Last: info.State.LastSeq}, nil
|
||||
}
|
||||
|
||||
// streamBinding is the module's binding of a stream by its local name, a work queue among them.
|
||||
func (c *Conn) streamBinding(module, local string) (membership.BusBinding, error) {
|
||||
return c.Binding(module, local, catalogue.StorageStream)
|
||||
}
|
||||
|
||||
// StreamFill is how full the module's stream is.
|
||||
func (c *Conn) StreamFill(module string, r buscalls.StreamRequest) (buscalls.StreamFilled, error) {
|
||||
b, err := c.streamBinding(module, r.Stream)
|
||||
if err != nil {
|
||||
return buscalls.StreamFilled{}, err
|
||||
}
|
||||
return c.streamFill(b.Stream)
|
||||
}
|
||||
|
||||
// StreamPublish says one event on the module's own bound stream, by an acknowledged publish on the binding's subject plus
|
||||
// "." plus the event, the id as the message id (design 42 zfs-owns-zfs, A-42). The module's own binding alone: another
|
||||
// module's, and a binding that is not a stream, are refused by Binding.
|
||||
func (c *Conn) StreamPublish(module string, r buscalls.StreamPublishRequest) (buscalls.StreamPublished, error) {
|
||||
b, err := c.streamBinding(module, r.Stream)
|
||||
if err != nil {
|
||||
return buscalls.StreamPublished{}, err
|
||||
}
|
||||
if b.Subject == "" {
|
||||
return buscalls.StreamPublished{}, fmt.Errorf("%s's stream %q has no subject to say an event on", module, r.Stream)
|
||||
}
|
||||
if r.Event == "" || strings.ContainsAny(r.Event, " *>\t\r\n") || strings.HasPrefix(r.Event, ".") || strings.HasSuffix(r.Event, ".") {
|
||||
return buscalls.StreamPublished{}, fmt.Errorf("the event %q is not a name to say on %s's stream %q", r.Event, module, r.Stream)
|
||||
}
|
||||
body := []byte(r.Body)
|
||||
if len(body) == 0 {
|
||||
body = []byte("null")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := c.JetStream()
|
||||
if err != nil {
|
||||
return buscalls.StreamPublished{}, err
|
||||
}
|
||||
var opts []jetstream.PublishOpt
|
||||
if r.ID != "" {
|
||||
opts = append(opts, jetstream.WithMsgID(r.ID))
|
||||
}
|
||||
ack, err := js.Publish(ctx, b.Subject+"."+r.Event, body, opts...)
|
||||
if err != nil {
|
||||
return buscalls.StreamPublished{}, fmt.Errorf("the event %s was not acknowledged on %s's stream %q: %w", r.Event, module, r.Stream, err)
|
||||
}
|
||||
return buscalls.StreamPublished{Sequence: ack.Sequence}, nil
|
||||
}
|
||||
|
||||
// StreamReadMost is the most messages one read answers.
|
||||
const StreamReadMost = 1000
|
||||
|
||||
// StreamRead is the module's stream's messages from a sequence on, Subjects narrowing them, at most Limit of them.
|
||||
func (c *Conn) StreamRead(module string, r buscalls.StreamReadRequest) (buscalls.StreamPage, error) {
|
||||
b, err := c.streamBinding(module, r.Stream)
|
||||
if err != nil {
|
||||
return buscalls.StreamPage{}, err
|
||||
}
|
||||
return c.readStream(b.Stream, r.Subjects, r.From, r.Limit)
|
||||
}
|
||||
|
||||
// readStream reads a stream by its bus name from a sequence on, the messages on the subjects filter names (every
|
||||
// subject when empty), at most limit of them.
|
||||
func (c *Conn) readStream(name, filter string, from uint64, limit int) (buscalls.StreamPage, error) {
|
||||
if limit <= 0 || limit > StreamReadMost {
|
||||
limit = StreamReadMost
|
||||
}
|
||||
if filter == "" {
|
||||
filter = ">"
|
||||
}
|
||||
if from == 0 {
|
||||
from = 1
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := c.JetStream()
|
||||
if err != nil {
|
||||
return buscalls.StreamPage{}, err
|
||||
}
|
||||
s, err := js.Stream(ctx, name)
|
||||
if errors.Is(err, jetstream.ErrStreamNotFound) {
|
||||
return buscalls.StreamPage{}, fmt.Errorf("the stream %s is bound and not on the bus", name)
|
||||
}
|
||||
if err != nil {
|
||||
return buscalls.StreamPage{}, err
|
||||
}
|
||||
info, err := s.Info(ctx)
|
||||
if err != nil {
|
||||
return buscalls.StreamPage{}, err
|
||||
}
|
||||
page := buscalls.StreamPage{Messages: []buscalls.StreamMessage{}, Last: info.State.LastSeq}
|
||||
for seq := from; seq <= info.State.LastSeq && len(page.Messages) < limit; {
|
||||
msg, err := s.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(filter))
|
||||
if errors.Is(err, jetstream.ErrMsgNotFound) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return page, err
|
||||
}
|
||||
body := json.RawMessage(msg.Data)
|
||||
if !json.Valid(msg.Data) {
|
||||
body, _ = json.Marshal(string(msg.Data))
|
||||
}
|
||||
page.Messages = append(page.Messages, buscalls.StreamMessage{Sequence: msg.Sequence, Subject: msg.Subject,
|
||||
Body: body, At: msg.Time})
|
||||
seq = msg.Sequence + 1
|
||||
if len(page.Messages) == limit && seq <= info.State.LastSeq {
|
||||
page.Next = seq
|
||||
}
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
|
||||
// WorkQueueSize is how many messages wait in the module's work queue, against its cap.
|
||||
func (c *Conn) WorkQueueSize(module string, r buscalls.WorkQueueRequest) (buscalls.WorkQueueSized, error) {
|
||||
b, err := c.streamBinding(module, r.Queue)
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueSized{}, err
|
||||
}
|
||||
fill, err := c.streamFill(b.Stream)
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueSized{}, err
|
||||
}
|
||||
return buscalls.WorkQueueSized{Fill: fill.Fill}, nil
|
||||
}
|
||||
|
||||
// WorkQueueDead is the module's work queue's dead letters: in a work queue a message is gone once settled, so one still
|
||||
// kept below a consumer's acknowledgement floor was delivered as often as the consumer allows and never settled.
|
||||
func (c *Conn) WorkQueueDead(module string, r buscalls.WorkQueueDeadRequest) (buscalls.WorkQueueDeadLetters, error) {
|
||||
b, err := c.streamBinding(module, r.Queue)
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueDeadLetters{}, err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := c.JetStream()
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueDeadLetters{}, err
|
||||
}
|
||||
s, err := js.Stream(ctx, b.Stream)
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueDeadLetters{}, fmt.Errorf("the work queue %s: %w", b.Stream, err)
|
||||
}
|
||||
out := buscalls.WorkQueueDeadLetters{Dead: []buscalls.DeadLetter{}}
|
||||
consumers := s.ListConsumers(ctx)
|
||||
for ci := range consumers.Info() {
|
||||
floor := ci.AckFloor.Stream
|
||||
if ci.Config.MaxDeliver <= 0 || floor == 0 {
|
||||
continue
|
||||
}
|
||||
page, err := c.readStream(b.Stream, ci.Config.FilterSubject, 1, r.Limit)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for _, m := range page.Messages {
|
||||
if m.Sequence > floor {
|
||||
break
|
||||
}
|
||||
out.Dead = append(out.Dead, buscalls.DeadLetter{Sequence: m.Sequence, Subject: m.Subject,
|
||||
Deliveries: ci.Config.MaxDeliver, Body: m.Body, At: m.At})
|
||||
}
|
||||
}
|
||||
if err := consumers.Err(); err != nil {
|
||||
return out, err
|
||||
}
|
||||
if r.Limit > 0 && len(out.Dead) > r.Limit {
|
||||
out.Dead = out.Dead[:r.Limit]
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
package bus_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// 1966/AC12: given a bundle whose membership binds objects `files`, when it puts and gets `a`, then it reads back what it
|
||||
// put; a local name it was not bound is refused with the reason, and so is another module's binding and a binding of
|
||||
// another storage type. A stream and a work queue bound alike answer their read-back, fill and size.
|
||||
func TestABundleReachesItsOwnBoundObjectsAndStreamsOnly(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
nc, err := nats.Connect(mt.URL(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(nc.Close)
|
||||
js, _ := jetstream.New(nc)
|
||||
ctx := context.Background()
|
||||
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: "m_files"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "m_events", Subjects: []string{"mesh.state.m.events.>"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, s := range []string{"mesh.state.m.events.one", "mesh.state.m.events.two"} {
|
||||
if _, err := js.Publish(ctx, s, []byte(`{"n":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
m := mt.MembershipOf("m", "anchor", false, nil)
|
||||
m.Bindings = []membership.BusBinding{
|
||||
{Local: "files", Storage: catalogue.StorageObjectStore, Template: catalogue.TemplateObjects, ObjectStore: "m_files"},
|
||||
{Local: "events", Storage: catalogue.StorageStream, Template: catalogue.TemplateEvents, Stream: "m_events",
|
||||
Subject: "mesh.state.m.events"},
|
||||
}
|
||||
other := mt.MembershipOf("other", "anchor", false, nil)
|
||||
mesh.Issue(t, m)
|
||||
mesh.Issue(t, other)
|
||||
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn.Logf = func(string, ...any) {}
|
||||
t.Cleanup(conn.Close)
|
||||
conn.Follow("m")
|
||||
conn.Follow("other")
|
||||
mt.Until(t, func() error {
|
||||
if conn.Membership("m") == nil || conn.Membership("other") == nil {
|
||||
return errors.New("not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
if _, err := conn.ObjectsPut("m", buscalls.ObjectsPutRequest{Objects: "files", Name: "a", Data: []byte("what was put")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := conn.ObjectsGet("m", buscalls.ObjectsNameRequest{Objects: "files", Name: "a"})
|
||||
if err != nil || got == nil || string(got.Data) != "what was put" || got.Info.Name != "a" {
|
||||
t.Fatalf("read back %+v, %v", got, err)
|
||||
}
|
||||
// Design 47 G §518: a value over one chunk is streamed into the store and read back out of it, its description kept.
|
||||
big := strings.Repeat("0123456789", 120<<10)
|
||||
info, err := conn.ObjectsPutStream(ctx, "m", buscalls.ObjectsNameRequest{Objects: "files", Name: "big"}, "streamed", strings.NewReader(big))
|
||||
if err != nil || info.Size != uint64(len(big)) || info.Description != "streamed" {
|
||||
t.Fatalf("streamed put %+v, %v", info, err)
|
||||
}
|
||||
opened, rd, found, err := conn.ObjectsOpen(ctx, "m", buscalls.ObjectsNameRequest{Objects: "files", Name: "big"})
|
||||
if err != nil || !found || opened.Size != uint64(len(big)) {
|
||||
t.Fatalf("opened %+v %v %v", opened, found, err)
|
||||
}
|
||||
if back, _ := io.ReadAll(rd); string(back) != big {
|
||||
t.Fatalf("read back %d bytes, want %d", len(back), len(big))
|
||||
}
|
||||
_ = rd.Close()
|
||||
if err := conn.ObjectsDelete("m", buscalls.ObjectsNameRequest{Objects: "files", Name: "big"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if listed, err := conn.ObjectsList("m", buscalls.ObjectsRequest{Objects: "files"}); err != nil || len(listed.Objects) != 1 {
|
||||
t.Fatalf("listed %+v, %v", listed, err)
|
||||
}
|
||||
page, err := conn.StreamRead("m", buscalls.StreamReadRequest{Stream: "events", Limit: 1})
|
||||
if err != nil || len(page.Messages) != 1 || page.Next != 2 || page.Last != 2 {
|
||||
t.Fatalf("read %+v, %v", page, err)
|
||||
}
|
||||
if fill, err := conn.StreamFill("m", buscalls.StreamRequest{Stream: "events"}); err != nil || fill.Count != 2 {
|
||||
t.Fatalf("fill %+v, %v", fill, err)
|
||||
}
|
||||
|
||||
// A-42: an event said on the module's own stream lands on the binding's subject plus the event, acknowledged, and a
|
||||
// second say with the same id is the same message.
|
||||
said, err := conn.StreamPublish("m", buscalls.StreamPublishRequest{Stream: "events", Event: "scrub.requested",
|
||||
Body: json.RawMessage(`{"pool":"tank"}`), ID: "id-1"})
|
||||
if err != nil || said.Sequence != 3 {
|
||||
t.Fatalf("said %+v, %v", said, err)
|
||||
}
|
||||
if again, err := conn.StreamPublish("m", buscalls.StreamPublishRequest{Stream: "events", Event: "scrub.requested",
|
||||
Body: json.RawMessage(`{"pool":"tank"}`), ID: "id-1"}); err != nil || again.Sequence != 3 {
|
||||
t.Fatalf("said again %+v, %v", again, err)
|
||||
}
|
||||
if page, err := conn.StreamRead("m", buscalls.StreamReadRequest{Stream: "events", From: 3,
|
||||
Subjects: "mesh.state.m.events.scrub.requested"}); err != nil || len(page.Messages) != 1 ||
|
||||
string(page.Messages[0].Body) != `{"pool":"tank"}` {
|
||||
t.Fatalf("read back %+v, %v", page, err)
|
||||
}
|
||||
|
||||
for _, c := range []struct {
|
||||
name, module, refusal string
|
||||
call func(module string) error
|
||||
}{
|
||||
{"a local name not bound", "m", `bound no object-store called "photos"`, func(module string) error {
|
||||
_, err := conn.ObjectsGet(module, buscalls.ObjectsNameRequest{Objects: "photos", Name: "a"})
|
||||
return err
|
||||
}},
|
||||
{"another module's binding", "other", `other was bound no object-store called "files"`, func(module string) error {
|
||||
_, err := conn.ObjectsGet(module, buscalls.ObjectsNameRequest{Objects: "files", Name: "a"})
|
||||
return err
|
||||
}},
|
||||
{"a binding of another storage type", "m", `"events" is bound with the storage type stream, not object-store`, func(module string) error {
|
||||
_, err := conn.ObjectsList(module, buscalls.ObjectsRequest{Objects: "events"})
|
||||
return err
|
||||
}},
|
||||
{"say on another module's stream", "other", `other was bound no stream called "events"`, func(module string) error {
|
||||
_, err := conn.StreamPublish(module, buscalls.StreamPublishRequest{Stream: "events", Event: "x"})
|
||||
return err
|
||||
}},
|
||||
{"say on a binding that is not a stream", "m", `"files" is bound with the storage type object-store, not stream`, func(module string) error {
|
||||
_, err := conn.StreamPublish(module, buscalls.StreamPublishRequest{Stream: "files", Event: "x"})
|
||||
return err
|
||||
}},
|
||||
{"a work queue not bound", "m", `bound no stream called "jobs"`, func(module string) error {
|
||||
_, err := conn.WorkQueueSize(module, buscalls.WorkQueueRequest{Queue: "jobs"})
|
||||
return err
|
||||
}},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
err := c.call(c.module)
|
||||
if err == nil || !strings.Contains(err.Error(), c.refusal) {
|
||||
t.Fatalf("refused with %v, want the reason %q", err, c.refusal)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+320
-107
@@ -10,6 +10,7 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
@@ -17,10 +18,12 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"log"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
@@ -29,13 +32,9 @@ import (
|
||||
"github.com/novox/mesh-tools/node-tools/internal/wire"
|
||||
)
|
||||
|
||||
// RequestTimeout is how long a call waits for its answer — what modules already expect. The
|
||||
// controller answers its verbs well inside it (its AnswerWithin, ten seconds), with a call id while a
|
||||
// verb still runs, so a caller is never left guessing past this (novox/hq issue 265).
|
||||
// RequestTimeout is how long a call waits for its answer — what modules already expect.
|
||||
const RequestTimeout = 30 * time.Second
|
||||
|
||||
const assignmentsStream = "ASSIGNMENTS"
|
||||
|
||||
// Credential is a broker credential as the mesh delivers it (novox/hq ADR 0120).
|
||||
type Credential struct {
|
||||
URL string `json:"url"`
|
||||
@@ -76,9 +75,9 @@ type Membership struct {
|
||||
// Hears are the module streams the module hears, each through a durable consumer on that stream the
|
||||
// runtime binds beside the one on EVENTS (ADR 0363 §8).
|
||||
Hears []HeardIssued `json:"hears,omitempty"`
|
||||
// Waits are the event families a session may wait on through the mesh MCP server (novox/hq ADR 0367 §2):
|
||||
// the tool runner's alone, composed from its manifest with the grant that lets it subscribe them.
|
||||
Waits []WaitIssued `json:"waits,omitempty"`
|
||||
// Bindings are the module's bus state in the new form, by local name (novox/hq ADR 0595 §1, mesh-sdk
|
||||
// membership.BusBinding): its object stores, streams, work queues and ledgers, its own alone (bindings.go).
|
||||
Bindings []membership.BusBinding `json:"bindings,omitempty"`
|
||||
}
|
||||
|
||||
// Served is an address a tool is answered on; `{tool}` stands for the tool's name.
|
||||
@@ -193,6 +192,8 @@ type Conn struct {
|
||||
calls callers
|
||||
// waits are the waits in flight by subject, each told when the bus refuses its subscription (wait.go).
|
||||
waits map[string]map[chan string]bool
|
||||
// inboxSaid is when a refused answer publish was last said, by subject (answerRefused).
|
||||
inboxSaid map[string]time.Time
|
||||
}
|
||||
|
||||
// answered is one subject the runtime answers on: what to subscribe again with, and how often it was
|
||||
@@ -230,6 +231,9 @@ func (c *Conn) refused(err error) {
|
||||
if c.publishRefused(err) {
|
||||
return
|
||||
}
|
||||
if c.answerRefused(err) {
|
||||
return
|
||||
}
|
||||
if subject := subjectRefused(err); subject != "" && c.waitRefused(subject) {
|
||||
return
|
||||
}
|
||||
@@ -286,9 +290,20 @@ func (c *Conn) refused(err error) {
|
||||
})
|
||||
}
|
||||
|
||||
// Connect dials the bus as the credential's module. A module's subjects come from its credential,
|
||||
// never from its calls (ADR 0074).
|
||||
// Connect dials the bus as the credential's module and follows its membership. A module's subjects come from its
|
||||
// credential, never from its calls (ADR 0074).
|
||||
func Connect(cred Credential) (*Conn, error) {
|
||||
c, err := Dial(cred)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.Follow(cred.Module)
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// Dial dials the bus as the credential's module and follows nothing: a connection that only publishes for its module,
|
||||
// as a module's ledger connection does (novox/hq ADR 0597 §9).
|
||||
func Dial(cred Credential) (*Conn, error) {
|
||||
if cred.Module == "" {
|
||||
return nil, errors.New("a broker credential with no module: the runtime derives its subjects " +
|
||||
"from the account the mesh issued, and cannot guess which module it is")
|
||||
@@ -327,7 +342,6 @@ func Connect(cred Credential) (*Conn, error) {
|
||||
}
|
||||
c = &Conn{nc: nc, js: js, self: cred.Module, node: cred.Node, cred: cred,
|
||||
issued: map[string]*Membership{}, Logf: log.Printf, answering: map[string]*answered{}}
|
||||
c.Follow(cred.Module)
|
||||
return c, nil
|
||||
}
|
||||
|
||||
@@ -340,13 +354,10 @@ func (c *Conn) Node() string { return c.node }
|
||||
// Credential is what this connection was opened with.
|
||||
func (c *Conn) Credential() Credential { return c.cred }
|
||||
|
||||
// MembershipSubject is the one address a runtime derives for an assignment (ADR 0160).
|
||||
func MembershipSubject(node, module string) string {
|
||||
return "mesh.assignment." + node + "." + module
|
||||
}
|
||||
|
||||
// Follow reads a module's membership on this machine once and follows it live, so its tools are
|
||||
// served where the mesh issued them (ADR 0175).
|
||||
// served where the mesh issued them (ADR 0175). It is read from the grants part's state memberships, its key
|
||||
// `<node>.<module>` (novox/hq ADR 0571 clause 7 and 10, mesh-sdk membership.GrantsBucket), and nowhere else:
|
||||
// the old assignments stream is not read and its subject is not heard.
|
||||
func (c *Conn) Follow(module string) {
|
||||
c.mu.Lock()
|
||||
if c.node == "" {
|
||||
@@ -359,44 +370,150 @@ func (c *Conn) Follow(module string) {
|
||||
}
|
||||
c.issued[module] = nil
|
||||
c.mu.Unlock()
|
||||
subject := MembershipSubject(c.node, module)
|
||||
// The subject-addressed direct get: the one address the mesh grants this account on the
|
||||
// stream's API.
|
||||
if got, err := c.nc.Request("$JS.API.DIRECT.GET."+assignmentsStream+"."+subject, nil, 5*time.Second); err == nil {
|
||||
if got.Header.Get("Status") == "" && len(got.Data) > 0 {
|
||||
var m Membership
|
||||
if json.Unmarshal(got.Data, &m) == nil {
|
||||
c.mu.Lock()
|
||||
c.issued[module] = &m
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
}
|
||||
c.followGrants(module)
|
||||
if c.Membership(module) == nil {
|
||||
c.Logf("[mesh-tools] no membership issued for %s on %s yet; serving the derived shape until one arrives", module, c.node)
|
||||
}
|
||||
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
|
||||
}
|
||||
|
||||
// GrantsPatience bounds the first read of a membership from the grants part's state.
|
||||
var GrantsPatience = 2 * time.Second
|
||||
|
||||
// GrantsRetry is the first wait before the memberships state is read again after it could not be; it doubles up to
|
||||
// GrantsRetryMost.
|
||||
var (
|
||||
GrantsRetry = time.Second
|
||||
GrantsRetryMost = 30 * time.Second
|
||||
)
|
||||
|
||||
// issue applies a membership that arrived for module, live, and re-serves on it.
|
||||
func (c *Conn) issue(module string, m Membership) {
|
||||
c.mu.Lock()
|
||||
c.issued[module] = &m
|
||||
handlers := make([]func(Membership), 0, len(c.onNew))
|
||||
for _, h := range c.onNew {
|
||||
handlers = append(handlers, *h)
|
||||
}
|
||||
c.mu.Unlock()
|
||||
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
|
||||
for _, h := range handlers {
|
||||
h(m)
|
||||
}
|
||||
_ = c.nc.Flush()
|
||||
}
|
||||
|
||||
// followGrants reads module's key of the grants part's state memberships and watches it: the value there now is applied
|
||||
// before Follow returns, each change after it live. A state that is not there yet, or not readable (a bus answering 503
|
||||
// right after its boot, a timeout), is said once and read again, with a growing wait, until it can be or the
|
||||
// connection closes: Follow runs once per module, so nothing else would ever read it again.
|
||||
func (c *Conn) followGrants(module string) {
|
||||
if c.watchGrants(module, true) {
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
wait := GrantsRetry
|
||||
for !c.nc.IsClosed() {
|
||||
time.Sleep(wait)
|
||||
if c.nc.IsClosed() {
|
||||
return
|
||||
}
|
||||
if c.watchGrants(module, false) {
|
||||
c.Logf("[mesh-tools] the grants part's memberships are read again; %s on %s follows its key", module, c.node)
|
||||
return
|
||||
}
|
||||
if wait *= 2; wait > GrantsRetryMost {
|
||||
wait = GrantsRetryMost
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// watchGrants is one attempt of followGrants: true once the key is watched. A failure is said when say is true.
|
||||
func (c *Conn) watchGrants(module string, say bool) bool {
|
||||
logf := func(format string, a ...any) {
|
||||
if say {
|
||||
c.Logf(format, a...)
|
||||
}
|
||||
}
|
||||
key := membership.GrantsKey(c.node, module)
|
||||
// First the key's direct get, which the account is granted alone: a bus that refuses it, or a state not there yet,
|
||||
// answers at once, where binding the bucket would wait out its bound on a refusal.
|
||||
bucket := membership.GrantsBucket
|
||||
probe, err := c.nc.Request("$JS.API.DIRECT.GET.KV_"+bucket+".$KV."+bucket+"."+key, nil, GrantsPatience)
|
||||
if err != nil || probe.Header.Get("Status") == "503" {
|
||||
why := "it did not answer"
|
||||
if err != nil {
|
||||
why = err.Error()
|
||||
}
|
||||
logf("[mesh-tools] the grants part's memberships (%s) cannot be read (%s); %s has no membership "+
|
||||
"until it can be", bucket, why, module)
|
||||
return false
|
||||
}
|
||||
js, err := jetstream.New(c.nc)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := js.KeyValue(ctx, bucket)
|
||||
if err != nil {
|
||||
logf("[mesh-tools] the grants part's memberships (%s) cannot be read (%v); %s has no membership "+
|
||||
"until it can be", membership.GrantsBucket, err, module)
|
||||
return false
|
||||
}
|
||||
w, err := kv.Watch(context.Background(), key)
|
||||
if err != nil {
|
||||
logf("[mesh-tools] %s's key %s of the grants part's memberships cannot be watched (%v); %s has no membership "+
|
||||
"until it can be", module, key, err, module)
|
||||
return false
|
||||
}
|
||||
apply := func(e jetstream.KeyValueEntry) {
|
||||
if e.Operation() != jetstream.KeyValuePut {
|
||||
return // a membership withdrawn is the stream's, or the module's unassignment's, to say
|
||||
}
|
||||
// Read through mesh-sdk's one codec of this state, the grants part's writer's (issue 774): it takes the bare
|
||||
// membership and a wrapper the part wrote before of one published or unchanged, and refuses a held-back one's
|
||||
// wrapper or a value naming no node or module, which then leaves the module with no membership.
|
||||
// A refused value arriving after a good one keeps the last good membership: nothing here clears it.
|
||||
var m Membership
|
||||
if err := json.Unmarshal(msg.Data, &m); err != nil {
|
||||
c.Logf("[mesh-tools] a membership arrived that is not one: %v", err)
|
||||
body, err := membership.BareGrantsMembership(e.Value())
|
||||
if err == nil {
|
||||
err = json.Unmarshal(body, &m)
|
||||
}
|
||||
if err != nil {
|
||||
c.Logf("[mesh-tools] the membership at %s is not one: %v", key, err)
|
||||
return
|
||||
}
|
||||
c.mu.Lock()
|
||||
c.issued[module] = &m
|
||||
handlers := make([]func(Membership), 0, len(c.onNew))
|
||||
for _, h := range c.onNew {
|
||||
handlers = append(handlers, *h)
|
||||
}
|
||||
c.mu.Unlock()
|
||||
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
|
||||
for _, h := range handlers {
|
||||
h(m)
|
||||
}
|
||||
_ = c.nc.Flush()
|
||||
})
|
||||
if err == nil {
|
||||
c.track(sub)
|
||||
c.issue(module, m)
|
||||
}
|
||||
timer := time.NewTimer(StateTimeout)
|
||||
defer timer.Stop()
|
||||
for current := true; current; {
|
||||
select {
|
||||
case e, ok := <-w.Updates():
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
if e == nil {
|
||||
current = false
|
||||
continue
|
||||
}
|
||||
apply(e)
|
||||
case <-timer.C:
|
||||
_ = w.Stop()
|
||||
logf("[mesh-tools] %s's key %s of the grants part's memberships did not answer in %s; %s has no membership "+
|
||||
"until it can be", module, key, StateTimeout, module)
|
||||
return false
|
||||
}
|
||||
}
|
||||
go func() {
|
||||
for e := range w.Updates() {
|
||||
if e != nil {
|
||||
apply(e)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return true
|
||||
}
|
||||
|
||||
// Membership is what the mesh issued a module here, or nil when nothing has been issued.
|
||||
@@ -686,88 +803,58 @@ func (c *Conn) Ask(key string, body any, on string) (Answered, error) {
|
||||
// account publish with its own name alone; asked again on the subject that names no caller only when the
|
||||
// first reached nobody — nothing serves it yet, or the bus does not grant it yet — so no call is done twice.
|
||||
if called := CalledSubject(subject, c.cred.User); called != "" {
|
||||
answered, handingOver, fallBack, err := c.askCalled(called, data)
|
||||
answered, fallBack, err := c.askCalled(called, data)
|
||||
if !fallBack {
|
||||
if !handingOver {
|
||||
return answered, err
|
||||
}
|
||||
time.Sleep(HandoverPause)
|
||||
answered, again, _, err := c.askCalled(called, data)
|
||||
if err != nil && again {
|
||||
return Answered{}, fmt.Errorf("asked twice, %s apart, and both times refused as a handover: %w", HandoverPause, err)
|
||||
}
|
||||
return answered, err
|
||||
}
|
||||
}
|
||||
answered, handingOver, err := c.askOnce(subject, data)
|
||||
if !handingOver {
|
||||
return answered, err
|
||||
}
|
||||
// **Refused because the controller is handing over: asked once more** (novox/hq issue 289). The
|
||||
// controller being replaced did nothing and said so; the one after it answers the same call. Once:
|
||||
// a second refusal is a controller that is not coming back, and the caller is told.
|
||||
time.Sleep(HandoverPause)
|
||||
answered, again, err := c.askOnce(subject, data)
|
||||
if err != nil && again {
|
||||
return Answered{}, fmt.Errorf("asked twice, %s apart, and both times refused as a handover: %w", HandoverPause, err)
|
||||
}
|
||||
return answered, err
|
||||
return c.askOnce(subject, data)
|
||||
}
|
||||
|
||||
// HandoverPause is how long a call refused as a handover waits before it is asked once more: the
|
||||
// supervisor's restart of the controller, which stops the one being replaced and starts the next. A
|
||||
// variable so a test need not wait.
|
||||
var HandoverPause = 3 * time.Second
|
||||
|
||||
// RetryHandingOver is the mark the controller puts on a call it refused because it is being replaced
|
||||
// (mesh-controller internal/link, `"retry": "handing-over"`).
|
||||
const RetryHandingOver = "handing-over"
|
||||
|
||||
// ErrNoResponders is the bus's own refusal: nothing subscribes to the subject asked. It is told apart
|
||||
// from a tool's own error by errors.Is, never by its text (a tool may say "503" itself, issue 1529).
|
||||
var ErrNoResponders = errors.New("503 no responders")
|
||||
|
||||
// askOnce asks once, and says whether the answer was a refusal marked to be asked again.
|
||||
func (c *Conn) askOnce(subject string, data []byte) (Answered, bool, error) {
|
||||
// askOnce asks once.
|
||||
func (c *Conn) askOnce(subject string, data []byte) (Answered, error) {
|
||||
msg, err := c.nc.Request(subject, data, RequestTimeout)
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return Answered{}, false, ErrNoResponders
|
||||
return Answered{}, ErrNoResponders
|
||||
}
|
||||
if errors.Is(err, nats.ErrTimeout) {
|
||||
return Answered{}, false, errors.New("timeout")
|
||||
return Answered{}, errors.New("timeout")
|
||||
}
|
||||
return Answered{}, false, err
|
||||
return Answered{}, err
|
||||
}
|
||||
return c.answerOf(subject, msg.Data)
|
||||
return answerOf(msg.Data)
|
||||
}
|
||||
|
||||
// answerOf reads one answer, asking the rest of a paged one on the subject it was asked on, and says whether it
|
||||
// was a refusal marked to be asked again.
|
||||
func (c *Conn) answerOf(subject string, data []byte) (Answered, bool, error) {
|
||||
// An answer larger than one message comes in parts (pages.go, novox/hq issue 314).
|
||||
body, err := c.whole(subject, data)
|
||||
if err != nil {
|
||||
return Answered{}, false, err
|
||||
}
|
||||
// answerOf reads one answer.
|
||||
func answerOf(data []byte) (Answered, error) {
|
||||
var r struct {
|
||||
Result json.RawMessage `json:"result"`
|
||||
Error string `json:"error"`
|
||||
Node string `json:"node"`
|
||||
Retry string `json:"retry"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &r); err != nil {
|
||||
return Answered{}, false, err
|
||||
if err := json.Unmarshal(data, &r); err != nil {
|
||||
return Answered{}, err
|
||||
}
|
||||
if r.Error != "" {
|
||||
return Answered{}, r.Retry == RetryHandingOver, errors.New(r.Error)
|
||||
return Answered{}, errors.New(r.Error)
|
||||
}
|
||||
return Answered{Result: r.Result, Node: r.Node}, false, nil
|
||||
return Answered{Result: r.Result, Node: r.Node}, nil
|
||||
}
|
||||
|
||||
// PublishAs emits an event as a module: published into JetStream and awaited, de-duplicated by its
|
||||
// own id (ADR 0042). The body is the payload; the metadata rides as headers.
|
||||
func (c *Conn) PublishAs(module string, env Envelope) error {
|
||||
_, err := c.PublishAsAcked(module, env)
|
||||
return err
|
||||
}
|
||||
|
||||
// PublishAsAcked is PublishAs that also answers the sequence the bus acknowledged the event at.
|
||||
func (c *Conn) PublishAsAcked(module string, env Envelope) (uint64, error) {
|
||||
// On the module's own stream when its membership names one carrying the event (ADR 0363 §2), on EVENTS
|
||||
// otherwise — and only there, as before, for a membership that names no stream.
|
||||
msg := nats.NewMsg(eventSubject(c.Membership(module), module, env.Key))
|
||||
@@ -789,8 +876,11 @@ func (c *Conn) PublishAs(module string, env Envelope) error {
|
||||
if id := env.Headers["x-event-id"]; id != "" {
|
||||
opts = append(opts, nats.MsgId(id))
|
||||
}
|
||||
_, err := c.js.PublishMsg(msg, opts...)
|
||||
return err
|
||||
ack, err := c.js.PublishMsg(msg, opts...)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return ack.Sequence, nil
|
||||
}
|
||||
|
||||
// Say publishes one message on core NATS, kept by nobody: a heartbeat, whose loss is the next one.
|
||||
@@ -966,8 +1056,8 @@ func (c *Conn) AskAs(module, key string, body any) (Answered, error) {
|
||||
}
|
||||
|
||||
// EventsStream is where every module's events land, and ConsumerOf the durable consumer the
|
||||
// controller makes for a module on a machine: the same names the module's own runtime bound
|
||||
// (node-tools/src/broker-nats.ts), so moving the module into the node's runtime neither loses an
|
||||
// mesh makes for a module on a machine: the same names the module's own runtime bound
|
||||
// (the retired TypeScript runtime's broker-nats.ts), so moving the module into the node's runtime neither loses an
|
||||
// event nor sees one twice.
|
||||
const EventsStream = "EVENTS"
|
||||
|
||||
@@ -981,7 +1071,7 @@ func ConsumerOf(node, module string) string {
|
||||
|
||||
// NakDelay is how long an event a handler failed waits before it is offered again: a transient cause
|
||||
// gets another attempt, a permanent one exhausts the consumer's max-deliver rather than spinning. The
|
||||
// consumer then gives the event up, and the controller keeps it in DEAD_LETTERS until a person delivers
|
||||
// consumer then gives the event up, and the mesh keeps it in DEAD_LETTERS until a person delivers
|
||||
// it again — under `mesh.again.<consumer>.…`, from which keyOf reads the same key — or drops it
|
||||
// (novox/hq issue 330).
|
||||
var NakDelay = 5 * time.Second
|
||||
@@ -995,24 +1085,112 @@ var EventHeartbeat = 10 * time.Second
|
||||
// ConsumeAs reads a module's durable consumer on EVENTS, and each one on a module's stream its
|
||||
// membership names (novox/hq ADR 0363 §8), and hands each event to deliver one at a time (novox/hq
|
||||
// ADR 0198): acknowledged when deliver returns nil, negatively acknowledged after NakDelay when it
|
||||
// returns an error, terminated when it is not an event at all. The consumers are the controller's to
|
||||
// returns an error, terminated when it is not an event at all. The consumers are the mesh's to
|
||||
// create; this binds to them and never makes one. Runs until stopped.
|
||||
func (c *Conn) ConsumeAs(module string, deliver func(Envelope) error) (func(), error) {
|
||||
return c.ConsumeAsBeating(module, deliver, nil)
|
||||
}
|
||||
|
||||
// IdleHeartbeat is how often the runtime tells a bundle that its module's consumer is bound and the bus answers,
|
||||
// while no event arrives (stdio.OnHeartbeat), so a subscription's silence is told from a quiet source.
|
||||
var IdleHeartbeat = 15 * time.Second
|
||||
|
||||
// Beat is the runtime's word on a module's consumer between events: it is bound and the bus answers (At), or the
|
||||
// bus's heartbeat stopped arriving or the bus ended the loop (Missed, with Why). Its JSON is the stdio Heartbeat.
|
||||
type Beat struct {
|
||||
At time.Time `json:"at"`
|
||||
Missed bool `json:"missed,omitempty"`
|
||||
Why string `json:"why,omitempty"`
|
||||
}
|
||||
|
||||
// ConsumeAsBeating is ConsumeAs that also says, to beat, how the module's consumer on EVENTS stands: a Beat at each
|
||||
// IdleHeartbeat in which no event arrived and the bus answered about the consumer, and one Missed (said once) when
|
||||
// the bus's own heartbeat to the loop stopped arriving, the bus ended the loop or no longer answers. A Beat that is
|
||||
// not Missed follows the first answer after one that was. A nil beat says nothing.
|
||||
func (c *Conn) ConsumeAsBeating(module string, deliver func(Envelope) error, beat func(Beat)) (func(), error) {
|
||||
return c.ConsumeAsBeatingWhen(module, deliver, beat, nil)
|
||||
}
|
||||
|
||||
// tickHook, when set before a consume starts, is called after each idle heartbeat interval was handled (tests only,
|
||||
// through SetTickHook), so a test counts beats instead of sleeping. A consume reads it once, when it starts.
|
||||
var tickHook func()
|
||||
|
||||
// ConsumeAsBeatingWhen is ConsumeAsBeating that asks the bus about the consumer, and says anything, only while wanted
|
||||
// answers true (nil is always): a module whose children asked for no heartbeat costs the bus no consumer-info request.
|
||||
func (c *Conn) ConsumeAsBeatingWhen(module string, deliver func(Envelope) error, beat func(Beat), wanted func() bool) (func(), error) {
|
||||
asked := func() bool { return beat != nil && (wanted == nil || wanted()) }
|
||||
durable := ConsumerOf(c.node, module)
|
||||
js, err := jetstream.New(c.nc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
h := newHearing(c, js, module, deliver)
|
||||
var lastEvent atomic.Int64
|
||||
var missed atomic.Bool
|
||||
say := func(why string) {
|
||||
if asked() && missed.CompareAndSwap(false, true) {
|
||||
beat(Beat{At: time.Now().UTC(), Missed: true, Why: why})
|
||||
}
|
||||
}
|
||||
h := newHearing(c, js, module, func(env Envelope) error {
|
||||
lastEvent.Store(time.Now().UnixNano())
|
||||
missed.Store(false) // an event arrived: the consumer is heard, whatever was said before
|
||||
return deliver(env)
|
||||
})
|
||||
// Bound by name, never created and never matched against a subject: the consumer and its
|
||||
// filters are the controller's (design 29 §3), exactly as the module's own runtime bound it.
|
||||
events, _, doing, err := h.bind(EventsStream, durable, nil)
|
||||
// filters are the mesh's (design 29 §3), exactly as the module's own runtime bound it.
|
||||
events, _, doing, err := h.bind(EventsStream, durable, func(err error) {
|
||||
if errors.Is(err, jetstream.ErrNoHeartbeat) {
|
||||
say("the bus's heartbeat to the consumer " + durable + " stopped arriving")
|
||||
}
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s %s's consumer %s: %w", doing, module, durable, err)
|
||||
}
|
||||
// And every consumer on a module's stream its membership names (novox/hq ADR 0363 §8), followed as the
|
||||
// membership changes. A membership naming none binds nothing more: the module hears EVENTS alone.
|
||||
h.start()
|
||||
if beat != nil {
|
||||
every := IdleHeartbeat
|
||||
hook := tickHook
|
||||
tickOnce := func() {
|
||||
select {
|
||||
case <-events.Closed():
|
||||
say("the bus ended reading the consumer " + durable)
|
||||
return
|
||||
default:
|
||||
}
|
||||
if !asked() {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), every)
|
||||
_, err := js.Consumer(ctx, EventsStream, durable)
|
||||
cancel()
|
||||
if err != nil {
|
||||
say("the bus does not answer about the consumer " + durable + ": " + err.Error())
|
||||
return
|
||||
}
|
||||
if time.Since(time.Unix(0, lastEvent.Load())) < every {
|
||||
return // an event arrived in this interval: it was the word
|
||||
}
|
||||
missed.Store(false)
|
||||
beat(Beat{At: time.Now().UTC()})
|
||||
}
|
||||
go func() {
|
||||
tick := time.NewTicker(every)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-h.done:
|
||||
return
|
||||
case <-tick.C:
|
||||
}
|
||||
tickOnce()
|
||||
if hook != nil {
|
||||
hook()
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
var once sync.Once
|
||||
return func() {
|
||||
once.Do(func() {
|
||||
@@ -1048,3 +1226,38 @@ func keyOf(subject string) string {
|
||||
}
|
||||
return event
|
||||
}
|
||||
|
||||
// answerRefused says a refused publish of an answer (to a caller's inbox: a permission, or the single-use
|
||||
// allowance of a reply expired) once per subject per minute, never silently (issue 1966: a caller waited out its
|
||||
// time for an answer the bus had refused). It reports whether the refusal was one.
|
||||
func (c *Conn) answerRefused(err error) bool {
|
||||
m := refusedPublish.FindStringSubmatch(err.Error())
|
||||
if len(m) < 2 || !strings.HasPrefix(strings.Trim(m[1], `"`), "_INBOX.") {
|
||||
return false
|
||||
}
|
||||
// Keyed without the reply's last token (a new one per request), so a caller's burst is one line.
|
||||
subject := strings.Trim(m[1], `"`)
|
||||
if dot := strings.LastIndex(subject, "."); dot > len("_INBOX.") {
|
||||
subject = subject[:dot]
|
||||
}
|
||||
c.mu.Lock()
|
||||
if c.inboxSaid == nil {
|
||||
c.inboxSaid = map[string]time.Time{}
|
||||
}
|
||||
now := time.Now()
|
||||
for k, at := range c.inboxSaid {
|
||||
if now.Sub(at) >= time.Minute {
|
||||
delete(c.inboxSaid, k)
|
||||
}
|
||||
}
|
||||
_, seen := c.inboxSaid[subject]
|
||||
say := !seen
|
||||
if say {
|
||||
c.inboxSaid[subject] = now
|
||||
}
|
||||
c.mu.Unlock()
|
||||
if say && c.Logf != nil {
|
||||
c.Logf("[mesh-tools] the bus refused this account an answer to %s (%v); the caller gets none", subject, err)
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -18,7 +18,7 @@ import (
|
||||
// mesh.seat.<seat>.call.<verb>.<caller> a mesh seat's verb
|
||||
// mesh.seat.<seat>.call.<verb>.<node>.<caller> a node seat's verb on one machine
|
||||
//
|
||||
// The last token is the bus user that published the call, and the controller grants each user the publish of
|
||||
// The last token is the bus user that published the call, and the mesh grants each user the publish of
|
||||
// these subjects with its own name there and no other — so the caller is a fact the bus server enforces, and
|
||||
// the runtime hands it to the module from the subject a call arrived on, never from the payload.
|
||||
//
|
||||
@@ -32,7 +32,7 @@ import (
|
||||
const CallKind = "call"
|
||||
|
||||
// A bus user's name carries dots (`<node>.<module>`, `person.<name>`); a subject token cannot. In the token
|
||||
// each dot is written `~`, which no part of a bus user's name may hold (the controller refuses any part that
|
||||
// each dot is written `~`, which no part of a bus user's name may hold (the mesh refuses any part that
|
||||
// is not letters, digits, `_` or `-`), so the token is the user's name and nothing else.
|
||||
const userDot = "~"
|
||||
|
||||
@@ -118,7 +118,7 @@ type callers struct {
|
||||
pending map[string]map[uint64]context.CancelCauseFunc
|
||||
}
|
||||
|
||||
// errCallRefused is the bus refusing a caller-named publish: this user is not granted it yet (the controller's
|
||||
// errCallRefused is the bus refusing a caller-named publish: this user is not granted it yet (the mesh's
|
||||
// grant not delivered to the bus), so the call is asked again on the subject that names no caller.
|
||||
var errCallRefused = errors.New("the bus refused the caller-named subject")
|
||||
|
||||
@@ -150,7 +150,7 @@ func (c *Conn) publishRefused(err error) bool {
|
||||
// askCalled asks once on a caller-named subject. fallBack says the call reached nobody — nothing serves that
|
||||
// subject yet, or the bus refused this user its publish — so it may be asked on the subject that names no
|
||||
// caller without being done twice.
|
||||
func (c *Conn) askCalled(subject string, data []byte) (answered Answered, handingOver, fallBack bool, err error) {
|
||||
func (c *Conn) askCalled(subject string, data []byte) (answered Answered, fallBack bool, err error) {
|
||||
ctx, cancel := context.WithCancelCause(context.Background())
|
||||
ctx, stop := context.WithTimeout(ctx, RequestTimeout)
|
||||
defer stop()
|
||||
@@ -180,22 +180,22 @@ func (c *Conn) askCalled(subject string, data []byte) (answered Answered, handin
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(context.Cause(ctx), errCallRefused):
|
||||
return Answered{}, false, true, nil
|
||||
return Answered{}, true, nil
|
||||
case errors.Is(err, nats.ErrNoResponders):
|
||||
c.saidFallBack(subject)
|
||||
return Answered{}, false, true, nil
|
||||
return Answered{}, true, nil
|
||||
case errors.Is(err, context.DeadlineExceeded) || errors.Is(err, nats.ErrTimeout):
|
||||
return Answered{}, false, false, errors.New("timeout")
|
||||
return Answered{}, false, errors.New("timeout")
|
||||
}
|
||||
return Answered{}, false, false, err
|
||||
return Answered{}, false, err
|
||||
}
|
||||
answered, handingOver, err = c.answerOf(subject, msg.Data)
|
||||
return answered, handingOver, false, err
|
||||
answered, err = answerOf(msg.Data)
|
||||
return answered, false, err
|
||||
}
|
||||
|
||||
// saidFallBack says, once per subject, that a caller-named call reached nobody and is asked on the subject that
|
||||
// names no caller: a runtime or holder not yet serving it, or a grant that drifted, is seen in the log rather
|
||||
// than read later as a quiet "unproven" (novox/hq issue 365; expected for the controller's seat until hq issue
|
||||
// than read later as a quiet "unproven" (novox/hq issue 365; expected until hq issue
|
||||
// 464).
|
||||
func (c *Conn) saidFallBack(subject string) {
|
||||
c.calls.mu.Lock()
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// A bus whose users are granted as the controller grants them (mesh-controller internal/broker): each named
|
||||
// A bus whose users are granted as the controller grants them (mesh-module-assigner internal/broker): each named
|
||||
// user publishes the subjects listed for it.
|
||||
func grantedBus(t *testing.T, publish map[string][]string) string {
|
||||
t.Helper()
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// FileField is the member of a chunked verb's answer that carries the file received: its whole bytes, base64 (design 47 G
|
||||
// A-249).
|
||||
const FileField = "file"
|
||||
|
||||
// AskAsChunked is AskAs of a seat's verb whose definition says `chunks` (design 47 G §681, A-237, A-249): it chooses the call
|
||||
// token (the body's `call`), hears the chunk subjects of that call before it asks, receives the chunks the verb's answer
|
||||
// counts with buscalls.SeatChunks, each acknowledged before the next, and answers the verb's own answer with the whole
|
||||
// file added as FileField. The asking bundle receives no chunk itself.
|
||||
func (c *Conn) AskAsChunked(module, key string, body any) (Answered, error) {
|
||||
// The verb may be asked of the machine that ran the ask (`@<node>`, A-254): the chunk subjects name the seat, the verb and
|
||||
// the call alone.
|
||||
name, _, _ := strings.Cut(key, "@")
|
||||
seat, verb, ok := strings.Cut(strings.TrimPrefix(name, "seat:"), ".")
|
||||
if !strings.HasPrefix(key, "seat:") || !ok {
|
||||
return Answered{}, fmt.Errorf("%q is no seat:<seat>.<verb>, which a chunked answer needs", key)
|
||||
}
|
||||
args := map[string]any{}
|
||||
if raw, err := json.Marshal(body); err != nil {
|
||||
return Answered{}, err
|
||||
} else if string(raw) != "null" {
|
||||
if err := json.Unmarshal(raw, &args); err != nil {
|
||||
return Answered{}, fmt.Errorf("the arguments of %s are not an object: %w", key, err)
|
||||
}
|
||||
}
|
||||
var bits [16]byte
|
||||
if _, err := rand.Read(bits[:]); err != nil {
|
||||
return Answered{}, err
|
||||
}
|
||||
send := buscalls.SeatChunks{Seat: seat, Verb: verb, Call: hex.EncodeToString(bits[:])}
|
||||
args["call"] = send.Call
|
||||
chunks := make(chan *nats.Msg, 4)
|
||||
sub, err := c.nc.ChanSubscribe(send.Chunks(), chunks)
|
||||
if err != nil {
|
||||
return Answered{}, fmt.Errorf("the chunks of %s cannot be heard: %w", key, err)
|
||||
}
|
||||
defer func() { _ = sub.Unsubscribe() }()
|
||||
answered, err := c.AskAs(module, key, args)
|
||||
if err != nil {
|
||||
return answered, err
|
||||
}
|
||||
fields := map[string]json.RawMessage{}
|
||||
if err := json.Unmarshal(answered.Result, &fields); err != nil {
|
||||
return answered, fmt.Errorf("%s answered what is no object: %w", key, err)
|
||||
}
|
||||
var count int64
|
||||
if raw, has := fields["chunks"]; has {
|
||||
if err := json.Unmarshal(raw, &count); err != nil {
|
||||
return answered, fmt.Errorf("%s answered chunks that is no count: %w", key, err)
|
||||
}
|
||||
}
|
||||
var file bytes.Buffer
|
||||
if count > 0 {
|
||||
next := func() (string, []byte, error) {
|
||||
select {
|
||||
case m := <-chunks:
|
||||
return m.Subject, m.Data, nil
|
||||
case <-time.After(buscalls.SeatAckWithin):
|
||||
return "", nil, buscalls.ErrSeatChunkMissed
|
||||
}
|
||||
}
|
||||
ack := func(subject string) error {
|
||||
_, err := c.nc.Request(subject, []byte("{}"), buscalls.SeatAckWithin)
|
||||
return err
|
||||
}
|
||||
if _, err := send.Receive(count, next, ack, &file); err != nil {
|
||||
return answered, fmt.Errorf("the chunks of %s: %w", key, err)
|
||||
}
|
||||
}
|
||||
if _, taken := fields[FileField]; taken {
|
||||
return answered, errors.New(key + " answers a member named " + FileField + " itself")
|
||||
}
|
||||
enc, _ := json.Marshal(base64.StdEncoding.EncodeToString(file.Bytes()))
|
||||
fields[FileField] = enc
|
||||
out, err := json.Marshal(fields)
|
||||
if err != nil {
|
||||
return answered, err
|
||||
}
|
||||
answered.Result = out
|
||||
return answered, nil
|
||||
}
|
||||
|
||||
// AnswerChunked is a seat verb's answer as its definition says `chunks` (design 47 G A-257, §681): a bundle answers the whole file
|
||||
// as the member FileField (base64), and the runner that serves the verb sends it in chunks with buscalls.SeatChunks on the call
|
||||
// token the body names, each acknowledged by the caller before the next, and answers the verb's own members with `chunks`, the
|
||||
// count, in place of the file. The bundle sends no chunk itself. An answer with no file is answered as it is. The send runs after
|
||||
// the answer is returned: the caller hears the chunks before it asks (AskAsChunked) and acknowledges each once it has the answer.
|
||||
func (c *Conn) AnswerChunked(seat, verb string, body json.RawMessage, answer any) (any, error) {
|
||||
raw, err := json.Marshal(answer)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
fields := map[string]json.RawMessage{}
|
||||
if err := json.Unmarshal(raw, &fields); err != nil {
|
||||
return answer, nil
|
||||
}
|
||||
enc, has := fields[FileField]
|
||||
if !has {
|
||||
return answer, nil
|
||||
}
|
||||
var text string
|
||||
if err := json.Unmarshal(enc, &text); err != nil {
|
||||
return nil, fmt.Errorf("%s.%s answers a %s that is no string: %w", seat, verb, FileField, err)
|
||||
}
|
||||
file, err := base64.StdEncoding.DecodeString(text)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s.%s answers a %s in no base64: %w", seat, verb, FileField, err)
|
||||
}
|
||||
var asked struct {
|
||||
Call string `json:"call"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &asked); err != nil || asked.Call == "" {
|
||||
return nil, fmt.Errorf("%s.%s is asked in chunks with no call token", seat, verb)
|
||||
}
|
||||
delete(fields, FileField)
|
||||
count := int64((len(file) + buscalls.SeatChunkBytes - 1) / buscalls.SeatChunkBytes)
|
||||
fields["chunks"], _ = json.Marshal(count)
|
||||
send := buscalls.SeatChunks{Seat: seat, Verb: verb, Call: asked.Call}
|
||||
go func() { _ = send.Send(c.nc, file, buscalls.SeatAckWithin, c.Logf) }()
|
||||
return json.RawMessage(mustMarshal(fields)), nil
|
||||
}
|
||||
|
||||
func mustMarshal(v any) []byte {
|
||||
b, _ := json.Marshal(v)
|
||||
return b
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
package bus_test
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// Design 47 G A-249, §681: the runner asks a chunked verb with a call token of its own, receives the chunks the answer counts
|
||||
// (each acknowledged before the next) and answers the verb's own answer with the whole file, so the asker receives none.
|
||||
func TestAChunkedSeatAnswerIsReceivedByTheRunnerAndHandedWhole(t *testing.T) {
|
||||
url := mt.URL(t)
|
||||
holder := dial(t, url, "mesh-delivery-indexer", "anchor", "")
|
||||
asker := dial(t, url, "mesh-build-go", "anchor", "")
|
||||
raw, err := nats.Connect(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(raw.Close)
|
||||
file := make([]byte, buscalls.SeatChunkBytes*2+5)
|
||||
for i := range file {
|
||||
file[i] = byte(i)
|
||||
}
|
||||
// the holder's process: answers the count, then sends each chunk and waits for its acknowledgement
|
||||
stop, err := holder.Handle("seat:mesh-catalog.input", func(body json.RawMessage) (any, error) {
|
||||
var in struct{ Call, Digest string }
|
||||
if err := json.Unmarshal(body, &in); err != nil || in.Call == "" {
|
||||
t.Errorf("the verb was asked with no call token: %s (%v)", body, err)
|
||||
return nil, err
|
||||
}
|
||||
send := buscalls.SeatChunks{Seat: "mesh-catalog", Verb: "input", Call: in.Call}
|
||||
go func() {
|
||||
for n := int64(1); n <= 3; n++ {
|
||||
end := min(int64(len(file)), n*buscalls.SeatChunkBytes)
|
||||
data, _ := json.Marshal(buscalls.SeatChunk{N: n, Of: 3, Data: file[(n-1)*buscalls.SeatChunkBytes : end]})
|
||||
ackSub, _ := raw.SubscribeSync(send.Ack(n))
|
||||
_ = raw.Flush()
|
||||
if err := raw.Publish(send.Chunk(n), data); err != nil {
|
||||
return
|
||||
}
|
||||
m, err := ackSub.NextMsg(buscalls.SeatAckWithin)
|
||||
if err != nil {
|
||||
t.Errorf("chunk %d not acknowledged: %v", n, err)
|
||||
return
|
||||
}
|
||||
_ = m.Respond([]byte("{}"))
|
||||
}
|
||||
}()
|
||||
return map[string]any{"digest": in.Digest, "chunks": 3}, nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer stop()
|
||||
holder.Flush() // the bus holds the verb's subscription before it is asked: the asker is another connection
|
||||
got, err := asker.AskAsChunked("mesh-build-go", "seat:mesh-catalog.input", map[string]any{"digest": "d1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var ans struct {
|
||||
Digest string `json:"digest"`
|
||||
Chunks int `json:"chunks"`
|
||||
File string `json:"file"`
|
||||
}
|
||||
if err := json.Unmarshal(got.Result, &ans); err != nil || ans.Digest != "d1" || ans.Chunks != 3 {
|
||||
t.Fatalf("the answer %s: %v", got.Result, err)
|
||||
}
|
||||
if body, err := base64.StdEncoding.DecodeString(ans.File); err != nil || string(body) != string(file) {
|
||||
t.Errorf("the file handed is %d bytes, want the %d sent: %v", len(body), len(file), err)
|
||||
}
|
||||
// Design 47 G A-257, A-254: a bundle answers a chunked verb with the whole file in the member `file`; the runner that serves the
|
||||
// verb sends the file in chunks (each acknowledged before the next) and answers the count, so the bundle sends none, and the asker
|
||||
// of one node's verb receives it whole.
|
||||
t.Run("ABundlesChunkedAnswerIsSentByTheRunner", func(t *testing.T) {
|
||||
url := mt.URL(t)
|
||||
holder := dial(t, url, "mesh-build-go", "anchor", "")
|
||||
asker := dial(t, url, "mesh-delivery-indexer", "anchor", "")
|
||||
file := make([]byte, buscalls.SeatChunkBytes*2+7)
|
||||
for i := range file {
|
||||
file[i] = byte(i * 7)
|
||||
}
|
||||
stop, err := holder.Handle("seat:node-build-go.output@anchor", func(body json.RawMessage) (any, error) {
|
||||
answer := map[string]any{"ask": "a1", "file": base64.StdEncoding.EncodeToString(file)}
|
||||
return holder.AnswerChunked("node-build-go", "output", body, answer)
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer stop()
|
||||
holder.Flush() // the bus holds the verb's subscription before it is asked: the asker is another connection
|
||||
got, err := asker.AskAsChunked("mesh-delivery-indexer", "seat:node-build-go.output@anchor", map[string]any{"ask": "a1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var ans struct {
|
||||
Ask string `json:"ask"`
|
||||
Chunks int `json:"chunks"`
|
||||
File string `json:"file"`
|
||||
}
|
||||
if err := json.Unmarshal(got.Result, &ans); err != nil || ans.Ask != "a1" || ans.Chunks != 3 {
|
||||
t.Fatalf("the answer %s: %v", got.Result, err)
|
||||
}
|
||||
if body, err := base64.StdEncoding.DecodeString(ans.File); err != nil || string(body) != string(file) {
|
||||
t.Errorf("the file handed is %d bytes, want the %d sent: %v", len(body), len(file), err)
|
||||
}
|
||||
// an answer with no file is the verb's own: no chunks are sent
|
||||
plain, err := holder.AnswerChunked("node-build-go", "output", json.RawMessage(`{"call":"c"}`), map[string]any{"ask": "x", "chunks": 0})
|
||||
if m, _ := plain.(map[string]any); err != nil || m["chunks"] != 0 {
|
||||
t.Fatalf("an answer with no file was %v, %v", plain, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -60,3 +61,96 @@ func TestASlowHandlerAndABurstAreEachHandedOverOnce(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The runtime tells a bundle how its module's consumer stands between events (stdio.OnHeartbeat): a word at each
|
||||
// idle heartbeat while the bus answers, once Missed when the bus ended the loop, and a word again when it is bound again.
|
||||
func TestAConsumerSaysItIsBoundWhileIdleAndOnceWhenTheBusEndedIt(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
restore := bus.IdleHeartbeat
|
||||
bus.IdleHeartbeat = 200 * time.Millisecond
|
||||
t.Cleanup(func() { bus.IdleHeartbeat = restore })
|
||||
ticks := make(chan struct{}, 64)
|
||||
t.Cleanup(bus.SetTickHook(func() {
|
||||
select {
|
||||
case ticks <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}))
|
||||
mesh.Consumer(t, "anchor", "quiet", []string{"mesh.mod.alpha.event.>"}, time.Second)
|
||||
c, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c.Logf = func(string, ...any) {}
|
||||
t.Cleanup(c.Close)
|
||||
|
||||
var mu sync.Mutex
|
||||
var beats []bus.Beat
|
||||
stop, err := c.ConsumeAsBeating("quiet", func(bus.Envelope) error { return nil }, func(b bus.Beat) {
|
||||
mu.Lock()
|
||||
beats = append(beats, b)
|
||||
mu.Unlock()
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(stop)
|
||||
// A consume nobody asked a heartbeat of says nothing and asks the bus nothing, however many intervals pass.
|
||||
var unaskedBeats atomic.Int32
|
||||
unasked, err := c.ConsumeAsBeatingWhen("quiet", func(bus.Envelope) error { return nil },
|
||||
func(bus.Beat) { unaskedBeats.Add(1) }, func() bool { return false })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(unasked)
|
||||
last := func() (bus.Beat, int) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(beats) == 0 {
|
||||
return bus.Beat{}, 0
|
||||
}
|
||||
return beats[len(beats)-1], len(beats)
|
||||
}
|
||||
mt.Until(t, func() error {
|
||||
if b, n := last(); n == 0 || b.Missed || b.At.IsZero() {
|
||||
return fmt.Errorf("no word that the consumer is bound yet: %d beats", n)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
mesh.RemoveConsumer(t, bus.EventsStream, "anchor", "quiet")
|
||||
mt.Until(t, func() error {
|
||||
if b, _ := last(); !b.Missed || b.Why == "" {
|
||||
return fmt.Errorf("the consumer is gone from the bus and no Missed word came: %+v", b)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
mu.Lock()
|
||||
missed := 0
|
||||
for _, b := range beats {
|
||||
if b.Missed {
|
||||
missed++
|
||||
}
|
||||
}
|
||||
mu.Unlock()
|
||||
for n := 0; n < 3; n++ { // three more intervals pass, counted by the loop's own tick
|
||||
select {
|
||||
case <-ticks:
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("the heartbeat loop handled no more intervals")
|
||||
}
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
after := 0
|
||||
for _, b := range beats {
|
||||
if b.Missed {
|
||||
after++
|
||||
}
|
||||
}
|
||||
if unaskedBeats.Load() != 0 {
|
||||
t.Errorf("a consume with no heartbeat asked said %d words", unaskedBeats.Load())
|
||||
}
|
||||
if after != missed || missed != 1 {
|
||||
t.Errorf("Missed is said once, not at every beat: %d then %d", missed, after)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
package bus
|
||||
|
||||
// SetTickHook makes the next consumes call f after each idle heartbeat interval was handled; the restore puts it back.
|
||||
func SetTickHook(f func()) (restore func()) {
|
||||
old := tickHook
|
||||
tickHook = f
|
||||
return func() { tickHook = old }
|
||||
}
|
||||
@@ -1,73 +0,0 @@
|
||||
package bus_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// **A call the controller refused because it is handing over is asked once more** (novox/hq issue 289):
|
||||
// the controller being replaced did nothing and marked its refusal, and the one after it answers. A
|
||||
// refusal not so marked is final, and a second handover refusal is said, not asked a third time.
|
||||
func TestAHandoverRefusalIsAskedOnceMore(t *testing.T) {
|
||||
was := bus.HandoverPause
|
||||
bus.HandoverPause = 10 * time.Millisecond
|
||||
t.Cleanup(func() { bus.HandoverPause = was })
|
||||
|
||||
url := meshtest.URL(t)
|
||||
seat, err := nats.Connect(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer seat.Close()
|
||||
c, err := bus.Connect(bus.Credential{URL: url, Module: "console", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer c.Close()
|
||||
|
||||
serve := func(t *testing.T, verb string, answers ...map[string]any) *atomic.Int32 {
|
||||
t.Helper()
|
||||
var asked atomic.Int32
|
||||
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", verb, "", ""), func(m *nats.Msg) {
|
||||
n := int(asked.Add(1)) - 1
|
||||
if n >= len(answers) {
|
||||
n = len(answers) - 1
|
||||
}
|
||||
body, _ := json.Marshal(answers[n])
|
||||
_ = m.Respond(body)
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
_ = seat.Flush()
|
||||
return &asked
|
||||
}
|
||||
handingOver := map[string]any{"error": "the controller is handing over to the next one; nothing was done, ask again",
|
||||
"retry": bus.RetryHandingOver}
|
||||
|
||||
asked := serve(t, "rotate", handingOver, map[string]any{"result": "rotated"})
|
||||
got, err := c.Ask("seat:mesh-controller.rotate", map[string]any{}, "")
|
||||
if err != nil || string(got.Result) != `"rotated"` || asked.Load() != 2 {
|
||||
t.Fatalf("after a handover refusal: %s %v, asked %d times", got.Result, err, asked.Load())
|
||||
}
|
||||
|
||||
asked = serve(t, "push", map[string]any{"error": "refused for its own reason"})
|
||||
if _, err := c.Ask("seat:mesh-controller.push", map[string]any{}, ""); err == nil || asked.Load() != 1 {
|
||||
t.Fatalf("an unmarked refusal was asked again: %v, asked %d times", err, asked.Load())
|
||||
}
|
||||
|
||||
asked = serve(t, "status", handingOver)
|
||||
_, err = c.Ask("seat:mesh-controller.status", map[string]any{}, "")
|
||||
if err == nil || asked.Load() != 2 || !strings.Contains(err.Error(), "both times refused as a handover") {
|
||||
t.Fatalf("two handover refusals: %v, asked %d times", err, asked.Load())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,427 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/buscompose"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/ledger"
|
||||
)
|
||||
|
||||
// A ledger on the bus (novox/hq ADR 0597, design 25 § 12.4; issue 1966): its history a stream with a subject per
|
||||
// instance under mesh-sdk's buscompose.LedgerHistorySubject, its state a key-value bucket, its snapshots a subject per
|
||||
// instance under buscompose.LedgerSnapshotSubject, the latest only (the stream that holds them keeps one per subject, as the bus module
|
||||
// creates it). **Only the module's ledger connection publishes on its history and its snapshots** (ADR 0597 §9): what
|
||||
// is read, and the state, go on the runner's own.
|
||||
|
||||
// LedgerStores are the stores of a module's ledger binding: reads and the state on c, the history's appends and the
|
||||
// snapshots' writes on writer, the module's ledger connection. The subjects are mesh-sdk's, the ones the grants part
|
||||
// grants the module's ledger account byte for byte; a binding naming others is refused, naming both.
|
||||
func (c *Conn) LedgerStores(module string, b membership.BusBinding, writer *Conn) (ledger.History, ledger.State, ledger.Snapshots, error) {
|
||||
history, snapshot := buscompose.LedgerHistorySubject(module, b.Local), buscompose.LedgerSnapshotSubject(module, b.Local)
|
||||
for _, named := range []struct{ what, bound, want string }{{"history", b.Subject, history}, {"snapshot", b.Snapshot, snapshot}} {
|
||||
if named.bound != "" && prefixOf(named.bound) != named.want {
|
||||
return nil, nil, nil, fmt.Errorf("%s's ledger %q is bound with the %s subject %s, and its ledger account is granted "+
|
||||
"%s alone (mesh-sdk buscompose); the binding is refused", module, b.Local, named.what, named.bound, named.want)
|
||||
}
|
||||
}
|
||||
return &busHistory{read: c, write: writer, stream: b.History, prefix: history},
|
||||
&busState{c: c, bucket: b.State},
|
||||
&busSnapshots{read: c, write: writer, prefix: snapshot}, nil
|
||||
}
|
||||
|
||||
// prefixOf is a binding's subject as the prefix an instance's key is added to: written with or without its `.>`.
|
||||
func prefixOf(subject string) string {
|
||||
return strings.TrimSuffix(strings.TrimSuffix(subject, ">"), ".")
|
||||
}
|
||||
|
||||
// publishExpecting publishes body on subject on the connection, written only when the subject's last message is still
|
||||
// at expect (0: it has none); ledger.ErrRaced when it is not.
|
||||
func publishExpecting(c *Conn, subject string, body []byte, expect uint64) (uint64, error) {
|
||||
msg := nats.NewMsg(subject)
|
||||
msg.Data = body
|
||||
msg.Header.Set(jetstream.ExpectedLastSubjSeqHeader, strconv.FormatUint(expect, 10))
|
||||
reply, err := c.nc.RequestMsg(msg, StateTimeout)
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return 0, fmt.Errorf("no stream on the bus keeps %s", subject)
|
||||
}
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var ack struct {
|
||||
Seq uint64 `json:"seq"`
|
||||
Error *struct {
|
||||
ErrCode uint16 `json:"err_code"`
|
||||
Description string `json:"description"`
|
||||
} `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(reply.Data, &ack); err != nil {
|
||||
return 0, fmt.Errorf("the bus answered the write to %s with what is not an acknowledgement: %q", subject, reply.Data)
|
||||
}
|
||||
if ack.Error != nil {
|
||||
if code := jetstream.ErrorCode(ack.Error.ErrCode); code == jetstream.JSErrCodeStreamWrongLastSequence ||
|
||||
code == jetstream.JSErrCodeStreamWrongLastSequenceConstant {
|
||||
return 0, ledger.ErrRaced
|
||||
}
|
||||
return 0, fmt.Errorf("the bus refused the write to %s: %s (%d)", subject, ack.Error.Description, ack.Error.ErrCode)
|
||||
}
|
||||
return ack.Seq, nil
|
||||
}
|
||||
|
||||
// lastOn is the last message on a subject of the stream that keeps it: its bytes and sequence, nil and 0 for none.
|
||||
func lastOn(c *Conn, stream, subject string) ([]byte, uint64, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := c.JetStream()
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
if stream == "" {
|
||||
if stream, err = js.StreamNameBySubject(ctx, subject); err != nil {
|
||||
return nil, 0, fmt.Errorf("no stream on the bus keeps %s: %w", subject, err)
|
||||
}
|
||||
}
|
||||
s, err := js.Stream(ctx, stream)
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("the stream %s: %w", stream, err)
|
||||
}
|
||||
m, err := s.GetLastMsgForSubject(ctx, subject)
|
||||
if errors.Is(err, jetstream.ErrMsgNotFound) {
|
||||
return nil, 0, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
return m.Data, m.Sequence, nil
|
||||
}
|
||||
|
||||
type busHistory struct {
|
||||
read, write *Conn
|
||||
stream, prefix string
|
||||
}
|
||||
|
||||
func (h *busHistory) Last(key string) (uint64, error) {
|
||||
_, seq, err := lastOn(h.read, h.stream, h.prefix+"."+key)
|
||||
return seq, err
|
||||
}
|
||||
|
||||
func (h *busHistory) Append(key string, op []byte, expect uint64) (uint64, error) {
|
||||
if h.write == nil {
|
||||
return 0, errors.New("the ledger's history is published only on its module's ledger connection, and none is held")
|
||||
}
|
||||
return publishExpecting(h.write, h.prefix+"."+key, op, expect)
|
||||
}
|
||||
|
||||
func (h *busHistory) Read(keys string, from uint64, limit int) ([]ledger.Kept, error) {
|
||||
var out []ledger.Kept
|
||||
for {
|
||||
want := StreamReadMost
|
||||
if limit > 0 && limit-len(out) < want {
|
||||
want = limit - len(out)
|
||||
}
|
||||
page, err := h.read.readStream(h.stream, h.prefix+"."+keys, from, want)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for _, m := range page.Messages {
|
||||
out = append(out, ledger.Kept{Sequence: m.Sequence, Data: m.Body})
|
||||
}
|
||||
if page.Next == 0 || (limit > 0 && len(out) >= limit) {
|
||||
return out, nil
|
||||
}
|
||||
from = page.Next
|
||||
}
|
||||
}
|
||||
|
||||
// Purge removes every operation of one instance from the history stream, on the module's ledger connection, the one
|
||||
// granted the purge of its own ledger's streams.
|
||||
func (h *busHistory) Purge(key string) error {
|
||||
return purgeSubject(h.read, h.write, h.stream, h.prefix+"."+key)
|
||||
}
|
||||
|
||||
// Fill is the history stream's fill against its cap, the cap the manifest gives the ledger.
|
||||
func (h *busHistory) Fill() (buscalls.Fill, error) {
|
||||
name := h.stream
|
||||
if name == "" {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := h.read.JetStream()
|
||||
if err != nil {
|
||||
return buscalls.Fill{}, err
|
||||
}
|
||||
if name, err = js.StreamNameBySubject(ctx, h.prefix+".>"); err != nil {
|
||||
return buscalls.Fill{}, fmt.Errorf("no stream on the bus keeps %s: %w", h.prefix, err)
|
||||
}
|
||||
}
|
||||
f, err := h.read.streamFill(name)
|
||||
return f.Fill, err
|
||||
}
|
||||
|
||||
// purgeSubject purges one subject's messages from the stream that keeps it: the stream named, or found on the runner's
|
||||
// own connection by its subject, purged on the ledger connection.
|
||||
func purgeSubject(read, write *Conn, stream, subject string) error {
|
||||
if write == nil {
|
||||
return errors.New("a ledger's streams are purged only on its module's ledger connection, and none is held")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
if stream == "" {
|
||||
js, err := read.JetStream()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if stream, err = js.StreamNameBySubject(ctx, subject); err != nil {
|
||||
return fmt.Errorf("no stream on the bus keeps %s: %w", subject, err)
|
||||
}
|
||||
}
|
||||
// The purge is asked of the stream by its name, with no read of its info first: the ledger account is granted the
|
||||
// purge of its own ledger's two streams and nothing else (PurgeAPI).
|
||||
body, err := json.Marshal(map[string]string{"filter": subject})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
reply, err := write.nc.RequestWithContext(ctx, PurgeAPI(stream), body)
|
||||
if err != nil {
|
||||
return fmt.Errorf("purging %s from %s: %w", subject, stream, err)
|
||||
}
|
||||
var ack struct {
|
||||
Success bool `json:"success"`
|
||||
Error *struct {
|
||||
Description string `json:"description"`
|
||||
} `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(reply.Data, &ack); err != nil || !ack.Success {
|
||||
why := string(reply.Data)
|
||||
if ack.Error != nil {
|
||||
why = ack.Error.Description
|
||||
}
|
||||
return fmt.Errorf("the bus refused the purge of %s from %s: %s", subject, stream, why)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PurgeAPI is the bus's subject that purges a stream, the one a ledger account is granted for its own ledger's streams.
|
||||
func PurgeAPI(stream string) string { return "$JS.API.STREAM.PURGE." + stream }
|
||||
|
||||
type busState struct {
|
||||
c *Conn
|
||||
bucket string
|
||||
}
|
||||
|
||||
func (s *busState) kv(ctx context.Context) (jetstream.KeyValue, error) {
|
||||
js, err := s.c.JetStream()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
kv, err := js.KeyValue(ctx, s.bucket)
|
||||
if errors.Is(err, jetstream.ErrBucketNotFound) {
|
||||
return nil, fmt.Errorf("the ledger's state %s is not on the bus", s.bucket)
|
||||
}
|
||||
return kv, err
|
||||
}
|
||||
|
||||
func (s *busState) Get(key string) ([]byte, uint64, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
e, err := kv.Get(ctx, key)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return nil, 0, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
return e.Value(), e.Revision(), nil
|
||||
}
|
||||
|
||||
// isFull is a write a bucket or stream refused because it holds as much as it may.
|
||||
func isFull(err error) bool {
|
||||
msg := err.Error()
|
||||
return strings.Contains(msg, "maximum bytes exceeded") || strings.Contains(msg, "maximum messages exceeded") ||
|
||||
strings.Contains(msg, "insufficient resources")
|
||||
}
|
||||
|
||||
func (s *busState) Write(key string, value []byte, revision uint64) (uint64, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var rev uint64
|
||||
if revision == 0 {
|
||||
rev, err = kv.Create(ctx, key, value)
|
||||
} else {
|
||||
rev, err = kv.Update(ctx, key, value, revision)
|
||||
}
|
||||
switch {
|
||||
case err == nil:
|
||||
return rev, nil
|
||||
case errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) || isWrongSequence(err):
|
||||
return 0, ledger.ErrChanged
|
||||
case isFull(err):
|
||||
return 0, fmt.Errorf("%w: %v", ledger.ErrFull, err)
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
|
||||
func (s *busState) Keys() ([]string, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
keys, err := kv.Keys(ctx)
|
||||
if errors.Is(err, jetstream.ErrNoKeysFound) {
|
||||
return nil, nil
|
||||
}
|
||||
return keys, err
|
||||
}
|
||||
|
||||
type busSnapshots struct {
|
||||
read, write *Conn
|
||||
prefix string
|
||||
}
|
||||
|
||||
func (s *busSnapshots) Get(key string) ([]byte, uint64, error) {
|
||||
return lastOn(s.read, "", s.prefix+"."+key)
|
||||
}
|
||||
|
||||
// Purge removes one instance's snapshot, on the module's ledger connection.
|
||||
func (s *busSnapshots) Purge(key string) error {
|
||||
return purgeSubject(s.read, s.write, "", s.prefix+"."+key)
|
||||
}
|
||||
|
||||
func (s *busSnapshots) Put(key string, value []byte, expect uint64) error {
|
||||
if s.write == nil {
|
||||
return errors.New("a ledger's snapshot is published only on its module's ledger connection, and none is held")
|
||||
}
|
||||
_, err := publishExpecting(s.write, s.prefix+"."+key, value, expect)
|
||||
return err
|
||||
}
|
||||
|
||||
// Watch hands deliver every instance's record under a key pattern as it is, then each change; a removed key is handed as
|
||||
// nil.
|
||||
func (s *busState) Watch(pattern string, deliver func(key string, value []byte, current bool) error) (func(), error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
kv, err := s.kv(ctx)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
w, err := kv.Watch(context.Background(), pattern)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
hand := func(e jetstream.KeyValueEntry, current bool) {
|
||||
var value []byte
|
||||
if e.Operation() == jetstream.KeyValuePut {
|
||||
value = e.Value()
|
||||
}
|
||||
if err := deliver(e.Key(), value, current); err != nil {
|
||||
s.c.Logf("[mesh-tools] a ledger watch of %s was not taken: %v", s.bucket, err)
|
||||
}
|
||||
}
|
||||
timer := time.NewTimer(StateTimeout)
|
||||
defer timer.Stop()
|
||||
for current := true; current; {
|
||||
select {
|
||||
case e, ok := <-w.Updates():
|
||||
if !ok {
|
||||
return nil, errors.New("the ledger's state watch ended before its current states were handed over")
|
||||
}
|
||||
if e == nil {
|
||||
current = false
|
||||
continue
|
||||
}
|
||||
hand(e, true)
|
||||
case <-timer.C:
|
||||
_ = w.Stop()
|
||||
return nil, fmt.Errorf("the ledger's state %s did not hand its current states in %s", s.bucket, StateTimeout)
|
||||
}
|
||||
}
|
||||
go func() {
|
||||
for e := range w.Updates() {
|
||||
if e != nil {
|
||||
hand(e, false)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return func() { _ = w.Stop() }, nil
|
||||
}
|
||||
|
||||
// LedgerIndexes are the buckets of a ledger binding's declared indexes, by index name, on the runner's own connection.
|
||||
func (c *Conn) LedgerIndexes(b membership.BusBinding) map[string]ledger.Index {
|
||||
out := map[string]ledger.Index{}
|
||||
for name, bucket := range b.Indexes {
|
||||
out[name] = &busState{c: c, bucket: bucket}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *busState) Entries(pattern string, limit int) ([]ledger.Indexed, error) {
|
||||
if pattern == "" {
|
||||
pattern = ">"
|
||||
}
|
||||
keys, err := s.Keys()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sort.Strings(keys)
|
||||
var out []ledger.Indexed
|
||||
for _, k := range keys {
|
||||
if limit > 0 && len(out) >= limit {
|
||||
break
|
||||
}
|
||||
if !SubjectMatches(pattern, k) {
|
||||
continue
|
||||
}
|
||||
v, _, err := s.Get(k)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if v != nil {
|
||||
out = append(out, ledger.Indexed{Key: k, Value: v})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (s *busState) Put(key string, value []byte) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = kv.Put(ctx, key, value)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *busState) Delete(key string) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := kv.Delete(ctx, key); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+172
-48
@@ -1,6 +1,7 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -8,6 +9,7 @@ import (
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
@@ -15,7 +17,7 @@ import (
|
||||
)
|
||||
|
||||
// A module's log on the bus (novox/hq ADR 0297): a stream of operations, one subject per key, that the
|
||||
// controller creates from what the module declared under `logs` and issues to the module's own instances
|
||||
// mesh creates from what the module declared under `logs` and issues to the module's own instances
|
||||
// in its membership. It mirrors a bucket (ADR 0201) in how it is reached: what a module may reach is
|
||||
// decided here, from its membership, and refused with the reason before anything is sent.
|
||||
//
|
||||
@@ -94,6 +96,10 @@ func logFilter(l LogIssued, pattern string) (string, error) {
|
||||
}
|
||||
parts := strings.Split(pattern, ".")
|
||||
for i, p := range parts {
|
||||
if p == "**" && i == len(parts)-1 {
|
||||
// `**` for the rest, as a state's key pattern and mesh-sdk's log reader say it.
|
||||
parts[i], p = ">", ">"
|
||||
}
|
||||
switch {
|
||||
case p == ">" && i == len(parts)-1, p == "*", logToken.MatchString(p):
|
||||
default:
|
||||
@@ -104,7 +110,7 @@ func logFilter(l LogIssued, pattern string) (string, error) {
|
||||
if len(pattern) > LogKeyMax {
|
||||
return "", fmt.Errorf("%q is longer than a key can be (%d bytes)", pattern, LogKeyMax)
|
||||
}
|
||||
return l.Subject + "." + pattern, nil
|
||||
return l.Subject + "." + strings.Join(parts, "."), nil
|
||||
}
|
||||
|
||||
// issuedLog is the log a module may reach by a name, or the reason it may not.
|
||||
@@ -126,13 +132,14 @@ func (c *Conn) issuedLog(module, name string) (LogIssued, error) {
|
||||
if len(names) > 0 {
|
||||
issued = strings.Join(names, ", ")
|
||||
}
|
||||
return LogIssued{}, fmt.Errorf("%s keeps no log called %q: a module declares the logs it keeps under `logs`, "+
|
||||
"and a log is its owner's alone; it was issued: %s (novox/hq ADR 0297)", module, name, issued)
|
||||
return LogIssued{}, fmt.Errorf("%s keeps no log called %q: a module declares the logs it keeps under `logs`, and "+
|
||||
"another module's it reads under `reads` as <module>.<log>; it was issued: %s (novox/hq ADR 0297, ADR 0571)",
|
||||
module, name, issued)
|
||||
}
|
||||
|
||||
// notOnTheBus says a log is issued and its stream is not there yet.
|
||||
func notOnTheBus(l LogIssued) error {
|
||||
return fmt.Errorf("the log %q is issued and its stream %s is not on the bus yet: the controller creates "+
|
||||
return fmt.Errorf("the log %q is issued and its stream %s is not on the bus yet: the mesh creates "+
|
||||
"it from the catalogue on its next raise (novox/hq ADR 0297)", l.Name, l.Stream)
|
||||
}
|
||||
|
||||
@@ -200,9 +207,10 @@ func (c *Conn) LogAppend(module, name, key string, value json.RawMessage, expect
|
||||
return ack.Seq, nil
|
||||
}
|
||||
|
||||
// LogRead is the entries from a sequence on, of the keys matching a pattern, at most limit of them and
|
||||
// about 4 MiB, by one batch direct get.
|
||||
func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int) (*LogPage, error) {
|
||||
// LogRead is the entries from a sequence on, or from a time on (since, when not zero), of the keys matching a pattern, at
|
||||
// most limit of them and about 4 MiB, by one batch direct get. A log another module keeps is read as its own is, by the
|
||||
// name its reads declare, `<module>.<log>` (novox/hq ADR 0571).
|
||||
func (c *Conn) LogRead(module, name string, from uint64, since time.Time, keys string, limit int) (*LogPage, error) {
|
||||
l, err := c.issuedLog(module, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -218,52 +226,24 @@ func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if from != 0 && !since.IsZero() {
|
||||
return nil, fmt.Errorf("a read of %s starts from a sequence or from a time, not both", name)
|
||||
}
|
||||
if from == 0 {
|
||||
from = 1
|
||||
}
|
||||
req, _ := json.Marshal(map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes})
|
||||
inbox := c.nc.NewInbox()
|
||||
sub, err := c.nc.SubscribeSync(inbox)
|
||||
req := map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes}
|
||||
if !since.IsZero() {
|
||||
delete(req, "seq")
|
||||
req["start_time"] = since.UTC()
|
||||
}
|
||||
entries, more, err := c.logBatch(l, req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = sub.Unsubscribe() }()
|
||||
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+l.Stream, inbox, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
page := &LogPage{Entries: []LogEntry{}}
|
||||
deadline := time.Now().Add(StateTimeout)
|
||||
for done := false; !done; {
|
||||
msg, err := sub.NextMsg(time.Until(deadline))
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return nil, notOnTheBus(l)
|
||||
}
|
||||
if errors.Is(err, nats.ErrTimeout) {
|
||||
return nil, fmt.Errorf("the read of %s did not end in %s", name, StateTimeout)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
switch status := msg.Header.Get("Status"); status {
|
||||
case "":
|
||||
e, err := logEntryOf(l, msg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
page.Entries = append(page.Entries, *e)
|
||||
case "204": // the end of the batch: what is still pending after the last entry sent
|
||||
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
|
||||
if pending > 0 && len(page.Entries) > 0 {
|
||||
page.Next = page.Entries[len(page.Entries)-1].Sequence + 1
|
||||
}
|
||||
done = true
|
||||
case "404": // nothing from that sequence on
|
||||
done = true
|
||||
case "503":
|
||||
return nil, notOnTheBus(l)
|
||||
default:
|
||||
return nil, fmt.Errorf("the read of %s was refused by the bus: %s %s", name, status, msg.Header.Get("Description"))
|
||||
}
|
||||
page := &LogPage{Entries: entries}
|
||||
if more && len(entries) > 0 {
|
||||
page.Next = entries[len(entries)-1].Sequence + 1
|
||||
}
|
||||
info, err := c.logInfo(l)
|
||||
if err != nil {
|
||||
@@ -273,6 +253,150 @@ func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int)
|
||||
return page, nil
|
||||
}
|
||||
|
||||
// LogNewest is the newest entry of each key matching a pattern, in the log's order, by one batch direct get (mesh-sdk
|
||||
// process.LogReader.Newest).
|
||||
func (c *Conn) LogNewest(module, name, keys string) ([]LogEntry, error) {
|
||||
l, err := c.issuedLog(module, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
filter, err := logFilter(l, keys)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries, more, err := c.logBatch(l, map[string]any{"multi_last": []string{filter}, "batch": LogReadMax})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if more {
|
||||
return nil, fmt.Errorf("%s holds more than %d keys matching %q: name fewer", name, LogReadMax, keys)
|
||||
}
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// logBatch is one batch direct get of a log: its entries, and whether more are pending after them.
|
||||
func (c *Conn) logBatch(l LogIssued, request map[string]any) ([]LogEntry, bool, error) {
|
||||
req, _ := json.Marshal(request)
|
||||
inbox := c.nc.NewInbox()
|
||||
sub, err := c.nc.SubscribeSync(inbox)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
defer func() { _ = sub.Unsubscribe() }()
|
||||
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+l.Stream, inbox, req); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
entries := []LogEntry{}
|
||||
deadline := time.Now().Add(StateTimeout)
|
||||
for {
|
||||
msg, err := sub.NextMsg(time.Until(deadline))
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return nil, false, notOnTheBus(l)
|
||||
}
|
||||
if errors.Is(err, nats.ErrTimeout) {
|
||||
return nil, false, fmt.Errorf("the read of %s did not end in %s", l.Name, StateTimeout)
|
||||
}
|
||||
return nil, false, err
|
||||
}
|
||||
switch status := msg.Header.Get("Status"); status {
|
||||
case "":
|
||||
e, err := logEntryOf(l, msg)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
entries = append(entries, *e)
|
||||
case "204": // the end of the batch: what is still pending after the last entry sent
|
||||
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
|
||||
return entries, pending > 0, nil
|
||||
case "404": // nothing from there on
|
||||
return entries, false, nil
|
||||
case "503":
|
||||
return nil, false, notOnTheBus(l)
|
||||
default:
|
||||
return nil, false, fmt.Errorf("the read of %s was refused by the bus: %s %s", l.Name, status, msg.Header.Get("Description"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// LogWatch hands deliver every entry of the keys matching a pattern from a sequence on (from), from a time on (since),
|
||||
// or, with neither, each new one, in the log's order, one at a time, through an ordered consumer it lets go of when
|
||||
// stopped (mesh-sdk process.LogReader.Watch). It returns once the entries there at its start are handed over; an error
|
||||
// from deliver is said, and the watch goes on with the next entry.
|
||||
func (c *Conn) LogWatch(module, name, keys string, from uint64, since time.Time, deliver func(LogEntry) error) (func(), error) {
|
||||
l, err := c.issuedLog(module, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
filter, err := logFilter(l, keys)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg := jetstream.OrderedConsumerConfig{FilterSubjects: []string{filter}, DeliverPolicy: jetstream.DeliverNewPolicy}
|
||||
switch {
|
||||
case from != 0 && !since.IsZero():
|
||||
return nil, fmt.Errorf("a watch of %s starts from a sequence or from a time, not both", name)
|
||||
case from != 0:
|
||||
cfg.DeliverPolicy, cfg.OptStartSeq = jetstream.DeliverByStartSequencePolicy, from
|
||||
case !since.IsZero():
|
||||
at := since.UTC()
|
||||
cfg.DeliverPolicy, cfg.OptStartTime = jetstream.DeliverByStartTimePolicy, &at
|
||||
}
|
||||
js, err := jetstream.New(c.nc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
cons, err := js.OrderedConsumer(ctx, l.Stream, cfg)
|
||||
if errors.Is(err, jetstream.ErrStreamNotFound) {
|
||||
return nil, notOnTheBus(l)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
|
||||
}
|
||||
info, err := cons.Info(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
|
||||
}
|
||||
it, err := cons.Messages()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
|
||||
}
|
||||
hand := func(m jetstream.Msg) error {
|
||||
meta, err := m.Metadata()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
e := LogEntry{Sequence: meta.Sequence.Stream, Key: strings.TrimPrefix(m.Subject(), l.Subject+"."),
|
||||
Value: valueOf(m.Data()), At: meta.Timestamp.UTC()}
|
||||
return deliver(e)
|
||||
}
|
||||
for pending := info.NumPending; pending > 0; pending-- {
|
||||
m, err := it.Next()
|
||||
if err == nil {
|
||||
err = hand(m)
|
||||
}
|
||||
if err != nil {
|
||||
it.Stop()
|
||||
return nil, fmt.Errorf("the watch of %s: %w", name, err)
|
||||
}
|
||||
}
|
||||
var stopped atomic.Bool
|
||||
go func() {
|
||||
for {
|
||||
m, err := it.Next()
|
||||
if err != nil || stopped.Load() {
|
||||
return // stopped, or the connection closed; a message fetched as it stopped is not handed over
|
||||
}
|
||||
if err := hand(m); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take entry of %s: %v; it is in the log, a read finds it", module, name, err)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return func() { stopped.Store(true); it.Stop() }, nil
|
||||
}
|
||||
|
||||
// LogLast is a key's last entry, or nil when it has none.
|
||||
func (c *Conn) LogLast(module, name, key string) (*LogEntry, error) {
|
||||
l, err := c.issuedLog(module, name)
|
||||
|
||||
@@ -1,13 +1,7 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/wire"
|
||||
)
|
||||
@@ -15,97 +9,8 @@ import (
|
||||
// An answer larger than the bus carries in one message (novox/hq issue 314).
|
||||
//
|
||||
// The bus carries one message of at most its max_payload, and the client library refuses a larger one
|
||||
// before it leaves the process that sent it. Until this, a holder whose answer was too large wrote one
|
||||
// line to its own log and its caller waited out RequestTimeout and read "did not answer" — the
|
||||
// controller's `conditions` and `status` for hours on 2026-10-08, and the operator's channel with them.
|
||||
//
|
||||
// The controller pages such an answer (mesh-controller internal/link, pages.go): it answers an error
|
||||
// carrying `paged` — the call it holds the answer under, its size and its parts — and a caller asks the
|
||||
// same subject again once per part with PageHeader, each part answered raw with PartHeader. Asking the
|
||||
// same subject means no grant beyond the one the caller already had. A module's answer too large is
|
||||
// refused here, loudly, in a message that fits: its instances may be several, and a part asked again
|
||||
// could reach another.
|
||||
|
||||
const (
|
||||
// PageHeader asks for one part of a paged answer: `<call> <part>`, from zero.
|
||||
PageHeader = "Mesh-Page"
|
||||
// PartHeader is on a part: `<part>/<parts>`.
|
||||
PartHeader = "Mesh-Part"
|
||||
// partTries is how often one part is asked: during a handover a part can reach the controller that
|
||||
// did not hold it.
|
||||
partTries = 3
|
||||
)
|
||||
|
||||
// PartTimeout bounds the asking of one part.
|
||||
var PartTimeout = 10 * time.Second
|
||||
|
||||
// paged is what a too-large answer says instead of itself.
|
||||
type paged struct {
|
||||
Call string `json:"call"`
|
||||
Bytes int `json:"bytes"`
|
||||
Parts int `json:"parts"`
|
||||
}
|
||||
|
||||
func pagedIn(data []byte) (paged, bool) {
|
||||
if !bytes.Contains(data, []byte(`"paged"`)) {
|
||||
return paged{}, false
|
||||
}
|
||||
var r struct {
|
||||
Paged *paged `json:"paged"`
|
||||
}
|
||||
if json.Unmarshal(data, &r) != nil || r.Paged == nil || r.Paged.Call == "" || r.Paged.Parts < 1 {
|
||||
return paged{}, false
|
||||
}
|
||||
return *r.Paged, true
|
||||
}
|
||||
|
||||
// whole is a reply's whole answer: the reply itself, or every part of a paged one, joined. A part that
|
||||
// cannot be had is an error naming it, never a shorter answer.
|
||||
func (c *Conn) whole(subject string, data []byte) ([]byte, error) {
|
||||
p, ok := pagedIn(data)
|
||||
if !ok {
|
||||
return data, nil
|
||||
}
|
||||
out := make([]byte, 0, p.Bytes)
|
||||
for n := 0; n < p.Parts; n++ {
|
||||
var part []byte
|
||||
var err error
|
||||
for try := 0; try < partTries; try++ {
|
||||
if part, err = c.askPart(subject, p, n); err == nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the answer was %d bytes, more than the bus carries in one message, paged as %s "+
|
||||
"in %d parts, and part %d could not be had: %w", p.Bytes, p.Call, p.Parts, n, err)
|
||||
}
|
||||
out = append(out, part...)
|
||||
}
|
||||
if len(out) != p.Bytes {
|
||||
return nil, fmt.Errorf("the answer paged as %s was %d bytes, and its parts joined are %d", p.Call, p.Bytes, len(out))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *Conn) askPart(subject string, p paged, n int) ([]byte, error) {
|
||||
ask := nats.NewMsg(subject)
|
||||
ask.Header.Set(PageHeader, fmt.Sprintf("%s %d", p.Call, n))
|
||||
ask.Data = []byte(`{}`)
|
||||
reply, err := c.nc.RequestMsg(ask, PartTimeout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if got := reply.Header.Get(PartHeader); got != fmt.Sprintf("%d/%d", n, p.Parts) {
|
||||
var r struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
if json.Unmarshal(reply.Data, &r) == nil && r.Error != "" {
|
||||
return nil, errors.New(r.Error)
|
||||
}
|
||||
return nil, fmt.Errorf("asked part %d/%d, answered %q", n, p.Parts, got)
|
||||
}
|
||||
return reply.Data, nil
|
||||
}
|
||||
// before it leaves the process that sent it. A module's answer too large is refused here, loudly, in a
|
||||
// message that fits, so its caller reads why instead of waiting out RequestTimeout.
|
||||
|
||||
// fits is an answer the bus can carry: the answer itself, or — when it is larger than one message —
|
||||
// an error that says so, which is.
|
||||
|
||||
@@ -2,7 +2,6 @@ package bus_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -13,8 +12,8 @@ import (
|
||||
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// novox/hq issue 314: an answer larger than the bus carries in one message arrives whole when the
|
||||
// controller pages it, and a module's is said to be too large — neither is a 30-second silence.
|
||||
// novox/hq issue 314: a module's answer larger than the bus carries in one message is said to be too large,
|
||||
// at once, not a 30-second silence. (The controller's paging ended with the controller, issue 1126.)
|
||||
func TestAnAnswerLargerThanOneMessage(t *testing.T) {
|
||||
url := meshtest.URL(t)
|
||||
seat, err := nats.Connect(url)
|
||||
@@ -27,43 +26,8 @@ func TestAnAnswerLargerThanOneMessage(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer c.Close()
|
||||
limit := int(seat.MaxPayload())
|
||||
large := strings.Repeat("a condition held past its bound; ", 3*int(seat.MaxPayload())/32)
|
||||
|
||||
// The controller as it pages (mesh-controller internal/link, pages.go): the first answer an error
|
||||
// with `paged`, each part asked again on the same subject with the page header.
|
||||
large := strings.Repeat("a condition held past its bound; ", 3*limit/32)
|
||||
whole, _ := json.Marshal(map[string]any{"result": large})
|
||||
partSize := limit - 1024
|
||||
parts := (len(whole) + partSize - 1) / partSize
|
||||
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", "conditions", "", ""), func(m *nats.Msg) {
|
||||
if page := m.Header.Get(bus.PageHeader); page != "" {
|
||||
var id string
|
||||
var n int
|
||||
fmt.Sscanf(page, "%s %d", &id, &n)
|
||||
end := min((n+1)*partSize, len(whole))
|
||||
part := nats.NewMsg(m.Reply)
|
||||
part.Header.Set(bus.PartHeader, fmt.Sprintf("%d/%d", n, parts))
|
||||
part.Data = whole[n*partSize : end]
|
||||
_ = m.RespondMsg(part)
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"error": "too large for one message",
|
||||
"paged": map[string]any{"call": "call-1-1", "bytes": len(whole), "parts": parts}})
|
||||
_ = m.Respond(body)
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer sub.Unsubscribe()
|
||||
_ = seat.Flush()
|
||||
|
||||
got, err := c.Ask("seat:mesh-controller.conditions", map[string]any{}, "")
|
||||
var said string
|
||||
if err != nil || json.Unmarshal(got.Result, &said) != nil || said != large {
|
||||
t.Fatalf("an answer of %d bytes in %d parts arrived as %d bytes: %v", len(whole), parts, len(got.Result), err)
|
||||
}
|
||||
|
||||
// A module's answer too large is refused here in words, at once.
|
||||
stop, err := c.HandleSubject("mesh.mod.journal.tool.lines", func(json.RawMessage) (any, error) { return large, nil })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/pernode"
|
||||
)
|
||||
|
||||
// A state its owner keeps one bucket per node (novox/hq ADR 0593 §6: each node's engine writes its own `composed-<node>`,
|
||||
// and the provisioner, the assigner and the view read them all) is served to a bundle as one state. The rules (a key's
|
||||
// first part names its node's bucket, keys and a watch over every bucket of the prefix, a bucket created later heard by
|
||||
// the bus's creation advisory) are mesh-sdk's pernode package, written once for a process and for this runtime; this
|
||||
// file adapts its entries to the runtime's shapes. A reader is granted the listing of stream names
|
||||
// (pernode.ListSubject) and the creation advisory (pernode.CreatedAdvisory) beside each bucket's read.
|
||||
|
||||
func perNodeRefusal(module, name, what string) error {
|
||||
return fmt.Errorf("%s's state %s is kept one bucket per node and read only: no %s of it as one state (novox/hq ADR 0593 §6)",
|
||||
module, name, what)
|
||||
}
|
||||
|
||||
func (c *Conn) family(s StateIssued) pernode.Family {
|
||||
return pernode.Family{Read: s.Name, Prefix: s.Bucket, Bus: pernode.OverNATS(c.nc)}
|
||||
}
|
||||
|
||||
func (c *Conn) perNodeGet(s StateIssued, key string) (*StateEntry, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
e, err := c.family(s).Get(ctx, key)
|
||||
if err != nil || e == nil {
|
||||
return nil, err
|
||||
}
|
||||
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision(), At: e.Created().UTC()}, nil
|
||||
}
|
||||
|
||||
func (c *Conn) perNodeKeys(s StateIssued) ([]string, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
return c.family(s).Keys(ctx)
|
||||
}
|
||||
|
||||
// perNodeWatch is StateWatch over every bucket of the family, and over a bucket created later. As in a single bucket's
|
||||
// watch, StateTimeout bounds the time without progress (issue 1726), and a value the bundle does not take is said and
|
||||
// the watch goes on.
|
||||
func (c *Conn) perNodeWatch(module string, s StateIssued, pattern string, deliver func(StateChange) error) (func(), error) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
progress := make(chan struct{}, 1)
|
||||
handle := func(ch pernode.Change) error {
|
||||
select {
|
||||
case progress <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
e := ch.Entry
|
||||
change := StateChange{State: s.Name, Key: e.Key(), Revision: e.Revision(), Current: ch.Current}
|
||||
switch e.Operation() {
|
||||
case jetstream.KeyValuePut:
|
||||
change.Op = "put"
|
||||
change.Value = valueOf(e.Value())
|
||||
default:
|
||||
if ch.Current {
|
||||
return nil
|
||||
}
|
||||
change.Op = "delete"
|
||||
if e.Operation() == jetstream.KeyValuePurge {
|
||||
change.Op = "expire"
|
||||
}
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
if err := deliver(change); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
|
||||
module, change.Op, change.Key, s.Name, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- c.family(s).Watch(ctx, pattern, handle) }()
|
||||
bound := time.NewTimer(StateTimeout)
|
||||
defer bound.Stop()
|
||||
for {
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
cancel()
|
||||
return nil, err
|
||||
}
|
||||
return cancel, nil
|
||||
case <-progress:
|
||||
if !bound.Stop() {
|
||||
select {
|
||||
case <-bound.C:
|
||||
default:
|
||||
}
|
||||
}
|
||||
bound.Reset(StateTimeout)
|
||||
case <-bound.C:
|
||||
cancel()
|
||||
return nil, fmt.Errorf("the current values of %s did not arrive in %s: none was handed over in that time", s.Name, StateTimeout)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,16 +14,16 @@ import (
|
||||
// ADR 0370 changes it, issue 685).
|
||||
//
|
||||
// While a module's code runs here, its asks reach the bus under this runtime's account, and the bus proves only
|
||||
// that account. So the runtime asks its own machine's client on ProvisionAskSubject — which the controller grants
|
||||
// that account. So the runtime asks its own machine's client on ProvisionAskSubject — which the mesh grants
|
||||
// this machine's tool runner alone to publish (ADR 0370 §3) — naming the module itself, from the bundle that asks
|
||||
// and never from the module's code or arguments (§4). The client answers that module's config values, for what its
|
||||
// manifest declares, and no secret: the vault hands a secret only to a caller the bus proves (§2).
|
||||
|
||||
// ProvisionAskSubject is where this machine's tool runner asks its provisioner client (mesh-controller
|
||||
// ProvisionAskSubject is where this machine's tool runner asks its provisioner client (the node-engine
|
||||
// internal/broker ProvisionerAskSubject): `prov.ask.<node>`.
|
||||
func ProvisionAskSubject(node string) string { return "prov.ask." + node }
|
||||
|
||||
// ProvisionChangedSubjects is every module's word of what moved on a machine (mesh-controller internal/broker
|
||||
// ProvisionChangedSubjects is every module's word of what moved on a machine (the node-engine
|
||||
// ProvisionerChangedSubject): `prov.changed.<node>.<module>`, a module's name dots and all.
|
||||
func ProvisionChangedSubjects(node string) string { return "prov.changed." + node + ".>" }
|
||||
|
||||
|
||||
@@ -84,3 +84,33 @@ func TestARefusedSubscriptionIsAskedForAgain(t *testing.T) {
|
||||
t.Error("a stopped subject is still tracked")
|
||||
}
|
||||
}
|
||||
|
||||
// Issue 1966: a refused publish of an answer (to a caller's inbox) is said, once per subject per minute, never dropped
|
||||
// in silence while the caller waits out its time.
|
||||
func TestARefusedAnswerIsSaidOncePerMinute(t *testing.T) {
|
||||
c := &Conn{}
|
||||
var said []string
|
||||
c.Logf = func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }
|
||||
refuse := func(subject string) bool {
|
||||
return c.answerRefused(fmt.Errorf("%w: Permissions Violation for Publish to %q", nats.ErrPermissionViolation, subject))
|
||||
}
|
||||
// A reply subject ends in a token new to every request: a burst under one prefix is one line.
|
||||
if !refuse("_INBOX.user.abc.t1") || !refuse("_INBOX.user.abc.t2") || !refuse("_INBOX.user.def.t1") {
|
||||
t.Fatal("a refused answer was not taken for one")
|
||||
}
|
||||
if len(said) != 2 {
|
||||
t.Fatalf("want one line per caller, got %v", said)
|
||||
}
|
||||
c.inboxSaid["_INBOX.user.abc"] = time.Now().Add(-2 * time.Minute)
|
||||
c.inboxSaid["_INBOX.user.old"] = time.Now().Add(-2 * time.Minute)
|
||||
refuse("_INBOX.user.abc.t3")
|
||||
if len(said) != 3 || !strings.Contains(said[2], "_INBOX.user.abc") {
|
||||
t.Fatalf("a minute later it is said again, got %v", said)
|
||||
}
|
||||
if _, kept := c.inboxSaid["_INBOX.user.old"]; kept || len(c.inboxSaid) != 2 {
|
||||
t.Fatalf("entries older than a minute are dropped on write, kept %v", c.inboxSaid)
|
||||
}
|
||||
if refuse("mesh.mod.x.tool.y") {
|
||||
t.Fatal("a refused publish that is no answer was taken for one")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,7 +24,7 @@ import (
|
||||
// and anything else is refused here, with the reason, before it is sent. A bundle reaches this only through
|
||||
// its own standard input and output: no tool call does.
|
||||
|
||||
// SeatTraffic is what a membership lists of a module's seat traffic (the controller's broker.SeatTraffic).
|
||||
// SeatTraffic is what a membership lists of a module's seat traffic (the grants part's record).
|
||||
type SeatTraffic struct {
|
||||
Publish []string `json:"publish,omitempty"`
|
||||
Subscribe []string `json:"subscribe,omitempty"`
|
||||
@@ -32,7 +32,7 @@ type SeatTraffic struct {
|
||||
Workers []SeatWorker `json:"workers,omitempty"`
|
||||
Records []string `json:"records,omitempty"`
|
||||
// Kinds are the holders of every kinded bench the module uses or watches, with what each promises: the
|
||||
// controller's record of their claims (novox/hq ADR 0259 §5).
|
||||
// record of their claims (novox/hq ADR 0259 §5).
|
||||
Kinds []KindHeld `json:"kinds,omitempty"`
|
||||
}
|
||||
|
||||
@@ -45,7 +45,7 @@ type KindHeld struct {
|
||||
Capabilities []string `json:"capabilities,omitempty"`
|
||||
}
|
||||
|
||||
// SeatKinds is the kinds the module's membership names, as the controller issued them.
|
||||
// SeatKinds is the kinds the module's membership names, as the grants part issued them.
|
||||
func (c *Conn) SeatKinds(module string) ([]KindHeld, error) {
|
||||
t, err := c.trafficOf(module)
|
||||
if err != nil {
|
||||
@@ -249,7 +249,7 @@ type Work struct {
|
||||
|
||||
// SeatTake reads a worker the module's seat traffic lists and hands each message to deliver: acknowledged
|
||||
// when deliver returns nil, offered again after NakDelay when it returns an error, ended when it is not
|
||||
// JSON. The worker is the controller's to create; this binds to it by name and never makes one.
|
||||
// JSON. The worker is the mesh's to create; this binds to it by name and never makes one.
|
||||
func (c *Conn) SeatTake(module, consumer string, deliver func(Work) error) (func(), error) {
|
||||
t, err := c.trafficOf(module)
|
||||
if err != nil {
|
||||
@@ -370,7 +370,13 @@ func (c *Conn) SeatRecord(module, bucket, key string) (*StateEntry, error) {
|
||||
}
|
||||
var revision uint64
|
||||
fmt.Sscan(reply.Header.Get("Nats-Sequence"), &revision)
|
||||
return &StateEntry{Key: module + "." + key, Value: valueOf(reply.Data), Revision: revision}, nil
|
||||
var at time.Time
|
||||
if stamp := reply.Header.Get("Nats-Time-Stamp"); stamp != "" {
|
||||
if t, err := time.Parse(time.RFC3339Nano, stamp); err == nil {
|
||||
at = t.UTC()
|
||||
}
|
||||
}
|
||||
return &StateEntry{Key: module + "." + key, Value: valueOf(reply.Data), Revision: revision, At: at}, nil
|
||||
}
|
||||
|
||||
// StateCreate writes one key only when it has no value, and answers its revision; a key that has one is
|
||||
|
||||
@@ -147,11 +147,27 @@ const (
|
||||
SettleNak = "nak" // handed back, to be delivered again at once
|
||||
SettleTerm = "term" // dropped
|
||||
SettleProgress = "progress" // still being worked: its acknowledgement wait starts again
|
||||
// SettleNakAfter hands the message back to be delivered again after a delay: the bus's own redelivery, so no part
|
||||
// keeps a timer for a step given again (novox/hq ADR 0664 §1; issue 1748). SeatSettleAfter gives it.
|
||||
SettleNakAfter = "nak-after"
|
||||
)
|
||||
|
||||
// SeatSettle gives the message module pulled under id its fate. A message pulled for another module, or settled
|
||||
// already, is refused.
|
||||
func (c *Conn) SeatSettle(module, id, fate string) error {
|
||||
return c.settle(module, id, fate, 0)
|
||||
}
|
||||
|
||||
// SeatSettleAfter hands the message module pulled under id back to be delivered again after delay (SettleNakAfter),
|
||||
// counting one more delivery. A delay of 0 or below is refused and settles nothing.
|
||||
func (c *Conn) SeatSettleAfter(module, id string, delay time.Duration) error {
|
||||
if delay <= 0 {
|
||||
return fmt.Errorf("a message given again after a delay needs a delay above 0, not %s", delay)
|
||||
}
|
||||
return c.settle(module, id, SettleNakAfter, delay)
|
||||
}
|
||||
|
||||
func (c *Conn) settle(module, id, fate string, delay time.Duration) error {
|
||||
seatPulls.mu.Lock()
|
||||
p, ok := seatPulls.held[id]
|
||||
if ok && p.module != module {
|
||||
@@ -173,9 +189,11 @@ func (c *Conn) SeatSettle(module, id, fate string) error {
|
||||
return p.msg.Term()
|
||||
case SettleProgress:
|
||||
return p.msg.InProgress()
|
||||
case SettleNakAfter:
|
||||
return p.msg.NakWithDelay(delay)
|
||||
}
|
||||
seatPulls.mu.Lock()
|
||||
seatPulls.held[id] = p // an unknown fate settles nothing
|
||||
seatPulls.mu.Unlock()
|
||||
return fmt.Errorf("a pulled message's fate is ack, nak, term or progress, not %q", fate)
|
||||
return fmt.Errorf("a pulled message's fate is ack, nak, nak-after, term or progress, not %q", fate)
|
||||
}
|
||||
|
||||
@@ -58,6 +58,21 @@ func TestAHolderPullsOneMessageWhenItHasRoom(t *testing.T) {
|
||||
if err != nil || again == nil || string(again.Body) != `{"id":"a2"}` || again.Delivered != 2 {
|
||||
t.Fatalf("the message handed back = %+v, %v", again, err)
|
||||
}
|
||||
// Issue 1748: a message settled with a delay is not given again at once, is after it, one delivery higher; a delay
|
||||
// of 0 settles nothing.
|
||||
if err := conn.SeatSettleAfter("messenger", again.ID, 0); err == nil {
|
||||
t.Error("a delay of 0 was accepted")
|
||||
}
|
||||
if err := conn.SeatSettleAfter("messenger", again.ID, 300*time.Millisecond); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if early, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 0); err != nil || early != nil {
|
||||
t.Fatalf("a message given again after a delay came at once: %+v, %v", early, err)
|
||||
}
|
||||
late, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 2*time.Second)
|
||||
if err != nil || late == nil || late.Delivered != 3 {
|
||||
t.Fatalf("the message given again after its delay = %+v, %v", late, err)
|
||||
}
|
||||
if _, err := conn.SeatPull("telegram", "SEAT_OPERATOR_CHANNEL_worker", 0); err == nil || !strings.Contains(err.Error(), "membership lists") {
|
||||
t.Errorf("a channel pulled the router's work: %v", err)
|
||||
}
|
||||
|
||||
@@ -98,7 +98,7 @@ func TestAModuleSubmitsAndSaysOnlyUnderItsOwnNameAndKind(t *testing.T) {
|
||||
t.Fatalf("an asker could not ask under its own name: %v", err)
|
||||
}
|
||||
for module, subject := range map[string]string{
|
||||
"mesh-delivery": "mesh.seat.operator-channel.accept.ask.mesh-controller",
|
||||
"mesh-delivery": "mesh.seat.operator-channel.accept.ask.mesh-module-assigner",
|
||||
"telegram": "mesh.seat.intake.event.choice.desktop",
|
||||
"messenger": "mesh.seat.intake.event.choice.telegram",
|
||||
} {
|
||||
@@ -206,13 +206,16 @@ func TestAnAskerReadsItsOwnRecordAndNoOther(t *testing.T) {
|
||||
if _, err := kv.Put("mesh-delivery.a1", []byte(`{"state":"open"}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := kv.Put("mesh-controller.a1", []byte(`{"state":"open"}`)); err != nil {
|
||||
if _, err := kv.Put("mesh-module-assigner.a1", []byte(`{"state":"open"}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := conn.SeatRecord("mesh-delivery", "messenger_asks", "a1")
|
||||
if err != nil || got == nil || string(got.Value) != `{"state":"open"}` {
|
||||
t.Fatalf("the asker could not read its own ask: %v %v", got, err)
|
||||
}
|
||||
if got.At.IsZero() || time.Since(got.At) > time.Minute {
|
||||
t.Errorf("the record does not say when the bus wrote it: at %v", got.At)
|
||||
}
|
||||
if missing, err := conn.SeatRecord("mesh-delivery", "messenger_asks", "a2"); err != nil || missing != nil {
|
||||
t.Fatalf("a missing ask read as %v, %v", missing, err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/stateobjects"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// SentGet reads one object of a sender's store of sent objects by the reference that carried it (ADR 0666 §16): the
|
||||
// store is the sender's `<module>_sent-objects`, and no other — a state's objects are reached through their owner's
|
||||
// bindings alone. It reads as this connection's account, which the bus lets read a sender's sent store only where the
|
||||
// grants part granted it (a bundle is held to what its runtime channel carries by the runtime call, not here), from the manifests' callers and consumers; a refusal says the store.
|
||||
func (c *Conn) SentGet(r buscalls.ObjectsSentGetRequest) (*buscalls.ObjectData, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
obs, err := c.sentStore(ctx, r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
info, err := obs.GetInfo(ctx, r.Object)
|
||||
if errors.Is(err, jetstream.ErrObjectNotFound) {
|
||||
return nil, fmt.Errorf("the sent object %s is not in %s: it is kept 7 days, then expires", r.Object, r.Store)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the sent object %s of %s: %w", r.Object, r.Store, err)
|
||||
}
|
||||
data, err := obs.GetBytes(ctx, r.Object)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the sent object %s of %s: %w", r.Object, r.Store, err)
|
||||
}
|
||||
return &buscalls.ObjectData{Info: objectInfoOf(info), Data: data}, nil
|
||||
}
|
||||
|
||||
// sentStore checks the reference r and opens the sender's sent store it names.
|
||||
func (c *Conn) sentStore(ctx context.Context, r buscalls.ObjectsSentGetRequest) (jetstream.ObjectStore, error) {
|
||||
ref := stateobjects.Ref{Store: r.Store, Object: r.Object, Size: r.Size, Digest: r.Digest, Form: stateobjects.FormVersion}
|
||||
if _, ok := stateobjects.ReadRef(stateobjects.Wrap(ref)); !ok {
|
||||
return nil, errors.New("a sent object is named by the reference's store, object, size and sha256, as the reply or event carried them")
|
||||
}
|
||||
if !strings.HasSuffix(r.Store, catalogue.BusNameSeparator+catalogue.SentObjectsLocal) {
|
||||
return nil, fmt.Errorf("%s is no sent store (a module's is `<module>%s%s`): only a sent object is read by its reference",
|
||||
r.Store, catalogue.BusNameSeparator, catalogue.SentObjectsLocal)
|
||||
}
|
||||
js, err := c.JetStream()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
obs, err := js.ObjectStore(ctx, r.Store)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the sent store %s cannot be read as %s: it is not on the bus, or the bus grants this account no "+
|
||||
"read of it (the manifests' callers and consumers are granted): %w", r.Store, c.cred.Module, err)
|
||||
}
|
||||
return obs, nil
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// The bus side of a bundle's put and get sessions (design 47 G §518, A-21, A-25): a put written into the module's object
|
||||
// store as its chunks arrive and a get read out of it as they are asked, through the store's own chunking, so no more than
|
||||
// one chunk is held in memory. The session itself is the runtime's (internal/runtime/sessions.go).
|
||||
|
||||
// ObjectsPutStream stores the object r.Name of the module's object store r.Objects from in, read to its end, with the
|
||||
// put's description (A-25); it answers once the object is sealed. A read error of in drops the object: nothing is left.
|
||||
func (c *Conn) ObjectsPutStream(ctx context.Context, module string, r buscalls.ObjectsNameRequest, description string, in io.Reader) (buscalls.ObjectInfo, error) {
|
||||
if r.Name == "" {
|
||||
return buscalls.ObjectInfo{}, errors.New("an object is put under a name")
|
||||
}
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
info, err := obs.Put(ctx, jetstream.ObjectMeta{Name: r.Name, Description: description}, in)
|
||||
if err != nil {
|
||||
_ = obs.Delete(context.WithoutCancel(ctx), r.Name)
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
return objectInfoOf(info), nil
|
||||
}
|
||||
|
||||
// ObjectsOpen is one object of the module's object store as a reader of its bytes beside its info; found is false when there
|
||||
// is none. The caller closes the reader.
|
||||
func (c *Conn) ObjectsOpen(ctx context.Context, module string, r buscalls.ObjectsNameRequest) (info buscalls.ObjectInfo, rd io.ReadCloser, found bool, err error) {
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return info, nil, false, err
|
||||
}
|
||||
res, err := obs.Get(ctx, r.Name)
|
||||
if errors.Is(err, jetstream.ErrObjectNotFound) {
|
||||
return info, nil, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return info, nil, false, err
|
||||
}
|
||||
i, err := res.Info()
|
||||
if err != nil {
|
||||
res.Close()
|
||||
return info, nil, false, err
|
||||
}
|
||||
return objectInfoOf(i), res, true, nil
|
||||
}
|
||||
|
||||
// SentOpen is a sent object, read by its reference as SentGet reads it, as a reader of its bytes beside its info.
|
||||
func (c *Conn) SentOpen(ctx context.Context, r buscalls.ObjectsSentGetRequest) (buscalls.ObjectInfo, io.ReadCloser, error) {
|
||||
obs, err := c.sentStore(ctx, r)
|
||||
if err != nil {
|
||||
return buscalls.ObjectInfo{}, nil, err
|
||||
}
|
||||
res, err := obs.Get(ctx, r.Object)
|
||||
if errors.Is(err, jetstream.ErrObjectNotFound) {
|
||||
return buscalls.ObjectInfo{}, nil, fmt.Errorf("the sent object %s is not in %s: it is kept 7 days, then expires", r.Object, r.Store)
|
||||
}
|
||||
if err != nil {
|
||||
return buscalls.ObjectInfo{}, nil, fmt.Errorf("the sent object %s of %s: %w", r.Object, r.Store, err)
|
||||
}
|
||||
i, err := res.Info()
|
||||
if err != nil {
|
||||
res.Close()
|
||||
return buscalls.ObjectInfo{}, nil, err
|
||||
}
|
||||
return objectInfoOf(i), res, nil
|
||||
}
|
||||
@@ -11,10 +11,12 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"git.novox.be/novox/mesh-sdk/go/pernode"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// A module's state on the bus (novox/hq ADR 0201): key-value buckets the controller creates from what
|
||||
// A module's state on the bus (novox/hq ADR 0201): key-value buckets the mesh creates from what
|
||||
// the module declared, and issues to each assignment in its membership by the name the module uses —
|
||||
// its own state by the local name, another module's as `<module>.<name>`.
|
||||
//
|
||||
@@ -28,6 +30,10 @@ type StateIssued struct {
|
||||
Name string `json:"name"`
|
||||
Bucket string `json:"bucket"`
|
||||
Writes bool `json:"writes,omitempty"`
|
||||
// PerNode says the state is kept one bucket per node (novox/hq ADR 0593 §6, mesh-sdk membership.StateIssued): Bucket is
|
||||
// then the buckets' prefix, each bucket the prefix and a node's name, and a key's first part names its node. It is read
|
||||
// only, served as one state (per_node.go).
|
||||
PerNode bool `json:"per-node,omitempty"`
|
||||
}
|
||||
|
||||
// StateEntry is one key's current value.
|
||||
@@ -35,9 +41,12 @@ type StateEntry struct {
|
||||
Key string `json:"key"`
|
||||
Value json.RawMessage `json:"value"`
|
||||
Revision uint64 `json:"revision"`
|
||||
// At is when the bus wrote this value (the entry's created time), as stdio.StateEntry.At reads it.
|
||||
At time.Time `json:"at,omitzero"`
|
||||
}
|
||||
|
||||
// StateChange is one change a watch delivers: a key put or deleted.
|
||||
// StateChange is one change a watch delivers: a key put, deleted, or expired (its own time to live, or the
|
||||
// bucket's age, ended; issue 1952).
|
||||
type StateChange struct {
|
||||
State string `json:"state"`
|
||||
Key string `json:"key"`
|
||||
@@ -63,21 +72,22 @@ func (c *Conn) issuedState(module, name string) (StateIssued, error) {
|
||||
return StateIssued{}, fmt.Errorf("%s has no membership issued on %s yet, so no state of it is reachable "+
|
||||
"until the mesh issues one (novox/hq ADR 0201)", module, c.node)
|
||||
}
|
||||
var names []string
|
||||
// What a membership issues by a local name is mesh-sdk's one rule (membership.StateIssued, issue 1966): the old
|
||||
// `state` list, then a key-value entry the module asks for in `requires` (ADR 0595 §1), by its local name.
|
||||
issuing := membership.Membership{Bindings: m.Bindings}
|
||||
for _, s := range m.State {
|
||||
if s.Name == name {
|
||||
return s, nil
|
||||
}
|
||||
names = append(names, s.Name)
|
||||
issuing.State = append(issuing.State, membership.StateIssued{Name: s.Name, Bucket: s.Bucket, Writes: s.Writes, PerNode: s.PerNode})
|
||||
}
|
||||
if s, ok := issuing.StateIssued(name); ok {
|
||||
return StateIssued{Name: s.Name, Bucket: s.Bucket, Writes: s.Writes, PerNode: s.PerNode}, nil
|
||||
}
|
||||
sort.Strings(names)
|
||||
issued := "none"
|
||||
if len(names) > 0 {
|
||||
if names := issuing.StateNames(); len(names) > 0 {
|
||||
issued = strings.Join(names, ", ")
|
||||
}
|
||||
return StateIssued{}, fmt.Errorf("%s keeps and reads no state called %q: it declares what it keeps under "+
|
||||
"`state` and what it reads under `reads` as <module>.<name>, and was issued: %s (novox/hq ADR 0201)",
|
||||
module, name, issued)
|
||||
"`state` or as a key-value entry in `requires`, and what it reads under `reads` as <module>.<name>, and was "+
|
||||
"issued: %s (novox/hq ADR 0201, ADR 0595 §1)", module, name, issued)
|
||||
}
|
||||
|
||||
func (c *Conn) bucket(ctx context.Context, s StateIssued) (jetstream.KeyValue, error) {
|
||||
@@ -88,7 +98,7 @@ func (c *Conn) bucket(ctx context.Context, s StateIssued) (jetstream.KeyValue, e
|
||||
kv, err := js.KeyValue(ctx, s.Bucket)
|
||||
if err != nil {
|
||||
if errors.Is(err, jetstream.ErrBucketNotFound) {
|
||||
return nil, fmt.Errorf("the state %q is issued and its bucket is not on the bus yet: the controller "+
|
||||
return nil, fmt.Errorf("the state %q is issued and its bucket is not on the bus yet: the mesh "+
|
||||
"creates it from the catalogue on its next raise", s.Name)
|
||||
}
|
||||
return nil, err
|
||||
@@ -113,6 +123,9 @@ func (c *Conn) StateGet(module, name, key string) (*StateEntry, error) {
|
||||
if err := checkKey(key); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.PerNode {
|
||||
return c.perNodeGet(s, key)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := c.bucket(ctx, s)
|
||||
@@ -126,7 +139,126 @@ func (c *Conn) StateGet(module, name, key string) (*StateEntry, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision()}, nil
|
||||
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision(), At: e.Created().UTC()}, nil
|
||||
}
|
||||
|
||||
// StatePage is the current values after a key, in key order, and the key to ask after for the rest.
|
||||
type StatePage struct {
|
||||
Entries []StateEntry `json:"entries"`
|
||||
Next string `json:"next,omitempty"`
|
||||
}
|
||||
|
||||
// StatePageBudget is the time a page spends reading when the asker names none: well under the 30 s a reader waits.
|
||||
const StatePageBudget = 10 * time.Second
|
||||
|
||||
// PageClock is the time a page's budget counts by, replaced by a test.
|
||||
var PageClock = time.Now
|
||||
|
||||
// StatePage reads the values in key order after `after` in one pass, as many as fit maxBytes (each key and value),
|
||||
// stopping when budget is spent, at least one value, so a bundle asks one request for many values, where StateGet
|
||||
// is one request each. Access is checked once and the bucket bound once. It is a read like StateGet: served to the
|
||||
// owner and to every reader, and writes nothing. Next is the last key given when more follow.
|
||||
func (c *Conn) StatePage(module, name, after string, maxBytes int, budget time.Duration) (StatePage, error) {
|
||||
s, err := c.issuedState(module, name)
|
||||
if err != nil {
|
||||
return StatePage{}, err
|
||||
}
|
||||
if budget <= 0 {
|
||||
budget = StatePageBudget
|
||||
}
|
||||
// The bind and the listing have a StateTimeout of their own, and each get another: the page's time budget is what
|
||||
// ends a slow page (with Next), never a deadline spread over the whole of it.
|
||||
// ctx covers the bind and the listing alone; every bus request inside the loop takes each().
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
each := func() (context.Context, context.CancelFunc) {
|
||||
return context.WithTimeout(context.Background(), StateTimeout)
|
||||
}
|
||||
var keys []string
|
||||
var get func(key string) (*StateEntry, error)
|
||||
if s.PerNode {
|
||||
fam := c.family(s)
|
||||
if keys, err = fam.Keys(ctx); err != nil {
|
||||
return StatePage{}, err
|
||||
}
|
||||
buckets := map[string]jetstream.KeyValue{} // each node's bucket bound once
|
||||
get = func(key string) (*StateEntry, error) {
|
||||
node, err := pernode.NodeOf(key)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
kv, bound := buckets[node]
|
||||
if !bound {
|
||||
octx, ocancel := each() // the open is a bus request in the loop: its own deadline, never ctx's
|
||||
kv, err = fam.Open(octx, node)
|
||||
ocancel()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
buckets[node] = kv
|
||||
}
|
||||
if kv == nil {
|
||||
return nil, nil
|
||||
}
|
||||
gctx, gcancel := each()
|
||||
defer gcancel()
|
||||
e, err := kv.Get(gctx, key)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision(), At: e.Created().UTC()}, nil
|
||||
}
|
||||
} else {
|
||||
kv, err := c.bucket(ctx, s)
|
||||
if err != nil {
|
||||
return StatePage{}, err
|
||||
}
|
||||
if keys, err = keysOf(ctx, kv); err != nil {
|
||||
return StatePage{}, err
|
||||
}
|
||||
get = func(key string) (*StateEntry, error) {
|
||||
gctx, gcancel := each()
|
||||
defer gcancel()
|
||||
e, err := kv.Get(gctx, key)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision(), At: e.Created().UTC()}, nil
|
||||
}
|
||||
}
|
||||
sort.Strings(keys)
|
||||
page := StatePage{Entries: []StateEntry{}}
|
||||
started := PageClock()
|
||||
size := 0
|
||||
for _, k := range keys {
|
||||
if after != "" && k <= after {
|
||||
continue
|
||||
}
|
||||
if len(page.Entries) > 0 && PageClock().Sub(started) >= budget {
|
||||
page.Next = page.Entries[len(page.Entries)-1].Key
|
||||
break
|
||||
}
|
||||
e, err := get(k)
|
||||
if err != nil {
|
||||
return StatePage{}, err
|
||||
}
|
||||
if e == nil {
|
||||
continue
|
||||
}
|
||||
if size > 0 && size+len(k)+len(e.Value) > maxBytes {
|
||||
page.Next = page.Entries[len(page.Entries)-1].Key
|
||||
break
|
||||
}
|
||||
page.Entries = append(page.Entries, *e)
|
||||
size += len(k) + len(e.Value)
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
|
||||
// StateKept is one value a bucket still keeps for a key: a put, or a delete (a purge reads as one).
|
||||
@@ -148,6 +280,9 @@ func (c *Conn) StateHistory(module, name, key string) ([]StateKept, error) {
|
||||
if err := checkKey(key); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.PerNode {
|
||||
return nil, perNodeRefusal(module, name, "history")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := c.bucket(ctx, s)
|
||||
@@ -173,6 +308,54 @@ func (c *Conn) StateHistory(module, name, key string) ([]StateKept, error) {
|
||||
return kept, nil
|
||||
}
|
||||
|
||||
// StateInfo is how full a state's bucket is against its caps (tools-are-how-a-module-is-spoken-to): a bundle's tools
|
||||
// say fill against cap from it.
|
||||
type StateInfo struct {
|
||||
State string `json:"state"`
|
||||
Bucket string `json:"bucket"`
|
||||
// Bytes is what the bucket holds now, every kept value and marker counted; Keys how many keys it holds, a key
|
||||
// whose last value is a delete or expiry marker counted until the marker goes.
|
||||
Bytes uint64 `json:"bytes"`
|
||||
Keys uint64 `json:"keys"`
|
||||
// MaxBytes and MaxValueBytes are the bucket's caps; -1 is none.
|
||||
MaxBytes int64 `json:"max_bytes"`
|
||||
MaxValueBytes int32 `json:"max_value_bytes"`
|
||||
// History is how many values a key keeps; TTLSeconds the bucket's age for a value (0 none); KeyTTL whether a key may
|
||||
// carry its own time to live (issue 1952).
|
||||
History int64 `json:"history"`
|
||||
TTLSeconds int64 `json:"ttl_seconds"`
|
||||
KeyTTL bool `json:"key_ttl"`
|
||||
}
|
||||
|
||||
// StateInfo reads a state's bucket's fill and caps. It is a read like StateGet: served to the owner and every reader.
|
||||
func (c *Conn) StateInfo(module, name string) (*StateInfo, error) {
|
||||
s, err := c.issuedState(module, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.PerNode {
|
||||
return nil, perNodeRefusal(module, name, "fill")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := c.bucket(ctx, s)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
status, err := kv.Status(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
b, ok := status.(*jetstream.KeyValueBucketStatus)
|
||||
if !ok || b.StreamInfo() == nil {
|
||||
return nil, fmt.Errorf("the bus answered no stream behind %s's state %s", module, name)
|
||||
}
|
||||
info := b.StreamInfo()
|
||||
return &StateInfo{State: name, Bucket: status.Bucket(), Bytes: info.State.Bytes, Keys: uint64(info.State.NumSubjects),
|
||||
MaxBytes: info.Config.MaxBytes, MaxValueBytes: info.Config.MaxMsgSize, History: status.History(),
|
||||
TTLSeconds: int64(status.TTL() / time.Second), KeyTTL: info.Config.AllowMsgTTL && status.LimitMarkerTTL() > 0}, nil
|
||||
}
|
||||
|
||||
// StatePut writes one key, as the module, where the module keeps the state. It answers the revision.
|
||||
func (c *Conn) StatePut(module, name, key string, value json.RawMessage) (uint64, error) {
|
||||
s, err := c.writable(module, name)
|
||||
@@ -199,6 +382,61 @@ func (c *Conn) StatePut(module, name, key string, value json.RawMessage) (uint64
|
||||
return kv.Put(ctx, key, value)
|
||||
}
|
||||
|
||||
// MinKeyTTL is the shortest time to live a key may be given: the bus counts a key's age in whole seconds (mesh-sdk
|
||||
// process.MinTTL).
|
||||
const MinKeyTTL = time.Second
|
||||
|
||||
// StateCreateTTL writes one key only when it has no value, with its own time to live (novox/hq ADR 0564 clauses 4 and
|
||||
// 7, issue 1952; mesh-sdk process.State.CreateTTL): the bus removes it when ttl passes and every watch hears the
|
||||
// removal as an expiry, so a bundle's wake needs no timer. A key that has a value is refused as ErrStateChanged. The
|
||||
// bucket must take keys with their own time to live (a state declared `key-ttl`); one that does not is refused, naming
|
||||
// it, before anything is written.
|
||||
func (c *Conn) StateCreateTTL(module, name, key string, value json.RawMessage, ttl time.Duration) (uint64, error) {
|
||||
if ttl < MinKeyTTL {
|
||||
return 0, fmt.Errorf("%s's state %s, key %s: a time to live of %s is shorter than the bus keeps (%s)",
|
||||
module, name, key, ttl, MinKeyTTL)
|
||||
}
|
||||
return c.stateWrite(module, name, key, value, func(ctx context.Context, kv jetstream.KeyValue) (uint64, error) {
|
||||
if err := takesKeyTTL(ctx, kv, module, name); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return kv.Create(ctx, key, value, jetstream.KeyTTL(ttl))
|
||||
})
|
||||
}
|
||||
|
||||
// StateRenewTTL writes one key whole with a fresh time to live, whatever it held (mesh-sdk process.State.RenewTTL): a
|
||||
// time to live is set only when a key is created, so a live key is deleted first, and its watches hear the delete and
|
||||
// then the new value.
|
||||
func (c *Conn) StateRenewTTL(module, name, key string, value json.RawMessage, ttl time.Duration) (uint64, error) {
|
||||
rev, err := c.StateCreateTTL(module, name, key, value, ttl)
|
||||
if !errors.Is(err, ErrStateChanged) {
|
||||
return rev, err
|
||||
}
|
||||
if err := c.StateDelete(module, name, key); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return c.StateCreateTTL(module, name, key, value, ttl)
|
||||
}
|
||||
|
||||
// takesKeyTTL refuses a bucket that does not allow a time to live per key, or leaves no marker when one ends: there the
|
||||
// server would refuse the write, or the key would vanish and no watch would hear it.
|
||||
func takesKeyTTL(ctx context.Context, kv jetstream.KeyValue, module, name string) error {
|
||||
status, err := kv.Status(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
allows := status.LimitMarkerTTL() > 0
|
||||
if b, ok := status.(*jetstream.KeyValueBucketStatus); ok && b.StreamInfo() != nil {
|
||||
allows = allows && b.StreamInfo().Config.AllowMsgTTL
|
||||
}
|
||||
if !allows {
|
||||
return fmt.Errorf("%s's state %s does not take keys with their own time to live: its bucket %s allows none, or "+
|
||||
"leaves no marker a watch hears when one ends; %s declares the state `key-ttl` for it (novox/hq ADR 0564, "+
|
||||
"issue 1952)", module, name, status.Bucket(), module)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// StateDelete removes one key, as the module, where the module keeps the state. A key that was not
|
||||
// there is not an error: what is asked for is that it is gone.
|
||||
func (c *Conn) StateDelete(module, name, key string) error {
|
||||
@@ -224,12 +462,20 @@ func (c *Conn) StateKeys(module, name string) ([]string, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.PerNode {
|
||||
return c.perNodeKeys(s)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := c.bucket(ctx, s)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return keysOf(ctx, kv)
|
||||
}
|
||||
|
||||
// keysOf is every key with a current value in one bucket, sorted.
|
||||
func keysOf(ctx context.Context, kv jetstream.KeyValue) ([]string, error) {
|
||||
lister, err := kv.ListKeys(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -291,12 +537,20 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
|
||||
}
|
||||
filter = strings.Join(parts, ".")
|
||||
}
|
||||
if s.PerNode {
|
||||
return c.perNodeWatch(module, s, pattern, deliver)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
kv, err := c.bucket(ctx, s)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.watchBucket(module, name, kv, filter, deliver)
|
||||
}
|
||||
|
||||
// watchBucket is StateWatch over one bucket: the current values, then every change.
|
||||
func (c *Conn) watchBucket(module, name string, kv jetstream.KeyValue, filter string, deliver func(StateChange) error) (stop func(), err error) {
|
||||
watching, stopWatching := context.WithCancel(context.Background())
|
||||
w, err := kv.Watch(watching, filter)
|
||||
if err != nil {
|
||||
@@ -322,6 +576,12 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
|
||||
go func() {
|
||||
initial := true
|
||||
for e := range w.Updates() {
|
||||
// A watch given up, or stopped, hands over nothing more, though values stay buffered in the
|
||||
// watcher: they would reach a watch id the bundle no longer holds (issue 774: 1257 orphan
|
||||
// changes after 51 failed watches).
|
||||
if watching.Err() != nil {
|
||||
break
|
||||
}
|
||||
if e == nil {
|
||||
// The end of what was there when the watch began.
|
||||
if initial {
|
||||
@@ -330,6 +590,11 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
|
||||
}
|
||||
continue
|
||||
}
|
||||
if initial {
|
||||
// Arrival from the bus is progress before the bundle is asked: the bound measures the current
|
||||
// values coming in, and restarts when the bundle has taken one.
|
||||
progress()
|
||||
}
|
||||
change := StateChange{State: name, Key: e.Key(), Revision: e.Revision(), Current: initial}
|
||||
switch e.Operation() {
|
||||
case jetstream.KeyValuePut:
|
||||
@@ -343,6 +608,15 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
|
||||
continue
|
||||
}
|
||||
change.Op = "delete"
|
||||
if e.Operation() == jetstream.KeyValuePurge {
|
||||
// The marker the server leaves when a key's time to live ends (Nats-Marker-Reason MaxAge,
|
||||
// read by the client as a purge): nothing else purges a module's state, since neither the
|
||||
// runtime nor mesh-sdk's State offers a purge (issue 1952).
|
||||
change.Op = "expire"
|
||||
}
|
||||
}
|
||||
if watching.Err() != nil {
|
||||
break
|
||||
}
|
||||
if err := deliver(change); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
|
||||
@@ -393,7 +667,7 @@ var credentialEndings = []string{"password", "passwd", "secret", "token", "crede
|
||||
|
||||
// credentialField is the first field anywhere in a JSON value whose name says it is a credential, or "". The one shape a
|
||||
// credential-shaped name may hold is a provider reference, an object of exactly `node` and `module`, both strings: a
|
||||
// placement's binding of the provision "secret" to the vault (issue 1126) names who provides it, never its value. Any
|
||||
// the assignment's binding of the provision "secret" to the vault (issue 1126) names who provides it, never its value. Any
|
||||
// other shape under such a name is refused.
|
||||
func credentialField(value json.RawMessage) string {
|
||||
var v any
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
//go:build integration
|
||||
|
||||
package bus_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// **A state its owner keeps one bucket per node is served to a bundle as one state** (novox/hq ADR 0593 §6: each node's
|
||||
// engine writes its own `composed-<node>`, and the provisioner reads them all). Get goes to the bucket the key's first
|
||||
// part names, Keys and Watch over every bucket of the prefix, a bucket created during the watch included, and a bucket
|
||||
// that only starts like the prefix is never read.
|
||||
func TestAPerNodeStateIsServedAcrossEveryNodesBucket(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
a, b := mesh.Bucket(t, "composed-a"), mesh.Bucket(t, "composed-b")
|
||||
other := mesh.Bucket(t, "composedx-z")
|
||||
for kv, key := range map[interface {
|
||||
Put(string, []byte) (uint64, error)
|
||||
}]string{a: "a.m1.f1", b: "b.m2.f2", other: "z.m3.f3"} {
|
||||
if _, err := kv.Put(key, []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
prov := mt.MembershipOf("mesh-provisioner", "anchor", false, nil)
|
||||
prov.State = []bus.StateIssued{{Name: "mesh-host.composed", Bucket: "composed-", PerNode: true}}
|
||||
mesh.Issue(t, prov)
|
||||
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn.Logf = func(string, ...any) {}
|
||||
t.Cleanup(conn.Close)
|
||||
conn.Follow("mesh-provisioner")
|
||||
mt.Until(t, func() error {
|
||||
if conn.Membership("mesh-provisioner") == nil {
|
||||
return errors.New("not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "b.m2.f2"); err != nil || e == nil || e.Key != "b.m2.f2" || e.At.IsZero() || time.Since(e.At) > time.Minute {
|
||||
t.Fatalf("get from the second node's bucket: %+v, %v", e, err)
|
||||
}
|
||||
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "q.m.f"); err != nil || e != nil {
|
||||
t.Fatalf("a key of a node with no bucket should read as none: %+v, %v", e, err)
|
||||
}
|
||||
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "z.m3.f3"); err != nil || e != nil {
|
||||
t.Fatalf("the look-alike bucket was read: %+v, %v", e, err)
|
||||
}
|
||||
if _, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "nonode"); err == nil {
|
||||
t.Fatal("a key naming no node should be refused")
|
||||
}
|
||||
if keys, err := conn.StateKeys("mesh-provisioner", "mesh-host.composed"); err != nil || !reflect.DeepEqual(keys, []string{"a.m1.f1", "b.m2.f2"}) {
|
||||
t.Fatalf("keys across the buckets: %v, %v", keys, err)
|
||||
}
|
||||
if _, err := conn.StatePut("mesh-provisioner", "mesh-host.composed", "a.m1.f9", []byte(`{}`)); err == nil {
|
||||
t.Fatal("a per-node state is read only")
|
||||
}
|
||||
|
||||
var mu sync.Mutex
|
||||
var current, live []string
|
||||
stop, err := conn.StateWatch("mesh-provisioner", "mesh-host.composed", "", func(ch bus.StateChange) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if ch.Current {
|
||||
current = append(current, ch.Key)
|
||||
} else {
|
||||
live = append(live, ch.Key)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(stop)
|
||||
mu.Lock()
|
||||
sort.Strings(current)
|
||||
if !reflect.DeepEqual(current, []string{"a.m1.f1", "b.m2.f2"}) {
|
||||
t.Fatalf("current values: %v", current)
|
||||
}
|
||||
mu.Unlock()
|
||||
|
||||
// A third node's bucket, created during the watch, and a change in a first one.
|
||||
c := mesh.Bucket(t, "composed-c")
|
||||
if _, err := c.Put("c.m4.f4", []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := a.Put("a.m1.f5", []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mt.Until(t, func() error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
got := strings.Join(append(append([]string(nil), current...), live...), ",")
|
||||
for _, want := range []string{"a.m1.f5", "c.m4.f4"} {
|
||||
if !strings.Contains(got, want) {
|
||||
return fmt.Errorf("%s not heard yet: current %v, changes %v", want, current, live)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
@@ -15,7 +15,7 @@ func TestACredentialNamedFieldIsFoundAnywhereInAValue(t *testing.T) {
|
||||
`{"clientSecret":"s"}`: "clientSecret",
|
||||
`{"maxTokens":4096,"name":"a","generation":3}`: "",
|
||||
`"just a string"`: "",
|
||||
// Issue 1126: a placement binds the provision "secret" to the vault; only an exact provider reference passes.
|
||||
// Issue 1126: an assignment binds the provision "secret" to the vault; only an exact provider reference passes.
|
||||
`{"bindings":{"secret":{"node":"anchor","module":"mesh-vault"}}}`: "",
|
||||
`{"secret":{"value":"s"}}`: "secret",
|
||||
`{"secret":{"node":"anchor","module":"mesh-vault","value":"s"}}`: "secret",
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -32,7 +33,7 @@ func TestAWatchHoldsWhileItsCurrentValuesKeepArriving(t *testing.T) {
|
||||
for i := 0; i < older; i++ {
|
||||
keys = append(keys, fmt.Sprintf("novox/mesh-catalog.%d.%040x.status=mesh/check", 400+i, i))
|
||||
}
|
||||
for _, member := range []string{"novox/mesh-sdk.76.34f210e5", "novox/mesh-controller.453.c06f75d0", "novox/mesh-delivery.33.28ee22ef"} {
|
||||
for _, member := range []string{"novox/mesh-sdk.76.34f210e5", "novox/mesh-module-assigner.453.c06f75d0", "novox/mesh-delivery.33.28ee22ef"} {
|
||||
keys = append(keys, member+strings.Repeat("0", 32)+".status=mesh/check")
|
||||
}
|
||||
for _, k := range keys {
|
||||
@@ -86,6 +87,74 @@ func TestAWatchHoldsWhileItsCurrentValuesKeepArriving(t *testing.T) {
|
||||
}
|
||||
return nil
|
||||
})
|
||||
// **A watch given up, or stopped, hands over nothing more** (issue 774). The runtime's watch stopped its watcher
|
||||
// when the current values did not arrive within the bound, but the watcher still held values, and its goroutine
|
||||
// went on handing them over: to a watch id the bundle no longer held (on the provisioner, 51 failed watches were
|
||||
// followed by 1257 orphan changes). Each row blocks the bundle on its first value, gives the watch up, lets the
|
||||
// bundle go and counts what it is still handed.
|
||||
t.Run("AWatchGivenUpHandsOverNothingMore", func(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
refused bool // the bound gives it up, or the caller stops it
|
||||
}{
|
||||
{"refused at the bound", true},
|
||||
{"stopped by its caller", false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
was := bus.StateTimeout
|
||||
bus.StateTimeout = 300 * time.Millisecond
|
||||
t.Cleanup(func() { bus.StateTimeout = was })
|
||||
conn, kv := owedWatcher(t)
|
||||
put := func(i int) {
|
||||
if _, err := kv.Put(fmt.Sprintf("novox/hq.%d.%040x.status=mesh/check", i+1, i), []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if tc.refused {
|
||||
for i := 0; i < 4; i++ {
|
||||
put(i)
|
||||
}
|
||||
}
|
||||
var calls atomic.Int32
|
||||
first := make(chan struct{}, 1)
|
||||
release := make(chan struct{})
|
||||
deliver := func(ch bus.StateChange) error {
|
||||
if calls.Add(1) == 1 {
|
||||
first <- struct{}{}
|
||||
<-release
|
||||
}
|
||||
return nil
|
||||
}
|
||||
var stop func()
|
||||
if tc.refused {
|
||||
if _, err := conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", deliver); err == nil {
|
||||
t.Fatal("the watch whose bundle took nothing was not refused")
|
||||
}
|
||||
} else {
|
||||
var err error
|
||||
if stop, err = conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", deliver); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 4; i++ {
|
||||
put(i)
|
||||
}
|
||||
select {
|
||||
case <-first:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("the bundle was handed no change")
|
||||
}
|
||||
stop()
|
||||
}
|
||||
// Let the watcher's server side deliver what it held, then free the bundle.
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
close(release)
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
if n := calls.Load(); n != 1 {
|
||||
t.Fatalf("the bundle was handed %d changes of a watch given up, want 1 (the one in hand)", n)
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A watch whose bundle takes nothing for longer than the bound is still refused, so a bundle that hangs
|
||||
@@ -116,6 +185,50 @@ func TestAWatchIsRefusedWhenNoCurrentValueIsTakenWithinTheBound(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// **A membership the grants part wrote in its old wrapper is served whole** (issue 774). Until 2026-10-30 the part wrote
|
||||
// its pass record, the membership inside, into its state memberships; the tool runner read the value as a bare
|
||||
// membership, and served no tool and no state. This is what the live mesh held: read through mesh-sdk's codec, its tools
|
||||
// are answered where the membership says and its state is reached. A held-back one's wrapper is refused, and the module has
|
||||
// no membership then (774/S32/AC2: no stream is read in its place).
|
||||
func TestAMembershipTheGrantsPartWroteWrappedIsServed(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
mesh.Bucket(t, "gitea_repos")
|
||||
gitea := mt.MembershipOf("gitea", "anchor", false, nil)
|
||||
gitea.State = []bus.StateIssued{{Name: "repos", Bucket: "gitea_repos", Writes: true}}
|
||||
mesh.IssueInGrantsWrapped(t, gitea, "published")
|
||||
// The old wrapper of a membership the part held back carries its unpublished composition: refused, never served.
|
||||
heldBack := mt.MembershipOf("pages", "anchor", true, nil)
|
||||
mesh.IssueInGrantsWrapped(t, heldBack, "not published, a shadow difference: publishing would lose the last holder of x")
|
||||
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var logged []string
|
||||
conn.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
|
||||
t.Cleanup(conn.Close)
|
||||
conn.Follow("gitea")
|
||||
mt.Until(t, func() error {
|
||||
if m := conn.Membership("gitea"); m == nil || len(m.Serves) == 0 {
|
||||
return fmt.Errorf("gitea's membership is %+v, want the one inside the wrapper", m)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if got, want := fmt.Sprint(conn.ServedOn("gitea", "gitea_list")), "[{mesh.mod.gitea.tool.gitea_list.anchor }]"; got != want {
|
||||
t.Errorf("gitea_list is served on %s, want %s", got, want)
|
||||
}
|
||||
if _, err := conn.StatePut("gitea", "repos", "novox.hq", []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatalf("the state the membership issues is refused: %v", err)
|
||||
}
|
||||
if e, err := conn.StateGet("gitea", "repos", "novox.hq"); err != nil || e == nil || string(e.Value) != `{"v":1}` || e.At.IsZero() || time.Since(e.At) > time.Minute {
|
||||
t.Errorf("the state reads back %+v, %v", e, err)
|
||||
}
|
||||
conn.Follow("pages")
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
if m := conn.Membership("pages"); m != nil || !strings.Contains(strings.Join(logged, "\n"), "no membership issued for pages") {
|
||||
t.Errorf("a held-back membership's wrapper was served or not said: %+v, %q", m, logged)
|
||||
}
|
||||
}
|
||||
|
||||
// owedWatcher is the gitea module's view of the checker's owed: the bucket, and a runtime connection
|
||||
// that serves gitea with the owed issued for reading.
|
||||
func owedWatcher(t *testing.T) (*bus.Conn, interface {
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
|
||||
// A module's own stream, as the runtime publishes to it and hears it (novox/hq ADR 0363 §8, issue 725).
|
||||
//
|
||||
// A module may declare a stream of its own; the controller creates it and says, in the module's
|
||||
// A module may declare a stream of its own; the mesh creates it and says, in the module's
|
||||
// membership, which of its events go there and on which subject (`streams`), and, in the membership of
|
||||
// each module granted to hear it, which durable consumer on it to bind (`hears`). The runtime publishes and
|
||||
// binds exactly as the membership says. A membership naming neither is what every membership was before:
|
||||
@@ -30,7 +30,7 @@ type StreamIssued struct {
|
||||
}
|
||||
|
||||
// HeardIssued is one module stream an assignment hears: the stream and the durable consumer on it the
|
||||
// runtime binds. The consumer is the controller's to make, to filter and to delete.
|
||||
// runtime binds. The consumer is the mesh's to make, to filter and to delete.
|
||||
type HeardIssued struct {
|
||||
Stream string `json:"stream"`
|
||||
Consumer string `json:"consumer"`
|
||||
@@ -73,7 +73,7 @@ func eventMatches(pattern, event string) bool {
|
||||
}
|
||||
|
||||
// RebindEvery is how often a consumer the membership names and the bus did not let the runtime bind is
|
||||
// tried again: the controller makes the consumer and sends the user list carrying its grant in the
|
||||
// tried again: the mesh makes the consumer and sends the user list carrying its grant in the
|
||||
// composition that issues the membership, so one may arrive before the other (ADR 0363 §6).
|
||||
var RebindEvery = 30 * time.Second
|
||||
|
||||
@@ -294,7 +294,7 @@ func (h *hearing) reconcile() {
|
||||
}
|
||||
}
|
||||
// **A loop the bus ended is no longer reading** (nats.go ends a pull loop on its own when its consumer is
|
||||
// deleted): the controller deletes and makes again under the same name a restricted consumer it cannot
|
||||
// deleted): the mesh deletes and makes again under the same name a restricted consumer it cannot
|
||||
// narrow (ADR 0363 §6), and the membership, unchanged, says nothing — so it is bound again here.
|
||||
// **A loop the bus did not end is asked about**: nats.go learns of a deleted consumer only from the answer
|
||||
// to a pull it has pending, and a consumer deleted and made again between two pulls, or while none is
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
package bus
|
||||
|
||||
import "testing"
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// novox/hq ADR 0363 §2, §3: an event goes to the stream whose pattern carries it, as the bus matches subjects.
|
||||
func TestAnEventFallsUnderAStreamsPatternAsTheBusMatchesIt(t *testing.T) {
|
||||
@@ -22,6 +27,24 @@ func TestAnEventFallsUnderAStreamsPatternAsTheBusMatchesIt(t *testing.T) {
|
||||
t.Errorf("%q under %q: %v, want %v", c.event, c.pattern, got, c.want)
|
||||
}
|
||||
}
|
||||
// Design 47 G §347, §59, §462: a sent object is read by the reference that carried it, from a sender's sent store only; a
|
||||
// state's objects, and a reference that is none are refused before the bus is asked.
|
||||
t.Run("SentGetRefusesWhatIsNoSentObjectsReference", func(t *testing.T) {
|
||||
digest := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
req buscalls.ObjectsSentGetRequest
|
||||
want string
|
||||
}{
|
||||
{"no digest", buscalls.ObjectsSentGetRequest{Store: "m_sent-objects", Object: "o", Size: 3}, "named by the reference"},
|
||||
{"a state's objects", buscalls.ObjectsSentGetRequest{Store: "m_asks-objects", Object: "o", Size: 3, Digest: digest}, "no sent store"},
|
||||
} {
|
||||
_, err := (&Conn{}).SentGet(c.req)
|
||||
if err == nil || !strings.Contains(err.Error(), c.want) {
|
||||
t.Errorf("%s: got %v, want an error containing %q", c.name, err, c.want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A membership naming no stream publishes every event on EVENTS, exactly as before (issue 725).
|
||||
|
||||
@@ -160,6 +160,68 @@ func TestAModuleHearsEveryStreamItsMembershipNames(t *testing.T) {
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
// The membership as the grants part keeps it (ADR 0571 clause 7, 10): read first and applied live, and nothing
|
||||
// published on the controller's subject is heard.
|
||||
granted := reader
|
||||
granted.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "from-grants"}}
|
||||
mesh.Issue(t, granted)
|
||||
again, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again.Logf = func(string, ...any) {}
|
||||
t.Cleanup(again.Close)
|
||||
again.Follow("reader")
|
||||
consumerOf := func() string {
|
||||
if m := again.Membership("reader"); m != nil && len(m.Hears) == 1 {
|
||||
return m.Hears[0].Consumer
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if got := consumerOf(); got != "from-grants" {
|
||||
t.Fatalf("the membership read at the follow names %q, want the grants part's", got)
|
||||
}
|
||||
granted.Hears[0].Consumer = "from-grants-2"
|
||||
mesh.Issue(t, granted)
|
||||
mt.Until(t, func() error {
|
||||
if got := consumerOf(); got != "from-grants-2" {
|
||||
return fmt.Errorf("the membership names %q, want the grants part's change", got)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
// 774/S32/AC1: a copy said on the controller's subject (mesh.assignment) changes nothing; there is no stream to read it from.
|
||||
streamed := reader
|
||||
streamed.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "from-subject"}}
|
||||
mesh.Publish(t, "mesh.assignment.anchor.reader", streamed)
|
||||
again.Flush()
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
if got := consumerOf(); got != "from-grants-2" {
|
||||
t.Fatalf("a membership said on the controller's subject was applied: the membership names %q", got)
|
||||
}
|
||||
// Review g1 round 1: memberships that cannot be read at the follow (a bus not ready after its boot) are read again until
|
||||
// they can be; the follow, which runs once per module, does not give the module up for good.
|
||||
was := bus.GrantsRetry
|
||||
bus.GrantsRetry = 50 * time.Millisecond
|
||||
t.Cleanup(func() { bus.GrantsRetry = was })
|
||||
mesh.DropGrants(t)
|
||||
late, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
late.Logf = func(string, ...any) {}
|
||||
t.Cleanup(late.Close)
|
||||
late.Follow("reader")
|
||||
if m := late.Membership("reader"); m != nil {
|
||||
t.Fatalf("a membership was served from a state not there: %+v", m)
|
||||
}
|
||||
mesh.Issue(t, granted)
|
||||
mt.Until(t, func() error {
|
||||
if m := late.Membership("reader"); m == nil || len(m.Hears) != 1 || m.Hears[0].Consumer != "from-grants-2" {
|
||||
return fmt.Errorf("the membership is %+v, want the one written once the state is there", m)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// novox/hq ADR 0363 §6 (issue 725): a consumer the membership names before the controller made it is bound once
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscompose"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// SubscribeNew hears one event, named `<emitter>.<event>`, from the next message on, on an ordered consumer of its own on each
|
||||
// stream that carries it, and hands every message to deliver as the envelope a module sees. The subjects the name lands on are
|
||||
// mesh-sdk's one rule (buscompose.EventSubjects, design 47 G §559, A-153), so a name with a wildcard is refused, naming the rule;
|
||||
// a name no stream on the bus carries is an error, never a subscription that hears nothing. stop ends every consumer.
|
||||
func (c *Conn) SubscribeNew(name string, deliver func(json.RawMessage) error) (func(), error) {
|
||||
subjects, err := buscompose.EventSubjects(name, "")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
js, err := c.JetStream()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
// The streams that carry each subject, found by the bus: a namespace no stream carries hears nothing and is skipped.
|
||||
byStream := map[string][]string{}
|
||||
for _, subject := range subjects {
|
||||
if strings.HasSuffix(subject, ".>") {
|
||||
continue
|
||||
}
|
||||
stream, err := js.StreamNameBySubject(ctx, subject)
|
||||
if err != nil || stream == "" {
|
||||
continue
|
||||
}
|
||||
byStream[stream] = append(byStream[stream], subject, subject+".>")
|
||||
}
|
||||
if len(byStream) == 0 {
|
||||
return nil, fmt.Errorf("no stream on the bus carries %s (%s): nothing would be heard", name, strings.Join(subjects, ", "))
|
||||
}
|
||||
var stops []func()
|
||||
stopAll := func() {
|
||||
for _, s := range stops {
|
||||
s()
|
||||
}
|
||||
}
|
||||
for _, stream := range slices.Sorted(mapKeys(byStream)) {
|
||||
cons, err := js.OrderedConsumer(ctx, stream, jetstream.OrderedConsumerConfig{FilterSubjects: byStream[stream],
|
||||
DeliverPolicy: jetstream.DeliverNewPolicy})
|
||||
if err != nil {
|
||||
stopAll()
|
||||
return nil, fmt.Errorf("an ordered consumer of %s on %s: %w", name, stream, err)
|
||||
}
|
||||
cc, err := cons.Consume(func(m jetstream.Msg) {
|
||||
env, ok := envelopeOf(m.Subject(), m.Headers(), m.Data())
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
raw, err := json.Marshal(env)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := deliver(raw); err != nil && c.Logf != nil {
|
||||
c.Logf("[mesh-tools] a subscription for new %s messages did not take one: %v", name, err)
|
||||
}
|
||||
})
|
||||
if err != nil {
|
||||
stopAll()
|
||||
return nil, fmt.Errorf("consuming %s on %s: %w", name, stream, err)
|
||||
}
|
||||
stops = append(stops, cc.Stop)
|
||||
}
|
||||
return stopAll, nil
|
||||
}
|
||||
|
||||
func mapKeys(m map[string][]string) func(func(string) bool) {
|
||||
return func(yield func(string) bool) {
|
||||
for k := range m {
|
||||
if !yield(k) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -16,38 +17,42 @@ import (
|
||||
// subscribes the subjects when called and unsubscribes them before it answers, so an event published
|
||||
// between two calls is not heard — a session reads the state from its owner after every wake (§4).
|
||||
|
||||
// WaitIssued is one event family the mesh issued this account to wait on (ADR 0367 §2): the name a session
|
||||
// asks for, and the subject it lands on. Only the tool runner's membership carries any.
|
||||
type WaitIssued struct {
|
||||
Event string `json:"event"`
|
||||
Subject string `json:"subject"`
|
||||
}
|
||||
|
||||
// Heard is one event a wait heard: the subject it arrived on, its body and when it arrived.
|
||||
// Heard is one event a wait heard: the subject it arrived on, its body, when it arrived, and when it was published, the
|
||||
// event's own header `x-time` as its emitter wrote it, empty for an event that carries none (design 34 every-event §12).
|
||||
type Heard struct {
|
||||
Subject string
|
||||
Body json.RawMessage
|
||||
At time.Time
|
||||
Subject string
|
||||
Body json.RawMessage
|
||||
At time.Time
|
||||
Published string
|
||||
}
|
||||
|
||||
// HearFirst subscribes the subjects for this call alone and answers the first message whose body match
|
||||
// accepts (every one, when match is nil), or ok false once `within` has passed. Every subscription is
|
||||
// closed before it returns. A subscription the bus refuses is said as an error naming the subject, at once;
|
||||
// a caller gone — ctx done — ends the wait at once with ctx's error, so nothing waits on for nobody.
|
||||
func (c *Conn) HearFirst(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (heard Heard, ok bool, err error) {
|
||||
// closed before it returns. A subscription the bus refuses is noted and the others go on waiting; refused lists
|
||||
// those noted by the time it answers (design 34 every-event §16). When the bus refused every subscription of the
|
||||
// call, it still waits out `within` and then answers the refusal, so a caller looping on it asks once a timeout and
|
||||
// never at once again. A caller gone — ctx done — ends the wait at once with ctx's error, so nothing waits on for
|
||||
// nobody.
|
||||
func (c *Conn) HearFirst(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (heard Heard, ok bool, refused []string, err error) {
|
||||
if len(subjects) == 0 {
|
||||
return Heard{}, false, fmt.Errorf("a wait names at least one subject")
|
||||
return Heard{}, false, nil, fmt.Errorf("a wait names at least one subject")
|
||||
}
|
||||
var unique []string
|
||||
for _, subject := range subjects {
|
||||
if !slices.Contains(unique, subject) {
|
||||
unique = append(unique, subject)
|
||||
}
|
||||
}
|
||||
got := make(chan Heard, 1)
|
||||
refused := make(chan string, len(subjects))
|
||||
refusals := make(chan string, len(unique))
|
||||
var subs []*nats.Subscription
|
||||
defer func() {
|
||||
for _, s := range subs {
|
||||
_ = s.Unsubscribe()
|
||||
}
|
||||
c.mu.Lock()
|
||||
for _, subject := range subjects {
|
||||
delete(c.waits[subject], refused)
|
||||
for _, subject := range unique {
|
||||
delete(c.waits[subject], refusals)
|
||||
if len(c.waits[subject]) == 0 {
|
||||
delete(c.waits, subject)
|
||||
}
|
||||
@@ -58,14 +63,14 @@ func (c *Conn) HearFirst(ctx context.Context, subjects []string, within time.Dur
|
||||
if c.waits == nil {
|
||||
c.waits = map[string]map[chan string]bool{}
|
||||
}
|
||||
for _, subject := range subjects {
|
||||
for _, subject := range unique {
|
||||
if c.waits[subject] == nil {
|
||||
c.waits[subject] = map[chan string]bool{}
|
||||
}
|
||||
c.waits[subject][refused] = true
|
||||
c.waits[subject][refusals] = true
|
||||
}
|
||||
c.mu.Unlock()
|
||||
for _, subject := range subjects {
|
||||
for _, subject := range unique {
|
||||
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
|
||||
body := json.RawMessage(msg.Data)
|
||||
if !json.Valid(body) {
|
||||
@@ -76,31 +81,39 @@ func (c *Conn) HearFirst(ctx context.Context, subjects []string, within time.Dur
|
||||
return
|
||||
}
|
||||
select {
|
||||
case got <- Heard{Subject: msg.Subject, Body: body, At: time.Now().UTC()}:
|
||||
case got <- Heard{Subject: msg.Subject, Body: body, At: time.Now().UTC(), Published: msg.Header.Get("x-time")}:
|
||||
default:
|
||||
}
|
||||
})
|
||||
if err != nil {
|
||||
return Heard{}, false, fmt.Errorf("could not wait on %s: %w", subject, err)
|
||||
return Heard{}, false, nil, fmt.Errorf("could not wait on %s: %w", subject, err)
|
||||
}
|
||||
subs = append(subs, sub)
|
||||
}
|
||||
// The bus has every subscription — or has said which it refused — once this returns.
|
||||
if err := c.nc.FlushTimeout(5 * time.Second); err != nil {
|
||||
return Heard{}, false, fmt.Errorf("the bus did not take the wait: %w", err)
|
||||
return Heard{}, false, nil, fmt.Errorf("the bus did not take the wait: %w", err)
|
||||
}
|
||||
timer := time.NewTimer(within)
|
||||
defer timer.Stop()
|
||||
select {
|
||||
case h := <-got:
|
||||
return h, true, nil
|
||||
case subject := <-refused:
|
||||
return Heard{}, false, fmt.Errorf("the bus refused this account a subscription on %s: it is not in the "+
|
||||
"account's wait grant (novox/hq ADR 0367 §2)", subject)
|
||||
case <-timer.C:
|
||||
return Heard{}, false, nil
|
||||
case <-ctx.Done():
|
||||
return Heard{}, false, ctx.Err()
|
||||
for {
|
||||
select {
|
||||
case h := <-got:
|
||||
return h, true, refused, nil
|
||||
case subject := <-refusals:
|
||||
if !slices.Contains(refused, subject) {
|
||||
refused = append(refused, subject)
|
||||
}
|
||||
case <-timer.C:
|
||||
if len(refused) == len(unique) {
|
||||
return Heard{}, false, refused, fmt.Errorf("the bus refused this account a subscription on %s: the "+
|
||||
"grants part grants this machine's tool runner every event subject (novox/hq ADR 0670 §1), so its "+
|
||||
"grant is not composed yet; read the state from its owner meanwhile", strings.Join(refused, ", "))
|
||||
}
|
||||
return Heard{}, false, refused, nil
|
||||
case <-ctx.Done():
|
||||
return Heard{}, false, refused, ctx.Err()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,22 +2,24 @@ package bus_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// novox/hq ADR 0367 §2, §5: the bus refuses the tool runner's account any subject its wait grant does not
|
||||
// name — another asker's `decided` among them — and a wait on one says the bus refused it, at once.
|
||||
// novox/hq ADR 0670 §1 (design 34 every-event §12, §16): a subscription the bus refuses is noted and the others go on
|
||||
// waiting, a heard event carries its own time; a wait whose every subscription is refused answers only after its
|
||||
// timeout, in words naming the grants part's grant of every event subject, never a wait grant (E-2).
|
||||
func TestTheBusRefusesAWaitOutsideTheGrant(t *testing.T) {
|
||||
granted := "mesh.seat.operator-channel.event.decided.claude-code"
|
||||
other := "mesh.seat.operator-channel.event.decided.shop"
|
||||
s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, NoLog: true, NoSigs: true,
|
||||
Users: []*server.User{{Username: "desk.node-tools", Password: "pw", Permissions: &server.Permissions{
|
||||
Publish: &server.SubjectPermission{Allow: []string{"$JS.API.DIRECT.GET.>", "mesh.>"}},
|
||||
@@ -33,33 +35,65 @@ func TestTheBusRefusesAWaitOutsideTheGrant(t *testing.T) {
|
||||
t.Cleanup(s.Shutdown)
|
||||
c := dial(t, s.ClientURL(), "node-tools", "desk", "desk.node-tools")
|
||||
|
||||
// Every subscription refused: answered after the timeout, never at once, in §16's words.
|
||||
start := time.Now()
|
||||
_, ok, err := c.HearFirst(context.Background(), []string{"mesh.seat.operator-channel.event.decided.shop"}, 5*time.Second, nil)
|
||||
if ok || err == nil || !strings.Contains(err.Error(), "refused") {
|
||||
t.Errorf("a wait on another asker's decided: heard %v, %v", ok, err)
|
||||
_, ok, refused, err := c.HearFirst(context.Background(), []string{other}, 300*time.Millisecond, nil)
|
||||
if took := time.Since(start); took < 300*time.Millisecond || took > 3*time.Second {
|
||||
t.Errorf("a wait refused whole answered after %s, want its timeout of 300ms", took)
|
||||
}
|
||||
if time.Since(start) > 3*time.Second {
|
||||
t.Errorf("the refusal took %s to be said", time.Since(start))
|
||||
if ok || err == nil || !strings.Contains(err.Error(), "the bus refused this account a subscription on "+other) ||
|
||||
!strings.Contains(err.Error(), "every event subject (novox/hq ADR 0670 §1)") ||
|
||||
strings.Contains(err.Error(), "wait grant") || strings.Contains(err.Error(), "0367") ||
|
||||
len(refused) != 1 || refused[0] != other {
|
||||
t.Errorf("a wait on another asker's decided: heard %v, refused %v, %v", ok, refused, err)
|
||||
}
|
||||
|
||||
done := make(chan bus.Heard, 1)
|
||||
go func() {
|
||||
h, ok, err := c.HearFirst(context.Background(), []string{granted}, 5*time.Second, func(body json.RawMessage) bool { return true })
|
||||
if ok && err == nil {
|
||||
done <- h
|
||||
}
|
||||
}()
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
if err := c.Say(granted, []byte(`{"ask":"a1"}`)); err != nil {
|
||||
// One refused beside one granted: the granted one hears, with the event's own time, and the refused one is listed.
|
||||
pub, err := nats.Connect(s.ClientURL(), nats.UserInfo("desk.node-tools", "pw"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
select {
|
||||
case h := <-done:
|
||||
if h.Subject != granted || string(h.Body) != `{"ask":"a1"}` {
|
||||
t.Errorf("heard %+v", h)
|
||||
t.Cleanup(pub.Close)
|
||||
for _, published := range []string{"2026-11-12T03:00:00.5Z", ""} {
|
||||
type result struct {
|
||||
h bus.Heard
|
||||
refused []string
|
||||
err error
|
||||
}
|
||||
done := make(chan result, 1)
|
||||
go func() {
|
||||
h, ok, refused, err := c.HearFirst(context.Background(), []string{other, granted}, 5*time.Second, nil)
|
||||
if !ok && err == nil {
|
||||
err = errors.New("timed out")
|
||||
}
|
||||
done <- result{h, refused, err}
|
||||
}()
|
||||
// The event is said again until the wait, subscribed in its own time, hears it.
|
||||
say := time.NewTicker(20 * time.Millisecond)
|
||||
var r result
|
||||
heard:
|
||||
for {
|
||||
select {
|
||||
case r = <-done:
|
||||
break heard
|
||||
case <-say.C:
|
||||
msg := nats.NewMsg(granted)
|
||||
msg.Data = []byte(`{"ask":"a1"}`)
|
||||
if published != "" {
|
||||
msg.Header.Set("x-time", published)
|
||||
}
|
||||
if err := pub.PublishMsg(msg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
say.Stop()
|
||||
if r.err != nil || r.h.Subject != granted || string(r.h.Body) != `{"ask":"a1"}` || r.h.Published != published {
|
||||
t.Errorf("heard %+v, %v; want published %q", r.h, r.err, published)
|
||||
}
|
||||
if len(r.refused) > 1 || len(r.refused) == 1 && r.refused[0] != other {
|
||||
t.Errorf("refused %v, want at most %s", r.refused, other)
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("a wait on the granted subject heard nothing")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -70,7 +104,7 @@ func TestAWaitEndsWhenItsCallerIsGone(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
time.AfterFunc(200*time.Millisecond, cancel)
|
||||
start := time.Now()
|
||||
_, ok, err := c.HearFirst(ctx, []string{"mesh.mod.mesh-knowledge.event.reviewed"}, time.Minute, nil)
|
||||
_, ok, _, err := c.HearFirst(ctx, []string{"mesh.mod.mesh-knowledge.event.reviewed"}, time.Minute, nil)
|
||||
if ok || !errors.Is(err, context.Canceled) {
|
||||
t.Errorf("a cancelled wait answered %v, %v", ok, err)
|
||||
}
|
||||
|
||||
@@ -17,8 +17,14 @@ package console
|
||||
// session — so a tool that arrived a minute ago is found without the client reconnecting.
|
||||
|
||||
import (
|
||||
assigner "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
|
||||
"git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
|
||||
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
@@ -27,6 +33,7 @@ import (
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/announce"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/firstview"
|
||||
)
|
||||
|
||||
// IndexKept is how long what the mesh answered is kept before it is asked again: long enough that
|
||||
@@ -35,19 +42,21 @@ var IndexKept = 5 * time.Second
|
||||
|
||||
// The seven tools the console announces. Names of the API's kind — letters, digits, `_`, `-`.
|
||||
const (
|
||||
verbOverview = "mesh_overview"
|
||||
verbMachine = "mesh_machine"
|
||||
verbSearch = "mesh_search"
|
||||
verbDescribe = "mesh_describe"
|
||||
verbCall = "mesh_call"
|
||||
verbRuntimes = "mesh_runtimes"
|
||||
verbWait = "mesh_wait"
|
||||
verbOverview = "mesh_overview"
|
||||
verbMachine = "mesh_machine"
|
||||
verbSearch = "mesh_search"
|
||||
verbDescribe = "mesh_describe"
|
||||
verbCall = "mesh_call"
|
||||
verbRuntimes = "mesh_runtimes"
|
||||
verbFirstViews = "mesh_first_views"
|
||||
verbFirstAgain = "mesh_first_view_again"
|
||||
verbWait = "mesh_wait"
|
||||
)
|
||||
|
||||
// searchCap is how many matches a search answers before it says how many more there were.
|
||||
const searchCap = 25
|
||||
|
||||
const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e.g. `mesh-controller.nodes`); " +
|
||||
const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e.g. `mesh-module-assigner.assignments`); " +
|
||||
"`<node>/<seat>.<verb>` for a seat every machine holds (e.g. `ace/node-packet-filter.rules`); " +
|
||||
"`<node>/<module>.<tool>` for a module on one machine (e.g. `novox/postgres.postgres_list_databases`); " +
|
||||
"and `<module>.<tool>` also for a module whose instances are interchangeable."
|
||||
@@ -86,13 +95,22 @@ func discovery() []map[string]any {
|
||||
"arrive, its size in bytes, how many modules and tools it announced, whether it was shortened to fit the " +
|
||||
"bus, and when it was last heard — and every runtime or machine expected that was not heard. Read-only; " +
|
||||
"for when an address is said to be missing or on another machine."},
|
||||
{"name": verbFirstViews, "inputSchema": obj(map[string]any{}),
|
||||
"description": "The tool runner's first views: the assigner's assignments and seat-holders, and the catalog half's registrations, the gate's source and the view of each watched branch (which modules exist, each with its manifest): for each, the owner, the tool " +
|
||||
"asked, the revision held and when it was made, how many keys, when it was asked and answered, how often " +
|
||||
"it was asked and how often by hand, whether its condition is raised, and why it was last asked. Read-only."},
|
||||
{"name": verbFirstAgain, "inputSchema": obj(map[string]any{
|
||||
"why": map[string]any{"type": "string", "description": "why it is asked again, kept in the log"},
|
||||
}), "description": "Ask the tool runner's four first views again, now, and say what each answers. A hand act, recorded in " +
|
||||
"the runner's log with its reason; for when a picture is doubted, or after an owner's state was made again."},
|
||||
waitTool(),
|
||||
approvalsTool(),
|
||||
}
|
||||
}
|
||||
|
||||
func isDiscovery(name string) bool {
|
||||
switch name {
|
||||
case verbOverview, verbMachine, verbSearch, verbDescribe, verbCall, verbRuntimes, verbWait:
|
||||
case verbOverview, verbMachine, verbSearch, verbDescribe, verbCall, verbRuntimes, verbFirstViews, verbFirstAgain, verbWait, verbApprovals:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
@@ -128,13 +146,17 @@ type index struct {
|
||||
NotAnswering []string
|
||||
Listing *Listing // the flat catalogue the call path resolves subjects with
|
||||
Discovery announce.Discovery
|
||||
// Recorded is where the controller's records place each module that declares tools.
|
||||
// Recorded is where the assigner places each module that declares tools.
|
||||
Recorded map[string][]string
|
||||
// Unheard is every runtime whose answer this index lacks: it said it was there and did not say
|
||||
// what it serves in time, or it answered before and not now. An address it might answer is never
|
||||
// called missing while it is here (2026-10-05).
|
||||
Unheard []unheard
|
||||
// Replaces is what each verb and tool replaces, from the controller's records (novox/hq ADR 0245):
|
||||
// Placed is the state seat-holders as the seat mesh-module-assigner's holder wrote it (novox/hq ADR 0571 clause 4), by its
|
||||
// keys `<seat>` and `<seat>.<node>`, read with assigner.HoldersOf; PlacedWhy says why it could not be read.
|
||||
Placed map[string][]byte
|
||||
PlacedWhy string
|
||||
// Replaces is what each verb and tool replaces (novox/hq ADR 0245):
|
||||
// `seat:<seat>.<verb>` and `<module>.<tool>`.
|
||||
Replaces map[string][]string
|
||||
// Earlier is every runtime not heard this round whose last answer, within RuntimeForgotten, this
|
||||
@@ -224,42 +246,8 @@ func findTool(tools []Tool, name string) *Tool {
|
||||
return nil
|
||||
}
|
||||
|
||||
// controllerOutput is a controller seat verb's answer: the command's printed output.
|
||||
func controllerOutput(conn *bus.Conn, verb string) (string, error) {
|
||||
got, err := conn.Ask("seat:mesh-controller."+verb, map[string]any{}, "")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var r struct {
|
||||
Output string `json:"output"`
|
||||
OK *bool `json:"ok"`
|
||||
}
|
||||
if json.Unmarshal(got.Result, &r) != nil {
|
||||
return "", fmt.Errorf("mesh-controller.%s answered something that is not its output", verb)
|
||||
}
|
||||
if r.OK != nil && !*r.OK {
|
||||
return "", fmt.Errorf("mesh-controller.%s: %s", verb, strings.TrimSpace(r.Output))
|
||||
}
|
||||
return r.Output, nil
|
||||
}
|
||||
|
||||
// jsonIn is the JSON document a command printed, after any lines it said first: a seat verb runs
|
||||
// the controller's command, and a command may warn before it answers.
|
||||
func jsonIn(output string) string {
|
||||
t := strings.TrimSpace(output)
|
||||
if strings.HasPrefix(t, "{") || strings.HasPrefix(t, "[") {
|
||||
return t
|
||||
}
|
||||
for _, open := range []string{"\n{", "\n["} {
|
||||
if i := strings.Index(output, open); i >= 0 {
|
||||
return strings.TrimSpace(output[i+1:])
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// recordedModule is a module as the controller's records hold it (`module list --json`): where the
|
||||
// mesh assigned it, and whether it declares tools — what should announce itself, and where.
|
||||
// recordedModule is a module as the bus holds it: where the assigner seat places it, and whether its manifest declares
|
||||
// tools — what should announce itself, and where (novox/hq ADR 0571).
|
||||
type recordedModule struct {
|
||||
Module string `json:"module"`
|
||||
On []string `json:"on"`
|
||||
@@ -268,33 +256,26 @@ type recordedModule struct {
|
||||
Replaces map[string][]string `json:"replaces,omitempty"`
|
||||
}
|
||||
|
||||
// recordedMachine is a machine as the controller's records hold it (`node list --json`).
|
||||
type recordedMachine struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// indexOn asks the mesh what it holds (novox/hq ADR 0197): what answers, from every runtime's own
|
||||
// announcement on the bus — one `$SRV.INFO` request — and what should, from the controller's records
|
||||
// announcement on the bus — one `$SRV.INFO` request — and what should, from the indexer's records
|
||||
// read as JSON. Nothing is inferred from a roster and nothing is parsed from print.
|
||||
//
|
||||
// `held` is what the console holds already, so an instance whose announcement is unchanged is not
|
||||
// asked for it again; `before` is the last answer of every runtime this round did not hear, which the
|
||||
// index holds instead, marked with its age (novox/hq issue 554).
|
||||
func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery) []earlier) (*index, error) {
|
||||
func indexOn(conn *bus.Conn, src Source, held announce.Known, before func(announce.Discovery) []earlier) (*index, error) {
|
||||
var wg sync.WaitGroup
|
||||
var nodesOut, modulesOut string
|
||||
var toolsAnswer json.RawMessage
|
||||
wg.Add(3)
|
||||
go func() { defer wg.Done(); nodesOut, _ = controllerOutput(conn, "nodes") }()
|
||||
go func() { defer wg.Done(); modulesOut, _ = controllerOutput(conn, "modules") }()
|
||||
// The seats' verbs as the records define them, for what each replaces (novox/hq ADR 0245): answered
|
||||
// in-process from the records, never a command.
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if got, err := conn.Ask("seat:mesh-controller.tools", map[string]any{}, ""); err == nil {
|
||||
toolsAnswer = got.Result
|
||||
}
|
||||
}()
|
||||
var placed map[string][]byte
|
||||
var placedErr error
|
||||
var recorded []recordedModule
|
||||
var known []string
|
||||
var recordsWhy []string
|
||||
wg.Add(2)
|
||||
// Which modules there are and where they are placed, and which machines the mesh has, from the bus (ADR 0571
|
||||
// clauses 4 and 5): the indexer's modules and manifests and the assigner seat's assignments, never any other record.
|
||||
go func() { defer wg.Done(); recorded, known, recordsWhy = recordsOn(src) }()
|
||||
// Who holds which seat on which node, from the assigner (ADR 0571 clause 4): no other call.
|
||||
go func() { defer wg.Done(); placed, placedErr = seatHoldersOn(src) }()
|
||||
d, err := announce.Gather(conn, held)
|
||||
wg.Wait()
|
||||
if err != nil {
|
||||
@@ -333,6 +314,10 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
|
||||
l := x.Listing
|
||||
x.Discovery = d
|
||||
x.Stale, x.Earlier = stale, earlierAt
|
||||
x.Placed = placed
|
||||
if placedErr != nil {
|
||||
x.PlacedWhy = placedErr.Error()
|
||||
}
|
||||
for _, s := range d.Silent {
|
||||
x.Unheard = append(x.Unheard, unheard{Runtime: s.Name, Machine: s.Machine,
|
||||
Why: fmt.Sprintf("it answered PING; what it serves did not arrive within %s", announce.Patience)})
|
||||
@@ -340,25 +325,24 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
|
||||
|
||||
// What should have answered: every assignment of a module that declares tools. Silence is named;
|
||||
// a module with no tools is never a name here.
|
||||
var recorded []recordedModule
|
||||
recordedOK := json.Unmarshal([]byte(jsonIn(modulesOut)), &recorded) == nil
|
||||
x.Replaces = replacesOf(toolsAnswer, recorded)
|
||||
if recordedOK {
|
||||
for _, m := range recorded {
|
||||
if !m.Tools {
|
||||
continue
|
||||
}
|
||||
x.Recorded[m.Module] = append([]string{}, m.On...)
|
||||
for _, n := range m.On {
|
||||
// A holder of a seat held once for the mesh announces no machine — the seat is the mesh's,
|
||||
// not a machine's — so what it announced without one answers for wherever it is assigned.
|
||||
if !announced[m.Module][n] && !announced[m.Module][""] {
|
||||
l.NotAnswering = append(l.NotAnswering, m.Module+" on "+n)
|
||||
}
|
||||
x.Replaces = replacesOf(catalogue.Seats(), recorded)
|
||||
for _, m := range recorded {
|
||||
if !m.Tools {
|
||||
continue
|
||||
}
|
||||
x.Recorded[m.Module] = append([]string{}, m.On...)
|
||||
for _, n := range m.On {
|
||||
// A holder of a seat held once for the mesh announces no machine — the seat is the mesh's,
|
||||
// not a machine's — so what it announced without one answers for wherever it is assigned.
|
||||
if !announced[m.Module][n] && !announced[m.Module][""] {
|
||||
l.NotAnswering = append(l.NotAnswering, m.Module+" on "+n)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
l.NotAnswering = append(l.NotAnswering, "mesh-controller (its records of the modules did not answer, so what is missing cannot be said)")
|
||||
}
|
||||
l.NotAnswering = append(l.NotAnswering, recordsWhy...)
|
||||
if x.PlacedWhy != "" {
|
||||
l.NotAnswering = append(l.NotAnswering, firstview.SeatHoldersRead+" (who holds which seat by assignment could not be read: "+
|
||||
x.PlacedWhy+")")
|
||||
}
|
||||
for _, u := range x.Unheard {
|
||||
l.NotAnswering = append(l.NotAnswering, "the runtime "+sayUnheard([]unheard{u}))
|
||||
@@ -366,13 +350,8 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
|
||||
sort.Strings(l.NotAnswering)
|
||||
x.NotAnswering = l.NotAnswering
|
||||
|
||||
var known []recordedMachine
|
||||
if json.Unmarshal([]byte(jsonIn(nodesOut)), &known) == nil {
|
||||
for _, n := range known {
|
||||
if n.Name != "" {
|
||||
machines[n.Name] = true
|
||||
}
|
||||
}
|
||||
for _, n := range known {
|
||||
machines[n] = true
|
||||
}
|
||||
for n := range machines {
|
||||
x.Machines = append(x.Machines, n)
|
||||
@@ -381,6 +360,107 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
|
||||
return x, nil
|
||||
}
|
||||
|
||||
// Source is where the console reads what it indexes the mesh by: the assigner's assignments and seat-holders, by
|
||||
// `<owner>.<state>` (firstview.AssignmentsRead and SeatHoldersRead), and the modules of the catalog half with their entries.
|
||||
// The tool runner's is its first views (firstview.Views), never the owners' state (novox/hq ADR 0615 §1; design 47 G §196, §594).
|
||||
type Source interface {
|
||||
// State is every current value of a state by key; an error says why it cannot be told yet.
|
||||
State(name string) (map[string][]byte, error)
|
||||
// Modules is every module the catalog half holds, by name, with its entry; an error says why it cannot be told yet.
|
||||
Modules() (map[string]meshcatalog.CatalogEntry, error)
|
||||
}
|
||||
|
||||
// seatHoldersOn reads the state seat-holders whole (novox/hq ADR 0571 clause 4), as firstview.SeatHoldersRead.
|
||||
func seatHoldersOn(src Source) (map[string][]byte, error) {
|
||||
return src.State(firstview.SeatHoldersRead)
|
||||
}
|
||||
|
||||
// manifestFacts is what the console takes of a manifest: the tools it declares and what each replaces.
|
||||
type manifestFacts struct {
|
||||
Tools []string `json:"tools"`
|
||||
Replaces map[string][]string `json:"replaces"`
|
||||
}
|
||||
|
||||
// recordsOn is every module the indexer holds, with where the assigner seat places it and whether it declares tools,
|
||||
// and every machine the assignments name (moduleNames, assigner.MachineNames). What cannot be read is said, one line
|
||||
// per state, for the listing's not answering.
|
||||
func recordsOn(src Source) (recorded []recordedModule, machines []string, why []string) {
|
||||
held, err := src.State(firstview.AssignmentsRead)
|
||||
if err != nil {
|
||||
why = append(why, firstview.AssignmentsRead+" (where modules are placed could not be read, so what is missing cannot "+
|
||||
"be said: "+err.Error()+")")
|
||||
}
|
||||
machines = assigner.MachineNames(held)
|
||||
on := map[string][]string{}
|
||||
// The assignments' keys in either form, a node key and its module keys or the form before (research 072 row 37).
|
||||
assembled, err := assigner.AssignmentsOf(held)
|
||||
if err != nil {
|
||||
why = append(why, firstview.AssignmentsRead+" (not every key is an assignment: "+err.Error()+")")
|
||||
}
|
||||
for _, node := range machines {
|
||||
for _, m := range assembled[node].Modules {
|
||||
on[m.Module] = append(on[m.Module], node)
|
||||
}
|
||||
}
|
||||
entries, err := src.Modules()
|
||||
if err != nil {
|
||||
return nil, machines, append(why, "the catalog half's modules (which modules there are could not be read: "+err.Error()+")")
|
||||
}
|
||||
names := make([]string, 0, len(entries))
|
||||
for name := range entries {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, name := range names {
|
||||
r := recordedModule{Module: name, On: on[name]}
|
||||
if m := entries[name].Manifest; len(m) > 0 {
|
||||
facts, err := manifestOf(m)
|
||||
if err != nil {
|
||||
why = append(why, name+" (its manifest could not be read, so whether it answers tools is not known: "+err.Error()+")")
|
||||
} else {
|
||||
r.Tools, r.Replaces = len(facts.Tools) > 0, facts.Replaces
|
||||
}
|
||||
}
|
||||
recorded = append(recorded, r)
|
||||
}
|
||||
return recorded, machines, why
|
||||
}
|
||||
|
||||
// manifestOf is what the console takes of the manifest an entry of the catalog half carries whole (design 47 G §443).
|
||||
func manifestOf(manifest map[string]any) (manifestFacts, error) {
|
||||
raw, err := json.Marshal(manifest)
|
||||
if err != nil {
|
||||
return manifestFacts{}, err
|
||||
}
|
||||
var facts manifestFacts
|
||||
if err := json.Unmarshal(raw, &facts); err != nil {
|
||||
return manifestFacts{}, err
|
||||
}
|
||||
return facts, nil
|
||||
}
|
||||
|
||||
// placedHolders is who holds seat (on node, for a seat held per machine) by assignment, or none.
|
||||
func (x *index) placedHolders(seat, node string) []meshmoduleassigner.SeatHoldersHoldersItem {
|
||||
if len(x.Placed) == 0 {
|
||||
return nil
|
||||
}
|
||||
hs, err := assigner.HoldersOf(x.Placed, seat, node)
|
||||
if err != nil || len(hs) > 0 || node != "" {
|
||||
return hs
|
||||
}
|
||||
// A seat held per machine, asked without one: every node's key of it.
|
||||
var all []meshmoduleassigner.SeatHoldersHoldersItem
|
||||
for key := range x.Placed {
|
||||
if s, n, ok := strings.Cut(key, "."); ok && s == seat {
|
||||
if more, err := assigner.HoldersOf(x.Placed, seat, n); err == nil {
|
||||
all = append(all, more...)
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Slice(all, func(i, j int) bool { return all[i].Node+"/"+all[i].Module < all[j].Node+"/"+all[j].Module })
|
||||
return all
|
||||
}
|
||||
|
||||
func containsHolder(hs []holder, h holder) bool {
|
||||
for _, x := range hs {
|
||||
if x == h {
|
||||
@@ -405,7 +485,7 @@ func (s *Surface) indexAsked(fresh bool) (*index, error) {
|
||||
return x, nil
|
||||
}
|
||||
s.mu.Unlock()
|
||||
x, err := indexOn(s.conn, s.held, s.before)
|
||||
x, err := indexOn(s.conn, s.src, s.held, s.before)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -606,6 +686,14 @@ func resolve(x *index, address string) (target, error) {
|
||||
return target{}, fmt.Errorf("%s is assigned to %s in the mesh's records, but nothing that answered discovery serves it: %s. "+
|
||||
"Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, strings.Join(on, ", "), why)
|
||||
}
|
||||
if hs := x.placedHolders(prefix, node); len(hs) > 0 {
|
||||
var held []string
|
||||
for _, h := range hs {
|
||||
held = append(held, h.Module+" on "+h.Node)
|
||||
}
|
||||
return target{}, fmt.Errorf("the seat %s is held by %s by assignment, but no holder of it announced itself on the bus. "+
|
||||
"Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, strings.Join(held, ", "))
|
||||
}
|
||||
if len(x.Unheard) > 0 {
|
||||
return target{}, fmt.Errorf("nothing that answered discovery is called %s, but not every runtime answered: %s — "+
|
||||
"%s may be theirs. Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, sayUnheard(x.Unheard), prefix)
|
||||
@@ -711,7 +799,7 @@ func firstLine(s string) string {
|
||||
// the machine, `node` is taken out: the machine is in the address. Where it does not — a seat held
|
||||
// once for the mesh, or an interchangeable module — `node` is the tool's own argument, and is kept.
|
||||
//
|
||||
// **Taking it out everywhere made the mesh's own verbs lie** (novox/hq issue 244): the controller's
|
||||
// **Taking it out everywhere made the mesh's own verbs lie** (novox/hq issue 244): the old controller verbs
|
||||
// `push`, `plan`, `assign`, `pin` and `settings` take the machine they act on as `node`, and were
|
||||
// described without it — `push` as taking nothing at all, so a push naming one machine arrived as a
|
||||
// push of every machine behind.
|
||||
@@ -744,6 +832,11 @@ func argumentsAsSent(t target, given map[string]any) (map[string]any, error) {
|
||||
if !t.Seat {
|
||||
return args, nil
|
||||
}
|
||||
t, held := seatDefinitionHeld(t)
|
||||
if !held {
|
||||
// Nothing here says what the verb takes: its holder judges the arguments (issue 1966).
|
||||
return args, nil
|
||||
}
|
||||
declared := map[string]bool{}
|
||||
var names []string
|
||||
if p, ok := schemaAsPassed(t)["properties"].(map[string]any); ok {
|
||||
@@ -771,6 +864,36 @@ func argumentsAsSent(t target, given map[string]any) (map[string]any, error) {
|
||||
return args, nil
|
||||
}
|
||||
|
||||
// seatDefinitionHeld is a seat verb's target with the input the console judges a call by, and whether anything here
|
||||
// says what the verb takes (issue 1966): the schema its holder announced when it declares an argument, else the verb's
|
||||
// input in the seat's definition this runner holds. Neither holds while the runner waits for the seat list holding
|
||||
// only the bootstrapped definitions and the holder announced an empty schema: then a call's arguments are passed on to
|
||||
// the holder, which judges its own input, and mesh_describe says the definition is not held.
|
||||
func seatDefinitionHeld(t target) (target, bool) {
|
||||
if p, ok := asSchema(t.Tool.Input)["properties"].(map[string]any); ok && len(p) > 0 {
|
||||
return t, true
|
||||
}
|
||||
seat, ok := catalogue.SeatNamed(t.Tool.Module)
|
||||
if !ok {
|
||||
return t, false
|
||||
}
|
||||
for _, v := range seat.Serves {
|
||||
if v.Name == t.Tool.Name && v.Input != nil {
|
||||
if raw, err := json.Marshal(v.Input); err == nil {
|
||||
t.Tool.Input = raw
|
||||
}
|
||||
}
|
||||
}
|
||||
return t, true
|
||||
}
|
||||
|
||||
// notHeld is what mesh_describe says of a seat verb whose definition this runner does not hold.
|
||||
func notHeld(seat string) string {
|
||||
return "the seat list this tool runner holds has no definition of " + seat + " yet (it holds the bootstrapped ones " +
|
||||
"until it reads the list): what the verb takes is not known here, so the arguments of a call are passed to " +
|
||||
"its holder, which judges them"
|
||||
}
|
||||
|
||||
// schemaWithoutNode is a tool's schema with `node` taken out, for an address that names the machine.
|
||||
func schemaWithoutNode(raw json.RawMessage) map[string]any {
|
||||
schema := asSchema(raw)
|
||||
@@ -804,6 +927,21 @@ func schemaWithoutNode(raw json.RawMessage) map[string]any {
|
||||
}
|
||||
|
||||
// answerText is a discovery verb's answer as MCP content: JSON, indented.
|
||||
// callFailed is why a call failed, as mesh_call says it: a marked tool or verb that reaches no responder names `withheld`
|
||||
// and the grants part's read (design 46 7-approval-mark §6), and a target known only from an earlier round says so.
|
||||
func (s *Surface) callFailed(t target, err error) string {
|
||||
why := whyItFailed(t.Address, err)
|
||||
if ok, _ := s.markedTool(t); ok && errors.Is(err, bus.ErrNoResponders) {
|
||||
// A marked subject withheld from a machine whose tool runner does not honour the mark (§6).
|
||||
why += "; it needs the operator's approval, so it may be withheld where the machine's tool runner does not state " +
|
||||
"approval-mark/1: mesh-grants.grants_user_list names it under withheld"
|
||||
}
|
||||
if t.Stale != nil {
|
||||
why += " (" + t.Stale.say() + ")"
|
||||
}
|
||||
return why
|
||||
}
|
||||
|
||||
func answerText(v any) map[string]any {
|
||||
var b strings.Builder
|
||||
enc := json.NewEncoder(&b)
|
||||
@@ -822,9 +960,15 @@ func (s *Surface) discover(ctx context.Context, name string, args map[string]any
|
||||
if name == verbRuntimes {
|
||||
return s.runtimesAnswer()
|
||||
}
|
||||
if name == verbFirstViews || name == verbFirstAgain {
|
||||
return s.firstViewsAnswer(ctx, name, args)
|
||||
}
|
||||
if name == verbWait {
|
||||
return s.waitAnswer(ctx, args)
|
||||
}
|
||||
if name == verbApprovals {
|
||||
return s.approvalsAnswer(args)
|
||||
}
|
||||
x, err := s.index()
|
||||
if err != nil {
|
||||
return failure("the mesh's discovery failed: " + err.Error())
|
||||
@@ -967,7 +1111,14 @@ func (s *Surface) discover(ctx context.Context, name string, args map[string]any
|
||||
if description == "" {
|
||||
description = t.Name
|
||||
}
|
||||
held := true
|
||||
if t.Seat {
|
||||
t, held = seatDefinitionHeld(t)
|
||||
}
|
||||
described := map[string]any{"address": t.Address, "description": description, "arguments": schemaAsPassed(t)}
|
||||
if !held {
|
||||
described["arguments"] = notHeld(t.Tool.Module)
|
||||
}
|
||||
key := t.Tool.Module + "." + t.Tool.Name
|
||||
if t.Seat {
|
||||
key = "seat:" + key
|
||||
@@ -978,6 +1129,10 @@ func (s *Surface) discover(ctx context.Context, name string, args map[string]any
|
||||
if t.Stale != nil {
|
||||
described["found"] = t.Stale.say()
|
||||
}
|
||||
if ok, source := s.markedTool(t); ok {
|
||||
described["needs-approval"] = map[string]any{"source": source,
|
||||
"says": "the first call asks the operator and runs nothing; the same call again within 30 minutes of the Approve runs once"}
|
||||
}
|
||||
return answerText(described)
|
||||
|
||||
case verbCall:
|
||||
@@ -997,13 +1152,13 @@ func (s *Surface) discover(ctx context.Context, name string, args map[string]any
|
||||
got, err = callTool(s.conn, t.Key, callArgs, nil, x.Listing)
|
||||
}
|
||||
if err != nil {
|
||||
why := whyItFailed(t.Address, err)
|
||||
if t.Stale != nil {
|
||||
why += " (" + t.Stale.say() + ")"
|
||||
}
|
||||
return failure(why)
|
||||
return failure(s.callFailed(t, err))
|
||||
}
|
||||
result, unread := resolveSent(s.conn, got.Result)
|
||||
content := []map[string]any{{"type": "text", "text": pretty(result)}}
|
||||
if unread != "" {
|
||||
content = append(content, map[string]any{"type": "text", "text": unread})
|
||||
}
|
||||
content := []map[string]any{{"type": "text", "text": pretty(got.Result)}}
|
||||
if got.Node != "" {
|
||||
content = append(content, map[string]any{"type": "text", "text": "answered by " + got.Node})
|
||||
}
|
||||
|
||||
@@ -70,35 +70,34 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
t.Cleanup(stop)
|
||||
}
|
||||
|
||||
// The controller: its records as JSON, as its seat verbs answer them, and its own seat's verbs
|
||||
// announced on the bus like every runtime's.
|
||||
controller := connect(t, "mesh-controller", "bench")
|
||||
out := func(s string) map[string]any { return map[string]any{"output": s, "ok": true} }
|
||||
serve := func(verb string, answer func() any) {
|
||||
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
|
||||
return answer(), nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(stop)
|
||||
// The controller: its own seat's verbs announced on the bus like every runtime's; its records are asked by no one.
|
||||
controller := connect(t, "mesh-module-assigner", "bench")
|
||||
// Its verb nodes, called through the console as any seat's verb is.
|
||||
stopNodes, err := controller.HandleSubject("mesh.seat.mesh-module-assigner.tool.nodes", func(json.RawMessage) (any, error) {
|
||||
return map[string]any{"ok": true, "output": `[{"name":"bench","mode":"converged"},{"name":"desk","mode":"converged"}]`}, nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
serve("nodes", func() any {
|
||||
return out("the bus's user list leaves out 2 user(s)\n" +
|
||||
`[{"name":"bench","heard":"3m ago","mode":"converged","id":"1f2e"},{"name":"desk","heard":"here","mode":"converged","id":"9a8b"}]`)
|
||||
t.Cleanup(stopNodes)
|
||||
// What the mesh holds, as the indexer and the assigner write it (ADR 0571 clauses 4, 5): never the controller's records.
|
||||
tools := map[string][]string{"alpha": {"one", "two"}, "beta": {"three"}, "gamma": {"given"}, "delta": nil,
|
||||
"epsilon": {"seven"}, "mesh-module-assigner": {"nodes"}}
|
||||
mesh.Placed(t, tools, map[string][]string{"desk": {"alpha", "beta", "delta"}, "bench": {"epsilon", "mesh-module-assigner"}})
|
||||
console := mt.MembershipOf("node-tools", "desk", false, nil)
|
||||
console.State = mt.ConsoleReads()
|
||||
mesh.Issue(t, console)
|
||||
mt.Until(t, func() error {
|
||||
if m := nodeTools.Membership("node-tools"); m == nil || len(m.State) == 0 {
|
||||
return fmt.Errorf("the console's reads are not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
var gammaOn atomic.Value
|
||||
gammaOn.Store("[]")
|
||||
serve("modules", func() any {
|
||||
return out(fmt.Sprintf(`[{"module":"alpha","on":["desk"],"tools":true},{"module":"beta","on":["desk"],"tools":true},`+
|
||||
`{"module":"gamma","on":%s,"tools":true},{"module":"delta","on":["desk"],"tools":false},`+
|
||||
`{"module":"epsilon","on":["bench"],"tools":true},{"module":"mesh-controller","on":["bench"],"tools":true}]`, gammaOn.Load().(string)))
|
||||
})
|
||||
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-controller", ID: "bench"}, func() []announce.Endpoint {
|
||||
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller", Scope: "mesh",
|
||||
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-module-assigner", ID: "bench"}, func() []announce.Endpoint {
|
||||
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-module-assigner", Seat: "mesh-module-assigner", Scope: "mesh",
|
||||
// As the live controller announces: a mesh seat's holder names no machine.
|
||||
Tool: "nodes", Node: "", Description: "Every machine the mesh knows.", Schema: json.RawMessage(`{}`),
|
||||
Subject: "mesh.seat.mesh-controller.tool.nodes"}}
|
||||
Subject: "mesh.seat.mesh-module-assigner.tool.nodes"}}
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -107,7 +106,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
controller.Flush()
|
||||
watcher.Flush()
|
||||
|
||||
up, err := Serve(NewSurface(nodeTools, "desk.node-tools"), "127.0.0.1:0")
|
||||
up, err := Serve(NewSurface(nodeTools, busSource{nodeTools}, "desk.node-tools"), "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -126,19 +125,19 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
if got := strings.Join(names, ","); got != "mesh_overview,mesh_machine,mesh_search,mesh_describe,mesh_call,mesh_runtimes,mesh_wait" {
|
||||
if got := strings.Join(names, ","); got != "mesh_overview,mesh_machine,mesh_search,mesh_describe,mesh_call,mesh_runtimes,mesh_first_views,mesh_first_view_again,mesh_wait" {
|
||||
t.Errorf("announced %s", got)
|
||||
}
|
||||
|
||||
// The overview names the mesh's seats, the machines' seats and the machines.
|
||||
overview, isErr := call(t, endpoint, "mesh_overview", nil)
|
||||
if isErr || !strings.Contains(overview, "mesh-controller.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
|
||||
if isErr || !strings.Contains(overview, "mesh-module-assigner.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
|
||||
!strings.Contains(overview, `"bench"`) || !strings.Contains(overview, `"desk"`) {
|
||||
t.Errorf("overview: %s", overview)
|
||||
}
|
||||
// Silence is named only where tools should have answered: epsilon declares tools on bench and
|
||||
// nothing there announced it; delta declares none and is never a name.
|
||||
if strings.Contains(overview, "mesh-controller on bench") {
|
||||
if strings.Contains(overview, "mesh-module-assigner on bench") {
|
||||
t.Errorf("a mesh seat's holder that announced no machine is called silent on its machine:\n%s", overview)
|
||||
}
|
||||
if !strings.Contains(overview, "epsilon on bench") || strings.Contains(overview, "delta") {
|
||||
@@ -151,7 +150,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
}
|
||||
|
||||
// A mesh seat, a node seat, an assignment and an interchangeable module, each by address.
|
||||
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-controller.nodes"}); isErr || !strings.Contains(got, "converged") {
|
||||
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-module-assigner.nodes"}); isErr || !strings.Contains(got, "converged") {
|
||||
t.Errorf("mesh seat: %s", got)
|
||||
}
|
||||
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "desk/node-shelf.list"}); isErr || !strings.Contains(got, `"a"`) {
|
||||
@@ -167,11 +166,11 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
|
||||
// Refused, by name, where the address does not say enough or says the wrong thing.
|
||||
for address, want := range map[string]string{
|
||||
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
|
||||
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
|
||||
"desk/mesh-controller.nodes": "held once for the whole mesh",
|
||||
"bench/beta.three": "beta does not run on bench; it runs on desk",
|
||||
"nonsense": "is not an address",
|
||||
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
|
||||
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
|
||||
"desk/mesh-module-assigner.nodes": "held once for the whole mesh",
|
||||
"bench/beta.three": "beta does not run on bench; it runs on desk",
|
||||
"nonsense": "is not an address",
|
||||
} {
|
||||
got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": address})
|
||||
if !isErr || !strings.Contains(got, want) {
|
||||
@@ -202,7 +201,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
t.Fatalf("gamma was found before it served: %s", got)
|
||||
}
|
||||
mesh.Issue(t, mt.MembershipOf("gamma", "desk", false, nil))
|
||||
gammaOn.Store(`["desk"]`)
|
||||
mesh.Placed(t, tools, map[string][]string{"desk": {"alpha", "beta", "delta", "gamma"}, "bench": {"epsilon", "mesh-module-assigner"}})
|
||||
late := connect(t, "node-tools", "desk")
|
||||
stopLate, err := runtime.Run(late, []runtime.Served{{Module: "gamma", Entrypoints: []string{mt.Fixture("env-gamma.serve.mjs")}}},
|
||||
nil, (&mt.Logs{}).Logf)
|
||||
@@ -233,12 +232,3 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
t.Errorf("an old name: %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheControllersRecordsAreReadAsJSON(t *testing.T) {
|
||||
if got := jsonIn("a warning printed first\n[{\"name\":\"ace\"}]"); got != `[{"name":"ace"}]` {
|
||||
t.Errorf("an array after a warning: %q", got)
|
||||
}
|
||||
if got := jsonIn(`{"seats":[]}`); got != `{"seats":[]}` {
|
||||
t.Errorf("an object: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
package console
|
||||
|
||||
import (
|
||||
"log"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
)
|
||||
|
||||
// verbApprovals is the mesh MCP server's tool that reads and steers what the gate asked on this machine (design 46
|
||||
// 7-approval-mark §18).
|
||||
const verbApprovals = "mesh_approvals"
|
||||
|
||||
// Approvals is the gate of this machine's tool runner, as mesh_approvals and mesh_describe read it.
|
||||
type Approvals interface {
|
||||
List() ([]approval.Listed, error)
|
||||
Read(id string) (approval.Whole, error)
|
||||
Applied() []approval.Marks
|
||||
Describe(module, seat, name string) (bool, string)
|
||||
Again(id string) error
|
||||
Cancel(id string) error
|
||||
// Fill is the state approvals' bytes against its cap, every machine's keys together.
|
||||
Fill() (bytes uint64, max int64, err error)
|
||||
}
|
||||
|
||||
func approvalsTool() map[string]any {
|
||||
str := func(desc string) map[string]any { return map[string]any{"type": "string", "description": desc} }
|
||||
return map[string]any{"name": verbApprovals, "inputSchema": map[string]any{"type": "object", "properties": map[string]any{
|
||||
"ask": str("an ask's id: its whole arguments and the act's digest"),
|
||||
"record": str("an ask's id: ask mesh-hand-acts.warranted again for its run, recorded in the hand-act log first"),
|
||||
"cancel": str("an ask's id: take that open ask back, recorded in the hand-act log first"),
|
||||
"why": str("why, for record and cancel, kept in the hand-act log"),
|
||||
}}, "description": "The calls that needed the operator's approval on this machine: with no argument, those asked in " +
|
||||
"the last day, newest first, with their state and fingerprint, the marks the tool runner applies (each module's " +
|
||||
"marked tools and verbs with their source and the catalog revision read) and its state's fill; `ask` reads one " +
|
||||
"whole, `record` asks its hand-act again, `cancel` takes an open ask back."}
|
||||
}
|
||||
|
||||
// approvalsAnswer is mesh_approvals.
|
||||
func (s *Surface) approvalsAnswer(args map[string]any) map[string]any {
|
||||
if s.Approvals == nil {
|
||||
return failure("this tool runner has no approval gate")
|
||||
}
|
||||
str := func(k string) string { v, _ := args[k].(string); return strings.TrimSpace(v) }
|
||||
why := str("why")
|
||||
if why == "" {
|
||||
why = "asked by hand with no reason given"
|
||||
}
|
||||
act := func(id, what, cause string, do func(string) error) map[string]any {
|
||||
// A hand act, recorded through the seat mesh-hand-acts before it is done (§18); one that cannot be recorded is refused.
|
||||
log.Printf("[mesh-tools] hand act: %s %s ask %s: %s", s.who, what, id, why)
|
||||
if err := s.recordAct(map[string]any{"what": what + " ask " + id, "why": why, "cause": cause, "verb": verbApprovals}); err != nil {
|
||||
return failure("the hand act was not recorded through mesh-hand-acts, so nothing was done: " + err.Error())
|
||||
}
|
||||
if err := do(id); err != nil {
|
||||
return failure(err.Error())
|
||||
}
|
||||
return answerText(map[string]any{"done": what + " ask " + id, "recorded": "mesh-hand-acts.hand-act"})
|
||||
}
|
||||
switch {
|
||||
case str("record") != "":
|
||||
return act(str("record"), "asked the hand-act of", "approval-record", s.Approvals.Again)
|
||||
case str("cancel") != "":
|
||||
return act(str("cancel"), "cancelled", "approval-cancel", s.Approvals.Cancel)
|
||||
case str("ask") != "":
|
||||
w, err := s.Approvals.Read(str("ask"))
|
||||
if err != nil {
|
||||
return failure(err.Error())
|
||||
}
|
||||
return answerText(w)
|
||||
}
|
||||
asks, err := s.Approvals.List()
|
||||
out := map[string]any{"asks": asks, "marks": s.Approvals.Applied()}
|
||||
if err != nil {
|
||||
out["asks"], out["asks_why"] = []approval.Listed{}, "the state approvals cannot be read: "+err.Error()
|
||||
}
|
||||
if b, max, err := s.Approvals.Fill(); err != nil {
|
||||
out["fill_why"] = err.Error()
|
||||
} else {
|
||||
out["fill"] = map[string]any{"bytes": b, "max_bytes": max, "of": "every machine's keys together"}
|
||||
}
|
||||
return answerText(out)
|
||||
}
|
||||
|
||||
// marked is what mesh_describe and a call that reached no one say of a marked tool or verb (§6, §18).
|
||||
func (s *Surface) marked(module, seat, name string) (bool, string) {
|
||||
if s.Approvals == nil {
|
||||
return false, ""
|
||||
}
|
||||
return s.Approvals.Describe(module, seat, name)
|
||||
}
|
||||
|
||||
// markedTool is marked for a resolved address.
|
||||
func (s *Surface) markedTool(t target) (bool, string) {
|
||||
if t.Seat {
|
||||
return s.marked("", t.Tool.Module, t.Tool.Name)
|
||||
}
|
||||
return s.marked(t.Tool.Module, "", t.Tool.Name)
|
||||
}
|
||||
@@ -63,8 +63,8 @@ func toolKey(name string, seats Seats) string {
|
||||
|
||||
// toolsOn is the flat catalogue (MESH_CONSOLE_FLAT=1): what announced itself on the bus, every
|
||||
// tool and seat verb, and the assignments with tools that did not (novox/hq ADR 0197).
|
||||
func toolsOn(conn *bus.Conn) (*Listing, error) {
|
||||
x, err := indexOn(conn, nil, nil)
|
||||
func toolsOn(conn *bus.Conn, src Source) (*Listing, error) {
|
||||
x, err := indexOn(conn, src, nil, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -118,9 +118,6 @@ var (
|
||||
timedOut = regexp.MustCompile(`(?i)timeout`)
|
||||
)
|
||||
|
||||
// controllerSeat is the seat whose holder keeps what came of every call (novox/hq issue 265).
|
||||
const controllerSeat = "mesh-controller"
|
||||
|
||||
// whyItFailed says why a call failed, so the remedy is in the words.
|
||||
func whyItFailed(key string, err error) string {
|
||||
if err == nil {
|
||||
@@ -145,12 +142,6 @@ func whyItFailed(key string, err error) string {
|
||||
said := key + " did not answer within " + bus.RequestTimeout.String() + ". That is not a failure: " +
|
||||
"something is serving it, so it may still be running and may already have done what was " +
|
||||
"asked — check before repeating a call that changes the mesh."
|
||||
if strings.HasPrefix(strings.TrimPrefix(key, "seat:"), controllerSeat+".") {
|
||||
// The controller answers every call within seconds, with an id while it runs, so silence
|
||||
// from it is an answer lost on the way — which it keeps.
|
||||
said += " The controller answers every call within seconds, so its answer was lost on the " +
|
||||
"way; " + controllerSeat + ".calls lists its recent calls and what came of each."
|
||||
}
|
||||
return said
|
||||
}
|
||||
return key + " failed: " + msg
|
||||
|
||||
@@ -3,6 +3,7 @@ package console
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
@@ -56,16 +57,21 @@ func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
|
||||
}
|
||||
defer stop()
|
||||
|
||||
// The controller's records (ADR 0197): ghost declares tools on desk and nothing announces it.
|
||||
controller := connect(t, "mesh-controller", "")
|
||||
stopSeat, _ := controller.HandleSubject("mesh.seat.mesh-controller.tool.modules", func(json.RawMessage) (any, error) {
|
||||
return map[string]any{"ok": true, "output": `[{"module":"alpha","on":["desk"],"tools":true},` +
|
||||
`{"module":"beta","on":["desk"],"tools":true},{"module":"ghost","on":["desk"],"tools":true}]`}, nil
|
||||
// What the mesh holds, as the indexer and the assigner write it (ADR 0571): ghost declares tools on desk and nothing
|
||||
// announces it.
|
||||
mesh.Placed(t, map[string][]string{"alpha": {"one", "two"}, "beta": {"three"}, "ghost": {"boo"}},
|
||||
map[string][]string{"desk": {"alpha", "beta", "ghost"}})
|
||||
console := mt.MembershipOf("node-tools", "desk", false, nil)
|
||||
console.State = mt.ConsoleReads()
|
||||
mesh.Issue(t, console)
|
||||
mt.Until(t, func() error {
|
||||
if m := nodeTools.Membership("node-tools"); m == nil || len(m.State) == 0 {
|
||||
return fmt.Errorf("the console's reads are not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
defer stopSeat()
|
||||
controller.Flush()
|
||||
|
||||
flat := NewSurface(nodeTools, "desk.node-tools")
|
||||
flat := NewSurface(nodeTools, busSource{nodeTools}, "desk.node-tools")
|
||||
flat.Flat = true // the whole catalogue, as before ADR 0195: still reachable, no longer announced
|
||||
up, err := Serve(flat, "127.0.0.1:0")
|
||||
if err != nil {
|
||||
@@ -126,7 +132,7 @@ func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestTheConsoleListensOnLoopbackAndNowhereElse(t *testing.T) {
|
||||
if _, err := Serve(NewSurface(nil, "x"), "0.0.0.0:0"); err == nil || !strings.Contains(err.Error(), "loopback and nowhere else") {
|
||||
if _, err := Serve(NewSurface(nil, nil, "x"), "0.0.0.0:0"); err == nil || !strings.Contains(err.Error(), "loopback and nowhere else") {
|
||||
t.Errorf("a non-loopback console was not refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -72,7 +72,7 @@ func TestACallAfterARoundThatHeardNothingUsesWhatWasHeardBefore(t *testing.T) {
|
||||
}
|
||||
return n
|
||||
}
|
||||
surface := NewSurface(console, "laptop.node-tools")
|
||||
surface := NewSurface(console, busSource{console}, "laptop.node-tools")
|
||||
up, err := Serve(surface, "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
package console
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/process"
|
||||
"git.novox.be/novox/mesh-sdk/go/views"
|
||||
)
|
||||
|
||||
// FirstViewer is the tool runner's four first views as the console shows them (novox/hq ADR 0615 §6; issue 1966 TLS-6): a
|
||||
// Source that also is one is asked by mesh_first_views and mesh_first_view_again.
|
||||
type FirstViewer interface {
|
||||
Status() []views.Status
|
||||
Again(ctx context.Context, why string) []error
|
||||
}
|
||||
|
||||
// entryLister is a FirstViewer that also keeps entries by fingerprint (design 47 G A-103): mesh_first_views lists them with their
|
||||
// state and since when.
|
||||
type entryLister interface{ Entries() any }
|
||||
|
||||
// ownBuilder is a FirstViewer that also says the build the tool runner itself was started with (design 47 G §205, §424):
|
||||
// mesh_first_views names it, or says `own build unknown` and why.
|
||||
type ownBuilder interface{ OwnBuild() process.Build }
|
||||
|
||||
// firstViewsAnswer is mesh_first_views and mesh_first_view_again.
|
||||
func (s *Surface) firstViewsAnswer(ctx context.Context, name string, args map[string]any) map[string]any {
|
||||
fv, ok := s.src.(FirstViewer)
|
||||
if !ok {
|
||||
return failure("this tool runner holds no first views: it reads the indexer's and the assigner's states through " +
|
||||
"another source")
|
||||
}
|
||||
if name == verbFirstViews {
|
||||
out := map[string]any{"first views": fv.Status()}
|
||||
if ob, ok := fv.(ownBuilder); ok {
|
||||
out["own build"] = ob.OwnBuild().String()
|
||||
}
|
||||
if el, ok := fv.(entryLister); ok {
|
||||
out["entries"] = el.Entries()
|
||||
}
|
||||
return answerText(out)
|
||||
}
|
||||
why, _ := args["why"].(string)
|
||||
if why = strings.TrimSpace(why); why == "" {
|
||||
why = "asked by hand with no reason given"
|
||||
}
|
||||
// A hand act, recorded through the seat mesh-hand-acts (novox/hq ADR 0615 §6, ADR 0565 §10) before it is done, and in the
|
||||
// runner's log. One that cannot be recorded is said, and the ask is not made.
|
||||
log.Printf("[mesh-tools] hand act: %s asked the first views again: %s", s.who, why)
|
||||
if err := s.recordAct(map[string]any{"what": "asked the tool runner's first views again", "why": why,
|
||||
"cause": "first-view-again", "verb": verbFirstAgain}); err != nil {
|
||||
return failure("the hand act was not recorded through mesh-hand-acts, so the first views were not asked again: " + err.Error())
|
||||
}
|
||||
out := map[string]any{"asked again": why, "recorded": "mesh-hand-acts.hand-act"}
|
||||
if errs := fv.Again(ctx, why); len(errs) > 0 {
|
||||
var msgs []string
|
||||
for _, e := range errs {
|
||||
msgs = append(msgs, e.Error())
|
||||
}
|
||||
out["not answered"] = msgs
|
||||
}
|
||||
out["first views"] = fv.Status()
|
||||
return answerText(out)
|
||||
}
|
||||
|
||||
// recordAct records one act done by hand through the seat mesh-hand-acts (its verb hand-act); Surface.Record replaces the call
|
||||
// in a test.
|
||||
func (s *Surface) recordAct(args map[string]any) error {
|
||||
if s.Record != nil {
|
||||
return s.Record(args)
|
||||
}
|
||||
if s.conn == nil {
|
||||
return errNoBus
|
||||
}
|
||||
_, err := s.conn.Ask("mesh-hand-acts.hand-act", args, "")
|
||||
return err
|
||||
}
|
||||
|
||||
var errNoBus = errors.New("no bus connection")
|
||||
@@ -0,0 +1,74 @@
|
||||
package console
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/process"
|
||||
"git.novox.be/novox/mesh-sdk/go/views"
|
||||
)
|
||||
|
||||
// viewer is a Source that also holds first views.
|
||||
type viewer struct {
|
||||
busSource
|
||||
asked []string
|
||||
build process.Build
|
||||
}
|
||||
|
||||
func (v *viewer) Status() []views.Status {
|
||||
return []views.Status{{Owner: "mesh-delivery-assigner", Tool: "assignments", State: "assignments", Revision: 4, Asks: 1}}
|
||||
}
|
||||
|
||||
// OwnBuild is the build the viewer was started with.
|
||||
func (v *viewer) OwnBuild() process.Build { return v.build }
|
||||
|
||||
// Entries are the entries by fingerprint the viewer keeps (design 47 G A-103).
|
||||
func (v *viewer) Entries() any { return []string{"fp:a asked"} }
|
||||
|
||||
func (v *viewer) Again(_ context.Context, why string) []error {
|
||||
v.asked = append(v.asked, why)
|
||||
return nil
|
||||
}
|
||||
|
||||
// issue 1966 TLS-6: the console's mesh_first_views shows each first view, mesh_first_view_again asks them again with its
|
||||
// reason, and a source holding none refuses both in words.
|
||||
func TestTheConsoleShowsAndAsksAgainTheRunnersFirstViews(t *testing.T) {
|
||||
text := func(r map[string]any) string {
|
||||
c, _ := json.Marshal(r["content"])
|
||||
return string(c)
|
||||
}
|
||||
v := &viewer{}
|
||||
var recorded []map[string]any
|
||||
s := &Surface{src: v, who: "desk.node-tools", Record: func(a map[string]any) error { recorded = append(recorded, a); return nil }}
|
||||
if got := text(s.discover(context.Background(), "mesh_first_views", nil)); !strings.Contains(got, "mesh-delivery-assigner") || !strings.Contains(got, "assignments") || !strings.Contains(got, "fp:a asked") {
|
||||
t.Errorf("mesh_first_views shows %s", got)
|
||||
}
|
||||
// The tool runner's own build is named, or said unknown with the reason (design 47 G §205, §424).
|
||||
for build, want := range map[string]string{"abcd1234": "abcd1234", "": "own build unknown"} {
|
||||
v.build = process.ReadOwnBuild(func(string) string { return build })
|
||||
if got := text(s.discover(context.Background(), "mesh_first_views", nil)); !strings.Contains(got, `own build`) || !strings.Contains(got, want) {
|
||||
t.Errorf("mesh_first_views with the build %q shows %s", build, got)
|
||||
}
|
||||
}
|
||||
got := s.discover(context.Background(), "mesh_first_view_again", map[string]any{"why": "doubted"})
|
||||
if len(v.asked) != 1 || v.asked[0] != "doubted" || !strings.Contains(text(got), "asked again") {
|
||||
t.Errorf("mesh_first_view_again asked %v and answered %s", v.asked, text(got))
|
||||
}
|
||||
if len(recorded) != 1 || recorded[0]["why"] != "doubted" || recorded[0]["cause"] != "first-view-again" {
|
||||
t.Errorf("the hand act recorded is %v", recorded)
|
||||
}
|
||||
// An act that cannot be recorded is not done.
|
||||
refusing := &Surface{src: v, who: "desk.node-tools", Record: func(map[string]any) error { return errors.New("no answer") }}
|
||||
if r := refusing.discover(context.Background(), "mesh_first_view_again", map[string]any{"why": "x"}); r["isError"] != true || len(v.asked) != 1 {
|
||||
t.Errorf("an unrecorded hand act was done: %s (asked %v)", text(r), v.asked)
|
||||
}
|
||||
for _, name := range []string{"mesh_first_views", "mesh_first_view_again"} {
|
||||
none := &Surface{src: busSource{}, who: "desk.node-tools"}
|
||||
if r := none.discover(context.Background(), name, nil); r["isError"] != true || !strings.Contains(text(r), "holds no first views") {
|
||||
t.Errorf("%s without first views answers %s", name, text(r))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -8,34 +8,90 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
// novox/hq ADR 0367 §5: the tool runner's manifest names the families a session waits on today, one by one,
|
||||
// and the asker's `decided` for the claude-code module alone. Adding one is a reviewed change to this file
|
||||
// (§6), so this list changes with it. The delivery's transitions are waited on under the gate's name alone (ADR 0450 §8, issues 1350
|
||||
// and 1406): today's module is forgotten.
|
||||
func TestTheManifestNamesTheFirstFamiliesOneByOne(t *testing.T) {
|
||||
// The tool runner's manifest names no events to wait on (novox/hq ADR 0670 §2): a session waits on any event by its
|
||||
// name. It holds each change of design 34 every-event §11 (E-1): no `waits`, `operator-channel` in `uses` for the asks of
|
||||
// marked waits, the per-machine state `approvals` in the old `state` section with its 48 hours and none in `requires`
|
||||
// (design 46 7-approval-mark §13), and the finding stream (design 47 G §387).
|
||||
func TestTheManifestNamesNoWaitsAndKeepsItsApprovals(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var keys map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &keys); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, ok := keys["waits"]; ok {
|
||||
t.Error("the manifest still carries waits (novox/hq ADR 0670 §2)")
|
||||
}
|
||||
var m struct {
|
||||
Module string `json:"module"`
|
||||
Uses []string `json:"uses"`
|
||||
State []json.RawMessage `json:"state"`
|
||||
Requires []json.RawMessage `json:"requires"`
|
||||
Consumes []string `json:"consumes"`
|
||||
Streams []struct {
|
||||
Name string `json:"name"`
|
||||
Events []string `json:"events"`
|
||||
} `json:"streams"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m.Module != "node-tools" || !reflect.DeepEqual(m.Uses, []string{"operator-channel"}) {
|
||||
t.Errorf("%s uses %v, want operator-channel (design 46 7-approval-mark §7)", m.Module, m.Uses)
|
||||
}
|
||||
type state struct {
|
||||
Name string `json:"name"`
|
||||
PerMachine bool `json:"per-machine"`
|
||||
TTLSeconds int `json:"ttl-seconds"`
|
||||
}
|
||||
var approvals []state
|
||||
for _, r := range m.State {
|
||||
var s state
|
||||
if json.Unmarshal(r, &s) == nil && s.Name == "approvals" {
|
||||
approvals = append(approvals, s)
|
||||
}
|
||||
}
|
||||
if want := []state{{Name: "approvals", PerMachine: true, TTLSeconds: 172800}}; !reflect.DeepEqual(approvals, want) {
|
||||
t.Errorf("the old state section's approvals is %+v, want %+v (design 46 7-approval-mark §13)", approvals, want)
|
||||
}
|
||||
for _, r := range m.Requires {
|
||||
if strings.Contains(string(r), `"approvals"`) {
|
||||
t.Errorf("approvals is asked in requires too: %s", r)
|
||||
}
|
||||
}
|
||||
if len(m.Streams) != 1 || m.Streams[0].Name != "node-tools-findings" ||
|
||||
!reflect.DeepEqual(m.Streams[0].Events, []string{"finding-written.*", "finding-cleared.*", "findings-stated.*"}) {
|
||||
t.Errorf("the finding stream is %+v, want node-tools-findings with the three finding events (design 47 G §387)", m.Streams)
|
||||
}
|
||||
// A wait is no consumer (ADR 0367 §2): what it consumes is the assigner's two views families (ADR 0615 §5, issue 1966
|
||||
// TLS-5), the events of the catalog half's branch follower (design 47 G §196, §614: `catalog-revision`,
|
||||
// `registration-changed`, and the gate's `source-moved`; never the indexer's `views.modules` or `views.manifests`), the two
|
||||
// events of the seat that serves the seat list (ADR 0636 §4) and the keeper's `findings-asked`, and nothing else.
|
||||
wantConsumes := []string{"mesh-catalog.catalog-revision", "mesh-catalog.registration-changed", "mesh-catalog.seat-definition-changed",
|
||||
"mesh-catalog.seat-definition-removed", "mesh-conditions.findings-asked", "seat:mesh-module-assigner.views.assignments",
|
||||
"seat:mesh-module-assigner.views.seat-holders", "mesh-delivery.source-moved"}
|
||||
if !reflect.DeepEqual(m.Consumes, wantConsumes) {
|
||||
t.Errorf("the tool runner consumes %v, want %v", m.Consumes, wantConsumes)
|
||||
}
|
||||
}
|
||||
|
||||
// issue 1966 TLS-1: the tool runner declares its own secret `ledgers`, each module's ledger credential, in the directory it
|
||||
// keeps its broker credential in, which is where the runner reads it (runtime.LedgersFileOf).
|
||||
func TestTheManifestDeclaresTheRunnersOwnSecretLedgers(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m struct {
|
||||
Module string `json:"module"`
|
||||
Waits []string `json:"waits"`
|
||||
Consumes []string `json:"consumes"`
|
||||
OwnSecrets map[string]string `json:"own-secrets"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{"mesh-knowledge.reviewed", "mesh-delivery-gate.transition", "gitea.forge.status",
|
||||
"mesh-controller.checked", "operator-channel.decided.claude-code"}
|
||||
if m.Module != "node-tools" || !reflect.DeepEqual(m.Waits, want) {
|
||||
t.Errorf("%s waits on %v, want %v", m.Module, m.Waits, want)
|
||||
}
|
||||
for _, w := range m.Waits {
|
||||
if strings.ContainsAny(w, "*>") {
|
||||
t.Errorf("a family named by a wildcard: %s", w)
|
||||
}
|
||||
}
|
||||
if len(m.Consumes) > 0 {
|
||||
t.Errorf("the tool runner consumes %v: a wait is no consumer (ADR 0367 §2)", m.Consumes)
|
||||
want := map[string]string{"broker": "${dir:mesh-state}/broker", "ledgers": "${dir:mesh-state}/ledgers"}
|
||||
if !reflect.DeepEqual(m.OwnSecrets, want) {
|
||||
t.Errorf("own secrets %v, want %v", m.OwnSecrets, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,6 +6,8 @@ package console
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -45,6 +47,7 @@ type RPCError struct {
|
||||
// Surface answers MCP requests over one bus connection, as one account.
|
||||
type Surface struct {
|
||||
conn *bus.Conn
|
||||
src Source
|
||||
who string
|
||||
mu sync.Mutex
|
||||
known *Listing
|
||||
@@ -58,11 +61,43 @@ type Surface struct {
|
||||
// Flat announces the whole catalogue, as the console did before ADR 0195: for a person reading
|
||||
// it or a client that wants it. Off by default; MESH_CONSOLE_FLAT=1 turns it on.
|
||||
Flat bool
|
||||
// Record, when set, records a hand act in place of the seat mesh-hand-acts' verb (a test's).
|
||||
Record func(args map[string]any) error
|
||||
// Approvals is this machine's approval gate, read by mesh_approvals and mesh_describe; nil when the runner has none.
|
||||
Approvals Approvals
|
||||
}
|
||||
|
||||
// MethodSeats is the console's JSON-RPC method beside MCP's that answers every seat's verbs and holders, in the shape
|
||||
// the old `tools` answered: `{seats: [{seat, scope, tools: [{name, description, input, replaces}], holders:
|
||||
// [{module, node}]}], holders_why}`.
|
||||
const MethodSeats = "mesh/seats"
|
||||
|
||||
// seatsAnswer is MethodSeats' answer: mesh-sdk's seat definitions, with each seat's holders by assignment.
|
||||
func (s *Surface) seatsAnswer() map[string]any {
|
||||
placed, err := seatHoldersOn(s.src)
|
||||
x := &index{Placed: placed}
|
||||
seats := []map[string]any{}
|
||||
for _, st := range catalogue.Seats() {
|
||||
holders := x.placedHolders(st.Name, "")
|
||||
if holders == nil {
|
||||
holders = []meshmoduleassigner.SeatHoldersHoldersItem{}
|
||||
}
|
||||
tools := st.Serves
|
||||
if tools == nil {
|
||||
tools = []catalogue.Verb{}
|
||||
}
|
||||
seats = append(seats, map[string]any{"seat": st.Name, "scope": st.Scope, "tools": tools, "holders": holders})
|
||||
}
|
||||
out := map[string]any{"seats": seats}
|
||||
if err != nil {
|
||||
out["holders_why"] = "who holds which seat by assignment could not be read: " + err.Error()
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// NewSurface is the surface over a connection, as `who`.
|
||||
func NewSurface(conn *bus.Conn, who string) *Surface {
|
||||
return &Surface{conn: conn, who: who, Flat: os.Getenv("MESH_CONSOLE_FLAT") == "1"}
|
||||
func NewSurface(conn *bus.Conn, src Source, who string) *Surface {
|
||||
return &Surface{conn: conn, src: src, who: who, Flat: os.Getenv("MESH_CONSOLE_FLAT") == "1"}
|
||||
}
|
||||
|
||||
func (s *Surface) listing() (*Listing, error) {
|
||||
@@ -73,7 +108,7 @@ func (s *Surface) listing() (*Listing, error) {
|
||||
return l, nil
|
||||
}
|
||||
s.mu.Unlock()
|
||||
l, err := toolsOn(s.conn)
|
||||
l, err := toolsOn(s.conn, s.src)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -125,6 +160,11 @@ func (s *Surface) HandleContext(ctx context.Context, r Request) *Reply {
|
||||
return nil
|
||||
}
|
||||
return answer(r.ID, map[string]any{})
|
||||
case MethodSeats:
|
||||
// The seats' contract and who holds each, for a client on this machine that has no seat definitions of its own
|
||||
// (the TypeScript `mesh`): read in-process from mesh-sdk and from the assigner, never asked of anything else
|
||||
// (novox/hq ADR 0571 clauses 4 and 5).
|
||||
return answer(r.ID, s.seatsAnswer())
|
||||
case "tools/list":
|
||||
if !s.Flat {
|
||||
return answer(r.ID, map[string]any{"tools": discovery()})
|
||||
@@ -303,7 +343,7 @@ func (s *Surface) instructions() string {
|
||||
"that serves it; what may be called was fixed when this account was issued, so a " +
|
||||
"refusal means the account, not the tool. The list is what the running modules " +
|
||||
"answered, plus every role's tools from the mesh's records — the mesh's own verbs " +
|
||||
"(mesh-controller.status, .push, .assign …) among them; a module that did not answer " +
|
||||
"(mesh-delivery.deliveries, mesh-module-assigner.propose …) among them; a module that did not answer " +
|
||||
"is named in the list's _meta and can still be called by <module>.<tool>."
|
||||
}
|
||||
return "The tools of a Novox mesh, reached as " + s.who + ", found by address rather than listed " +
|
||||
|
||||
@@ -11,9 +11,9 @@ import (
|
||||
// 2026-10-05 the controller's push was described as taking nothing, and a push naming one machine
|
||||
// ran as a push of every machine behind.
|
||||
func TestAMeshSeatsNodeIsItsOwnArgument(t *testing.T) {
|
||||
push := Tool{Module: "mesh-controller", Name: "push", Seat: true, Scope: "mesh",
|
||||
push := Tool{Module: "mesh-module-assigner", Name: "push", Seat: true, Scope: "mesh",
|
||||
Input: json.RawMessage(`{"type":"object","properties":{"node":{"type":"string"},"behind":{"type":"string"}}}`)}
|
||||
mesh := target{Address: "mesh-controller.push", Tool: push, Seat: true}
|
||||
mesh := target{Address: "mesh-module-assigner.push", Tool: push, Seat: true}
|
||||
|
||||
props := schemaAsPassed(mesh)["properties"].(map[string]any)
|
||||
if _, has := props["node"]; !has {
|
||||
@@ -44,6 +44,27 @@ func TestAMeshSeatsNodeIsItsOwnArgument(t *testing.T) {
|
||||
t.Fatalf("another machine in the arguments than in the address was dropped: %v", err)
|
||||
}
|
||||
|
||||
// Issue 1966: a seat verb its holder announced with an empty schema is judged by the definition this runner holds;
|
||||
// where it holds none (it waits for the seat list, holding the bootstrapped ones), the arguments go to the holder,
|
||||
// which judges them, and mesh_describe says the definition is not held.
|
||||
empty := json.RawMessage(`{"type":"object","properties":{}}`)
|
||||
propose := target{Address: "mesh-module-assigner.propose", Seat: true,
|
||||
Tool: Tool{Module: "mesh-module-assigner", Name: "propose", Seat: true, Scope: "mesh", Input: empty}}
|
||||
if sent, err := argumentsAsSent(propose, map[string]any{"changes": []any{}, "why": "w"}); err != nil || sent["why"] != "w" {
|
||||
t.Fatalf("a held definition's arguments were refused: %v %v", sent, err)
|
||||
}
|
||||
if _, err := argumentsAsSent(propose, map[string]any{"chnages": 1}); err == nil || !strings.Contains(err.Error(), "chnages") {
|
||||
t.Fatalf("an argument the held definition does not declare was sent: %v", err)
|
||||
}
|
||||
unheld := target{Address: "mesh-not-yet-read.act", Seat: true,
|
||||
Tool: Tool{Module: "mesh-not-yet-read", Name: "act", Seat: true, Scope: "mesh", Input: empty}}
|
||||
if sent, err := argumentsAsSent(unheld, map[string]any{"changes": 1}); err != nil || sent["changes"] != 1 {
|
||||
t.Fatalf("a seat whose definition is not held refused its arguments: %v %v", sent, err)
|
||||
}
|
||||
if _, held := seatDefinitionHeld(unheld); held || !strings.Contains(notHeld("mesh-not-yet-read"), "no definition of mesh-not-yet-read") {
|
||||
t.Fatal("a definition not held is said to be held")
|
||||
}
|
||||
|
||||
// A module's tool is its own: what it takes beyond node is the module's to judge.
|
||||
beta := target{Address: "desk/beta.three", Node: "desk", Tool: Tool{Module: "beta", Name: "three"}}
|
||||
if sent, err := argumentsAsSent(beta, map[string]any{"node": "desk", "verbose": true}); err != nil || sent["verbose"] != true || sent["node"] != nil {
|
||||
|
||||
@@ -11,18 +11,33 @@ import (
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
|
||||
// A tool's own error that mentions 503 is the tool's answer, not a routing fault (issue 1529).
|
||||
func TestAToolsOwn503IsNotNothingServes(t *testing.T) {
|
||||
// A marked tool or verb that reaches no responder names withheld and the grants part's read (design 46 7-approval-mark §6);
|
||||
// an unmarked one, or a marked one failing otherwise, does not. A tool's own error that mentions 503 is the tool's answer, not a routing fault (issue 1529); a real no-responders still says
|
||||
// nothing serves.
|
||||
func TestARealNoRespondersStillSaysNothingServes(t *testing.T) {
|
||||
got := whyItFailed("scanner.scan", errors.New("the scanner refused the scan job: 503 Service Unavailable"))
|
||||
if strings.Contains(got, "nothing serves") || !strings.HasPrefix(got, "scanner.scan failed: ") || !strings.Contains(got, "503") {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARealNoRespondersStillSaysNothingServes(t *testing.T) {
|
||||
for _, err := range []error{bus.ErrNoResponders, fmt.Errorf("ask: %w", bus.ErrNoResponders), nats.ErrNoResponders} {
|
||||
if got := whyItFailed("a.b", err); !strings.HasPrefix(got, "nothing serves a.b") {
|
||||
t.Fatalf("%v: got %q", err, got)
|
||||
}
|
||||
}
|
||||
s := &Surface{Approvals: &fakeApprovals{marked: "shell.wipe"}}
|
||||
for _, c := range []struct {
|
||||
target target
|
||||
err error
|
||||
hinted bool
|
||||
}{
|
||||
{target{Address: "shell.wipe", Tool: Tool{Module: "shell", Name: "wipe"}}, bus.ErrNoResponders, true},
|
||||
{target{Address: "node-login-shell.execute", Seat: true, Tool: Tool{Module: "node-login-shell", Name: "execute"}}, bus.ErrNoResponders, false},
|
||||
{target{Address: "shell.tidy", Tool: Tool{Module: "shell", Name: "tidy"}}, bus.ErrNoResponders, false},
|
||||
{target{Address: "shell.wipe", Tool: Tool{Module: "shell", Name: "wipe"}}, errors.New("its own failure"), false},
|
||||
} {
|
||||
got := s.callFailed(c.target, c.err)
|
||||
if hinted := strings.Contains(got, "approval-mark/1: mesh-grants.grants_user_list names it under withheld"); hinted != c.hinted {
|
||||
t.Errorf("%s (%v): %q", c.target.Address, c.err, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,9 +1,18 @@
|
||||
package console
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/stateobjects"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// A refusal is said only for the bus's own refusals; a tool's answer that mentions authorization is the
|
||||
@@ -19,19 +28,71 @@ func TestARefusalIsSaidOnlyForTheBussOwn(t *testing.T) {
|
||||
t.Errorf("%q read as a refusal: %v, want %v", msg, got, refusal)
|
||||
}
|
||||
}
|
||||
// ADR 0666 §16: an answer a module sent as a reference is read from its sent store for the person asking; one the bus does
|
||||
// not let the runner read is shown as its reference with its size and why it was not read.
|
||||
t.Run("AnAnswerSentAsAReferenceIsReadForThePersonAsking", func(t *testing.T) {
|
||||
nc, err := nats.Connect(meshtest.URL(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer nc.Close()
|
||||
js, _ := jetstream.New(nc)
|
||||
obs, err := js.CreateObjectStore(context.Background(), jetstream.ObjectStoreConfig{Bucket: "m_sent-objects"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn, err := bus.Connect(bus.Credential{URL: meshtest.URL(t), Module: "console", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
body := []byte(`{"lines":"` + strings.Repeat("x", 9<<20) + `"}`)
|
||||
sum := sha256.Sum256(body)
|
||||
ref := stateobjects.Ref{Store: "m_sent-objects", Object: "tool.lines." + hex.EncodeToString(sum[:]), Size: int64(len(body)),
|
||||
Digest: hex.EncodeToString(sum[:]), Form: stateobjects.FormVersion}
|
||||
if _, err := obs.PutBytes(context.Background(), ref.Object, body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, unread := resolveSent(conn, stateobjects.Wrap(ref)); unread != "" || string(got) != string(body) {
|
||||
t.Fatalf("read %d bytes of %d, %q", len(got), len(body), unread)
|
||||
}
|
||||
missing := ref
|
||||
missing.Store = "gone_sent-objects"
|
||||
wrapped := stateobjects.Wrap(missing)
|
||||
got, unread := resolveSent(conn, wrapped)
|
||||
if string(got) != string(wrapped) || !strings.Contains(unread, "9437") || !strings.Contains(unread, "gone_sent-objects") {
|
||||
t.Fatalf("an unread answer showed %q and %q", got, unread)
|
||||
}
|
||||
if got, unread := resolveSent(conn, []byte(`{"a":1}`)); unread != "" || string(got) != `{"a":1}` {
|
||||
t.Fatalf("a plain answer showed %q, %q", got, unread)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// **A timeout says it is not a failure** (novox/hq issue 265): a push that outlasted the wait had
|
||||
// pushed, and the caller read "did not answer in time". And from the controller, where its answer is.
|
||||
// pushed, and the caller read "did not answer in time". No hint names a seat that is gone (novox/hq issue 1126):
|
||||
// the grammar's example, the handshake's instructions and a timeout's remedy named the controller's verbs.
|
||||
func TestATimeoutIsNotAFailure(t *testing.T) {
|
||||
got := whyItFailed("seat:mesh-controller.push", errors.New("timeout"))
|
||||
for _, want := range []string{"not a failure", "may already have done", "mesh-controller.calls"} {
|
||||
got := whyItFailed("seat:mesh-delivery.deliveries", errors.New("timeout"))
|
||||
for _, want := range []string{"not a failure", "may already have done"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("a controller timeout does not say %q: %s", want, got)
|
||||
t.Errorf("a seat's timeout does not say %q: %s", want, got)
|
||||
}
|
||||
}
|
||||
if other := whyItFailed("novox/postgres.postgres_list_databases", errors.New("timeout")); !strings.Contains(other, "not a failure") ||
|
||||
strings.Contains(other, "calls") {
|
||||
t.Errorf("a module's timeout: %s", other)
|
||||
for name, text := range map[string]string{
|
||||
"grammar": grammar,
|
||||
"flat instructions": (&Surface{Flat: true, who: "x"}).instructions(),
|
||||
"deep instructions": (&Surface{who: "x"}).instructions(),
|
||||
"timeout": got,
|
||||
} {
|
||||
if strings.Contains(text, "controller") {
|
||||
t.Errorf("%s names the controller: %s", name, text)
|
||||
}
|
||||
// The seats were renamed (ADR 0634 §9): no hint names the seat that placed modules or enrolled a machine by its old name.
|
||||
for _, old := range []string{"mesh-placement", "mesh-enrolment"} {
|
||||
if strings.Contains(text, old) {
|
||||
t.Errorf("%s names the seat %s, which is now mesh-module-assigner or mesh-node-enroller: %s", name, old, text)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,14 +2,19 @@ package console
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
assigner "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
|
||||
seatassigner "git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
|
||||
"github.com/nats-io/nats.go/micro"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/announce"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
@@ -52,7 +57,25 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
|
||||
// Kept for the test: one round of discovery answers every question but mesh_runtimes, which asks anew.
|
||||
IndexKept, announce.Patience = time.Minute, 2500*time.Millisecond
|
||||
t.Cleanup(func() { IndexKept, announce.Patience = keptWas, patienceWas })
|
||||
mt.New(t)
|
||||
mesh := mt.New(t)
|
||||
// What the mesh holds, as the indexer and the assigner write it (ADR 0571): never the controller's records.
|
||||
mesh.Placed(t, map[string][]string{"slack": {"slack_check"}, "nap": {"doze"}, "nextcloud-client": {"nextcloud_check"}},
|
||||
map[string][]string{"laptop": {"slack", "nextcloud-client"}, "sleeper": {"nap", "nextcloud-client"},
|
||||
"desktop": {"nextcloud-client"}, "bench": nil})
|
||||
// Who holds which seat, by assignment (ADR 0571 clause 4), as the seat mesh-module-assigner's holder composes it.
|
||||
placedKV := mesh.Bucket(t, meshtest.SeatHoldersBucket)
|
||||
holdersKey, err := assigner.SeatHoldersKey("mesh-wants", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, _ := json.Marshal(seatassigner.SeatHolders{V: assigner.Version, Seat: "mesh-wants", Scope: "mesh",
|
||||
Holders: []seatassigner.SeatHoldersHoldersItem{{Module: "mesh-delivery-assigner", Node: "bench"}}})
|
||||
if _, err := placedKV.Put(holdersKey, raw); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
consoleMembership := mt.MembershipOf("node-tools", "laptop-console", false, nil)
|
||||
consoleMembership.State = mt.ConsoleReads()
|
||||
mesh.Issue(t, consoleMembership)
|
||||
|
||||
laptop := connect(t, "node-tools", "laptop")
|
||||
stopTool, err := laptop.HandleSubject("mesh.mod.slack.tool.slack_check.laptop", func(json.RawMessage) (any, error) {
|
||||
@@ -76,14 +99,17 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
|
||||
Subject: "mesh.mod.nextcloud-client.tool.nextcloud_check.desktop"},
|
||||
})
|
||||
|
||||
controller := connect(t, "mesh-controller", "bench")
|
||||
controller := connect(t, "mesh-module-assigner", "bench")
|
||||
var toolsAsked atomic.Int32
|
||||
for verb, output := range map[string]string{
|
||||
"nodes": `[{"name":"laptop"},{"name":"sleeper"},{"name":"desktop"},{"name":"bench"}]`,
|
||||
"modules": `[{"module":"slack","on":["laptop"],"tools":true},{"module":"nap","on":["sleeper"],"tools":true},` +
|
||||
`{"module":"nextcloud-client","on":["laptop","desktop","sleeper"],"tools":true}]`,
|
||||
// The controller's records are served here only to see that none is asked (ADR 0571 clauses 4 and 5).
|
||||
"nodes": `[{"name":"laptop"}]`,
|
||||
"modules": `[]`,
|
||||
"tools": `{"seats":[]}`,
|
||||
} {
|
||||
output := output
|
||||
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
|
||||
output, verb := output, verb
|
||||
stop, err := controller.HandleSubject("mesh.seat.mesh-module-assigner.tool."+verb, func(json.RawMessage) (any, error) {
|
||||
toolsAsked.Add(1)
|
||||
return map[string]any{"output": output, "ok": true}, nil
|
||||
})
|
||||
if err != nil {
|
||||
@@ -94,7 +120,14 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
|
||||
controller.Flush()
|
||||
|
||||
console := connect(t, "node-tools", "laptop-console")
|
||||
up, err := Serve(NewSurface(console, "laptop.node-tools"), "127.0.0.1:0")
|
||||
console.Follow("node-tools")
|
||||
mt.Until(t, func() error {
|
||||
if console.Membership("node-tools") == nil {
|
||||
return fmt.Errorf("the console's membership is not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
up, err := Serve(NewSurface(console, busSource{console}, "laptop.node-tools"), "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -113,8 +146,20 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
|
||||
t.Errorf("the late runtime's machine lists nothing: %s", machine)
|
||||
}
|
||||
|
||||
// The seats' contract and holders, for a client on this machine with no seat definitions (the TypeScript `mesh`).
|
||||
seats := post(t, endpoint, `{"jsonrpc":"2.0","id":9,"method":"mesh/seats"}`)
|
||||
if raw, _ := json.Marshal(seats["result"]); !strings.Contains(string(raw), `"holders":[{"module":"mesh-delivery-assigner","node":"bench"}],"scope":"mesh","seat":"mesh-wants"`) ||
|
||||
!strings.Contains(string(raw), `"replaces":["journalctl"]`) {
|
||||
t.Errorf("mesh/seats answered %s", raw)
|
||||
}
|
||||
if n := toolsAsked.Load(); n != 0 {
|
||||
t.Errorf("the console asked the controller's records %d times", n)
|
||||
}
|
||||
|
||||
// The silent one is named, never called missing.
|
||||
for address, want := range map[string]string{
|
||||
// A seat whose holder announced nothing is named with its holder by assignment, never called missing.
|
||||
"mesh-wants.read": "the seat mesh-wants is held by mesh-delivery-assigner on bench by assignment",
|
||||
"sleeper/nap.doze": "nap is assigned to sleeper in the mesh's records, but nothing that answered discovery serves it: discovery did not hear in full from node-tools on sleeper (it answered PING",
|
||||
"sleeper/nextcloud-client.nextcloud_check": "nextcloud-client on sleeper did not answer discovery: discovery did not hear in full from node-tools on sleeper",
|
||||
} {
|
||||
|
||||
@@ -19,7 +19,8 @@ package console
|
||||
// Ties go to a seat — the place tools belong — then to the shorter name, then by address.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
@@ -265,27 +266,16 @@ func (x *index) search(query string) []hit {
|
||||
return hits
|
||||
}
|
||||
|
||||
// replacesOf is what every verb and tool replaces, from the controller's records (novox/hq ADR 0245):
|
||||
// a seat's verbs from its `tools` answer, keyed `seat:<seat>.<verb>`, and a module's own tools from
|
||||
// its `modules` answer, keyed `<module>.<tool>`. A controller older than the field answers neither,
|
||||
// and search matches names and descriptions as before.
|
||||
func replacesOf(toolsAnswer json.RawMessage, modules []recordedModule) map[string][]string {
|
||||
// replacesOf is what every verb and tool replaces (novox/hq ADR 0245): a seat's verbs from the seat's contract in
|
||||
// mesh-sdk, keyed `seat:<seat>.<verb>`, read in-process and asked of no part (ADR 0571 clause 5: no part keeps a
|
||||
// central record of tools); and a module's own tools from its records, keyed `<module>.<tool>`. A verb or a tool with
|
||||
// no `replaces` is matched by its name and description as before.
|
||||
func replacesOf(seats []catalogue.Seat, modules []recordedModule) map[string][]string {
|
||||
out := map[string][]string{}
|
||||
var seats struct {
|
||||
Seats []struct {
|
||||
Seat string `json:"seat"`
|
||||
Tools []struct {
|
||||
Name string `json:"name"`
|
||||
Replaces []string `json:"replaces"`
|
||||
} `json:"tools"`
|
||||
} `json:"seats"`
|
||||
}
|
||||
if json.Unmarshal(toolsAnswer, &seats) == nil {
|
||||
for _, s := range seats.Seats {
|
||||
for _, t := range s.Tools {
|
||||
if len(t.Replaces) > 0 {
|
||||
out["seat:"+s.Seat+"."+t.Name] = t.Replaces
|
||||
}
|
||||
for _, s := range seats {
|
||||
for _, v := range s.Serves {
|
||||
if len(v.Replaces) > 0 {
|
||||
out["seat:"+s.Name+"."+v.Name] = append([]string(nil), v.Replaces...)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
package console
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -19,6 +20,7 @@ func meshToSearch(t *testing.T) *index {
|
||||
"status": "One unit as the service manager sees it now: its states, whether it starts at boot.",
|
||||
"restart": "Restart one unit.",
|
||||
"journal": "The last lines of one unit's journal (at most 2000).",
|
||||
"failed": "The units that failed.",
|
||||
}
|
||||
for _, node := range []string{"anchor", "laptop"} {
|
||||
for verb, does := range verbs {
|
||||
@@ -31,8 +33,8 @@ func meshToSearch(t *testing.T) *index {
|
||||
"node": "What one machine reported it can do, what it is assigned, and why.",
|
||||
"nodes": "Every machine the mesh knows, with whether it is converged or adopted.",
|
||||
} {
|
||||
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller",
|
||||
Tool: verb, Scope: "mesh", Description: does, Subject: "mesh.seat.mesh-controller.tool." + verb})
|
||||
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-module-assigner", Seat: "mesh-module-assigner",
|
||||
Tool: verb, Scope: "mesh", Description: does, Subject: "mesh.seat.mesh-module-assigner.tool." + verb})
|
||||
}
|
||||
for tool, does := range map[string]string{
|
||||
"docker_list": "Every container on this machine: name, image, state, ports.",
|
||||
@@ -44,13 +46,7 @@ func meshToSearch(t *testing.T) *index {
|
||||
Description: does, Subject: "mesh.mod.docker.tool." + tool + ".anchor"})
|
||||
}
|
||||
x, _, _ := indexOfEndpoints(endpoints)
|
||||
tools := json.RawMessage(`{"seats":[{"seat":"node-service-manager","scope":"node","tools":[
|
||||
{"name":"units","replaces":["systemctl list-units","systemctl --failed"]},
|
||||
{"name":"status","replaces":["systemctl status","systemctl is-active"]},
|
||||
{"name":"restart","replaces":["systemctl restart"]},
|
||||
{"name":"journal","replaces":["journalctl"]}]},
|
||||
{"seat":"mesh-controller","scope":"mesh","tools":[{"name":"node","replaces":["hostnamectl","uptime"]},{"name":"nodes"}]}]}`)
|
||||
x.Replaces = replacesOf(tools, []recordedModule{{Module: "docker", On: []string{"anchor"}, Tools: true,
|
||||
x.Replaces = replacesOf(catalogue.Seats(), []recordedModule{{Module: "docker", On: []string{"anchor"}, Tools: true,
|
||||
Replaces: map[string][]string{"docker_list": {"docker ps"}, "docker_logs": {"docker logs"}, "docker_restart": {"docker restart"}}}})
|
||||
return x
|
||||
}
|
||||
@@ -60,22 +56,21 @@ func meshToSearch(t *testing.T) *index {
|
||||
func TestSearchFindsTheToolThatReplacesACommandFirst(t *testing.T) {
|
||||
x := meshToSearch(t)
|
||||
for query, want := range map[string]string{
|
||||
"journalctl": "anchor/node-service-manager.journal",
|
||||
"journalctl -u mesh-controller -n 50": "anchor/node-service-manager.journal",
|
||||
"sudo journalctl -fu sshd": "anchor/node-service-manager.journal",
|
||||
"/usr/bin/journalctl": "anchor/node-service-manager.journal",
|
||||
"logs": "anchor/node-service-manager.journal",
|
||||
"systemctl status": "anchor/node-service-manager.status",
|
||||
"systemctl status nats.service": "anchor/node-service-manager.status",
|
||||
"systemctl restart mesh-controller": "anchor/node-service-manager.restart",
|
||||
"systemctl --failed": "anchor/node-service-manager.units",
|
||||
"docker ps": "anchor/docker.docker_list",
|
||||
"docker ps -a": "anchor/docker.docker_list",
|
||||
"docker logs": "anchor/docker.docker_logs",
|
||||
"docker logs --tail 100 nats": "anchor/docker.docker_logs",
|
||||
"docker restart nats": "anchor/docker.docker_restart",
|
||||
"container logs": "anchor/docker.docker_logs",
|
||||
"uptime": "mesh-controller.node",
|
||||
"journalctl": "anchor/node-service-manager.journal",
|
||||
"journalctl -u mesh-module-assigner -n 50": "anchor/node-service-manager.journal",
|
||||
"sudo journalctl -fu sshd": "anchor/node-service-manager.journal",
|
||||
"/usr/bin/journalctl": "anchor/node-service-manager.journal",
|
||||
"logs": "anchor/node-service-manager.journal",
|
||||
"systemctl status": "anchor/node-service-manager.status",
|
||||
"systemctl status nats.service": "anchor/node-service-manager.status",
|
||||
"systemctl restart mesh-module-assigner": "anchor/node-service-manager.restart",
|
||||
"systemctl --failed": "anchor/node-service-manager.failed",
|
||||
"docker ps": "anchor/docker.docker_list",
|
||||
"docker ps -a": "anchor/docker.docker_list",
|
||||
"docker logs": "anchor/docker.docker_logs",
|
||||
"docker logs --tail 100 nats": "anchor/docker.docker_logs",
|
||||
"docker restart nats": "anchor/docker.docker_restart",
|
||||
"container logs": "anchor/docker.docker_logs",
|
||||
} {
|
||||
hits := x.search(query)
|
||||
if len(hits) == 0 {
|
||||
@@ -116,7 +111,7 @@ func TestASearchByCommandSaysWhatItReplaces(t *testing.T) {
|
||||
// A synonym is a word, never part of one: `logs` matches through `log`, and finds no `login`.
|
||||
func TestSearchStillMatchesEveryWord(t *testing.T) {
|
||||
x := meshToSearch(t)
|
||||
if hits := x.search("machine converged"); len(hits) != 1 || hits[0].Address != "mesh-controller.nodes" {
|
||||
if hits := x.search("machine converged"); len(hits) != 1 || hits[0].Address != "mesh-module-assigner.nodes" {
|
||||
t.Errorf("words in a description: %+v", hits)
|
||||
}
|
||||
if hits := x.search("journalctl kubernetes"); len(hits) != 1 || hits[0].Address != "anchor/node-service-manager.journal" {
|
||||
@@ -134,10 +129,10 @@ func TestSearchStillMatchesEveryWord(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A controller older than `replaces` answers without it: search matches names and descriptions as before.
|
||||
// A seat's contract and a module's records without `replaces`: search matches names and descriptions as before.
|
||||
func TestSearchWithoutReplacesMatchesAsBefore(t *testing.T) {
|
||||
x := meshToSearch(t)
|
||||
x.Replaces = replacesOf(json.RawMessage(`{"seats":[{"seat":"node-service-manager","tools":[{"name":"journal"}]}]}`),
|
||||
x.Replaces = replacesOf([]catalogue.Seat{{Name: "node-service-manager", Serves: []catalogue.Verb{{Name: "journal"}}}},
|
||||
[]recordedModule{{Module: "docker"}})
|
||||
if len(x.Replaces) != 0 {
|
||||
t.Fatalf("replaces from records that have none: %v", x.Replaces)
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
package console
|
||||
|
||||
import (
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatlist/seatlisttest"
|
||||
)
|
||||
|
||||
// The seats the tests describe are given to catalogue as the tool runner gives the ones it read from the bus
|
||||
// (novox/hq ADR 0635 §1); catalogue compiles none.
|
||||
func init() { catalogue.UseSeats(catalogue.SeatsFromDefinitions(seatlisttest.Definitions())) }
|
||||
@@ -0,0 +1,31 @@
|
||||
package console
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/stateobjects"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
|
||||
// resolveSent is the answer a person reads (ADR 0666 §16): an answer a module sent as a reference to an object of its
|
||||
// sent store is read from it when the bus grants this runner's account the read (the standing grant the manifests give
|
||||
// the callers of the verb), else shown as its reference with its size and why it was not read.
|
||||
func resolveSent(conn *bus.Conn, result json.RawMessage) (json.RawMessage, string) {
|
||||
ref, ok := stateobjects.ReadRef(result)
|
||||
if !ok {
|
||||
return result, ""
|
||||
}
|
||||
whole, err := stateobjects.Receive(result, func(r stateobjects.Ref) ([]byte, error) {
|
||||
got, err := conn.SentGet(buscalls.ObjectsSentGetRequest{Store: r.Store, Object: r.Object, Size: r.Size, Digest: r.Digest})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return got.Data, nil
|
||||
})
|
||||
if err != nil {
|
||||
return result, fmt.Sprintf("the answer is %d bytes, held as the object %s of %s, and was not read: %v", ref.Size, ref.Object, ref.Store, err)
|
||||
}
|
||||
return whole, ""
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package console
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"sync"
|
||||
|
||||
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// busSource is a Source over a bus connection's own states, which a test's fixture mesh fills: the console's tests stand a
|
||||
// membership that reads the four states beside it, where the tool runner itself holds first views (firstview.Views).
|
||||
type busSource struct{ conn *bus.Conn }
|
||||
|
||||
func (b busSource) State(name string) (map[string][]byte, error) {
|
||||
var mu sync.Mutex
|
||||
keys := map[string][]byte{}
|
||||
stop, err := b.conn.StateWatch(b.conn.Credential().Module, name, "", func(c bus.StateChange) error {
|
||||
if c.Current && c.Op == "put" {
|
||||
mu.Lock()
|
||||
keys[c.Key] = c.Value
|
||||
mu.Unlock()
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
stop()
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return keys, nil
|
||||
}
|
||||
|
||||
func (b busSource) Entry(name, key string) ([]byte, error) {
|
||||
entry, err := b.conn.StateGet(b.conn.Credential().Module, name, key)
|
||||
if err != nil || entry == nil {
|
||||
return nil, err
|
||||
}
|
||||
return entry.Value, nil
|
||||
}
|
||||
|
||||
// Modules reads the test bed's entries of the catalog half (meshtest.ModulesRead), as the tool runner's first views hold them.
|
||||
func (b busSource) Modules() (map[string]meshcatalog.CatalogEntry, error) {
|
||||
raw, err := b.State(mt.ModulesRead)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := map[string]meshcatalog.CatalogEntry{}
|
||||
for module, value := range raw {
|
||||
var e meshcatalog.CatalogEntry
|
||||
if err := json.Unmarshal(value, &e); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[module] = e
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -2,9 +2,9 @@ package console
|
||||
|
||||
// mesh_wait: a session waits on the bus for one event instead of polling (novox/hq ADR 0367).
|
||||
//
|
||||
// **Only the families the tool runner was issued** (§2, §6): its membership names them, composed by the
|
||||
// controller from the tool runner's manifest together with the grant that lets this account subscribe them.
|
||||
// A name not listed there is refused here, before the bus would refuse it; no argument or setting adds one.
|
||||
// **Any event, by its name** (novox/hq ADR 0670 §1, ADR 0671 §1; design 34 every-event §13, §26): the grants part
|
||||
// grants this machine's tool runner every event subject, a seat's, a module's and a module stream's, and a name
|
||||
// becomes its subjects through mesh-sdk's buscompose.EventSubjects; no membership and no manifest names a list.
|
||||
// **An event only wakes the session** (§4): what it says is read again from the verb that owns it.
|
||||
|
||||
import (
|
||||
@@ -13,9 +13,15 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"sort"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
|
||||
"git.novox.be/novox/mesh-sdk/go/buscompose"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
)
|
||||
|
||||
// WaitDefault is how long a wait lasts when no timeout is given, and WaitLongest the most it may last
|
||||
@@ -25,20 +31,29 @@ var (
|
||||
WaitLongest = 10 * time.Minute
|
||||
)
|
||||
|
||||
// waitMostNames is the most event names one call gives (design 34 every-event §13, §26): each name is six
|
||||
// subscriptions for the call, so 16 names are 96, far above what a watcher names.
|
||||
const waitMostNames = 16
|
||||
|
||||
func waitTool() map[string]any {
|
||||
return map[string]any{"name": verbWait, "inputSchema": map[string]any{"type": "object", "properties": map[string]any{
|
||||
"subjects": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
|
||||
"description": "one or more event families to wait on, by name, e.g. `mesh-knowledge.reviewed`; only " +
|
||||
"those the mesh granted this machine's tool runner — a refusal names them"},
|
||||
"description": fmt.Sprintf("one or more event names, `<seat or module>.<event>`, such as "+
|
||||
"`mesh-delivery-view.delivered`, at most %d; an event whose subject goes on after the name (an asker, a "+
|
||||
"kind, a module) is heard by the name too", waitMostNames)},
|
||||
"from": map[string]any{"type": "string", "enum": []string{"seat", "module", "stream"},
|
||||
"description": "narrows every name to a seat's events, a module's or a module stream's, for an emitter " +
|
||||
"whose name is more than one of them (optional)"},
|
||||
"filter": map[string]any{"type": "object",
|
||||
"description": "fields of the event's body that must equal the given values, e.g. {\"kind\": \"decision\", \"number\": 367} (optional)"},
|
||||
"description": "fields of the event's body that must equal the given values, e.g. {\"kind\": \"decision\", \"number\": 367}: the filter narrows by the body's fields (optional)"},
|
||||
"timeout": map[string]any{"type": "number",
|
||||
"description": fmt.Sprintf("seconds to wait: %d when absent, %d at most", int(WaitDefault.Seconds()), int(WaitLongest.Seconds()))},
|
||||
}, "required": []string{"subjects"}},
|
||||
"description": "Wait for the first event of the named families that matches the filter, and answer it with its " +
|
||||
"family, body and time — or say it timed out. Call it in a loop from a background command, and after every " +
|
||||
"answer read the state from the verb that owns it: an event only wakes you, and one published between two " +
|
||||
"waits is not heard (novox/hq ADR 0367)."}
|
||||
"description": "Wait for the first event of the named events that matches the filter, and answer it with its " +
|
||||
"name, where it came from, body and times — or say it timed out. Call it in a loop from a background command, " +
|
||||
"and after every answer read the state from the verb that owns it: an event only wakes you, and one " +
|
||||
"published between two waits is not heard (novox/hq ADR 0367). An operator's answer is read only with " +
|
||||
"claude_code_answer, never from the event."}
|
||||
}
|
||||
|
||||
// waitAnswer answers one mesh_wait call.
|
||||
@@ -56,51 +71,195 @@ func (s *Surface) waitAnswer(ctx context.Context, args map[string]any) map[strin
|
||||
return failure("mesh_wait's filter is an object of body fields to the values they must equal")
|
||||
}
|
||||
|
||||
issued := map[string]string{} // name → subject
|
||||
if m := s.conn.Membership(s.conn.Module()); m != nil {
|
||||
for _, w := range m.Waits {
|
||||
issued[w.Event] = w.Subject
|
||||
}
|
||||
from, _ := args["from"].(string)
|
||||
if args["from"] != nil && from != "seat" && from != "module" && from != "stream" {
|
||||
return failure(fmt.Sprintf("mesh_wait's from is seat, module or stream, not %v", args["from"]))
|
||||
}
|
||||
if len(issued) == 0 {
|
||||
return failure("this machine's tool runner was issued no event families to wait on: the mesh grants them from " +
|
||||
"the tool runner's manifest (novox/hq ADR 0367 §2), and its membership names none yet")
|
||||
if len(names) > waitMostNames {
|
||||
return failure(fmt.Sprintf("mesh_wait names at most %d events in one call, not %d: each name is six "+
|
||||
"subscriptions for the call (design 34 every-event §13)", waitMostNames, len(names)))
|
||||
}
|
||||
byName := make([]string, 0, len(issued))
|
||||
for name := range issued {
|
||||
byName = append(byName, name)
|
||||
}
|
||||
sort.Strings(byName)
|
||||
family := map[string]string{} // subject → name
|
||||
var subjects []string
|
||||
known, unchecked := s.emitters()
|
||||
var subjects, undeclared []string
|
||||
for _, name := range names {
|
||||
subject, granted := issued[name]
|
||||
if !granted {
|
||||
return failure(fmt.Sprintf("mesh_wait may not wait on %q: this machine's tool runner is granted %s alone. "+
|
||||
"A family is added only by a reviewed change to the tool runner's manifest (node-tools' module.json, "+
|
||||
"novox/hq ADR 0367 §6); no argument, call or setting adds one", name, strings.Join(byName, ", ")))
|
||||
subs, err := buscompose.EventSubjects(name, from)
|
||||
if err != nil {
|
||||
return failure("mesh_wait may not wait on it: " + err.Error())
|
||||
}
|
||||
if _, twice := family[subject]; !twice {
|
||||
subjects = append(subjects, subject)
|
||||
// A misspelt emitter is refused, never waited on for ever; a known emitter's undeclared event is waited on and
|
||||
// its timed-out answer says so (design 34 every-event §14).
|
||||
if unchecked == "" {
|
||||
emitter, event, _ := strings.Cut(name, ".")
|
||||
seatEvents, isSeat := known.seats[emitter]
|
||||
moduleEvents, isModule := known.modules[emitter]
|
||||
isSeat = isSeat && from != "module" && from != "stream"
|
||||
isModule = isModule && from != "seat"
|
||||
if !isSeat && !isModule {
|
||||
return failure(fmt.Sprintf("mesh_wait may not wait on %q: %s is neither a seat nor a module of %s",
|
||||
name, emitter, known.source))
|
||||
}
|
||||
if !(isSeat && declares(seatEvents, event)) && !(isModule && declares(moduleEvents, event)) {
|
||||
undeclared = append(undeclared, name)
|
||||
}
|
||||
}
|
||||
for _, subject := range subs {
|
||||
if !slices.Contains(subjects, subject) {
|
||||
subjects = append(subjects, subject)
|
||||
}
|
||||
}
|
||||
family[subject] = name
|
||||
}
|
||||
|
||||
heard, ok, err := s.conn.HearFirst(ctx, subjects, timeout, func(body json.RawMessage) bool { return bodyMatches(body, filter) })
|
||||
listen := func(within time.Duration) (any, error) {
|
||||
return s.hear(ctx, names, subjects, undeclared, known.source, unchecked, filter, within)
|
||||
}
|
||||
// A wait whose subjects overlap a marked event's runs only on the operator's warrant for that one wait (design 34
|
||||
// every-event §34 to §36), judged by the runner's approval gate.
|
||||
var answer any
|
||||
if g, ok := s.Approvals.(waitGate); ok {
|
||||
given := map[string]any{}
|
||||
for _, k := range []string{"subjects", "from", "filter", "timeout"} {
|
||||
if v, has := args[k]; has {
|
||||
given[k] = v
|
||||
}
|
||||
}
|
||||
raw, _ := json.Marshal(given)
|
||||
answer, err = g.Waited(ctx, approval.Call{Address: verbWait, Name: verbWait, Caller: s.who, Args: raw},
|
||||
approval.Wait{Names: names, From: from, Subjects: subjects, Filter: filter, Timeout: timeout}, listen)
|
||||
} else {
|
||||
answer, err = listen(timeout)
|
||||
}
|
||||
if err != nil {
|
||||
return failure("mesh_wait could not wait: " + err.Error())
|
||||
return failure(err.Error())
|
||||
}
|
||||
if !ok {
|
||||
return answerText(map[string]any{"timed-out": true, "waited": timeout.String(), "subjects": names,
|
||||
"read": "nothing matching was heard; read the state once from the verb that owns it, then wait again"})
|
||||
}
|
||||
return answerText(map[string]any{
|
||||
"heard": map[string]any{"subject": family[heard.Subject], "body": heard.Body, "at": heard.At.Format(time.RFC3339Nano)},
|
||||
"read": "an event only wakes you: read the state from the verb that owns it before acting (novox/hq ADR 0367 §4)",
|
||||
})
|
||||
return answerText(answer)
|
||||
}
|
||||
|
||||
// waitNames is the families a call names: a list of names, or one name.
|
||||
// waitGate is the runner's approval gate as mesh_wait goes through it (design 34 every-event §34 to §36).
|
||||
type waitGate interface {
|
||||
Waited(ctx context.Context, c approval.Call, w approval.Wait, hear func(within time.Duration) (any, error)) (any, error)
|
||||
}
|
||||
|
||||
// hear subscribes the wait's subjects for within and answers the first matching event, or that it timed out.
|
||||
func (s *Surface) hear(ctx context.Context, names, subjects, undeclared []string, source, unchecked string, filter map[string]any,
|
||||
timeout time.Duration) (map[string]any, error) {
|
||||
heard, ok, refused, err := s.conn.HearFirst(ctx, subjects, timeout, func(body json.RawMessage) bool { return bodyMatches(body, filter) })
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("mesh_wait could not wait: %w", err)
|
||||
}
|
||||
var answer map[string]any
|
||||
if !ok {
|
||||
answer = map[string]any{"timed-out": true, "waited": timeout.String(), "subjects": names,
|
||||
"read": "nothing matching was heard; read the state once from the verb that owns it, then wait again"}
|
||||
if len(undeclared) > 0 {
|
||||
answer["undeclared"] = fmt.Sprintf("no declaration names %s: no seat definition and no module's entry in %s "+
|
||||
"declares that event", strings.Join(undeclared, ", "), source)
|
||||
}
|
||||
} else {
|
||||
name, from, _ := buscompose.EventName(heard.Subject)
|
||||
h := map[string]any{"subject": name, "from": from, "body": heard.Body, "at": heard.At.Format(time.RFC3339Nano)}
|
||||
if heard.Published != "" {
|
||||
h["published"] = heard.Published
|
||||
}
|
||||
read := "an event only wakes you: read the state from the verb that owns it before acting (novox/hq ADR 0367 §4)"
|
||||
if strings.HasPrefix(name, "operator-channel.decided") {
|
||||
read = "an event only wakes you: the operator's answer is read only with claude_code_answer, never from " +
|
||||
"the event's body (novox/hq ADR 0670 §5)"
|
||||
}
|
||||
answer = map[string]any{"heard": h, "read": read}
|
||||
}
|
||||
if unchecked != "" {
|
||||
answer["unchecked"] = "the names were not checked: " + unchecked
|
||||
}
|
||||
// A subscription the bus refused while others stood is said (design 34 every-event §16).
|
||||
if len(refused) > 0 {
|
||||
answer["refused"] = refused
|
||||
}
|
||||
return answer, nil
|
||||
}
|
||||
|
||||
// halfReader is a Source that also holds the catalog half of each followed branch (firstview.Views.Halves): mesh_wait
|
||||
// checks an event name's emitter against it (design 34 every-event §14).
|
||||
type halfReader interface {
|
||||
Halves() (map[string]meshcatalogclient.Half, error)
|
||||
}
|
||||
|
||||
// knownEmitters is what the catalog half says emits what: each seat's declared events and each module's, and the
|
||||
// source a refusal names, with its revisions.
|
||||
type knownEmitters struct {
|
||||
seats, modules map[string][]string
|
||||
source string
|
||||
}
|
||||
|
||||
// emitters is the catalog half's emitters, or why they are not checked: the half not yet read (its first view,
|
||||
// mesh_first_views), or no half held at all.
|
||||
func (s *Surface) emitters() (known knownEmitters, unchecked string) {
|
||||
hr, ok := s.src.(halfReader)
|
||||
if !ok {
|
||||
return known, "this tool runner holds no catalog half"
|
||||
}
|
||||
halves, err := hr.Halves()
|
||||
if err != nil {
|
||||
return known, "the catalog half is not yet read (" + err.Error() + "); its first view is in mesh_first_views"
|
||||
}
|
||||
seatEvents := map[string][]string{} // a seat's events, as its definition in the seat list declares them
|
||||
for _, st := range catalogue.Seats() {
|
||||
seatEvents[st.Name] = st.Emits
|
||||
}
|
||||
known.seats, known.modules = map[string][]string{}, map[string][]string{}
|
||||
branches := make([]string, 0, len(halves))
|
||||
for b := range halves {
|
||||
branches = append(branches, b)
|
||||
}
|
||||
slices.Sort(branches)
|
||||
var revisions []string
|
||||
for _, b := range branches {
|
||||
half := halves[b]
|
||||
revisions = append(revisions, fmt.Sprintf("revision %d of %s", half.Revision.Revision, b))
|
||||
for _, d := range half.Revision.Definitions {
|
||||
known.seats[d.Seat] = seatEvents[d.Seat]
|
||||
}
|
||||
for m, entry := range half.Entries {
|
||||
if _, taken := known.modules[m]; !taken {
|
||||
known.modules[m] = eventsDeclared(entry.Manifest)
|
||||
}
|
||||
}
|
||||
}
|
||||
known.source = "the catalog half (mesh-catalog.revision, " + strings.Join(revisions, "; ") + ")"
|
||||
return known, ""
|
||||
}
|
||||
|
||||
// eventsDeclared is the events a module's manifest declares: its `emits` and its streams' `events`.
|
||||
func eventsDeclared(manifest map[string]any) []string {
|
||||
var out []string
|
||||
add := func(list any) {
|
||||
items, _ := list.([]any)
|
||||
for _, x := range items {
|
||||
if e, ok := x.(string); ok {
|
||||
out = append(out, e)
|
||||
}
|
||||
}
|
||||
}
|
||||
add(manifest["emits"])
|
||||
streams, _ := manifest["streams"].([]any)
|
||||
for _, st := range streams {
|
||||
if m, ok := st.(map[string]any); ok {
|
||||
add(m["events"])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// declares says whether one of the declared events is the event a name gives, or one its name is followed by.
|
||||
func declares(declared []string, event string) bool {
|
||||
for _, d := range declared {
|
||||
if buscompose.Overlaps(d, event) || buscompose.Overlaps(d, event+".>") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// waitNames is the event names a call gives: a list of names, or one name.
|
||||
func waitNames(given any) ([]string, error) {
|
||||
var names []string
|
||||
switch v := given.(type) {
|
||||
@@ -110,7 +269,7 @@ func waitNames(given any) ([]string, error) {
|
||||
for _, x := range v {
|
||||
name, ok := x.(string)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("mesh_wait's subjects are event names, such as mesh-knowledge.reviewed; %v is not one", x)
|
||||
return nil, fmt.Errorf("mesh_wait's subjects are event names, such as mesh-delivery-view.delivered; %v is not one", x)
|
||||
}
|
||||
names = append(names, name)
|
||||
}
|
||||
@@ -119,7 +278,7 @@ func waitNames(given any) ([]string, error) {
|
||||
names[i] = strings.TrimSpace(names[i])
|
||||
}
|
||||
if len(names) == 0 {
|
||||
return nil, fmt.Errorf("mesh_wait names at least one event family in `subjects`, such as mesh-knowledge.reviewed")
|
||||
return nil, fmt.Errorf("mesh_wait names at least one event in `subjects`, such as mesh-delivery-view.delivered")
|
||||
}
|
||||
return names, nil
|
||||
}
|
||||
|
||||
@@ -3,37 +3,65 @@ package console
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
|
||||
opseat "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
|
||||
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
|
||||
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatdef"
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
runnerapproval "github.com/novox/mesh-tools/node-tools/internal/approval"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// waitingRuntime is the tool runner on a machine, issued the families ADR 0367 §5 names, as the controller
|
||||
// composes them from its manifest: the surface over its connection, and the mesh to publish on.
|
||||
// waitingRuntime is the tool runner on a machine, issued no waits (novox/hq ADR 0670 §2): the surface over its
|
||||
// connection, and the mesh to publish on.
|
||||
func waitingRuntime(t *testing.T) (*Surface, *mt.Mesh) {
|
||||
t.Helper()
|
||||
mesh := mt.New(t)
|
||||
m := mt.MembershipOf("node-tools", "desk", false, nil)
|
||||
m.Waits = []bus.WaitIssued{
|
||||
{Event: "mesh-knowledge.reviewed", Subject: "mesh.mod.mesh-knowledge.event.reviewed"},
|
||||
{Event: "operator-channel.decided.claude-code", Subject: "mesh.seat.operator-channel.event.decided.claude-code"},
|
||||
}
|
||||
mesh.Issue(t, m)
|
||||
mesh.Issue(t, mt.MembershipOf("node-tools", "desk", false, nil))
|
||||
conn := connect(t, "node-tools", "desk")
|
||||
mt.Until(t, func() error {
|
||||
if got := conn.Membership("node-tools"); got == nil || len(got.Waits) == 0 {
|
||||
if conn.Membership("node-tools") == nil {
|
||||
return os.ErrNotExist
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return NewSurface(conn, busSource{conn}, "desk.node-tools"), mesh
|
||||
}
|
||||
|
||||
// halfSource is a Source that also holds a catalog half, as the tool runner's first views do.
|
||||
type halfSource struct {
|
||||
Source
|
||||
halves map[string]meshcatalogclient.Half
|
||||
}
|
||||
|
||||
func (h halfSource) Halves() (map[string]meshcatalogclient.Half, error) { return h.halves, nil }
|
||||
|
||||
// listening waits until the surface's connection holds n more subscriptions than before: a wait has subscribed.
|
||||
func listening(t *testing.T, s *Surface, before, n int) {
|
||||
t.Helper()
|
||||
mt.Until(t, func() error {
|
||||
if s.conn.Subscriptions() < before+n {
|
||||
return os.ErrNotExist
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return NewSurface(conn, "desk.node-tools"), mesh
|
||||
}
|
||||
|
||||
// callWait is one tools/call of mesh_wait, answered as an agent host reads it.
|
||||
@@ -50,8 +78,9 @@ func callWait(s *Surface, args map[string]any) (text string, isError bool) {
|
||||
return strings.Join(parts, "\n"), isErr
|
||||
}
|
||||
|
||||
// novox/hq ADR 0367 §1: mesh_wait answers the first event that matches — by its family and its filter — with
|
||||
// its subject, body and time, within a second of its publication, and ignores the others.
|
||||
// novox/hq ADR 0367 §1, ADR 0670 §1 (design 34 every-event §13, §15, §19): mesh_wait answers the first event that
|
||||
// matches — by its name and its filter — with no membership naming it: its whole name, the namespace it arrived on, its
|
||||
// body, when it arrived and when it was published, within a second of its publication, and ignores the others.
|
||||
func TestMeshWaitAnswersTheFirstMatchingEvent(t *testing.T) {
|
||||
s, mesh := waitingRuntime(t)
|
||||
type answer struct {
|
||||
@@ -59,18 +88,39 @@ func TestMeshWaitAnswersTheFirstMatchingEvent(t *testing.T) {
|
||||
isErr bool
|
||||
at time.Time
|
||||
}
|
||||
type heardAnswer struct {
|
||||
Heard struct {
|
||||
Subject string `json:"subject"`
|
||||
From string `json:"from"`
|
||||
Body map[string]any `json:"body"`
|
||||
At time.Time `json:"at"`
|
||||
Published *string `json:"published"`
|
||||
} `json:"heard"`
|
||||
Read string `json:"read"`
|
||||
}
|
||||
nc, err := nats.Connect(mt.URL(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(nc.Close)
|
||||
done := make(chan answer, 1)
|
||||
before := s.conn.Subscriptions()
|
||||
go func() {
|
||||
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-knowledge.reviewed"},
|
||||
"filter": map[string]any{"kind": "decision", "number": 367}, "timeout": 20})
|
||||
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-delivery-view.delivered"},
|
||||
"filter": map[string]any{"module": "netcheck"}, "timeout": 20})
|
||||
done <- answer{text, isErr, time.Now()}
|
||||
}()
|
||||
// Once it listens: another record's verdict, another family's event, then the one it waits for.
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
mesh.Emit(t, "mesh.mod.mesh-knowledge.event.reviewed", map[string]any{"kind": "decision", "number": 366, "what": "approved"})
|
||||
mesh.Emit(t, "mesh.mod.mesh-knowledge.event.proposed", map[string]any{"kind": "decision", "number": 367})
|
||||
// Once it listens: another module's delivery, another event, then the one it waits for, with its own time.
|
||||
listening(t, s, before, 6)
|
||||
mesh.Emit(t, "mesh.mod.mesh-delivery-view.event.delivered", map[string]any{"module": "grafana"})
|
||||
mesh.Emit(t, "mesh.mod.mesh-delivery-view.event.planned", map[string]any{"module": "netcheck"})
|
||||
msg := nats.NewMsg("mesh.mod.mesh-delivery-view.event.delivered")
|
||||
msg.Header.Set("x-time", "2026-11-12T03:00:00.5Z")
|
||||
msg.Data = []byte(`{"module":"netcheck"}`)
|
||||
published := time.Now()
|
||||
mesh.Emit(t, "mesh.mod.mesh-knowledge.event.reviewed", map[string]any{"kind": "decision", "number": 367, "what": "approved"})
|
||||
if err := nc.PublishMsg(msg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var got answer
|
||||
select {
|
||||
case got = <-done:
|
||||
@@ -83,32 +133,32 @@ func TestMeshWaitAnswersTheFirstMatchingEvent(t *testing.T) {
|
||||
if got.at.Sub(published) > time.Second {
|
||||
t.Errorf("answered %s after the event was published, want within a second", got.at.Sub(published))
|
||||
}
|
||||
var heard struct {
|
||||
Heard struct {
|
||||
Subject string `json:"subject"`
|
||||
Body map[string]any `json:"body"`
|
||||
At time.Time `json:"at"`
|
||||
} `json:"heard"`
|
||||
}
|
||||
var heard heardAnswer
|
||||
if err := json.Unmarshal([]byte(got.text), &heard); err != nil {
|
||||
t.Fatalf("not an answer: %s", got.text)
|
||||
}
|
||||
if heard.Heard.Subject != "mesh-knowledge.reviewed" || heard.Heard.Body["number"] != float64(367) || heard.Heard.At.IsZero() {
|
||||
t.Errorf("answered %+v, want record 367's verdict with its time", heard.Heard)
|
||||
if h := heard.Heard; h.Subject != "mesh-delivery-view.delivered" || h.From != "module" || h.Body["module"] != "netcheck" ||
|
||||
h.At.IsZero() || h.Published == nil || *h.Published != "2026-11-12T03:00:00.5Z" || !strings.Contains(heard.Read, "only wakes you") {
|
||||
t.Errorf("answered %s, want netcheck's delivery from a module with its times", got.text)
|
||||
}
|
||||
// And the subject it is granted on the asker's own `decided`, by the name a session asks for.
|
||||
// And a seat's `decided` of one asker, heard by `operator-channel.decided`: its whole name, from the seat, with no
|
||||
// time of its own, and the answer read only with claude_code_answer.
|
||||
before = s.conn.Subscriptions()
|
||||
go func() {
|
||||
text, isErr := callWait(s, map[string]any{"subjects": "operator-channel.decided.claude-code", "timeout": 20})
|
||||
text, isErr := callWait(s, map[string]any{"subjects": "operator-channel.decided", "timeout": 20})
|
||||
done <- answer{text, isErr, time.Now()}
|
||||
}()
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
listening(t, s, before, 6)
|
||||
// A seat's event, said by the router on core NATS here: no stream of the test's bus carries the seat.
|
||||
if err := connect(t, "messenger", "").Say("mesh.seat.operator-channel.event.decided.claude-code", []byte(`{"ask":"a1"}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
select {
|
||||
case got = <-done:
|
||||
if got.isErr || !strings.Contains(got.text, `"subject": "operator-channel.decided.claude-code"`) {
|
||||
heard = heardAnswer{}
|
||||
_ = json.Unmarshal([]byte(got.text), &heard)
|
||||
if h := heard.Heard; got.isErr || h.Subject != "operator-channel.decided.claude-code" || h.From != "seat" ||
|
||||
h.Published != nil || !strings.Contains(heard.Read, "claude_code_answer") {
|
||||
t.Errorf("the asker's decided: %s", got.text)
|
||||
}
|
||||
case <-time.After(10 * time.Second):
|
||||
@@ -125,7 +175,8 @@ func TestMeshWaitSaysItTimedOutAndKeepsNothing(t *testing.T) {
|
||||
if took := time.Since(start); took < time.Second || took > 3*time.Second {
|
||||
t.Errorf("a one-second wait took %s", took)
|
||||
}
|
||||
if isErr || !strings.Contains(text, `"timed-out": true`) {
|
||||
// Before the catalog half is read, names are not checked, and the answer says so (design 34 every-event §14).
|
||||
if isErr || !strings.Contains(text, `"timed-out": true`) || !strings.Contains(text, `"unchecked"`) {
|
||||
t.Errorf("a wait that heard nothing: %s", text)
|
||||
}
|
||||
if after := s.conn.Subscriptions(); after != before {
|
||||
@@ -148,39 +199,357 @@ func TestMeshWaitsTimeoutIsBounded(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// novox/hq ADR 0367 §2 and §6: an event outside the granted families is refused, naming the families and how
|
||||
// one is added; no argument — a wildcard, another asker's `decided`, a subject spelled out — widens it.
|
||||
func TestMeshWaitRefusesAnEventOutsideTheGrantedFamilies(t *testing.T) {
|
||||
// novox/hq ADR 0670 §1, §2 (design 34 every-event §13, §16, §19): mesh_wait refuses loudly and never loops: a name with
|
||||
// a wildcard, of one token, more than 16 names, none, or a from of another word is refused at once naming the rule; a
|
||||
// wait whose every subscription the bus refuses answers after its timeout, in words naming no wait grant.
|
||||
func TestMeshWaitRefusesLoudlyAndNeverLoops(t *testing.T) {
|
||||
s, _ := waitingRuntime(t)
|
||||
for _, subjects := range []any{
|
||||
[]any{"mesh-knowledge.proposed"},
|
||||
[]any{"mesh-knowledge.reviewed", "gitea.forge.push"},
|
||||
[]any{"mesh-knowledge.*"},
|
||||
[]any{"operator-channel.decided.shop"},
|
||||
[]any{"operator-channel.decided.*"},
|
||||
[]any{"mesh.mod.mesh-knowledge.event.reviewed"},
|
||||
[]any{">"},
|
||||
[]any{},
|
||||
var seventeen []any
|
||||
for i := 0; i < 17; i++ {
|
||||
seventeen = append(seventeen, fmt.Sprintf("mesh-knowledge.reviewed%d", i))
|
||||
}
|
||||
for _, c := range []struct {
|
||||
args map[string]any
|
||||
says string
|
||||
}{
|
||||
{map[string]any{"subjects": []any{"mesh-knowledge.*"}}, "[A-Za-z0-9_-]+"},
|
||||
{map[string]any{"subjects": []any{"operator-channel.decided.>"}}, "[A-Za-z0-9_-]+"},
|
||||
{map[string]any{"subjects": []any{"mesh-knowledge"}}, "fewer than two tokens"},
|
||||
{map[string]any{"subjects": seventeen}, "at most 16"},
|
||||
{map[string]any{"subjects": []any{}}, "at least one event"},
|
||||
{map[string]any{"subjects": []any{"mesh-knowledge.reviewed"}, "from": "bench"}, "seat, module or stream"},
|
||||
} {
|
||||
text, isErr := callWait(s, map[string]any{"subjects": subjects, "timeout": 1})
|
||||
if !isErr {
|
||||
t.Errorf("%v was waited on: %s", subjects, text)
|
||||
continue
|
||||
c.args["timeout"] = 1
|
||||
start := time.Now()
|
||||
text, isErr := callWait(s, c.args)
|
||||
if !isErr || !strings.Contains(text, c.says) || strings.Contains(text, "manifest") {
|
||||
t.Errorf("%v: %s, want refused saying %q", c.args, text, c.says)
|
||||
}
|
||||
if len(subjects.([]any)) > 0 && (!strings.Contains(text, "mesh-knowledge.reviewed") || !strings.Contains(text, "manifest")) {
|
||||
t.Errorf("%v refused without naming the families and how one is added: %s", subjects, text)
|
||||
if took := time.Since(start); took > 500*time.Millisecond {
|
||||
t.Errorf("%v was refused after %s, want at once", c.args, took)
|
||||
}
|
||||
}
|
||||
|
||||
t.Run("the catalog half", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
// E §14: against the catalog half, a misspelt emitter is refused naming the half and its revision, a known emitter's
|
||||
// undeclared event is waited on and its timeout says so, and a declared one says nothing of it.
|
||||
half := NewSurface(s.conn, halfSource{busSource{s.conn}, map[string]meshcatalogclient.Half{"main": {
|
||||
Revision: catalogseat.CatalogRevision{Revision: 7, Definitions: []catalogseat.CatalogRevisionDefinitionsItem{{Seat: "mesh-catalog"}}},
|
||||
Entries: map[string]catalogseat.CatalogEntry{"mesh-delivery-view": {Module: "mesh-delivery-view",
|
||||
Manifest: map[string]any{"emits": []any{"delivered"}}}},
|
||||
}}}, "desk.node-tools")
|
||||
for _, c := range []struct {
|
||||
name, says string
|
||||
refused bool
|
||||
}{
|
||||
{"mesh-delivry-view.delivered", "mesh-delivry-view is neither a seat nor a module of the catalog half (mesh-catalog.revision, revision 7 of main)", true},
|
||||
{"mesh-delivery-view.planned", "no declaration names mesh-delivery-view.planned", false},
|
||||
{"mesh-catalog.catalog-revision", "", false},
|
||||
} {
|
||||
text, isErr := callWait(half, map[string]any{"subjects": []any{c.name}, "timeout": 1})
|
||||
if isErr != c.refused || c.says != "" && !strings.Contains(text, c.says) ||
|
||||
c.says == "" && (strings.Contains(text, "undeclared") || strings.Contains(text, "unchecked")) {
|
||||
t.Errorf("%s against the catalog half: %s", c.name, text)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a bus that refuses every subscription", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
// The bus refusing every subscription of the call: answered after the timeout, in §16's words.
|
||||
b, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, NoLog: true, NoSigs: true,
|
||||
Users: []*server.User{{Username: "desk.node-tools", Password: "pw", Permissions: &server.Permissions{
|
||||
Subscribe: &server.SubjectPermission{Allow: []string{"_INBOX.desk.node-tools.>", "mesh.assignment.desk.*"}},
|
||||
}}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
go b.Start()
|
||||
if !b.ReadyForConnections(10 * time.Second) {
|
||||
t.Fatal("the bus did not come up")
|
||||
}
|
||||
t.Cleanup(b.Shutdown)
|
||||
conn, err := bus.Connect(bus.Credential{URL: b.ClientURL(), Module: "node-tools", Node: "desk", User: "desk.node-tools", Password: "pw"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn.Logf = func(string, ...any) {}
|
||||
t.Cleanup(conn.Close)
|
||||
start := time.Now()
|
||||
text, isErr := callWait(NewSurface(conn, nil, "desk.node-tools"), map[string]any{"subjects": []any{"mesh-delivery-view.delivered"}, "timeout": 1})
|
||||
if took := time.Since(start); took < time.Second || took > 3*time.Second {
|
||||
t.Errorf("a wait the bus refused whole answered after %s, want its timeout of 1s", took)
|
||||
}
|
||||
if !isErr || !strings.Contains(text, "the bus refused this account a subscription on mesh.seat.mesh-delivery-view.event.delivered") ||
|
||||
!strings.Contains(text, "novox/hq ADR 0670 §1") || strings.Contains(text, "wait grant") || strings.Contains(text, "0367 §2") {
|
||||
t.Errorf("a wait the bus refused whole: %s", text)
|
||||
}
|
||||
})
|
||||
|
||||
// Design 34 every-event §37: a wait whose subjects overlap a marked event's, or a stream's with a `grant`, runs only on the
|
||||
// operator's warrant for that one wait; with none it subscribes nothing, asks once and answers at its timeout or the choice.
|
||||
// Each group of rows has a runtime of its own, so the waits of the groups run side by side (A-311: the one-second floor of
|
||||
// a wait's timeout stands, and no clock is injected).
|
||||
for name, group := range map[string]func(*testing.T, *Surface, *mt.Mesh){
|
||||
"no warrant": markedWaitsWithNoWarrant, "a warrant": markedWaitsOnAWarrant, "the catalog half unread": markedWaitsBeforeTheHalfIsRead,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
s, mesh := waitingRuntime(t)
|
||||
group(t, s, mesh)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// markedWaitsWithNoWarrant are §37's rows of a wait with no warrant, against a gate on the test bus, the router's records written there as the router writes them.
|
||||
func markedWaitsWithNoWarrant(t *testing.T, s *Surface, mesh *mt.Mesh) {
|
||||
gw := newGatedWaits(t, s, mesh, "src2:runner")
|
||||
g := gw.surface
|
||||
took := func(args map[string]any) (string, bool, time.Duration) {
|
||||
start := time.Now()
|
||||
text, isErr := callWait(g, args)
|
||||
return text, isErr, time.Since(start)
|
||||
}
|
||||
// A fixture module's marked event, no warrant: nothing of it subscribed (only its ask's decided), asked once, §12's
|
||||
// words at its timeout; the same wait again asks nothing new.
|
||||
before := g.conn.Subscriptions()
|
||||
go func() {
|
||||
listening(t, g, before, 1)
|
||||
mesh.Emit(t, "mesh.mod.mail.event.message", map[string]any{"n": 1})
|
||||
}()
|
||||
text, isErr, d := took(map[string]any{"subjects": []any{"mail.message"}, "timeout": 1})
|
||||
if !isErr || !strings.Contains(text, "this wait needs the operator's approval. Asked on telegram") || gw.asks() != 1 ||
|
||||
d < time.Second || strings.Contains(text, "heard") {
|
||||
t.Errorf("a marked event's wait, no warrant: %s after %s, %d asks", text, d, gw.asks())
|
||||
}
|
||||
if text, isErr, _ = took(map[string]any{"subjects": []any{"mail.message"}, "timeout": 1}); !isErr ||
|
||||
!strings.Contains(text, "still waiting") || gw.asks() != 1 {
|
||||
t.Errorf("the same marked wait again: %s, %d asks", text, gw.asks())
|
||||
}
|
||||
// A stream with a `grant` is marked whole; a name with the node after a marked event's name is marked.
|
||||
for i, name := range []string{"mail.inbox", "mail.message.desk"} {
|
||||
if text, isErr, _ := took(map[string]any{"subjects": []any{name}, "timeout": 1}); !isErr || !strings.Contains(text, "Not run") || gw.asks() != 2+i {
|
||||
t.Errorf("%s: %s, %d asks", name, text, gw.asks())
|
||||
}
|
||||
}
|
||||
// The same names from a seat ask nothing.
|
||||
if text, isErr, _ := took(map[string]any{"subjects": []any{"mail.message"}, "from": "seat", "timeout": 1}); isErr ||
|
||||
!strings.Contains(text, `"timed-out": true`) || gw.asks() != 3 {
|
||||
t.Errorf("from a seat: %s, %d asks", text, gw.asks())
|
||||
}
|
||||
}
|
||||
|
||||
// markedWaitsOnAWarrant are §37's rows of a wait chosen while it waits and of one on a warrant.
|
||||
func markedWaitsOnAWarrant(t *testing.T, s *Surface, mesh *mt.Mesh) {
|
||||
gw := newGatedWaits(t, s, mesh, "src2:runner")
|
||||
g := gw.surface
|
||||
took := func(args map[string]any) (string, bool, time.Duration) {
|
||||
start := time.Now()
|
||||
text, isErr := callWait(g, args)
|
||||
return text, isErr, time.Since(start)
|
||||
}
|
||||
// The most timeout a wait takes, its ask chosen while it waits: answered at the choice; the same call made then
|
||||
// subscribes and hears the event published after it, answered with its approval; a warrant for other names, another
|
||||
// filter or another timeout is refused, asked anew.
|
||||
long := map[string]any{"subjects": []any{"mail.message"}, "filter": map[string]any{"n": 2}, "timeout": 600}
|
||||
done := make(chan string, 1)
|
||||
go func() { text, _, d := took(long); done <- fmt.Sprintf("%s after %s", text, d) }()
|
||||
mt.Until(t, func() error {
|
||||
if gw.asks() < 1 {
|
||||
return os.ErrNotExist
|
||||
}
|
||||
return nil
|
||||
})
|
||||
gw.approve(t)
|
||||
select {
|
||||
case got := <-done:
|
||||
if !strings.Contains(got, "make this wait again within 30 minutes") {
|
||||
t.Errorf("a long wait chosen while it waits: %s", got)
|
||||
}
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("a long marked wait was not answered at the choice")
|
||||
}
|
||||
before := g.conn.Subscriptions()
|
||||
go func() {
|
||||
listening(t, g, before, 6)
|
||||
mesh.Emit(t, "mesh.mod.mail.event.message", map[string]any{"n": 2})
|
||||
}()
|
||||
if text, isErr, _ := took(long); isErr || !strings.Contains(text, `"approval": "run on the operator's approval`) ||
|
||||
!strings.Contains(text, `"subject": "mail.message"`) {
|
||||
t.Errorf("the same wait on its warrant: %s", text)
|
||||
}
|
||||
for _, other := range []map[string]any{
|
||||
{"subjects": []any{"mail.message", "mail.inbox"}, "filter": map[string]any{"n": 2}, "timeout": 1},
|
||||
{"subjects": []any{"mail.message"}, "filter": map[string]any{"n": 3}, "timeout": 1},
|
||||
{"subjects": []any{"mail.message"}, "filter": map[string]any{"n": 2}, "timeout": 2},
|
||||
} {
|
||||
asked := gw.asks()
|
||||
if text, isErr, _ := took(other); !isErr || !strings.Contains(text, "Not run") || gw.asks() != asked+1 {
|
||||
t.Errorf("another wait than the warrant's %v: %s", other, text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A tool runner issued no families waits on nothing, and says so.
|
||||
func TestMeshWaitWithNoFamiliesIssuedWaitsOnNothing(t *testing.T) {
|
||||
mt.New(t)
|
||||
s := NewSurface(connect(t, "node-tools", "desk"), "desk.node-tools")
|
||||
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-knowledge.reviewed"}, "timeout": 1})
|
||||
if !isErr || !strings.Contains(text, "no event families") {
|
||||
t.Errorf("a runtime issued no families: %s", text)
|
||||
// markedWaitsBeforeTheHalfIsRead are §37's rows of a wait before the catalog half is read, and of one whose runner build is unknown.
|
||||
func markedWaitsBeforeTheHalfIsRead(t *testing.T, s *Surface, mesh *mt.Mesh) {
|
||||
gw := newGatedWaits(t, s, mesh, "src2:runner")
|
||||
g := gw.surface
|
||||
took := func(args map[string]any) (string, bool, time.Duration) {
|
||||
start := time.Now()
|
||||
text, isErr := callWait(g, args)
|
||||
return text, isErr, time.Since(start)
|
||||
}
|
||||
// Before the catalog half is read nothing is asked: judged when its first view comes within the timeout, refused at
|
||||
// the timeout when it does not.
|
||||
gw.cat.mu.Lock()
|
||||
gw.cat.ready = errors.New("no first view yet")
|
||||
gw.cat.mu.Unlock()
|
||||
asked := gw.asks()
|
||||
if text, isErr, d := took(map[string]any{"subjects": []any{"mail.message"}, "timeout": 1}); !isErr ||
|
||||
!strings.Contains(text, "the marks are not yet read") || gw.asks() != asked || d < time.Second {
|
||||
t.Errorf("before the catalog half is read: %s after %s", text, d)
|
||||
}
|
||||
gw.cat.came = make(chan chan struct{})
|
||||
go func() { close(<-gw.cat.came) }()
|
||||
if text, isErr, _ := took(map[string]any{"subjects": []any{"mail.message"}, "filter": map[string]any{"n": 9}, "timeout": 1}); !isErr ||
|
||||
!strings.Contains(text, "this wait needs the operator's approval") || gw.asks() != asked+1 {
|
||||
t.Errorf("the catalog half read within the timeout: %s", text)
|
||||
}
|
||||
// While the runner's own fingerprint is unknown a marked wait is refused, saying so.
|
||||
unknown := newGatedWaits(t, s, mesh, "")
|
||||
if text, isErr := callWait(unknown.surface, map[string]any{"subjects": []any{"mail.message"}, "timeout": 1}); !isErr ||
|
||||
!strings.Contains(text, "own fingerprint is unknown") || unknown.asks() != 0 {
|
||||
t.Errorf("the runner's fingerprint unknown: %s", text)
|
||||
}
|
||||
}
|
||||
|
||||
// gatedWaits is the surface with the runner's approval gate: the fixture module mail marks its event message and declares
|
||||
// the stream inbox with a grant; the router takes each ask as it is published, its records on the test bus.
|
||||
type gatedWaits struct {
|
||||
surface *Surface
|
||||
cat *waitCatalog
|
||||
records nats.KeyValue
|
||||
mu sync.Mutex
|
||||
asked []opseat.Ask
|
||||
say *bus.Conn
|
||||
}
|
||||
|
||||
func newGatedWaits(t *testing.T, s *Surface, mesh *mt.Mesh, build string) *gatedWaits {
|
||||
gw := &gatedWaits{cat: &waitCatalog{entries: []catalogseat.CatalogEntry{{Module: "mail", Fp: "src2:mail", Manifest: map[string]any{
|
||||
"emits": []any{"message"}, "needs-approval": []any{"message"},
|
||||
"streams": []any{map[string]any{"name": "inbox", "events": []any{"inbox.*"}, "grant": map[string]any{"subscribe": []any{"x"}}}}}}}},
|
||||
records: mesh.Bucket(t, "operator-channel-asks"), say: connect(t, "messenger", "")}
|
||||
state := mt.NewState()
|
||||
gate := approval.New(approval.Deps{Node: "desk", Runner: "node-tools", Build: build, Catalog: gw.cat, Wait: gw.cat.wait,
|
||||
State: state, Holder: func(string) (bool, error) { return true, nil },
|
||||
Hear: func(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (bool, error) {
|
||||
_, ok, _, err := s.conn.HearFirst(ctx, subjects, within, match)
|
||||
return ok, err
|
||||
},
|
||||
Channel: &operatorchannelclient.GateChannel{State: state, KeyPrefix: "desk.", Publish: func(module, subject string, body any, id string) error {
|
||||
a, ok := body.(opseat.Ask)
|
||||
if subject != opseat.AcceptSubjectAsk(module) || !ok {
|
||||
return nil
|
||||
}
|
||||
gw.mu.Lock()
|
||||
gw.asked = append(gw.asked, a)
|
||||
gw.mu.Unlock()
|
||||
gw.write(t, opseat.AsksByAskerIDValue{Asker: module, Key: module + "." + a.ID, State: "open", Headline: a.Headline,
|
||||
Expires: a.Expires, Opened: operatorchannelclient.FormatTime(time.Now()), OfferedOn: []string{"telegram"}})
|
||||
return nil
|
||||
},
|
||||
ReadRecord: func(module, id string) (*stdio.StateEntry, error) {
|
||||
e, err := gw.records.Get(module + "." + id)
|
||||
if errors.Is(err, nats.ErrKeyNotFound) {
|
||||
return nil, nil
|
||||
} else if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &stdio.StateEntry{Key: e.Key(), Value: e.Value(), Revision: e.Revision()}, nil
|
||||
}}})
|
||||
gw.surface = NewSurface(s.conn, busSource{s.conn}, "desk.node-tools")
|
||||
gw.surface.Approvals = gateApprovals{gate}
|
||||
return gw
|
||||
}
|
||||
|
||||
func (gw *gatedWaits) write(t *testing.T, rec opseat.AsksByAskerIDValue) {
|
||||
raw, _ := json.Marshal(rec)
|
||||
if _, err := gw.records.Put(rec.Key, raw); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
|
||||
func (gw *gatedWaits) asks() int {
|
||||
gw.mu.Lock()
|
||||
defer gw.mu.Unlock()
|
||||
return len(gw.asked)
|
||||
}
|
||||
|
||||
// approve writes the router's record of the newest ask as a person's Approve, and says its decided.
|
||||
func (gw *gatedWaits) approve(t *testing.T) {
|
||||
gw.mu.Lock()
|
||||
a := gw.asked[len(gw.asked)-1]
|
||||
gw.mu.Unlock()
|
||||
gw.write(t, opseat.AsksByAskerIDValue{Asker: "node-tools", Key: "node-tools." + a.ID, State: "chosen", Expires: a.Expires,
|
||||
OfferedOn: []string{"telegram"}, Warrant: &opseat.Warrant{Ask: a.ID, Asker: "node-tools", Outcome: operatorchannelclient.OutcomeChosen,
|
||||
Option: approval.OptionApprove, Level: operatorchannelclient.Approve, At: operatorchannelclient.FormatTime(time.Now()),
|
||||
AskDigest: operatorchannelclient.Digest(a), Channel: "telegram",
|
||||
By: &opseat.Person{Who: operatorchannelclient.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}})
|
||||
if err := gw.say.Say(opseat.SubjectDecided("node-tools"), []byte(`{"ask":"`+a.ID+`"}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// gateApprovals is the gate as the surface holds it: mesh_approvals' reads unused here, mesh_wait's gate.
|
||||
type gateApprovals struct{ *approval.Gate }
|
||||
|
||||
func (gateApprovals) Fill() (uint64, int64, error) { return 0, 0, nil }
|
||||
|
||||
// waitCatalog is a catalog half as a fixture: its entries, read or not; with came set, its first view comes once a wait
|
||||
// waits for it (waiting is told on it, then it closes).
|
||||
type waitCatalog struct {
|
||||
entries []catalogseat.CatalogEntry
|
||||
mu sync.Mutex
|
||||
ready error
|
||||
came chan chan struct{}
|
||||
}
|
||||
|
||||
func (c *waitCatalog) Ready() error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.ready
|
||||
}
|
||||
func (c *waitCatalog) wait(within time.Duration) error {
|
||||
if c.Ready() == nil {
|
||||
return nil
|
||||
}
|
||||
if c.came != nil {
|
||||
view := make(chan struct{})
|
||||
c.came <- view
|
||||
<-view
|
||||
c.mu.Lock()
|
||||
c.ready = nil
|
||||
c.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
<-time.After(within) // the bound of the runner's own wait for the first view
|
||||
return c.Ready()
|
||||
}
|
||||
func (c *waitCatalog) Revision() string { return "7" }
|
||||
func (c *waitCatalog) Served(string) (approval.Entry, bool, string) {
|
||||
return approval.Entry{}, false, "not-found"
|
||||
}
|
||||
func (c *waitCatalog) Newest(string) (approval.Entry, bool) {
|
||||
return approval.Entry{}, false
|
||||
}
|
||||
func (c *waitCatalog) Definitions() map[string]seatdef.Definition { return nil }
|
||||
func (c *waitCatalog) Entries() []approval.Entry {
|
||||
var out []approval.Entry
|
||||
for _, e := range c.entries {
|
||||
out = append(out, runnerapproval.Entry(e))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// The mesh MCP server announces seven tools (ADR 0367 v13): ADR 0195's five, mesh_runtimes and mesh_wait, its timeout bounded.
|
||||
@@ -194,7 +563,7 @@ func TestMeshWaitIsAnnouncedAmongSeven(t *testing.T) {
|
||||
}
|
||||
}
|
||||
sort.Strings(names)
|
||||
want := []string{"mesh_call", "mesh_describe", "mesh_machine", "mesh_overview", "mesh_runtimes", "mesh_search", "mesh_wait"}
|
||||
want := []string{"mesh_approvals", "mesh_call", "mesh_describe", "mesh_first_view_again", "mesh_first_views", "mesh_machine", "mesh_overview", "mesh_runtimes", "mesh_search", "mesh_wait"}
|
||||
if !reflect.DeepEqual(names, want) {
|
||||
t.Errorf("announced %v, want %v", names, want)
|
||||
}
|
||||
@@ -207,8 +576,49 @@ func TestMeshWaitIsAnnouncedAmongSeven(t *testing.T) {
|
||||
t.Errorf("mesh_wait's schema has no %s: %s", word, schema)
|
||||
}
|
||||
}
|
||||
// Design 46 7-approval-mark §18: mesh_approvals lists, and records a hand act before it cancels; refused without a record.
|
||||
gate := &fakeApprovals{}
|
||||
var recorded []string
|
||||
s := &Surface{who: "n1.node-tools", Approvals: gate, Record: func(a map[string]any) error {
|
||||
if a["why"] == "unrecordable" {
|
||||
return errors.New("mesh-hand-acts does not answer")
|
||||
}
|
||||
recorded = append(recorded, a["what"].(string))
|
||||
return nil
|
||||
}}
|
||||
if out := s.discover(context.Background(), verbApprovals, map[string]any{}); out["isError"] == true || !strings.Contains(fmtAnswer(out), "every machine") {
|
||||
t.Errorf("the list %v", out)
|
||||
}
|
||||
if out := s.discover(context.Background(), verbApprovals, map[string]any{"cancel": "a1", "why": "no longer needed"}); out["isError"] == true ||
|
||||
gate.cancelled != "a1" || len(recorded) != 1 || recorded[0] != "cancelled ask a1" {
|
||||
t.Errorf("a cancel %v, recorded %v", out, recorded)
|
||||
}
|
||||
if out := s.discover(context.Background(), verbApprovals, map[string]any{"cancel": "a2", "why": "unrecordable"}); out["isError"] != true || gate.cancelled != "a1" {
|
||||
t.Errorf("a cancel without a record %v", out)
|
||||
}
|
||||
}
|
||||
|
||||
func fmtAnswer(out map[string]any) string { raw, _ := json.Marshal(out); return string(raw) }
|
||||
|
||||
// fakeApprovals is a gate as mesh_approvals reads it.
|
||||
type fakeApprovals struct {
|
||||
cancelled string
|
||||
// marked is the one `<module>.<tool>` its Describe says is marked.
|
||||
marked string
|
||||
}
|
||||
|
||||
func (f *fakeApprovals) List() ([]approval.Listed, error) {
|
||||
return []approval.Listed{{Ask: "a1", State: "asked"}}, nil
|
||||
}
|
||||
func (f *fakeApprovals) Read(id string) (approval.Whole, error) { return approval.Whole{}, nil }
|
||||
func (f *fakeApprovals) Applied() []approval.Marks { return nil }
|
||||
func (f *fakeApprovals) Describe(module, _, name string) (bool, string) {
|
||||
return f.marked != "" && f.marked == module+"."+name, "entry " + module
|
||||
}
|
||||
func (f *fakeApprovals) Again(string) error { return nil }
|
||||
func (f *fakeApprovals) Cancel(id string) error { f.cancelled = id; return nil }
|
||||
func (f *fakeApprovals) Fill() (uint64, int64, error) { return 10, 64 << 20, nil }
|
||||
|
||||
// A client that hangs up or cancels ends its wait at once: the subscription is closed and nothing is left
|
||||
// waiting for the timeout (the review of issue 745). Over the console's own HTTP, as an agent host calls it.
|
||||
func TestMeshWaitEndsWhenItsClientHangsUp(t *testing.T) {
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
package firstview
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatdef"
|
||||
|
||||
runnerapproval "github.com/novox/mesh-tools/node-tools/internal/approval"
|
||||
)
|
||||
|
||||
// Findings is the runner's held finding writer, the one the gate writes its findings through too (design 46 7-approval-mark
|
||||
// §4, §16, §19); nil when it was not made.
|
||||
func (v *Views) Findings() runnerapproval.HeldWriter {
|
||||
if v.deps.Findings == nil {
|
||||
return nil
|
||||
}
|
||||
return v.deps.Findings
|
||||
}
|
||||
|
||||
// ApprovalCatalog is the catalog half as the gate reads its marks (design 46 7-approval-mark §5): the entries the branch
|
||||
// follower holds, an entry by fingerprint asked of `mesh-catalog.entry` when no branch lists it, and the seat definitions
|
||||
// the seat list holds (defs).
|
||||
func (v *Views) ApprovalCatalog(defs func() map[string]seatdef.Definition) approval.Catalog {
|
||||
return approvalCatalog{v: v, defs: defs}
|
||||
}
|
||||
|
||||
type approvalCatalog struct {
|
||||
v *Views
|
||||
defs func() map[string]seatdef.Definition
|
||||
}
|
||||
|
||||
func (c approvalCatalog) Ready() error {
|
||||
if c.v.branches == nil {
|
||||
return errNotFollowed
|
||||
}
|
||||
return c.v.branches.Ready()
|
||||
}
|
||||
|
||||
func (c approvalCatalog) Revision() string {
|
||||
if c.v.branches == nil {
|
||||
return ""
|
||||
}
|
||||
for _, branch := range c.v.branches.Followed() {
|
||||
if cat, ok := c.v.branches.Catalog(branch); ok {
|
||||
if half, ok := cat.View(); ok {
|
||||
return strconv.FormatInt(half.Revision.Revision, 10)
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (c approvalCatalog) Served(fingerprint string) (approval.Entry, bool, string) {
|
||||
if c.v.branches == nil {
|
||||
return approval.Entry{}, false, errNotFollowed.Error()
|
||||
}
|
||||
e, state := c.v.branches.Resolve(context.Background(), fingerprint)
|
||||
return runnerapproval.Entry(e), state == meshcatalogclient.EntryAnswered, string(state)
|
||||
}
|
||||
|
||||
func (c approvalCatalog) Newest(module string) (approval.Entry, bool) {
|
||||
if c.v.branches == nil {
|
||||
return approval.Entry{}, false
|
||||
}
|
||||
e, ok, err := c.v.branches.Entry(module)
|
||||
return runnerapproval.Entry(e), ok && err == nil
|
||||
}
|
||||
|
||||
func (c approvalCatalog) Definitions() map[string]seatdef.Definition {
|
||||
if c.defs == nil {
|
||||
return nil
|
||||
}
|
||||
return c.defs()
|
||||
}
|
||||
|
||||
func (c approvalCatalog) Entries() []approval.Entry {
|
||||
halves, err := c.v.Halves()
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []approval.Entry
|
||||
for _, half := range halves {
|
||||
for _, e := range half.Entries {
|
||||
out = append(out, runnerapproval.Entry(e))
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type notFollowed struct{}
|
||||
|
||||
func (notFollowed) Error() string { return "the catalog half is not followed by this runner" }
|
||||
|
||||
var errNotFollowed error = notFollowed{}
|
||||
|
||||
// heardOne wakes every wait on the picture.
|
||||
func (v *Views) heardOne() {
|
||||
v.mu.Lock()
|
||||
defer v.mu.Unlock()
|
||||
if v.heard != nil {
|
||||
close(v.heard)
|
||||
}
|
||||
v.heard = make(chan struct{})
|
||||
}
|
||||
|
||||
// heardNext is closed at the next event heard.
|
||||
func (v *Views) heardNext() <-chan struct{} {
|
||||
v.mu.Lock()
|
||||
defer v.mu.Unlock()
|
||||
if v.heard == nil {
|
||||
v.heard = make(chan struct{})
|
||||
}
|
||||
return v.heard
|
||||
}
|
||||
|
||||
// WaitCatalog waits within the bound for the catalog half's first view of every watched branch (design 46 7-approval-mark
|
||||
// §19), waking at each event the runner hears and at each first view answered; nil once it is read, else why not.
|
||||
func (v *Views) WaitCatalog(within time.Duration) error {
|
||||
c := approvalCatalog{v: v}
|
||||
timer := time.NewTimer(within)
|
||||
defer timer.Stop()
|
||||
for {
|
||||
next := v.heardNext()
|
||||
err := c.Ready()
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
var ready []<-chan struct{}
|
||||
if v.branches != nil {
|
||||
for _, branch := range v.branches.Followed() {
|
||||
if cat, ok := v.branches.Catalog(branch); ok {
|
||||
select {
|
||||
case <-cat.Ready(): // read already: waits on nothing
|
||||
default:
|
||||
ready = append(ready, cat.Ready())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if !waitAny(next, timer.C, ready) {
|
||||
return c.Ready()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// waitAny waits for next or one of ready, true, or for the bound, false.
|
||||
func waitAny(next <-chan struct{}, bound <-chan time.Time, ready []<-chan struct{}) bool {
|
||||
woke := make(chan struct{}, 1)
|
||||
stop := make(chan struct{})
|
||||
defer close(stop)
|
||||
for _, r := range ready {
|
||||
go func(r <-chan struct{}) {
|
||||
select {
|
||||
case <-r:
|
||||
woke <- struct{}{}
|
||||
case <-stop:
|
||||
}
|
||||
}(r)
|
||||
}
|
||||
select {
|
||||
case <-next:
|
||||
case <-woke:
|
||||
case <-bound:
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// OnSeatHolder calls f whenever the assigner's seat-holders view changes for the seat (lead answer A-157: the gate asks
|
||||
// mesh-hand-acts.warranted again when the hand-act seat's holder is heard again).
|
||||
func (v *Views) OnSeatHolder(seat string, f func()) {
|
||||
v.mu.Lock()
|
||||
defer v.mu.Unlock()
|
||||
if v.onKey == nil {
|
||||
v.onKey = map[string][]func(){}
|
||||
}
|
||||
k := "seat-holders " + seat
|
||||
v.onKey[k] = append(v.onKey[k], f)
|
||||
}
|
||||
|
||||
// heardKey calls the hearers of a key of an owner's state just changed: a seat's holders, under the seat's own key and
|
||||
// its `<seat>.<node>` keys.
|
||||
func (v *Views) heardKey(state, key string) {
|
||||
seat, _, _ := strings.Cut(key, ".")
|
||||
v.mu.Lock()
|
||||
fs := append([]func(){}, v.onKey[state+" "+seat]...)
|
||||
v.mu.Unlock()
|
||||
for _, f := range fs {
|
||||
go f()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,572 @@
|
||||
// Package firstview is the tool runner's picture of what it reads of the assigner and the catalog half: the assigner's
|
||||
// assignments and seat-holders (novox/hq ADR 0615 §1, §3; ADR 0621 §2; issue 1966 TLS-5), and which modules exist with the
|
||||
// manifest each entry carries, followed per watched branch through meshcatalogclient's branch follower on this process's own
|
||||
// consumer (design 47 G §196, §594, §614; lead answers A-5, A-27, A-103), never the indexer's state `modules` or `manifests`.
|
||||
//
|
||||
// **It never reads the owners' state.** At start it asks each owner's first-view tool until answered (the owner answers in a
|
||||
// runtime of its own, never through this one: ADR 0621 §1), then follows the owner's views family, one event per changed key;
|
||||
// a gap in the revisions, or an owner whose state was made again, asks the first view once more, said in the log. Past two
|
||||
// minutes without an answer it writes a finding through its held finding writer (design 47 G A-27, A-28), which keeps it in its
|
||||
// own state `findings` and says it; the answer clears it.
|
||||
package firstview
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
|
||||
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
|
||||
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
|
||||
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
"git.novox.be/novox/mesh-sdk/go/keys"
|
||||
"git.novox.be/novox/mesh-sdk/go/process"
|
||||
"git.novox.be/novox/mesh-sdk/go/views"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
|
||||
// AssignerSeat is the seat whose holder's views the runner follows: the holder is read from the catalog half's claims of the
|
||||
// seat, never named in code (design 47 G §616, §428), and the names the console reads the two states by are the seat's
|
||||
// (`<seat>.<state>`; the manifest's `consumes` names their views families by the seat, as `seat:<seat>.views.<state>`).
|
||||
const AssignerSeat = meshmoduleassigner.Seat
|
||||
|
||||
const (
|
||||
AssignmentsRead = AssignerSeat + ".assignments"
|
||||
SeatHoldersRead = AssignerSeat + ".seat-holders"
|
||||
)
|
||||
|
||||
// Spec is one first view the runner holds of the seat's holder: its first-view tool and the state.
|
||||
type Spec struct{ Tool, State string }
|
||||
|
||||
// Specs are the two states asked of the holder's tool; the catalog half's views are adopted by the branch follower.
|
||||
var Specs = []Spec{
|
||||
{"assignments", "assignments"},
|
||||
{"seat-holders", "seat-holders"},
|
||||
}
|
||||
|
||||
// Name is the name a state is read by, `<seat>.<state>`: AssignmentsRead and SeatHoldersRead.
|
||||
func (s Spec) Name() string { return AssignerSeat + "." + s.State }
|
||||
|
||||
// Views is the first views: the assigner's two, and the catalog half's per watched branch.
|
||||
type Views struct {
|
||||
logf func(string, ...any)
|
||||
mu sync.Mutex
|
||||
followers map[string]*views.Follower // by `<owner>.<state>`
|
||||
byState map[string]*views.Follower // by the state an event names, the owners' views families
|
||||
adopted map[string]*adoption // by `<owner>.<state>`, the views the branch follower adopted
|
||||
hearers map[string][]*adoption // event key `<seat>.<event>` -> the adopted views that hear it
|
||||
branches *meshcatalogclient.Branches
|
||||
holder *holding // the module that claims the assigner seat, once the catalog half names it
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
wg sync.WaitGroup
|
||||
deps Deps
|
||||
// heard is closed and made again at every event heard, so a wait for the picture wakes on events, never on a timer.
|
||||
heard chan struct{}
|
||||
// onKey are the hearers of one key of an owner's state, by `<state> <key>` (OnSeatHolder).
|
||||
onKey map[string][]func()
|
||||
}
|
||||
|
||||
// FindingsState is the runner's own state the findings are written in.
|
||||
const FindingsState = "findings"
|
||||
|
||||
// Deps is what the views reach the bus through: Start makes them from the runner's connection, a test from fakes.
|
||||
type Deps struct {
|
||||
// Module is the runner's own name, the observer of its findings.
|
||||
Module string
|
||||
// Ask calls an owner's tool, `<owner>.<tool>`.
|
||||
Ask func(address string, body any) (json.RawMessage, error)
|
||||
// Consume hands each event of the owners' views families to deliver, until the stop is called.
|
||||
Consume func(deliver func(bus.Envelope) error) (stop func(), err error)
|
||||
// Findings is the runner's held finding writer (views.HeldWriter): a follower's unanswered first view is written and cleared
|
||||
// through it with the contract's Finding (design 47 G A-28). Nil says it in the log only.
|
||||
Findings FindingWriter
|
||||
// StartFindings starts that writer, and its stop ends it; nil starts none.
|
||||
StartFindings func(ctx context.Context) (stop func(), err error)
|
||||
// Build is the build this process was started with (process.OwnBuild, read once at the start): its fingerprint, or
|
||||
// why it is unknown (design 47 G §205, §424).
|
||||
Build process.Build
|
||||
// Node is the machine the runner is on, named in its findings' subject.
|
||||
Node string
|
||||
// Bound and Wait are the follower's (zero: two minutes, five seconds).
|
||||
Bound, Wait time.Duration
|
||||
}
|
||||
|
||||
// FindingWriter is the writer the runner's findings go through: views.HeldWriter over the contract's Finding.
|
||||
type FindingWriter interface {
|
||||
Write(ctx context.Context, finding condseat.Finding) error
|
||||
Clear(ctx context.Context, scope, id, kind, why string) error
|
||||
}
|
||||
|
||||
// Start asks the first views, each until answered, and follows the owners' views families on the runner's own
|
||||
// consumers. It returns at once: the picture fills as the owners answer.
|
||||
func Start(parent context.Context, conn *bus.Conn, logf func(string, ...any)) (*Views, func()) {
|
||||
return StartWith(parent, DepsOf(conn, logf), logf)
|
||||
}
|
||||
|
||||
// DepsOf is the views' reach through the runner's own connection: its tool calls, its consumer and its held finding writer, whose
|
||||
// state adapter, JetStream and node are this connection's, in this one place (design 47 G A-16, A-27).
|
||||
func DepsOf(conn *bus.Conn, logf func(string, ...any)) Deps {
|
||||
me := conn.Module()
|
||||
d := Deps{Module: me, Node: conn.Node(), Build: process.OwnBuild(),
|
||||
Ask: func(address string, body any) (json.RawMessage, error) {
|
||||
got, err := conn.Ask(address, body, "")
|
||||
return got.Result, err
|
||||
},
|
||||
Consume: func(deliver func(bus.Envelope) error) (func(), error) { return conn.ConsumeAs(me, deliver) },
|
||||
}
|
||||
js, err := conn.JetStream()
|
||||
if err != nil {
|
||||
logf("[mesh-tools] the runner's findings are not written (%v): the first views' findings are in the log only", err)
|
||||
return d
|
||||
}
|
||||
w, err := views.NewHeldWriter(views.WriterConfig[condseat.Finding, condseat.FindingCleared, condseat.FindingsStated, condseat.FindingsAskedEvent]{
|
||||
Asked: condseat.SubjectFindingsAsked, Module: me, Node: conn.Node(), State: findingsState{conn: conn, module: me}, JS: js, Logf: logf,
|
||||
Kinds: append([]string{views.UnansweredKind}, approval.Kinds...)})
|
||||
if err != nil {
|
||||
logf("[mesh-tools] the runner's findings are not written (%v): the first views' findings are in the log only", err)
|
||||
return d
|
||||
}
|
||||
d.Findings = w
|
||||
d.StartFindings = func(ctx context.Context) (func(), error) { return w.Close, w.Start(ctx) }
|
||||
return d
|
||||
}
|
||||
|
||||
// findingsState is the runner's own state `findings` (views.FindingState) over its connection.
|
||||
type findingsState struct {
|
||||
conn *bus.Conn
|
||||
module string
|
||||
}
|
||||
|
||||
func (s findingsState) Get(key string) (*stdio.StateEntry, error) {
|
||||
e, err := s.conn.StateGet(s.module, FindingsState, key)
|
||||
if err != nil || e == nil {
|
||||
return nil, err
|
||||
}
|
||||
return &stdio.StateEntry{Key: e.Key, Value: e.Value, Revision: e.Revision, At: e.At}, nil
|
||||
}
|
||||
|
||||
func (s findingsState) Put(key string, value any) (uint64, error) {
|
||||
raw, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return s.conn.StatePut(s.module, FindingsState, key, raw)
|
||||
}
|
||||
|
||||
func (s findingsState) Delete(key string) error {
|
||||
return s.conn.StateDelete(s.module, FindingsState, key)
|
||||
}
|
||||
|
||||
func (s findingsState) Keys() ([]string, error) { return s.conn.StateKeys(s.module, FindingsState) }
|
||||
|
||||
// StartWith is Start over deps.
|
||||
func StartWith(parent context.Context, d Deps, logf func(string, ...any)) (*Views, func()) {
|
||||
ctx, cancel := context.WithCancel(parent)
|
||||
v := &Views{logf: logf, ctx: ctx, cancel: cancel, followers: map[string]*views.Follower{}, byState: map[string]*views.Follower{},
|
||||
adopted: map[string]*adoption{}, hearers: map[string][]*adoption{}, deps: d}
|
||||
stopFindings := func() {}
|
||||
if d.StartFindings != nil {
|
||||
if stop, err := d.StartFindings(ctx); err != nil {
|
||||
logf("[mesh-tools] the runner's finding writer does not start (%v): the first views' findings are in the log only", err)
|
||||
} else {
|
||||
stopFindings = stop
|
||||
}
|
||||
}
|
||||
// The catalog half is followed by the shared branch follower on this process's own wiring (design 47 G §614, A-103): the
|
||||
// registrations, the gate's source and a view per watched branch, each heard on this consumer.
|
||||
branches, err := meshcatalogclient.NewBranches(wiring{v}, func(key string, body any) (json.RawMessage, error) { return d.Ask(key, body) })
|
||||
if err != nil {
|
||||
logf("[mesh-tools] the catalog half is not followed (%v): the console cannot say which modules exist", err)
|
||||
} else {
|
||||
branches.Logf = func(format string, args ...any) { logf("[mesh-tools] "+format, args...) }
|
||||
branches.Changed = func(_, module string, entry json.RawMessage, deleted bool, _ time.Time) {
|
||||
v.claimed(module, entry, deleted)
|
||||
}
|
||||
v.branches = branches
|
||||
}
|
||||
stop, err := d.Consume(v.hear)
|
||||
if err != nil {
|
||||
logf("[mesh-tools] the owners' views are not followed (%v): the pictures are the first views alone until the runner starts again", err)
|
||||
stop = func() {}
|
||||
}
|
||||
return v, func() { stop(); cancel(); v.wg.Wait(); stopFindings() }
|
||||
}
|
||||
|
||||
// hear applies one views event; a gap asks the first view once more. Every event heard wakes what waits on the picture
|
||||
// (heardOne).
|
||||
func (v *Views) hear(env bus.Envelope) error {
|
||||
defer v.heardOne()
|
||||
v.mu.Lock()
|
||||
hs := append([]*adoption(nil), v.hearers[env.Key]...)
|
||||
v.mu.Unlock()
|
||||
if len(hs) > 0 { // an event of the catalog half or the gate's source, heard by the views the branch follower adopted
|
||||
for _, a := range hs {
|
||||
if err := a.hear(v.ctx, env.Body); err != nil {
|
||||
v.logf("[mesh-tools] %v", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
ev, err := views.DecodeEvent(env.Body)
|
||||
if err != nil {
|
||||
return nil // not a views event: nothing to hold, nothing to retry
|
||||
}
|
||||
v.mu.Lock()
|
||||
f := v.byState[ev.State]
|
||||
v.mu.Unlock()
|
||||
if f == nil || f.Owner != ev.Owner {
|
||||
return nil
|
||||
}
|
||||
defer v.heardKey(ev.State, ev.Key)
|
||||
if gap, why := f.Apply(ev); gap {
|
||||
v.logf("[mesh-tools] a gap in %s's %s (%s): its first view %s is asked once more", f.Owner, f.State, why, f.Tool)
|
||||
go func() { _ = f.Again(v.ctx, "a gap: "+why) }()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// finding writes or clears the runner's finding of a first view unanswered past its bound, through its writer (design 47 G A-28).
|
||||
func (v *Views) finding(f condseat.Finding, tool string, raised bool) {
|
||||
if v.deps.Findings == nil {
|
||||
v.logf("[mesh-tools] %s", f.Summary)
|
||||
return
|
||||
}
|
||||
var err error
|
||||
if raised {
|
||||
err = v.deps.Findings.Write(context.Background(), f)
|
||||
} else {
|
||||
err = v.deps.Findings.Clear(context.Background(), f.Scope, f.ID, f.Kind, "the owner answered")
|
||||
}
|
||||
if err != nil {
|
||||
v.logf("[mesh-tools] the finding of the first view %s is not kept: %v", tool, err)
|
||||
}
|
||||
}
|
||||
|
||||
// State is the picture of the state spec names (`<owner>.<state>`), every current value by key; an error says the owner has
|
||||
// not answered yet, so nothing is claimed of it.
|
||||
func (v *Views) State(name string) (map[string][]byte, error) {
|
||||
v.mu.Lock()
|
||||
if state, ok := strings.CutPrefix(name, AssignerSeat+"."); ok { // the seat's name stands for its holder's
|
||||
if v.holder == nil {
|
||||
v.mu.Unlock()
|
||||
return nil, fmt.Errorf("the catalog half names no holder of the seat %s yet, so %s has no first view", AssignerSeat, name)
|
||||
}
|
||||
name = v.holder.module + "." + state
|
||||
}
|
||||
f := v.followers[name]
|
||||
v.mu.Unlock()
|
||||
if f == nil {
|
||||
return nil, fmt.Errorf("the tool runner holds no first view of %s", name)
|
||||
}
|
||||
select {
|
||||
case <-f.Ready():
|
||||
default:
|
||||
return nil, fmt.Errorf("%s's first view %s has not been answered yet", f.Owner, f.Tool)
|
||||
}
|
||||
out := map[string][]byte{}
|
||||
for k, raw := range f.Values() {
|
||||
out[k] = raw
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Entry is one value of the picture, nil when the owner holds none.
|
||||
func (v *Views) Entry(name, key string) ([]byte, error) {
|
||||
all, err := v.State(name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return all[key], nil
|
||||
}
|
||||
|
||||
// Status is each first view as it stands, by `<owner>.<state>`, sorted.
|
||||
func (v *Views) Status() []views.Status {
|
||||
var out []views.Status
|
||||
for _, f := range v.all() {
|
||||
out = append(out, f.Status())
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Owner+out[i].State < out[j].Owner+out[j].State })
|
||||
return out
|
||||
}
|
||||
|
||||
// OwnBuild is the build the tool runner was started with: the fingerprint of the want its node-engine applied, or why it is
|
||||
// unknown (`own build unknown: started without MESH_FINGERPRINT`).
|
||||
func (v *Views) OwnBuild() process.Build { return v.deps.Build }
|
||||
|
||||
// Entries are the entries by fingerprint the catalog half's follower keeps, with their state and since when (design 47 G A-103).
|
||||
func (v *Views) Entries() any {
|
||||
if v.branches == nil {
|
||||
return []meshcatalogclient.EntryStatus{}
|
||||
}
|
||||
return v.branches.EntryStatuses()
|
||||
}
|
||||
|
||||
// all is every first view held now: the owners', then the adopted ones.
|
||||
func (v *Views) all() []*views.Follower {
|
||||
v.mu.Lock()
|
||||
defer v.mu.Unlock()
|
||||
out := make([]*views.Follower, 0, len(v.followers))
|
||||
for _, f := range v.followers {
|
||||
out = append(out, f)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Again asks every first view once more, by hand, for why.
|
||||
func (v *Views) Again(ctx context.Context, why string) []error {
|
||||
var errs []error
|
||||
for _, f := range v.all() {
|
||||
if err := f.Again(ctx, why); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
return errs
|
||||
}
|
||||
|
||||
// unansweredFinding is the finding of a first view unanswered past its bound, in the contract's form (design 47 G A-28): about the
|
||||
// runner's module, a warning, its id the view's name escaped as a key's name is.
|
||||
func unansweredFinding(module, node string, st views.Status, at time.Time) condseat.Finding {
|
||||
return condseat.Finding{Kind: views.UnansweredKind, Scope: scopeModule, ID: keys.EscapeName(module + "." + st.Owner + "." + st.State),
|
||||
Severity: condseat.FindingSeverityWarning, Source: "views", Observer: module,
|
||||
Subject: condseat.Subject{Scope: scopeModule, ID: module, Machine: node},
|
||||
Summary: fmt.Sprintf("%s has no first view of %s's %s: %s has not answered since %s (%s)", module, st.Owner, st.State, st.Tool,
|
||||
st.Asked.Format(time.RFC3339), strings.TrimSpace(st.Error)),
|
||||
Headline: fmt.Sprintf("%s waits for %s", module, st.Owner), At: at.UTC()}
|
||||
}
|
||||
|
||||
// scopeModule is the scope of a finding about a module.
|
||||
const scopeModule = "module"
|
||||
|
||||
// viewWaitsFinding is the finding of a view waited for past views.WaitsBound, in the contract's form: about the runner's module,
|
||||
// a warning, its id the view's name escaped as a key's name is.
|
||||
func viewWaitsFinding(module, node string, st views.Status, at time.Time) condseat.Finding {
|
||||
return condseat.Finding{Kind: views.ViewWaitsKind, Scope: scopeModule, ID: keys.EscapeName(module + "." + st.Owner + "." + st.State),
|
||||
Severity: condseat.FindingSeverityWarning, Source: "views", Observer: module,
|
||||
Subject: condseat.Subject{Scope: scopeModule, ID: module, Machine: node},
|
||||
Summary: fmt.Sprintf("%s waits for the view %s.%s at revision %s since %s and acts on nothing new until it holds it; it holds %s",
|
||||
module, st.Owner, st.State, st.WaitsFor, st.WaitingSince.UTC().Format(time.RFC3339), orNone(st.View)),
|
||||
Headline: fmt.Sprintf("%s waits for a view of %s", module, st.Owner), At: at.UTC()}
|
||||
}
|
||||
|
||||
func orNone(s string) string {
|
||||
if s == "" {
|
||||
return "none"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// adoption is a view the branch follower put on this process's consumer: its follower, the event it hears and its handler.
|
||||
type adoption struct {
|
||||
name, key string
|
||||
f *views.Follower
|
||||
hear func(context.Context, []byte) error
|
||||
cancel context.CancelFunc
|
||||
}
|
||||
|
||||
// wiring is the branch follower's wiring over this process's own consumer (meshcatalogclient.Wiring; design 47 G §614, A-103):
|
||||
// an adopted view is held with the owners', takes its first view at once, hears its seat's event on the runner's consumer,
|
||||
// writes its findings through the runner's held writer, and a dropped one clears them.
|
||||
type wiring struct{ v *Views }
|
||||
|
||||
func (w wiring) Adopt(f *views.Follower, seat, event string, hear func(context.Context, []byte) error) error {
|
||||
v := w.v
|
||||
if f == nil || f.Owner == "" || f.State == "" || seat == "" || event == "" || hear == nil {
|
||||
return fmt.Errorf("the tool runner adopts a view that names its owner, state, seat's event and handler")
|
||||
}
|
||||
name := f.Owner + "." + f.State
|
||||
key := seat + "." + event
|
||||
v.mu.Lock()
|
||||
if v.followers[name] != nil {
|
||||
v.mu.Unlock()
|
||||
return fmt.Errorf("%s is followed already", name)
|
||||
}
|
||||
if f.Logf == nil {
|
||||
f.Logf = v.logf
|
||||
}
|
||||
if f.Bound == 0 {
|
||||
f.Bound = v.deps.Bound
|
||||
}
|
||||
if f.Wait == 0 {
|
||||
f.Wait = v.deps.Wait
|
||||
}
|
||||
held := func() bool { v.mu.Lock(); defer v.mu.Unlock(); return v.followers[name] == f }
|
||||
f.Raise = func(st views.Status) {
|
||||
if held() {
|
||||
v.finding(unansweredFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, true)
|
||||
}
|
||||
}
|
||||
f.Clear = func(st views.Status) {
|
||||
v.finding(unansweredFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
|
||||
}
|
||||
f.RaiseWaits = func(st views.Status) {
|
||||
if held() {
|
||||
v.finding(viewWaitsFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, true)
|
||||
}
|
||||
}
|
||||
f.ClearWaits = func(st views.Status) {
|
||||
v.finding(viewWaitsFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(v.ctx)
|
||||
a := &adoption{name: name, key: key, f: f, hear: hear, cancel: cancel}
|
||||
v.followers[name] = f
|
||||
v.adopted[name] = a
|
||||
v.hearers[key] = append(v.hearers[key], a)
|
||||
v.mu.Unlock()
|
||||
v.wg.Add(1)
|
||||
go func() { defer v.wg.Done(); _ = f.Start(ctx) }()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w wiring) Drop(owner, state string) error {
|
||||
v := w.v
|
||||
name := owner + "." + state
|
||||
v.mu.Lock()
|
||||
a := v.adopted[name]
|
||||
if a == nil {
|
||||
v.mu.Unlock()
|
||||
return fmt.Errorf("%s is not a view this tool runner adopted", name)
|
||||
}
|
||||
delete(v.followers, name)
|
||||
delete(v.adopted, name)
|
||||
v.hearers[a.key] = slices.DeleteFunc(v.hearers[a.key], func(o *adoption) bool { return o == a })
|
||||
v.mu.Unlock()
|
||||
a.cancel()
|
||||
st := a.f.Status()
|
||||
if st.Raised {
|
||||
v.finding(unansweredFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
|
||||
}
|
||||
if st.WaitsRaised {
|
||||
v.finding(viewWaitsFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Halves is the catalog half of each followed branch, by branch: its revision, with the seat definitions it was judged
|
||||
// with, and its entries; mesh_wait checks an event name's emitter against them (design 34 every-event §14). An error says
|
||||
// the half has no first view yet, never that no module exists.
|
||||
func (v *Views) Halves() (map[string]meshcatalogclient.Half, error) {
|
||||
if v.branches == nil {
|
||||
return nil, fmt.Errorf("the tool runner does not follow the catalog half")
|
||||
}
|
||||
if err := v.branches.Ready(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := map[string]meshcatalogclient.Half{}
|
||||
for _, branch := range v.branches.Followed() {
|
||||
if c, ok := v.branches.Catalog(branch); ok {
|
||||
out[branch], _ = c.View() // a branch at none-yet is ready and empty (§614)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Modules is every module the catalog half holds, by name, with its entry: the entries of the watched branches' newest trunk
|
||||
// revisions, the branch followed first naming a module more than one hold (design 47 G §196, §594). An error says the half has
|
||||
// no first view yet, never that no module exists.
|
||||
func (v *Views) Modules() (map[string]catalogseat.CatalogEntry, error) {
|
||||
if v.branches == nil {
|
||||
return nil, fmt.Errorf("the tool runner does not follow the catalog half")
|
||||
}
|
||||
if err := v.branches.Ready(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := map[string]catalogseat.CatalogEntry{}
|
||||
for _, branch := range v.branches.Followed() {
|
||||
c, ok := v.branches.Catalog(branch)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
half, _ := c.View() // a branch at none-yet is ready and empty (§614)
|
||||
for m, e := range half.Entries {
|
||||
if _, taken := out[m]; !taken {
|
||||
out[m] = e
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// holding is the views of the module that claims the assigner seat, and what stops them.
|
||||
type holding struct {
|
||||
module string
|
||||
cancel context.CancelFunc
|
||||
views []*views.Follower
|
||||
}
|
||||
|
||||
// claimed is the branch follower's word of one entry changed: an entry that serves a verb of the assigner seat makes its module
|
||||
// the holder whose views the runner follows (a holder before it is dropped), the deletion of the holder's entry drops them
|
||||
// (design 47 G §616, §428). It acts on the event alone; nothing waits on a timer for the claim.
|
||||
func (v *Views) claimed(module string, entry json.RawMessage, deleted bool) {
|
||||
claims := false
|
||||
if !deleted {
|
||||
var e catalogseat.CatalogEntry
|
||||
if json.Unmarshal(entry, &e) == nil {
|
||||
claims = slices.ContainsFunc(e.Serves, func(s string) bool { return strings.HasPrefix(s, AssignerSeat+".") })
|
||||
}
|
||||
}
|
||||
v.mu.Lock()
|
||||
cur := v.holder
|
||||
switch {
|
||||
case claims && cur != nil && cur.module == module, !claims && (cur == nil || cur.module != module), v.ctx.Err() != nil:
|
||||
v.mu.Unlock()
|
||||
return
|
||||
}
|
||||
if cur != nil {
|
||||
v.dropHolderLocked()
|
||||
}
|
||||
if claims {
|
||||
v.startHolderLocked(module)
|
||||
}
|
||||
v.mu.Unlock()
|
||||
}
|
||||
|
||||
// startHolderLocked follows the first views of module, the holder of the seat, on the runner's own consumer.
|
||||
func (v *Views) startHolderLocked(module string) {
|
||||
ctx, cancel := context.WithCancel(v.ctx)
|
||||
h := &holding{module: module, cancel: cancel}
|
||||
d := v.deps
|
||||
for _, s := range Specs {
|
||||
s := s
|
||||
call := func(tool string, body any) (json.RawMessage, error) { return d.Ask(module+"."+tool, body) }
|
||||
f := &views.Follower{Owner: module, Tool: s.Tool, State: s.State, Ask: views.AskVia(call, s.Tool), Logf: v.logf,
|
||||
Bound: d.Bound, Wait: d.Wait,
|
||||
Raise: func(st views.Status) {
|
||||
v.finding(unansweredFinding(d.Module, d.Node, st, time.Now()), st.Tool, true)
|
||||
},
|
||||
Clear: func(st views.Status) {
|
||||
v.finding(unansweredFinding(d.Module, d.Node, st, time.Now()), st.Tool, false)
|
||||
}}
|
||||
v.followers[module+"."+s.State] = f
|
||||
v.byState[s.State] = f
|
||||
h.views = append(h.views, f)
|
||||
v.wg.Add(1)
|
||||
go func() { defer v.wg.Done(); _ = f.Start(ctx) }()
|
||||
}
|
||||
v.holder = h
|
||||
v.logf("[mesh-tools] the seat %s is held by %s: its views are followed", AssignerSeat, module)
|
||||
}
|
||||
|
||||
// dropHolderLocked stops the views of the holder and clears the findings they raised.
|
||||
func (v *Views) dropHolderLocked() {
|
||||
h := v.holder
|
||||
v.holder = nil
|
||||
h.cancel()
|
||||
for _, f := range h.views {
|
||||
delete(v.followers, f.Owner+"."+f.State)
|
||||
delete(v.byState, f.State)
|
||||
if st := f.Status(); st.Raised {
|
||||
go v.finding(unansweredFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
|
||||
}
|
||||
}
|
||||
v.logf("[mesh-tools] %s no longer holds the seat %s: its views are no longer followed", h.module, AssignerSeat)
|
||||
}
|
||||
@@ -0,0 +1,204 @@
|
||||
//go:build integration
|
||||
|
||||
package firstview
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/process"
|
||||
"git.novox.be/novox/mesh-sdk/go/views"
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
|
||||
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
delivery "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// issue 1966 TLS-5 [i] (ADR 0615 §2, §4, §6): against the real bus, with the owners down and then up, the runner's own
|
||||
// consumers on the owners' own views streams carry the manifest's four `consumes`: the first views are asked until answered,
|
||||
// the unanswered ones are findings that the answer clears, and each owner's next event reaches its picture.
|
||||
func TestTheRunnersConsumerCarriesTheOwnersViewsFamiliesWhenTheOwnersAreDownThenUp(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
url := mt.URL(t)
|
||||
|
||||
// The owners publish their views on their own streams (views.Emit: `mesh.stream.<owner>.event.views.<state>`, on
|
||||
// assigner-views and indexer-views), never on EVENTS. The consumers on them are the mesh's, one filter per family the
|
||||
// manifest's `consumes` names, and the runner's membership names them as the mesh issues it (ADR 0363 §8).
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var manifest struct {
|
||||
Consumes []string `json:"consumes"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &manifest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// the manifest names the assigner's views by the seat; the module the catalog half says claims it is the fixture's
|
||||
const holder = "assigner-fixture"
|
||||
streamOf := map[string]string{"seat:" + AssignerSeat: catalogue.StreamBusName("assigner-views")}
|
||||
filters := map[string][]string{}
|
||||
families := 0
|
||||
for _, c := range manifest.Consumes {
|
||||
emitter, event, isView := strings.Cut(c, ".views.")
|
||||
if !isView { // the seats' events (the seat list's, the catalog half's), on EVENTS (cmd/node-tools/seatlist.go)
|
||||
continue
|
||||
}
|
||||
stream := streamOf[emitter]
|
||||
if stream == "" {
|
||||
t.Fatalf("the manifest consumes %s, whose views stream the test does not know", c)
|
||||
}
|
||||
filters[stream] = append(filters[stream], process.ModuleStreamSubject(holder, views.EventPrefix+event))
|
||||
families++
|
||||
}
|
||||
if families != len(Specs) {
|
||||
t.Fatalf("the manifest consumes %d families, the runner holds %d first views", families, len(Specs))
|
||||
}
|
||||
runner := mt.MembershipOf("node-tools", "anchor", false, nil)
|
||||
for _, seat := range []string{"seat:" + AssignerSeat} {
|
||||
stream := streamOf[seat]
|
||||
mesh.ModuleStream(t, holder, stream)
|
||||
mesh.ConsumerOn(t, stream, "anchor", "node-tools", filters[stream], 2*time.Second)
|
||||
runner.Hears = append(runner.Hears, bus.HeardIssued{Stream: stream, Consumer: "anchor_node-tools"})
|
||||
}
|
||||
mesh.Consumer(t, "anchor", "node-tools", []string{"mesh.mod.mesh-knowledge.event.reviewed"}, 2*time.Second) // its `waits`, on EVENTS
|
||||
mesh.Issue(t, runner)
|
||||
|
||||
conn, err := bus.Connect(bus.Credential{URL: url, Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn.Logf = func(string, ...any) {}
|
||||
t.Cleanup(conn.Close)
|
||||
conn.Follow("node-tools")
|
||||
mt.Until(t, func() error {
|
||||
if conn.Membership("node-tools") == nil {
|
||||
return errFindings(-1)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
var mu sync.Mutex
|
||||
findings := map[string]bool{}
|
||||
deps := DepsOf(conn, t.Logf)
|
||||
deps.Bound, deps.Wait = 100*time.Millisecond, 20*time.Millisecond
|
||||
deps.Findings = writes{&mu, findings}
|
||||
deps.StartFindings = nil
|
||||
v, stop := StartWith(context.Background(), deps, func(string, ...any) {})
|
||||
t.Cleanup(stop)
|
||||
|
||||
// The owners are down: no tool answers, nothing is claimed, and past the bound each view is a finding.
|
||||
if _, err := v.State(AssignmentsRead); err == nil {
|
||||
t.Fatal("a state its owner has not answered is claimed")
|
||||
}
|
||||
mt.Until(t, func() error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(findings) != catalogViews {
|
||||
return errFindings(len(findings))
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
// The owners come up and answer their first views.
|
||||
nc, err := nats.Connect(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(nc.Close)
|
||||
respond := func(subject string, answer any) {
|
||||
if _, err := nc.Subscribe(subject, func(m *nats.Msg) {
|
||||
raw, _ := json.Marshal(answer)
|
||||
body, _ := json.Marshal(map[string]json.RawMessage{"result": raw})
|
||||
_ = m.Respond(body)
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
// the catalog half answers: the gate's source names main, whose entries say the fixture claims the assigner seat
|
||||
respond("mesh.seat.mesh-delivery.tool.source", delivery.SourceAnswer{Branches: []string{"main"}})
|
||||
respond("mesh.seat.mesh-catalog.tool.registrations", catalogseat.RegistrationsAnswer{Registrations: []catalogseat.Registration{}})
|
||||
respond("mesh.seat.mesh-catalog.tool.revision", catalogseat.RevisionAnswer{
|
||||
Revision: &catalogseat.CatalogRevision{Revision: 3, Kind: meshcatalogclient.KindTrunk, Branch: "main"},
|
||||
Entries: []catalogseat.CatalogEntry{{Module: holder, Serves: []string{AssignerSeat + ".assignments"}}}})
|
||||
for _, s := range Specs {
|
||||
s := s
|
||||
if _, err := nc.Subscribe("mesh.mod."+holder+".tool."+s.Tool, func(m *nats.Msg) {
|
||||
answer, _ := json.Marshal(views.Answer{V: 1, Owner: holder, State: s.State, Revision: 4, Created: time.Unix(1000, 0).UTC(),
|
||||
Values: map[string]json.RawMessage{"first": json.RawMessage(`{"n":1}`)}})
|
||||
body, _ := json.Marshal(map[string]json.RawMessage{"result": answer})
|
||||
_ = m.Respond(body)
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
_ = nc.Flush()
|
||||
for _, s := range Specs {
|
||||
name := s.Name()
|
||||
mt.Until(t, func() error { _, err := v.Entry(name, "first"); return err })
|
||||
}
|
||||
mt.Until(t, func() error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(findings) != 0 { // every view answered clears its finding
|
||||
return errFindings(len(findings))
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
// Each owner's next event arrives on the runner's consumer: all four families are bound.
|
||||
for _, s := range Specs {
|
||||
ev, _ := json.Marshal(views.Event{V: 1, Owner: holder, State: s.State, Key: "next", Revision: 5, Created: time.Unix(1000, 0).UTC(),
|
||||
Value: json.RawMessage(`{"n":2}`)})
|
||||
mesh.Emit(t, process.ModuleStreamSubject(holder, views.EventPrefix+s.State), json.RawMessage(ev))
|
||||
}
|
||||
for _, s := range Specs {
|
||||
name := s.Name()
|
||||
mt.Until(t, func() error {
|
||||
got, err := v.Entry(name, "next")
|
||||
if err != nil || string(got) != `{"n":2}` {
|
||||
return errFindings(-1)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// catalogViews is the first views the branch follower adopts before any branch is known: the registrations and the gate's source
|
||||
// (design 47 G §614), each a finding while the seats' verbs go unanswered.
|
||||
const catalogViews = 2
|
||||
|
||||
type errFindings int
|
||||
|
||||
func (e errFindings) Error() string { return "not yet" }
|
||||
|
||||
// writes is the runner's finding writer in this test: the findings written and not cleared.
|
||||
type writes struct {
|
||||
mu *sync.Mutex
|
||||
by map[string]bool
|
||||
}
|
||||
|
||||
func (w writes) Write(_ context.Context, f condseat.Finding) error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
w.by[views.FindingKey(f.Scope, f.ID, f.Kind)] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w writes) Clear(_ context.Context, scope, id, kind, _ string) error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
delete(w.by, views.FindingKey(scope, id, kind))
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,295 @@
|
||||
package firstview
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
|
||||
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
|
||||
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
delivery "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
|
||||
"git.novox.be/novox/mesh-sdk/go/views"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
|
||||
// owners is the four owners' first-view tools as a test answers them: a state's values at a revision, or nothing while down.
|
||||
type owners struct {
|
||||
mu sync.Mutex
|
||||
down bool
|
||||
state map[string]map[string]string // tool → key → value
|
||||
rev map[string]uint64
|
||||
asked map[string]int
|
||||
deliver func(bus.Envelope) error
|
||||
finds map[string]bool
|
||||
// catalog is the revision of the watched branch main the seat mesh-catalog answers, and modules its entries
|
||||
catalog int64
|
||||
modules []string
|
||||
// holder is the module among them that claims the assigner seat
|
||||
holder string
|
||||
}
|
||||
|
||||
func newOwners() *owners {
|
||||
return &owners{state: map[string]map[string]string{"assignments": {"anchor": `{"p":1}`}, "seat-holders": {"seat": `{"h":1}`}},
|
||||
rev: map[string]uint64{"assignments": 4, "seat-holders": 2}, asked: map[string]int{}, finds: map[string]bool{},
|
||||
catalog: 3, modules: []string{"web", fixtureHolder}, holder: fixtureHolder}
|
||||
}
|
||||
|
||||
// Write and Clear are the runner's finding writer in a test: the findings written and not cleared, by their key.
|
||||
func (o *owners) Write(_ context.Context, f condseat.Finding) error {
|
||||
o.mu.Lock()
|
||||
defer o.mu.Unlock()
|
||||
o.finds[views.FindingKey(f.Scope, f.ID, f.Kind)] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
func (o *owners) Clear(_ context.Context, scope, id, kind, _ string) error {
|
||||
o.mu.Lock()
|
||||
defer o.mu.Unlock()
|
||||
delete(o.finds, views.FindingKey(scope, id, kind))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (o *owners) ask(address string, _ any) (json.RawMessage, error) {
|
||||
o.mu.Lock()
|
||||
defer o.mu.Unlock()
|
||||
tool := address[strings.LastIndex(address, ".")+1:]
|
||||
o.asked[tool]++
|
||||
if o.down {
|
||||
return nil, errors.New("503 no responders")
|
||||
}
|
||||
switch address {
|
||||
case "seat:mesh-delivery.source":
|
||||
return json.Marshal(delivery.SourceAnswer{Branches: []string{"main"}})
|
||||
case "seat:mesh-catalog.registrations":
|
||||
return json.Marshal(catalogseat.RegistrationsAnswer{Registrations: []catalogseat.Registration{}})
|
||||
case "seat:mesh-catalog.revision":
|
||||
a := catalogseat.RevisionAnswer{Revision: &catalogseat.CatalogRevision{Revision: o.catalog, Kind: meshcatalogclient.KindTrunk, Branch: "main"}}
|
||||
for _, m := range o.modules {
|
||||
e := catalogseat.CatalogEntry{Module: m, Manifest: map[string]any{"module": m, "tools": []string{m + "_check"}}}
|
||||
if m == o.holder {
|
||||
e.Serves = []string{AssignerSeat + ".assignments", AssignerSeat + ".seat-holders"}
|
||||
}
|
||||
a.Entries = append(a.Entries, e)
|
||||
}
|
||||
return json.Marshal(a)
|
||||
}
|
||||
vals := map[string]json.RawMessage{}
|
||||
for k, v := range o.state[tool] {
|
||||
vals[k] = json.RawMessage(v)
|
||||
}
|
||||
return json.Marshal(views.Answer{V: 1, Owner: "o", State: tool, Revision: o.rev[tool], Created: time.Unix(1000, 0).UTC(), Values: vals})
|
||||
}
|
||||
|
||||
func (o *owners) say(t *testing.T, state, key, value string, revision uint64, gap bool) {
|
||||
t.Helper()
|
||||
ev := views.Event{V: 1, Owner: owner(state), State: state, Key: key, Revision: revision, Created: time.Unix(1000, 0).UTC(), Gap: gap}
|
||||
if value != "" {
|
||||
ev.Value = json.RawMessage(value)
|
||||
}
|
||||
raw, _ := json.Marshal(ev)
|
||||
if err := o.deliver(bus.Envelope{Key: ev.Owner + ".views." + state, Body: raw}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// fixtureHolder is the module the fixture catalog half says claims the assigner seat: no name is in the code.
|
||||
const fixtureHolder = "assigner-fixture"
|
||||
|
||||
func owner(state string) string { return fixtureHolder }
|
||||
|
||||
func eventually(t *testing.T, what string, ok func() bool) {
|
||||
t.Helper()
|
||||
for end := time.Now().Add(3 * time.Second); !ok(); time.Sleep(5 * time.Millisecond) {
|
||||
if time.Now().After(end) {
|
||||
t.Fatal(what)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// issue 1966 TLS-5 (ADR 0615 §2, §4, §5, §6): the runner holds the four states from the owners' first views and their
|
||||
// events alone; nothing is claimed of a state not yet answered; a gap asks the first view once more; an owner that stays down
|
||||
// past the bound is a finding, cleared by the answer; the status and the hand ask show them.
|
||||
func TestTheRunnerHoldsTheFourStatesFromTheOwnersFirstViewsAndTheirEvents(t *testing.T) {
|
||||
o := newOwners()
|
||||
o.down = true
|
||||
var logs []string
|
||||
var lmu sync.Mutex
|
||||
deps := Deps{Module: "node-tools", Ask: o.ask, Bound: 30 * time.Millisecond, Wait: 10 * time.Millisecond,
|
||||
Consume: func(deliver func(bus.Envelope) error) (func(), error) { o.deliver = deliver; return func() {}, nil },
|
||||
Findings: o}
|
||||
v, stop := StartWith(context.Background(), deps, func(f string, a ...any) { lmu.Lock(); logs = append(logs, f); lmu.Unlock() })
|
||||
defer stop()
|
||||
|
||||
// Down: nothing is claimed, and past the bound each unanswered view is a finding.
|
||||
if _, err := v.State(AssignmentsRead); err == nil || !strings.Contains(err.Error(), "names no holder") {
|
||||
t.Fatalf("a state whose holder is not known is claimed: %v", err)
|
||||
}
|
||||
eventually(t, "no finding past the bound", func() bool { o.mu.Lock(); defer o.mu.Unlock(); return len(o.finds) == 2 })
|
||||
|
||||
// Up: answered, the findings are cleared, and the pictures hold the owners' values.
|
||||
o.mu.Lock()
|
||||
o.down = false
|
||||
o.mu.Unlock()
|
||||
eventually(t, "not answered", func() bool { _, err := v.State(SeatHoldersRead); return err == nil })
|
||||
eventually(t, "findings not cleared", func() bool { o.mu.Lock(); defer o.mu.Unlock(); return len(o.finds) == 0 })
|
||||
if got, _ := v.Entry(AssignerSeat+".seat-holders", "none"); got != nil {
|
||||
t.Errorf("an absent key is %q", got)
|
||||
}
|
||||
|
||||
// The catalog half is followed by the shared branch follower on the runner's own consumer: the gate's source names main,
|
||||
// main's view is adopted, and its entries are the modules, each with its manifest (design 47 G §196, §614).
|
||||
eventually(t, "the catalog half holds no modules", func() bool { m, err := v.Modules(); return err == nil && len(m) == 2 })
|
||||
m, _ := v.Modules()
|
||||
if m["web"].Manifest["module"] != "web" {
|
||||
t.Errorf("the module web is %+v", m["web"])
|
||||
}
|
||||
// A catalog-revision of the branch is heard on the same consumer and asks the view once more.
|
||||
o.mu.Lock()
|
||||
o.catalog, o.modules = 4, []string{"web", "db", fixtureHolder}
|
||||
o.mu.Unlock()
|
||||
ev, _ := json.Marshal(catalogseat.CatalogRevisionEvent{V: 1, Branch: "main", Kind: meshcatalogclient.KindTrunk, Revision: 4})
|
||||
if err := o.deliver(bus.Envelope{Key: catalogseat.Seat + "." + catalogseat.EventCatalogRevision, Body: ev}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eventually(t, "the catalog-revision was not taken", func() bool { m, err := v.Modules(); return err == nil && len(m) == 3 })
|
||||
// And the halves by branch, with their revision, which mesh_wait names checking an emitter (design 34 every-event §14).
|
||||
if h, err := v.Halves(); err != nil || h["main"].Revision.Revision != 4 || len(h["main"].Entries) != 3 {
|
||||
t.Errorf("the halves: %+v, %v", h, err)
|
||||
}
|
||||
|
||||
// Events: the next revision is applied; an old one passed over; a deletion removes; none of it asks again.
|
||||
asked := func(tool string) int { o.mu.Lock(); defer o.mu.Unlock(); return o.asked[tool] }
|
||||
before := asked("assignments")
|
||||
o.say(t, "assignments", "laptop", `{"p":2}`, 5, false)
|
||||
o.say(t, "assignments", "laptop", `{"p":0}`, 5, false)
|
||||
o.say(t, "assignments", "anchor", "", 6, false)
|
||||
// (a deleted mark carries no value: Deleted is the encoding's, set below)
|
||||
if got, _ := v.Entry(AssignerSeat+".assignments", "laptop"); string(got) != `{"p":2}` {
|
||||
t.Errorf("the next revision is not applied: %q", got)
|
||||
}
|
||||
if n := asked("assignments"); n != before {
|
||||
t.Errorf("an event in order asked the first view again (%d asks, was %d)", n, before)
|
||||
}
|
||||
|
||||
// A gap (revision 9 where 7 was next) asks the first view once more, said in the log.
|
||||
o.say(t, "assignments", "x", `{"p":9}`, 9, false)
|
||||
eventually(t, "a gap did not ask the first view again", func() bool { return asked("assignments") > before })
|
||||
|
||||
// The status says owner, tool, revision and the asks; the hand ask asks every view again.
|
||||
var assignments views.Status
|
||||
for _, s := range v.Status() {
|
||||
if s.State == "assignments" {
|
||||
assignments = s
|
||||
}
|
||||
}
|
||||
if assignments.Owner != fixtureHolder || assignments.Tool != "assignments" || assignments.Asks < 2 || assignments.Answered.IsZero() {
|
||||
t.Errorf("the status of assignments is %+v", assignments)
|
||||
}
|
||||
manifests := asked("seat-holders")
|
||||
if errs := v.Again(context.Background(), "doubted"); len(errs) != 0 {
|
||||
t.Fatalf("the hand ask failed: %v", errs)
|
||||
}
|
||||
if asked("seat-holders") != manifests+1 {
|
||||
t.Errorf("the hand ask did not ask the seat-holders again")
|
||||
}
|
||||
for _, st := range v.Status() {
|
||||
if st.State == "seat-holders" && st.ByHand != 1 {
|
||||
t.Errorf("the hand ask is not counted: %+v", st)
|
||||
}
|
||||
}
|
||||
lmu.Lock()
|
||||
defer lmu.Unlock()
|
||||
if !strings.Contains(strings.Join(logs, "\n"), "a gap in") {
|
||||
t.Errorf("the gap was not said in the log: %v", logs)
|
||||
}
|
||||
}
|
||||
|
||||
// The four first views are the manifest's `consumes` families, and the console reads each by the name here.
|
||||
func TestTheFirstViewsAreTheManifestsViews(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m struct {
|
||||
Consumes []string `json:"consumes"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]bool{AssignmentsRead: true, SeatHoldersRead: true}
|
||||
for _, s := range Specs {
|
||||
if !want[s.Name()] {
|
||||
t.Errorf("the first view %s is read by no name here", s.Name())
|
||||
}
|
||||
delete(want, s.Name())
|
||||
// the family is named by the seat, never by its holder (design 47 G §616)
|
||||
if view := "seat:" + AssignerSeat + ".views." + s.Tool; !slices.Contains(m.Consumes, view) {
|
||||
t.Errorf("the first view %s has no family in the manifest's consumes %v", view, m.Consumes)
|
||||
}
|
||||
}
|
||||
if len(want) > 0 {
|
||||
t.Errorf("no first view reads %v", want)
|
||||
}
|
||||
for _, c := range m.Consumes {
|
||||
if strings.Contains(c, "assigner") && !strings.HasPrefix(c, "seat:") {
|
||||
t.Errorf("the manifest names the assigner by a module: %s", c)
|
||||
}
|
||||
}
|
||||
// the catalog half's follower hears these events and asks these verbs, and the manifest says so (design 47 G §614)
|
||||
for _, ev := range []string{"mesh-catalog.catalog-revision", "mesh-catalog.registration-changed", "mesh-delivery.source-moved"} {
|
||||
if !slices.Contains(m.Consumes, ev) {
|
||||
t.Errorf("the catalog half's event %s is not in the manifest's consumes %v", ev, m.Consumes)
|
||||
}
|
||||
}
|
||||
for _, ev := range []string{"mesh-delivery-indexer.views.modules", "mesh-delivery-indexer.views.manifests"} {
|
||||
if slices.Contains(m.Consumes, ev) {
|
||||
t.Errorf("the manifest still consumes the indexer's %s", ev)
|
||||
}
|
||||
}
|
||||
// design 47 G §616: the holder of the assigner seat is whichever module the catalog half says serves it; when another one
|
||||
// claims it the views are of the new holder, and when none does the seat's names answer that the catalog half names no holder.
|
||||
t.Run("TheAssignersViewsFollowTheModuleThatClaimsTheSeat", func(t *testing.T) {
|
||||
o := newOwners()
|
||||
deps := Deps{Module: "node-tools", Ask: o.ask, Bound: time.Second, Wait: 10 * time.Millisecond,
|
||||
Consume: func(deliver func(bus.Envelope) error) (func(), error) { o.deliver = deliver; return func() {}, nil }, Findings: o}
|
||||
v, stop := StartWith(context.Background(), deps, func(string, ...any) {})
|
||||
defer stop()
|
||||
eventually(t, "not answered", func() bool { _, err := v.State(AssignmentsRead); return err == nil })
|
||||
ownerOf := func() string {
|
||||
for _, s := range v.Status() {
|
||||
if s.State == "assignments" {
|
||||
return s.Owner
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if ownerOf() != fixtureHolder {
|
||||
t.Fatalf("the views are of %q", ownerOf())
|
||||
}
|
||||
revise := func(rev int64, holder string, modules ...string) {
|
||||
o.mu.Lock()
|
||||
o.catalog, o.modules, o.holder = rev, modules, holder
|
||||
o.mu.Unlock()
|
||||
ev, _ := json.Marshal(catalogseat.CatalogRevisionEvent{V: 1, Branch: "main", Kind: meshcatalogclient.KindTrunk, Revision: rev})
|
||||
if err := o.deliver(bus.Envelope{Key: catalogseat.Seat + "." + catalogseat.EventCatalogRevision, Body: ev}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
revise(4, "other-holder", "web", "other-holder")
|
||||
eventually(t, "the views did not move to the new holder", func() bool { return ownerOf() == "other-holder" })
|
||||
eventually(t, "not answered", func() bool { _, err := v.State(SeatHoldersRead); return err == nil })
|
||||
revise(5, "", "web")
|
||||
eventually(t, "the views stayed after the claim went", func() bool {
|
||||
_, err := v.State(AssignmentsRead)
|
||||
return err != nil && strings.Contains(err.Error(), "names no holder")
|
||||
})
|
||||
})
|
||||
}
|
||||
@@ -19,6 +19,9 @@ while IFS= read -r line; do
|
||||
*'"method":"tools/list"'*)
|
||||
printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
|
||||
printf '{"jsonrpc":"2.0","id":"s1","method":"mesh/subscribe","params":{"pattern":"#"}}\n' ;;
|
||||
*'"method":"mesh/heartbeat"'*)
|
||||
params=$(printf '%s' "$line" | sed -n 's/.*"params":\(.*\)}$/\1/p')
|
||||
printf '{"jsonrpc":"2.0","id":"h1","method":"mesh/publish","params":%s}\n' "$params" ;;
|
||||
*'"method":"mesh/event"'*)
|
||||
case "$ANSWER" in
|
||||
error) printf '{"jsonrpc":"2.0","id":%s,"error":{"code":-32000,"message":"Unexpected end of JSON input"}}\n' "$id" ;;
|
||||
@@ -30,11 +33,18 @@ done
|
||||
`
|
||||
|
||||
type subscribing struct {
|
||||
mu sync.Mutex
|
||||
deliver func(json.RawMessage) error
|
||||
mu sync.Mutex
|
||||
deliver func(json.RawMessage) error
|
||||
beat func(json.RawMessage)
|
||||
published []json.RawMessage
|
||||
}
|
||||
|
||||
func (b *subscribing) Publish(json.RawMessage) error { return nil }
|
||||
func (b *subscribing) Publish(p json.RawMessage) error {
|
||||
b.mu.Lock()
|
||||
b.published = append(b.published, p)
|
||||
b.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
func (b *subscribing) Ask(json.RawMessage) (json.RawMessage, error) { return nil, nil }
|
||||
func (b *subscribing) State(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
|
||||
func (b *subscribing) Log(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
|
||||
@@ -48,10 +58,16 @@ func (b *subscribing) Provision(string, json.RawMessage) (json.RawMessage, error
|
||||
func (b *subscribing) ProvisionWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
|
||||
return func() {}, nil
|
||||
}
|
||||
func (b *subscribing) Subscribe(d func(json.RawMessage) error) error {
|
||||
func (b *subscribing) BusState(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
|
||||
func (b *subscribing) BusStateWatch(string, json.RawMessage, func(json.RawMessage) error) (func(), error) {
|
||||
return func() {}, nil
|
||||
}
|
||||
func (b *subscribing) Begin(string) (string, func()) { return "", func() {} }
|
||||
func (b *subscribing) Subscribe(d func(json.RawMessage) error, beat func(json.RawMessage)) error {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
b.deliver = d
|
||||
b.beat = beat
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -95,3 +111,49 @@ func TestAnEventIsTakenByAnyAnswerThatIsNotAnError(t *testing.T) {
|
||||
t.Fatalf("an error answer was not the bundle's refusal of mesh/event: %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (b *subscribing) LogWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
|
||||
return func() {}, nil
|
||||
}
|
||||
|
||||
// The runtime's word on the module's consumer between events reaches the child that subscribed as `mesh/heartbeat`,
|
||||
// unchanged (stdio.OnHeartbeat reads it in the bundle).
|
||||
func TestTheWordOnTheConsumerBetweenEventsReachesTheBundleThatSubscribed(t *testing.T) {
|
||||
entry := filepath.Join(t.TempDir(), "bundle")
|
||||
if err := os.WriteFile(entry, []byte(answeringBundle), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b := &subscribing{}
|
||||
l, err := Start("plex", entry, append(os.Environ(), "ANSWER=result"), b, t.Logf)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(l.Stop)
|
||||
var beat func(json.RawMessage)
|
||||
for i := 0; beat == nil; i++ {
|
||||
b.mu.Lock()
|
||||
beat = b.beat
|
||||
b.mu.Unlock()
|
||||
if i > 100 {
|
||||
t.Fatal("the bundle never subscribed")
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
word := json.RawMessage(`{"at":"2026-11-04T13:00:00Z","missed":true,"why":"the bus ended it"}`)
|
||||
beat(word)
|
||||
for i := 0; ; i++ {
|
||||
b.mu.Lock()
|
||||
got := b.published
|
||||
b.mu.Unlock()
|
||||
if len(got) == 1 {
|
||||
if string(got[0]) != string(word) {
|
||||
t.Fatalf("the bundle was told %s, not %s", got[0], word)
|
||||
}
|
||||
return
|
||||
}
|
||||
if i > 100 {
|
||||
t.Fatal("the bundle was told nothing")
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,6 +19,8 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"golang.org/x/sys/unix"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/wire"
|
||||
@@ -38,6 +40,9 @@ type Tool struct {
|
||||
Name string
|
||||
Description string
|
||||
Input json.RawMessage
|
||||
// Effect is what the tool does, reads, acts or dangerous (novox/hq ADR 0387 §1), where the runtime knows it: its own
|
||||
// tools say it; a bundle's tool is marked by its manifest.
|
||||
Effect string
|
||||
// Run calls the tool for caller: the bus user the call's subject named, or "" when it named none. A
|
||||
// caller is handed to the bundle in the call's `_meta` (MetaCaller), never among its arguments.
|
||||
Run func(caller string, args json.RawMessage) (json.RawMessage, error)
|
||||
@@ -48,12 +53,21 @@ type Tool struct {
|
||||
// stdio.CallerOf, and says Unproven when it is absent.
|
||||
const MetaCaller = "mesh/caller"
|
||||
|
||||
// callParams is a `tools/call`'s params: the tool, its arguments as the caller sent them, and — only when the
|
||||
// subject named one — the caller, in `_meta`, which the runtime writes and no caller can.
|
||||
func callParams(name, caller string, args json.RawMessage) map[string]any {
|
||||
// callParams is a `tools/call`'s params: the tool, its arguments as the caller sent them, and in `_meta`, which the
|
||||
// runtime writes and no caller can, the caller — only when the subject named one — and the id of the call in flight
|
||||
// (mesh-sdk buscalls.MetaCall), which a ledger's apply names so the runtime stamps the caller it proved (novox/hq ADR
|
||||
// 0597 §7).
|
||||
func callParams(name, caller, call string, args json.RawMessage) map[string]any {
|
||||
params := map[string]any{"name": name, "arguments": args}
|
||||
meta := map[string]any{}
|
||||
if caller != "" {
|
||||
params["_meta"] = map[string]any{MetaCaller: caller}
|
||||
meta[MetaCaller] = caller
|
||||
}
|
||||
if call != "" {
|
||||
meta[buscalls.MetaCall] = call
|
||||
}
|
||||
if len(meta) > 0 {
|
||||
params["_meta"] = meta
|
||||
}
|
||||
return params
|
||||
}
|
||||
@@ -67,7 +81,9 @@ type Registration struct {
|
||||
// Bus is what a launched bundle reaches the mesh through (novox/hq ADR 0193, ADR 0198): the runtime
|
||||
// publishes, asks and subscribes on the module's behalf. Subscribe is asked each time the module's
|
||||
// code subscribes; the runtime binds the module's consumer once and calls deliver for every event,
|
||||
// acknowledging it on the bus when deliver returns nil.
|
||||
// acknowledging it on the bus when deliver returns nil. beat is the runtime's word on that consumer between
|
||||
// events (a bus.Beat as JSON), which the launch hands the child that subscribed as `mesh/heartbeat`, so a bundle
|
||||
// can tell a silent source from a quiet one (stdio.OnHeartbeat).
|
||||
//
|
||||
// State and Watch reach the module's state (novox/hq ADR 0201): State answers `get`, `put`, `delete`
|
||||
// and `keys`; Watch hands deliver the current values and then every change, and returns once the
|
||||
@@ -75,11 +91,13 @@ type Registration struct {
|
||||
type Bus interface {
|
||||
Publish(params json.RawMessage) error
|
||||
Ask(params json.RawMessage) (json.RawMessage, error)
|
||||
Subscribe(deliver func(envelope json.RawMessage) error) error
|
||||
Subscribe(deliver func(envelope json.RawMessage) error, beat func(heartbeat json.RawMessage)) error
|
||||
State(verb string, params json.RawMessage) (json.RawMessage, error)
|
||||
Watch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
|
||||
// Log answers a bundle's `append`, `read`, `last` and `info` on its module's log (novox/hq ADR 0297).
|
||||
Log(verb string, params json.RawMessage) (json.RawMessage, error)
|
||||
// LogWatch hands deliver each entry of a log from where the bundle asked, one at a time (novox/hq ADR 0571).
|
||||
LogWatch(params json.RawMessage, deliver func(entry json.RawMessage) error) (stop func(), err error)
|
||||
// Seat answers a bundle's seat traffic (novox/hq ADR 0259 §3): `publish`, `prove` and `record` at once;
|
||||
// `take` and `answer` bind once and hand each message or proof on through handOn, to whichever child
|
||||
// of the module asked last — a restarted child asks again as its code runs again.
|
||||
@@ -89,8 +107,29 @@ type Bus interface {
|
||||
// (novox/hq ADR 0349 §14, ADR 0370 §4): its config, and its bindings' values that are not secret.
|
||||
Provision(verb string, params json.RawMessage) (json.RawMessage, error)
|
||||
ProvisionWatch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
|
||||
// BusState answers a bundle's calls on its module's bus state in the new form (mesh-sdk buscalls; novox/hq ADR 0595
|
||||
// §9, design 25 § 12.5): its objects, a stream's read-back and fill, a work queue's size and dead letters, and its
|
||||
// ledgers, by the method the bundle asked, from its own module's bindings alone.
|
||||
BusState(method string, params json.RawMessage) (json.RawMessage, error)
|
||||
// BusStateWatch hands deliver each change of an objects or a ledger watch, the current ones first.
|
||||
BusStateWatch(method string, params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
|
||||
// Begin says a tool call of the module is in flight for the caller the runtime proved, and answers its id, which the
|
||||
// bundle is handed in the call's `_meta`, and the end to call once the call is answered (novox/hq ADR 0597 §7).
|
||||
Begin(caller string) (id string, end func())
|
||||
}
|
||||
|
||||
// busStateCalls are the methods of mesh-sdk buscalls a bundle asks and BusState answers.
|
||||
var busStateCalls = map[string]bool{
|
||||
buscalls.ObjectsPut: true, buscalls.ObjectsGet: true, buscalls.ObjectsList: true, buscalls.ObjectsDelete: true,
|
||||
buscalls.ObjectsInfo: true, buscalls.StreamRead: true, buscalls.StreamFill: true, buscalls.StreamPublish: true, buscalls.WorkQueueSize: true,
|
||||
buscalls.WorkQueueDead: true, buscalls.LedgerApply: true, buscalls.LedgerGet: true, buscalls.LedgerHistory: true,
|
||||
buscalls.LedgerQuery: true, buscalls.LedgerRebuild: true, buscalls.LedgerReindex: true, buscalls.LedgerSnapshot: true,
|
||||
buscalls.LedgerLag: true, buscalls.ObjectsSentGet: true, buscalls.ObjectsPutChunk: true, buscalls.ObjectsGetChunk: true,
|
||||
}
|
||||
|
||||
// busStateWatches are the watches of mesh-sdk buscalls, each with the method its changes are handed to the bundle by.
|
||||
var busStateWatches = map[string]string{buscalls.ObjectsWatch: buscalls.ObjectsWatched, buscalls.LedgerWatch: buscalls.LedgerWatched}
|
||||
|
||||
// EventTimeout bounds how long a bundle has to handle one event before it is offered again.
|
||||
var EventTimeout = 2 * time.Minute
|
||||
|
||||
@@ -168,11 +207,33 @@ type child struct {
|
||||
// restart decided — which Stop waits for (novox/hq issue 476).
|
||||
handled chan struct{}
|
||||
why error
|
||||
// endedFor is why the runner itself ended the bundle (a line over the cap); its calls fail with it.
|
||||
endedFor string
|
||||
// watches are the state watches this child asked for, stopped when it exits (ADR 0201): a child
|
||||
// that starts again watches again, as its code runs again.
|
||||
watches []func()
|
||||
}
|
||||
|
||||
// maxLine is the longest line the runner reads from a bundle; a variable so a test need not write megabytes.
|
||||
var maxLine = 16 << 20
|
||||
|
||||
// errNoAnswer is a call the bundle did not answer within its bound.
|
||||
var errNoAnswer = errors.New("the bundle did not answer in time")
|
||||
|
||||
// readCapped reads one line of at most max bytes. tooLong says it was longer; the rest is not read.
|
||||
func readCapped(r *bufio.Reader, max int) (line []byte, tooLong bool, err error) {
|
||||
for {
|
||||
part, isPrefix, e := r.ReadLine()
|
||||
line = append(line, part...)
|
||||
if len(line) > max {
|
||||
return nil, true, nil
|
||||
}
|
||||
if e != nil || !isPrefix {
|
||||
return line, false, e
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *child) write(m any) error {
|
||||
b, err := wire.Marshal(m)
|
||||
if err != nil {
|
||||
@@ -225,11 +286,23 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
_, err := c.ask(module, "mesh/event", map[string]any{"envelope": envelope}, EventTimeout)
|
||||
return err
|
||||
}
|
||||
// The heartbeat of the module's consumer goes to the child that last subscribed, as a notification: the
|
||||
// bundle answers nothing, and a child not running misses it, as it misses the events it did not take.
|
||||
beat := func(heartbeat json.RawMessage) {
|
||||
mu.Lock()
|
||||
c := subscriber
|
||||
mu.Unlock()
|
||||
if c == nil {
|
||||
return
|
||||
}
|
||||
_ = c.write(map[string]any{"jsonrpc": "2.0", "method": "mesh/heartbeat", "params": heartbeat})
|
||||
}
|
||||
var restart func(after time.Duration)
|
||||
var startedAt time.Time
|
||||
// The restart pauses are read once, here: the exit handling runs on its own goroutine and must not
|
||||
// read the package's settings while a caller (a test) changes them (novox/hq issue 476).
|
||||
pauseFirst, pauseMost, settle := RestartFirst, RestartMost, RestartSettle
|
||||
lineCap := maxLine
|
||||
pause := pauseFirst
|
||||
|
||||
start := func() (*child, error) {
|
||||
@@ -275,10 +348,25 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
}()
|
||||
go func() {
|
||||
defer close(c.handled)
|
||||
scan := bufio.NewScanner(stdout)
|
||||
scan.Buffer(make([]byte, 64*1024), 16<<20)
|
||||
for scan.Scan() {
|
||||
line := strings.TrimSpace(scan.Text())
|
||||
rd := bufio.NewReaderSize(stdout, 64*1024)
|
||||
for {
|
||||
raw, tooLong, rerr := readCapped(rd, lineCap)
|
||||
if tooLong {
|
||||
// Never left frozen (issue 1966): a reader that stops while the bundle lives leaves the bundle's
|
||||
// next write blocked for good. Said, and the bundle ended; the exit path below fails what waits
|
||||
// on it and starts it again.
|
||||
reason := fmt.Sprintf("%s's bundle wrote a line over %d bytes (a tool's answer the bus cannot carry)", module, lineCap)
|
||||
logf("[mesh-tools] %s; ending the bundle", reason)
|
||||
c.mu.Lock()
|
||||
c.endedFor = reason
|
||||
c.mu.Unlock()
|
||||
_ = cmd.Process.Kill()
|
||||
break
|
||||
}
|
||||
if rerr != nil && len(raw) == 0 {
|
||||
break
|
||||
}
|
||||
line := strings.TrimSpace(string(raw))
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
@@ -296,19 +384,84 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
go func(m message) {
|
||||
var result any = map[string]any{}
|
||||
var err error
|
||||
asked := time.Now()
|
||||
switch m.Method {
|
||||
case "mesh/publish":
|
||||
err = mesh.Publish(m.Params)
|
||||
var said struct {
|
||||
Ack bool `json:"ack"`
|
||||
}
|
||||
_ = json.Unmarshal(m.Params, &said)
|
||||
if !said.Ack {
|
||||
err = mesh.Publish(m.Params)
|
||||
break
|
||||
}
|
||||
// With ack the runtime answers only once the bus acknowledged the event: its sequence, or {error}
|
||||
// (design 47 G §559). A refusal is an answer, so a writer says it as a finding's resend.
|
||||
acker, ok := mesh.(interface {
|
||||
PublishAcked(params json.RawMessage) (uint64, error)
|
||||
})
|
||||
switch {
|
||||
case !ok:
|
||||
result = map[string]any{"error": "this tool runner's bus publishes no event with an acknowledgement"}
|
||||
default:
|
||||
if seq, perr := acker.PublishAcked(m.Params); perr != nil {
|
||||
result = map[string]any{"error": perr.Error()}
|
||||
} else {
|
||||
result = map[string]any{"sequence": seq}
|
||||
}
|
||||
}
|
||||
case "mesh/ask":
|
||||
var asked json.RawMessage
|
||||
if asked, err = mesh.Ask(m.Params); err == nil {
|
||||
result = asked
|
||||
}
|
||||
case stdio.CapabilitiesCall:
|
||||
// The capability words this runner states (design 47 G §576); a Bus that states none answers none.
|
||||
words := []string{}
|
||||
if w, ok := mesh.(interface{ CapabilityWords() []string }); ok {
|
||||
words = w.CapabilityWords()
|
||||
}
|
||||
var raw []byte
|
||||
if raw, err = json.Marshal(map[string]any{"words": words}); err == nil {
|
||||
result = json.RawMessage(raw)
|
||||
}
|
||||
case "mesh/subscribe":
|
||||
var said struct {
|
||||
Pattern string `json:"pattern"`
|
||||
Deliver string `json:"deliver"`
|
||||
}
|
||||
_ = json.Unmarshal(m.Params, &said)
|
||||
if said.Deliver == "new" {
|
||||
// An ordered consumer of its own from the next message on (design 47 G §559, A-153): each message
|
||||
// is handed to this child as an event is, and the consumer ends with the child.
|
||||
newer, ok := mesh.(interface {
|
||||
SubscribeNew(pattern string, deliver func(json.RawMessage) error) (func(), error)
|
||||
})
|
||||
if !ok {
|
||||
err = errors.New("this tool runner's bus makes no subscription for new messages")
|
||||
break
|
||||
}
|
||||
var stop func()
|
||||
stop, err = newer.SubscribeNew(said.Pattern, func(envelope json.RawMessage) error {
|
||||
_, err := c.ask(module, "mesh/event", map[string]any{"envelope": envelope}, EventTimeout)
|
||||
return err
|
||||
})
|
||||
if err == nil {
|
||||
c.mu.Lock()
|
||||
if c.why != nil {
|
||||
c.mu.Unlock()
|
||||
stop()
|
||||
} else {
|
||||
c.watches = append(c.watches, stop)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
mu.Lock()
|
||||
subscriber = c
|
||||
mu.Unlock()
|
||||
err = mesh.Subscribe(deliver)
|
||||
err = mesh.Subscribe(deliver, beat)
|
||||
case "mesh/seat.publish", "mesh/seat.prove", "mesh/seat.record", "mesh/seat.kinds", "mesh/seat.pull", "mesh/seat.settle":
|
||||
var answered json.RawMessage
|
||||
if answered, err = mesh.Seat(strings.TrimPrefix(m.Method, "mesh/seat."), m.Params, nil); err == nil {
|
||||
@@ -323,12 +476,12 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
result = answered
|
||||
}
|
||||
case "mesh/state.get", "mesh/state.put", "mesh/state.delete", "mesh/state.keys",
|
||||
"mesh/state.create", "mesh/state.update", "mesh/state.history":
|
||||
"mesh/state.create", "mesh/state.update", "mesh/state.history", "mesh/state.info", "mesh/state.page":
|
||||
var answered json.RawMessage
|
||||
if answered, err = mesh.State(strings.TrimPrefix(m.Method, "mesh/state."), m.Params); err == nil {
|
||||
result = answered
|
||||
}
|
||||
case "mesh/log.append", "mesh/log.read", "mesh/log.last", "mesh/log.info":
|
||||
case "mesh/log.append", "mesh/log.read", "mesh/log.newest", "mesh/log.last", "mesh/log.info":
|
||||
var answered json.RawMessage
|
||||
if answered, err = mesh.Log(strings.TrimPrefix(m.Method, "mesh/log."), m.Params); err == nil {
|
||||
result = answered
|
||||
@@ -357,6 +510,25 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
case "mesh/log.watch":
|
||||
// As a state's watch: each entry asked of this child in order, `mesh/log {watch, entry}`,
|
||||
// naming the watch by the id the child asked it with (novox/hq ADR 0571).
|
||||
var stop func()
|
||||
watch := m.ID
|
||||
stop, err = mesh.LogWatch(m.Params, func(entry json.RawMessage) error {
|
||||
_, err := c.ask(module, "mesh/log", map[string]any{"watch": watch, "entry": entry}, EventTimeout)
|
||||
return err
|
||||
})
|
||||
if err == nil {
|
||||
c.mu.Lock()
|
||||
if c.why != nil {
|
||||
c.mu.Unlock()
|
||||
stop()
|
||||
} else {
|
||||
c.watches = append(c.watches, stop)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
case "mesh/state.watch":
|
||||
// Each change is asked of this child, in order, naming the watch it is for by
|
||||
// the id the child asked it with — two watches of one state are two handlers;
|
||||
@@ -378,13 +550,47 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
case buscalls.ObjectsWatch, buscalls.LedgerWatch:
|
||||
// As a state's watch: each change asked of this child in order, by the method of its
|
||||
// family; the change names the watch by the id the bundle gave in its request.
|
||||
var stop func()
|
||||
told := busStateWatches[m.Method]
|
||||
stop, err = mesh.BusStateWatch(m.Method, m.Params, func(change json.RawMessage) error {
|
||||
_, err := c.ask(module, told, change, EventTimeout)
|
||||
return err
|
||||
})
|
||||
if err == nil {
|
||||
c.mu.Lock()
|
||||
if c.why != nil {
|
||||
c.mu.Unlock()
|
||||
stop()
|
||||
} else {
|
||||
c.watches = append(c.watches, stop)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
default:
|
||||
if busStateCalls[m.Method] {
|
||||
var answered json.RawMessage
|
||||
if answered, err = mesh.BusState(m.Method, m.Params); err == nil {
|
||||
result = answered
|
||||
}
|
||||
break
|
||||
}
|
||||
if len(m.ID) > 0 {
|
||||
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID,
|
||||
"error": map[string]any{"code": -32601, "message": "the runtime answers no " + m.Method + " from a bundle"}})
|
||||
}
|
||||
return
|
||||
}
|
||||
// A state or log operation the bus refused or let time out reaches a caller as a call that
|
||||
// does not answer: say it here, with how long it took, so the next hang is readable.
|
||||
// A lost compare-and-set is the normal answer of two writers to one key, retried by its caller: not said.
|
||||
if err != nil && errorCode(err) != CodeChanged && (strings.HasPrefix(m.Method, "mesh/state.") ||
|
||||
strings.HasPrefix(m.Method, "mesh/log.") || busStateCalls[m.Method]) {
|
||||
logf("[mesh-tools] %s's bundle asked %s and it failed after %s: %v", module, m.Method,
|
||||
time.Since(asked).Round(time.Millisecond), err)
|
||||
}
|
||||
if len(m.ID) == 0 {
|
||||
return
|
||||
}
|
||||
@@ -426,6 +632,9 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
}
|
||||
saidMu.Unlock()
|
||||
c.mu.Lock()
|
||||
if c.endedFor != "" {
|
||||
why = c.endedFor + "; the bundle was ended (" + code + ")"
|
||||
}
|
||||
c.why = errors.New(why)
|
||||
watches := c.watches
|
||||
c.watches = nil
|
||||
@@ -433,6 +642,10 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
for _, stop := range watches {
|
||||
stop()
|
||||
}
|
||||
// A session of the bundle's put or get ends with its channel (design 47 G §518).
|
||||
if d, ok := mesh.(interface{ DropSessions() }); ok {
|
||||
d.DropSessions()
|
||||
}
|
||||
close(c.dead)
|
||||
mu.Lock()
|
||||
// Only a child that was serving is brought back; one that died in its own handshake was
|
||||
@@ -590,8 +803,14 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
if len(args) == 0 || string(args) == "null" {
|
||||
args = json.RawMessage("{}")
|
||||
}
|
||||
res, err := asking("tools/call", callParams(full, caller, args), CallTimeout)
|
||||
call, end := mesh.Begin(caller)
|
||||
defer end()
|
||||
res, err := asking("tools/call", callParams(full, caller, call, args), CallTimeout)
|
||||
if err != nil {
|
||||
if errors.Is(err, errNoAnswer) {
|
||||
// Its caller is gone by now and nobody else would hear of it (issue 1966).
|
||||
logf("[mesh-tools] %s.%s called by %q: %v", module, full, caller, err)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
var called struct {
|
||||
@@ -689,7 +908,7 @@ func (c *child) ask(module, method string, params any, timeout time.Duration) (j
|
||||
c.mu.Lock()
|
||||
delete(c.pending, id)
|
||||
c.mu.Unlock()
|
||||
return nil, fmt.Errorf("%s's bundle did not answer %s in %ds", module, method, int(timeout/time.Second))
|
||||
return nil, fmt.Errorf("%s's bundle did not answer %s in %ds: %w", module, method, int(timeout/time.Second), errNoAnswer)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -105,3 +105,7 @@ func TestABundleAsksForItsValuesAndIsHandedItsWatchsChanges(t *testing.T) {
|
||||
t.Errorf("the runtime's bus was asked %q", b.asked)
|
||||
}
|
||||
}
|
||||
|
||||
func (b *provisioning) LogWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
|
||||
return func() {}, nil
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -68,6 +69,65 @@ func TestABundleTakesASeatsWorkThroughItsOwnChannel(t *testing.T) {
|
||||
if err != nil || string(got) != `{"taken":true}` {
|
||||
t.Fatalf("the work was not handed to the bundle: %s %v", got, err)
|
||||
}
|
||||
// Design 47 G §576, §518: a bundle reads the runner's capability words with mesh/capabilities, then runs its put session
|
||||
// through the runner's put-chunk, and the sessions of its channel are dropped when the channel closes.
|
||||
t.Run("ABundleReadsTheCapabilityWordsRunsAPutSessionAndItsChannelsClosingDropsThem", func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
entry := filepath.Join(dir, "bundle")
|
||||
if err := os.WriteFile(entry, []byte(puttingBundle), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b := &sessionBus{dropped: make(chan struct{})}
|
||||
l, err := Start("putter", entry, os.Environ(), b, func(string, ...any) {})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(l.Stop)
|
||||
select {
|
||||
case <-b.dropped:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("the bundle's sessions were not dropped when its channel closed")
|
||||
}
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if len(b.chunks) != 1 || !strings.HasPrefix(b.chunks[0], "mesh/objects.put-chunk ") || !strings.Contains(b.chunks[0], `"seq":1`) {
|
||||
t.Fatalf("the runner took %v", b.chunks)
|
||||
}
|
||||
})
|
||||
// Design 47 G §559: `mesh/publish` with `ack` is answered with its sequence once the bus acknowledged it, or `{error}`;
|
||||
// `mesh/subscribe` with `deliver: new` makes a subscription for new messages that ends with the bundle's channel.
|
||||
t.Run("APublishWithAckIsAnsweredWithItsSequenceAndASubscriptionForNewMessagesEndsWithTheChannel", func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
entry, out := filepath.Join(dir, "bundle"), filepath.Join(dir, "out")
|
||||
if err := os.WriteFile(entry, []byte(findingBundle), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b := &findingBus{stopped: make(chan struct{})}
|
||||
l, err := Start("writer", entry, append(os.Environ(), "FINDING_OUT="+out), b, func(string, ...any) {})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(l.Stop)
|
||||
select {
|
||||
case <-b.stopped:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("the subscription for new messages was not ended with the bundle's channel")
|
||||
}
|
||||
got, _ := os.ReadFile(out)
|
||||
for _, want := range []string{`"sequence":42`, `"error":"the bus refused it"`} {
|
||||
if !strings.Contains(string(got), want) {
|
||||
t.Errorf("the bundle was answered %s, want it to hold %s", got, want)
|
||||
}
|
||||
}
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if b.pattern != "mesh-conditions.findings-asked" {
|
||||
t.Errorf("the subscription was made for %q", b.pattern)
|
||||
}
|
||||
if len(b.published) != 0 {
|
||||
t.Errorf("a publish with ack went the unacknowledged way: %v", b.published)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A bundle that pulls one ask from its worker when it is listed.
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
package launch
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// puttingBundle asks the capability words, and once they are answered with put-sessions sends the one chunk of a put
|
||||
// session, as mesh-sdk's bundle does; it then ends, which closes its channel.
|
||||
const puttingBundle = `#!/bin/sh
|
||||
while IFS= read -r line; do
|
||||
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
|
||||
case "$line" in
|
||||
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
|
||||
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
|
||||
printf '{"jsonrpc":"2.0","id":"w1","method":"mesh/capabilities","params":{}}\n' ;;
|
||||
*'"id":"w1"'**put-sessions*) printf '{"jsonrpc":"2.0","id":"p1","method":"mesh/objects.put-chunk","params":{"objects":"o","name":"n","call":"c","seq":1,"last":true,"data":"aGk="}}\n' ;;
|
||||
*'"id":"p1"'*) exit 0 ;;
|
||||
esac
|
||||
done
|
||||
`
|
||||
|
||||
type sessionBus struct {
|
||||
subscribing
|
||||
mu sync.Mutex
|
||||
chunks []string
|
||||
dropped chan struct{}
|
||||
}
|
||||
|
||||
func (b *sessionBus) CapabilityWords() []string { return []string{"put-sessions"} }
|
||||
func (b *sessionBus) BusState(method string, params json.RawMessage) (json.RawMessage, error) {
|
||||
b.mu.Lock()
|
||||
b.chunks = append(b.chunks, method+" "+string(params))
|
||||
b.mu.Unlock()
|
||||
return json.RawMessage(`{"name":"n"}`), nil
|
||||
}
|
||||
func (b *sessionBus) DropSessions() { close(b.dropped) }
|
||||
|
||||
// findingBundle asks, as mesh-sdk's finding writer does, a publish with ack that the bus takes, one it refuses, and a
|
||||
// subscription for new messages; each answer it gets is written to $FINDING_OUT, and it ends after the subscription's.
|
||||
const findingBundle = `#!/bin/sh
|
||||
while IFS= read -r line; do
|
||||
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
|
||||
case "$line" in
|
||||
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
|
||||
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
|
||||
printf '{"jsonrpc":"2.0","id":"a1","method":"mesh/publish","params":{"key":"taken","ack":true,"body":{}}}\n' ;;
|
||||
*'"id":"a1"'*) printf '%s\n' "$line" >>"$FINDING_OUT"
|
||||
printf '{"jsonrpc":"2.0","id":"a2","method":"mesh/publish","params":{"key":"refused","ack":true,"body":{}}}\n' ;;
|
||||
*'"id":"a2"'*) printf '%s\n' "$line" >>"$FINDING_OUT"
|
||||
printf '{"jsonrpc":"2.0","id":"s1","method":"mesh/subscribe","params":{"pattern":"mesh-conditions.findings-asked","deliver":"new"}}\n' ;;
|
||||
*'"id":"s1"'*) printf '%s\n' "$line" >>"$FINDING_OUT"; exit 0 ;;
|
||||
esac
|
||||
done
|
||||
`
|
||||
|
||||
type findingBus struct {
|
||||
subscribing
|
||||
mu sync.Mutex
|
||||
pattern string
|
||||
stopped chan struct{}
|
||||
}
|
||||
|
||||
func (b *findingBus) PublishAcked(params json.RawMessage) (uint64, error) {
|
||||
if strings.Contains(string(params), `"refused"`) {
|
||||
return 0, errors.New("the bus refused it")
|
||||
}
|
||||
return 42, nil
|
||||
}
|
||||
|
||||
func (b *findingBus) SubscribeNew(pattern string, _ func(json.RawMessage) error) (func(), error) {
|
||||
b.mu.Lock()
|
||||
b.pattern = pattern
|
||||
b.mu.Unlock()
|
||||
return func() { close(b.stopped) }, nil
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
package launch
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// changedError is a lost compare-and-set as the bus answers it: told by its code (novox/hq ADR 0259).
|
||||
type changedError struct{}
|
||||
|
||||
func (changedError) Error() string { return "the key changed since revision 3" }
|
||||
func (changedError) ErrorCode() int { return CodeChanged }
|
||||
|
||||
// refusingState answers the state "lost" with a lost compare-and-set and every other state with a time-out.
|
||||
type refusingState struct{ subscribing }
|
||||
|
||||
func (*refusingState) State(_ string, params json.RawMessage) (json.RawMessage, error) {
|
||||
if strings.Contains(string(params), `"state":"lost"`) {
|
||||
return nil, changedError{}
|
||||
}
|
||||
return nil, errors.New("the bus did not answer in 10s")
|
||||
}
|
||||
|
||||
// A bundle that asks the runtime for the state "lost" once it is started, and for the state "copied" once that is
|
||||
// answered, so the first answer's handling has ended before the second is asked.
|
||||
const askingBundle = `#!/bin/sh
|
||||
while IFS= read -r line; do
|
||||
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
|
||||
case "$line" in
|
||||
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
|
||||
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
|
||||
printf '{"jsonrpc":"2.0","id":"q1","method":"mesh/state.update","params":{"state":"lost","key":"k"}}\n' ;;
|
||||
*'"id":"q1"'*) printf '{"jsonrpc":"2.0","id":"q2","method":"mesh/state.get","params":{"state":"copied","key":"k"}}\n' ;;
|
||||
esac
|
||||
done
|
||||
`
|
||||
|
||||
// A state operation the bus refuses or lets time out is said in the runtime's log with its method, its bundle and its
|
||||
// time (issue 1966: a hand-act seat whose calls hung said nothing); a lost compare-and-set, asked first, is not.
|
||||
func TestAFailedStateOperationIsSaidInTheRuntimesLog(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
entry := filepath.Join(dir, "bundle")
|
||||
if err := os.WriteFile(entry, []byte(askingBundle), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
said := make(chan string, 16)
|
||||
logf := func(f string, a ...any) { said <- fmt.Sprintf(f, a...) }
|
||||
l, err := Start("asker", entry, os.Environ(), &refusingState{}, logf)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(l.Stop)
|
||||
var heard []string
|
||||
timeout := time.After(5 * time.Second)
|
||||
for {
|
||||
select {
|
||||
case line := <-said:
|
||||
heard = append(heard, line)
|
||||
if strings.Contains(line, "mesh/state.update") {
|
||||
t.Fatalf("a lost compare-and-set was said as a failure: %q", line)
|
||||
}
|
||||
if strings.Contains(line, "asker's bundle asked mesh/state.get and it failed after") && strings.Contains(line, "did not answer in 10s") {
|
||||
return
|
||||
}
|
||||
case <-timeout:
|
||||
t.Fatalf("the failed state operation was not said; the log: %v", heard)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A bundle that lists one tool, "t", and answers a call to it as its MODE says: "long" with a line over the runner's cap,
|
||||
// "silent" with nothing at all.
|
||||
const callingBundle = `#!/bin/sh
|
||||
while IFS= read -r line; do
|
||||
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
|
||||
case "$line" in
|
||||
*'"method":"initialize"'*) echo started >&2; printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
|
||||
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[{"name":"t","description":"d"}]}}\n' "$id" ;;
|
||||
*'"method":"tools/call"'*) [ "$MODE" = long ] && printf '%4096s\n' x ;;
|
||||
esac
|
||||
done
|
||||
`
|
||||
|
||||
// Issue 1966: a bundle's line over the runner's cap, and a call the bundle never answers, are said in the runtime's log;
|
||||
// the over-long line ends the bundle and fails the call at once, never leaving the runner reading nothing while the
|
||||
// bundle's next write blocks.
|
||||
func TestAnOverlongLineAndAnUnansweredCallAreSaid(t *testing.T) {
|
||||
oldMax, oldCall, oldFirst := maxLine, CallTimeout, RestartFirst
|
||||
maxLine, RestartFirst = 1024, 10*time.Millisecond
|
||||
t.Cleanup(func() { maxLine, CallTimeout, RestartFirst = oldMax, oldCall, oldFirst })
|
||||
for _, row := range []struct {
|
||||
mode string
|
||||
timeout time.Duration
|
||||
}{
|
||||
// The call's own bound is long: only the runner ending the bundle fails it within a second.
|
||||
{"long", 10 * time.Second},
|
||||
{"silent", 300 * time.Millisecond},
|
||||
} {
|
||||
t.Run(row.mode, func(t *testing.T) {
|
||||
CallTimeout = row.timeout
|
||||
dir := t.TempDir()
|
||||
entry := filepath.Join(dir, "bundle")
|
||||
if err := os.WriteFile(entry, []byte(callingBundle), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
said := make(chan string, 32)
|
||||
logf := func(f string, a ...any) { said <- fmt.Sprintf(f, a...) }
|
||||
l, err := Start("caller", entry, append(os.Environ(), "MODE="+row.mode), &refusingState{}, logf)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(l.Stop)
|
||||
begun := time.Now()
|
||||
_, err = l.Registrations[0].Tools[0].Run("caller-1", nil)
|
||||
if err == nil {
|
||||
t.Fatal("the call was answered")
|
||||
}
|
||||
var heard []string
|
||||
timeout := time.After(5 * time.Second)
|
||||
want := "caller.t called by \"caller-1\""
|
||||
if row.mode == "long" {
|
||||
want = "ending the bundle"
|
||||
if took := time.Since(begun); took > time.Second {
|
||||
t.Fatalf("the call hung for %s", took)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "over 1024 bytes") || strings.Contains(err.Error(), "did not answer") {
|
||||
t.Fatalf("the call failed with %q, not the cap's reason", err)
|
||||
}
|
||||
}
|
||||
// For "long" the bundle is also started again: it says "started" on each start, so twice in all.
|
||||
started, found := 0, false
|
||||
for !found || (row.mode == "long" && started < 2) {
|
||||
select {
|
||||
case line := <-said:
|
||||
heard = append(heard, line)
|
||||
found = found || strings.Contains(line, want)
|
||||
if line == "[caller] started" {
|
||||
started++
|
||||
}
|
||||
case <-timeout:
|
||||
t.Fatalf("not said, or the bundle not started again; the log: %v", heard)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
package launch
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// A-42: a launched bundle may ask mesh/stream.publish.
|
||||
func TestABundleMayAskStreamPublish(t *testing.T) {
|
||||
if !busStateCalls[buscalls.StreamPublish] {
|
||||
t.Fatal("mesh/stream.publish is not among the methods a bundle may ask")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,576 @@
|
||||
// Package ledger is the tool runner's side of a ledger (novox/hq ADR 0597, design 25 § 12.4; issue 1966): a storage type
|
||||
// whose state is derived from its history. `apply` checks an operation against its type, appends it to the instance's
|
||||
// history with the last sequence expected, derives the state by the operation's effect and writes it as an update of the
|
||||
// revision read, and writes the instance's snapshot. What the history, the state and the snapshots are kept in is given
|
||||
// (History, State, Snapshots): the bus in the runtime, fakes in a test.
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// The refusals the stores answer, told by errors.Is.
|
||||
var (
|
||||
// ErrRaced is an append or a snapshot written with an expected sequence that is no longer the last: another writer
|
||||
// wrote first.
|
||||
ErrRaced = errors.New("another writer wrote the instance's history first")
|
||||
// ErrChanged is a state written at a revision that is no longer the key's.
|
||||
ErrChanged = errors.New("the state was written by another writer first")
|
||||
// ErrFull is a state bucket that refuses a write because it is full.
|
||||
ErrFull = errors.New("the state bucket is full")
|
||||
)
|
||||
|
||||
// CodeChanged is the error code a bundle is answered a lost race with, as a lost state update (mesh-sdk stdio.CodeChanged).
|
||||
const CodeChanged = -32010
|
||||
|
||||
// raced is an apply refused because another writer appended to the instance first (ADR 0597 §5).
|
||||
type raced struct{ instance string }
|
||||
|
||||
func (e raced) Error() string {
|
||||
return "the operation on " + e.instance + " is not taken: another writer appended to its history first; read it and apply again (novox/hq ADR 0597 §5)"
|
||||
}
|
||||
func (e raced) Is(target error) bool { return target == ErrRaced }
|
||||
func (e raced) ErrorCode() int { return CodeChanged }
|
||||
|
||||
// Kept is one message a history keeps: its sequence and its bytes.
|
||||
type Kept struct {
|
||||
Sequence uint64
|
||||
Data []byte
|
||||
}
|
||||
|
||||
// History is a ledger's history: a stream with a subject per instance, each instance named by its key.
|
||||
type History interface {
|
||||
// Last is the sequence of the instance's last operation, 0 when it has none.
|
||||
Last(key string) (uint64, error)
|
||||
// Append writes one operation, expecting the instance's last sequence to be expect (0: it has none); ErrRaced when
|
||||
// it is not.
|
||||
Append(key string, op []byte, expect uint64) (uint64, error)
|
||||
// Read is the operations of the instances keys names (a key, with `*` for one name and `>` for the rest) from a
|
||||
// sequence on, oldest first, at most limit of them (every one when limit is 0).
|
||||
Read(keys string, from uint64, limit int) ([]Kept, error)
|
||||
// Purge removes every operation of the instance key names, and nothing of another (an instance's removal).
|
||||
Purge(key string) error
|
||||
// Fill is the history's fill against its cap, the ledger's cap (buscalls.LedgerFill; design 47 G §587).
|
||||
Fill() (buscalls.Fill, error)
|
||||
}
|
||||
|
||||
// State is a ledger's state bucket: each instance's record under its key.
|
||||
type State interface {
|
||||
// Get is the key's value and revision; nil and 0 when it has none.
|
||||
Get(key string) ([]byte, uint64, error)
|
||||
// Write writes the key's value at the revision read (0: only when it has none); ErrChanged when the key is no longer
|
||||
// at it, ErrFull when the bucket is full.
|
||||
Write(key string, value []byte, revision uint64) (uint64, error)
|
||||
// Keys is every key the bucket holds.
|
||||
Keys() ([]string, error)
|
||||
// Delete removes the key; a key it does not hold is no fault.
|
||||
Delete(key string) error
|
||||
}
|
||||
|
||||
// Snapshots are a ledger's snapshots: the latest record of each instance, on a subject of its own, never expiring.
|
||||
type Snapshots interface {
|
||||
// Get is the key's latest snapshot and the sequence it was written at; nil and 0 when it has none.
|
||||
Get(key string) ([]byte, uint64, error)
|
||||
// Put writes the key's snapshot, expecting its last to be at expect (0: none); ErrRaced when it is not.
|
||||
Put(key string, value []byte, expect uint64) error
|
||||
// Purge removes the key's snapshot; one it does not hold is no fault.
|
||||
Purge(key string) error
|
||||
}
|
||||
|
||||
// Ledger is one ledger of one module: its types and where it is kept.
|
||||
type Ledger struct {
|
||||
Module string
|
||||
Name string
|
||||
// Account is the owning module's authenticated account, which every operation it applies is stamped with.
|
||||
Account string
|
||||
Types []catalogue.LedgerType
|
||||
History History
|
||||
State State
|
||||
Snapshots Snapshots
|
||||
// Indexes are its declared indexes' buckets, by index name.
|
||||
Indexes map[string]Index
|
||||
// Now is the clock an operation is stamped by; time.Now when nil.
|
||||
Now func() time.Time
|
||||
// Logf says what an apply could not do after its operation was taken; nothing is said when nil.
|
||||
Logf func(format string, args ...any)
|
||||
}
|
||||
|
||||
func (l *Ledger) say(format string, args ...any) {
|
||||
if l.Logf != nil {
|
||||
l.Logf(format, args...)
|
||||
}
|
||||
}
|
||||
|
||||
// Record is what the state and a snapshot keep of an instance: its state and the sequence of the last operation in it.
|
||||
type Record struct {
|
||||
Sequence uint64 `json:"sequence"`
|
||||
State json.RawMessage `json:"state"`
|
||||
}
|
||||
|
||||
// segment is one part of an instance's path.
|
||||
var segment = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
|
||||
|
||||
// KeyOf is the key an instance is kept under, in the state and as its history's and snapshot's subject token: its path's
|
||||
// parts joined by dots (`decision/0290` is `decision.0290`).
|
||||
func KeyOf(instance string) (string, error) {
|
||||
parts := strings.Split(instance, "/")
|
||||
if len(parts) < 2 {
|
||||
return "", fmt.Errorf("%q is not an instance: an instance is named by a path whose first part is its type's name, "+
|
||||
"such as decision/0290 (novox/hq ADR 0597 §1)", instance)
|
||||
}
|
||||
for _, p := range parts {
|
||||
if !segment.MatchString(p) {
|
||||
return "", fmt.Errorf("%q is not an instance: each part of its path holds letters, digits, - and _ alone", instance)
|
||||
}
|
||||
}
|
||||
return strings.Join(parts, "."), nil
|
||||
}
|
||||
|
||||
// InstanceOf is the instance a key names.
|
||||
func InstanceOf(key string) string { return strings.ReplaceAll(key, ".", "/") }
|
||||
|
||||
func (l *Ledger) now() time.Time {
|
||||
if l.Now != nil {
|
||||
return l.Now().UTC()
|
||||
}
|
||||
return time.Now().UTC()
|
||||
}
|
||||
|
||||
// typeOf is the instance's type, by its path's first part.
|
||||
func (l *Ledger) typeOf(instance string) (catalogue.LedgerType, error) {
|
||||
name, _, _ := strings.Cut(instance, "/")
|
||||
for _, t := range l.Types {
|
||||
if t.Name == name {
|
||||
return t, nil
|
||||
}
|
||||
}
|
||||
return catalogue.LedgerType{}, fmt.Errorf("%s's ledger %s has no type %q, which the instance %s names", l.Module, l.Name, name, instance)
|
||||
}
|
||||
|
||||
// operationOf is the operation a type allows by its name.
|
||||
func operationOf(t catalogue.LedgerType, name string) (catalogue.LedgerOperation, error) {
|
||||
for _, op := range t.Operations {
|
||||
if op.Name == name {
|
||||
return op, nil
|
||||
}
|
||||
}
|
||||
return catalogue.LedgerOperation{}, fmt.Errorf("the type %s allows no operation %q", t.Name, name)
|
||||
}
|
||||
|
||||
// Digest is the digest a type pairs a text field with (ADR 0597 §2): sha256 of the text, as the knowledge base writes it.
|
||||
func Digest(text string) string {
|
||||
sum := sha256.Sum256([]byte(text))
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// kinds is the JSON kind each schema type name is held to; a name not here is the judge's to know.
|
||||
var kinds = map[string]func(any) bool{
|
||||
"string": func(v any) bool { _, ok := v.(string); return ok },
|
||||
"number": func(v any) bool { _, ok := v.(float64); return ok },
|
||||
"integer": func(v any) bool { f, ok := v.(float64); return ok && f == float64(int64(f)) },
|
||||
"boolean": func(v any) bool { _, ok := v.(bool); return ok },
|
||||
"object": func(v any) bool { _, ok := v.(map[string]any); return ok },
|
||||
"array": func(v any) bool { _, ok := v.([]any); return ok },
|
||||
}
|
||||
|
||||
// checked is an operation as apply checked it: its instance's key, its operation and its body.
|
||||
type checked struct {
|
||||
key string
|
||||
op catalogue.LedgerOperation
|
||||
body map[string]any
|
||||
}
|
||||
|
||||
// check holds an operation to its type (ADR 0597 §2): a type the instance names, an operation it allows, a body that is an
|
||||
// object whose fields fit the schema, the key its effect names, and every digest matching its text.
|
||||
func (l *Ledger) check(r buscalls.ApplyRequest) (checked, error) {
|
||||
key, err := KeyOf(r.Instance)
|
||||
if err != nil {
|
||||
return checked{}, err
|
||||
}
|
||||
t, err := l.typeOf(r.Instance)
|
||||
if err != nil {
|
||||
return checked{}, err
|
||||
}
|
||||
op, err := operationOf(t, r.Operation)
|
||||
if err != nil {
|
||||
return checked{}, err
|
||||
}
|
||||
body := map[string]any{}
|
||||
if len(r.Body) > 0 && string(r.Body) != "null" {
|
||||
if err := json.Unmarshal(r.Body, &body); err != nil {
|
||||
return checked{}, fmt.Errorf("the operation %s's body is a JSON object: %w", r.Operation, err)
|
||||
}
|
||||
}
|
||||
for name, field := range t.Schema {
|
||||
v, has := body[name]
|
||||
if !has {
|
||||
continue
|
||||
}
|
||||
if fits, known := kinds[field.Type]; known && !fits(v) {
|
||||
return checked{}, fmt.Errorf("the operation %s's field %s is a %s in the type %s", r.Operation, name, field.Type, t.Name)
|
||||
}
|
||||
}
|
||||
for name, field := range t.Schema {
|
||||
if field.DigestOf == "" {
|
||||
continue
|
||||
}
|
||||
digest, hasDigest := body[name]
|
||||
text, hasText := body[field.DigestOf]
|
||||
if !hasDigest && !hasText {
|
||||
continue
|
||||
}
|
||||
s, isText := text.(string)
|
||||
if !hasDigest || !hasText || !isText || digest != Digest(s) {
|
||||
return checked{}, fmt.Errorf("the operation %s's %s is not the digest of its %s, so it is refused (novox/hq ADR 0597 §2)",
|
||||
r.Operation, name, field.DigestOf)
|
||||
}
|
||||
}
|
||||
switch op.Effect {
|
||||
case catalogue.EffectSetFields, catalogue.EffectAppend:
|
||||
case catalogue.EffectSetEntry, catalogue.EffectRemove:
|
||||
if op.Effect == catalogue.EffectRemove && op.Into == "" {
|
||||
break
|
||||
}
|
||||
if _, ok := body[op.Key].(string); !ok {
|
||||
return checked{}, fmt.Errorf("the operation %s names its entry by its field %s, a text, which its body lacks", r.Operation, op.Key)
|
||||
}
|
||||
default:
|
||||
return checked{}, fmt.Errorf("the operation %s's effect %q is not one the mesh offers (novox/hq ADR 0597 §2)", r.Operation, op.Effect)
|
||||
}
|
||||
return checked{key: key, op: op, body: body}, nil
|
||||
}
|
||||
|
||||
// effect is an operation's effect on a state (ADR 0597 §2): it copies only the fields the operation names, and the
|
||||
// operation's sequence and proven author where it says so; null is a removed instance.
|
||||
func effect(state json.RawMessage, op catalogue.LedgerOperation, body map[string]any, seq uint64, author buscalls.Author) (json.RawMessage, error) {
|
||||
s := map[string]any{}
|
||||
if len(state) > 0 && string(state) != "null" {
|
||||
if err := json.Unmarshal(state, &s); err != nil {
|
||||
return nil, fmt.Errorf("the state held is not an object: %w", err)
|
||||
}
|
||||
}
|
||||
copied := map[string]any{}
|
||||
for _, f := range op.Fields {
|
||||
if v, ok := body[f]; ok {
|
||||
copied[f] = v
|
||||
}
|
||||
}
|
||||
if op.Sequence {
|
||||
copied["sequence"] = seq
|
||||
}
|
||||
// Written last, so a body's own `author` never stands for the proven one (ADR 0597 §7).
|
||||
if op.Author {
|
||||
copied["author"] = author
|
||||
}
|
||||
switch op.Effect {
|
||||
case catalogue.EffectSetFields:
|
||||
for k, v := range copied {
|
||||
s[k] = v
|
||||
}
|
||||
case catalogue.EffectSetEntry:
|
||||
m, _ := s[op.Into].(map[string]any)
|
||||
if m == nil {
|
||||
m = map[string]any{}
|
||||
}
|
||||
m[body[op.Key].(string)] = copied
|
||||
s[op.Into] = m
|
||||
case catalogue.EffectAppend:
|
||||
list, _ := s[op.Into].([]any)
|
||||
s[op.Into] = append(list, copied)
|
||||
case catalogue.EffectRemove:
|
||||
if op.Into == "" {
|
||||
return json.RawMessage("null"), nil
|
||||
}
|
||||
if m, ok := s[op.Into].(map[string]any); ok {
|
||||
delete(m, body[op.Key].(string))
|
||||
}
|
||||
}
|
||||
return json.Marshal(s)
|
||||
}
|
||||
|
||||
// fold applies one kept operation to a record, as apply derived it.
|
||||
func (l *Ledger) fold(rec Record, k Kept) (Record, error) {
|
||||
var o buscalls.Operation
|
||||
if err := json.Unmarshal(k.Data, &o); err != nil {
|
||||
return rec, fmt.Errorf("the history's operation %d is not one: %w", k.Sequence, err)
|
||||
}
|
||||
t, err := l.typeOf(o.Instance)
|
||||
if err != nil {
|
||||
return rec, err
|
||||
}
|
||||
op, err := operationOf(t, o.Operation)
|
||||
if err != nil {
|
||||
return rec, err
|
||||
}
|
||||
body := map[string]any{}
|
||||
if len(o.Body) > 0 && string(o.Body) != "null" {
|
||||
if err := json.Unmarshal(o.Body, &body); err != nil {
|
||||
return rec, err
|
||||
}
|
||||
}
|
||||
if (op.Effect == catalogue.EffectSetEntry || (op.Effect == catalogue.EffectRemove && op.Into != "")) && !isText(body[op.Key]) {
|
||||
return rec, fmt.Errorf("the history's operation %d names no entry", k.Sequence)
|
||||
}
|
||||
state, err := effect(rec.State, op, body, k.Sequence, o.Author)
|
||||
if err != nil {
|
||||
return rec, err
|
||||
}
|
||||
return Record{Sequence: k.Sequence, State: state}, nil
|
||||
}
|
||||
|
||||
func isText(v any) bool { _, ok := v.(string); return ok }
|
||||
|
||||
// replay folds the instance's operations after rec's sequence, up to upto (every one when 0), onto rec.
|
||||
func (l *Ledger) replay(key string, rec Record, upto uint64) (Record, error) {
|
||||
kept, err := l.History.Read(key, rec.Sequence+1, 0)
|
||||
if err != nil {
|
||||
return rec, err
|
||||
}
|
||||
for _, k := range kept {
|
||||
if upto != 0 && k.Sequence > upto {
|
||||
break
|
||||
}
|
||||
if k.Sequence <= rec.Sequence {
|
||||
continue
|
||||
}
|
||||
if rec, err = l.fold(rec, k); err != nil {
|
||||
return rec, err
|
||||
}
|
||||
}
|
||||
return rec, nil
|
||||
}
|
||||
|
||||
// readState is the instance's record in the state and its revision; an empty record when it has none.
|
||||
func (l *Ledger) readState(key string) (Record, uint64, error) {
|
||||
raw, rev, err := l.State.Get(key)
|
||||
if err != nil || raw == nil {
|
||||
return Record{}, rev, err
|
||||
}
|
||||
var rec Record
|
||||
if err := json.Unmarshal(raw, &rec); err != nil {
|
||||
return Record{}, rev, fmt.Errorf("the state of %s is not a record: %w", key, err)
|
||||
}
|
||||
return rec, rev, nil
|
||||
}
|
||||
|
||||
// readSnapshot is the instance's snapshot and the sequence it was written at; an empty record when it has none.
|
||||
func (l *Ledger) readSnapshot(key string) (Record, uint64, error) {
|
||||
raw, at, err := l.Snapshots.Get(key)
|
||||
if err != nil || raw == nil {
|
||||
return Record{}, at, err
|
||||
}
|
||||
var rec Record
|
||||
if err := json.Unmarshal(raw, &rec); err != nil {
|
||||
return Record{}, at, fmt.Errorf("the snapshot of %s is not a record: %w", key, err)
|
||||
}
|
||||
return rec, at, nil
|
||||
}
|
||||
|
||||
// derive is the instance's record caught up with its history up to last: from the state, or from the snapshot when it
|
||||
// is further on (an instance whose state was lost, or whose operations expired, ADR 0597 §8).
|
||||
func (l *Ledger) derive(key string, base Record, last uint64) (Record, error) {
|
||||
if base.Sequence >= last {
|
||||
return base, nil
|
||||
}
|
||||
snap, _, err := l.readSnapshot(key)
|
||||
if err != nil {
|
||||
return base, err
|
||||
}
|
||||
if snap.Sequence > base.Sequence {
|
||||
base = snap
|
||||
}
|
||||
return l.replay(key, base, last)
|
||||
}
|
||||
|
||||
// settle writes want into the state at the revision read (ADR 0597 §5): a lost update re-reads, and catches the state
|
||||
// read up to want's sequence from the history, never appending again; a state already further on is left as it is. It
|
||||
// answers what the state holds after it, and whether it is behind because the bucket is full (ADR 0597 §6).
|
||||
func (l *Ledger) settle(key string, want Record, rev uint64) (Record, bool, error) {
|
||||
for tries := 0; tries < 16; tries++ {
|
||||
raw, err := json.Marshal(want)
|
||||
if err != nil {
|
||||
return want, false, err
|
||||
}
|
||||
_, err = l.State.Write(key, raw, rev)
|
||||
switch {
|
||||
case err == nil:
|
||||
return want, false, nil
|
||||
case errors.Is(err, ErrFull):
|
||||
return want, true, nil
|
||||
case !errors.Is(err, ErrChanged):
|
||||
return want, false, err
|
||||
}
|
||||
cur, curRev, err := l.readState(key)
|
||||
if err != nil {
|
||||
return want, false, err
|
||||
}
|
||||
if cur.Sequence >= want.Sequence {
|
||||
return cur, false, nil
|
||||
}
|
||||
if want, err = l.replay(key, cur, want.Sequence); err != nil {
|
||||
return want, false, err
|
||||
}
|
||||
rev = curRev
|
||||
}
|
||||
return want, false, fmt.Errorf("the state of %s kept being written by other writers; the operation is in its history and the next apply catches it up", key)
|
||||
}
|
||||
|
||||
// writeSnapshot writes rec as the instance's snapshot unless the snapshot is already as far on (ADR 0597 §8).
|
||||
func (l *Ledger) writeSnapshot(key string, rec Record) error {
|
||||
for tries := 0; tries < 16; tries++ {
|
||||
snap, at, err := l.readSnapshot(key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if at != 0 && snap.Sequence >= rec.Sequence {
|
||||
return nil
|
||||
}
|
||||
raw, err := json.Marshal(rec)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = l.Snapshots.Put(key, raw, at); err == nil || !errors.Is(err, ErrRaced) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("the snapshot of %s kept being written by other writers", key)
|
||||
}
|
||||
|
||||
// Apply applies one operation to one instance, stamped with author (ADR 0597 §3, §5, §6, §8; design 25 § 12.4 steps 1-5):
|
||||
// checked against its type, appended with the instance's last sequence expected, the state derived by its effect and
|
||||
// written at the revision read, and the snapshot written. A racing writer's append is refused; a full state bucket
|
||||
// answers taken with the state behind.
|
||||
func (l *Ledger) Apply(r buscalls.ApplyRequest, author buscalls.Author) (buscalls.Applied, error) {
|
||||
c, err := l.check(r)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
last, err := l.History.Last(c.key)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
held, rev, err := l.readState(c.key)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
base, err := l.derive(c.key, held, last)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
body := r.Body
|
||||
if len(body) == 0 {
|
||||
body = json.RawMessage("{}")
|
||||
}
|
||||
op, err := json.Marshal(buscalls.Operation{Instance: r.Instance, Operation: r.Operation, Body: body, Author: author, At: l.now()})
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
seq, err := l.History.Append(c.key, op, last)
|
||||
if errors.Is(err, ErrRaced) {
|
||||
return buscalls.Applied{}, raced{r.Instance}
|
||||
}
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
state, err := effect(base.State, c.op, c.body, seq, author)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
// From here the operation is taken: it is in the history, and a fault below is the state's to catch up, said here
|
||||
// and answered as the state behind, never as a refusal of what later takes effect (ADR 0597 §6).
|
||||
now, behind, err := l.settle(c.key, Record{Sequence: seq, State: state}, rev)
|
||||
if err != nil {
|
||||
l.say("[mesh-tools] %s's ledger %s took operation %d on %s and its state could not follow: %v; the next apply "+
|
||||
"or a rebuild catches it up", l.Module, l.Name, seq, r.Instance, err)
|
||||
behind, now = true, Record{Sequence: seq, State: state}
|
||||
}
|
||||
if behind {
|
||||
now = Record{Sequence: seq, State: state}
|
||||
}
|
||||
if err := l.writeSnapshot(c.key, now); err != nil {
|
||||
l.say("[mesh-tools] %s's ledger %s took operation %d on %s and its snapshot could not be written: %v; the next "+
|
||||
"apply or a snapshot repair writes it", l.Module, l.Name, seq, r.Instance, err)
|
||||
}
|
||||
return buscalls.Applied{Taken: true, Sequence: seq, StateBehind: behind}, nil
|
||||
}
|
||||
|
||||
// Fill is the fill of the ledger's history against the ledger's cap (buscalls.LedgerFill; design 47 G §587): its messages,
|
||||
// its bytes and the cap in bytes. The history holds every operation, and the cap is its alone: neither the fullest of the
|
||||
// stores nor their sum is answered. A history that cannot say its fill is an error naming the ledger, never a fill of
|
||||
// nothing.
|
||||
func (l *Ledger) Fill(buscalls.LedgerRequest) (buscalls.Fill, error) {
|
||||
f, err := l.History.Fill()
|
||||
if err != nil {
|
||||
return buscalls.Fill{}, fmt.Errorf("%s's ledger %s cannot say the fill of its history: %w", l.Module, l.Name, err)
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// Remove removes one instance with everything it kept (buscalls.LedgerRemove; design 47 G §587, §155): its operations, its
|
||||
// snapshot, its key in the state bucket and its keys in every declared index, and answers how many operations and how many
|
||||
// bytes of the history that was. An instance the ledger does not hold answers 0 and 0, and a removal that left something
|
||||
// (said, and answered as an error) is made again by calling it again, which removes the rest.
|
||||
func (l *Ledger) Remove(r buscalls.LedgerRemoveRequest) (buscalls.LedgerRemoved, error) {
|
||||
key, err := KeyOf(r.Instance)
|
||||
if err != nil {
|
||||
return buscalls.LedgerRemoved{}, err
|
||||
}
|
||||
out := buscalls.LedgerRemoved{Instance: r.Instance}
|
||||
before, err := l.History.Fill()
|
||||
if err != nil {
|
||||
return out, fmt.Errorf("%s's ledger %s cannot say the fill of its history, so a removal is not made: %w", l.Module, l.Name, err)
|
||||
}
|
||||
ops, err := l.History.Read(key, 1, 0)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for _, step := range []struct {
|
||||
what string
|
||||
remove func(string) error
|
||||
}{{"operations", l.History.Purge}, {"snapshot", l.Snapshots.Purge}, {"state", l.State.Delete}, {"index entries", l.dropFromIndexes}} {
|
||||
if err := step.remove(key); err != nil {
|
||||
l.say("[mesh-tools] %s's ledger %s removes %s and its %s could not be removed: %v; removing it again removes what "+
|
||||
"is left", l.Module, l.Name, r.Instance, step.what, err)
|
||||
return out, fmt.Errorf("%s's ledger %s removed part of %s and its %s are left: %w", l.Module, l.Name, r.Instance, step.what, err)
|
||||
}
|
||||
}
|
||||
after, err := l.History.Fill()
|
||||
if err != nil {
|
||||
return out, fmt.Errorf("%s's ledger %s removed %s and cannot say the fill of its history after: %w", l.Module, l.Name, r.Instance, err)
|
||||
}
|
||||
out.Operations = uint64(len(ops))
|
||||
if before.Bytes > after.Bytes {
|
||||
out.Bytes = before.Bytes - after.Bytes
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// dropFromIndexes removes the entries of the instance key names from every index bucket the ledger is bound to.
|
||||
func (l *Ledger) dropFromIndexes(key string) error {
|
||||
instance := InstanceOf(key)
|
||||
for name, ix := range l.Indexes {
|
||||
held, err := ix.Entries("", 0)
|
||||
if err != nil {
|
||||
return fmt.Errorf("the index %s: %w", name, err)
|
||||
}
|
||||
for _, e := range held {
|
||||
var v buscalls.IndexEntry
|
||||
if json.Unmarshal(e.Value, &v) == nil && v.Instance == instance {
|
||||
if err := ix.Delete(e.Key); err != nil {
|
||||
return fmt.Errorf("the index %s: %w", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
var owner = buscalls.Author{Account: "ledger.anchor.m"}
|
||||
|
||||
// 1966/AC13, 1966/AC14 and the checks of an operation against its type (ADR 0597 §2, §5, §6, §8; design 25 § 12.4).
|
||||
func TestApplyAppendsDerivesAndSnapshots(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
run func(t *testing.T, l *Ledger, f fakes)
|
||||
}{
|
||||
{"1966/AC13 an instance at sequence 4 is taken with 5, its state and snapshot the effect's result", func(t *testing.T, l *Ledger, f fakes) {
|
||||
for i := 1; i <= 4; i++ {
|
||||
applied(t, l, "decision/0290", "note", fmt.Sprintf(`{"says":"n%d"}`, i), owner)
|
||||
}
|
||||
if last, _ := f.history.Last("decision.0290"); last != 4 {
|
||||
t.Fatalf("the instance is at %d, want 4", last)
|
||||
}
|
||||
got := applied(t, l, "decision/0290", "title", `{"title":"A ledger","ignored":"x"}`, owner)
|
||||
if !got.Taken || got.Sequence != 5 || got.StateBehind {
|
||||
t.Fatalf("answered %+v, want taken with 5", got)
|
||||
}
|
||||
raw, _, _ := f.state.Get("decision.0290")
|
||||
state := record(t, raw)
|
||||
want := `{"author":{"account":"ledger.anchor.m"},"notes":[{"says":"n1"},{"says":"n2"},{"says":"n3"},{"says":"n4"}],"sequence":5,"title":"A ledger"}`
|
||||
if state.Sequence != 5 || string(state.State) != want {
|
||||
t.Fatalf("the state is %d %s, want 5 %s", state.Sequence, state.State, want)
|
||||
}
|
||||
snap, _, _ := f.snapshots.Get("decision.0290")
|
||||
if string(snap) != string(raw) {
|
||||
t.Fatalf("the snapshot is %s, the state %s", snap, raw)
|
||||
}
|
||||
}},
|
||||
{"1966/AC14 a full state bucket answers taken and state behind, and the next apply after room catches up", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/0290", "note", `{"says":"first"}`, owner)
|
||||
f.state.Full = true
|
||||
got := applied(t, l, "decision/0290", "note", `{"says":"while full"}`, owner)
|
||||
if !got.Taken || got.Sequence != 2 || !got.StateBehind {
|
||||
t.Fatalf("answered %+v, want taken with 2 and state behind", got)
|
||||
}
|
||||
if raw, _, _ := f.state.Get("decision.0290"); record(t, raw).Sequence != 1 {
|
||||
t.Fatalf("the full state moved: %s", raw)
|
||||
}
|
||||
f.state.Full = false
|
||||
got = applied(t, l, "decision/0290", "note", `{"says":"after room"}`, owner)
|
||||
raw, _, _ := f.state.Get("decision.0290")
|
||||
state := record(t, raw)
|
||||
if got.StateBehind || state.Sequence != 3 || string(state.State) != `{"notes":[{"says":"first"},{"says":"while full"},{"says":"after room"}]}` {
|
||||
t.Fatalf("after room: %+v, the state %d %s", got, state.Sequence, state.State)
|
||||
}
|
||||
}},
|
||||
{"each effect: set an entry, remove it, remove the instance", func(t *testing.T, l *Ledger, f fakes) {
|
||||
text := "the section's text"
|
||||
applied(t, l, "decision/1", "section", fmt.Sprintf(`{"id":"a","text":%q,"digest":%q}`, text, Digest(text)), owner)
|
||||
applied(t, l, "decision/1", "section", fmt.Sprintf(`{"id":"b","text":"b","digest":%q}`, Digest("b")), owner)
|
||||
applied(t, l, "decision/1", "drop-section", `{"id":"a"}`, owner)
|
||||
raw, _, _ := f.state.Get("decision.1")
|
||||
if got := string(record(t, raw).State); got != fmt.Sprintf(`{"sections":{"b":{"digest":%q}}}`, Digest("b")) {
|
||||
t.Fatalf("the state is %s", got)
|
||||
}
|
||||
applied(t, l, "decision/1", "withdraw", `{}`, owner)
|
||||
raw, _, _ = f.state.Get("decision.1")
|
||||
if got := record(t, raw); got.Sequence != 4 || string(got.State) != "null" {
|
||||
t.Fatalf("the removed instance is %d %s", got.Sequence, got.State)
|
||||
}
|
||||
}},
|
||||
{"an operation outside its type, a body off its schema and a digest off its text are refused", func(t *testing.T, l *Ledger, f fakes) {
|
||||
for _, c := range []struct{ instance, operation, body, refusal string }{
|
||||
{"listing/a17", "title", `{}`, `no type "listing"`},
|
||||
{"decision/0290", "accept", `{}`, `no operation "accept"`},
|
||||
{"decision/0290", "title", `{"title":7}`, "field title is a string"},
|
||||
{"decision/0290", "section", `{"id":"a","text":"x","digest":"sha256:00"}`, "is not the digest of its text"},
|
||||
{"decision/0290", "section", `{"text":"x"}`, "is not the digest"},
|
||||
{"decision/0290", "drop-section", `{}`, "names its entry by its field id"},
|
||||
{"decision", "title", `{}`, "is not an instance"},
|
||||
{"decision/a.b", "title", `{}`, "is not an instance"},
|
||||
} {
|
||||
_, err := l.Apply(buscalls.ApplyRequest{Instance: c.instance, Operation: c.operation, Body: json.RawMessage(c.body)}, owner)
|
||||
if err == nil || !strings.Contains(err.Error(), c.refusal) {
|
||||
t.Errorf("%s %s %s: %v, want %q", c.instance, c.operation, c.body, err, c.refusal)
|
||||
}
|
||||
}
|
||||
if last, _ := f.history.Last("decision.0290"); last != 0 {
|
||||
t.Fatalf("a refused operation was appended: %d", last)
|
||||
}
|
||||
}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
l, f := newLedger()
|
||||
c.run(t, l, f)
|
||||
})
|
||||
}
|
||||
// design 47 G §587: a ledger's fill is its history's against the ledger's cap, a store that cannot say it is an error naming
|
||||
// the ledger; a removal takes an instance's operations, snapshot, state and index entries and answers what that was, an
|
||||
// unknown instance answers none, and nothing of another instance goes.
|
||||
t.Run("TheLedgerSaysItsFillAndRemovesAnInstanceWhole", func(t *testing.T) {
|
||||
l, f := newLedger()
|
||||
f.history.MaxBytes = 1 << 20
|
||||
for _, n := range []string{"a", "b"} {
|
||||
applied(t, l, "decision/0290", "note", `{"says":"`+n+`"}`, owner)
|
||||
}
|
||||
applied(t, l, "decision/0291", "note", `{"says":"other"}`, owner)
|
||||
entry, _ := json.Marshal(buscalls.IndexEntry{Instance: "decision/0290", Sequence: 2})
|
||||
other, _ := json.Marshal(buscalls.IndexEntry{Instance: "decision/0291", Sequence: 1})
|
||||
_ = f.index.Put("t.decision.0290", entry)
|
||||
_ = f.index.Put("t.decision.0291", other)
|
||||
|
||||
fill, err := l.Fill(buscalls.LedgerRequest{Ledger: "records"})
|
||||
if err != nil || fill.Count != 3 || fill.Bytes == 0 || fill.MaxBytes != 1<<20 {
|
||||
t.Fatalf("the fill is %+v (%v), want 3 messages against 1 MiB", fill, err)
|
||||
}
|
||||
got, err := l.Remove(buscalls.LedgerRemoveRequest{Ledger: "records", Instance: "decision/0290"})
|
||||
if err != nil || got.Instance != "decision/0290" || got.Operations != 2 || got.Bytes == 0 {
|
||||
t.Fatalf("the removal answered %+v (%v), want 2 operations and their bytes", got, err)
|
||||
}
|
||||
if kept, _ := f.history.Read("decision.0290", 1, 0); len(kept) != 0 {
|
||||
t.Errorf("operations are left: %d", len(kept))
|
||||
}
|
||||
if raw, _, _ := f.state.Get("decision.0290"); raw != nil {
|
||||
t.Errorf("the state is left: %s", raw)
|
||||
}
|
||||
if raw, _, _ := f.snapshots.Get("decision.0290"); raw != nil {
|
||||
t.Errorf("the snapshot is left: %s", raw)
|
||||
}
|
||||
if held, _ := f.index.Entries("", 0); len(held) != 1 || held[0].Key != "t.decision.0291" {
|
||||
t.Errorf("the index holds %v, want the other instance's entry alone", held)
|
||||
}
|
||||
if kept, _ := f.history.Read("decision.0291", 1, 0); len(kept) != 1 {
|
||||
t.Errorf("another instance lost operations: %d left", len(kept))
|
||||
}
|
||||
if again, err := l.Remove(buscalls.LedgerRemoveRequest{Ledger: "records", Instance: "decision/0290"}); err != nil || again.Operations != 0 || again.Bytes != 0 {
|
||||
t.Errorf("a removal said twice answered %+v (%v), want none", again, err)
|
||||
}
|
||||
f.history.FillFails = true
|
||||
if _, err := l.Fill(buscalls.LedgerRequest{}); err == nil || !strings.Contains(err.Error(), "ledger records cannot say the fill") {
|
||||
t.Errorf("a history that cannot say its fill answered %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// The memory stores keep what the bus keeps, as the bus answers it, for a test of a ledger without a bus: a stream whose
|
||||
// sequence counts across every instance, a bucket with revisions that may be full, and snapshots written with an expected
|
||||
// sequence.
|
||||
|
||||
// MemoryHistory is a history in memory.
|
||||
type MemoryHistory struct {
|
||||
// MaxBytes is the cap its fill is said against.
|
||||
MaxBytes uint64
|
||||
// FillFails makes Fill an error, as a store that cannot say its fill.
|
||||
FillFails bool
|
||||
mu sync.Mutex
|
||||
seq uint64
|
||||
kept []struct {
|
||||
key string
|
||||
Kept
|
||||
}
|
||||
}
|
||||
|
||||
// Matches is whether a key pattern (`*` one name, `>` the rest) names a key.
|
||||
func Matches(pattern, key string) bool {
|
||||
p, k := strings.Split(pattern, "."), strings.Split(key, ".")
|
||||
for i, part := range p {
|
||||
if part == ">" {
|
||||
return len(k) > i
|
||||
}
|
||||
if i >= len(k) || (part != "*" && part != k[i]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return len(p) == len(k)
|
||||
}
|
||||
|
||||
func (h *MemoryHistory) Last(key string) (uint64, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
last := uint64(0)
|
||||
for _, k := range h.kept {
|
||||
if k.key == key {
|
||||
last = k.Sequence
|
||||
}
|
||||
}
|
||||
return last, nil
|
||||
}
|
||||
|
||||
func (h *MemoryHistory) Append(key string, op []byte, expect uint64) (uint64, error) {
|
||||
last, _ := h.Last(key)
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if last != expect {
|
||||
return 0, ErrRaced
|
||||
}
|
||||
h.seq++
|
||||
h.kept = append(h.kept, struct {
|
||||
key string
|
||||
Kept
|
||||
}{key, Kept{Sequence: h.seq, Data: op}})
|
||||
return h.seq, nil
|
||||
}
|
||||
|
||||
func (h *MemoryHistory) Read(keys string, from uint64, limit int) ([]Kept, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
var out []Kept
|
||||
for _, k := range h.kept {
|
||||
if k.Sequence >= from && Matches(keys, k.key) && (limit == 0 || len(out) < limit) {
|
||||
out = append(out, k.Kept)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (h *MemoryHistory) Purge(key string) error {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
kept := h.kept[:0]
|
||||
for _, k := range h.kept {
|
||||
if k.key != key {
|
||||
kept = append(kept, k)
|
||||
}
|
||||
}
|
||||
h.kept = kept
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *MemoryHistory) Fill() (buscalls.Fill, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if h.FillFails {
|
||||
return buscalls.Fill{}, errors.New("the history does not say its fill")
|
||||
}
|
||||
f := buscalls.Fill{Count: uint64(len(h.kept)), MaxBytes: h.MaxBytes}
|
||||
for _, k := range h.kept {
|
||||
f.Bytes += uint64(len(k.Data))
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// Expire drops every operation of the history, as a ledger's keeping days end them.
|
||||
func (h *MemoryHistory) Expire() {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.kept = nil
|
||||
}
|
||||
|
||||
type memoryValue struct {
|
||||
value []byte
|
||||
rev uint64
|
||||
}
|
||||
|
||||
// MemoryState is a state bucket in memory.
|
||||
type MemoryState struct {
|
||||
mu sync.Mutex
|
||||
rev uint64
|
||||
// Full makes every write refused as a full bucket's.
|
||||
Full bool
|
||||
kv map[string]memoryValue
|
||||
}
|
||||
|
||||
func (s *MemoryState) Get(key string) ([]byte, uint64, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
v := s.kv[key]
|
||||
return v.value, v.rev, nil
|
||||
}
|
||||
|
||||
func (s *MemoryState) Write(key string, value []byte, revision uint64) (uint64, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.Full {
|
||||
return 0, ErrFull
|
||||
}
|
||||
if s.kv[key].rev != revision {
|
||||
return 0, ErrChanged
|
||||
}
|
||||
if s.kv == nil {
|
||||
s.kv = map[string]memoryValue{}
|
||||
}
|
||||
s.rev++
|
||||
s.kv[key] = memoryValue{value, s.rev}
|
||||
return s.rev, nil
|
||||
}
|
||||
|
||||
func (s *MemoryState) Keys() ([]string, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var out []string
|
||||
for k := range s.kv {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (s *MemoryState) Delete(key string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
delete(s.kv, key)
|
||||
return nil
|
||||
}
|
||||
|
||||
// MemorySnapshots are snapshots in memory.
|
||||
type MemorySnapshots struct {
|
||||
mu sync.Mutex
|
||||
at uint64
|
||||
kv map[string]memoryValue
|
||||
}
|
||||
|
||||
func (s *MemorySnapshots) Get(key string) ([]byte, uint64, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
v := s.kv[key]
|
||||
return v.value, v.rev, nil
|
||||
}
|
||||
|
||||
func (s *MemorySnapshots) Put(key string, value []byte, expect uint64) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.kv[key].rev != expect {
|
||||
return ErrRaced
|
||||
}
|
||||
if s.kv == nil {
|
||||
s.kv = map[string]memoryValue{}
|
||||
}
|
||||
s.at++
|
||||
s.kv[key] = memoryValue{value, s.at}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *MemorySnapshots) Purge(key string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
delete(s.kv, key)
|
||||
return nil
|
||||
}
|
||||
|
||||
// MemoryIndex is an index's bucket in memory.
|
||||
type MemoryIndex struct {
|
||||
mu sync.Mutex
|
||||
kv map[string][]byte
|
||||
}
|
||||
|
||||
func (x *MemoryIndex) Entries(pattern string, limit int) ([]Indexed, error) {
|
||||
x.mu.Lock()
|
||||
defer x.mu.Unlock()
|
||||
if pattern == "" {
|
||||
pattern = ">"
|
||||
}
|
||||
var keys []string
|
||||
for k := range x.kv {
|
||||
if Matches(pattern, k) {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
}
|
||||
sort.Strings(keys)
|
||||
var out []Indexed
|
||||
for _, k := range keys {
|
||||
if limit > 0 && len(out) >= limit {
|
||||
break
|
||||
}
|
||||
out = append(out, Indexed{Key: k, Value: x.kv[k]})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (x *MemoryIndex) Put(key string, value []byte) error {
|
||||
x.mu.Lock()
|
||||
defer x.mu.Unlock()
|
||||
if x.kv == nil {
|
||||
x.kv = map[string][]byte{}
|
||||
}
|
||||
x.kv[key] = value
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *MemoryIndex) Delete(key string) error {
|
||||
x.mu.Lock()
|
||||
defer x.mu.Unlock()
|
||||
delete(x.kv, key)
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// Reading a ledger (ADR 0597 §10, design 25 § 12.5; issue 1966 T3): an instance's state, its history, a watch of its
|
||||
// states, and a declared index. Only the owning module reads its ledger: the runtime opens a ledger for the module whose
|
||||
// bundle asks, never one the request names.
|
||||
|
||||
// Index is one declared index's bucket, kept by a watcher in the owning module's process and rebuilt by reindex (ADR
|
||||
// 0597 §4).
|
||||
type Index interface {
|
||||
// Entries is the entries whose key a pattern names (`*` one name, `>` the rest; empty is every key), at most limit
|
||||
// (every one when 0), each as the bucket holds it.
|
||||
Entries(pattern string, limit int) ([]Indexed, error)
|
||||
// Put writes one entry; Delete removes one.
|
||||
Put(key string, value []byte) error
|
||||
Delete(key string) error
|
||||
}
|
||||
|
||||
// Indexed is one key of an index's bucket and its value, a buscalls.IndexEntry as JSON.
|
||||
type Indexed struct {
|
||||
Key string
|
||||
Value []byte
|
||||
}
|
||||
|
||||
// Watcher is a state bucket that hands each instance's record as it is and as it changes, under a key pattern.
|
||||
type Watcher interface {
|
||||
Watch(pattern string, deliver func(key string, value []byte, current bool) error) (func(), error)
|
||||
}
|
||||
|
||||
// HistoryMost is the most operations one history read answers.
|
||||
const HistoryMost = 1000
|
||||
|
||||
// Get is an instance's state as the state holds it, and the sequence of its last operation in it; null and 0 for an
|
||||
// instance with none.
|
||||
func (l *Ledger) Get(r buscalls.InstanceRequest) (buscalls.InstanceState, error) {
|
||||
key, err := KeyOf(r.Instance)
|
||||
if err != nil {
|
||||
return buscalls.InstanceState{}, err
|
||||
}
|
||||
rec, _, err := l.readState(key)
|
||||
if err != nil {
|
||||
return buscalls.InstanceState{}, err
|
||||
}
|
||||
state := rec.State
|
||||
if len(state) == 0 {
|
||||
state = json.RawMessage("null")
|
||||
}
|
||||
return buscalls.InstanceState{Instance: r.Instance, State: state, Sequence: rec.Sequence}, nil
|
||||
}
|
||||
|
||||
// operationsOf reads kept operations as the history answers them, each with its sequence.
|
||||
func operationsOf(kept []Kept) ([]buscalls.Operation, error) {
|
||||
out := make([]buscalls.Operation, 0, len(kept))
|
||||
for _, k := range kept {
|
||||
var o buscalls.Operation
|
||||
if err := json.Unmarshal(k.Data, &o); err != nil {
|
||||
return nil, fmt.Errorf("the history's operation %d is not one: %w", k.Sequence, err)
|
||||
}
|
||||
o.Sequence = k.Sequence
|
||||
out = append(out, o)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ReadHistory is an instance's operations from a sequence on, oldest first, each with its sequence and its proven author.
|
||||
func (l *Ledger) ReadHistory(r buscalls.HistoryRequest) (buscalls.HistoryPage, error) {
|
||||
key, err := KeyOf(r.Instance)
|
||||
if err != nil {
|
||||
return buscalls.HistoryPage{}, err
|
||||
}
|
||||
limit := r.Limit
|
||||
if limit <= 0 || limit > HistoryMost {
|
||||
limit = HistoryMost
|
||||
}
|
||||
last, err := l.History.Last(key)
|
||||
if err != nil {
|
||||
return buscalls.HistoryPage{}, err
|
||||
}
|
||||
kept, err := l.History.Read(key, r.From, limit+1)
|
||||
if err != nil {
|
||||
return buscalls.HistoryPage{}, err
|
||||
}
|
||||
page := buscalls.HistoryPage{Last: last}
|
||||
if len(kept) > limit {
|
||||
page.Next = kept[limit].Sequence
|
||||
kept = kept[:limit]
|
||||
}
|
||||
if page.Operations, err = operationsOf(kept); err != nil {
|
||||
return buscalls.HistoryPage{}, err
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
|
||||
// Watch hands deliver the state of every instance Instances names (`*` one name, `>` the rest, in an instance's path),
|
||||
// the current ones first, then each change.
|
||||
func (l *Ledger) Watch(r buscalls.LedgerWatchRequest, deliver func(buscalls.InstanceChange) error) (func(), error) {
|
||||
w, ok := l.State.(Watcher)
|
||||
if !ok {
|
||||
return nil, errors.New("this ledger's state cannot be watched")
|
||||
}
|
||||
pattern := ">"
|
||||
if r.Instances != "" {
|
||||
pattern = keyPattern(r.Instances)
|
||||
}
|
||||
return w.Watch(pattern, func(key string, value []byte, current bool) error {
|
||||
var rec Record
|
||||
if value != nil {
|
||||
if err := json.Unmarshal(value, &rec); err != nil {
|
||||
return fmt.Errorf("the state of %s is not a record: %w", key, err)
|
||||
}
|
||||
}
|
||||
state := rec.State
|
||||
if len(state) == 0 {
|
||||
state = json.RawMessage("null")
|
||||
}
|
||||
return deliver(buscalls.InstanceChange{Watch: r.Watch, Instance: InstanceOf(key), State: state, Sequence: rec.Sequence,
|
||||
Current: current})
|
||||
})
|
||||
}
|
||||
|
||||
// keyPattern is an instance pattern as a key pattern: its path's parts joined by dots.
|
||||
func keyPattern(instances string) string {
|
||||
out := []byte(instances)
|
||||
for i, c := range out {
|
||||
if c == '/' {
|
||||
out[i] = '.'
|
||||
}
|
||||
}
|
||||
return string(out)
|
||||
}
|
||||
|
||||
// index is a declared index by its name, or the reason there is none.
|
||||
func (l *Ledger) index(name string) (Index, error) {
|
||||
if ix, ok := l.Indexes[name]; ok {
|
||||
return ix, nil
|
||||
}
|
||||
return nil, fmt.Errorf("%s's ledger %s declares no index %q", l.Module, l.Name, name)
|
||||
}
|
||||
|
||||
// Query is a declared index's entries whose key Key names.
|
||||
func (l *Ledger) Query(r buscalls.QueryRequest) (buscalls.QueryAnswer, error) {
|
||||
ix, err := l.index(r.Index)
|
||||
if err != nil {
|
||||
return buscalls.QueryAnswer{}, err
|
||||
}
|
||||
held, err := ix.Entries(r.Key, r.Limit)
|
||||
if err != nil {
|
||||
return buscalls.QueryAnswer{}, err
|
||||
}
|
||||
entries := make([]buscalls.IndexEntry, 0, len(held))
|
||||
for _, h := range held {
|
||||
var e buscalls.IndexEntry
|
||||
if err := json.Unmarshal(h.Value, &e); err != nil {
|
||||
return buscalls.QueryAnswer{}, fmt.Errorf("the index %s's entry %s is not one: %w", r.Index, h.Key, err)
|
||||
}
|
||||
entries = append(entries, e)
|
||||
}
|
||||
return buscalls.QueryAnswer{Entries: entries}, nil
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// 1966/AC15, the other reads of a ledger (get, query) and its repairs (ADR 0597 §8, §10; design 25 § 12.5).
|
||||
func TestReadsAndRepairsOfALedger(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
run func(t *testing.T, l *Ledger, f fakes)
|
||||
}{
|
||||
{"1966/AC15 history answers an instance's operations oldest first with sequence and author", func(t *testing.T, l *Ledger, f fakes) {
|
||||
op := buscalls.Author{Account: "ledger.anchor.m", Caller: "op@shanks"}
|
||||
applied(t, l, "decision/1", "note", `{"says":"one"}`, owner)
|
||||
applied(t, l, "decision/2", "note", `{"says":"elsewhere"}`, owner)
|
||||
applied(t, l, "decision/1", "title", `{"title":"Two"}`, op)
|
||||
applied(t, l, "decision/1", "note", `{"says":"three"}`, owner)
|
||||
page, err := l.ReadHistory(buscalls.HistoryRequest{Instance: "decision/1", Limit: 2})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var got []string
|
||||
for _, o := range page.Operations {
|
||||
got = append(got, fmt.Sprintf("%s@%d %s %s", o.Operation, o.Sequence, o.Author.Account, o.Author.Caller))
|
||||
}
|
||||
if strings.Join(got, "; ") != "note@1 ledger.anchor.m ; title@3 ledger.anchor.m op@shanks" || page.Next != 4 || page.Last != 4 {
|
||||
t.Fatalf("the history is %q, next %d, last %d", got, page.Next, page.Last)
|
||||
}
|
||||
rest, _ := l.ReadHistory(buscalls.HistoryRequest{Instance: "decision/1", From: page.Next})
|
||||
if len(rest.Operations) != 1 || rest.Operations[0].Sequence != 4 || rest.Next != 0 {
|
||||
t.Fatalf("read on: %+v", rest)
|
||||
}
|
||||
}},
|
||||
{"get answers the state and its sequence, null for an instance with none", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
|
||||
got, err := l.Get(buscalls.InstanceRequest{Instance: "decision/1"})
|
||||
if err != nil || got.Sequence != 1 || !strings.Contains(string(got.State), `"title":"One"`) {
|
||||
t.Fatalf("got %+v, %v", got, err)
|
||||
}
|
||||
if none, err := l.Get(buscalls.InstanceRequest{Instance: "decision/9"}); err != nil || string(none.State) != "null" {
|
||||
t.Fatalf("none is %+v, %v", none, err)
|
||||
}
|
||||
}},
|
||||
{"query reads a declared index by key, and an undeclared one is refused", func(t *testing.T, l *Ledger, f fakes) {
|
||||
_ = f.index.Put("One.decision.1", []byte(`{"instance":"decision/1","sequence":1}`))
|
||||
_ = f.index.Put("Two.decision.2", []byte(`{"instance":"decision/2","sequence":2}`))
|
||||
got, err := l.Query(buscalls.QueryRequest{Index: "by-title", Key: "One.>"})
|
||||
if err != nil || len(got.Entries) != 1 || got.Entries[0] != (buscalls.IndexEntry{Instance: "decision/1", Sequence: 1}) {
|
||||
t.Fatalf("queried %+v, %v", got, err)
|
||||
}
|
||||
if _, err := l.Query(buscalls.QueryRequest{Index: "by-date"}); err == nil || !strings.Contains(err.Error(), `no index "by-date"`) {
|
||||
t.Fatalf("an undeclared index: %v", err)
|
||||
}
|
||||
}},
|
||||
{"a lost state rebuilds from the history, an expired one from its snapshot, and lag says what is behind", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
|
||||
applied(t, l, "decision/1", "note", `{"says":"kept"}`, owner)
|
||||
want, _, _ := f.state.Get("decision.1")
|
||||
f.state.kv = nil
|
||||
lag, _ := l.Lag(buscalls.LedgerRequest{})
|
||||
if len(lag.Instances) != 1 || lag.Instances[0].History != 2 || lag.Instances[0].State != 0 || lag.Instances[0].Snapshot != 2 {
|
||||
t.Fatalf("lag says %+v", lag)
|
||||
}
|
||||
if got, err := l.Rebuild(buscalls.RebuildRequest{}); err != nil || got.Written != 1 {
|
||||
t.Fatalf("rebuilt %+v, %v", got, err)
|
||||
}
|
||||
if got, _, _ := f.state.Get("decision.1"); string(record(t, got).State) != string(record(t, want).State) {
|
||||
t.Fatalf("rebuilt from the history %s, want %s", got, want)
|
||||
}
|
||||
f.state.kv = nil
|
||||
f.history.Expire()
|
||||
if got, err := l.Rebuild(buscalls.RebuildRequest{Instance: "decision/1", From: buscalls.RebuildFromSnapshot}); err != nil || got.Written != 1 {
|
||||
t.Fatalf("rebuilt %+v, %v", got, err)
|
||||
}
|
||||
if got, _, _ := f.state.Get("decision.1"); string(got) != string(want) {
|
||||
t.Fatalf("rebuilt from the snapshot %s, want %s", got, want)
|
||||
}
|
||||
}},
|
||||
{"reindex builds a declared index from the states, and lag says an index behind until it does", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
|
||||
applied(t, l, "decision/2", "note", `{"says":"untitled"}`, owner)
|
||||
_ = f.index.Put("stale.decision.9", []byte(`{}`))
|
||||
if lag, _ := l.Lag(buscalls.LedgerRequest{}); len(lag.Indexes) != 1 || lag.Indexes[0].Behind != 1 {
|
||||
t.Fatalf("lag says %+v", lag)
|
||||
}
|
||||
if got, err := l.Reindex(buscalls.ReindexRequest{Index: "by-title"}); err != nil || got.Written != 1 {
|
||||
t.Fatalf("reindexed %+v, %v", got, err)
|
||||
}
|
||||
entries, _ := f.index.Entries("", 0)
|
||||
if len(entries) != 1 || entries[0].Key != buscalls.IndexKey([]string{"One"}, "decision/1") || string(entries[0].Value) != `{"instance":"decision/1","sequence":1}` {
|
||||
t.Fatalf("the index holds %+v", entries)
|
||||
}
|
||||
if lag, _ := l.Lag(buscalls.LedgerRequest{}); len(lag.Indexes) != 0 || len(lag.Instances) != 0 {
|
||||
t.Fatalf("lag after reindex says %+v", lag)
|
||||
}
|
||||
}},
|
||||
{"snapshot rewrites a snapshot behind its state", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
|
||||
f.snapshots.kv = nil
|
||||
if got, err := l.Snapshot(buscalls.InstanceRequest{}); err != nil || got.Written != 1 {
|
||||
t.Fatalf("snapshot %+v, %v", got, err)
|
||||
}
|
||||
state, _, _ := f.state.Get("decision.1")
|
||||
if snap, _, _ := f.snapshots.Get("decision.1"); string(snap) != string(state) {
|
||||
t.Fatalf("the snapshot %s, the state %s", snap, state)
|
||||
}
|
||||
}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
l, f := newLedger()
|
||||
c.run(t, l, f)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,304 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// The ledger's repairs (ADR 0597 §8, §10; design 25 § 12.5; issue 1966 T4): rebuild an instance's state from its history
|
||||
// or its snapshot, rewrite a snapshot behind its state, build a declared index again, and say what is behind its history.
|
||||
// The tool runner serves them as tools of every module owning a ledger, its own ledgers only (the runtime's repairs.go).
|
||||
|
||||
// instances is every instance the ledger knows, by key: those its state holds and those its history names.
|
||||
func (l *Ledger) instances() ([]string, error) {
|
||||
seen := map[string]bool{}
|
||||
keys, err := l.State.Keys()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, k := range keys {
|
||||
seen[k] = true
|
||||
}
|
||||
kept, err := l.History.Read(">", 1, 0)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ops, err := operationsOf(kept)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, o := range ops {
|
||||
if k, err := KeyOf(o.Instance); err == nil {
|
||||
seen[k] = true
|
||||
}
|
||||
}
|
||||
out := make([]string, 0, len(seen))
|
||||
for k := range seen {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// keysOf is the one instance a request names, or every instance when it names none.
|
||||
func (l *Ledger) keysOf(instance string) ([]string, error) {
|
||||
if instance == "" {
|
||||
return l.instances()
|
||||
}
|
||||
k, err := KeyOf(instance)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return []string{k}, nil
|
||||
}
|
||||
|
||||
// Rebuild derives the state of an instance (every instance when it names none) again, from its whole history or from its
|
||||
// snapshot and the operations after it, and writes it, and its snapshot when that is behind. A derivation further back
|
||||
// than the state holds is not written: the state never goes back (ADR 0597 §5).
|
||||
func (l *Ledger) Rebuild(r buscalls.RebuildRequest) (buscalls.Repaired, error) {
|
||||
from := r.From
|
||||
if from == "" {
|
||||
from = buscalls.RebuildFromHistory
|
||||
}
|
||||
if from != buscalls.RebuildFromHistory && from != buscalls.RebuildFromSnapshot {
|
||||
return buscalls.Repaired{}, fmt.Errorf("a rebuild is from %q or %q, not %q", buscalls.RebuildFromHistory, buscalls.RebuildFromSnapshot, from)
|
||||
}
|
||||
keys, err := l.keysOf(r.Instance)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
var out buscalls.Repaired
|
||||
for _, key := range keys {
|
||||
base := Record{}
|
||||
if from == buscalls.RebuildFromSnapshot {
|
||||
if base, _, err = l.readSnapshot(key); err != nil {
|
||||
return out, err
|
||||
}
|
||||
}
|
||||
derived, err := l.replay(key, base, 0)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
held, rev, err := l.readState(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if derived.Sequence < held.Sequence {
|
||||
continue
|
||||
}
|
||||
now, _, err := l.settle(key, derived, rev)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if err := l.writeSnapshot(key, now); err != nil {
|
||||
return out, err
|
||||
}
|
||||
out.Written++
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Snapshot rewrites the snapshot of an instance (every instance when it names none) that is behind its state.
|
||||
func (l *Ledger) Snapshot(r buscalls.InstanceRequest) (buscalls.Repaired, error) {
|
||||
keys, err := l.keysOf(r.Instance)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
var out buscalls.Repaired
|
||||
for _, key := range keys {
|
||||
held, _, err := l.readState(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
snap, at, err := l.readSnapshot(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if held.Sequence == 0 || (at != 0 && snap.Sequence >= held.Sequence) {
|
||||
continue
|
||||
}
|
||||
if err := l.writeSnapshot(key, held); err != nil {
|
||||
return out, err
|
||||
}
|
||||
out.Written++
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// indexEntryOf is an instance's entry in a declared index, as reindex writes it and the owner's watcher keeps it
|
||||
// (mesh-sdk buscalls.IndexKey, buscalls.IndexEntry): its key the values of the fields the index is on, in their order,
|
||||
// then the instance; its value the instance and the sequence of the state it was taken from. False when the state holds
|
||||
// none of those fields. A value that is not text is written as its JSON.
|
||||
func indexEntryOf(on []string, key string, rec Record) (string, []byte, bool) {
|
||||
var state map[string]any
|
||||
if json.Unmarshal(rec.State, &state) != nil || state == nil {
|
||||
return "", nil, false
|
||||
}
|
||||
values := make([]string, 0, len(on))
|
||||
found := false
|
||||
for _, f := range on {
|
||||
v, has := state[f]
|
||||
found = found || has
|
||||
s, ok := v.(string)
|
||||
if !ok {
|
||||
raw, _ := json.Marshal(v)
|
||||
s = string(raw)
|
||||
}
|
||||
values = append(values, s)
|
||||
}
|
||||
if !found {
|
||||
return "", nil, false
|
||||
}
|
||||
instance := InstanceOf(key)
|
||||
value, _ := json.Marshal(buscalls.IndexEntry{Instance: instance, Sequence: rec.Sequence})
|
||||
return buscalls.IndexKey(values, instance), value, true
|
||||
}
|
||||
|
||||
// declared is the fields an index is on and the types declaring it.
|
||||
func (l *Ledger) declared(index string) ([]string, map[string]bool, error) {
|
||||
types := map[string]bool{}
|
||||
var on []string
|
||||
for _, t := range l.Types {
|
||||
for _, ix := range t.Indexes {
|
||||
if ix.Name == index {
|
||||
types[t.Name] = true
|
||||
on = ix.On
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(types) == 0 {
|
||||
return nil, nil, fmt.Errorf("%s's ledger %s declares no index %q", l.Module, l.Name, index)
|
||||
}
|
||||
return on, types, nil
|
||||
}
|
||||
|
||||
// wanted is every entry an index should hold, from the states of the types declaring it.
|
||||
func (l *Ledger) wanted(index string) (map[string][]byte, map[string]uint64, error) {
|
||||
on, types, err := l.declared(index)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
keys, err := l.instances()
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
want, seqs := map[string][]byte{}, map[string]uint64{}
|
||||
for _, key := range keys {
|
||||
name, _, _ := strings.Cut(key, ".")
|
||||
if !types[name] {
|
||||
continue
|
||||
}
|
||||
held, _, err := l.readState(key)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if k, v, ok := indexEntryOf(on, key, held); ok {
|
||||
want[k], seqs[k] = v, held.Sequence
|
||||
}
|
||||
}
|
||||
return want, seqs, nil
|
||||
}
|
||||
|
||||
// Reindex builds a declared index again from the ledger's states, derived from its history: every entry it should hold
|
||||
// written, every other removed.
|
||||
func (l *Ledger) Reindex(r buscalls.ReindexRequest) (buscalls.Repaired, error) {
|
||||
ix, err := l.index(r.Index)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
want, _, err := l.wanted(r.Index)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
held, err := ix.Entries("", 0)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
var out buscalls.Repaired
|
||||
for _, e := range held {
|
||||
if _, keep := want[e.Key]; !keep {
|
||||
if err := ix.Delete(e.Key); err != nil {
|
||||
return out, err
|
||||
}
|
||||
}
|
||||
}
|
||||
keys := make([]string, 0, len(want))
|
||||
for k := range want {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
if err := ix.Put(k, want[k]); err != nil {
|
||||
return out, err
|
||||
}
|
||||
out.Written++
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Lag is every instance whose state or snapshot is behind its history, and how many entries each declared index lacks
|
||||
// or holds from an older state; nothing when nothing is behind.
|
||||
func (l *Ledger) Lag(buscalls.LedgerRequest) (buscalls.Lagged, error) {
|
||||
keys, err := l.instances()
|
||||
if err != nil {
|
||||
return buscalls.Lagged{}, err
|
||||
}
|
||||
var out buscalls.Lagged
|
||||
for _, key := range keys {
|
||||
last, err := l.History.Last(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
held, _, err := l.readState(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
snap, _, err := l.readSnapshot(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if held.Sequence < last || snap.Sequence < last {
|
||||
out.Instances = append(out.Instances, buscalls.InstanceLag{Instance: InstanceOf(key), History: last,
|
||||
State: held.Sequence, Snapshot: snap.Sequence})
|
||||
}
|
||||
}
|
||||
names := make([]string, 0, len(l.Indexes))
|
||||
for name := range l.Indexes {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, name := range names {
|
||||
if _, _, err := l.declared(name); err != nil {
|
||||
// A bucket bound for an index no type declares has nothing it should hold.
|
||||
continue
|
||||
}
|
||||
want, seqs, err := l.wanted(name)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
held, err := l.Indexes[name].Entries("", 0)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
at := map[string]uint64{}
|
||||
for _, e := range held {
|
||||
var v buscalls.IndexEntry
|
||||
_ = json.Unmarshal(e.Value, &v)
|
||||
at[e.Key] = v.Sequence
|
||||
}
|
||||
behind := uint64(0)
|
||||
for k := range want {
|
||||
if s, has := at[k]; !has || s < seqs[k] {
|
||||
behind++
|
||||
}
|
||||
}
|
||||
if behind > 0 {
|
||||
out.Indexes = append(out.Indexes, buscalls.IndexLag{Index: name, Behind: behind})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// records is a ledger type of each effect: a title set, a section set as an entry, a note appended, a section dropped,
|
||||
// and the whole instance removed.
|
||||
var records = catalogue.LedgerType{
|
||||
Name: "decision",
|
||||
Schema: map[string]catalogue.LedgerField{
|
||||
"title": {Type: "string"}, "text": {Type: "string"}, "digest": {Type: "string", DigestOf: "text"},
|
||||
},
|
||||
Operations: []catalogue.LedgerOperation{
|
||||
{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}, Sequence: true, Author: true},
|
||||
{Name: "section", Effect: catalogue.EffectSetEntry, Into: "sections", Key: "id", Fields: []string{"digest"}},
|
||||
{Name: "note", Effect: catalogue.EffectAppend, Into: "notes", Fields: []string{"says"}},
|
||||
{Name: "drop-section", Effect: catalogue.EffectRemove, Into: "sections", Key: "id"},
|
||||
{Name: "withdraw", Effect: catalogue.EffectRemove},
|
||||
},
|
||||
Indexes: []catalogue.LedgerIndex{{Name: "by-title", On: []string{"title"}}},
|
||||
}
|
||||
|
||||
type fakes struct {
|
||||
history *MemoryHistory
|
||||
state *MemoryState
|
||||
snapshots *MemorySnapshots
|
||||
index *MemoryIndex
|
||||
}
|
||||
|
||||
func newLedger() (*Ledger, fakes) {
|
||||
f := fakes{&MemoryHistory{}, &MemoryState{}, &MemorySnapshots{}, &MemoryIndex{}}
|
||||
return &Ledger{Module: "m", Name: "records", Types: []catalogue.LedgerType{records}, History: f.history, State: f.state,
|
||||
Snapshots: f.snapshots, Indexes: map[string]Index{"by-title": f.index}}, f
|
||||
}
|
||||
|
||||
// applied applies an operation and fails the test on a refusal.
|
||||
func applied(t *testing.T, l *Ledger, instance, operation, body string, author buscalls.Author) buscalls.Applied {
|
||||
t.Helper()
|
||||
got, err := l.Apply(buscalls.ApplyRequest{Ledger: l.Name, Instance: instance, Operation: operation, Body: json.RawMessage(body)}, author)
|
||||
if err != nil {
|
||||
t.Fatalf("%s %s: %v", operation, body, err)
|
||||
}
|
||||
return got
|
||||
}
|
||||
|
||||
// record is the instance's record a store holds, read as a Record.
|
||||
func record(t *testing.T, raw []byte) Record {
|
||||
t.Helper()
|
||||
var r Record
|
||||
if raw != nil {
|
||||
if err := json.Unmarshal(raw, &r); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return r
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
// Package meshtest raises what the controller would, for tests against a real bus: the ASSIGNMENTS
|
||||
// Package meshtest raises what the mesh would, for tests against a real bus: the ASSIGNMENTS
|
||||
// and EVENTS streams, memberships issued by hand, and a fixture's path.
|
||||
//
|
||||
// **Every test has a bus of its own** (novox/hq ADR 0237, the controller's internal/testbus): a server
|
||||
// **Every test has a bus of its own** (novox/hq ADR 0237, the old testbus): a server
|
||||
// linked in at the release go.mod pins — the release the mesh runs, held so by a test beside this one —
|
||||
// started for the one test and gone after it. The packages shared one bus and had to run one at a time:
|
||||
// each test raised the streams afresh, and the console discovers every runtime that announces itself on
|
||||
@@ -11,6 +11,7 @@
|
||||
package meshtest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -20,8 +21,14 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
assigner "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
|
||||
seatassigner "git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
|
||||
meshcatalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
@@ -67,7 +74,7 @@ func URL(t *testing.T) string {
|
||||
// buses are the running tests' buses, by test.
|
||||
var buses sync.Map
|
||||
|
||||
// Mesh is the controller's job, done by hand.
|
||||
// Mesh is the mesh's job, done by hand.
|
||||
type Mesh struct {
|
||||
nc *nats.Conn
|
||||
js nats.JetStreamContext
|
||||
@@ -81,11 +88,10 @@ func New(t *testing.T) *Mesh {
|
||||
t.Fatal(err)
|
||||
}
|
||||
js, _ := nc.JetStream()
|
||||
for _, s := range []string{"ASSIGNMENTS", "EVENTS"} {
|
||||
_ = js.DeleteStream(s)
|
||||
}
|
||||
if _, err := js.AddStream(&nats.StreamConfig{Name: "ASSIGNMENTS", Subjects: []string{"mesh.assignment.>"},
|
||||
MaxMsgsPerSubject: 1, AllowDirect: true}); err != nil {
|
||||
_ = js.DeleteStream("EVENTS")
|
||||
// The grants part's state memberships, where a runtime reads what is issued it (ADR 0571 clause 10).
|
||||
_ = js.DeleteKeyValue(membership.GrantsBucket)
|
||||
if _, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: membership.GrantsBucket, History: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := js.AddStream(&nats.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
|
||||
@@ -95,15 +101,124 @@ func New(t *testing.T) *Mesh {
|
||||
return &Mesh{nc: nc, js: js}
|
||||
}
|
||||
|
||||
// Issue publishes a membership.
|
||||
// Issue writes a membership where the grants part keeps it, its state memberships, key `<node>.<module>` (novox/hq ADR
|
||||
// 0571 clause 7 and 10; mesh-sdk membership.GrantsBucket), the one place a runtime reads it. It writes with
|
||||
// mesh-sdk's encoder, the grants part's own, never json of this package's type: a value built here by hand agreed with
|
||||
// the reader's mistake and hid that the part wrote another shape (issue 774).
|
||||
func (m *Mesh) Issue(t *testing.T, mem bus.Membership) {
|
||||
t.Helper()
|
||||
body, _ := json.Marshal(mem)
|
||||
if _, err := m.js.Publish(bus.MembershipSubject(mem.Node, mem.Module), body); err != nil {
|
||||
m.putGrants(t, mem, membership.EncodeGrantsMembership(sdkMembership(t, mem)))
|
||||
}
|
||||
|
||||
// IssueInGrantsWrapped writes a membership as the grants part wrote it before mesh-sdk's codec (until 2026-10-30, what
|
||||
// the live mesh holds until the part writes again): its pass record, the membership inside, published saying what
|
||||
// became of it ("published", "unchanged", or why it was held back).
|
||||
func (m *Mesh) IssueInGrantsWrapped(t *testing.T, mem bus.Membership, published string) {
|
||||
t.Helper()
|
||||
body, _ := json.Marshal(map[string]any{"node": mem.Node, "module": mem.Module, "digest": "sha256:0",
|
||||
"membership": sdkMembership(t, mem), "published": published})
|
||||
m.putGrants(t, mem, body)
|
||||
}
|
||||
|
||||
func (m *Mesh) putGrants(t *testing.T, mem bus.Membership, body []byte) {
|
||||
t.Helper()
|
||||
kv, err := m.js.KeyValue(membership.GrantsBucket)
|
||||
if err != nil {
|
||||
if kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: membership.GrantsBucket, History: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if _, err := kv.Put(membership.GrantsKey(mem.Node, mem.Module), body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// DropGrants deletes the grants part's state memberships, as a bus that has not made it yet holds none; Issue makes it again.
|
||||
func (m *Mesh) DropGrants(t *testing.T) {
|
||||
t.Helper()
|
||||
if err := m.js.DeleteKeyValue(membership.GrantsBucket); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// sdkMembership is a membership of this package's type as mesh-sdk's, which the grants part composes and writes.
|
||||
func sdkMembership(t *testing.T, mem bus.Membership) membership.Membership {
|
||||
t.Helper()
|
||||
raw, _ := json.Marshal(mem)
|
||||
var out membership.Membership
|
||||
if err := json.Unmarshal(raw, &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// The states the console reads, and their buckets (the test bed's own names: the indexer's client names
|
||||
// no holder).
|
||||
const (
|
||||
catalogHolder = "mesh-catalog"
|
||||
assignerHolder = "mesh-module-assigner" // the seat's name stands for its holder in the console's state names (firstview.AssignmentsRead)
|
||||
// AssignmentsBucket and SeatHoldersBucket are the assigner's states assignments and seat-holders.
|
||||
AssignmentsBucket = assignerHolder + "_assignments"
|
||||
SeatHoldersBucket = assignerHolder + "_seat-holders"
|
||||
// ModulesBucket holds the catalog half's entries by module, the test bed's own stand-in for the runner's first views of it.
|
||||
ModulesBucket = catalogHolder + "_modules"
|
||||
)
|
||||
|
||||
// ModulesRead names the test bed's state of the catalog half's entries.
|
||||
const ModulesRead = catalogHolder + ".modules"
|
||||
|
||||
// ConsoleReads are the states a console's runtime reads to say what the mesh holds (novox/hq ADR 0571 clauses 4, 5):
|
||||
// assignments, the catalog half's entries, and the seat holders.
|
||||
func ConsoleReads() []bus.StateIssued {
|
||||
return []bus.StateIssued{
|
||||
{Name: assignerHolder + ".assignments", Bucket: AssignmentsBucket},
|
||||
{Name: ModulesRead, Bucket: ModulesBucket},
|
||||
{Name: assignerHolder + ".seat-holders", Bucket: SeatHoldersBucket},
|
||||
}
|
||||
}
|
||||
|
||||
// Placed writes what the mesh holds as its parts write it, with mesh-sdk's encoders: each module the indexer holds with
|
||||
// its manifest (tools, the tools it declares; none for a module without), and each node's assignment of modules.
|
||||
func (m *Mesh) Placed(t *testing.T, tools map[string][]string, on map[string][]string) {
|
||||
t.Helper()
|
||||
kv := func(bucket string) nats.KeyValue {
|
||||
_ = m.js.DeleteKeyValue(bucket)
|
||||
b, err := m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: bucket, History: 1, MaxValueSize: 256 * 1024})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
modules, assigned := kv(ModulesBucket), kv(AssignmentsBucket)
|
||||
kv(SeatHoldersBucket)
|
||||
for module, declared := range tools {
|
||||
manifest := map[string]any{"module": module, "tools": declared}
|
||||
raw, err := json.Marshal(meshcatalog.CatalogEntry{Module: module, Manifest: manifest, Commit: strings.Repeat("a", 40)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := modules.Put(module, raw); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for node, placed := range on {
|
||||
p := seatassigner.Assignments{V: assigner.Version, Node: node}
|
||||
for _, module := range placed {
|
||||
p.Modules = append(p.Modules, seatassigner.PlacedModule{Module: module})
|
||||
}
|
||||
// As the seat writes it: its node key and a key per module (research 072 row 37).
|
||||
values, err := assigner.AssignmentsValues(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for key, raw := range values {
|
||||
if _, err := assigned.Put(key, raw); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// NextEvent is the subject the next event under a pattern lands on.
|
||||
func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
|
||||
t.Helper()
|
||||
@@ -122,7 +237,7 @@ func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
|
||||
return ch
|
||||
}
|
||||
|
||||
// MembershipOf is a membership as the controller issues one on a machine.
|
||||
// MembershipOf is a membership as the mesh issues one on a machine.
|
||||
func MembershipOf(module, node string, plain bool, seats map[string][]string) bus.Membership {
|
||||
own := "mesh.mod." + module
|
||||
serves := []bus.Served{{Subject: own + ".tool.{tool}." + node}}
|
||||
@@ -183,7 +298,7 @@ func (l *Logs) Has(fragments ...string) bool {
|
||||
// All is every line.
|
||||
func (l *Logs) All() string { return strings.Join(l.lines, "\n") }
|
||||
|
||||
// Consumer makes a module's durable consumer on a machine as the controller does: pull, on the
|
||||
// Consumer makes a module's durable consumer on a machine as the mesh does: pull, on the
|
||||
// EVENTS stream, filtered to what the module consumes, with a short ack wait so a test sees a
|
||||
// redelivery in seconds rather than the mesh's minutes.
|
||||
func (m *Mesh) Consumer(t *testing.T, node, module string, filters []string, ackWait time.Duration) {
|
||||
@@ -209,6 +324,18 @@ func (m *Mesh) Emit(t *testing.T, subject string, body any) {
|
||||
}
|
||||
}
|
||||
|
||||
// Publish says body on a plain subject, as a core publish with no stream behind it.
|
||||
func (m *Mesh) Publish(t *testing.T, subject string, body any) {
|
||||
t.Helper()
|
||||
data, _ := json.Marshal(body)
|
||||
if err := m.nc.Publish(subject, data); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := m.nc.Flush(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Pending is what a module's consumer still holds: delivered and not acknowledged, and not yet
|
||||
// delivered.
|
||||
func (m *Mesh) Pending(t *testing.T, node, module string) (ackPending, notDelivered uint64) {
|
||||
@@ -231,7 +358,7 @@ func (m *Mesh) BucketKeeping(t *testing.T, name string, history uint8) nats.KeyV
|
||||
return kv
|
||||
}
|
||||
|
||||
// Bucket makes a module's state afresh as the controller does from the catalogue (novox/hq ADR 0201),
|
||||
// Bucket makes a module's state afresh as the mesh does from the catalogue (novox/hq ADR 0201),
|
||||
// and answers it for writing what is there before a bundle starts.
|
||||
func (m *Mesh) Bucket(t *testing.T, name string) nats.KeyValue {
|
||||
t.Helper()
|
||||
@@ -243,10 +370,27 @@ func (m *Mesh) Bucket(t *testing.T, name string) nats.KeyValue {
|
||||
return kv
|
||||
}
|
||||
|
||||
// Log makes a module's log afresh as the controller does from the catalogue (novox/hq ADR 0297): a file
|
||||
// BucketKeyTTL makes a module's state afresh as the mesh does for one declared `key-ttl` (mesh-sdk buscompose:
|
||||
// a time to live per key allowed, its end leaving a marker kept a minute; issue 1952).
|
||||
func (m *Mesh) BucketKeyTTL(t *testing.T, name string) {
|
||||
t.Helper()
|
||||
_ = m.js.DeleteKeyValue(name)
|
||||
js, err := jetstream.New(m.nc)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: 1, MaxValueSize: 256 * 1024,
|
||||
LimitMarkerTTL: time.Minute}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Log makes a module's log afresh as the mesh does from the catalogue (novox/hq ADR 0297): a file
|
||||
// stream named LOG_<module>_<name> on mesh.log.<module>.<name>.>, limits retention, no age, no cap per
|
||||
// key, the declared cap, 260 KiB a message (a 256 KiB value and its headers), refusing new entries when full, direct get, no delete or
|
||||
// purge. It answers the membership entry the controller issues for it.
|
||||
// purge. It answers the membership entry the mesh issues for it.
|
||||
func (m *Mesh) Log(t *testing.T, module, name string, maxBytes int64) bus.LogIssued {
|
||||
t.Helper()
|
||||
stream := "LOG_" + module + "_" + name
|
||||
@@ -272,7 +416,7 @@ func (m *Mesh) Consumers(t *testing.T, stream string) int {
|
||||
return info.State.Consumers
|
||||
}
|
||||
|
||||
// ModuleStream makes a module's own stream afresh as the controller does from the catalogue (novox/hq ADR
|
||||
// ModuleStream makes a module's own stream afresh as the mesh does from the catalogue (novox/hq ADR
|
||||
// 0363 §1): named in capitals, taking `mesh.stream.<module>.event.>`, a subject space EVENTS does not take.
|
||||
func (m *Mesh) ModuleStream(t *testing.T, module, name string) {
|
||||
t.Helper()
|
||||
@@ -318,7 +462,7 @@ func (m *Mesh) Messages(t *testing.T, stream string) uint64 {
|
||||
return info.State.Msgs
|
||||
}
|
||||
|
||||
// RemoveConsumer deletes a module's consumer on a machine on a stream, as the controller does when a grant
|
||||
// RemoveConsumer deletes a module's consumer on a machine on a stream, as the mesh does when a grant
|
||||
// it cannot narrow is withdrawn (novox/hq ADR 0363 §6).
|
||||
func (m *Mesh) RemoveConsumer(t *testing.T, stream, node, module string) {
|
||||
t.Helper()
|
||||
|
||||
@@ -110,6 +110,13 @@ func TestTheRepositoryHoldsNoCheckScriptAndNodeToolsIsItsOnlyModule(t *testing.T
|
||||
if !d.IsDir() && d.Name() == "module.json" {
|
||||
found = append(found, filepath.ToSlash(p))
|
||||
}
|
||||
// ADR 0604 section 4: the TypeScript that called the old catalog module's tools is gone, and nothing
|
||||
// under node-tools names one (the needle is built so this file does not name it either).
|
||||
if !d.IsDir() && strings.Contains(filepath.ToSlash(p), "/node-tools/") {
|
||||
if body, rerr := os.ReadFile(p); rerr == nil && strings.Contains(string(body), "catalog"+"_modules") {
|
||||
t.Errorf("%s names a catalog tool of the retired module", p)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
package meshtest
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/keys"
|
||||
)
|
||||
|
||||
// State is a module's state as a fixture, the shape a gate's Store takes: Fail makes every read and write fail, as a bus
|
||||
// refusing them; Vals are its values by key. A key the bus refuses (a name outside letters, digits and -/_=, in
|
||||
// dot-separated names) is refused here too, so no test passes a key the live bus refuses (lead answer A-209).
|
||||
type State struct {
|
||||
mu sync.Mutex
|
||||
Vals map[string]json.RawMessage
|
||||
revs map[string]uint64
|
||||
n uint64
|
||||
Fail error
|
||||
}
|
||||
|
||||
// checkKey refuses a key the bus would.
|
||||
func checkKey(k string) error {
|
||||
for _, part := range strings.Split(k, ".") {
|
||||
if keys.CheckName("key", part) != nil {
|
||||
return fmt.Errorf("%q is not a key the bus can hold: letters, digits and -/_=, in dot-separated names", k)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// NewState is an empty State.
|
||||
func NewState() *State { return &State{Vals: map[string]json.RawMessage{}, revs: map[string]uint64{}} }
|
||||
|
||||
func (m *State) Get(k string) (*stdio.StateEntry, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if err := checkKey(k); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if m.Fail != nil {
|
||||
return nil, m.Fail
|
||||
}
|
||||
v, ok := m.Vals[k]
|
||||
if !ok {
|
||||
return nil, nil
|
||||
}
|
||||
return &stdio.StateEntry{Key: k, Value: v, Revision: m.revs[k]}, nil
|
||||
}
|
||||
func (m *State) Create(k string, v any) (uint64, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if err := checkKey(k); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if m.Fail != nil {
|
||||
return 0, m.Fail
|
||||
}
|
||||
if _, ok := m.Vals[k]; ok {
|
||||
return 0, stdio.ErrChanged
|
||||
}
|
||||
return m.put(k, v)
|
||||
}
|
||||
func (m *State) Update(k string, v any, rev uint64) (uint64, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if err := checkKey(k); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if m.Fail != nil {
|
||||
return 0, m.Fail
|
||||
}
|
||||
if rev == 0 || m.revs[k] != rev {
|
||||
return 0, stdio.ErrChanged
|
||||
}
|
||||
return m.put(k, v)
|
||||
}
|
||||
func (m *State) Put(k string, v any) (uint64, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if err := checkKey(k); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if m.Fail != nil {
|
||||
return 0, m.Fail
|
||||
}
|
||||
return m.put(k, v)
|
||||
}
|
||||
func (m *State) Keys() ([]string, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if m.Fail != nil {
|
||||
return nil, m.Fail
|
||||
}
|
||||
var out []string
|
||||
for k := range m.Vals {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
func (m *State) put(k string, v any) (uint64, error) {
|
||||
raw, ok := v.(json.RawMessage)
|
||||
if !ok {
|
||||
raw, _ = json.Marshal(v)
|
||||
}
|
||||
m.n++
|
||||
m.Vals[k], m.revs[k] = raw, m.n
|
||||
return m.n, nil
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
// Package provide is the runtime's half of node-tools being a provider of `mcp-endpoint`: it runs the mesh SDK's
|
||||
// provider helper, so a consumer handed to the provisioner (claude-code's `${bound:mcp-endpoint:port}`) is answered
|
||||
// from an entry node-tools writes in its own bucket, not from the controller's binding file (novox/hq ADR 0349
|
||||
// from an entry node-tools writes in its own bucket, not from the old binding file (novox/hq ADR 0349
|
||||
// clause 12, issue 791).
|
||||
//
|
||||
// The offer carries no credential: the console's authority is the machine's login (ADR 0152, 0175), so the helper
|
||||
@@ -15,7 +15,8 @@ import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/provider"
|
||||
provisions "git.novox.be/novox/mesh-contracts/provisioning/go/clients/meshprovisionerclient"
|
||||
"git.novox.be/novox/mesh-contracts/provisioning/go/clients/meshprovisionerclient/provider"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
@@ -37,16 +38,23 @@ func Offers() map[string]provider.Offer {
|
||||
}}
|
||||
}
|
||||
|
||||
// Provisioner is the module holding the seat mesh-provisioner, whose provisions and plans the helper reads: named from this
|
||||
// module's manifest (`reads`; TestTheOfferServesThePortTheManifestServes), since no client names a holder (novox/hq ADR 0625 §4).
|
||||
const Provisioner = "mesh-provisioner"
|
||||
|
||||
// busOf is the helper's bus over the runtime's own state: the provider's bucket is its state `bound`, and the
|
||||
// controller's assignments are the state it reads, `mesh-controller.assignments` (both declared in the manifest).
|
||||
// provisions and plans of the seat mesh-provisioner are the states it reads, by mesh-sdk's names (novox/hq ADR 0570,
|
||||
// issue 1954 L6-T1). The old assignments are read no more (mesh-sdk 692dbf9, L6-P3).
|
||||
type busOf struct{ conn *bus.Conn }
|
||||
|
||||
func (b busOf) state(bucket string) (string, error) {
|
||||
switch bucket {
|
||||
case provider.BucketOf(Module):
|
||||
return "bound", nil
|
||||
case provider.AssignmentsBucket:
|
||||
return "mesh-controller.assignments", nil
|
||||
case provisions.ProvisionsBucketOf(Provisioner):
|
||||
return provisions.ProvisionsReadOf(Provisioner), nil
|
||||
case provisions.PlansBucketOf(Provisioner):
|
||||
return provisions.PlansReadOf(Provisioner), nil
|
||||
}
|
||||
return "", fmt.Errorf("%s is not a bucket this provider keeps or reads", bucket)
|
||||
}
|
||||
@@ -57,7 +65,8 @@ func (b busOf) Watch(ctx context.Context, bucket string, handle func(key string,
|
||||
return err
|
||||
}
|
||||
stop, err := b.conn.StateWatch(Module, name, "", func(ch bus.StateChange) error {
|
||||
if ch.Op == "delete" {
|
||||
if ch.Op != "put" {
|
||||
// A delete, or an expiry (issue 1952): the key is gone.
|
||||
handle(ch.Key, nil)
|
||||
} else {
|
||||
handle(ch.Key, ch.Value)
|
||||
@@ -92,7 +101,7 @@ func (b busOf) Delete(_ context.Context, bucket, key string) error {
|
||||
func New(conn *bus.Conn, node string, every time.Duration, logf func(string, ...any)) *provider.Provider {
|
||||
return provider.New(provider.Options{
|
||||
Node: node,
|
||||
Module: Module,
|
||||
Module: Module, Provisioner: Provisioner,
|
||||
Offers: Offers(),
|
||||
Bus: busOf{conn: conn},
|
||||
Every: every,
|
||||
|
||||
@@ -7,6 +7,10 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
provisions "git.novox.be/novox/mesh-contracts/provisioning/go/clients/meshprovisionerclient"
|
||||
"git.novox.be/novox/mesh-contracts/provisioning/go/clients/meshprovisionerclient/provider"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
@@ -41,19 +45,39 @@ func fill(t *testing.T, template string, entry map[string]any) string {
|
||||
func TestTheRuntimeAnswersAHandedConsumersMcpEndpointThroughTheHelper(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
bound := mesh.Bucket(t, "node-tools_bound")
|
||||
assignments := mesh.Bucket(t, "mesh-controller_assignments")
|
||||
plans := mesh.Bucket(t, provisions.PlansBucketOf(Provisioner))
|
||||
provs := mesh.Bucket(t, provisions.ProvisionsBucketOf(Provisioner))
|
||||
|
||||
mem := mt.MembershipOf("node-tools", "anchor", false, nil)
|
||||
mem.State = []bus.StateIssued{
|
||||
{Name: "bound", Bucket: "node-tools_bound", Writes: true},
|
||||
{Name: "mesh-controller.assignments", Bucket: "mesh-controller_assignments"},
|
||||
{Name: provisions.PlansReadOf(Provisioner), Bucket: provisions.PlansBucketOf(Provisioner)},
|
||||
{Name: provisions.ProvisionsReadOf(Provisioner), Bucket: provisions.ProvisionsBucketOf(Provisioner)},
|
||||
}
|
||||
mesh.Issue(t, mem)
|
||||
|
||||
// claude-code, handed on this machine, is bound to node-tools here for mcp-endpoint (issue 788's plan entry).
|
||||
plan := `{"node":"anchor","module":"claude-code","provisioning":{"provisions":[{"provision":"mcp-endpoint",` +
|
||||
`"provider":"anchor","provider-module":"node-tools","at":"127.0.0.1","from":"anchor","as":"mesh_anchor_claude_code"}]}}`
|
||||
if _, err := assignments.Put("anchor.claude-code", []byte(plan)); err != nil {
|
||||
// claude-code, handed on this machine, is bound to node-tools here for mcp-endpoint: its plan and its provision
|
||||
// key, written as mesh-provisioner writes them, with mesh-sdk's encoders (ADR 0570 clauses 1-3, issue 1954).
|
||||
plan, err := provisions.EncodePlan(provisions.Plan{Node: "anchor", Module: "claude-code", Plan: catalogue.Plan{
|
||||
Provisions: []catalogue.PlanProvision{{Provision: "mcp-endpoint", Provider: "anchor", ProviderModule: "node-tools",
|
||||
At: "127.0.0.1", From: "anchor", As: "mesh_anchor_claude_code"}}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := plans.Put("anchor.claude-code", plan); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key, err := provisions.Key("anchor", "claude-code", "mcp-endpoint", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
asked, err := provisions.Encode(provisions.Provision{Provision: "mcp-endpoint", State: provisions.Asked,
|
||||
Consumer: provisions.End{Node: "anchor", Module: "claude-code"}, Provider: provisions.End{Node: "anchor", Module: "node-tools"},
|
||||
At: time.Now()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := provs.Put(key, asked); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -102,6 +126,8 @@ func TestTheOfferServesThePortTheManifestServes(t *testing.T) {
|
||||
var m struct {
|
||||
Serves map[string]map[string]any `json:"serves"`
|
||||
Moved bool `json:"moved"`
|
||||
State []json.RawMessage `json:"state"`
|
||||
Reads []string `json:"reads"`
|
||||
Provides []struct {
|
||||
Name string `json:"name"`
|
||||
AnsweredBy string `json:"answered-by"`
|
||||
@@ -133,11 +159,41 @@ func TestTheOfferServesThePortTheManifestServes(t *testing.T) {
|
||||
t.Error("the manifest still carries a consumers file the helper does not need")
|
||||
}
|
||||
}
|
||||
for _, r := range m.Reads {
|
||||
if r == "mesh-module-assigner.assignments" {
|
||||
t.Error("the manifest still reads mesh-module-assigner.assignments, which the sdk's helper no longer reads (issue 1954)")
|
||||
}
|
||||
}
|
||||
for _, a := range m.Build.Artifacts {
|
||||
if a.Name == "files" || len(a.Env) > 0 {
|
||||
t.Errorf("the build still carries %+v, which only a moved module needs", a)
|
||||
}
|
||||
}
|
||||
// Every bucket the sdk's helper reaches is a state the manifest keeps or reads, by the sdk's names: the provider's
|
||||
// own and the seat mesh-provisioner's provisions and plans (issue 1954 L6-T1); the controller's assignments no more.
|
||||
declared := map[string]bool{}
|
||||
for _, n := range m.Reads {
|
||||
declared[n] = true
|
||||
}
|
||||
for _, r := range m.State { // an old state entry is its name, or an object naming it (approvals, per machine)
|
||||
var e struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
if json.Unmarshal(r, &e.Name) != nil {
|
||||
_ = json.Unmarshal(r, &e)
|
||||
}
|
||||
declared[e.Name] = true
|
||||
}
|
||||
for bucket, want := range map[string]string{
|
||||
provider.BucketOf(Module): "bound",
|
||||
provisions.ProvisionsBucketOf(Provisioner): "mesh-provisioner.provisions",
|
||||
provisions.PlansBucketOf(Provisioner): "mesh-provisioner.plans",
|
||||
} {
|
||||
got, err := (busOf{}).state(bucket)
|
||||
if err != nil || got != want || !declared[got] {
|
||||
t.Errorf("the helper's bucket %s reaches state %q (%v), want %q declared in the manifest", bucket, got, err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Only the machine's runtime is the provider: a process on another credential, or with no machine, answers nobody.
|
||||
@@ -147,3 +203,24 @@ func TestOnlyTheMachinesRuntimeStartsTheHelper(t *testing.T) {
|
||||
Start(nil, c.module, c.node, time.Second, said.Logf)()
|
||||
}
|
||||
}
|
||||
|
||||
// 1126/B19 (ADR 0590): this manifest is not refused by the check that refuses any manifest naming the controller:
|
||||
// it reads, invokes, claims and requires nothing of it.
|
||||
func TestTheManifestNamesNothingOfTheController(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, err := catalogue.Decode(raw)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Read as text, as mesh-catalog's controllerdeps check reads its own: mesh-sdk names no module (issue 774), and the
|
||||
// manifest says it is the tool runner rather than being known by its name.
|
||||
if strings.Contains(string(raw), "mesh-controller") {
|
||||
t.Error("node-tools' manifest names the ended controller")
|
||||
}
|
||||
if !catalogue.IsToolRunner(m) {
|
||||
t.Error("node-tools' manifest does not say it is the tool runner (\"tool-runner\": true)")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,14 +2,19 @@ package runtime
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/micro"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/announce"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
@@ -100,10 +105,6 @@ func TestTheRuntimeAnnouncesWhatItServesInTheServicesProtocol(t *testing.T) {
|
||||
if msg, err := nc.Request("$SRV.PING", nil, 2*time.Second); err != nil || !strings.Contains(string(msg.Data), micro.PingResponseType) {
|
||||
t.Errorf("ping: %v %v", msg, err)
|
||||
}
|
||||
if _, err := nc.Request("$SRV.INFO.somebody-else", nil, 300*time.Millisecond); err == nil {
|
||||
t.Error("answered a request for another service")
|
||||
}
|
||||
|
||||
// alpha is re-issued a plain subject: the announcement says so, and says it is interchangeable.
|
||||
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", true, nil))
|
||||
var after string
|
||||
@@ -120,3 +121,86 @@ func TestTheRuntimeAnnouncesWhatItServesInTheServicesProtocol(t *testing.T) {
|
||||
}
|
||||
_ = bus.Served{}
|
||||
}
|
||||
|
||||
// issue 1966 TLS-3 (ENG-2): the runner states the capabilities of the calls it answers in its PING, on its own instance, which
|
||||
// is what its machine's node-engine asks and takes into the node's record; no bundle is needed for it.
|
||||
func TestTheRunnerStatesTheCapabilitiesOfItsCallsInItsPing(t *testing.T) {
|
||||
t.Setenv(FingerprintWord, "sha256:runner")
|
||||
mesh := mt.New(t)
|
||||
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", false, nil))
|
||||
nodeTools := connect(t, "node-tools", "anchor")
|
||||
stop, err := Run(nodeTools, nil, nil, (&mt.Logs{}).Logf)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer stop()
|
||||
nc, err := nats.Connect(mt.URL(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer nc.Close()
|
||||
msg, err := nc.Request("$SRV.PING.node-tools.anchor", nil, 2*time.Second)
|
||||
if err != nil {
|
||||
t.Fatalf("ping of the instance: %v", err)
|
||||
}
|
||||
var ping micro.Ping
|
||||
if err := json.Unmarshal(msg.Data, &ping); err != nil || ping.Metadata[announce.Capabilities] != "approval-mark/1,bus-state-calls,finding-calls,ledger-calls,ledger-upkeep-calls,put-sessions,seat-settle-delay,sent-objects-calls,stream-publish" {
|
||||
t.Errorf("the runner states %q (%v), want its bus-state and ledger calls", ping.Metadata[announce.Capabilities], err)
|
||||
}
|
||||
// Its own fingerprint beside them (design 46 approval mark §6).
|
||||
if ping.Metadata[announce.Fingerprint] != "sha256:runner" {
|
||||
t.Errorf("the runner states its fingerprint as %q", ping.Metadata[announce.Fingerprint])
|
||||
}
|
||||
// Design 47 G §462, §488, §622: each call is stated under the capability word of its own family, the words mesh-sdk's buscalls
|
||||
// names: the read of a sent object under sent-objects-calls (no longer under bus-state-calls), the ledger's fill and removal
|
||||
// under ledger-upkeep-calls, the others where they were.
|
||||
t.Run("EachCallIsStatedUnderTheWordOfItsFamily", func(t *testing.T) {
|
||||
for method, word := range map[string]string{
|
||||
buscalls.ObjectsSentGet: buscalls.SentObjectsCalls,
|
||||
buscalls.LedgerFill: buscalls.LedgerUpkeepCalls,
|
||||
buscalls.LedgerRemove: buscalls.LedgerUpkeepCalls,
|
||||
buscalls.LedgerApply: buscalls.LedgerCalls,
|
||||
buscalls.ObjectsPut: buscalls.BusStateCalls,
|
||||
buscalls.StreamPublish: buscalls.StreamPublishWord,
|
||||
} {
|
||||
var in []string
|
||||
for c, methods := range callFamilies {
|
||||
if slices.Contains(methods, method) {
|
||||
in = append(in, c)
|
||||
}
|
||||
}
|
||||
if len(in) != 1 || in[0] != word {
|
||||
t.Errorf("%s is stated under %v, want %s alone", method, in, word)
|
||||
}
|
||||
}
|
||||
})
|
||||
// Design 47 G §545: the words the tool runner states are exactly its manifest's artifact `states`, so the delivery plan's edge
|
||||
// of §605 runs to it for each word a module's artifact `calls`.
|
||||
t.Run("TheWordsTheRunnerStatesAreItsManifestsStates", func(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m struct {
|
||||
Build struct {
|
||||
Artifacts []struct {
|
||||
Name string `json:"name"`
|
||||
States []string `json:"states"`
|
||||
} `json:"artifacts"`
|
||||
} `json:"build"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var declared []string
|
||||
for _, a := range m.Build.Artifacts {
|
||||
if a.Name == "runtime" {
|
||||
declared = a.States
|
||||
}
|
||||
}
|
||||
slices.Sort(declared)
|
||||
if got := CallCapabilities(); !slices.Equal(got, declared) {
|
||||
t.Errorf("the runner states %v, its manifest's states are %v", got, declared)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
|
||||
// The capabilities the tool runner states of the calls it answers (novox/hq ADR 0595 §9, ADR 0597; design 25 § 12.4): each is
|
||||
// a family of the methods a bundle asks it, named so a manifest's `capabilities` can ask for it, and stated in its announcement
|
||||
// (announce.Capabilities) for its machine's node-engine to take into the node's record. A module whose manifest asks for one is
|
||||
// held off a machine whose runner does not state it.
|
||||
const (
|
||||
CapBusStateCalls = buscalls.BusStateCalls
|
||||
CapLedgerCalls = buscalls.LedgerCalls
|
||||
// CapLedgerUpkeepCalls is the ledger's fill and removal (§462, §488) and CapSentObjectsCalls the read of a sent object (§488):
|
||||
// each is stated once the runner answers its family, the words mesh-sdk's buscalls names (§622).
|
||||
CapLedgerUpkeepCalls = buscalls.LedgerUpkeepCalls
|
||||
CapSentObjectsCalls = buscalls.SentObjectsCalls
|
||||
// CapPutSessions is the put and get sessions' word (design 47 G §518, §622), mesh-sdk's.
|
||||
CapPutSessions = stdio.CapPutSessions
|
||||
// CapApprovalMark says the runner honours the approval mark: its gate sits on every handler it serves (design 46
|
||||
// 7-approval-mark §6, §20), so the grants part composes no deny of its modules' marked subjects.
|
||||
CapApprovalMark = "approval-mark/1"
|
||||
// CapStreamPublish is the call that says an event on the module's own bound stream (design 42 zfs-owns-zfs, A-44), the
|
||||
// word mesh-sdk's buscalls names (§622).
|
||||
CapStreamPublish = buscalls.StreamPublishWord
|
||||
)
|
||||
|
||||
// callFamilies are the methods each capability stands for, the ones BusState and BusStateWatch answer.
|
||||
var callFamilies = map[string][]string{
|
||||
CapBusStateCalls: {buscalls.ObjectsPut, buscalls.ObjectsGet, buscalls.ObjectsList, buscalls.ObjectsDelete, buscalls.ObjectsInfo,
|
||||
buscalls.ObjectsWatch, buscalls.StreamRead, buscalls.StreamFill, buscalls.WorkQueueSize, buscalls.WorkQueueDead},
|
||||
CapStreamPublish: {buscalls.StreamPublish},
|
||||
CapLedgerCalls: {buscalls.LedgerApply, buscalls.LedgerGet, buscalls.LedgerHistory, buscalls.LedgerWatch, buscalls.LedgerQuery,
|
||||
buscalls.LedgerRebuild, buscalls.LedgerReindex, buscalls.LedgerSnapshot, buscalls.LedgerLag},
|
||||
CapLedgerUpkeepCalls: {buscalls.LedgerFill, buscalls.LedgerRemove},
|
||||
CapSentObjectsCalls: {buscalls.ObjectsSentGet},
|
||||
CapPutSessions: {buscalls.ObjectsPutChunk, buscalls.ObjectsGetChunk},
|
||||
}
|
||||
|
||||
// CallCapabilities is every capability of the calls this runner answers, sorted. The delayed settle of a pulled seat
|
||||
// message names a fate, not a method, so it stands beside the method families (novox/hq ADR 0664 §5); so does `finding-calls`,
|
||||
// the acknowledged `mesh/publish` and the `mesh/subscribe` for new messages the launch answers (design 47 G §559).
|
||||
func CallCapabilities() []string {
|
||||
out := []string{stdio.CapSeatSettleDelay, stdio.CapFindingCalls, CapApprovalMark}
|
||||
for c := range callFamilies {
|
||||
out = append(out, c)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// BusState answers a bundle's calls on its module's bus state in the new form (mesh-sdk buscalls; novox/hq ADR 0595
|
||||
// §9, design 25 § 12.5; issue 1966): each as the module whose bundle asks, never a module the request names, so a
|
||||
// bundle reaches its own module's bindings alone.
|
||||
func (b *moduleBus) BusState(method string, params json.RawMessage) (json.RawMessage, error) {
|
||||
conn := b.all.conn
|
||||
var answer any
|
||||
var err error
|
||||
switch method {
|
||||
case buscalls.ObjectsPut:
|
||||
var r buscalls.ObjectsPutRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.ObjectsPut(b.module, r)
|
||||
}
|
||||
case buscalls.ObjectsGet:
|
||||
var r buscalls.ObjectsNameRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = b.all.objectsGet(b.module, r)
|
||||
if err == nil && answer == nil {
|
||||
return json.RawMessage("null"), nil
|
||||
}
|
||||
}
|
||||
case buscalls.ObjectsDelete:
|
||||
var r buscalls.ObjectsNameRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
err = conn.ObjectsDelete(b.module, r)
|
||||
answer = buscalls.Done{}
|
||||
}
|
||||
case buscalls.ObjectsSentGet:
|
||||
var r buscalls.ObjectsSentGetRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = b.all.sentGet(b.module, r)
|
||||
}
|
||||
case buscalls.ObjectsPutChunk:
|
||||
answer, err = b.all.putChunk(b.module, params)
|
||||
case buscalls.ObjectsGetChunk:
|
||||
answer, err = b.all.getChunk(b.module, params)
|
||||
case buscalls.ObjectsList:
|
||||
var r buscalls.ObjectsRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.ObjectsList(b.module, r)
|
||||
}
|
||||
case buscalls.ObjectsInfo:
|
||||
var r buscalls.ObjectsRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.ObjectsInfo(b.module, r)
|
||||
}
|
||||
case buscalls.StreamRead:
|
||||
var r buscalls.StreamReadRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.StreamRead(b.module, r)
|
||||
}
|
||||
case buscalls.StreamFill:
|
||||
var r buscalls.StreamRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.StreamFill(b.module, r)
|
||||
}
|
||||
case buscalls.StreamPublish:
|
||||
var r buscalls.StreamPublishRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.StreamPublish(b.module, r)
|
||||
}
|
||||
case buscalls.WorkQueueSize:
|
||||
var r buscalls.WorkQueueRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.WorkQueueSize(b.module, r)
|
||||
}
|
||||
case buscalls.WorkQueueDead:
|
||||
var r buscalls.WorkQueueDeadRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.WorkQueueDead(b.module, r)
|
||||
}
|
||||
case buscalls.LedgerApply, buscalls.LedgerGet, buscalls.LedgerHistory, buscalls.LedgerQuery, buscalls.LedgerRebuild,
|
||||
buscalls.LedgerReindex, buscalls.LedgerSnapshot, buscalls.LedgerLag, buscalls.LedgerFill, buscalls.LedgerRemove:
|
||||
answer, err = b.ledgerCall(method, params)
|
||||
default:
|
||||
return nil, fmt.Errorf("the runtime answers no %s", method)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return json.Marshal(answer)
|
||||
}
|
||||
|
||||
// BusStateWatch hands a bundle each change of its objects or its ledger, the current ones first.
|
||||
func (b *moduleBus) BusStateWatch(method string, params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
|
||||
switch method {
|
||||
case buscalls.ObjectsWatch:
|
||||
var r buscalls.ObjectsWatchRequest
|
||||
if err := decode(method, params, &r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return b.all.conn.ObjectsWatch(b.module, r, func(change buscalls.ObjectsChange) error {
|
||||
raw, err := json.Marshal(change)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return deliver(raw)
|
||||
})
|
||||
case buscalls.LedgerWatch:
|
||||
return b.ledgerWatch(params, deliver)
|
||||
}
|
||||
return nil, fmt.Errorf("the runtime answers no %s", method)
|
||||
}
|
||||
|
||||
// decode reads a bus-state call's request.
|
||||
func decode(method string, params json.RawMessage, into any) error {
|
||||
if err := json.Unmarshal(params, into); err != nil {
|
||||
return fmt.Errorf("%s's request is not of its shape (mesh-sdk buscalls): %w", method, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// objectsGet answers a get: the object whole when it is at most one chunk, else the session that reads it chunk by chunk
|
||||
// (design 47 G §490, §518); nil when there is none.
|
||||
func (c *consumers) objectsGet(module string, r buscalls.ObjectsNameRequest) (any, error) {
|
||||
st := c.objects()
|
||||
if st == nil {
|
||||
return nil, fmt.Errorf("the runner has no bus to read %s of %s from", r.Name, module)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), bus.StateTimeout)
|
||||
defer cancel()
|
||||
info, rd, found, err := st.ObjectsOpen(ctx, module, r)
|
||||
if err != nil || !found {
|
||||
return nil, err
|
||||
}
|
||||
if info.Size > buscalls.ChunkBytes {
|
||||
return c.openGet(module, info, rd, sessionDigest(info.Digest), false)
|
||||
}
|
||||
defer rd.Close()
|
||||
data, err := io.ReadAll(rd)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &buscalls.ObjectData{Info: info, Data: data}, nil
|
||||
}
|
||||
|
||||
// sentGet answers a get of a sent object by its reference: whole when the reference's size is at most one chunk, else a
|
||||
// session (design 47 G §620).
|
||||
func (c *consumers) sentGet(module string, r buscalls.ObjectsSentGetRequest) (any, error) {
|
||||
if r.Size <= buscalls.ChunkBytes {
|
||||
got, err := c.conn.SentGet(r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return got, nil
|
||||
}
|
||||
st := c.objects()
|
||||
if st == nil {
|
||||
return nil, fmt.Errorf("the runner has no bus to read %s from", r.Object)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), bus.StateTimeout)
|
||||
defer cancel()
|
||||
info, rd, err := st.SentOpen(ctx, r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.openGet(module, info, rd, sessionDigest(r.Digest), true)
|
||||
}
|
||||
@@ -2,9 +2,13 @@ package runtime
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatdef"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
@@ -13,6 +17,7 @@ import (
|
||||
|
||||
type saidCaller struct {
|
||||
Caller string `json:"caller"`
|
||||
Call string `json:"call"`
|
||||
Arguments map[string]any `json:"arguments"`
|
||||
}
|
||||
|
||||
@@ -23,7 +28,7 @@ func TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller(t *testing.T)
|
||||
mesh := mt.New(t)
|
||||
mesh.Issue(t, mt.MembershipOf("ledger", "anchor", true, map[string][]string{"node-desk": {"who"}}))
|
||||
nodeTools := connect(t, "node-tools", "anchor")
|
||||
stop, err := Run(nodeTools, []Served{{"ledger", []string{mt.Fixture("caller-echo.py")}}}, nil, func(string, ...any) {})
|
||||
stop, err := Run(nodeTools, []Served{{"ledger", []string{mt.Fixture("caller-echo.py")}}}, nil, func(string, ...any) {}, deskDefined)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -36,6 +41,7 @@ func TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller(t *testing.T)
|
||||
asker.Logf = func(string, ...any) {}
|
||||
defer asker.Close()
|
||||
|
||||
calls := map[string]bool{}
|
||||
for _, key := range []string{"ledger.who", "ledger.who@anchor", "seat:node-desk.who@anchor"} {
|
||||
var got saidCaller
|
||||
mt.Until(t, func() error {
|
||||
@@ -49,19 +55,31 @@ func TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller(t *testing.T)
|
||||
if got.Caller != "shanks.node-tools" {
|
||||
t.Errorf("%s reached the module with caller %q, want the asker's bus user", key, got.Caller)
|
||||
}
|
||||
// And the id of the call in flight, one per call, which a ledger's apply names (novox/hq ADR 0597 §7).
|
||||
if got.Call == "" || calls[got.Call] {
|
||||
t.Errorf("%s reached the module with call id %q, want one of its own", key, got.Call)
|
||||
}
|
||||
calls[got.Call] = true
|
||||
if got.Arguments["caller"] != "novox.node-tools" {
|
||||
t.Errorf("%s: the arguments did not reach the module as sent: %v", key, got.Arguments)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The subject that names no caller is still served, for one release (retired by hq issue 464), and says so:
|
||||
// the module is named no caller, whatever the body claims.
|
||||
// deskDefined is the seat list's definition of the fixture seat node-desk, its verb who taking its class's default.
|
||||
var deskDefined = WithSeatDefinitions(func() map[string]seatdef.Definition {
|
||||
return map[string]seatdef.Definition{"node-desk": {Seat: "node-desk", Verbs: []seatdef.Verb{{Name: "who"}}}}
|
||||
})
|
||||
|
||||
// The subject that names no caller is still served for a module's tool, for one release (retired by hq issue 464), and says
|
||||
// so: the module is named no caller, whatever the body claims. A seat's verb refuses it before the bundle runs (design 47 G
|
||||
// §610; lead answers A-165, A-179), and refuses every caller while the seat's definition is unread, names no such verb or
|
||||
// opens it to none.
|
||||
func TestTheSubjectThatNamesNoCallerStillWorksAndNamesNone(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
mesh.Issue(t, mt.MembershipOf("ledger", "anchor", true, map[string][]string{"node-desk": {"who"}}))
|
||||
nodeTools := connect(t, "node-tools", "anchor")
|
||||
stop, err := Run(nodeTools, []Served{{"ledger", []string{mt.Fixture("caller-echo.py")}}}, nil, func(string, ...any) {})
|
||||
stop, err := Run(nodeTools, []Served{{"ledger", []string{mt.Fixture("caller-echo.py")}}}, nil, func(string, ...any) {}, deskDefined)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -84,8 +102,96 @@ func TestTheSubjectThatNamesNoCallerStillWorksAndNamesNone(t *testing.T) {
|
||||
}
|
||||
return err
|
||||
})
|
||||
if strings.HasPrefix(subject, "mesh.seat.") {
|
||||
if !strings.Contains(reply.Error, "node-desk.who takes no unproven caller") || reply.Result.Call != "" {
|
||||
t.Errorf("%s: an unproven caller of a seat's verb was answered %+v (error %q), want refused before the bundle", subject, reply.Result, reply.Error)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if reply.Error != "" || reply.Result.Caller != "" {
|
||||
t.Errorf("%s: the module was named caller %q (error %q), want none", subject, reply.Result.Caller, reply.Error)
|
||||
}
|
||||
}
|
||||
for _, c := range []struct {
|
||||
defs map[string]seatdef.Definition
|
||||
says string
|
||||
}{
|
||||
{nil, "is not yet read"},
|
||||
{map[string]seatdef.Definition{"node-desk": {Seat: "node-desk"}}, "names no verb who"},
|
||||
{map[string]seatdef.Definition{"node-desk": {Seat: "node-desk", Verbs: []seatdef.Verb{{Name: "who", Callers: &seatdef.Callers{List: []string{}}}}}}, "open to no caller"},
|
||||
{map[string]seatdef.Definition{"node-desk": {Seat: "node-desk", Verbs: []seatdef.Verb{{Name: "who", Class: seatdef.ClassAct}}}}, ""},
|
||||
} {
|
||||
defs := c.defs
|
||||
err := (&consumers{definitions: func() map[string]seatdef.Definition { return defs }}).admitted("node-desk", "who", "workstation.node-tools")
|
||||
if c.says == "" && err != nil || c.says != "" && (err == nil || !strings.Contains(err.Error(), c.says)) {
|
||||
t.Errorf("a proven caller with the definitions %v: %v, want %q", c.defs, err, c.says)
|
||||
}
|
||||
}
|
||||
// Design 47 G A-249: a bundle's seat call is received by the runner in chunks only for a verb whose definition says `chunks`,
|
||||
// of any machine: a verb a node holds is asked of the node that ran the ask (A-254).
|
||||
t.Run("OnlyAChunkedSeatVerbIsAskedInChunks", func(t *testing.T) {
|
||||
c := &consumers{definitions: func() map[string]seatdef.Definition {
|
||||
return map[string]seatdef.Definition{"mesh-catalog": {Verbs: []seatdef.Verb{{Name: "input", Chunks: true}, {Name: "entry"}}}}
|
||||
}}
|
||||
for key, want := range map[string]bool{
|
||||
"seat:mesh-catalog.input": true,
|
||||
"seat:mesh-catalog.entry": false,
|
||||
"seat:mesh-catalog.input@home-server": true,
|
||||
"seat:mesh-catalog.entry@home-server": false,
|
||||
"seat:mesh-unread.input": false,
|
||||
"mesh-delivery-indexer.indexer": false,
|
||||
} {
|
||||
if got := c.chunked(key); got != want {
|
||||
t.Errorf("chunked(%s) = %v, want %v", key, got, want)
|
||||
}
|
||||
}
|
||||
if (&consumers{}).chunked("seat:mesh-catalog.input") {
|
||||
t.Error("a runner that holds no seat definitions asks no chunks")
|
||||
}
|
||||
})
|
||||
// **The tool runner answers a held seat's marked verb by caller and serves the seat's `approvals`** (design 47 G §677; lead
|
||||
// answers A-265, A-271), with zsh's node-login-shell `execute` as the case: the marked verb reaches the gate with the caller
|
||||
// its subject names, and `approvals` is answered from the runner's own record of the calls its gate holds, with the
|
||||
// arguments for the call's own caller only, though no bundle names the verb.
|
||||
t.Run("TheRunnerServesApprovalsForTheSeatsItsBundlesHold", func(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
mesh.Issue(t, mt.MembershipOf("zsh", "anchor", true, map[string][]string{"node-login-shell": {"execute", seatdef.ApprovalsVerb}}))
|
||||
nodeTools := connect(t, "node-tools", "anchor")
|
||||
gate := &gateHolding{}
|
||||
defined := WithSeatDefinitions(func() map[string]seatdef.Definition {
|
||||
return map[string]seatdef.Definition{"node-login-shell": {Seat: "node-login-shell", Verbs: []seatdef.Verb{
|
||||
{Name: "execute", NeedsApproval: true}, {Name: seatdef.ApprovalsVerb}}}}
|
||||
})
|
||||
stop, err := Run(nodeTools, []Served{{"zsh", []string{mt.Fixture("login-shell-execute.py")}}}, nil, func(string, ...any) {}, defined, WithGate(gate))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer stop()
|
||||
asker, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "workstation", User: "workstation.node-tools", Password: "unused"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
asker.Logf = func(string, ...any) {}
|
||||
defer asker.Close()
|
||||
|
||||
mt.Until(t, func() error {
|
||||
_, err := asker.Ask("seat:node-login-shell.execute@anchor", map[string]any{"command": "ls"}, "")
|
||||
return err
|
||||
})
|
||||
if len(gate.calls) != 1 || gate.calls[0].Seat != "node-login-shell" || gate.calls[0].Name != "execute" || gate.calls[0].Caller != "workstation.node-tools" {
|
||||
t.Fatalf("the marked verb reached the gate as %+v, want the seat's execute by workstation.node-tools", gate.calls)
|
||||
}
|
||||
var got approval.Approvals
|
||||
mt.Until(t, func() error {
|
||||
answered, err := asker.Ask("seat:node-login-shell.approvals@anchor", map[string]any{}, "")
|
||||
if err == nil {
|
||||
err = json.Unmarshal(answered.Result, &got)
|
||||
}
|
||||
return err
|
||||
})
|
||||
if len(got.Calls) != 1 || got.Calls[0].Caller != "workstation.node-tools" || got.Calls[0].Verb != "node-login-shell.execute" ||
|
||||
string(got.Calls[0].Arguments) != `{"command":"ls"}` {
|
||||
t.Fatalf("approvals answered %+v, want the one call the gate holds, with its caller's arguments", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -157,3 +159,37 @@ func TestALongRunningBundleThatExitsIsStartedAgain(t *testing.T) {
|
||||
|
||||
func fmtSprintf(format string, args ...any) string { return fmt.Sprintf(format, args...) }
|
||||
func errorf(format string, args ...any) error { return fmt.Errorf(format, args...) }
|
||||
|
||||
// A bundle loaded again sets its event and heartbeat handlers while the consumer's loop hands a word or an event on: no data race (-race),
|
||||
// and the heartbeat is asked of the bus only while some bundle holds a handler.
|
||||
func TestAHeartbeatHandlerSetAgainWhileWordsAreHandedOnIsNotARace(t *testing.T) {
|
||||
m := &moduleEvents{}
|
||||
all := &consumers{logf: t.Logf, of: map[string]*moduleEvents{"mod": m}}
|
||||
slot := &beatSlot{}
|
||||
m.beats = []*beatSlot{slot}
|
||||
if m.beatAsked() {
|
||||
t.Fatal("a heartbeat is asked while no bundle gave a handler")
|
||||
}
|
||||
took := &deliverSlot{}
|
||||
took.set(func(json.RawMessage) error { return nil })
|
||||
m.delivers = []*deliverSlot{took}
|
||||
var got atomic.Int32
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
for i := 0; i < 200; i++ {
|
||||
slot.set(func(json.RawMessage) { got.Add(1) })
|
||||
took.set(func(json.RawMessage) error { got.Add(1); return nil })
|
||||
}
|
||||
}()
|
||||
for i := 0; i < 200; i++ {
|
||||
all.handBeat(m, bus.Beat{At: time.Now()})
|
||||
if err := all.handOut("mod", m, bus.Envelope{Key: "a.b", Body: json.RawMessage(`{}`)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
<-done
|
||||
if !m.beatAsked() {
|
||||
t.Error("a bundle gave a handler and no heartbeat is asked")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
"git.novox.be/novox/mesh-sdk/go/seatdef"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/launch"
|
||||
)
|
||||
|
||||
// FingerprintWord is the word of a module's composed environment that names the build its bundle serves (design 47 G §424).
|
||||
const FingerprintWord = "MESH_FINGERPRINT"
|
||||
|
||||
// Gate is the approval gate the runner serves through (design 46 7-approval-mark §4, §20; node-tools/internal/approval).
|
||||
type Gate interface {
|
||||
// Unserved is, of a module's marked tools, those its bundle does not register, saying it as the module's finding.
|
||||
Unserved(module, fingerprint string, registered []string) []string
|
||||
// Run is a call through the gate: run is called only when the call may run now, once.
|
||||
Run(call approval.Call, run func() (any, error)) (any, error)
|
||||
// Approvals is the seat verb `approvals` for a caller: the calls the gate holds, its acts and claims (A-265, A-271).
|
||||
Approvals(caller string) (approval.Approvals, error)
|
||||
}
|
||||
|
||||
// gatedRun is how a handler calls a tool: through the gate where there is one.
|
||||
type gatedRun func(call approval.Call, t launch.Tool, caller string) (any, error)
|
||||
|
||||
// gated calls the tool through the gate, or straight where the runner has none.
|
||||
func (c *consumers) gated(call approval.Call, t launch.Tool, caller string) (any, error) {
|
||||
run := func() (any, error) { return t.Run(caller, call.Args) }
|
||||
if c.gate == nil {
|
||||
return run()
|
||||
}
|
||||
return c.gate.Run(call, run)
|
||||
}
|
||||
|
||||
// approvalsOf is how the runner answers a held seat's verb `approvals`: from its gate's record, nil without a gate.
|
||||
func (r *runner) approvalsOf() func(caller string) (any, error) {
|
||||
if r.events.gate == nil {
|
||||
return nil
|
||||
}
|
||||
return func(caller string) (any, error) { return r.events.gate.Approvals(caller) }
|
||||
}
|
||||
|
||||
// WithGate serves every call through the approval gate.
|
||||
func WithGate(g Gate) Option { return func(c *consumers) { c.gate = g } }
|
||||
|
||||
var _ Gate = (*approval.Gate)(nil)
|
||||
|
||||
// WithSeatDefinitions refuses, where a seat's verb is answered, a caller outside the verb's effective callers, before the
|
||||
// bundle runs (design 47 G §610; lead answers A-165, A-179): defs are the seat definitions the runner's seat list holds.
|
||||
// Without it a verb is answered for every caller, as before.
|
||||
func WithSeatDefinitions(defs func() map[string]seatdef.Definition) Option {
|
||||
return func(c *consumers) { c.definitions = defs }
|
||||
}
|
||||
|
||||
// admitted says why a seat's verb is refused to a caller, nil when it is answered: an unproven caller is refused, and so is
|
||||
// every caller while the seat's definition is not read, names no such verb, or the verb's effective callers are empty; a
|
||||
// proven caller is among them, since the grants part composes a verb's call subject only for its effective callers (A-179).
|
||||
func (c *consumers) admitted(seat, verb, caller string) error {
|
||||
if c.definitions == nil {
|
||||
return nil
|
||||
}
|
||||
if caller == "" {
|
||||
return fmt.Errorf("Not run: %s.%s takes no unproven caller: call it on its subject that names its caller, "+
|
||||
"mesh.seat.%s.call.%s.<caller>", seat, verb, seat, verb)
|
||||
}
|
||||
def, ok := c.definitions()[seat]
|
||||
if !ok {
|
||||
return fmt.Errorf("Not run: the definition of the seat %s is not yet read by this tool runner's seat list, so no caller "+
|
||||
"of %s.%s is admitted", seat, seat, verb)
|
||||
}
|
||||
for _, v := range def.Verbs {
|
||||
if v.Name == verb {
|
||||
if len(v.EffectiveCallers()) == 0 {
|
||||
return fmt.Errorf("Not run: %s.%s is open to no caller", seat, verb)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("Not run: the seat %s's definition names no verb %s, so no caller is admitted", seat, verb)
|
||||
}
|
||||
|
||||
// unservedMarks is, of the module's marked tools, those its bundle's registrations under its own name do not register.
|
||||
func (r *runner) unservedMarks(module string, regs []launch.Registration) []string {
|
||||
if r.events.gate == nil {
|
||||
return nil
|
||||
}
|
||||
var registered []string
|
||||
for _, reg := range regs {
|
||||
if reg.Module == module {
|
||||
for _, t := range reg.Tools {
|
||||
registered = append(registered, t.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
return r.events.gate.Unserved(module, r.fingerprintOf(module), registered)
|
||||
}
|
||||
|
||||
// chunked says whether key names a seat's verb whose definition says `chunks` (design 47 G A-249): its answer is received in
|
||||
// chunks by the runner, whose call hands the bundle the whole.
|
||||
func (c *consumers) chunked(key string) bool {
|
||||
if c.definitions == nil || !strings.HasPrefix(key, "seat:") {
|
||||
return false
|
||||
}
|
||||
name, _, _ := strings.Cut(key, "@") // a verb asked of the machine that ran the ask is chunked as the verb is (A-254)
|
||||
seat, verb, ok := strings.Cut(strings.TrimPrefix(name, "seat:"), ".")
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
for _, v := range c.definitions()[seat].Verbs {
|
||||
if v.Name == verb {
|
||||
return v.Chunks
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"git.novox.be/novox/mesh-sdk/go/approval"
|
||||
)
|
||||
|
||||
// gateHolding is the runner's gate as far as the runtime reaches it: it records the calls it is run for and answers
|
||||
// `approvals` from them.
|
||||
type gateHolding struct{ calls []approval.Call }
|
||||
|
||||
func (g *gateHolding) Unserved(string, string, []string) []string { return nil }
|
||||
func (g *gateHolding) Run(c approval.Call, run func() (any, error)) (any, error) {
|
||||
g.calls = append(g.calls, c)
|
||||
return run()
|
||||
}
|
||||
func (g *gateHolding) Approvals(caller string) (approval.Approvals, error) {
|
||||
out := approval.Approvals{Calls: []approval.ListedCall{}, Acts: []approval.ListedAct{}, Claims: []string{}}
|
||||
for i, c := range g.calls {
|
||||
l := approval.ListedCall{Call: string(rune('a' + i)), Caller: c.Caller, Verb: c.Seat + "." + c.Name, State: "waiting"}
|
||||
if caller == c.Caller {
|
||||
l.Arguments = c.Args
|
||||
}
|
||||
out.Calls = append(out.Calls, l)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"sync"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
|
||||
// The ledger credentials (novox/hq ADR 0597 §9, design 25 § 12.4; issue 1966 T6): one per module assignment, held by
|
||||
// the machine's tool runner in its own secret `ledgers`, a map of module to credential (`{}` when none), which the vault
|
||||
// rewrites on each mint and retirement. **A module's ledger history is appended, and its snapshots written, only on
|
||||
// that module's own ledger connection**; everything else stays on the runner's own.
|
||||
|
||||
// LedgersFile is where the runner reads its own secret `ledgers`: MESH_LEDGERS_FILE when set, else beside its broker
|
||||
// credential, both in its directory mesh-state (`${dir:mesh-state}/ledgers`). Its manifest declares the secret only once
|
||||
// the vault holds it on every machine (B3 of issue 1966): until then a file not there means no ledger credentials, and
|
||||
// every ledger write is refused naming the missing one.
|
||||
const LedgersFile = "MESH_LEDGERS_FILE"
|
||||
|
||||
// LedgersFileOf is the path of the secret `ledgers` for a runtime whose broker credential is at broker.
|
||||
func LedgersFileOf(broker string) string {
|
||||
if file := os.Getenv(LedgersFile); file != "" {
|
||||
return file
|
||||
}
|
||||
if broker == "" {
|
||||
return ""
|
||||
}
|
||||
return filepath.Join(filepath.Dir(broker), "ledgers")
|
||||
}
|
||||
|
||||
// LedgerCredentials are the ledger connections of the modules on this machine, read from the secret `ledgers` and
|
||||
// followed as it changes, with no restart.
|
||||
type LedgerCredentials struct {
|
||||
file string
|
||||
dial func(bus.Credential) (*bus.Conn, error)
|
||||
logf func(string, ...any)
|
||||
|
||||
mu sync.Mutex
|
||||
creds map[string]bus.Credential
|
||||
conns map[string]*bus.Conn
|
||||
why string // why the file could not be read, when it could not
|
||||
stop func()
|
||||
}
|
||||
|
||||
// FollowLedgerCredentials reads the secret `ledgers` at file and follows it (WatchFile), dialling a module's connection
|
||||
// with dial the first time it is asked for.
|
||||
func FollowLedgerCredentials(file string, dial func(bus.Credential) (*bus.Conn, error), logf func(string, ...any)) *LedgerCredentials {
|
||||
l := &LedgerCredentials{file: file, dial: dial, logf: logf, creds: map[string]bus.Credential{}, conns: map[string]*bus.Conn{}}
|
||||
l.read()
|
||||
stop, err := WatchFile(file, l.read, logf, "a change of the ledger credentials is taken only after a restart")
|
||||
if err != nil {
|
||||
logf("[mesh-tools] cannot follow the ledger credentials at %s: %v; a change is taken only after a restart", file, err)
|
||||
} else {
|
||||
l.stop = stop
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
// read takes the file as it is now: a module whose credential changed or went has its connection closed, and is
|
||||
// dialled again from the new one when next asked for.
|
||||
func (l *LedgerCredentials) read() {
|
||||
creds, why := map[string]bus.Credential{}, ""
|
||||
raw, err := os.ReadFile(l.file)
|
||||
switch {
|
||||
case errors.Is(err, os.ErrNotExist):
|
||||
why = "the file " + l.file + " is not there yet"
|
||||
case err != nil:
|
||||
why = err.Error()
|
||||
case len(bytes.TrimSpace(raw)) == 0:
|
||||
why = "the file " + l.file + " is empty"
|
||||
default:
|
||||
if err := json.Unmarshal(raw, &creds); err != nil {
|
||||
why = "the file " + l.file + " is not a map of module to credential: " + err.Error()
|
||||
creds = map[string]bus.Credential{}
|
||||
}
|
||||
}
|
||||
l.mu.Lock()
|
||||
var closing []*bus.Conn
|
||||
for module, conn := range l.conns {
|
||||
if now, has := creds[module]; !has || !reflect.DeepEqual(now, l.creds[module]) {
|
||||
closing = append(closing, conn)
|
||||
delete(l.conns, module)
|
||||
}
|
||||
}
|
||||
saidBefore := l.why
|
||||
l.creds, l.why = creds, why
|
||||
l.mu.Unlock()
|
||||
for _, c := range closing {
|
||||
c.Close()
|
||||
}
|
||||
if why != "" && why != saidBefore {
|
||||
l.logf("[mesh-tools] the ledger credentials cannot be read (%s); no ledger is written until they can", why)
|
||||
}
|
||||
}
|
||||
|
||||
// Writer is a module's ledger connection, dialled from its credential the first time it is asked for; refused, naming
|
||||
// what is missing, when the secret holds none for it.
|
||||
func (l *LedgerCredentials) Writer(module string) (*bus.Conn, error) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if c := l.conns[module]; c != nil {
|
||||
return c, nil
|
||||
}
|
||||
cred, has := l.creds[module]
|
||||
if !has {
|
||||
why := ""
|
||||
if l.why != "" {
|
||||
why = " (" + l.why + ")"
|
||||
}
|
||||
return nil, fmt.Errorf("%s's ledger credential is missing from the tool runner's own secret ledgers%s: its history "+
|
||||
"is published only on its own ledger connection, which the grants part issues with its membership and the "+
|
||||
"vault writes there (novox/hq ADR 0597 §9)", module, why)
|
||||
}
|
||||
if cred.Module == "" {
|
||||
cred.Module = module
|
||||
}
|
||||
c, err := l.dial(cred)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s's ledger connection could not be opened: %w", module, err)
|
||||
}
|
||||
c.Logf = l.logf
|
||||
l.conns[module] = c
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// Stop ends the following and closes every ledger connection.
|
||||
func (l *LedgerCredentials) Stop() {
|
||||
if l.stop != nil {
|
||||
l.stop()
|
||||
}
|
||||
l.mu.Lock()
|
||||
conns := l.conns
|
||||
l.conns = map[string]*bus.Conn{}
|
||||
l.mu.Unlock()
|
||||
for _, c := range conns {
|
||||
c.Close()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/buscompose"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// 1966/AC22: given the secret ledgers holding m's credential, when m applies, then the append goes on m's connection
|
||||
// (the runner's own account may not publish on m's ledger, as the grants part grants it); given it missing, apply is
|
||||
// refused naming it; a change of the file is taken without a restart.
|
||||
func TestALedgerIsWrittenOnlyOnItsModulesLedgerConnection(t *testing.T) {
|
||||
ledgerSubjects := []string{buscompose.LedgerHistorySubject("m", "records") + ".>", buscompose.LedgerSnapshotSubject("m", "records") + ".>"}
|
||||
users := []*server.User{
|
||||
{Username: "node-tools", Password: "pw", Permissions: &server.Permissions{
|
||||
Publish: &server.SubjectPermission{Allow: []string{">"}, Deny: ledgerSubjects}}},
|
||||
// As the grants part composes a ledger account (mesh-grants buscompose ledgerPermissions): publish on its
|
||||
// module's ledger histories and snapshots, subscribe its own inbox, nothing else.
|
||||
{Username: "ledger.anchor.m", Password: "pw", Permissions: &server.Permissions{
|
||||
Publish: &server.SubjectPermission{Allow: ledgerSubjects},
|
||||
Subscribe: &server.SubjectPermission{Allow: []string{"_INBOX.ledger.anchor.m.>"}}}},
|
||||
{Username: "admin", Password: "pw"},
|
||||
}
|
||||
s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: t.TempDir(),
|
||||
NoLog: true, NoSigs: true, Users: users})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
go s.Start()
|
||||
if !s.ReadyForConnections(30 * time.Second) {
|
||||
t.Fatal("the bus did not come up")
|
||||
}
|
||||
t.Cleanup(s.Shutdown)
|
||||
url := s.ClientURL()
|
||||
|
||||
admin, err := nats.Connect(url, nats.UserInfo("admin", "pw"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(admin.Close)
|
||||
js, _ := jetstream.New(admin)
|
||||
ctx := context.Background()
|
||||
for _, cfg := range []jetstream.StreamConfig{
|
||||
{Name: "LEDGER_m_records", Subjects: []string{"mesh.ledger.m.records.history.>"}},
|
||||
{Name: "SNAPSHOT_m_records", Subjects: []string{"mesh.ledger.m.records.snapshot.>"}, MaxMsgsPerSubject: 1},
|
||||
} {
|
||||
if _, err := js.CreateStream(ctx, cfg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, b := range []string{"m_records", membership.GrantsBucket} {
|
||||
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
mem := mt.MembershipOf("m", "anchor", false, nil)
|
||||
mem.Bindings = []membership.BusBinding{{Local: "records", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger,
|
||||
History: "LEDGER_m_records", Subject: "mesh.ledger.m.records.history", State: "m_records", Snapshot: "mesh.ledger.m.records.snapshot",
|
||||
Types: []catalogue.LedgerType{{Name: "decision", Operations: []catalogue.LedgerOperation{
|
||||
{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}}}}}}}
|
||||
raw, _ := json.Marshal(mem)
|
||||
var sdk membership.Membership
|
||||
_ = json.Unmarshal(raw, &sdk)
|
||||
grants, _ := js.KeyValue(ctx, membership.GrantsBucket)
|
||||
if _, err := grants.Put(ctx, membership.GrantsKey("anchor", "m"), membership.EncodeGrantsMembership(sdk)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
conn, err := bus.Connect(bus.Credential{URL: url, Module: "node-tools", Node: "anchor", User: "node-tools", Password: "pw"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn.Logf = func(string, ...any) {}
|
||||
t.Cleanup(conn.Close)
|
||||
conn.Follow("m")
|
||||
mt.Until(t, func() error {
|
||||
if conn.Membership("m") == nil {
|
||||
return os.ErrNotExist
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
// Not there yet, as on a machine whose vault has not made it: no ledger credentials, and the runner serves on.
|
||||
file := filepath.Join(t.TempDir(), "ledgers")
|
||||
creds := FollowLedgerCredentials(file, bus.Dial, t.Logf)
|
||||
t.Cleanup(creds.Stop)
|
||||
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
|
||||
WithLedgerWriters(creds.Writer)(all)
|
||||
m := all.forModule("m").(*moduleBus)
|
||||
apply := func(instance string) error {
|
||||
_, err := m.BusState(buscalls.LedgerApply, json.RawMessage(`{"ledger":"records","instance":"`+instance+`","operation":"title","body":{"title":"T"}}`))
|
||||
return err
|
||||
}
|
||||
|
||||
// Missing: refused, naming it and why.
|
||||
if err := apply("decision/1"); err == nil || !strings.Contains(err.Error(), "m's ledger credential is missing from the tool runner's own secret ledgers (the file "+file+" is not there yet)") {
|
||||
t.Fatalf("an apply with no credential: %v", err)
|
||||
}
|
||||
// Written: taken without a restart, on m's connection (the runner's own account is denied m's ledger subjects, so an
|
||||
// append that lands went on m's), whose account stamps the operation.
|
||||
cred, _ := json.Marshal(map[string]bus.Credential{"m": {URL: url, Node: "anchor", Module: "m", User: "ledger.anchor.m", Password: "pw"}})
|
||||
if err := os.WriteFile(file, cred, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mt.Until(t, func() error { return apply("decision/1") })
|
||||
page, err := m.BusState(buscalls.LedgerHistory, json.RawMessage(`{"ledger":"records","instance":"decision/1"}`))
|
||||
if err != nil || !strings.Contains(string(page), `"author":{"account":"ledger.anchor.m"}`) {
|
||||
t.Fatalf("the history holds %s, %v", page, err)
|
||||
}
|
||||
// Gone again: refused again, the change taken without a restart.
|
||||
if err := os.WriteFile(file, []byte(`{}`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mt.Until(t, func() error {
|
||||
if err := apply("decision/2"); err == nil || !strings.Contains(err.Error(), "credential is missing") {
|
||||
return os.ErrExist
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user