Compare commits
72
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
84ede95c4a | ||
|
|
7fd9d893c0 | ||
|
|
0fb4591c9d | ||
|
|
7da4c55752 | ||
|
|
b4819dc1fb | ||
|
|
40e0e8282b | ||
|
|
01ee2ef394 | ||
|
|
8b8a760047 | ||
|
|
db3dd04be7 | ||
|
|
d13890b944 | ||
|
|
b3903446f9 | ||
|
|
e550c87a29 | ||
|
|
ff7661a136 | ||
|
|
84533edf2e | ||
|
|
ea516ece5a | ||
|
|
8c1e04dd37 | ||
|
|
43d3fba501 | ||
|
|
df929b48b5 | ||
|
|
961ec0ef96 | ||
|
|
5a9e997349 | ||
|
|
fadc73f73b | ||
|
|
d206b0dde1 | ||
|
|
2bb0a6461d | ||
|
|
38a634723b | ||
|
|
8fe7ef3b82 | ||
|
|
63c1e3faa2 | ||
|
|
ad22c5e9fe | ||
|
|
d6c516ab44 | ||
|
|
b8e42cf54c | ||
|
|
24c0ec0d32 | ||
|
|
bd24ceafa9 | ||
|
|
3f538027e9 | ||
|
|
8114b45884 | ||
|
|
d7d908cf67 | ||
|
|
e475d9f849 | ||
|
|
e6f27a5f74 | ||
|
|
c678511871 | ||
|
|
4f636b7080 | ||
|
|
49f9269376 | ||
|
|
29fd452628 | ||
|
|
331fa06b54 | ||
|
|
5b2ff5e321 | ||
|
|
4fbfdfad13 | ||
|
|
75943ca7b0 | ||
|
|
fb6631aee1 | ||
|
|
d41bee158c | ||
|
|
110e55cb48 | ||
|
|
c494018074 | ||
|
|
4cdf63443d | ||
|
|
1e9995faba | ||
|
|
f6c401ecd2 | ||
|
|
c2f01bf10b | ||
|
|
be2fc2e9b6 | ||
|
|
aa05629375 | ||
|
|
35a7ff70fe | ||
|
|
747bfa71aa | ||
|
|
316a5a2fc3 | ||
|
|
0edcd2501d | ||
|
|
ac645e2cae | ||
|
|
a479a90fb2 | ||
|
|
3c9068f99a | ||
|
|
250aa73a56 | ||
|
|
e32eef2379 | ||
|
|
070ae53128 | ||
|
|
8c0618cfc5 | ||
|
|
54680238a0 | ||
|
|
200249a84f | ||
|
|
67850457ef | ||
|
|
9d67b0995f | ||
|
|
e9009b7f04 | ||
|
|
9fc97e45c9 | ||
|
|
1583fa85d9 |
@@ -23,8 +23,7 @@ marked executable — is **launched** rather than imported (novox/hq ADR 0188):
|
||||
as a child with its own environment and speaks MCP over stdio to it, `tools/list` once and
|
||||
`tools/call` per call. A tool it lists as `<seat>.<verb>` is the seat's implementation. A child that
|
||||
exits is named in the log and started again on its next call. So a tools bundle may be written in
|
||||
any language; the mesh's SDK for each is the stdio loop and nothing more (`node-tools/src/launch.ts`
|
||||
is the runtime's side of it).
|
||||
any language; the mesh's SDK for each is the stdio loop and nothing more (`node-tools/internal/launch` is the runtime's side of it).
|
||||
|
||||
Everything hard — dispatch, collection, duplicate-name safety — is the sdk's. This is the thin
|
||||
wrapper that binds the bus and loads the modules. Keeping the bus client here, out of the sdk, is
|
||||
@@ -45,7 +44,7 @@ MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME where thei
|
||||
MESH_BROKER_URL a plain URL instead of the credential, for the bootstrap case
|
||||
```
|
||||
|
||||
`node dist/main.js`. On a node the controller composes the variables and the host supervises the
|
||||
`node dist/main.js`. On a node the mesh composes the variables and the node-engine supervises the
|
||||
process like any other host-side workload (novox/hq to-be 38). As `node-tools` the same process is
|
||||
the console: MCP on `127.0.0.1:4270` (or `MESH_CONSOLE_LISTEN`).
|
||||
|
||||
@@ -66,14 +65,14 @@ answer took, its size in bytes, how many modules and tools it announced, whether
|
||||
fit the bus, when it was last heard — and who was expected and not heard).
|
||||
|
||||
`mesh_search` also finds a tool by the shell command it replaces (novox/hq ADR 0245): each seat verb and
|
||||
module tool says what it replaces in the controller's records (`replaces`, read from the controller's
|
||||
module tool says what it replaces in the indexer's records (`replaces`, read from its
|
||||
`tools` and `modules` answers), so `journalctl -u x`, `systemctl status x` or `docker ps` puts the verb
|
||||
for it first, and says the command it matched as `instead_of`. A word also matches through a short table
|
||||
of the words agents use for the mesh's (`logs` finds the journal). Best first: a replaced command, then
|
||||
words in a name or a replaced command, then words in a description; seats before modules on a tie.
|
||||
|
||||
Discovery asks the bus (ADR 0197): every runtime answers the NATS services protocol's `$SRV.PING` and
|
||||
`$SRV.INFO` with what it serves at that moment, and the console reads where the controller's records
|
||||
`$SRV.INFO` with what it serves at that moment, and the console reads where the placement seat's records
|
||||
place each module. The console waits at least 750 ms, and up to 5 s for every runtime that answered
|
||||
PING to send what it serves — a large runtime's answer crossing the bus to a broker on another machine
|
||||
can arrive well after the rest. A runtime that said it is there and did not say what it serves, or
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
The node's tool runtime as a module (novox/hq ADR 0175, to-be 38 WP3). Assigned to a machine, it is
|
||||
one process the host runs from this bundle, as the operator's account: it serves every assigned
|
||||
module's tools and every held seat's verbs on the bus, and answers MCP on the machine's loopback —
|
||||
the console (design 34). The controller composes the process (which bundles to load, where the
|
||||
the console (design 34). The mesh composes the process (which bundles to load, where the
|
||||
credential is, whose machine it is); this manifest says only what the machine must have for it: the
|
||||
interpreter, a place for the credential, the loopback port, and leave to call every tool.
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
// wherever MESH_CONSOLE_LISTEN says — it is also the console, MCP over HTTP on loopback.
|
||||
//
|
||||
// MESH_BROKER_FILE the credential the mesh sealed to this machine for the runtime
|
||||
// MESH_LEDGERS_FILE its own secret ledgers, each module's ledger credential; beside the broker file when unset
|
||||
// MESH_TOOL_MODULES <module>=<entrypoint>,… the bundles to launch
|
||||
// MESH_TOOL_ENV {"<module>": {"<word>": "<value>"}} what each is given (ADR 0192)
|
||||
// MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME their home
|
||||
@@ -56,7 +57,15 @@ func main() {
|
||||
if err != nil {
|
||||
log.Fatalf("node-tools: %v", err)
|
||||
}
|
||||
stop, err := runtime.Run(conn, served, envs, log.Printf)
|
||||
// Its own secret ledgers: each module's ledger connection, the only one its history is published on (novox/hq ADR
|
||||
// 0597 §9). Followed, so a credential minted or retired is taken with no restart.
|
||||
var opts []runtime.Option
|
||||
if file := runtime.LedgersFileOf(os.Getenv("MESH_BROKER_FILE")); file != "" {
|
||||
ledgers := runtime.FollowLedgerCredentials(file, bus.Dial, log.Printf)
|
||||
defer ledgers.Stop()
|
||||
opts = append(opts, runtime.WithLedgerWriters(ledgers.Writer))
|
||||
}
|
||||
stop, err := runtime.Run(conn, served, envs, log.Printf, opts...)
|
||||
if err != nil {
|
||||
log.Fatalf("node-tools: %v", err)
|
||||
}
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ require (
|
||||
)
|
||||
|
||||
require (
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103221822-85ba0a9a70d8
|
||||
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
|
||||
github.com/google/go-tpm v0.9.8 // indirect
|
||||
github.com/klauspost/compress v1.20.0 // indirect
|
||||
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0=
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103221822-85ba0a9a70d8 h1:XEbIVImp6yGjA91rp0iZBoDdL5KfamjOuDLwWvqh0hc=
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103221822-85ba0a9a70d8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
|
||||
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
|
||||
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
|
||||
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
// **A machine heard and its runtime gone is a machine nobody can ask anything.** The node-engine says
|
||||
// every minute that the machine is there; nothing said whether the runtime serving every module's
|
||||
// tools and every held seat's verbs was. Now it says so too, on its own subject, every interval, with
|
||||
// the interval — the controller's watchdog raises `tools-silent` after three missed — and its build.
|
||||
// the interval — the mesh's watchdog raises `tools-silent` after three missed — and its build.
|
||||
// Core NATS, kept by nobody, as the host's heartbeat: a lost one is the next one.
|
||||
package alive
|
||||
|
||||
|
||||
@@ -0,0 +1,398 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// A module's bus state in the new form (novox/hq ADR 0595, design 25 § 12.5; issue 1966 T1): an object store, a stream,
|
||||
// a work queue, a ledger, each asked for in `requires` by a local name and bound in the module's membership by the bus
|
||||
// names the bus module created. **A bundle reaches only its own module's bindings**: one account per machine carries
|
||||
// every module on it, so what a module may reach is decided here, from its membership, and refused with the reason
|
||||
// before anything is sent, as its state is (state.go).
|
||||
|
||||
// Binding is the module's binding of a local name, of the storage type storage, or the reason there is none.
|
||||
func (c *Conn) Binding(module, local, storage string) (membership.BusBinding, error) {
|
||||
m := c.Membership(module)
|
||||
if m == nil {
|
||||
return membership.BusBinding{}, fmt.Errorf("%s has no membership issued on %s yet, so none of its bus state is "+
|
||||
"reachable until the mesh issues one (novox/hq ADR 0595)", module, c.node)
|
||||
}
|
||||
var names []string
|
||||
for _, b := range m.Bindings {
|
||||
if b.Local != local {
|
||||
names = append(names, b.Local+" ("+b.Storage+")")
|
||||
continue
|
||||
}
|
||||
if b.Storage != storage {
|
||||
return membership.BusBinding{}, fmt.Errorf("%s's %q is bound with the storage type %s, not %s (novox/hq ADR 0595)",
|
||||
module, local, b.Storage, storage)
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
sort.Strings(names)
|
||||
bound := "none"
|
||||
if len(names) > 0 {
|
||||
bound = strings.Join(names, ", ")
|
||||
}
|
||||
return membership.BusBinding{}, fmt.Errorf("%s was bound no %s called %q: a module reaches only the bus state its "+
|
||||
"manifest asks for in `requires`, by its local name, once the mesh created it; it was bound: %s (novox/hq ADR 0595)",
|
||||
module, storage, local, bound)
|
||||
}
|
||||
|
||||
// Bindings is every binding a module's membership carries now.
|
||||
func (c *Conn) Bindings(module string) []membership.BusBinding {
|
||||
if m := c.Membership(module); m != nil {
|
||||
return append([]membership.BusBinding(nil), m.Bindings...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// jetStream is a JetStream handle on this connection.
|
||||
func (c *Conn) jetStream() (jetstream.JetStream, error) { return jetstream.New(c.nc) }
|
||||
|
||||
func (c *Conn) objectStore(ctx context.Context, module, local string) (jetstream.ObjectStore, membership.BusBinding, error) {
|
||||
b, err := c.Binding(module, local, catalogue.StorageObjectStore)
|
||||
if err != nil {
|
||||
return nil, b, err
|
||||
}
|
||||
js, err := c.jetStream()
|
||||
if err != nil {
|
||||
return nil, b, err
|
||||
}
|
||||
obs, err := js.ObjectStore(ctx, b.ObjectStore)
|
||||
if errors.Is(err, jetstream.ErrBucketNotFound) || errors.Is(err, jetstream.ErrStreamNotFound) {
|
||||
return nil, b, fmt.Errorf("%s's objects %q are bound and their store %s is not on the bus", module, local, b.ObjectStore)
|
||||
}
|
||||
return obs, b, err
|
||||
}
|
||||
|
||||
func objectInfoOf(i *jetstream.ObjectInfo) buscalls.ObjectInfo {
|
||||
return buscalls.ObjectInfo{Name: i.Name, Size: i.Size, Chunks: i.Chunks, Digest: i.Digest, Description: i.Description,
|
||||
Modified: i.ModTime, Deleted: i.Deleted}
|
||||
}
|
||||
|
||||
// ObjectsPut stores one object whole under its name in the module's object store.
|
||||
func (c *Conn) ObjectsPut(module string, r buscalls.ObjectsPutRequest) (buscalls.ObjectInfo, error) {
|
||||
if r.Name == "" {
|
||||
return buscalls.ObjectInfo{}, errors.New("an object is put under a name")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
info, err := obs.PutBytes(ctx, r.Name, r.Data)
|
||||
if err != nil {
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
if r.Description != "" {
|
||||
if err := obs.UpdateMeta(ctx, r.Name, jetstream.ObjectMeta{Name: r.Name, Description: r.Description}); err != nil {
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
if info, err = obs.GetInfo(ctx, r.Name); err != nil {
|
||||
return buscalls.ObjectInfo{}, err
|
||||
}
|
||||
}
|
||||
return objectInfoOf(info), nil
|
||||
}
|
||||
|
||||
// ObjectsGet is one object of the module's object store, its info and its bytes; nil when there is none.
|
||||
func (c *Conn) ObjectsGet(module string, r buscalls.ObjectsNameRequest) (*buscalls.ObjectData, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
res, err := obs.Get(ctx, r.Name)
|
||||
if errors.Is(err, jetstream.ErrObjectNotFound) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer res.Close()
|
||||
info, err := res.Info()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
data, err := obs.GetBytes(ctx, r.Name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &buscalls.ObjectData{Info: objectInfoOf(info), Data: data}, nil
|
||||
}
|
||||
|
||||
// ObjectsDelete deletes one object of the module's object store; one that is not there is not an error.
|
||||
func (c *Conn) ObjectsDelete(module string, r buscalls.ObjectsNameRequest) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := obs.Delete(ctx, r.Name); err != nil && !errors.Is(err, jetstream.ErrObjectNotFound) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ObjectsList is every object the module's object store holds.
|
||||
func (c *Conn) ObjectsList(module string, r buscalls.ObjectsRequest) (buscalls.ObjectsListed, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
if err != nil {
|
||||
return buscalls.ObjectsListed{}, err
|
||||
}
|
||||
out := buscalls.ObjectsListed{Objects: []buscalls.ObjectInfo{}}
|
||||
infos, err := obs.List(ctx)
|
||||
if errors.Is(err, jetstream.ErrNoObjectsFound) {
|
||||
return out, nil
|
||||
}
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for _, i := range infos {
|
||||
out.Objects = append(out.Objects, objectInfoOf(i))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ObjectsInfo is how full the module's object store is: its objects, its bytes and its cap.
|
||||
func (c *Conn) ObjectsInfo(module string, r buscalls.ObjectsRequest) (buscalls.Fill, error) {
|
||||
listed, err := c.ObjectsList(module, r)
|
||||
if err != nil {
|
||||
return buscalls.Fill{}, err
|
||||
}
|
||||
b, _ := c.Binding(module, r.Objects, catalogue.StorageObjectStore)
|
||||
fill, err := c.streamFill("OBJ_" + b.ObjectStore)
|
||||
if err != nil {
|
||||
return buscalls.Fill{}, err
|
||||
}
|
||||
fill.Count = uint64(len(listed.Objects))
|
||||
return fill.Fill, nil
|
||||
}
|
||||
|
||||
// ObjectsWatch hands deliver every object of the module's object store as it is, then each stored or deleted.
|
||||
func (c *Conn) ObjectsWatch(module string, r buscalls.ObjectsWatchRequest, deliver func(buscalls.ObjectsChange) error) (func(), error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
obs, _, err := c.objectStore(ctx, module, r.Objects)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
watchCtx, stop := context.WithCancel(context.Background())
|
||||
w, err := obs.Watch(watchCtx, jetstream.IncludeHistory())
|
||||
if err != nil {
|
||||
stop()
|
||||
return nil, err
|
||||
}
|
||||
current := true
|
||||
for current {
|
||||
select {
|
||||
case i, ok := <-w.Updates():
|
||||
if !ok {
|
||||
stop()
|
||||
return nil, errors.New("the object store's watch ended before its current objects were handed over")
|
||||
}
|
||||
if i == nil {
|
||||
current = false
|
||||
continue
|
||||
}
|
||||
if err := deliver(buscalls.ObjectsChange{Watch: r.Watch, Object: objectInfoOf(i), Current: true}); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take an object of %s: %v", module, r.Objects, err)
|
||||
}
|
||||
case <-time.After(StateTimeout):
|
||||
_ = w.Stop()
|
||||
stop()
|
||||
return nil, fmt.Errorf("the object store %s did not hand its current objects in %s", r.Objects, StateTimeout)
|
||||
}
|
||||
}
|
||||
go func() {
|
||||
for i := range w.Updates() {
|
||||
if i == nil {
|
||||
continue
|
||||
}
|
||||
if err := deliver(buscalls.ObjectsChange{Watch: r.Watch, Object: objectInfoOf(i)}); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take an object of %s: %v", module, r.Objects, err)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return func() { _ = w.Stop(); stop() }, nil
|
||||
}
|
||||
|
||||
// streamFill is a stream's fill and its first and last sequence, by its name on the bus.
|
||||
func (c *Conn) streamFill(name string) (buscalls.StreamFilled, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := c.jetStream()
|
||||
if err != nil {
|
||||
return buscalls.StreamFilled{}, err
|
||||
}
|
||||
s, err := js.Stream(ctx, name)
|
||||
if errors.Is(err, jetstream.ErrStreamNotFound) {
|
||||
return buscalls.StreamFilled{}, fmt.Errorf("the stream %s is bound and not on the bus", name)
|
||||
}
|
||||
if err != nil {
|
||||
return buscalls.StreamFilled{}, err
|
||||
}
|
||||
info, err := s.Info(ctx)
|
||||
if err != nil {
|
||||
return buscalls.StreamFilled{}, err
|
||||
}
|
||||
max := uint64(0)
|
||||
if info.Config.MaxBytes > 0 {
|
||||
max = uint64(info.Config.MaxBytes)
|
||||
}
|
||||
return buscalls.StreamFilled{Fill: buscalls.Fill{Count: info.State.Msgs, Bytes: info.State.Bytes, MaxBytes: max},
|
||||
First: info.State.FirstSeq, Last: info.State.LastSeq}, nil
|
||||
}
|
||||
|
||||
// streamBinding is the module's binding of a stream by its local name, a work queue among them.
|
||||
func (c *Conn) streamBinding(module, local string) (membership.BusBinding, error) {
|
||||
return c.Binding(module, local, catalogue.StorageStream)
|
||||
}
|
||||
|
||||
// StreamFill is how full the module's stream is.
|
||||
func (c *Conn) StreamFill(module string, r buscalls.StreamRequest) (buscalls.StreamFilled, error) {
|
||||
b, err := c.streamBinding(module, r.Stream)
|
||||
if err != nil {
|
||||
return buscalls.StreamFilled{}, err
|
||||
}
|
||||
return c.streamFill(b.Stream)
|
||||
}
|
||||
|
||||
// StreamReadMost is the most messages one read answers.
|
||||
const StreamReadMost = 1000
|
||||
|
||||
// StreamRead is the module's stream's messages from a sequence on, Subjects narrowing them, at most Limit of them.
|
||||
func (c *Conn) StreamRead(module string, r buscalls.StreamReadRequest) (buscalls.StreamPage, error) {
|
||||
b, err := c.streamBinding(module, r.Stream)
|
||||
if err != nil {
|
||||
return buscalls.StreamPage{}, err
|
||||
}
|
||||
return c.readStream(b.Stream, r.Subjects, r.From, r.Limit)
|
||||
}
|
||||
|
||||
// readStream reads a stream by its bus name from a sequence on, the messages on the subjects filter names (every
|
||||
// subject when empty), at most limit of them.
|
||||
func (c *Conn) readStream(name, filter string, from uint64, limit int) (buscalls.StreamPage, error) {
|
||||
if limit <= 0 || limit > StreamReadMost {
|
||||
limit = StreamReadMost
|
||||
}
|
||||
if filter == "" {
|
||||
filter = ">"
|
||||
}
|
||||
if from == 0 {
|
||||
from = 1
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := c.jetStream()
|
||||
if err != nil {
|
||||
return buscalls.StreamPage{}, err
|
||||
}
|
||||
s, err := js.Stream(ctx, name)
|
||||
if errors.Is(err, jetstream.ErrStreamNotFound) {
|
||||
return buscalls.StreamPage{}, fmt.Errorf("the stream %s is bound and not on the bus", name)
|
||||
}
|
||||
if err != nil {
|
||||
return buscalls.StreamPage{}, err
|
||||
}
|
||||
info, err := s.Info(ctx)
|
||||
if err != nil {
|
||||
return buscalls.StreamPage{}, err
|
||||
}
|
||||
page := buscalls.StreamPage{Messages: []buscalls.StreamMessage{}, Last: info.State.LastSeq}
|
||||
for seq := from; seq <= info.State.LastSeq && len(page.Messages) < limit; {
|
||||
msg, err := s.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(filter))
|
||||
if errors.Is(err, jetstream.ErrMsgNotFound) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return page, err
|
||||
}
|
||||
body := json.RawMessage(msg.Data)
|
||||
if !json.Valid(msg.Data) {
|
||||
body, _ = json.Marshal(string(msg.Data))
|
||||
}
|
||||
page.Messages = append(page.Messages, buscalls.StreamMessage{Sequence: msg.Sequence, Subject: msg.Subject,
|
||||
Body: body, At: msg.Time})
|
||||
seq = msg.Sequence + 1
|
||||
if len(page.Messages) == limit && seq <= info.State.LastSeq {
|
||||
page.Next = seq
|
||||
}
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
|
||||
// WorkQueueSize is how many messages wait in the module's work queue, against its cap.
|
||||
func (c *Conn) WorkQueueSize(module string, r buscalls.WorkQueueRequest) (buscalls.WorkQueueSized, error) {
|
||||
b, err := c.streamBinding(module, r.Queue)
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueSized{}, err
|
||||
}
|
||||
fill, err := c.streamFill(b.Stream)
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueSized{}, err
|
||||
}
|
||||
return buscalls.WorkQueueSized{Fill: fill.Fill}, nil
|
||||
}
|
||||
|
||||
// WorkQueueDead is the module's work queue's dead letters: in a work queue a message is gone once settled, so one still
|
||||
// kept below a consumer's acknowledgement floor was delivered as often as the consumer allows and never settled.
|
||||
func (c *Conn) WorkQueueDead(module string, r buscalls.WorkQueueDeadRequest) (buscalls.WorkQueueDeadLetters, error) {
|
||||
b, err := c.streamBinding(module, r.Queue)
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueDeadLetters{}, err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := c.jetStream()
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueDeadLetters{}, err
|
||||
}
|
||||
s, err := js.Stream(ctx, b.Stream)
|
||||
if err != nil {
|
||||
return buscalls.WorkQueueDeadLetters{}, fmt.Errorf("the work queue %s: %w", b.Stream, err)
|
||||
}
|
||||
out := buscalls.WorkQueueDeadLetters{Dead: []buscalls.DeadLetter{}}
|
||||
consumers := s.ListConsumers(ctx)
|
||||
for ci := range consumers.Info() {
|
||||
floor := ci.AckFloor.Stream
|
||||
if ci.Config.MaxDeliver <= 0 || floor == 0 {
|
||||
continue
|
||||
}
|
||||
page, err := c.readStream(b.Stream, ci.Config.FilterSubject, 1, r.Limit)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for _, m := range page.Messages {
|
||||
if m.Sequence > floor {
|
||||
break
|
||||
}
|
||||
out.Dead = append(out.Dead, buscalls.DeadLetter{Sequence: m.Sequence, Subject: m.Subject,
|
||||
Deliveries: ci.Config.MaxDeliver, Body: m.Body, At: m.At})
|
||||
}
|
||||
}
|
||||
if err := consumers.Err(); err != nil {
|
||||
return out, err
|
||||
}
|
||||
if r.Limit > 0 && len(out.Dead) > r.Limit {
|
||||
out.Dead = out.Dead[:r.Limit]
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package bus_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// 1966/AC12: given a bundle whose membership binds objects `files`, when it puts and gets `a`, then it reads back what it
|
||||
// put; a local name it was not bound is refused with the reason, and so is another module's binding and a binding of
|
||||
// another storage type. A stream and a work queue bound alike answer their read-back, fill and size.
|
||||
func TestABundleReachesItsOwnBoundObjectsAndStreamsOnly(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
nc, err := nats.Connect(mt.URL(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(nc.Close)
|
||||
js, _ := jetstream.New(nc)
|
||||
ctx := context.Background()
|
||||
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: "m_files"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "m_events", Subjects: []string{"mesh.state.m.events.>"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, s := range []string{"mesh.state.m.events.one", "mesh.state.m.events.two"} {
|
||||
if _, err := js.Publish(ctx, s, []byte(`{"n":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
m := mt.MembershipOf("m", "anchor", false, nil)
|
||||
m.Bindings = []membership.BusBinding{
|
||||
{Local: "files", Storage: catalogue.StorageObjectStore, Template: catalogue.TemplateObjects, ObjectStore: "m_files"},
|
||||
{Local: "events", Storage: catalogue.StorageStream, Template: catalogue.TemplateEvents, Stream: "m_events",
|
||||
Subject: "mesh.state.m.events"},
|
||||
}
|
||||
other := mt.MembershipOf("other", "anchor", false, nil)
|
||||
mesh.Issue(t, m)
|
||||
mesh.Issue(t, other)
|
||||
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn.Logf = func(string, ...any) {}
|
||||
t.Cleanup(conn.Close)
|
||||
conn.Follow("m")
|
||||
conn.Follow("other")
|
||||
mt.Until(t, func() error {
|
||||
if conn.Membership("m") == nil || conn.Membership("other") == nil {
|
||||
return errors.New("not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
if _, err := conn.ObjectsPut("m", buscalls.ObjectsPutRequest{Objects: "files", Name: "a", Data: []byte("what was put")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := conn.ObjectsGet("m", buscalls.ObjectsNameRequest{Objects: "files", Name: "a"})
|
||||
if err != nil || got == nil || string(got.Data) != "what was put" || got.Info.Name != "a" {
|
||||
t.Fatalf("read back %+v, %v", got, err)
|
||||
}
|
||||
if listed, err := conn.ObjectsList("m", buscalls.ObjectsRequest{Objects: "files"}); err != nil || len(listed.Objects) != 1 {
|
||||
t.Fatalf("listed %+v, %v", listed, err)
|
||||
}
|
||||
page, err := conn.StreamRead("m", buscalls.StreamReadRequest{Stream: "events", Limit: 1})
|
||||
if err != nil || len(page.Messages) != 1 || page.Next != 2 || page.Last != 2 {
|
||||
t.Fatalf("read %+v, %v", page, err)
|
||||
}
|
||||
if fill, err := conn.StreamFill("m", buscalls.StreamRequest{Stream: "events"}); err != nil || fill.Count != 2 {
|
||||
t.Fatalf("fill %+v, %v", fill, err)
|
||||
}
|
||||
|
||||
for _, c := range []struct {
|
||||
name, module, refusal string
|
||||
call func(module string) error
|
||||
}{
|
||||
{"a local name not bound", "m", `bound no object-store called "photos"`, func(module string) error {
|
||||
_, err := conn.ObjectsGet(module, buscalls.ObjectsNameRequest{Objects: "photos", Name: "a"})
|
||||
return err
|
||||
}},
|
||||
{"another module's binding", "other", `other was bound no object-store called "files"`, func(module string) error {
|
||||
_, err := conn.ObjectsGet(module, buscalls.ObjectsNameRequest{Objects: "files", Name: "a"})
|
||||
return err
|
||||
}},
|
||||
{"a binding of another storage type", "m", `"events" is bound with the storage type stream, not object-store`, func(module string) error {
|
||||
_, err := conn.ObjectsList(module, buscalls.ObjectsRequest{Objects: "events"})
|
||||
return err
|
||||
}},
|
||||
{"a work queue not bound", "m", `bound no stream called "jobs"`, func(module string) error {
|
||||
_, err := conn.WorkQueueSize(module, buscalls.WorkQueueRequest{Queue: "jobs"})
|
||||
return err
|
||||
}},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
err := c.call(c.module)
|
||||
if err == nil || !strings.Contains(err.Error(), c.refusal) {
|
||||
t.Fatalf("refused with %v, want the reason %q", err, c.refusal)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+179
-100
@@ -10,6 +10,7 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
@@ -17,6 +18,8 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"git.novox.be/novox/mesh-sdk/go/buscompose"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"log"
|
||||
"regexp"
|
||||
"strings"
|
||||
@@ -29,13 +32,9 @@ import (
|
||||
"github.com/novox/mesh-tools/node-tools/internal/wire"
|
||||
)
|
||||
|
||||
// RequestTimeout is how long a call waits for its answer — what modules already expect. The
|
||||
// controller answers its verbs well inside it (its AnswerWithin, ten seconds), with a call id while a
|
||||
// verb still runs, so a caller is never left guessing past this (novox/hq issue 265).
|
||||
// RequestTimeout is how long a call waits for its answer — what modules already expect.
|
||||
const RequestTimeout = 30 * time.Second
|
||||
|
||||
const assignmentsStream = "ASSIGNMENTS"
|
||||
|
||||
// Credential is a broker credential as the mesh delivers it (novox/hq ADR 0120).
|
||||
type Credential struct {
|
||||
URL string `json:"url"`
|
||||
@@ -79,6 +78,9 @@ type Membership struct {
|
||||
// Waits are the event families a session may wait on through the mesh MCP server (novox/hq ADR 0367 §2):
|
||||
// the tool runner's alone, composed from its manifest with the grant that lets it subscribe them.
|
||||
Waits []WaitIssued `json:"waits,omitempty"`
|
||||
// Bindings are the module's bus state in the new form, by local name (novox/hq ADR 0595 §1, mesh-sdk
|
||||
// membership.BusBinding): its object stores, streams, work queues and ledgers, its own alone (bindings.go).
|
||||
Bindings []membership.BusBinding `json:"bindings,omitempty"`
|
||||
}
|
||||
|
||||
// Served is an address a tool is answered on; `{tool}` stands for the tool's name.
|
||||
@@ -286,9 +288,20 @@ func (c *Conn) refused(err error) {
|
||||
})
|
||||
}
|
||||
|
||||
// Connect dials the bus as the credential's module. A module's subjects come from its credential,
|
||||
// never from its calls (ADR 0074).
|
||||
// Connect dials the bus as the credential's module and follows its membership. A module's subjects come from its
|
||||
// credential, never from its calls (ADR 0074).
|
||||
func Connect(cred Credential) (*Conn, error) {
|
||||
c, err := Dial(cred)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.Follow(cred.Module)
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// Dial dials the bus as the credential's module and follows nothing: a connection that only publishes for its module,
|
||||
// as a module's ledger connection does (novox/hq ADR 0597 §9).
|
||||
func Dial(cred Credential) (*Conn, error) {
|
||||
if cred.Module == "" {
|
||||
return nil, errors.New("a broker credential with no module: the runtime derives its subjects " +
|
||||
"from the account the mesh issued, and cannot guess which module it is")
|
||||
@@ -327,7 +340,6 @@ func Connect(cred Credential) (*Conn, error) {
|
||||
}
|
||||
c = &Conn{nc: nc, js: js, self: cred.Module, node: cred.Node, cred: cred,
|
||||
issued: map[string]*Membership{}, Logf: log.Printf, answering: map[string]*answered{}}
|
||||
c.Follow(cred.Module)
|
||||
return c, nil
|
||||
}
|
||||
|
||||
@@ -340,13 +352,10 @@ func (c *Conn) Node() string { return c.node }
|
||||
// Credential is what this connection was opened with.
|
||||
func (c *Conn) Credential() Credential { return c.cred }
|
||||
|
||||
// MembershipSubject is the one address a runtime derives for an assignment (ADR 0160).
|
||||
func MembershipSubject(node, module string) string {
|
||||
return "mesh.assignment." + node + "." + module
|
||||
}
|
||||
|
||||
// Follow reads a module's membership on this machine once and follows it live, so its tools are
|
||||
// served where the mesh issued them (ADR 0175).
|
||||
// served where the mesh issued them (ADR 0175). It is read from the grants part's state memberships, its key
|
||||
// `<node>.<module>` (novox/hq ADR 0571 clause 7 and 10, mesh-sdk buscompose.GrantsMembershipsBucket), and nowhere else:
|
||||
// the old assignments stream is not read and its subject is not heard.
|
||||
func (c *Conn) Follow(module string) {
|
||||
c.mu.Lock()
|
||||
if c.node == "" {
|
||||
@@ -359,44 +368,150 @@ func (c *Conn) Follow(module string) {
|
||||
}
|
||||
c.issued[module] = nil
|
||||
c.mu.Unlock()
|
||||
subject := MembershipSubject(c.node, module)
|
||||
// The subject-addressed direct get: the one address the mesh grants this account on the
|
||||
// stream's API.
|
||||
if got, err := c.nc.Request("$JS.API.DIRECT.GET."+assignmentsStream+"."+subject, nil, 5*time.Second); err == nil {
|
||||
if got.Header.Get("Status") == "" && len(got.Data) > 0 {
|
||||
var m Membership
|
||||
if json.Unmarshal(got.Data, &m) == nil {
|
||||
c.mu.Lock()
|
||||
c.issued[module] = &m
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
}
|
||||
c.followGrants(module)
|
||||
if c.Membership(module) == nil {
|
||||
c.Logf("[mesh-tools] no membership issued for %s on %s yet; serving the derived shape until one arrives", module, c.node)
|
||||
}
|
||||
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
|
||||
}
|
||||
|
||||
// GrantsPatience bounds the first read of a membership from the grants part's state.
|
||||
var GrantsPatience = 2 * time.Second
|
||||
|
||||
// GrantsRetry is the first wait before the memberships state is read again after it could not be; it doubles up to
|
||||
// GrantsRetryMost.
|
||||
var (
|
||||
GrantsRetry = time.Second
|
||||
GrantsRetryMost = 30 * time.Second
|
||||
)
|
||||
|
||||
// issue applies a membership that arrived for module, live, and re-serves on it.
|
||||
func (c *Conn) issue(module string, m Membership) {
|
||||
c.mu.Lock()
|
||||
c.issued[module] = &m
|
||||
handlers := make([]func(Membership), 0, len(c.onNew))
|
||||
for _, h := range c.onNew {
|
||||
handlers = append(handlers, *h)
|
||||
}
|
||||
c.mu.Unlock()
|
||||
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
|
||||
for _, h := range handlers {
|
||||
h(m)
|
||||
}
|
||||
_ = c.nc.Flush()
|
||||
}
|
||||
|
||||
// followGrants reads module's key of the grants part's state memberships and watches it: the value there now is applied
|
||||
// before Follow returns, each change after it live. A state that is not there yet, or not readable (a bus answering 503
|
||||
// right after its boot, a timeout), is said once and read again, with a growing wait, until it can be or the
|
||||
// connection closes: Follow runs once per module, so nothing else would ever read it again.
|
||||
func (c *Conn) followGrants(module string) {
|
||||
if c.watchGrants(module, true) {
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
wait := GrantsRetry
|
||||
for !c.nc.IsClosed() {
|
||||
time.Sleep(wait)
|
||||
if c.nc.IsClosed() {
|
||||
return
|
||||
}
|
||||
if c.watchGrants(module, false) {
|
||||
c.Logf("[mesh-tools] the grants part's memberships are read again; %s on %s follows its key", module, c.node)
|
||||
return
|
||||
}
|
||||
if wait *= 2; wait > GrantsRetryMost {
|
||||
wait = GrantsRetryMost
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// watchGrants is one attempt of followGrants: true once the key is watched. A failure is said when say is true.
|
||||
func (c *Conn) watchGrants(module string, say bool) bool {
|
||||
logf := func(format string, a ...any) {
|
||||
if say {
|
||||
c.Logf(format, a...)
|
||||
}
|
||||
}
|
||||
key := buscompose.MembershipKey(c.node, module)
|
||||
// First the key's direct get, which the account is granted alone: a bus that refuses it, or a state not there yet,
|
||||
// answers at once, where binding the bucket would wait out its bound on a refusal.
|
||||
bucket := buscompose.GrantsMembershipsBucket
|
||||
probe, err := c.nc.Request("$JS.API.DIRECT.GET.KV_"+bucket+".$KV."+bucket+"."+key, nil, GrantsPatience)
|
||||
if err != nil || probe.Header.Get("Status") == "503" {
|
||||
why := "it did not answer"
|
||||
if err != nil {
|
||||
why = err.Error()
|
||||
}
|
||||
logf("[mesh-tools] the grants part's memberships (%s) cannot be read (%s); %s has no membership "+
|
||||
"until it can be", bucket, why, module)
|
||||
return false
|
||||
}
|
||||
js, err := jetstream.New(c.nc)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := js.KeyValue(ctx, bucket)
|
||||
if err != nil {
|
||||
logf("[mesh-tools] the grants part's memberships (%s) cannot be read (%v); %s has no membership "+
|
||||
"until it can be", buscompose.GrantsMembershipsBucket, err, module)
|
||||
return false
|
||||
}
|
||||
w, err := kv.Watch(context.Background(), key)
|
||||
if err != nil {
|
||||
logf("[mesh-tools] %s's key %s of the grants part's memberships cannot be watched (%v); %s has no membership "+
|
||||
"until it can be", module, key, err, module)
|
||||
return false
|
||||
}
|
||||
apply := func(e jetstream.KeyValueEntry) {
|
||||
if e.Operation() != jetstream.KeyValuePut {
|
||||
return // a membership withdrawn is the stream's, or the module's unassignment's, to say
|
||||
}
|
||||
// Read through mesh-sdk's one codec of this state, the grants part's writer's (issue 774): it takes the bare
|
||||
// membership and a wrapper the part wrote before of one published or unchanged, and refuses a held-back one's
|
||||
// wrapper or a value naming no node or module, which then leaves the module with no membership.
|
||||
// A refused value arriving after a good one keeps the last good membership: nothing here clears it.
|
||||
var m Membership
|
||||
if err := json.Unmarshal(msg.Data, &m); err != nil {
|
||||
c.Logf("[mesh-tools] a membership arrived that is not one: %v", err)
|
||||
body, err := membership.BareGrantsMembership(e.Value())
|
||||
if err == nil {
|
||||
err = json.Unmarshal(body, &m)
|
||||
}
|
||||
if err != nil {
|
||||
c.Logf("[mesh-tools] the membership at %s is not one: %v", key, err)
|
||||
return
|
||||
}
|
||||
c.mu.Lock()
|
||||
c.issued[module] = &m
|
||||
handlers := make([]func(Membership), 0, len(c.onNew))
|
||||
for _, h := range c.onNew {
|
||||
handlers = append(handlers, *h)
|
||||
}
|
||||
c.mu.Unlock()
|
||||
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
|
||||
for _, h := range handlers {
|
||||
h(m)
|
||||
}
|
||||
_ = c.nc.Flush()
|
||||
})
|
||||
if err == nil {
|
||||
c.track(sub)
|
||||
c.issue(module, m)
|
||||
}
|
||||
timer := time.NewTimer(StateTimeout)
|
||||
defer timer.Stop()
|
||||
for current := true; current; {
|
||||
select {
|
||||
case e, ok := <-w.Updates():
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
if e == nil {
|
||||
current = false
|
||||
continue
|
||||
}
|
||||
apply(e)
|
||||
case <-timer.C:
|
||||
_ = w.Stop()
|
||||
logf("[mesh-tools] %s's key %s of the grants part's memberships did not answer in %s; %s has no membership "+
|
||||
"until it can be", module, key, StateTimeout, module)
|
||||
return false
|
||||
}
|
||||
}
|
||||
go func() {
|
||||
for e := range w.Updates() {
|
||||
if e != nil {
|
||||
apply(e)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return true
|
||||
}
|
||||
|
||||
// Membership is what the mesh issued a module here, or nil when nothing has been issued.
|
||||
@@ -686,83 +801,47 @@ func (c *Conn) Ask(key string, body any, on string) (Answered, error) {
|
||||
// account publish with its own name alone; asked again on the subject that names no caller only when the
|
||||
// first reached nobody — nothing serves it yet, or the bus does not grant it yet — so no call is done twice.
|
||||
if called := CalledSubject(subject, c.cred.User); called != "" {
|
||||
answered, handingOver, fallBack, err := c.askCalled(called, data)
|
||||
answered, fallBack, err := c.askCalled(called, data)
|
||||
if !fallBack {
|
||||
if !handingOver {
|
||||
return answered, err
|
||||
}
|
||||
time.Sleep(HandoverPause)
|
||||
answered, again, _, err := c.askCalled(called, data)
|
||||
if err != nil && again {
|
||||
return Answered{}, fmt.Errorf("asked twice, %s apart, and both times refused as a handover: %w", HandoverPause, err)
|
||||
}
|
||||
return answered, err
|
||||
}
|
||||
}
|
||||
answered, handingOver, err := c.askOnce(subject, data)
|
||||
if !handingOver {
|
||||
return answered, err
|
||||
}
|
||||
// **Refused because the controller is handing over: asked once more** (novox/hq issue 289). The
|
||||
// controller being replaced did nothing and said so; the one after it answers the same call. Once:
|
||||
// a second refusal is a controller that is not coming back, and the caller is told.
|
||||
time.Sleep(HandoverPause)
|
||||
answered, again, err := c.askOnce(subject, data)
|
||||
if err != nil && again {
|
||||
return Answered{}, fmt.Errorf("asked twice, %s apart, and both times refused as a handover: %w", HandoverPause, err)
|
||||
}
|
||||
return answered, err
|
||||
return c.askOnce(subject, data)
|
||||
}
|
||||
|
||||
// HandoverPause is how long a call refused as a handover waits before it is asked once more: the
|
||||
// supervisor's restart of the controller, which stops the one being replaced and starts the next. A
|
||||
// variable so a test need not wait.
|
||||
var HandoverPause = 3 * time.Second
|
||||
|
||||
// RetryHandingOver is the mark the controller puts on a call it refused because it is being replaced
|
||||
// (mesh-controller internal/link, `"retry": "handing-over"`).
|
||||
const RetryHandingOver = "handing-over"
|
||||
|
||||
// ErrNoResponders is the bus's own refusal: nothing subscribes to the subject asked. It is told apart
|
||||
// from a tool's own error by errors.Is, never by its text (a tool may say "503" itself, issue 1529).
|
||||
var ErrNoResponders = errors.New("503 no responders")
|
||||
|
||||
// askOnce asks once, and says whether the answer was a refusal marked to be asked again.
|
||||
func (c *Conn) askOnce(subject string, data []byte) (Answered, bool, error) {
|
||||
// askOnce asks once.
|
||||
func (c *Conn) askOnce(subject string, data []byte) (Answered, error) {
|
||||
msg, err := c.nc.Request(subject, data, RequestTimeout)
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return Answered{}, false, ErrNoResponders
|
||||
return Answered{}, ErrNoResponders
|
||||
}
|
||||
if errors.Is(err, nats.ErrTimeout) {
|
||||
return Answered{}, false, errors.New("timeout")
|
||||
return Answered{}, errors.New("timeout")
|
||||
}
|
||||
return Answered{}, false, err
|
||||
return Answered{}, err
|
||||
}
|
||||
return c.answerOf(subject, msg.Data)
|
||||
return answerOf(msg.Data)
|
||||
}
|
||||
|
||||
// answerOf reads one answer, asking the rest of a paged one on the subject it was asked on, and says whether it
|
||||
// was a refusal marked to be asked again.
|
||||
func (c *Conn) answerOf(subject string, data []byte) (Answered, bool, error) {
|
||||
// An answer larger than one message comes in parts (pages.go, novox/hq issue 314).
|
||||
body, err := c.whole(subject, data)
|
||||
if err != nil {
|
||||
return Answered{}, false, err
|
||||
}
|
||||
// answerOf reads one answer.
|
||||
func answerOf(data []byte) (Answered, error) {
|
||||
var r struct {
|
||||
Result json.RawMessage `json:"result"`
|
||||
Error string `json:"error"`
|
||||
Node string `json:"node"`
|
||||
Retry string `json:"retry"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &r); err != nil {
|
||||
return Answered{}, false, err
|
||||
if err := json.Unmarshal(data, &r); err != nil {
|
||||
return Answered{}, err
|
||||
}
|
||||
if r.Error != "" {
|
||||
return Answered{}, r.Retry == RetryHandingOver, errors.New(r.Error)
|
||||
return Answered{}, errors.New(r.Error)
|
||||
}
|
||||
return Answered{Result: r.Result, Node: r.Node}, false, nil
|
||||
return Answered{Result: r.Result, Node: r.Node}, nil
|
||||
}
|
||||
|
||||
// PublishAs emits an event as a module: published into JetStream and awaited, de-duplicated by its
|
||||
@@ -966,8 +1045,8 @@ func (c *Conn) AskAs(module, key string, body any) (Answered, error) {
|
||||
}
|
||||
|
||||
// EventsStream is where every module's events land, and ConsumerOf the durable consumer the
|
||||
// controller makes for a module on a machine: the same names the module's own runtime bound
|
||||
// (node-tools/src/broker-nats.ts), so moving the module into the node's runtime neither loses an
|
||||
// mesh makes for a module on a machine: the same names the module's own runtime bound
|
||||
// (the retired TypeScript runtime's broker-nats.ts), so moving the module into the node's runtime neither loses an
|
||||
// event nor sees one twice.
|
||||
const EventsStream = "EVENTS"
|
||||
|
||||
@@ -981,7 +1060,7 @@ func ConsumerOf(node, module string) string {
|
||||
|
||||
// NakDelay is how long an event a handler failed waits before it is offered again: a transient cause
|
||||
// gets another attempt, a permanent one exhausts the consumer's max-deliver rather than spinning. The
|
||||
// consumer then gives the event up, and the controller keeps it in DEAD_LETTERS until a person delivers
|
||||
// consumer then gives the event up, and the mesh keeps it in DEAD_LETTERS until a person delivers
|
||||
// it again — under `mesh.again.<consumer>.…`, from which keyOf reads the same key — or drops it
|
||||
// (novox/hq issue 330).
|
||||
var NakDelay = 5 * time.Second
|
||||
@@ -995,7 +1074,7 @@ var EventHeartbeat = 10 * time.Second
|
||||
// ConsumeAs reads a module's durable consumer on EVENTS, and each one on a module's stream its
|
||||
// membership names (novox/hq ADR 0363 §8), and hands each event to deliver one at a time (novox/hq
|
||||
// ADR 0198): acknowledged when deliver returns nil, negatively acknowledged after NakDelay when it
|
||||
// returns an error, terminated when it is not an event at all. The consumers are the controller's to
|
||||
// returns an error, terminated when it is not an event at all. The consumers are the mesh's to
|
||||
// create; this binds to them and never makes one. Runs until stopped.
|
||||
func (c *Conn) ConsumeAs(module string, deliver func(Envelope) error) (func(), error) {
|
||||
durable := ConsumerOf(c.node, module)
|
||||
@@ -1005,7 +1084,7 @@ func (c *Conn) ConsumeAs(module string, deliver func(Envelope) error) (func(), e
|
||||
}
|
||||
h := newHearing(c, js, module, deliver)
|
||||
// Bound by name, never created and never matched against a subject: the consumer and its
|
||||
// filters are the controller's (design 29 §3), exactly as the module's own runtime bound it.
|
||||
// filters are the mesh's (design 29 §3), exactly as the module's own runtime bound it.
|
||||
events, _, doing, err := h.bind(EventsStream, durable, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s %s's consumer %s: %w", doing, module, durable, err)
|
||||
|
||||
@@ -18,7 +18,7 @@ import (
|
||||
// mesh.seat.<seat>.call.<verb>.<caller> a mesh seat's verb
|
||||
// mesh.seat.<seat>.call.<verb>.<node>.<caller> a node seat's verb on one machine
|
||||
//
|
||||
// The last token is the bus user that published the call, and the controller grants each user the publish of
|
||||
// The last token is the bus user that published the call, and the mesh grants each user the publish of
|
||||
// these subjects with its own name there and no other — so the caller is a fact the bus server enforces, and
|
||||
// the runtime hands it to the module from the subject a call arrived on, never from the payload.
|
||||
//
|
||||
@@ -32,7 +32,7 @@ import (
|
||||
const CallKind = "call"
|
||||
|
||||
// A bus user's name carries dots (`<node>.<module>`, `person.<name>`); a subject token cannot. In the token
|
||||
// each dot is written `~`, which no part of a bus user's name may hold (the controller refuses any part that
|
||||
// each dot is written `~`, which no part of a bus user's name may hold (the mesh refuses any part that
|
||||
// is not letters, digits, `_` or `-`), so the token is the user's name and nothing else.
|
||||
const userDot = "~"
|
||||
|
||||
@@ -118,7 +118,7 @@ type callers struct {
|
||||
pending map[string]map[uint64]context.CancelCauseFunc
|
||||
}
|
||||
|
||||
// errCallRefused is the bus refusing a caller-named publish: this user is not granted it yet (the controller's
|
||||
// errCallRefused is the bus refusing a caller-named publish: this user is not granted it yet (the mesh's
|
||||
// grant not delivered to the bus), so the call is asked again on the subject that names no caller.
|
||||
var errCallRefused = errors.New("the bus refused the caller-named subject")
|
||||
|
||||
@@ -150,7 +150,7 @@ func (c *Conn) publishRefused(err error) bool {
|
||||
// askCalled asks once on a caller-named subject. fallBack says the call reached nobody — nothing serves that
|
||||
// subject yet, or the bus refused this user its publish — so it may be asked on the subject that names no
|
||||
// caller without being done twice.
|
||||
func (c *Conn) askCalled(subject string, data []byte) (answered Answered, handingOver, fallBack bool, err error) {
|
||||
func (c *Conn) askCalled(subject string, data []byte) (answered Answered, fallBack bool, err error) {
|
||||
ctx, cancel := context.WithCancelCause(context.Background())
|
||||
ctx, stop := context.WithTimeout(ctx, RequestTimeout)
|
||||
defer stop()
|
||||
@@ -180,22 +180,22 @@ func (c *Conn) askCalled(subject string, data []byte) (answered Answered, handin
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(context.Cause(ctx), errCallRefused):
|
||||
return Answered{}, false, true, nil
|
||||
return Answered{}, true, nil
|
||||
case errors.Is(err, nats.ErrNoResponders):
|
||||
c.saidFallBack(subject)
|
||||
return Answered{}, false, true, nil
|
||||
return Answered{}, true, nil
|
||||
case errors.Is(err, context.DeadlineExceeded) || errors.Is(err, nats.ErrTimeout):
|
||||
return Answered{}, false, false, errors.New("timeout")
|
||||
return Answered{}, false, errors.New("timeout")
|
||||
}
|
||||
return Answered{}, false, false, err
|
||||
return Answered{}, false, err
|
||||
}
|
||||
answered, handingOver, err = c.answerOf(subject, msg.Data)
|
||||
return answered, handingOver, false, err
|
||||
answered, err = answerOf(msg.Data)
|
||||
return answered, false, err
|
||||
}
|
||||
|
||||
// saidFallBack says, once per subject, that a caller-named call reached nobody and is asked on the subject that
|
||||
// names no caller: a runtime or holder not yet serving it, or a grant that drifted, is seen in the log rather
|
||||
// than read later as a quiet "unproven" (novox/hq issue 365; expected for the controller's seat until hq issue
|
||||
// than read later as a quiet "unproven" (novox/hq issue 365; expected until hq issue
|
||||
// 464).
|
||||
func (c *Conn) saidFallBack(subject string) {
|
||||
c.calls.mu.Lock()
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// A bus whose users are granted as the controller grants them (mesh-controller internal/broker): each named
|
||||
// A bus whose users are granted as the controller grants them (mesh-placement internal/broker): each named
|
||||
// user publishes the subjects listed for it.
|
||||
func grantedBus(t *testing.T, publish map[string][]string) string {
|
||||
t.Helper()
|
||||
|
||||
@@ -1,73 +0,0 @@
|
||||
package bus_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// **A call the controller refused because it is handing over is asked once more** (novox/hq issue 289):
|
||||
// the controller being replaced did nothing and marked its refusal, and the one after it answers. A
|
||||
// refusal not so marked is final, and a second handover refusal is said, not asked a third time.
|
||||
func TestAHandoverRefusalIsAskedOnceMore(t *testing.T) {
|
||||
was := bus.HandoverPause
|
||||
bus.HandoverPause = 10 * time.Millisecond
|
||||
t.Cleanup(func() { bus.HandoverPause = was })
|
||||
|
||||
url := meshtest.URL(t)
|
||||
seat, err := nats.Connect(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer seat.Close()
|
||||
c, err := bus.Connect(bus.Credential{URL: url, Module: "console", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer c.Close()
|
||||
|
||||
serve := func(t *testing.T, verb string, answers ...map[string]any) *atomic.Int32 {
|
||||
t.Helper()
|
||||
var asked atomic.Int32
|
||||
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", verb, "", ""), func(m *nats.Msg) {
|
||||
n := int(asked.Add(1)) - 1
|
||||
if n >= len(answers) {
|
||||
n = len(answers) - 1
|
||||
}
|
||||
body, _ := json.Marshal(answers[n])
|
||||
_ = m.Respond(body)
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
_ = seat.Flush()
|
||||
return &asked
|
||||
}
|
||||
handingOver := map[string]any{"error": "the controller is handing over to the next one; nothing was done, ask again",
|
||||
"retry": bus.RetryHandingOver}
|
||||
|
||||
asked := serve(t, "rotate", handingOver, map[string]any{"result": "rotated"})
|
||||
got, err := c.Ask("seat:mesh-controller.rotate", map[string]any{}, "")
|
||||
if err != nil || string(got.Result) != `"rotated"` || asked.Load() != 2 {
|
||||
t.Fatalf("after a handover refusal: %s %v, asked %d times", got.Result, err, asked.Load())
|
||||
}
|
||||
|
||||
asked = serve(t, "push", map[string]any{"error": "refused for its own reason"})
|
||||
if _, err := c.Ask("seat:mesh-controller.push", map[string]any{}, ""); err == nil || asked.Load() != 1 {
|
||||
t.Fatalf("an unmarked refusal was asked again: %v, asked %d times", err, asked.Load())
|
||||
}
|
||||
|
||||
asked = serve(t, "status", handingOver)
|
||||
_, err = c.Ask("seat:mesh-controller.status", map[string]any{}, "")
|
||||
if err == nil || asked.Load() != 2 || !strings.Contains(err.Error(), "both times refused as a handover") {
|
||||
t.Fatalf("two handover refusals: %v, asked %d times", err, asked.Load())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,351 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscompose"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/ledger"
|
||||
)
|
||||
|
||||
// A ledger on the bus (novox/hq ADR 0597, design 25 § 12.4; issue 1966): its history a stream with a subject per
|
||||
// instance under mesh-sdk's buscompose.LedgerHistorySubject, its state a key-value bucket, its snapshots a subject per
|
||||
// instance under buscompose.LedgerSnapshotSubject, the latest only (the stream that holds them keeps one per subject, as the bus module
|
||||
// creates it). **Only the module's ledger connection publishes on its history and its snapshots** (ADR 0597 §9): what
|
||||
// is read, and the state, go on the runner's own.
|
||||
|
||||
// LedgerStores are the stores of a module's ledger binding: reads and the state on c, the history's appends and the
|
||||
// snapshots' writes on writer, the module's ledger connection. The subjects are mesh-sdk's, the ones the grants part
|
||||
// grants the module's ledger account byte for byte; a binding naming others is refused, naming both.
|
||||
func (c *Conn) LedgerStores(module string, b membership.BusBinding, writer *Conn) (ledger.History, ledger.State, ledger.Snapshots, error) {
|
||||
history, snapshot := buscompose.LedgerHistorySubject(module, b.Local), buscompose.LedgerSnapshotSubject(module, b.Local)
|
||||
for _, named := range []struct{ what, bound, want string }{{"history", b.Subject, history}, {"snapshot", b.Snapshot, snapshot}} {
|
||||
if named.bound != "" && prefixOf(named.bound) != named.want {
|
||||
return nil, nil, nil, fmt.Errorf("%s's ledger %q is bound with the %s subject %s, and its ledger account is granted "+
|
||||
"%s alone (mesh-sdk buscompose); the binding is refused", module, b.Local, named.what, named.bound, named.want)
|
||||
}
|
||||
}
|
||||
return &busHistory{read: c, write: writer, stream: b.History, prefix: history},
|
||||
&busState{c: c, bucket: b.State},
|
||||
&busSnapshots{read: c, write: writer, prefix: snapshot}, nil
|
||||
}
|
||||
|
||||
// prefixOf is a binding's subject as the prefix an instance's key is added to: written with or without its `.>`.
|
||||
func prefixOf(subject string) string {
|
||||
return strings.TrimSuffix(strings.TrimSuffix(subject, ">"), ".")
|
||||
}
|
||||
|
||||
// publishExpecting publishes body on subject on the connection, written only when the subject's last message is still
|
||||
// at expect (0: it has none); ledger.ErrRaced when it is not.
|
||||
func publishExpecting(c *Conn, subject string, body []byte, expect uint64) (uint64, error) {
|
||||
msg := nats.NewMsg(subject)
|
||||
msg.Data = body
|
||||
msg.Header.Set(jetstream.ExpectedLastSubjSeqHeader, strconv.FormatUint(expect, 10))
|
||||
reply, err := c.nc.RequestMsg(msg, StateTimeout)
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return 0, fmt.Errorf("no stream on the bus keeps %s", subject)
|
||||
}
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var ack struct {
|
||||
Seq uint64 `json:"seq"`
|
||||
Error *struct {
|
||||
ErrCode uint16 `json:"err_code"`
|
||||
Description string `json:"description"`
|
||||
} `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(reply.Data, &ack); err != nil {
|
||||
return 0, fmt.Errorf("the bus answered the write to %s with what is not an acknowledgement: %q", subject, reply.Data)
|
||||
}
|
||||
if ack.Error != nil {
|
||||
if code := jetstream.ErrorCode(ack.Error.ErrCode); code == jetstream.JSErrCodeStreamWrongLastSequence ||
|
||||
code == jetstream.JSErrCodeStreamWrongLastSequenceConstant {
|
||||
return 0, ledger.ErrRaced
|
||||
}
|
||||
return 0, fmt.Errorf("the bus refused the write to %s: %s (%d)", subject, ack.Error.Description, ack.Error.ErrCode)
|
||||
}
|
||||
return ack.Seq, nil
|
||||
}
|
||||
|
||||
// lastOn is the last message on a subject of the stream that keeps it: its bytes and sequence, nil and 0 for none.
|
||||
func lastOn(c *Conn, stream, subject string) ([]byte, uint64, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
js, err := c.jetStream()
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
if stream == "" {
|
||||
if stream, err = js.StreamNameBySubject(ctx, subject); err != nil {
|
||||
return nil, 0, fmt.Errorf("no stream on the bus keeps %s: %w", subject, err)
|
||||
}
|
||||
}
|
||||
s, err := js.Stream(ctx, stream)
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("the stream %s: %w", stream, err)
|
||||
}
|
||||
m, err := s.GetLastMsgForSubject(ctx, subject)
|
||||
if errors.Is(err, jetstream.ErrMsgNotFound) {
|
||||
return nil, 0, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
return m.Data, m.Sequence, nil
|
||||
}
|
||||
|
||||
type busHistory struct {
|
||||
read, write *Conn
|
||||
stream, prefix string
|
||||
}
|
||||
|
||||
func (h *busHistory) Last(key string) (uint64, error) {
|
||||
_, seq, err := lastOn(h.read, h.stream, h.prefix+"."+key)
|
||||
return seq, err
|
||||
}
|
||||
|
||||
func (h *busHistory) Append(key string, op []byte, expect uint64) (uint64, error) {
|
||||
if h.write == nil {
|
||||
return 0, errors.New("the ledger's history is published only on its module's ledger connection, and none is held")
|
||||
}
|
||||
return publishExpecting(h.write, h.prefix+"."+key, op, expect)
|
||||
}
|
||||
|
||||
func (h *busHistory) Read(keys string, from uint64, limit int) ([]ledger.Kept, error) {
|
||||
var out []ledger.Kept
|
||||
for {
|
||||
want := StreamReadMost
|
||||
if limit > 0 && limit-len(out) < want {
|
||||
want = limit - len(out)
|
||||
}
|
||||
page, err := h.read.readStream(h.stream, h.prefix+"."+keys, from, want)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for _, m := range page.Messages {
|
||||
out = append(out, ledger.Kept{Sequence: m.Sequence, Data: m.Body})
|
||||
}
|
||||
if page.Next == 0 || (limit > 0 && len(out) >= limit) {
|
||||
return out, nil
|
||||
}
|
||||
from = page.Next
|
||||
}
|
||||
}
|
||||
|
||||
type busState struct {
|
||||
c *Conn
|
||||
bucket string
|
||||
}
|
||||
|
||||
func (s *busState) kv(ctx context.Context) (jetstream.KeyValue, error) {
|
||||
js, err := s.c.jetStream()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
kv, err := js.KeyValue(ctx, s.bucket)
|
||||
if errors.Is(err, jetstream.ErrBucketNotFound) {
|
||||
return nil, fmt.Errorf("the ledger's state %s is not on the bus", s.bucket)
|
||||
}
|
||||
return kv, err
|
||||
}
|
||||
|
||||
func (s *busState) Get(key string) ([]byte, uint64, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
e, err := kv.Get(ctx, key)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return nil, 0, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
return e.Value(), e.Revision(), nil
|
||||
}
|
||||
|
||||
// isFull is a write a bucket or stream refused because it holds as much as it may.
|
||||
func isFull(err error) bool {
|
||||
msg := err.Error()
|
||||
return strings.Contains(msg, "maximum bytes exceeded") || strings.Contains(msg, "maximum messages exceeded") ||
|
||||
strings.Contains(msg, "insufficient resources")
|
||||
}
|
||||
|
||||
func (s *busState) Write(key string, value []byte, revision uint64) (uint64, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var rev uint64
|
||||
if revision == 0 {
|
||||
rev, err = kv.Create(ctx, key, value)
|
||||
} else {
|
||||
rev, err = kv.Update(ctx, key, value, revision)
|
||||
}
|
||||
switch {
|
||||
case err == nil:
|
||||
return rev, nil
|
||||
case errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) || isWrongSequence(err):
|
||||
return 0, ledger.ErrChanged
|
||||
case isFull(err):
|
||||
return 0, fmt.Errorf("%w: %v", ledger.ErrFull, err)
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
|
||||
func (s *busState) Keys() ([]string, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
keys, err := kv.Keys(ctx)
|
||||
if errors.Is(err, jetstream.ErrNoKeysFound) {
|
||||
return nil, nil
|
||||
}
|
||||
return keys, err
|
||||
}
|
||||
|
||||
type busSnapshots struct {
|
||||
read, write *Conn
|
||||
prefix string
|
||||
}
|
||||
|
||||
func (s *busSnapshots) Get(key string) ([]byte, uint64, error) {
|
||||
return lastOn(s.read, "", s.prefix+"."+key)
|
||||
}
|
||||
|
||||
func (s *busSnapshots) Put(key string, value []byte, expect uint64) error {
|
||||
if s.write == nil {
|
||||
return errors.New("a ledger's snapshot is published only on its module's ledger connection, and none is held")
|
||||
}
|
||||
_, err := publishExpecting(s.write, s.prefix+"."+key, value, expect)
|
||||
return err
|
||||
}
|
||||
|
||||
// Watch hands deliver every instance's record under a key pattern as it is, then each change; a removed key is handed as
|
||||
// nil.
|
||||
func (s *busState) Watch(pattern string, deliver func(key string, value []byte, current bool) error) (func(), error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
kv, err := s.kv(ctx)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
w, err := kv.Watch(context.Background(), pattern)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
hand := func(e jetstream.KeyValueEntry, current bool) {
|
||||
var value []byte
|
||||
if e.Operation() == jetstream.KeyValuePut {
|
||||
value = e.Value()
|
||||
}
|
||||
if err := deliver(e.Key(), value, current); err != nil {
|
||||
s.c.Logf("[mesh-tools] a ledger watch of %s was not taken: %v", s.bucket, err)
|
||||
}
|
||||
}
|
||||
timer := time.NewTimer(StateTimeout)
|
||||
defer timer.Stop()
|
||||
for current := true; current; {
|
||||
select {
|
||||
case e, ok := <-w.Updates():
|
||||
if !ok {
|
||||
return nil, errors.New("the ledger's state watch ended before its current states were handed over")
|
||||
}
|
||||
if e == nil {
|
||||
current = false
|
||||
continue
|
||||
}
|
||||
hand(e, true)
|
||||
case <-timer.C:
|
||||
_ = w.Stop()
|
||||
return nil, fmt.Errorf("the ledger's state %s did not hand its current states in %s", s.bucket, StateTimeout)
|
||||
}
|
||||
}
|
||||
go func() {
|
||||
for e := range w.Updates() {
|
||||
if e != nil {
|
||||
hand(e, false)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return func() { _ = w.Stop() }, nil
|
||||
}
|
||||
|
||||
// LedgerIndexes are the buckets of a ledger binding's declared indexes, by index name, on the runner's own connection.
|
||||
func (c *Conn) LedgerIndexes(b membership.BusBinding) map[string]ledger.Index {
|
||||
out := map[string]ledger.Index{}
|
||||
for name, bucket := range b.Indexes {
|
||||
out[name] = &busState{c: c, bucket: bucket}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *busState) Entries(pattern string, limit int) ([]ledger.Indexed, error) {
|
||||
if pattern == "" {
|
||||
pattern = ">"
|
||||
}
|
||||
keys, err := s.Keys()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sort.Strings(keys)
|
||||
var out []ledger.Indexed
|
||||
for _, k := range keys {
|
||||
if limit > 0 && len(out) >= limit {
|
||||
break
|
||||
}
|
||||
if !SubjectMatches(pattern, k) {
|
||||
continue
|
||||
}
|
||||
v, _, err := s.Get(k)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if v != nil {
|
||||
out = append(out, ledger.Indexed{Key: k, Value: v})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (s *busState) Put(key string, value []byte) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = kv.Put(ctx, key, value)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *busState) Delete(key string) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := s.kv(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := kv.Delete(ctx, key); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+172
-48
@@ -1,6 +1,7 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -8,6 +9,7 @@ import (
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
@@ -15,7 +17,7 @@ import (
|
||||
)
|
||||
|
||||
// A module's log on the bus (novox/hq ADR 0297): a stream of operations, one subject per key, that the
|
||||
// controller creates from what the module declared under `logs` and issues to the module's own instances
|
||||
// mesh creates from what the module declared under `logs` and issues to the module's own instances
|
||||
// in its membership. It mirrors a bucket (ADR 0201) in how it is reached: what a module may reach is
|
||||
// decided here, from its membership, and refused with the reason before anything is sent.
|
||||
//
|
||||
@@ -94,6 +96,10 @@ func logFilter(l LogIssued, pattern string) (string, error) {
|
||||
}
|
||||
parts := strings.Split(pattern, ".")
|
||||
for i, p := range parts {
|
||||
if p == "**" && i == len(parts)-1 {
|
||||
// `**` for the rest, as a state's key pattern and mesh-sdk's log reader say it.
|
||||
parts[i], p = ">", ">"
|
||||
}
|
||||
switch {
|
||||
case p == ">" && i == len(parts)-1, p == "*", logToken.MatchString(p):
|
||||
default:
|
||||
@@ -104,7 +110,7 @@ func logFilter(l LogIssued, pattern string) (string, error) {
|
||||
if len(pattern) > LogKeyMax {
|
||||
return "", fmt.Errorf("%q is longer than a key can be (%d bytes)", pattern, LogKeyMax)
|
||||
}
|
||||
return l.Subject + "." + pattern, nil
|
||||
return l.Subject + "." + strings.Join(parts, "."), nil
|
||||
}
|
||||
|
||||
// issuedLog is the log a module may reach by a name, or the reason it may not.
|
||||
@@ -126,13 +132,14 @@ func (c *Conn) issuedLog(module, name string) (LogIssued, error) {
|
||||
if len(names) > 0 {
|
||||
issued = strings.Join(names, ", ")
|
||||
}
|
||||
return LogIssued{}, fmt.Errorf("%s keeps no log called %q: a module declares the logs it keeps under `logs`, "+
|
||||
"and a log is its owner's alone; it was issued: %s (novox/hq ADR 0297)", module, name, issued)
|
||||
return LogIssued{}, fmt.Errorf("%s keeps no log called %q: a module declares the logs it keeps under `logs`, and "+
|
||||
"another module's it reads under `reads` as <module>.<log>; it was issued: %s (novox/hq ADR 0297, ADR 0571)",
|
||||
module, name, issued)
|
||||
}
|
||||
|
||||
// notOnTheBus says a log is issued and its stream is not there yet.
|
||||
func notOnTheBus(l LogIssued) error {
|
||||
return fmt.Errorf("the log %q is issued and its stream %s is not on the bus yet: the controller creates "+
|
||||
return fmt.Errorf("the log %q is issued and its stream %s is not on the bus yet: the mesh creates "+
|
||||
"it from the catalogue on its next raise (novox/hq ADR 0297)", l.Name, l.Stream)
|
||||
}
|
||||
|
||||
@@ -200,9 +207,10 @@ func (c *Conn) LogAppend(module, name, key string, value json.RawMessage, expect
|
||||
return ack.Seq, nil
|
||||
}
|
||||
|
||||
// LogRead is the entries from a sequence on, of the keys matching a pattern, at most limit of them and
|
||||
// about 4 MiB, by one batch direct get.
|
||||
func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int) (*LogPage, error) {
|
||||
// LogRead is the entries from a sequence on, or from a time on (since, when not zero), of the keys matching a pattern, at
|
||||
// most limit of them and about 4 MiB, by one batch direct get. A log another module keeps is read as its own is, by the
|
||||
// name its reads declare, `<module>.<log>` (novox/hq ADR 0571).
|
||||
func (c *Conn) LogRead(module, name string, from uint64, since time.Time, keys string, limit int) (*LogPage, error) {
|
||||
l, err := c.issuedLog(module, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -218,52 +226,24 @@ func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if from != 0 && !since.IsZero() {
|
||||
return nil, fmt.Errorf("a read of %s starts from a sequence or from a time, not both", name)
|
||||
}
|
||||
if from == 0 {
|
||||
from = 1
|
||||
}
|
||||
req, _ := json.Marshal(map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes})
|
||||
inbox := c.nc.NewInbox()
|
||||
sub, err := c.nc.SubscribeSync(inbox)
|
||||
req := map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes}
|
||||
if !since.IsZero() {
|
||||
delete(req, "seq")
|
||||
req["start_time"] = since.UTC()
|
||||
}
|
||||
entries, more, err := c.logBatch(l, req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = sub.Unsubscribe() }()
|
||||
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+l.Stream, inbox, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
page := &LogPage{Entries: []LogEntry{}}
|
||||
deadline := time.Now().Add(StateTimeout)
|
||||
for done := false; !done; {
|
||||
msg, err := sub.NextMsg(time.Until(deadline))
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return nil, notOnTheBus(l)
|
||||
}
|
||||
if errors.Is(err, nats.ErrTimeout) {
|
||||
return nil, fmt.Errorf("the read of %s did not end in %s", name, StateTimeout)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
switch status := msg.Header.Get("Status"); status {
|
||||
case "":
|
||||
e, err := logEntryOf(l, msg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
page.Entries = append(page.Entries, *e)
|
||||
case "204": // the end of the batch: what is still pending after the last entry sent
|
||||
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
|
||||
if pending > 0 && len(page.Entries) > 0 {
|
||||
page.Next = page.Entries[len(page.Entries)-1].Sequence + 1
|
||||
}
|
||||
done = true
|
||||
case "404": // nothing from that sequence on
|
||||
done = true
|
||||
case "503":
|
||||
return nil, notOnTheBus(l)
|
||||
default:
|
||||
return nil, fmt.Errorf("the read of %s was refused by the bus: %s %s", name, status, msg.Header.Get("Description"))
|
||||
}
|
||||
page := &LogPage{Entries: entries}
|
||||
if more && len(entries) > 0 {
|
||||
page.Next = entries[len(entries)-1].Sequence + 1
|
||||
}
|
||||
info, err := c.logInfo(l)
|
||||
if err != nil {
|
||||
@@ -273,6 +253,150 @@ func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int)
|
||||
return page, nil
|
||||
}
|
||||
|
||||
// LogNewest is the newest entry of each key matching a pattern, in the log's order, by one batch direct get (mesh-sdk
|
||||
// process.LogReader.Newest).
|
||||
func (c *Conn) LogNewest(module, name, keys string) ([]LogEntry, error) {
|
||||
l, err := c.issuedLog(module, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
filter, err := logFilter(l, keys)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries, more, err := c.logBatch(l, map[string]any{"multi_last": []string{filter}, "batch": LogReadMax})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if more {
|
||||
return nil, fmt.Errorf("%s holds more than %d keys matching %q: name fewer", name, LogReadMax, keys)
|
||||
}
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// logBatch is one batch direct get of a log: its entries, and whether more are pending after them.
|
||||
func (c *Conn) logBatch(l LogIssued, request map[string]any) ([]LogEntry, bool, error) {
|
||||
req, _ := json.Marshal(request)
|
||||
inbox := c.nc.NewInbox()
|
||||
sub, err := c.nc.SubscribeSync(inbox)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
defer func() { _ = sub.Unsubscribe() }()
|
||||
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+l.Stream, inbox, req); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
entries := []LogEntry{}
|
||||
deadline := time.Now().Add(StateTimeout)
|
||||
for {
|
||||
msg, err := sub.NextMsg(time.Until(deadline))
|
||||
if err != nil {
|
||||
if errors.Is(err, nats.ErrNoResponders) {
|
||||
return nil, false, notOnTheBus(l)
|
||||
}
|
||||
if errors.Is(err, nats.ErrTimeout) {
|
||||
return nil, false, fmt.Errorf("the read of %s did not end in %s", l.Name, StateTimeout)
|
||||
}
|
||||
return nil, false, err
|
||||
}
|
||||
switch status := msg.Header.Get("Status"); status {
|
||||
case "":
|
||||
e, err := logEntryOf(l, msg)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
entries = append(entries, *e)
|
||||
case "204": // the end of the batch: what is still pending after the last entry sent
|
||||
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
|
||||
return entries, pending > 0, nil
|
||||
case "404": // nothing from there on
|
||||
return entries, false, nil
|
||||
case "503":
|
||||
return nil, false, notOnTheBus(l)
|
||||
default:
|
||||
return nil, false, fmt.Errorf("the read of %s was refused by the bus: %s %s", l.Name, status, msg.Header.Get("Description"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// LogWatch hands deliver every entry of the keys matching a pattern from a sequence on (from), from a time on (since),
|
||||
// or, with neither, each new one, in the log's order, one at a time, through an ordered consumer it lets go of when
|
||||
// stopped (mesh-sdk process.LogReader.Watch). It returns once the entries there at its start are handed over; an error
|
||||
// from deliver is said, and the watch goes on with the next entry.
|
||||
func (c *Conn) LogWatch(module, name, keys string, from uint64, since time.Time, deliver func(LogEntry) error) (func(), error) {
|
||||
l, err := c.issuedLog(module, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
filter, err := logFilter(l, keys)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg := jetstream.OrderedConsumerConfig{FilterSubjects: []string{filter}, DeliverPolicy: jetstream.DeliverNewPolicy}
|
||||
switch {
|
||||
case from != 0 && !since.IsZero():
|
||||
return nil, fmt.Errorf("a watch of %s starts from a sequence or from a time, not both", name)
|
||||
case from != 0:
|
||||
cfg.DeliverPolicy, cfg.OptStartSeq = jetstream.DeliverByStartSequencePolicy, from
|
||||
case !since.IsZero():
|
||||
at := since.UTC()
|
||||
cfg.DeliverPolicy, cfg.OptStartTime = jetstream.DeliverByStartTimePolicy, &at
|
||||
}
|
||||
js, err := jetstream.New(c.nc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
cons, err := js.OrderedConsumer(ctx, l.Stream, cfg)
|
||||
if errors.Is(err, jetstream.ErrStreamNotFound) {
|
||||
return nil, notOnTheBus(l)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
|
||||
}
|
||||
info, err := cons.Info(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
|
||||
}
|
||||
it, err := cons.Messages()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
|
||||
}
|
||||
hand := func(m jetstream.Msg) error {
|
||||
meta, err := m.Metadata()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
e := LogEntry{Sequence: meta.Sequence.Stream, Key: strings.TrimPrefix(m.Subject(), l.Subject+"."),
|
||||
Value: valueOf(m.Data()), At: meta.Timestamp.UTC()}
|
||||
return deliver(e)
|
||||
}
|
||||
for pending := info.NumPending; pending > 0; pending-- {
|
||||
m, err := it.Next()
|
||||
if err == nil {
|
||||
err = hand(m)
|
||||
}
|
||||
if err != nil {
|
||||
it.Stop()
|
||||
return nil, fmt.Errorf("the watch of %s: %w", name, err)
|
||||
}
|
||||
}
|
||||
var stopped atomic.Bool
|
||||
go func() {
|
||||
for {
|
||||
m, err := it.Next()
|
||||
if err != nil || stopped.Load() {
|
||||
return // stopped, or the connection closed; a message fetched as it stopped is not handed over
|
||||
}
|
||||
if err := hand(m); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take entry of %s: %v; it is in the log, a read finds it", module, name, err)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return func() { stopped.Store(true); it.Stop() }, nil
|
||||
}
|
||||
|
||||
// LogLast is a key's last entry, or nil when it has none.
|
||||
func (c *Conn) LogLast(module, name, key string) (*LogEntry, error) {
|
||||
l, err := c.issuedLog(module, name)
|
||||
|
||||
@@ -1,13 +1,7 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/wire"
|
||||
)
|
||||
@@ -15,97 +9,8 @@ import (
|
||||
// An answer larger than the bus carries in one message (novox/hq issue 314).
|
||||
//
|
||||
// The bus carries one message of at most its max_payload, and the client library refuses a larger one
|
||||
// before it leaves the process that sent it. Until this, a holder whose answer was too large wrote one
|
||||
// line to its own log and its caller waited out RequestTimeout and read "did not answer" — the
|
||||
// controller's `conditions` and `status` for hours on 2026-10-08, and the operator's channel with them.
|
||||
//
|
||||
// The controller pages such an answer (mesh-controller internal/link, pages.go): it answers an error
|
||||
// carrying `paged` — the call it holds the answer under, its size and its parts — and a caller asks the
|
||||
// same subject again once per part with PageHeader, each part answered raw with PartHeader. Asking the
|
||||
// same subject means no grant beyond the one the caller already had. A module's answer too large is
|
||||
// refused here, loudly, in a message that fits: its instances may be several, and a part asked again
|
||||
// could reach another.
|
||||
|
||||
const (
|
||||
// PageHeader asks for one part of a paged answer: `<call> <part>`, from zero.
|
||||
PageHeader = "Mesh-Page"
|
||||
// PartHeader is on a part: `<part>/<parts>`.
|
||||
PartHeader = "Mesh-Part"
|
||||
// partTries is how often one part is asked: during a handover a part can reach the controller that
|
||||
// did not hold it.
|
||||
partTries = 3
|
||||
)
|
||||
|
||||
// PartTimeout bounds the asking of one part.
|
||||
var PartTimeout = 10 * time.Second
|
||||
|
||||
// paged is what a too-large answer says instead of itself.
|
||||
type paged struct {
|
||||
Call string `json:"call"`
|
||||
Bytes int `json:"bytes"`
|
||||
Parts int `json:"parts"`
|
||||
}
|
||||
|
||||
func pagedIn(data []byte) (paged, bool) {
|
||||
if !bytes.Contains(data, []byte(`"paged"`)) {
|
||||
return paged{}, false
|
||||
}
|
||||
var r struct {
|
||||
Paged *paged `json:"paged"`
|
||||
}
|
||||
if json.Unmarshal(data, &r) != nil || r.Paged == nil || r.Paged.Call == "" || r.Paged.Parts < 1 {
|
||||
return paged{}, false
|
||||
}
|
||||
return *r.Paged, true
|
||||
}
|
||||
|
||||
// whole is a reply's whole answer: the reply itself, or every part of a paged one, joined. A part that
|
||||
// cannot be had is an error naming it, never a shorter answer.
|
||||
func (c *Conn) whole(subject string, data []byte) ([]byte, error) {
|
||||
p, ok := pagedIn(data)
|
||||
if !ok {
|
||||
return data, nil
|
||||
}
|
||||
out := make([]byte, 0, p.Bytes)
|
||||
for n := 0; n < p.Parts; n++ {
|
||||
var part []byte
|
||||
var err error
|
||||
for try := 0; try < partTries; try++ {
|
||||
if part, err = c.askPart(subject, p, n); err == nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the answer was %d bytes, more than the bus carries in one message, paged as %s "+
|
||||
"in %d parts, and part %d could not be had: %w", p.Bytes, p.Call, p.Parts, n, err)
|
||||
}
|
||||
out = append(out, part...)
|
||||
}
|
||||
if len(out) != p.Bytes {
|
||||
return nil, fmt.Errorf("the answer paged as %s was %d bytes, and its parts joined are %d", p.Call, p.Bytes, len(out))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *Conn) askPart(subject string, p paged, n int) ([]byte, error) {
|
||||
ask := nats.NewMsg(subject)
|
||||
ask.Header.Set(PageHeader, fmt.Sprintf("%s %d", p.Call, n))
|
||||
ask.Data = []byte(`{}`)
|
||||
reply, err := c.nc.RequestMsg(ask, PartTimeout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if got := reply.Header.Get(PartHeader); got != fmt.Sprintf("%d/%d", n, p.Parts) {
|
||||
var r struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
if json.Unmarshal(reply.Data, &r) == nil && r.Error != "" {
|
||||
return nil, errors.New(r.Error)
|
||||
}
|
||||
return nil, fmt.Errorf("asked part %d/%d, answered %q", n, p.Parts, got)
|
||||
}
|
||||
return reply.Data, nil
|
||||
}
|
||||
// before it leaves the process that sent it. A module's answer too large is refused here, loudly, in a
|
||||
// message that fits, so its caller reads why instead of waiting out RequestTimeout.
|
||||
|
||||
// fits is an answer the bus can carry: the answer itself, or — when it is larger than one message —
|
||||
// an error that says so, which is.
|
||||
|
||||
@@ -2,7 +2,6 @@ package bus_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -13,8 +12,8 @@ import (
|
||||
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// novox/hq issue 314: an answer larger than the bus carries in one message arrives whole when the
|
||||
// controller pages it, and a module's is said to be too large — neither is a 30-second silence.
|
||||
// novox/hq issue 314: a module's answer larger than the bus carries in one message is said to be too large,
|
||||
// at once, not a 30-second silence. (The controller's paging ended with the controller, issue 1126.)
|
||||
func TestAnAnswerLargerThanOneMessage(t *testing.T) {
|
||||
url := meshtest.URL(t)
|
||||
seat, err := nats.Connect(url)
|
||||
@@ -27,43 +26,8 @@ func TestAnAnswerLargerThanOneMessage(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer c.Close()
|
||||
limit := int(seat.MaxPayload())
|
||||
large := strings.Repeat("a condition held past its bound; ", 3*int(seat.MaxPayload())/32)
|
||||
|
||||
// The controller as it pages (mesh-controller internal/link, pages.go): the first answer an error
|
||||
// with `paged`, each part asked again on the same subject with the page header.
|
||||
large := strings.Repeat("a condition held past its bound; ", 3*limit/32)
|
||||
whole, _ := json.Marshal(map[string]any{"result": large})
|
||||
partSize := limit - 1024
|
||||
parts := (len(whole) + partSize - 1) / partSize
|
||||
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", "conditions", "", ""), func(m *nats.Msg) {
|
||||
if page := m.Header.Get(bus.PageHeader); page != "" {
|
||||
var id string
|
||||
var n int
|
||||
fmt.Sscanf(page, "%s %d", &id, &n)
|
||||
end := min((n+1)*partSize, len(whole))
|
||||
part := nats.NewMsg(m.Reply)
|
||||
part.Header.Set(bus.PartHeader, fmt.Sprintf("%d/%d", n, parts))
|
||||
part.Data = whole[n*partSize : end]
|
||||
_ = m.RespondMsg(part)
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"error": "too large for one message",
|
||||
"paged": map[string]any{"call": "call-1-1", "bytes": len(whole), "parts": parts}})
|
||||
_ = m.Respond(body)
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer sub.Unsubscribe()
|
||||
_ = seat.Flush()
|
||||
|
||||
got, err := c.Ask("seat:mesh-controller.conditions", map[string]any{}, "")
|
||||
var said string
|
||||
if err != nil || json.Unmarshal(got.Result, &said) != nil || said != large {
|
||||
t.Fatalf("an answer of %d bytes in %d parts arrived as %d bytes: %v", len(whole), parts, len(got.Result), err)
|
||||
}
|
||||
|
||||
// A module's answer too large is refused here in words, at once.
|
||||
stop, err := c.HandleSubject("mesh.mod.journal.tool.lines", func(json.RawMessage) (any, error) { return large, nil })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
package bus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/pernode"
|
||||
)
|
||||
|
||||
// A state its owner keeps one bucket per node (novox/hq ADR 0593 §6: each node's engine writes its own `composed-<node>`,
|
||||
// and the provisioner, the assigner and the view read them all) is served to a bundle as one state. The rules (a key's
|
||||
// first part names its node's bucket, keys and a watch over every bucket of the prefix, a bucket created later heard by
|
||||
// the bus's creation advisory) are mesh-sdk's pernode package, written once for a process and for this runtime; this
|
||||
// file adapts its entries to the runtime's shapes. A reader is granted the listing of stream names
|
||||
// (pernode.ListSubject) and the creation advisory (pernode.CreatedAdvisory) beside each bucket's read.
|
||||
|
||||
func perNodeRefusal(module, name, what string) error {
|
||||
return fmt.Errorf("%s's state %s is kept one bucket per node and read only: no %s of it as one state (novox/hq ADR 0593 §6)",
|
||||
module, name, what)
|
||||
}
|
||||
|
||||
func (c *Conn) family(s StateIssued) pernode.Family {
|
||||
return pernode.Family{Read: s.Name, Prefix: s.Bucket, Bus: pernode.OverNATS(c.nc)}
|
||||
}
|
||||
|
||||
func (c *Conn) perNodeGet(s StateIssued, key string) (*StateEntry, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
e, err := c.family(s).Get(ctx, key)
|
||||
if err != nil || e == nil {
|
||||
return nil, err
|
||||
}
|
||||
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision()}, nil
|
||||
}
|
||||
|
||||
func (c *Conn) perNodeKeys(s StateIssued) ([]string, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
return c.family(s).Keys(ctx)
|
||||
}
|
||||
|
||||
// perNodeWatch is StateWatch over every bucket of the family, and over a bucket created later. As in a single bucket's
|
||||
// watch, StateTimeout bounds the time without progress (issue 1726), and a value the bundle does not take is said and
|
||||
// the watch goes on.
|
||||
func (c *Conn) perNodeWatch(module string, s StateIssued, pattern string, deliver func(StateChange) error) (func(), error) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
progress := make(chan struct{}, 1)
|
||||
handle := func(ch pernode.Change) error {
|
||||
select {
|
||||
case progress <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
e := ch.Entry
|
||||
change := StateChange{State: s.Name, Key: e.Key(), Revision: e.Revision(), Current: ch.Current}
|
||||
switch e.Operation() {
|
||||
case jetstream.KeyValuePut:
|
||||
change.Op = "put"
|
||||
change.Value = valueOf(e.Value())
|
||||
default:
|
||||
if ch.Current {
|
||||
return nil
|
||||
}
|
||||
change.Op = "delete"
|
||||
if e.Operation() == jetstream.KeyValuePurge {
|
||||
change.Op = "expire"
|
||||
}
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
if err := deliver(change); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
|
||||
module, change.Op, change.Key, s.Name, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- c.family(s).Watch(ctx, pattern, handle) }()
|
||||
bound := time.NewTimer(StateTimeout)
|
||||
defer bound.Stop()
|
||||
for {
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
cancel()
|
||||
return nil, err
|
||||
}
|
||||
return cancel, nil
|
||||
case <-progress:
|
||||
if !bound.Stop() {
|
||||
select {
|
||||
case <-bound.C:
|
||||
default:
|
||||
}
|
||||
}
|
||||
bound.Reset(StateTimeout)
|
||||
case <-bound.C:
|
||||
cancel()
|
||||
return nil, fmt.Errorf("the current values of %s did not arrive in %s: none was handed over in that time", s.Name, StateTimeout)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,16 +14,16 @@ import (
|
||||
// ADR 0370 changes it, issue 685).
|
||||
//
|
||||
// While a module's code runs here, its asks reach the bus under this runtime's account, and the bus proves only
|
||||
// that account. So the runtime asks its own machine's client on ProvisionAskSubject — which the controller grants
|
||||
// that account. So the runtime asks its own machine's client on ProvisionAskSubject — which the mesh grants
|
||||
// this machine's tool runner alone to publish (ADR 0370 §3) — naming the module itself, from the bundle that asks
|
||||
// and never from the module's code or arguments (§4). The client answers that module's config values, for what its
|
||||
// manifest declares, and no secret: the vault hands a secret only to a caller the bus proves (§2).
|
||||
|
||||
// ProvisionAskSubject is where this machine's tool runner asks its provisioner client (mesh-controller
|
||||
// ProvisionAskSubject is where this machine's tool runner asks its provisioner client (the node-engine
|
||||
// internal/broker ProvisionerAskSubject): `prov.ask.<node>`.
|
||||
func ProvisionAskSubject(node string) string { return "prov.ask." + node }
|
||||
|
||||
// ProvisionChangedSubjects is every module's word of what moved on a machine (mesh-controller internal/broker
|
||||
// ProvisionChangedSubjects is every module's word of what moved on a machine (the node-engine
|
||||
// ProvisionerChangedSubject): `prov.changed.<node>.<module>`, a module's name dots and all.
|
||||
func ProvisionChangedSubjects(node string) string { return "prov.changed." + node + ".>" }
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ import (
|
||||
// and anything else is refused here, with the reason, before it is sent. A bundle reaches this only through
|
||||
// its own standard input and output: no tool call does.
|
||||
|
||||
// SeatTraffic is what a membership lists of a module's seat traffic (the controller's broker.SeatTraffic).
|
||||
// SeatTraffic is what a membership lists of a module's seat traffic (the grants part's record).
|
||||
type SeatTraffic struct {
|
||||
Publish []string `json:"publish,omitempty"`
|
||||
Subscribe []string `json:"subscribe,omitempty"`
|
||||
@@ -32,7 +32,7 @@ type SeatTraffic struct {
|
||||
Workers []SeatWorker `json:"workers,omitempty"`
|
||||
Records []string `json:"records,omitempty"`
|
||||
// Kinds are the holders of every kinded bench the module uses or watches, with what each promises: the
|
||||
// controller's record of their claims (novox/hq ADR 0259 §5).
|
||||
// record of their claims (novox/hq ADR 0259 §5).
|
||||
Kinds []KindHeld `json:"kinds,omitempty"`
|
||||
}
|
||||
|
||||
@@ -45,7 +45,7 @@ type KindHeld struct {
|
||||
Capabilities []string `json:"capabilities,omitempty"`
|
||||
}
|
||||
|
||||
// SeatKinds is the kinds the module's membership names, as the controller issued them.
|
||||
// SeatKinds is the kinds the module's membership names, as the grants part issued them.
|
||||
func (c *Conn) SeatKinds(module string) ([]KindHeld, error) {
|
||||
t, err := c.trafficOf(module)
|
||||
if err != nil {
|
||||
@@ -249,7 +249,7 @@ type Work struct {
|
||||
|
||||
// SeatTake reads a worker the module's seat traffic lists and hands each message to deliver: acknowledged
|
||||
// when deliver returns nil, offered again after NakDelay when it returns an error, ended when it is not
|
||||
// JSON. The worker is the controller's to create; this binds to it by name and never makes one.
|
||||
// JSON. The worker is the mesh's to create; this binds to it by name and never makes one.
|
||||
func (c *Conn) SeatTake(module, consumer string, deliver func(Work) error) (func(), error) {
|
||||
t, err := c.trafficOf(module)
|
||||
if err != nil {
|
||||
|
||||
@@ -98,7 +98,7 @@ func TestAModuleSubmitsAndSaysOnlyUnderItsOwnNameAndKind(t *testing.T) {
|
||||
t.Fatalf("an asker could not ask under its own name: %v", err)
|
||||
}
|
||||
for module, subject := range map[string]string{
|
||||
"mesh-delivery": "mesh.seat.operator-channel.accept.ask.mesh-controller",
|
||||
"mesh-delivery": "mesh.seat.operator-channel.accept.ask.mesh-placement",
|
||||
"telegram": "mesh.seat.intake.event.choice.desktop",
|
||||
"messenger": "mesh.seat.intake.event.choice.telegram",
|
||||
} {
|
||||
@@ -206,7 +206,7 @@ func TestAnAskerReadsItsOwnRecordAndNoOther(t *testing.T) {
|
||||
if _, err := kv.Put("mesh-delivery.a1", []byte(`{"state":"open"}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := kv.Put("mesh-controller.a1", []byte(`{"state":"open"}`)); err != nil {
|
||||
if _, err := kv.Put("mesh-placement.a1", []byte(`{"state":"open"}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := conn.SeatRecord("mesh-delivery", "messenger_asks", "a1")
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// A module's state on the bus (novox/hq ADR 0201): key-value buckets the controller creates from what
|
||||
// A module's state on the bus (novox/hq ADR 0201): key-value buckets the mesh creates from what
|
||||
// the module declared, and issues to each assignment in its membership by the name the module uses —
|
||||
// its own state by the local name, another module's as `<module>.<name>`.
|
||||
//
|
||||
@@ -28,6 +28,10 @@ type StateIssued struct {
|
||||
Name string `json:"name"`
|
||||
Bucket string `json:"bucket"`
|
||||
Writes bool `json:"writes,omitempty"`
|
||||
// PerNode says the state is kept one bucket per node (novox/hq ADR 0593 §6, mesh-sdk membership.StateIssued): Bucket is
|
||||
// then the buckets' prefix, each bucket the prefix and a node's name, and a key's first part names its node. It is read
|
||||
// only, served as one state (per_node.go).
|
||||
PerNode bool `json:"per-node,omitempty"`
|
||||
}
|
||||
|
||||
// StateEntry is one key's current value.
|
||||
@@ -37,7 +41,8 @@ type StateEntry struct {
|
||||
Revision uint64 `json:"revision"`
|
||||
}
|
||||
|
||||
// StateChange is one change a watch delivers: a key put or deleted.
|
||||
// StateChange is one change a watch delivers: a key put, deleted, or expired (its own time to live, or the
|
||||
// bucket's age, ended; issue 1952).
|
||||
type StateChange struct {
|
||||
State string `json:"state"`
|
||||
Key string `json:"key"`
|
||||
@@ -88,7 +93,7 @@ func (c *Conn) bucket(ctx context.Context, s StateIssued) (jetstream.KeyValue, e
|
||||
kv, err := js.KeyValue(ctx, s.Bucket)
|
||||
if err != nil {
|
||||
if errors.Is(err, jetstream.ErrBucketNotFound) {
|
||||
return nil, fmt.Errorf("the state %q is issued and its bucket is not on the bus yet: the controller "+
|
||||
return nil, fmt.Errorf("the state %q is issued and its bucket is not on the bus yet: the mesh "+
|
||||
"creates it from the catalogue on its next raise", s.Name)
|
||||
}
|
||||
return nil, err
|
||||
@@ -113,6 +118,9 @@ func (c *Conn) StateGet(module, name, key string) (*StateEntry, error) {
|
||||
if err := checkKey(key); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.PerNode {
|
||||
return c.perNodeGet(s, key)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := c.bucket(ctx, s)
|
||||
@@ -148,6 +156,9 @@ func (c *Conn) StateHistory(module, name, key string) ([]StateKept, error) {
|
||||
if err := checkKey(key); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.PerNode {
|
||||
return nil, perNodeRefusal(module, name, "history")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := c.bucket(ctx, s)
|
||||
@@ -173,6 +184,54 @@ func (c *Conn) StateHistory(module, name, key string) ([]StateKept, error) {
|
||||
return kept, nil
|
||||
}
|
||||
|
||||
// StateInfo is how full a state's bucket is against its caps (tools-are-how-a-module-is-spoken-to): a bundle's tools
|
||||
// say fill against cap from it.
|
||||
type StateInfo struct {
|
||||
State string `json:"state"`
|
||||
Bucket string `json:"bucket"`
|
||||
// Bytes is what the bucket holds now, every kept value and marker counted; Keys how many keys it holds, a key
|
||||
// whose last value is a delete or expiry marker counted until the marker goes.
|
||||
Bytes uint64 `json:"bytes"`
|
||||
Keys uint64 `json:"keys"`
|
||||
// MaxBytes and MaxValueBytes are the bucket's caps; -1 is none.
|
||||
MaxBytes int64 `json:"max_bytes"`
|
||||
MaxValueBytes int32 `json:"max_value_bytes"`
|
||||
// History is how many values a key keeps; TTLSeconds the bucket's age for a value (0 none); KeyTTL whether a key may
|
||||
// carry its own time to live (issue 1952).
|
||||
History int64 `json:"history"`
|
||||
TTLSeconds int64 `json:"ttl_seconds"`
|
||||
KeyTTL bool `json:"key_ttl"`
|
||||
}
|
||||
|
||||
// StateInfo reads a state's bucket's fill and caps. It is a read like StateGet: served to the owner and every reader.
|
||||
func (c *Conn) StateInfo(module, name string) (*StateInfo, error) {
|
||||
s, err := c.issuedState(module, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.PerNode {
|
||||
return nil, perNodeRefusal(module, name, "fill")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := c.bucket(ctx, s)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
status, err := kv.Status(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
b, ok := status.(*jetstream.KeyValueBucketStatus)
|
||||
if !ok || b.StreamInfo() == nil {
|
||||
return nil, fmt.Errorf("the bus answered no stream behind %s's state %s", module, name)
|
||||
}
|
||||
info := b.StreamInfo()
|
||||
return &StateInfo{State: name, Bucket: status.Bucket(), Bytes: info.State.Bytes, Keys: uint64(info.State.NumSubjects),
|
||||
MaxBytes: info.Config.MaxBytes, MaxValueBytes: info.Config.MaxMsgSize, History: status.History(),
|
||||
TTLSeconds: int64(status.TTL() / time.Second), KeyTTL: info.Config.AllowMsgTTL && status.LimitMarkerTTL() > 0}, nil
|
||||
}
|
||||
|
||||
// StatePut writes one key, as the module, where the module keeps the state. It answers the revision.
|
||||
func (c *Conn) StatePut(module, name, key string, value json.RawMessage) (uint64, error) {
|
||||
s, err := c.writable(module, name)
|
||||
@@ -199,6 +258,61 @@ func (c *Conn) StatePut(module, name, key string, value json.RawMessage) (uint64
|
||||
return kv.Put(ctx, key, value)
|
||||
}
|
||||
|
||||
// MinKeyTTL is the shortest time to live a key may be given: the bus counts a key's age in whole seconds (mesh-sdk
|
||||
// process.MinTTL).
|
||||
const MinKeyTTL = time.Second
|
||||
|
||||
// StateCreateTTL writes one key only when it has no value, with its own time to live (novox/hq ADR 0564 clauses 4 and
|
||||
// 7, issue 1952; mesh-sdk process.State.CreateTTL): the bus removes it when ttl passes and every watch hears the
|
||||
// removal as an expiry, so a bundle's wake needs no timer. A key that has a value is refused as ErrStateChanged. The
|
||||
// bucket must take keys with their own time to live (a state declared `key-ttl`); one that does not is refused, naming
|
||||
// it, before anything is written.
|
||||
func (c *Conn) StateCreateTTL(module, name, key string, value json.RawMessage, ttl time.Duration) (uint64, error) {
|
||||
if ttl < MinKeyTTL {
|
||||
return 0, fmt.Errorf("%s's state %s, key %s: a time to live of %s is shorter than the bus keeps (%s)",
|
||||
module, name, key, ttl, MinKeyTTL)
|
||||
}
|
||||
return c.stateWrite(module, name, key, value, func(ctx context.Context, kv jetstream.KeyValue) (uint64, error) {
|
||||
if err := takesKeyTTL(ctx, kv, module, name); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return kv.Create(ctx, key, value, jetstream.KeyTTL(ttl))
|
||||
})
|
||||
}
|
||||
|
||||
// StateRenewTTL writes one key whole with a fresh time to live, whatever it held (mesh-sdk process.State.RenewTTL): a
|
||||
// time to live is set only when a key is created, so a live key is deleted first, and its watches hear the delete and
|
||||
// then the new value.
|
||||
func (c *Conn) StateRenewTTL(module, name, key string, value json.RawMessage, ttl time.Duration) (uint64, error) {
|
||||
rev, err := c.StateCreateTTL(module, name, key, value, ttl)
|
||||
if !errors.Is(err, ErrStateChanged) {
|
||||
return rev, err
|
||||
}
|
||||
if err := c.StateDelete(module, name, key); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return c.StateCreateTTL(module, name, key, value, ttl)
|
||||
}
|
||||
|
||||
// takesKeyTTL refuses a bucket that does not allow a time to live per key, or leaves no marker when one ends: there the
|
||||
// server would refuse the write, or the key would vanish and no watch would hear it.
|
||||
func takesKeyTTL(ctx context.Context, kv jetstream.KeyValue, module, name string) error {
|
||||
status, err := kv.Status(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
allows := status.LimitMarkerTTL() > 0
|
||||
if b, ok := status.(*jetstream.KeyValueBucketStatus); ok && b.StreamInfo() != nil {
|
||||
allows = allows && b.StreamInfo().Config.AllowMsgTTL
|
||||
}
|
||||
if !allows {
|
||||
return fmt.Errorf("%s's state %s does not take keys with their own time to live: its bucket %s allows none, or "+
|
||||
"leaves no marker a watch hears when one ends; %s declares the state `key-ttl` for it (novox/hq ADR 0564, "+
|
||||
"issue 1952)", module, name, status.Bucket(), module)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// StateDelete removes one key, as the module, where the module keeps the state. A key that was not
|
||||
// there is not an error: what is asked for is that it is gone.
|
||||
func (c *Conn) StateDelete(module, name, key string) error {
|
||||
@@ -224,12 +338,20 @@ func (c *Conn) StateKeys(module, name string) ([]string, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.PerNode {
|
||||
return c.perNodeKeys(s)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
defer cancel()
|
||||
kv, err := c.bucket(ctx, s)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return keysOf(ctx, kv)
|
||||
}
|
||||
|
||||
// keysOf is every key with a current value in one bucket, sorted.
|
||||
func keysOf(ctx context.Context, kv jetstream.KeyValue) ([]string, error) {
|
||||
lister, err := kv.ListKeys(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -291,12 +413,20 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
|
||||
}
|
||||
filter = strings.Join(parts, ".")
|
||||
}
|
||||
if s.PerNode {
|
||||
return c.perNodeWatch(module, s, pattern, deliver)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
|
||||
kv, err := c.bucket(ctx, s)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.watchBucket(module, name, kv, filter, deliver)
|
||||
}
|
||||
|
||||
// watchBucket is StateWatch over one bucket: the current values, then every change.
|
||||
func (c *Conn) watchBucket(module, name string, kv jetstream.KeyValue, filter string, deliver func(StateChange) error) (stop func(), err error) {
|
||||
watching, stopWatching := context.WithCancel(context.Background())
|
||||
w, err := kv.Watch(watching, filter)
|
||||
if err != nil {
|
||||
@@ -322,6 +452,12 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
|
||||
go func() {
|
||||
initial := true
|
||||
for e := range w.Updates() {
|
||||
// A watch given up, or stopped, hands over nothing more, though values stay buffered in the
|
||||
// watcher: they would reach a watch id the bundle no longer holds (issue 774: 1257 orphan
|
||||
// changes after 51 failed watches).
|
||||
if watching.Err() != nil {
|
||||
break
|
||||
}
|
||||
if e == nil {
|
||||
// The end of what was there when the watch began.
|
||||
if initial {
|
||||
@@ -330,6 +466,11 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
|
||||
}
|
||||
continue
|
||||
}
|
||||
if initial {
|
||||
// Arrival from the bus is progress before the bundle is asked: the bound measures the current
|
||||
// values coming in, and restarts when the bundle has taken one.
|
||||
progress()
|
||||
}
|
||||
change := StateChange{State: name, Key: e.Key(), Revision: e.Revision(), Current: initial}
|
||||
switch e.Operation() {
|
||||
case jetstream.KeyValuePut:
|
||||
@@ -343,6 +484,15 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
|
||||
continue
|
||||
}
|
||||
change.Op = "delete"
|
||||
if e.Operation() == jetstream.KeyValuePurge {
|
||||
// The marker the server leaves when a key's time to live ends (Nats-Marker-Reason MaxAge,
|
||||
// read by the client as a purge): nothing else purges a module's state, since neither the
|
||||
// runtime nor mesh-sdk's State offers a purge (issue 1952).
|
||||
change.Op = "expire"
|
||||
}
|
||||
}
|
||||
if watching.Err() != nil {
|
||||
break
|
||||
}
|
||||
if err := deliver(change); err != nil {
|
||||
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
//go:build integration
|
||||
|
||||
package bus_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// **A state its owner keeps one bucket per node is served to a bundle as one state** (novox/hq ADR 0593 §6: each node's
|
||||
// engine writes its own `composed-<node>`, and the provisioner reads them all). Get goes to the bucket the key's first
|
||||
// part names, Keys and Watch over every bucket of the prefix, a bucket created during the watch included, and a bucket
|
||||
// that only starts like the prefix is never read.
|
||||
func TestAPerNodeStateIsServedAcrossEveryNodesBucket(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
a, b := mesh.Bucket(t, "composed-a"), mesh.Bucket(t, "composed-b")
|
||||
other := mesh.Bucket(t, "composedx-z")
|
||||
for kv, key := range map[interface {
|
||||
Put(string, []byte) (uint64, error)
|
||||
}]string{a: "a.m1.f1", b: "b.m2.f2", other: "z.m3.f3"} {
|
||||
if _, err := kv.Put(key, []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
prov := mt.MembershipOf("mesh-provisioner", "anchor", false, nil)
|
||||
prov.State = []bus.StateIssued{{Name: "mesh-host.composed", Bucket: "composed-", PerNode: true}}
|
||||
mesh.Issue(t, prov)
|
||||
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn.Logf = func(string, ...any) {}
|
||||
t.Cleanup(conn.Close)
|
||||
conn.Follow("mesh-provisioner")
|
||||
mt.Until(t, func() error {
|
||||
if conn.Membership("mesh-provisioner") == nil {
|
||||
return errors.New("not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "b.m2.f2"); err != nil || e == nil || e.Key != "b.m2.f2" {
|
||||
t.Fatalf("get from the second node's bucket: %+v, %v", e, err)
|
||||
}
|
||||
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "q.m.f"); err != nil || e != nil {
|
||||
t.Fatalf("a key of a node with no bucket should read as none: %+v, %v", e, err)
|
||||
}
|
||||
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "z.m3.f3"); err != nil || e != nil {
|
||||
t.Fatalf("the look-alike bucket was read: %+v, %v", e, err)
|
||||
}
|
||||
if _, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "nonode"); err == nil {
|
||||
t.Fatal("a key naming no node should be refused")
|
||||
}
|
||||
if keys, err := conn.StateKeys("mesh-provisioner", "mesh-host.composed"); err != nil || !reflect.DeepEqual(keys, []string{"a.m1.f1", "b.m2.f2"}) {
|
||||
t.Fatalf("keys across the buckets: %v, %v", keys, err)
|
||||
}
|
||||
if _, err := conn.StatePut("mesh-provisioner", "mesh-host.composed", "a.m1.f9", []byte(`{}`)); err == nil {
|
||||
t.Fatal("a per-node state is read only")
|
||||
}
|
||||
|
||||
var mu sync.Mutex
|
||||
var current, live []string
|
||||
stop, err := conn.StateWatch("mesh-provisioner", "mesh-host.composed", "", func(ch bus.StateChange) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if ch.Current {
|
||||
current = append(current, ch.Key)
|
||||
} else {
|
||||
live = append(live, ch.Key)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(stop)
|
||||
mu.Lock()
|
||||
sort.Strings(current)
|
||||
if !reflect.DeepEqual(current, []string{"a.m1.f1", "b.m2.f2"}) {
|
||||
t.Fatalf("current values: %v", current)
|
||||
}
|
||||
mu.Unlock()
|
||||
|
||||
// A third node's bucket, created during the watch, and a change in a first one.
|
||||
c := mesh.Bucket(t, "composed-c")
|
||||
if _, err := c.Put("c.m4.f4", []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := a.Put("a.m1.f5", []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mt.Until(t, func() error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
got := strings.Join(append(append([]string(nil), current...), live...), ",")
|
||||
for _, want := range []string{"a.m1.f5", "c.m4.f4"} {
|
||||
if !strings.Contains(got, want) {
|
||||
return fmt.Errorf("%s not heard yet: current %v, changes %v", want, current, live)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -32,7 +33,7 @@ func TestAWatchHoldsWhileItsCurrentValuesKeepArriving(t *testing.T) {
|
||||
for i := 0; i < older; i++ {
|
||||
keys = append(keys, fmt.Sprintf("novox/mesh-catalog.%d.%040x.status=mesh/check", 400+i, i))
|
||||
}
|
||||
for _, member := range []string{"novox/mesh-sdk.76.34f210e5", "novox/mesh-controller.453.c06f75d0", "novox/mesh-delivery.33.28ee22ef"} {
|
||||
for _, member := range []string{"novox/mesh-sdk.76.34f210e5", "novox/mesh-placement.453.c06f75d0", "novox/mesh-delivery.33.28ee22ef"} {
|
||||
keys = append(keys, member+strings.Repeat("0", 32)+".status=mesh/check")
|
||||
}
|
||||
for _, k := range keys {
|
||||
@@ -116,6 +117,50 @@ func TestAWatchIsRefusedWhenNoCurrentValueIsTakenWithinTheBound(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// **A membership the grants part wrote in its old wrapper is served whole** (issue 774). Until 2026-10-30 the part wrote
|
||||
// its pass record, the membership inside, into its state memberships; the tool runner read the value as a bare
|
||||
// membership, and served no tool and no state. This is what the live mesh held: read through mesh-sdk's codec, its tools
|
||||
// are answered where the membership says and its state is reached. A held-back one's wrapper is refused, and the module has
|
||||
// no membership then (774/S32/AC2: no stream is read in its place).
|
||||
func TestAMembershipTheGrantsPartWroteWrappedIsServed(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
mesh.Bucket(t, "gitea_repos")
|
||||
gitea := mt.MembershipOf("gitea", "anchor", false, nil)
|
||||
gitea.State = []bus.StateIssued{{Name: "repos", Bucket: "gitea_repos", Writes: true}}
|
||||
mesh.IssueInGrantsWrapped(t, gitea, "published")
|
||||
// The old wrapper of a membership the part held back carries its unpublished composition: refused, never served.
|
||||
heldBack := mt.MembershipOf("pages", "anchor", true, nil)
|
||||
mesh.IssueInGrantsWrapped(t, heldBack, "not published, a shadow difference: publishing would lose the last holder of x")
|
||||
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var logged []string
|
||||
conn.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
|
||||
t.Cleanup(conn.Close)
|
||||
conn.Follow("gitea")
|
||||
mt.Until(t, func() error {
|
||||
if m := conn.Membership("gitea"); m == nil || len(m.Serves) == 0 {
|
||||
return fmt.Errorf("gitea's membership is %+v, want the one inside the wrapper", m)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if got, want := fmt.Sprint(conn.ServedOn("gitea", "gitea_list")), "[{mesh.mod.gitea.tool.gitea_list.anchor }]"; got != want {
|
||||
t.Errorf("gitea_list is served on %s, want %s", got, want)
|
||||
}
|
||||
if _, err := conn.StatePut("gitea", "repos", "novox.hq", []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatalf("the state the membership issues is refused: %v", err)
|
||||
}
|
||||
if e, err := conn.StateGet("gitea", "repos", "novox.hq"); err != nil || e == nil || string(e.Value) != `{"v":1}` {
|
||||
t.Errorf("the state reads back %+v, %v", e, err)
|
||||
}
|
||||
conn.Follow("pages")
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
if m := conn.Membership("pages"); m != nil || !strings.Contains(strings.Join(logged, "\n"), "no membership issued for pages") {
|
||||
t.Errorf("a held-back membership's wrapper was served or not said: %+v, %q", m, logged)
|
||||
}
|
||||
}
|
||||
|
||||
// owedWatcher is the gitea module's view of the checker's owed: the bucket, and a runtime connection
|
||||
// that serves gitea with the owed issued for reading.
|
||||
func owedWatcher(t *testing.T) (*bus.Conn, interface {
|
||||
@@ -142,3 +187,72 @@ func owedWatcher(t *testing.T) (*bus.Conn, interface {
|
||||
})
|
||||
return conn, kv
|
||||
}
|
||||
|
||||
// **A watch given up, or stopped, hands over nothing more** (issue 774). The runtime's watch stopped its watcher
|
||||
// when the current values did not arrive within the bound, but the watcher still held values, and its goroutine
|
||||
// went on handing them over: to a watch id the bundle no longer held (on the provisioner, 51 failed watches were
|
||||
// followed by 1257 orphan changes). Each row blocks the bundle on its first value, gives the watch up, lets the
|
||||
// bundle go and counts what it is still handed.
|
||||
func TestAWatchGivenUpHandsOverNothingMore(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
refused bool // the bound gives it up, or the caller stops it
|
||||
}{
|
||||
{"refused at the bound", true},
|
||||
{"stopped by its caller", false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
was := bus.StateTimeout
|
||||
bus.StateTimeout = 300 * time.Millisecond
|
||||
t.Cleanup(func() { bus.StateTimeout = was })
|
||||
conn, kv := owedWatcher(t)
|
||||
put := func(i int) {
|
||||
if _, err := kv.Put(fmt.Sprintf("novox/hq.%d.%040x.status=mesh/check", i+1, i), []byte(`{"v":1}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if tc.refused {
|
||||
for i := 0; i < 4; i++ {
|
||||
put(i)
|
||||
}
|
||||
}
|
||||
var calls atomic.Int32
|
||||
first := make(chan struct{}, 1)
|
||||
release := make(chan struct{})
|
||||
deliver := func(ch bus.StateChange) error {
|
||||
if calls.Add(1) == 1 {
|
||||
first <- struct{}{}
|
||||
<-release
|
||||
}
|
||||
return nil
|
||||
}
|
||||
var stop func()
|
||||
if tc.refused {
|
||||
if _, err := conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", deliver); err == nil {
|
||||
t.Fatal("the watch whose bundle took nothing was not refused")
|
||||
}
|
||||
} else {
|
||||
var err error
|
||||
if stop, err = conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", deliver); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 4; i++ {
|
||||
put(i)
|
||||
}
|
||||
select {
|
||||
case <-first:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("the bundle was handed no change")
|
||||
}
|
||||
stop()
|
||||
}
|
||||
// Let the watcher's server side deliver what it held, then free the bundle.
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
close(release)
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
if n := calls.Load(); n != 1 {
|
||||
t.Fatalf("the bundle was handed %d changes of a watch given up, want 1 (the one in hand)", n)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
|
||||
// A module's own stream, as the runtime publishes to it and hears it (novox/hq ADR 0363 §8, issue 725).
|
||||
//
|
||||
// A module may declare a stream of its own; the controller creates it and says, in the module's
|
||||
// A module may declare a stream of its own; the mesh creates it and says, in the module's
|
||||
// membership, which of its events go there and on which subject (`streams`), and, in the membership of
|
||||
// each module granted to hear it, which durable consumer on it to bind (`hears`). The runtime publishes and
|
||||
// binds exactly as the membership says. A membership naming neither is what every membership was before:
|
||||
@@ -30,7 +30,7 @@ type StreamIssued struct {
|
||||
}
|
||||
|
||||
// HeardIssued is one module stream an assignment hears: the stream and the durable consumer on it the
|
||||
// runtime binds. The consumer is the controller's to make, to filter and to delete.
|
||||
// runtime binds. The consumer is the mesh's to make, to filter and to delete.
|
||||
type HeardIssued struct {
|
||||
Stream string `json:"stream"`
|
||||
Consumer string `json:"consumer"`
|
||||
@@ -73,7 +73,7 @@ func eventMatches(pattern, event string) bool {
|
||||
}
|
||||
|
||||
// RebindEvery is how often a consumer the membership names and the bus did not let the runtime bind is
|
||||
// tried again: the controller makes the consumer and sends the user list carrying its grant in the
|
||||
// tried again: the mesh makes the consumer and sends the user list carrying its grant in the
|
||||
// composition that issues the membership, so one may arrive before the other (ADR 0363 §6).
|
||||
var RebindEvery = 30 * time.Second
|
||||
|
||||
@@ -294,7 +294,7 @@ func (h *hearing) reconcile() {
|
||||
}
|
||||
}
|
||||
// **A loop the bus ended is no longer reading** (nats.go ends a pull loop on its own when its consumer is
|
||||
// deleted): the controller deletes and makes again under the same name a restricted consumer it cannot
|
||||
// deleted): the mesh deletes and makes again under the same name a restricted consumer it cannot
|
||||
// narrow (ADR 0363 §6), and the membership, unchanged, says nothing — so it is bound again here.
|
||||
// **A loop the bus did not end is asked about**: nats.go learns of a deleted consumer only from the answer
|
||||
// to a pull it has pending, and a consumer deleted and made again between two pulls, or while none is
|
||||
|
||||
@@ -160,6 +160,68 @@ func TestAModuleHearsEveryStreamItsMembershipNames(t *testing.T) {
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
// The membership as the grants part keeps it (ADR 0571 clause 7, 10): read first and applied live, and nothing
|
||||
// published on the controller's subject is heard.
|
||||
granted := reader
|
||||
granted.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "from-grants"}}
|
||||
mesh.Issue(t, granted)
|
||||
again, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again.Logf = func(string, ...any) {}
|
||||
t.Cleanup(again.Close)
|
||||
again.Follow("reader")
|
||||
consumerOf := func() string {
|
||||
if m := again.Membership("reader"); m != nil && len(m.Hears) == 1 {
|
||||
return m.Hears[0].Consumer
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if got := consumerOf(); got != "from-grants" {
|
||||
t.Fatalf("the membership read at the follow names %q, want the grants part's", got)
|
||||
}
|
||||
granted.Hears[0].Consumer = "from-grants-2"
|
||||
mesh.Issue(t, granted)
|
||||
mt.Until(t, func() error {
|
||||
if got := consumerOf(); got != "from-grants-2" {
|
||||
return fmt.Errorf("the membership names %q, want the grants part's change", got)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
// 774/S32/AC1: a copy said on the controller's subject (mesh.assignment) changes nothing; there is no stream to read it from.
|
||||
streamed := reader
|
||||
streamed.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "from-subject"}}
|
||||
mesh.Publish(t, "mesh.assignment.anchor.reader", streamed)
|
||||
again.Flush()
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
if got := consumerOf(); got != "from-grants-2" {
|
||||
t.Fatalf("a membership said on the controller's subject was applied: the membership names %q", got)
|
||||
}
|
||||
// Review g1 round 1: memberships that cannot be read at the follow (a bus not ready after its boot) are read again until
|
||||
// they can be; the follow, which runs once per module, does not give the module up for good.
|
||||
was := bus.GrantsRetry
|
||||
bus.GrantsRetry = 50 * time.Millisecond
|
||||
t.Cleanup(func() { bus.GrantsRetry = was })
|
||||
mesh.DropGrants(t)
|
||||
late, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
late.Logf = func(string, ...any) {}
|
||||
t.Cleanup(late.Close)
|
||||
late.Follow("reader")
|
||||
if m := late.Membership("reader"); m != nil {
|
||||
t.Fatalf("a membership was served from a state not there: %+v", m)
|
||||
}
|
||||
mesh.Issue(t, granted)
|
||||
mt.Until(t, func() error {
|
||||
if m := late.Membership("reader"); m == nil || len(m.Hears) != 1 || m.Hears[0].Consumer != "from-grants-2" {
|
||||
return fmt.Errorf("the membership is %+v, want the one written once the state is there", m)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// novox/hq ADR 0363 §6 (issue 725): a consumer the membership names before the controller made it is bound once
|
||||
|
||||
@@ -17,6 +17,9 @@ package console
|
||||
// session — so a tool that arrived a minute ago is found without the client reconnecting.
|
||||
|
||||
import (
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
@@ -47,7 +50,7 @@ const (
|
||||
// searchCap is how many matches a search answers before it says how many more there were.
|
||||
const searchCap = 25
|
||||
|
||||
const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e.g. `mesh-controller.nodes`); " +
|
||||
const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e.g. `mesh-placement.placement`); " +
|
||||
"`<node>/<seat>.<verb>` for a seat every machine holds (e.g. `ace/node-packet-filter.rules`); " +
|
||||
"`<node>/<module>.<tool>` for a module on one machine (e.g. `novox/postgres.postgres_list_databases`); " +
|
||||
"and `<module>.<tool>` also for a module whose instances are interchangeable."
|
||||
@@ -128,13 +131,17 @@ type index struct {
|
||||
NotAnswering []string
|
||||
Listing *Listing // the flat catalogue the call path resolves subjects with
|
||||
Discovery announce.Discovery
|
||||
// Recorded is where the controller's records place each module that declares tools.
|
||||
// Recorded is where placement places each module that declares tools.
|
||||
Recorded map[string][]string
|
||||
// Unheard is every runtime whose answer this index lacks: it said it was there and did not say
|
||||
// what it serves in time, or it answered before and not now. An address it might answer is never
|
||||
// called missing while it is here (2026-10-05).
|
||||
Unheard []unheard
|
||||
// Replaces is what each verb and tool replaces, from the controller's records (novox/hq ADR 0245):
|
||||
// Placed is the state seat-holders as the seat mesh-placement's holder wrote it (novox/hq ADR 0571 clause 4), by its
|
||||
// keys `<seat>` and `<seat>.<node>`, read with delivery.HoldersOf; PlacedWhy says why it could not be read.
|
||||
Placed map[string][]byte
|
||||
PlacedWhy string
|
||||
// Replaces is what each verb and tool replaces (novox/hq ADR 0245):
|
||||
// `seat:<seat>.<verb>` and `<module>.<tool>`.
|
||||
Replaces map[string][]string
|
||||
// Earlier is every runtime not heard this round whose last answer, within RuntimeForgotten, this
|
||||
@@ -224,42 +231,8 @@ func findTool(tools []Tool, name string) *Tool {
|
||||
return nil
|
||||
}
|
||||
|
||||
// controllerOutput is a controller seat verb's answer: the command's printed output.
|
||||
func controllerOutput(conn *bus.Conn, verb string) (string, error) {
|
||||
got, err := conn.Ask("seat:mesh-controller."+verb, map[string]any{}, "")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var r struct {
|
||||
Output string `json:"output"`
|
||||
OK *bool `json:"ok"`
|
||||
}
|
||||
if json.Unmarshal(got.Result, &r) != nil {
|
||||
return "", fmt.Errorf("mesh-controller.%s answered something that is not its output", verb)
|
||||
}
|
||||
if r.OK != nil && !*r.OK {
|
||||
return "", fmt.Errorf("mesh-controller.%s: %s", verb, strings.TrimSpace(r.Output))
|
||||
}
|
||||
return r.Output, nil
|
||||
}
|
||||
|
||||
// jsonIn is the JSON document a command printed, after any lines it said first: a seat verb runs
|
||||
// the controller's command, and a command may warn before it answers.
|
||||
func jsonIn(output string) string {
|
||||
t := strings.TrimSpace(output)
|
||||
if strings.HasPrefix(t, "{") || strings.HasPrefix(t, "[") {
|
||||
return t
|
||||
}
|
||||
for _, open := range []string{"\n{", "\n["} {
|
||||
if i := strings.Index(output, open); i >= 0 {
|
||||
return strings.TrimSpace(output[i+1:])
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// recordedModule is a module as the controller's records hold it (`module list --json`): where the
|
||||
// mesh assigned it, and whether it declares tools — what should announce itself, and where.
|
||||
// recordedModule is a module as the bus holds it: where the placement seat places it, and whether its manifest declares
|
||||
// tools — what should announce itself, and where (novox/hq ADR 0571).
|
||||
type recordedModule struct {
|
||||
Module string `json:"module"`
|
||||
On []string `json:"on"`
|
||||
@@ -268,13 +241,8 @@ type recordedModule struct {
|
||||
Replaces map[string][]string `json:"replaces,omitempty"`
|
||||
}
|
||||
|
||||
// recordedMachine is a machine as the controller's records hold it (`node list --json`).
|
||||
type recordedMachine struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// indexOn asks the mesh what it holds (novox/hq ADR 0197): what answers, from every runtime's own
|
||||
// announcement on the bus — one `$SRV.INFO` request — and what should, from the controller's records
|
||||
// announcement on the bus — one `$SRV.INFO` request — and what should, from the indexer's records
|
||||
// read as JSON. Nothing is inferred from a roster and nothing is parsed from print.
|
||||
//
|
||||
// `held` is what the console holds already, so an instance whose announcement is unchanged is not
|
||||
@@ -282,19 +250,17 @@ type recordedMachine struct {
|
||||
// index holds instead, marked with its age (novox/hq issue 554).
|
||||
func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery) []earlier) (*index, error) {
|
||||
var wg sync.WaitGroup
|
||||
var nodesOut, modulesOut string
|
||||
var toolsAnswer json.RawMessage
|
||||
wg.Add(3)
|
||||
go func() { defer wg.Done(); nodesOut, _ = controllerOutput(conn, "nodes") }()
|
||||
go func() { defer wg.Done(); modulesOut, _ = controllerOutput(conn, "modules") }()
|
||||
// The seats' verbs as the records define them, for what each replaces (novox/hq ADR 0245): answered
|
||||
// in-process from the records, never a command.
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if got, err := conn.Ask("seat:mesh-controller.tools", map[string]any{}, ""); err == nil {
|
||||
toolsAnswer = got.Result
|
||||
}
|
||||
}()
|
||||
var placed map[string][]byte
|
||||
var placedErr error
|
||||
var recorded []recordedModule
|
||||
var known []string
|
||||
var recordsWhy []string
|
||||
wg.Add(2)
|
||||
// Which modules there are and where they are placed, and which machines the mesh has, from the bus (ADR 0571
|
||||
// clauses 4 and 5): the indexer's modules and manifests and the placement seat's placements, never any other record.
|
||||
go func() { defer wg.Done(); recorded, known, recordsWhy = recordsOn(conn) }()
|
||||
// Who holds which seat on which node, from placement (ADR 0571 clause 4): no other call.
|
||||
go func() { defer wg.Done(); placed, placedErr = seatHoldersOn(conn) }()
|
||||
d, err := announce.Gather(conn, held)
|
||||
wg.Wait()
|
||||
if err != nil {
|
||||
@@ -333,6 +299,10 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
|
||||
l := x.Listing
|
||||
x.Discovery = d
|
||||
x.Stale, x.Earlier = stale, earlierAt
|
||||
x.Placed = placed
|
||||
if placedErr != nil {
|
||||
x.PlacedWhy = placedErr.Error()
|
||||
}
|
||||
for _, s := range d.Silent {
|
||||
x.Unheard = append(x.Unheard, unheard{Runtime: s.Name, Machine: s.Machine,
|
||||
Why: fmt.Sprintf("it answered PING; what it serves did not arrive within %s", announce.Patience)})
|
||||
@@ -340,25 +310,24 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
|
||||
|
||||
// What should have answered: every assignment of a module that declares tools. Silence is named;
|
||||
// a module with no tools is never a name here.
|
||||
var recorded []recordedModule
|
||||
recordedOK := json.Unmarshal([]byte(jsonIn(modulesOut)), &recorded) == nil
|
||||
x.Replaces = replacesOf(toolsAnswer, recorded)
|
||||
if recordedOK {
|
||||
for _, m := range recorded {
|
||||
if !m.Tools {
|
||||
continue
|
||||
}
|
||||
x.Recorded[m.Module] = append([]string{}, m.On...)
|
||||
for _, n := range m.On {
|
||||
// A holder of a seat held once for the mesh announces no machine — the seat is the mesh's,
|
||||
// not a machine's — so what it announced without one answers for wherever it is assigned.
|
||||
if !announced[m.Module][n] && !announced[m.Module][""] {
|
||||
l.NotAnswering = append(l.NotAnswering, m.Module+" on "+n)
|
||||
}
|
||||
x.Replaces = replacesOf(catalogue.DefaultSeats(), recorded)
|
||||
for _, m := range recorded {
|
||||
if !m.Tools {
|
||||
continue
|
||||
}
|
||||
x.Recorded[m.Module] = append([]string{}, m.On...)
|
||||
for _, n := range m.On {
|
||||
// A holder of a seat held once for the mesh announces no machine — the seat is the mesh's,
|
||||
// not a machine's — so what it announced without one answers for wherever it is assigned.
|
||||
if !announced[m.Module][n] && !announced[m.Module][""] {
|
||||
l.NotAnswering = append(l.NotAnswering, m.Module+" on "+n)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
l.NotAnswering = append(l.NotAnswering, "mesh-controller (its records of the modules did not answer, so what is missing cannot be said)")
|
||||
}
|
||||
l.NotAnswering = append(l.NotAnswering, recordsWhy...)
|
||||
if x.PlacedWhy != "" {
|
||||
l.NotAnswering = append(l.NotAnswering, delivery.SeatHoldersRead+" (who holds which seat by placement could not be read: "+
|
||||
x.PlacedWhy+")")
|
||||
}
|
||||
for _, u := range x.Unheard {
|
||||
l.NotAnswering = append(l.NotAnswering, "the runtime "+sayUnheard([]unheard{u}))
|
||||
@@ -366,13 +335,8 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
|
||||
sort.Strings(l.NotAnswering)
|
||||
x.NotAnswering = l.NotAnswering
|
||||
|
||||
var known []recordedMachine
|
||||
if json.Unmarshal([]byte(jsonIn(nodesOut)), &known) == nil {
|
||||
for _, n := range known {
|
||||
if n.Name != "" {
|
||||
machines[n.Name] = true
|
||||
}
|
||||
}
|
||||
for _, n := range known {
|
||||
machines[n] = true
|
||||
}
|
||||
for n := range machines {
|
||||
x.Machines = append(x.Machines, n)
|
||||
@@ -381,6 +345,138 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
|
||||
return x, nil
|
||||
}
|
||||
|
||||
// stateOn reads a state the runtime's membership reads, whole: its current values by key, a deleted key absent.
|
||||
func stateOn(conn *bus.Conn, name string) (map[string][]byte, error) {
|
||||
var mu sync.Mutex
|
||||
keys := map[string][]byte{}
|
||||
stop, err := conn.StateWatch(conn.Credential().Module, name, "", func(c bus.StateChange) error {
|
||||
if c.Current && c.Op == "put" {
|
||||
mu.Lock()
|
||||
keys[c.Key] = c.Value
|
||||
mu.Unlock()
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
stop()
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return keys, nil
|
||||
}
|
||||
|
||||
// seatHoldersOn reads the state seat-holders whole (novox/hq ADR 0571 clause 4), as delivery.SeatHoldersRead.
|
||||
func seatHoldersOn(conn *bus.Conn) (map[string][]byte, error) {
|
||||
return stateOn(conn, delivery.SeatHoldersRead)
|
||||
}
|
||||
|
||||
// The indexer's states the console reads (novox/hq ADR 0571 clauses 4 and 5): every module, and each manifest by its
|
||||
// digest, by the names a reader's manifest declares.
|
||||
var (
|
||||
modulesRead = delivery.IndexerModule + ".modules"
|
||||
manifestsRead = delivery.IndexerModule + ".manifests"
|
||||
)
|
||||
|
||||
// manifestFacts is what the console takes of a manifest: the tools it declares and what each replaces. A manifest's
|
||||
// digest never names another, so what is read once is kept.
|
||||
type manifestFacts struct {
|
||||
Tools []string `json:"tools"`
|
||||
Replaces map[string][]string `json:"replaces"`
|
||||
}
|
||||
|
||||
var manifestsKept sync.Map // digest → manifestFacts
|
||||
|
||||
// recordsOn is every module the indexer holds, with where the placement seat places it and whether it declares tools,
|
||||
// and every machine placement names (delivery.ModuleNames, delivery.MachineNames). What cannot be read is said, one line
|
||||
// per state, for the listing's not answering.
|
||||
func recordsOn(conn *bus.Conn) (recorded []recordedModule, machines []string, why []string) {
|
||||
placements, err := stateOn(conn, delivery.PlacementRead)
|
||||
if err != nil {
|
||||
why = append(why, delivery.PlacementRead+" (where modules are placed could not be read, so what is missing cannot "+
|
||||
"be said: "+err.Error()+")")
|
||||
}
|
||||
machines = delivery.MachineNames(placements)
|
||||
on := map[string][]string{}
|
||||
// The placement's keys in either form, a node key and its module keys or the form before (research 072 row 37).
|
||||
assembled, err := delivery.PlacementsOf(placements)
|
||||
if err != nil {
|
||||
why = append(why, delivery.PlacementRead+" (not every key is a placement: "+err.Error()+")")
|
||||
}
|
||||
for _, node := range machines {
|
||||
for _, m := range assembled[node].Modules {
|
||||
on[m.Module] = append(on[m.Module], node)
|
||||
}
|
||||
}
|
||||
modules, err := stateOn(conn, modulesRead)
|
||||
if err != nil {
|
||||
return nil, machines, append(why, modulesRead+" (which modules there are could not be read: "+err.Error()+")")
|
||||
}
|
||||
names, err := delivery.ModuleNames(modules)
|
||||
if err != nil {
|
||||
return nil, machines, append(why, modulesRead+" ("+err.Error()+")")
|
||||
}
|
||||
for _, name := range names {
|
||||
r := recordedModule{Module: name, On: on[name]}
|
||||
if m, err := delivery.DecodeModule(modules[name]); err == nil && m.Manifest != "" {
|
||||
facts, err := manifestOf(conn, m.Manifest)
|
||||
if err != nil {
|
||||
why = append(why, name+" (its manifest could not be read, so whether it answers tools is not known: "+err.Error()+")")
|
||||
} else {
|
||||
r.Tools, r.Replaces = len(facts.Tools) > 0, facts.Replaces
|
||||
}
|
||||
}
|
||||
recorded = append(recorded, r)
|
||||
}
|
||||
return recorded, machines, why
|
||||
}
|
||||
|
||||
// manifestOf is what the console takes of the manifest of a digest, read from the indexer's manifests once.
|
||||
func manifestOf(conn *bus.Conn, digest string) (manifestFacts, error) {
|
||||
if kept, ok := manifestsKept.Load(digest); ok {
|
||||
return kept.(manifestFacts), nil
|
||||
}
|
||||
entry, err := conn.StateGet(conn.Credential().Module, manifestsRead, digest)
|
||||
if err != nil {
|
||||
return manifestFacts{}, err
|
||||
}
|
||||
if entry == nil {
|
||||
return manifestFacts{}, fmt.Errorf("the indexer holds no manifest %s", digest)
|
||||
}
|
||||
m, err := delivery.DecodeManifest(entry.Value)
|
||||
if err != nil {
|
||||
return manifestFacts{}, err
|
||||
}
|
||||
var facts manifestFacts
|
||||
if err := json.Unmarshal([]byte(m.JSON), &facts); err != nil {
|
||||
return manifestFacts{}, err
|
||||
}
|
||||
manifestsKept.Store(digest, facts)
|
||||
return facts, nil
|
||||
}
|
||||
|
||||
// placedHolders is who holds seat (on node, for a seat held per machine) by placement, or none.
|
||||
func (x *index) placedHolders(seat, node string) []delivery.SeatHolder {
|
||||
if len(x.Placed) == 0 {
|
||||
return nil
|
||||
}
|
||||
hs, err := delivery.HoldersOf(x.Placed, seat, node)
|
||||
if err != nil || len(hs) > 0 || node != "" {
|
||||
return hs
|
||||
}
|
||||
// A seat held per machine, asked without one: every node's key of it.
|
||||
var all []delivery.SeatHolder
|
||||
for key := range x.Placed {
|
||||
if s, n, ok := strings.Cut(key, "."); ok && s == seat {
|
||||
if more, err := delivery.HoldersOf(x.Placed, seat, n); err == nil {
|
||||
all = append(all, more...)
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Slice(all, func(i, j int) bool { return all[i].Node+"/"+all[i].Module < all[j].Node+"/"+all[j].Module })
|
||||
return all
|
||||
}
|
||||
|
||||
func containsHolder(hs []holder, h holder) bool {
|
||||
for _, x := range hs {
|
||||
if x == h {
|
||||
@@ -606,6 +702,14 @@ func resolve(x *index, address string) (target, error) {
|
||||
return target{}, fmt.Errorf("%s is assigned to %s in the mesh's records, but nothing that answered discovery serves it: %s. "+
|
||||
"Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, strings.Join(on, ", "), why)
|
||||
}
|
||||
if hs := x.placedHolders(prefix, node); len(hs) > 0 {
|
||||
var held []string
|
||||
for _, h := range hs {
|
||||
held = append(held, h.Module+" on "+h.Node)
|
||||
}
|
||||
return target{}, fmt.Errorf("the seat %s is held by %s by placement, but no holder of it announced itself on the bus. "+
|
||||
"Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, strings.Join(held, ", "))
|
||||
}
|
||||
if len(x.Unheard) > 0 {
|
||||
return target{}, fmt.Errorf("nothing that answered discovery is called %s, but not every runtime answered: %s — "+
|
||||
"%s may be theirs. Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, sayUnheard(x.Unheard), prefix)
|
||||
@@ -711,7 +815,7 @@ func firstLine(s string) string {
|
||||
// the machine, `node` is taken out: the machine is in the address. Where it does not — a seat held
|
||||
// once for the mesh, or an interchangeable module — `node` is the tool's own argument, and is kept.
|
||||
//
|
||||
// **Taking it out everywhere made the mesh's own verbs lie** (novox/hq issue 244): the controller's
|
||||
// **Taking it out everywhere made the mesh's own verbs lie** (novox/hq issue 244): the old controller verbs
|
||||
// `push`, `plan`, `assign`, `pin` and `settings` take the machine they act on as `node`, and were
|
||||
// described without it — `push` as taking nothing at all, so a push naming one machine arrived as a
|
||||
// push of every machine behind.
|
||||
|
||||
@@ -70,35 +70,34 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
t.Cleanup(stop)
|
||||
}
|
||||
|
||||
// The controller: its records as JSON, as its seat verbs answer them, and its own seat's verbs
|
||||
// announced on the bus like every runtime's.
|
||||
controller := connect(t, "mesh-controller", "bench")
|
||||
out := func(s string) map[string]any { return map[string]any{"output": s, "ok": true} }
|
||||
serve := func(verb string, answer func() any) {
|
||||
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
|
||||
return answer(), nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(stop)
|
||||
// The controller: its own seat's verbs announced on the bus like every runtime's; its records are asked by no one.
|
||||
controller := connect(t, "mesh-placement", "bench")
|
||||
// Its verb nodes, called through the console as any seat's verb is.
|
||||
stopNodes, err := controller.HandleSubject("mesh.seat.mesh-placement.tool.nodes", func(json.RawMessage) (any, error) {
|
||||
return map[string]any{"ok": true, "output": `[{"name":"bench","mode":"converged"},{"name":"desk","mode":"converged"}]`}, nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
serve("nodes", func() any {
|
||||
return out("the bus's user list leaves out 2 user(s)\n" +
|
||||
`[{"name":"bench","heard":"3m ago","mode":"converged","id":"1f2e"},{"name":"desk","heard":"here","mode":"converged","id":"9a8b"}]`)
|
||||
t.Cleanup(stopNodes)
|
||||
// What the mesh holds, as the indexer and placement write it (ADR 0571 clauses 4, 5): never the controller's records.
|
||||
tools := map[string][]string{"alpha": {"one", "two"}, "beta": {"three"}, "gamma": {"given"}, "delta": nil,
|
||||
"epsilon": {"seven"}, "mesh-placement": {"nodes"}}
|
||||
mesh.Placed(t, tools, map[string][]string{"desk": {"alpha", "beta", "delta"}, "bench": {"epsilon", "mesh-placement"}})
|
||||
console := mt.MembershipOf("node-tools", "desk", false, nil)
|
||||
console.State = mt.ConsoleReads()
|
||||
mesh.Issue(t, console)
|
||||
mt.Until(t, func() error {
|
||||
if m := nodeTools.Membership("node-tools"); m == nil || len(m.State) == 0 {
|
||||
return fmt.Errorf("the console's reads are not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
var gammaOn atomic.Value
|
||||
gammaOn.Store("[]")
|
||||
serve("modules", func() any {
|
||||
return out(fmt.Sprintf(`[{"module":"alpha","on":["desk"],"tools":true},{"module":"beta","on":["desk"],"tools":true},`+
|
||||
`{"module":"gamma","on":%s,"tools":true},{"module":"delta","on":["desk"],"tools":false},`+
|
||||
`{"module":"epsilon","on":["bench"],"tools":true},{"module":"mesh-controller","on":["bench"],"tools":true}]`, gammaOn.Load().(string)))
|
||||
})
|
||||
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-controller", ID: "bench"}, func() []announce.Endpoint {
|
||||
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller", Scope: "mesh",
|
||||
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-placement", ID: "bench"}, func() []announce.Endpoint {
|
||||
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-placement", Seat: "mesh-placement", Scope: "mesh",
|
||||
// As the live controller announces: a mesh seat's holder names no machine.
|
||||
Tool: "nodes", Node: "", Description: "Every machine the mesh knows.", Schema: json.RawMessage(`{}`),
|
||||
Subject: "mesh.seat.mesh-controller.tool.nodes"}}
|
||||
Subject: "mesh.seat.mesh-placement.tool.nodes"}}
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -132,13 +131,13 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
|
||||
// The overview names the mesh's seats, the machines' seats and the machines.
|
||||
overview, isErr := call(t, endpoint, "mesh_overview", nil)
|
||||
if isErr || !strings.Contains(overview, "mesh-controller.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
|
||||
if isErr || !strings.Contains(overview, "mesh-placement.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
|
||||
!strings.Contains(overview, `"bench"`) || !strings.Contains(overview, `"desk"`) {
|
||||
t.Errorf("overview: %s", overview)
|
||||
}
|
||||
// Silence is named only where tools should have answered: epsilon declares tools on bench and
|
||||
// nothing there announced it; delta declares none and is never a name.
|
||||
if strings.Contains(overview, "mesh-controller on bench") {
|
||||
if strings.Contains(overview, "mesh-placement on bench") {
|
||||
t.Errorf("a mesh seat's holder that announced no machine is called silent on its machine:\n%s", overview)
|
||||
}
|
||||
if !strings.Contains(overview, "epsilon on bench") || strings.Contains(overview, "delta") {
|
||||
@@ -151,7 +150,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
}
|
||||
|
||||
// A mesh seat, a node seat, an assignment and an interchangeable module, each by address.
|
||||
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-controller.nodes"}); isErr || !strings.Contains(got, "converged") {
|
||||
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-placement.nodes"}); isErr || !strings.Contains(got, "converged") {
|
||||
t.Errorf("mesh seat: %s", got)
|
||||
}
|
||||
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "desk/node-shelf.list"}); isErr || !strings.Contains(got, `"a"`) {
|
||||
@@ -167,11 +166,11 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
|
||||
// Refused, by name, where the address does not say enough or says the wrong thing.
|
||||
for address, want := range map[string]string{
|
||||
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
|
||||
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
|
||||
"desk/mesh-controller.nodes": "held once for the whole mesh",
|
||||
"bench/beta.three": "beta does not run on bench; it runs on desk",
|
||||
"nonsense": "is not an address",
|
||||
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
|
||||
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
|
||||
"desk/mesh-placement.nodes": "held once for the whole mesh",
|
||||
"bench/beta.three": "beta does not run on bench; it runs on desk",
|
||||
"nonsense": "is not an address",
|
||||
} {
|
||||
got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": address})
|
||||
if !isErr || !strings.Contains(got, want) {
|
||||
@@ -202,7 +201,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
t.Fatalf("gamma was found before it served: %s", got)
|
||||
}
|
||||
mesh.Issue(t, mt.MembershipOf("gamma", "desk", false, nil))
|
||||
gammaOn.Store(`["desk"]`)
|
||||
mesh.Placed(t, tools, map[string][]string{"desk": {"alpha", "beta", "delta", "gamma"}, "bench": {"epsilon", "mesh-placement"}})
|
||||
late := connect(t, "node-tools", "desk")
|
||||
stopLate, err := runtime.Run(late, []runtime.Served{{Module: "gamma", Entrypoints: []string{mt.Fixture("env-gamma.serve.mjs")}}},
|
||||
nil, (&mt.Logs{}).Logf)
|
||||
@@ -233,12 +232,3 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
|
||||
t.Errorf("an old name: %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheControllersRecordsAreReadAsJSON(t *testing.T) {
|
||||
if got := jsonIn("a warning printed first\n[{\"name\":\"ace\"}]"); got != `[{"name":"ace"}]` {
|
||||
t.Errorf("an array after a warning: %q", got)
|
||||
}
|
||||
if got := jsonIn(`{"seats":[]}`); got != `{"seats":[]}` {
|
||||
t.Errorf("an object: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -118,9 +118,6 @@ var (
|
||||
timedOut = regexp.MustCompile(`(?i)timeout`)
|
||||
)
|
||||
|
||||
// controllerSeat is the seat whose holder keeps what came of every call (novox/hq issue 265).
|
||||
const controllerSeat = "mesh-controller"
|
||||
|
||||
// whyItFailed says why a call failed, so the remedy is in the words.
|
||||
func whyItFailed(key string, err error) string {
|
||||
if err == nil {
|
||||
@@ -145,12 +142,6 @@ func whyItFailed(key string, err error) string {
|
||||
said := key + " did not answer within " + bus.RequestTimeout.String() + ". That is not a failure: " +
|
||||
"something is serving it, so it may still be running and may already have done what was " +
|
||||
"asked — check before repeating a call that changes the mesh."
|
||||
if strings.HasPrefix(strings.TrimPrefix(key, "seat:"), controllerSeat+".") {
|
||||
// The controller answers every call within seconds, with an id while it runs, so silence
|
||||
// from it is an answer lost on the way — which it keeps.
|
||||
said += " The controller answers every call within seconds, so its answer was lost on the " +
|
||||
"way; " + controllerSeat + ".calls lists its recent calls and what came of each."
|
||||
}
|
||||
return said
|
||||
}
|
||||
return key + " failed: " + msg
|
||||
|
||||
@@ -3,6 +3,7 @@ package console
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
@@ -56,14 +57,19 @@ func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
|
||||
}
|
||||
defer stop()
|
||||
|
||||
// The controller's records (ADR 0197): ghost declares tools on desk and nothing announces it.
|
||||
controller := connect(t, "mesh-controller", "")
|
||||
stopSeat, _ := controller.HandleSubject("mesh.seat.mesh-controller.tool.modules", func(json.RawMessage) (any, error) {
|
||||
return map[string]any{"ok": true, "output": `[{"module":"alpha","on":["desk"],"tools":true},` +
|
||||
`{"module":"beta","on":["desk"],"tools":true},{"module":"ghost","on":["desk"],"tools":true}]`}, nil
|
||||
// What the mesh holds, as the indexer and placement write it (ADR 0571): ghost declares tools on desk and nothing
|
||||
// announces it.
|
||||
mesh.Placed(t, map[string][]string{"alpha": {"one", "two"}, "beta": {"three"}, "ghost": {"boo"}},
|
||||
map[string][]string{"desk": {"alpha", "beta", "ghost"}})
|
||||
console := mt.MembershipOf("node-tools", "desk", false, nil)
|
||||
console.State = mt.ConsoleReads()
|
||||
mesh.Issue(t, console)
|
||||
mt.Until(t, func() error {
|
||||
if m := nodeTools.Membership("node-tools"); m == nil || len(m.State) == 0 {
|
||||
return fmt.Errorf("the console's reads are not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
defer stopSeat()
|
||||
controller.Flush()
|
||||
|
||||
flat := NewSurface(nodeTools, "desk.node-tools")
|
||||
flat.Flat = true // the whole catalogue, as before ADR 0195: still reachable, no longer announced
|
||||
|
||||
@@ -26,7 +26,7 @@ func TestTheManifestNamesTheFirstFamiliesOneByOne(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{"mesh-knowledge.reviewed", "mesh-delivery-gate.transition", "gitea.forge.status",
|
||||
"mesh-controller.checked", "operator-channel.decided.claude-code"}
|
||||
"mesh-self-check.checked", "operator-channel.decided.claude-code"}
|
||||
if m.Module != "node-tools" || !reflect.DeepEqual(m.Waits, want) {
|
||||
t.Errorf("%s waits on %v, want %v", m.Module, m.Waits, want)
|
||||
}
|
||||
|
||||
@@ -6,6 +6,8 @@ package console
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -60,6 +62,34 @@ type Surface struct {
|
||||
Flat bool
|
||||
}
|
||||
|
||||
// MethodSeats is the console's JSON-RPC method beside MCP's that answers every seat's verbs and holders, in the shape
|
||||
// the old `tools` answered: `{seats: [{seat, scope, tools: [{name, description, input, replaces}], holders:
|
||||
// [{module, node}]}], holders_why}`.
|
||||
const MethodSeats = "mesh/seats"
|
||||
|
||||
// seatsAnswer is MethodSeats' answer: mesh-sdk's seat definitions, with each seat's holders by placement.
|
||||
func (s *Surface) seatsAnswer() map[string]any {
|
||||
placed, err := seatHoldersOn(s.conn)
|
||||
x := &index{Placed: placed}
|
||||
seats := []map[string]any{}
|
||||
for _, st := range catalogue.DefaultSeats() {
|
||||
holders := x.placedHolders(st.Name, "")
|
||||
if holders == nil {
|
||||
holders = []delivery.SeatHolder{}
|
||||
}
|
||||
tools := st.Serves
|
||||
if tools == nil {
|
||||
tools = []catalogue.Verb{}
|
||||
}
|
||||
seats = append(seats, map[string]any{"seat": st.Name, "scope": st.Scope, "tools": tools, "holders": holders})
|
||||
}
|
||||
out := map[string]any{"seats": seats}
|
||||
if err != nil {
|
||||
out["holders_why"] = "who holds which seat by placement could not be read: " + err.Error()
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// NewSurface is the surface over a connection, as `who`.
|
||||
func NewSurface(conn *bus.Conn, who string) *Surface {
|
||||
return &Surface{conn: conn, who: who, Flat: os.Getenv("MESH_CONSOLE_FLAT") == "1"}
|
||||
@@ -125,6 +155,11 @@ func (s *Surface) HandleContext(ctx context.Context, r Request) *Reply {
|
||||
return nil
|
||||
}
|
||||
return answer(r.ID, map[string]any{})
|
||||
case MethodSeats:
|
||||
// The seats' contract and who holds each, for a client on this machine that has no seat definitions of its own
|
||||
// (the TypeScript `mesh`): read in-process from mesh-sdk and from placement, never asked of anything else
|
||||
// (novox/hq ADR 0571 clauses 4 and 5).
|
||||
return answer(r.ID, s.seatsAnswer())
|
||||
case "tools/list":
|
||||
if !s.Flat {
|
||||
return answer(r.ID, map[string]any{"tools": discovery()})
|
||||
@@ -303,7 +338,7 @@ func (s *Surface) instructions() string {
|
||||
"that serves it; what may be called was fixed when this account was issued, so a " +
|
||||
"refusal means the account, not the tool. The list is what the running modules " +
|
||||
"answered, plus every role's tools from the mesh's records — the mesh's own verbs " +
|
||||
"(mesh-controller.status, .push, .assign …) among them; a module that did not answer " +
|
||||
"(mesh-delivery.deliveries, mesh-placement.propose …) among them; a module that did not answer " +
|
||||
"is named in the list's _meta and can still be called by <module>.<tool>."
|
||||
}
|
||||
return "The tools of a Novox mesh, reached as " + s.who + ", found by address rather than listed " +
|
||||
|
||||
@@ -11,9 +11,9 @@ import (
|
||||
// 2026-10-05 the controller's push was described as taking nothing, and a push naming one machine
|
||||
// ran as a push of every machine behind.
|
||||
func TestAMeshSeatsNodeIsItsOwnArgument(t *testing.T) {
|
||||
push := Tool{Module: "mesh-controller", Name: "push", Seat: true, Scope: "mesh",
|
||||
push := Tool{Module: "mesh-placement", Name: "push", Seat: true, Scope: "mesh",
|
||||
Input: json.RawMessage(`{"type":"object","properties":{"node":{"type":"string"},"behind":{"type":"string"}}}`)}
|
||||
mesh := target{Address: "mesh-controller.push", Tool: push, Seat: true}
|
||||
mesh := target{Address: "mesh-placement.push", Tool: push, Seat: true}
|
||||
|
||||
props := schemaAsPassed(mesh)["properties"].(map[string]any)
|
||||
if _, has := props["node"]; !has {
|
||||
|
||||
@@ -22,16 +22,23 @@ func TestARefusalIsSaidOnlyForTheBussOwn(t *testing.T) {
|
||||
}
|
||||
|
||||
// **A timeout says it is not a failure** (novox/hq issue 265): a push that outlasted the wait had
|
||||
// pushed, and the caller read "did not answer in time". And from the controller, where its answer is.
|
||||
// pushed, and the caller read "did not answer in time". No hint names a seat that is gone (novox/hq issue 1126):
|
||||
// the grammar's example, the handshake's instructions and a timeout's remedy named the controller's verbs.
|
||||
func TestATimeoutIsNotAFailure(t *testing.T) {
|
||||
got := whyItFailed("seat:mesh-controller.push", errors.New("timeout"))
|
||||
for _, want := range []string{"not a failure", "may already have done", "mesh-controller.calls"} {
|
||||
got := whyItFailed("seat:mesh-delivery.deliveries", errors.New("timeout"))
|
||||
for _, want := range []string{"not a failure", "may already have done"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("a controller timeout does not say %q: %s", want, got)
|
||||
t.Errorf("a seat's timeout does not say %q: %s", want, got)
|
||||
}
|
||||
}
|
||||
if other := whyItFailed("novox/postgres.postgres_list_databases", errors.New("timeout")); !strings.Contains(other, "not a failure") ||
|
||||
strings.Contains(other, "calls") {
|
||||
t.Errorf("a module's timeout: %s", other)
|
||||
for name, text := range map[string]string{
|
||||
"grammar": grammar,
|
||||
"flat instructions": (&Surface{Flat: true, who: "x"}).instructions(),
|
||||
"deep instructions": (&Surface{who: "x"}).instructions(),
|
||||
"timeout": got,
|
||||
} {
|
||||
if strings.Contains(text, "controller") {
|
||||
t.Errorf("%s names the controller: %s", name, text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,10 +2,13 @@ package console
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
"github.com/nats-io/nats.go/micro"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/announce"
|
||||
@@ -52,7 +55,23 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
|
||||
// Kept for the test: one round of discovery answers every question but mesh_runtimes, which asks anew.
|
||||
IndexKept, announce.Patience = time.Minute, 2500*time.Millisecond
|
||||
t.Cleanup(func() { IndexKept, announce.Patience = keptWas, patienceWas })
|
||||
mt.New(t)
|
||||
mesh := mt.New(t)
|
||||
// What the mesh holds, as the indexer and placement write it (ADR 0571): never the controller's records.
|
||||
mesh.Placed(t, map[string][]string{"slack": {"slack_check"}, "nap": {"doze"}, "nextcloud-client": {"nextcloud_check"}},
|
||||
map[string][]string{"laptop": {"slack", "nextcloud-client"}, "sleeper": {"nap", "nextcloud-client"},
|
||||
"desktop": {"nextcloud-client"}, "bench": nil})
|
||||
// Who holds which seat, by placement (ADR 0571 clause 4), as the seat mesh-placement's holder composes it.
|
||||
placedKV := mesh.Bucket(t, delivery.SeatHoldersBucket)
|
||||
for key, sh := range delivery.SeatHoldersOf(map[string]delivery.Placement{"bench": {Node: "bench",
|
||||
Holds: []delivery.Holding{{Claim: "mesh-wants", Scope: "mesh", Module: "mesh-delivery-assigner"}}}}) {
|
||||
raw, _ := json.Marshal(sh)
|
||||
if _, err := placedKV.Put(key, raw); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
consoleMembership := mt.MembershipOf("node-tools", "laptop-console", false, nil)
|
||||
consoleMembership.State = mt.ConsoleReads()
|
||||
mesh.Issue(t, consoleMembership)
|
||||
|
||||
laptop := connect(t, "node-tools", "laptop")
|
||||
stopTool, err := laptop.HandleSubject("mesh.mod.slack.tool.slack_check.laptop", func(json.RawMessage) (any, error) {
|
||||
@@ -76,14 +95,17 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
|
||||
Subject: "mesh.mod.nextcloud-client.tool.nextcloud_check.desktop"},
|
||||
})
|
||||
|
||||
controller := connect(t, "mesh-controller", "bench")
|
||||
controller := connect(t, "mesh-placement", "bench")
|
||||
var toolsAsked atomic.Int32
|
||||
for verb, output := range map[string]string{
|
||||
"nodes": `[{"name":"laptop"},{"name":"sleeper"},{"name":"desktop"},{"name":"bench"}]`,
|
||||
"modules": `[{"module":"slack","on":["laptop"],"tools":true},{"module":"nap","on":["sleeper"],"tools":true},` +
|
||||
`{"module":"nextcloud-client","on":["laptop","desktop","sleeper"],"tools":true}]`,
|
||||
// The controller's records are served here only to see that none is asked (ADR 0571 clauses 4 and 5).
|
||||
"nodes": `[{"name":"laptop"}]`,
|
||||
"modules": `[]`,
|
||||
"tools": `{"seats":[]}`,
|
||||
} {
|
||||
output := output
|
||||
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
|
||||
output, verb := output, verb
|
||||
stop, err := controller.HandleSubject("mesh.seat.mesh-placement.tool."+verb, func(json.RawMessage) (any, error) {
|
||||
toolsAsked.Add(1)
|
||||
return map[string]any{"output": output, "ok": true}, nil
|
||||
})
|
||||
if err != nil {
|
||||
@@ -94,6 +116,13 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
|
||||
controller.Flush()
|
||||
|
||||
console := connect(t, "node-tools", "laptop-console")
|
||||
console.Follow("node-tools")
|
||||
mt.Until(t, func() error {
|
||||
if console.Membership("node-tools") == nil {
|
||||
return fmt.Errorf("the console's membership is not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
up, err := Serve(NewSurface(console, "laptop.node-tools"), "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -113,8 +142,20 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
|
||||
t.Errorf("the late runtime's machine lists nothing: %s", machine)
|
||||
}
|
||||
|
||||
// The seats' contract and holders, for a client on this machine with no seat definitions (the TypeScript `mesh`).
|
||||
seats := post(t, endpoint, `{"jsonrpc":"2.0","id":9,"method":"mesh/seats"}`)
|
||||
if raw, _ := json.Marshal(seats["result"]); !strings.Contains(string(raw), `"holders":[{"module":"mesh-delivery-assigner","node":"bench"}],"scope":"mesh","seat":"mesh-wants"`) ||
|
||||
!strings.Contains(string(raw), `"replaces":["journalctl"]`) {
|
||||
t.Errorf("mesh/seats answered %s", raw)
|
||||
}
|
||||
if n := toolsAsked.Load(); n != 0 {
|
||||
t.Errorf("the console asked the controller's records %d times", n)
|
||||
}
|
||||
|
||||
// The silent one is named, never called missing.
|
||||
for address, want := range map[string]string{
|
||||
// A seat whose holder announced nothing is named with its holder by placement, never called missing.
|
||||
"mesh-wants.read": "the seat mesh-wants is held by mesh-delivery-assigner on bench by placement",
|
||||
"sleeper/nap.doze": "nap is assigned to sleeper in the mesh's records, but nothing that answered discovery serves it: discovery did not hear in full from node-tools on sleeper (it answered PING",
|
||||
"sleeper/nextcloud-client.nextcloud_check": "nextcloud-client on sleeper did not answer discovery: discovery did not hear in full from node-tools on sleeper",
|
||||
} {
|
||||
|
||||
@@ -19,7 +19,8 @@ package console
|
||||
// Ties go to a seat — the place tools belong — then to the shorter name, then by address.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
@@ -265,27 +266,16 @@ func (x *index) search(query string) []hit {
|
||||
return hits
|
||||
}
|
||||
|
||||
// replacesOf is what every verb and tool replaces, from the controller's records (novox/hq ADR 0245):
|
||||
// a seat's verbs from its `tools` answer, keyed `seat:<seat>.<verb>`, and a module's own tools from
|
||||
// its `modules` answer, keyed `<module>.<tool>`. A controller older than the field answers neither,
|
||||
// and search matches names and descriptions as before.
|
||||
func replacesOf(toolsAnswer json.RawMessage, modules []recordedModule) map[string][]string {
|
||||
// replacesOf is what every verb and tool replaces (novox/hq ADR 0245): a seat's verbs from the seat's contract in
|
||||
// mesh-sdk, keyed `seat:<seat>.<verb>`, read in-process and asked of no part (ADR 0571 clause 5: no part keeps a
|
||||
// central record of tools); and a module's own tools from its records, keyed `<module>.<tool>`. A verb or a tool with
|
||||
// no `replaces` is matched by its name and description as before.
|
||||
func replacesOf(seats []catalogue.Seat, modules []recordedModule) map[string][]string {
|
||||
out := map[string][]string{}
|
||||
var seats struct {
|
||||
Seats []struct {
|
||||
Seat string `json:"seat"`
|
||||
Tools []struct {
|
||||
Name string `json:"name"`
|
||||
Replaces []string `json:"replaces"`
|
||||
} `json:"tools"`
|
||||
} `json:"seats"`
|
||||
}
|
||||
if json.Unmarshal(toolsAnswer, &seats) == nil {
|
||||
for _, s := range seats.Seats {
|
||||
for _, t := range s.Tools {
|
||||
if len(t.Replaces) > 0 {
|
||||
out["seat:"+s.Seat+"."+t.Name] = t.Replaces
|
||||
}
|
||||
for _, s := range seats {
|
||||
for _, v := range s.Serves {
|
||||
if len(v.Replaces) > 0 {
|
||||
out["seat:"+s.Name+"."+v.Name] = append([]string(nil), v.Replaces...)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
package console
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -19,6 +20,7 @@ func meshToSearch(t *testing.T) *index {
|
||||
"status": "One unit as the service manager sees it now: its states, whether it starts at boot.",
|
||||
"restart": "Restart one unit.",
|
||||
"journal": "The last lines of one unit's journal (at most 2000).",
|
||||
"failed": "The units that failed.",
|
||||
}
|
||||
for _, node := range []string{"anchor", "laptop"} {
|
||||
for verb, does := range verbs {
|
||||
@@ -31,8 +33,8 @@ func meshToSearch(t *testing.T) *index {
|
||||
"node": "What one machine reported it can do, what it is assigned, and why.",
|
||||
"nodes": "Every machine the mesh knows, with whether it is converged or adopted.",
|
||||
} {
|
||||
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller",
|
||||
Tool: verb, Scope: "mesh", Description: does, Subject: "mesh.seat.mesh-controller.tool." + verb})
|
||||
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-placement", Seat: "mesh-placement",
|
||||
Tool: verb, Scope: "mesh", Description: does, Subject: "mesh.seat.mesh-placement.tool." + verb})
|
||||
}
|
||||
for tool, does := range map[string]string{
|
||||
"docker_list": "Every container on this machine: name, image, state, ports.",
|
||||
@@ -44,13 +46,7 @@ func meshToSearch(t *testing.T) *index {
|
||||
Description: does, Subject: "mesh.mod.docker.tool." + tool + ".anchor"})
|
||||
}
|
||||
x, _, _ := indexOfEndpoints(endpoints)
|
||||
tools := json.RawMessage(`{"seats":[{"seat":"node-service-manager","scope":"node","tools":[
|
||||
{"name":"units","replaces":["systemctl list-units","systemctl --failed"]},
|
||||
{"name":"status","replaces":["systemctl status","systemctl is-active"]},
|
||||
{"name":"restart","replaces":["systemctl restart"]},
|
||||
{"name":"journal","replaces":["journalctl"]}]},
|
||||
{"seat":"mesh-controller","scope":"mesh","tools":[{"name":"node","replaces":["hostnamectl","uptime"]},{"name":"nodes"}]}]}`)
|
||||
x.Replaces = replacesOf(tools, []recordedModule{{Module: "docker", On: []string{"anchor"}, Tools: true,
|
||||
x.Replaces = replacesOf(catalogue.DefaultSeats(), []recordedModule{{Module: "docker", On: []string{"anchor"}, Tools: true,
|
||||
Replaces: map[string][]string{"docker_list": {"docker ps"}, "docker_logs": {"docker logs"}, "docker_restart": {"docker restart"}}}})
|
||||
return x
|
||||
}
|
||||
@@ -60,22 +56,21 @@ func meshToSearch(t *testing.T) *index {
|
||||
func TestSearchFindsTheToolThatReplacesACommandFirst(t *testing.T) {
|
||||
x := meshToSearch(t)
|
||||
for query, want := range map[string]string{
|
||||
"journalctl": "anchor/node-service-manager.journal",
|
||||
"journalctl -u mesh-controller -n 50": "anchor/node-service-manager.journal",
|
||||
"sudo journalctl -fu sshd": "anchor/node-service-manager.journal",
|
||||
"/usr/bin/journalctl": "anchor/node-service-manager.journal",
|
||||
"logs": "anchor/node-service-manager.journal",
|
||||
"systemctl status": "anchor/node-service-manager.status",
|
||||
"systemctl status nats.service": "anchor/node-service-manager.status",
|
||||
"systemctl restart mesh-controller": "anchor/node-service-manager.restart",
|
||||
"systemctl --failed": "anchor/node-service-manager.units",
|
||||
"docker ps": "anchor/docker.docker_list",
|
||||
"docker ps -a": "anchor/docker.docker_list",
|
||||
"docker logs": "anchor/docker.docker_logs",
|
||||
"docker logs --tail 100 nats": "anchor/docker.docker_logs",
|
||||
"docker restart nats": "anchor/docker.docker_restart",
|
||||
"container logs": "anchor/docker.docker_logs",
|
||||
"uptime": "mesh-controller.node",
|
||||
"journalctl": "anchor/node-service-manager.journal",
|
||||
"journalctl -u mesh-placement -n 50": "anchor/node-service-manager.journal",
|
||||
"sudo journalctl -fu sshd": "anchor/node-service-manager.journal",
|
||||
"/usr/bin/journalctl": "anchor/node-service-manager.journal",
|
||||
"logs": "anchor/node-service-manager.journal",
|
||||
"systemctl status": "anchor/node-service-manager.status",
|
||||
"systemctl status nats.service": "anchor/node-service-manager.status",
|
||||
"systemctl restart mesh-placement": "anchor/node-service-manager.restart",
|
||||
"systemctl --failed": "anchor/node-service-manager.failed",
|
||||
"docker ps": "anchor/docker.docker_list",
|
||||
"docker ps -a": "anchor/docker.docker_list",
|
||||
"docker logs": "anchor/docker.docker_logs",
|
||||
"docker logs --tail 100 nats": "anchor/docker.docker_logs",
|
||||
"docker restart nats": "anchor/docker.docker_restart",
|
||||
"container logs": "anchor/docker.docker_logs",
|
||||
} {
|
||||
hits := x.search(query)
|
||||
if len(hits) == 0 {
|
||||
@@ -116,7 +111,7 @@ func TestASearchByCommandSaysWhatItReplaces(t *testing.T) {
|
||||
// A synonym is a word, never part of one: `logs` matches through `log`, and finds no `login`.
|
||||
func TestSearchStillMatchesEveryWord(t *testing.T) {
|
||||
x := meshToSearch(t)
|
||||
if hits := x.search("machine converged"); len(hits) != 1 || hits[0].Address != "mesh-controller.nodes" {
|
||||
if hits := x.search("machine converged"); len(hits) != 1 || hits[0].Address != "mesh-placement.nodes" {
|
||||
t.Errorf("words in a description: %+v", hits)
|
||||
}
|
||||
if hits := x.search("journalctl kubernetes"); len(hits) != 1 || hits[0].Address != "anchor/node-service-manager.journal" {
|
||||
@@ -134,10 +129,10 @@ func TestSearchStillMatchesEveryWord(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A controller older than `replaces` answers without it: search matches names and descriptions as before.
|
||||
// A seat's contract and a module's records without `replaces`: search matches names and descriptions as before.
|
||||
func TestSearchWithoutReplacesMatchesAsBefore(t *testing.T) {
|
||||
x := meshToSearch(t)
|
||||
x.Replaces = replacesOf(json.RawMessage(`{"seats":[{"seat":"node-service-manager","tools":[{"name":"journal"}]}]}`),
|
||||
x.Replaces = replacesOf([]catalogue.Seat{{Name: "node-service-manager", Serves: []catalogue.Verb{{Name: "journal"}}}},
|
||||
[]recordedModule{{Module: "docker"}})
|
||||
if len(x.Replaces) != 0 {
|
||||
t.Fatalf("replaces from records that have none: %v", x.Replaces)
|
||||
|
||||
@@ -3,7 +3,7 @@ package console
|
||||
// mesh_wait: a session waits on the bus for one event instead of polling (novox/hq ADR 0367).
|
||||
//
|
||||
// **Only the families the tool runner was issued** (§2, §6): its membership names them, composed by the
|
||||
// controller from the tool runner's manifest together with the grant that lets this account subscribe them.
|
||||
// mesh from the tool runner's manifest together with the grant that lets this account subscribe them.
|
||||
// A name not listed there is refused here, before the bus would refuse it; no argument or setting adds one.
|
||||
// **An event only wakes the session** (§4): what it says is read again from the verb that owns it.
|
||||
|
||||
|
||||
@@ -48,6 +48,11 @@ func (b *subscribing) Provision(string, json.RawMessage) (json.RawMessage, error
|
||||
func (b *subscribing) ProvisionWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
|
||||
return func() {}, nil
|
||||
}
|
||||
func (b *subscribing) BusState(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
|
||||
func (b *subscribing) BusStateWatch(string, json.RawMessage, func(json.RawMessage) error) (func(), error) {
|
||||
return func() {}, nil
|
||||
}
|
||||
func (b *subscribing) Begin(string) (string, func()) { return "", func() {} }
|
||||
func (b *subscribing) Subscribe(d func(json.RawMessage) error) error {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
@@ -95,3 +100,7 @@ func TestAnEventIsTakenByAnyAnswerThatIsNotAnError(t *testing.T) {
|
||||
t.Fatalf("an error answer was not the bundle's refusal of mesh/event: %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (b *subscribing) LogWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
|
||||
return func() {}, nil
|
||||
}
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"golang.org/x/sys/unix"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/wire"
|
||||
@@ -38,6 +39,9 @@ type Tool struct {
|
||||
Name string
|
||||
Description string
|
||||
Input json.RawMessage
|
||||
// Effect is what the tool does, reads, acts or dangerous (novox/hq ADR 0387 §1), where the runtime knows it: its own
|
||||
// tools say it; a bundle's tool is marked by its manifest.
|
||||
Effect string
|
||||
// Run calls the tool for caller: the bus user the call's subject named, or "" when it named none. A
|
||||
// caller is handed to the bundle in the call's `_meta` (MetaCaller), never among its arguments.
|
||||
Run func(caller string, args json.RawMessage) (json.RawMessage, error)
|
||||
@@ -48,12 +52,21 @@ type Tool struct {
|
||||
// stdio.CallerOf, and says Unproven when it is absent.
|
||||
const MetaCaller = "mesh/caller"
|
||||
|
||||
// callParams is a `tools/call`'s params: the tool, its arguments as the caller sent them, and — only when the
|
||||
// subject named one — the caller, in `_meta`, which the runtime writes and no caller can.
|
||||
func callParams(name, caller string, args json.RawMessage) map[string]any {
|
||||
// callParams is a `tools/call`'s params: the tool, its arguments as the caller sent them, and in `_meta`, which the
|
||||
// runtime writes and no caller can, the caller — only when the subject named one — and the id of the call in flight
|
||||
// (mesh-sdk buscalls.MetaCall), which a ledger's apply names so the runtime stamps the caller it proved (novox/hq ADR
|
||||
// 0597 §7).
|
||||
func callParams(name, caller, call string, args json.RawMessage) map[string]any {
|
||||
params := map[string]any{"name": name, "arguments": args}
|
||||
meta := map[string]any{}
|
||||
if caller != "" {
|
||||
params["_meta"] = map[string]any{MetaCaller: caller}
|
||||
meta[MetaCaller] = caller
|
||||
}
|
||||
if call != "" {
|
||||
meta[buscalls.MetaCall] = call
|
||||
}
|
||||
if len(meta) > 0 {
|
||||
params["_meta"] = meta
|
||||
}
|
||||
return params
|
||||
}
|
||||
@@ -80,6 +93,8 @@ type Bus interface {
|
||||
Watch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
|
||||
// Log answers a bundle's `append`, `read`, `last` and `info` on its module's log (novox/hq ADR 0297).
|
||||
Log(verb string, params json.RawMessage) (json.RawMessage, error)
|
||||
// LogWatch hands deliver each entry of a log from where the bundle asked, one at a time (novox/hq ADR 0571).
|
||||
LogWatch(params json.RawMessage, deliver func(entry json.RawMessage) error) (stop func(), err error)
|
||||
// Seat answers a bundle's seat traffic (novox/hq ADR 0259 §3): `publish`, `prove` and `record` at once;
|
||||
// `take` and `answer` bind once and hand each message or proof on through handOn, to whichever child
|
||||
// of the module asked last — a restarted child asks again as its code runs again.
|
||||
@@ -89,8 +104,29 @@ type Bus interface {
|
||||
// (novox/hq ADR 0349 §14, ADR 0370 §4): its config, and its bindings' values that are not secret.
|
||||
Provision(verb string, params json.RawMessage) (json.RawMessage, error)
|
||||
ProvisionWatch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
|
||||
// BusState answers a bundle's calls on its module's bus state in the new form (mesh-sdk buscalls; novox/hq ADR 0595
|
||||
// §9, design 25 § 12.5): its objects, a stream's read-back and fill, a work queue's size and dead letters, and its
|
||||
// ledgers, by the method the bundle asked, from its own module's bindings alone.
|
||||
BusState(method string, params json.RawMessage) (json.RawMessage, error)
|
||||
// BusStateWatch hands deliver each change of an objects or a ledger watch, the current ones first.
|
||||
BusStateWatch(method string, params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
|
||||
// Begin says a tool call of the module is in flight for the caller the runtime proved, and answers its id, which the
|
||||
// bundle is handed in the call's `_meta`, and the end to call once the call is answered (novox/hq ADR 0597 §7).
|
||||
Begin(caller string) (id string, end func())
|
||||
}
|
||||
|
||||
// busStateCalls are the methods of mesh-sdk buscalls a bundle asks and BusState answers.
|
||||
var busStateCalls = map[string]bool{
|
||||
buscalls.ObjectsPut: true, buscalls.ObjectsGet: true, buscalls.ObjectsList: true, buscalls.ObjectsDelete: true,
|
||||
buscalls.ObjectsInfo: true, buscalls.StreamRead: true, buscalls.StreamFill: true, buscalls.WorkQueueSize: true,
|
||||
buscalls.WorkQueueDead: true, buscalls.LedgerApply: true, buscalls.LedgerGet: true, buscalls.LedgerHistory: true,
|
||||
buscalls.LedgerQuery: true, buscalls.LedgerRebuild: true, buscalls.LedgerReindex: true, buscalls.LedgerSnapshot: true,
|
||||
buscalls.LedgerLag: true,
|
||||
}
|
||||
|
||||
// busStateWatches are the watches of mesh-sdk buscalls, each with the method its changes are handed to the bundle by.
|
||||
var busStateWatches = map[string]string{buscalls.ObjectsWatch: buscalls.ObjectsWatched, buscalls.LedgerWatch: buscalls.LedgerWatched}
|
||||
|
||||
// EventTimeout bounds how long a bundle has to handle one event before it is offered again.
|
||||
var EventTimeout = 2 * time.Minute
|
||||
|
||||
@@ -323,12 +359,12 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
result = answered
|
||||
}
|
||||
case "mesh/state.get", "mesh/state.put", "mesh/state.delete", "mesh/state.keys",
|
||||
"mesh/state.create", "mesh/state.update", "mesh/state.history":
|
||||
"mesh/state.create", "mesh/state.update", "mesh/state.history", "mesh/state.info":
|
||||
var answered json.RawMessage
|
||||
if answered, err = mesh.State(strings.TrimPrefix(m.Method, "mesh/state."), m.Params); err == nil {
|
||||
result = answered
|
||||
}
|
||||
case "mesh/log.append", "mesh/log.read", "mesh/log.last", "mesh/log.info":
|
||||
case "mesh/log.append", "mesh/log.read", "mesh/log.newest", "mesh/log.last", "mesh/log.info":
|
||||
var answered json.RawMessage
|
||||
if answered, err = mesh.Log(strings.TrimPrefix(m.Method, "mesh/log."), m.Params); err == nil {
|
||||
result = answered
|
||||
@@ -357,6 +393,25 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
case "mesh/log.watch":
|
||||
// As a state's watch: each entry asked of this child in order, `mesh/log {watch, entry}`,
|
||||
// naming the watch by the id the child asked it with (novox/hq ADR 0571).
|
||||
var stop func()
|
||||
watch := m.ID
|
||||
stop, err = mesh.LogWatch(m.Params, func(entry json.RawMessage) error {
|
||||
_, err := c.ask(module, "mesh/log", map[string]any{"watch": watch, "entry": entry}, EventTimeout)
|
||||
return err
|
||||
})
|
||||
if err == nil {
|
||||
c.mu.Lock()
|
||||
if c.why != nil {
|
||||
c.mu.Unlock()
|
||||
stop()
|
||||
} else {
|
||||
c.watches = append(c.watches, stop)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
case "mesh/state.watch":
|
||||
// Each change is asked of this child, in order, naming the watch it is for by
|
||||
// the id the child asked it with — two watches of one state are two handlers;
|
||||
@@ -378,7 +433,33 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
case buscalls.ObjectsWatch, buscalls.LedgerWatch:
|
||||
// As a state's watch: each change asked of this child in order, by the method of its
|
||||
// family; the change names the watch by the id the bundle gave in its request.
|
||||
var stop func()
|
||||
told := busStateWatches[m.Method]
|
||||
stop, err = mesh.BusStateWatch(m.Method, m.Params, func(change json.RawMessage) error {
|
||||
_, err := c.ask(module, told, change, EventTimeout)
|
||||
return err
|
||||
})
|
||||
if err == nil {
|
||||
c.mu.Lock()
|
||||
if c.why != nil {
|
||||
c.mu.Unlock()
|
||||
stop()
|
||||
} else {
|
||||
c.watches = append(c.watches, stop)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
default:
|
||||
if busStateCalls[m.Method] {
|
||||
var answered json.RawMessage
|
||||
if answered, err = mesh.BusState(m.Method, m.Params); err == nil {
|
||||
result = answered
|
||||
}
|
||||
break
|
||||
}
|
||||
if len(m.ID) > 0 {
|
||||
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID,
|
||||
"error": map[string]any{"code": -32601, "message": "the runtime answers no " + m.Method + " from a bundle"}})
|
||||
@@ -590,7 +671,9 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
|
||||
if len(args) == 0 || string(args) == "null" {
|
||||
args = json.RawMessage("{}")
|
||||
}
|
||||
res, err := asking("tools/call", callParams(full, caller, args), CallTimeout)
|
||||
call, end := mesh.Begin(caller)
|
||||
defer end()
|
||||
res, err := asking("tools/call", callParams(full, caller, call, args), CallTimeout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -105,3 +105,7 @@ func TestABundleAsksForItsValuesAndIsHandedItsWatchsChanges(t *testing.T) {
|
||||
t.Errorf("the runtime's bus was asked %q", b.asked)
|
||||
}
|
||||
}
|
||||
|
||||
func (b *provisioning) LogWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
|
||||
return func() {}, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,497 @@
|
||||
// Package ledger is the tool runner's side of a ledger (novox/hq ADR 0597, design 25 § 12.4; issue 1966): a storage type
|
||||
// whose state is derived from its history. `apply` checks an operation against its type, appends it to the instance's
|
||||
// history with the last sequence expected, derives the state by the operation's effect and writes it as an update of the
|
||||
// revision read, and writes the instance's snapshot. What the history, the state and the snapshots are kept in is given
|
||||
// (History, State, Snapshots): the bus in the runtime, fakes in a test.
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// The refusals the stores answer, told by errors.Is.
|
||||
var (
|
||||
// ErrRaced is an append or a snapshot written with an expected sequence that is no longer the last: another writer
|
||||
// wrote first.
|
||||
ErrRaced = errors.New("another writer wrote the instance's history first")
|
||||
// ErrChanged is a state written at a revision that is no longer the key's.
|
||||
ErrChanged = errors.New("the state was written by another writer first")
|
||||
// ErrFull is a state bucket that refuses a write because it is full.
|
||||
ErrFull = errors.New("the state bucket is full")
|
||||
)
|
||||
|
||||
// CodeChanged is the error code a bundle is answered a lost race with, as a lost state update (mesh-sdk stdio.CodeChanged).
|
||||
const CodeChanged = -32010
|
||||
|
||||
// raced is an apply refused because another writer appended to the instance first (ADR 0597 §5).
|
||||
type raced struct{ instance string }
|
||||
|
||||
func (e raced) Error() string {
|
||||
return "the operation on " + e.instance + " is not taken: another writer appended to its history first; read it and apply again (novox/hq ADR 0597 §5)"
|
||||
}
|
||||
func (e raced) Is(target error) bool { return target == ErrRaced }
|
||||
func (e raced) ErrorCode() int { return CodeChanged }
|
||||
|
||||
// Kept is one message a history keeps: its sequence and its bytes.
|
||||
type Kept struct {
|
||||
Sequence uint64
|
||||
Data []byte
|
||||
}
|
||||
|
||||
// History is a ledger's history: a stream with a subject per instance, each instance named by its key.
|
||||
type History interface {
|
||||
// Last is the sequence of the instance's last operation, 0 when it has none.
|
||||
Last(key string) (uint64, error)
|
||||
// Append writes one operation, expecting the instance's last sequence to be expect (0: it has none); ErrRaced when
|
||||
// it is not.
|
||||
Append(key string, op []byte, expect uint64) (uint64, error)
|
||||
// Read is the operations of the instances keys names (a key, with `*` for one name and `>` for the rest) from a
|
||||
// sequence on, oldest first, at most limit of them (every one when limit is 0).
|
||||
Read(keys string, from uint64, limit int) ([]Kept, error)
|
||||
}
|
||||
|
||||
// State is a ledger's state bucket: each instance's record under its key.
|
||||
type State interface {
|
||||
// Get is the key's value and revision; nil and 0 when it has none.
|
||||
Get(key string) ([]byte, uint64, error)
|
||||
// Write writes the key's value at the revision read (0: only when it has none); ErrChanged when the key is no longer
|
||||
// at it, ErrFull when the bucket is full.
|
||||
Write(key string, value []byte, revision uint64) (uint64, error)
|
||||
// Keys is every key the bucket holds.
|
||||
Keys() ([]string, error)
|
||||
}
|
||||
|
||||
// Snapshots are a ledger's snapshots: the latest record of each instance, on a subject of its own, never expiring.
|
||||
type Snapshots interface {
|
||||
// Get is the key's latest snapshot and the sequence it was written at; nil and 0 when it has none.
|
||||
Get(key string) ([]byte, uint64, error)
|
||||
// Put writes the key's snapshot, expecting its last to be at expect (0: none); ErrRaced when it is not.
|
||||
Put(key string, value []byte, expect uint64) error
|
||||
}
|
||||
|
||||
// Ledger is one ledger of one module: its types and where it is kept.
|
||||
type Ledger struct {
|
||||
Module string
|
||||
Name string
|
||||
// Account is the owning module's authenticated account, which every operation it applies is stamped with.
|
||||
Account string
|
||||
Types []catalogue.LedgerType
|
||||
History History
|
||||
State State
|
||||
Snapshots Snapshots
|
||||
// Indexes are its declared indexes' buckets, by index name.
|
||||
Indexes map[string]Index
|
||||
// Now is the clock an operation is stamped by; time.Now when nil.
|
||||
Now func() time.Time
|
||||
// Logf says what an apply could not do after its operation was taken; nothing is said when nil.
|
||||
Logf func(format string, args ...any)
|
||||
}
|
||||
|
||||
func (l *Ledger) say(format string, args ...any) {
|
||||
if l.Logf != nil {
|
||||
l.Logf(format, args...)
|
||||
}
|
||||
}
|
||||
|
||||
// Record is what the state and a snapshot keep of an instance: its state and the sequence of the last operation in it.
|
||||
type Record struct {
|
||||
Sequence uint64 `json:"sequence"`
|
||||
State json.RawMessage `json:"state"`
|
||||
}
|
||||
|
||||
// segment is one part of an instance's path.
|
||||
var segment = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
|
||||
|
||||
// KeyOf is the key an instance is kept under, in the state and as its history's and snapshot's subject token: its path's
|
||||
// parts joined by dots (`decision/0290` is `decision.0290`).
|
||||
func KeyOf(instance string) (string, error) {
|
||||
parts := strings.Split(instance, "/")
|
||||
if len(parts) < 2 {
|
||||
return "", fmt.Errorf("%q is not an instance: an instance is named by a path whose first part is its type's name, "+
|
||||
"such as decision/0290 (novox/hq ADR 0597 §1)", instance)
|
||||
}
|
||||
for _, p := range parts {
|
||||
if !segment.MatchString(p) {
|
||||
return "", fmt.Errorf("%q is not an instance: each part of its path holds letters, digits, - and _ alone", instance)
|
||||
}
|
||||
}
|
||||
return strings.Join(parts, "."), nil
|
||||
}
|
||||
|
||||
// InstanceOf is the instance a key names.
|
||||
func InstanceOf(key string) string { return strings.ReplaceAll(key, ".", "/") }
|
||||
|
||||
func (l *Ledger) now() time.Time {
|
||||
if l.Now != nil {
|
||||
return l.Now().UTC()
|
||||
}
|
||||
return time.Now().UTC()
|
||||
}
|
||||
|
||||
// typeOf is the instance's type, by its path's first part.
|
||||
func (l *Ledger) typeOf(instance string) (catalogue.LedgerType, error) {
|
||||
name, _, _ := strings.Cut(instance, "/")
|
||||
for _, t := range l.Types {
|
||||
if t.Name == name {
|
||||
return t, nil
|
||||
}
|
||||
}
|
||||
return catalogue.LedgerType{}, fmt.Errorf("%s's ledger %s has no type %q, which the instance %s names", l.Module, l.Name, name, instance)
|
||||
}
|
||||
|
||||
// operationOf is the operation a type allows by its name.
|
||||
func operationOf(t catalogue.LedgerType, name string) (catalogue.LedgerOperation, error) {
|
||||
for _, op := range t.Operations {
|
||||
if op.Name == name {
|
||||
return op, nil
|
||||
}
|
||||
}
|
||||
return catalogue.LedgerOperation{}, fmt.Errorf("the type %s allows no operation %q", t.Name, name)
|
||||
}
|
||||
|
||||
// Digest is the digest a type pairs a text field with (ADR 0597 §2): sha256 of the text, as the knowledge base writes it.
|
||||
func Digest(text string) string {
|
||||
sum := sha256.Sum256([]byte(text))
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// kinds is the JSON kind each schema type name is held to; a name not here is the judge's to know.
|
||||
var kinds = map[string]func(any) bool{
|
||||
"string": func(v any) bool { _, ok := v.(string); return ok },
|
||||
"number": func(v any) bool { _, ok := v.(float64); return ok },
|
||||
"integer": func(v any) bool { f, ok := v.(float64); return ok && f == float64(int64(f)) },
|
||||
"boolean": func(v any) bool { _, ok := v.(bool); return ok },
|
||||
"object": func(v any) bool { _, ok := v.(map[string]any); return ok },
|
||||
"array": func(v any) bool { _, ok := v.([]any); return ok },
|
||||
}
|
||||
|
||||
// checked is an operation as apply checked it: its instance's key, its operation and its body.
|
||||
type checked struct {
|
||||
key string
|
||||
op catalogue.LedgerOperation
|
||||
body map[string]any
|
||||
}
|
||||
|
||||
// check holds an operation to its type (ADR 0597 §2): a type the instance names, an operation it allows, a body that is an
|
||||
// object whose fields fit the schema, the key its effect names, and every digest matching its text.
|
||||
func (l *Ledger) check(r buscalls.ApplyRequest) (checked, error) {
|
||||
key, err := KeyOf(r.Instance)
|
||||
if err != nil {
|
||||
return checked{}, err
|
||||
}
|
||||
t, err := l.typeOf(r.Instance)
|
||||
if err != nil {
|
||||
return checked{}, err
|
||||
}
|
||||
op, err := operationOf(t, r.Operation)
|
||||
if err != nil {
|
||||
return checked{}, err
|
||||
}
|
||||
body := map[string]any{}
|
||||
if len(r.Body) > 0 && string(r.Body) != "null" {
|
||||
if err := json.Unmarshal(r.Body, &body); err != nil {
|
||||
return checked{}, fmt.Errorf("the operation %s's body is a JSON object: %w", r.Operation, err)
|
||||
}
|
||||
}
|
||||
for name, field := range t.Schema {
|
||||
v, has := body[name]
|
||||
if !has {
|
||||
continue
|
||||
}
|
||||
if fits, known := kinds[field.Type]; known && !fits(v) {
|
||||
return checked{}, fmt.Errorf("the operation %s's field %s is a %s in the type %s", r.Operation, name, field.Type, t.Name)
|
||||
}
|
||||
}
|
||||
for name, field := range t.Schema {
|
||||
if field.DigestOf == "" {
|
||||
continue
|
||||
}
|
||||
digest, hasDigest := body[name]
|
||||
text, hasText := body[field.DigestOf]
|
||||
if !hasDigest && !hasText {
|
||||
continue
|
||||
}
|
||||
s, isText := text.(string)
|
||||
if !hasDigest || !hasText || !isText || digest != Digest(s) {
|
||||
return checked{}, fmt.Errorf("the operation %s's %s is not the digest of its %s, so it is refused (novox/hq ADR 0597 §2)",
|
||||
r.Operation, name, field.DigestOf)
|
||||
}
|
||||
}
|
||||
switch op.Effect {
|
||||
case catalogue.EffectSetFields, catalogue.EffectAppend:
|
||||
case catalogue.EffectSetEntry, catalogue.EffectRemove:
|
||||
if op.Effect == catalogue.EffectRemove && op.Into == "" {
|
||||
break
|
||||
}
|
||||
if _, ok := body[op.Key].(string); !ok {
|
||||
return checked{}, fmt.Errorf("the operation %s names its entry by its field %s, a text, which its body lacks", r.Operation, op.Key)
|
||||
}
|
||||
default:
|
||||
return checked{}, fmt.Errorf("the operation %s's effect %q is not one the mesh offers (novox/hq ADR 0597 §2)", r.Operation, op.Effect)
|
||||
}
|
||||
return checked{key: key, op: op, body: body}, nil
|
||||
}
|
||||
|
||||
// effect is an operation's effect on a state (ADR 0597 §2): it copies only the fields the operation names, and the
|
||||
// operation's sequence and proven author where it says so; null is a removed instance.
|
||||
func effect(state json.RawMessage, op catalogue.LedgerOperation, body map[string]any, seq uint64, author buscalls.Author) (json.RawMessage, error) {
|
||||
s := map[string]any{}
|
||||
if len(state) > 0 && string(state) != "null" {
|
||||
if err := json.Unmarshal(state, &s); err != nil {
|
||||
return nil, fmt.Errorf("the state held is not an object: %w", err)
|
||||
}
|
||||
}
|
||||
copied := map[string]any{}
|
||||
for _, f := range op.Fields {
|
||||
if v, ok := body[f]; ok {
|
||||
copied[f] = v
|
||||
}
|
||||
}
|
||||
if op.Sequence {
|
||||
copied["sequence"] = seq
|
||||
}
|
||||
// Written last, so a body's own `author` never stands for the proven one (ADR 0597 §7).
|
||||
if op.Author {
|
||||
copied["author"] = author
|
||||
}
|
||||
switch op.Effect {
|
||||
case catalogue.EffectSetFields:
|
||||
for k, v := range copied {
|
||||
s[k] = v
|
||||
}
|
||||
case catalogue.EffectSetEntry:
|
||||
m, _ := s[op.Into].(map[string]any)
|
||||
if m == nil {
|
||||
m = map[string]any{}
|
||||
}
|
||||
m[body[op.Key].(string)] = copied
|
||||
s[op.Into] = m
|
||||
case catalogue.EffectAppend:
|
||||
list, _ := s[op.Into].([]any)
|
||||
s[op.Into] = append(list, copied)
|
||||
case catalogue.EffectRemove:
|
||||
if op.Into == "" {
|
||||
return json.RawMessage("null"), nil
|
||||
}
|
||||
if m, ok := s[op.Into].(map[string]any); ok {
|
||||
delete(m, body[op.Key].(string))
|
||||
}
|
||||
}
|
||||
return json.Marshal(s)
|
||||
}
|
||||
|
||||
// fold applies one kept operation to a record, as apply derived it.
|
||||
func (l *Ledger) fold(rec Record, k Kept) (Record, error) {
|
||||
var o buscalls.Operation
|
||||
if err := json.Unmarshal(k.Data, &o); err != nil {
|
||||
return rec, fmt.Errorf("the history's operation %d is not one: %w", k.Sequence, err)
|
||||
}
|
||||
t, err := l.typeOf(o.Instance)
|
||||
if err != nil {
|
||||
return rec, err
|
||||
}
|
||||
op, err := operationOf(t, o.Operation)
|
||||
if err != nil {
|
||||
return rec, err
|
||||
}
|
||||
body := map[string]any{}
|
||||
if len(o.Body) > 0 && string(o.Body) != "null" {
|
||||
if err := json.Unmarshal(o.Body, &body); err != nil {
|
||||
return rec, err
|
||||
}
|
||||
}
|
||||
if (op.Effect == catalogue.EffectSetEntry || (op.Effect == catalogue.EffectRemove && op.Into != "")) && !isText(body[op.Key]) {
|
||||
return rec, fmt.Errorf("the history's operation %d names no entry", k.Sequence)
|
||||
}
|
||||
state, err := effect(rec.State, op, body, k.Sequence, o.Author)
|
||||
if err != nil {
|
||||
return rec, err
|
||||
}
|
||||
return Record{Sequence: k.Sequence, State: state}, nil
|
||||
}
|
||||
|
||||
func isText(v any) bool { _, ok := v.(string); return ok }
|
||||
|
||||
// replay folds the instance's operations after rec's sequence, up to upto (every one when 0), onto rec.
|
||||
func (l *Ledger) replay(key string, rec Record, upto uint64) (Record, error) {
|
||||
kept, err := l.History.Read(key, rec.Sequence+1, 0)
|
||||
if err != nil {
|
||||
return rec, err
|
||||
}
|
||||
for _, k := range kept {
|
||||
if upto != 0 && k.Sequence > upto {
|
||||
break
|
||||
}
|
||||
if k.Sequence <= rec.Sequence {
|
||||
continue
|
||||
}
|
||||
if rec, err = l.fold(rec, k); err != nil {
|
||||
return rec, err
|
||||
}
|
||||
}
|
||||
return rec, nil
|
||||
}
|
||||
|
||||
// readState is the instance's record in the state and its revision; an empty record when it has none.
|
||||
func (l *Ledger) readState(key string) (Record, uint64, error) {
|
||||
raw, rev, err := l.State.Get(key)
|
||||
if err != nil || raw == nil {
|
||||
return Record{}, rev, err
|
||||
}
|
||||
var rec Record
|
||||
if err := json.Unmarshal(raw, &rec); err != nil {
|
||||
return Record{}, rev, fmt.Errorf("the state of %s is not a record: %w", key, err)
|
||||
}
|
||||
return rec, rev, nil
|
||||
}
|
||||
|
||||
// readSnapshot is the instance's snapshot and the sequence it was written at; an empty record when it has none.
|
||||
func (l *Ledger) readSnapshot(key string) (Record, uint64, error) {
|
||||
raw, at, err := l.Snapshots.Get(key)
|
||||
if err != nil || raw == nil {
|
||||
return Record{}, at, err
|
||||
}
|
||||
var rec Record
|
||||
if err := json.Unmarshal(raw, &rec); err != nil {
|
||||
return Record{}, at, fmt.Errorf("the snapshot of %s is not a record: %w", key, err)
|
||||
}
|
||||
return rec, at, nil
|
||||
}
|
||||
|
||||
// derive is the instance's record caught up with its history up to last: from the state, or from the snapshot when it
|
||||
// is further on (an instance whose state was lost, or whose operations expired, ADR 0597 §8).
|
||||
func (l *Ledger) derive(key string, base Record, last uint64) (Record, error) {
|
||||
if base.Sequence >= last {
|
||||
return base, nil
|
||||
}
|
||||
snap, _, err := l.readSnapshot(key)
|
||||
if err != nil {
|
||||
return base, err
|
||||
}
|
||||
if snap.Sequence > base.Sequence {
|
||||
base = snap
|
||||
}
|
||||
return l.replay(key, base, last)
|
||||
}
|
||||
|
||||
// settle writes want into the state at the revision read (ADR 0597 §5): a lost update re-reads, and catches the state
|
||||
// read up to want's sequence from the history, never appending again; a state already further on is left as it is. It
|
||||
// answers what the state holds after it, and whether it is behind because the bucket is full (ADR 0597 §6).
|
||||
func (l *Ledger) settle(key string, want Record, rev uint64) (Record, bool, error) {
|
||||
for tries := 0; tries < 16; tries++ {
|
||||
raw, err := json.Marshal(want)
|
||||
if err != nil {
|
||||
return want, false, err
|
||||
}
|
||||
_, err = l.State.Write(key, raw, rev)
|
||||
switch {
|
||||
case err == nil:
|
||||
return want, false, nil
|
||||
case errors.Is(err, ErrFull):
|
||||
return want, true, nil
|
||||
case !errors.Is(err, ErrChanged):
|
||||
return want, false, err
|
||||
}
|
||||
cur, curRev, err := l.readState(key)
|
||||
if err != nil {
|
||||
return want, false, err
|
||||
}
|
||||
if cur.Sequence >= want.Sequence {
|
||||
return cur, false, nil
|
||||
}
|
||||
if want, err = l.replay(key, cur, want.Sequence); err != nil {
|
||||
return want, false, err
|
||||
}
|
||||
rev = curRev
|
||||
}
|
||||
return want, false, fmt.Errorf("the state of %s kept being written by other writers; the operation is in its history and the next apply catches it up", key)
|
||||
}
|
||||
|
||||
// writeSnapshot writes rec as the instance's snapshot unless the snapshot is already as far on (ADR 0597 §8).
|
||||
func (l *Ledger) writeSnapshot(key string, rec Record) error {
|
||||
for tries := 0; tries < 16; tries++ {
|
||||
snap, at, err := l.readSnapshot(key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if at != 0 && snap.Sequence >= rec.Sequence {
|
||||
return nil
|
||||
}
|
||||
raw, err := json.Marshal(rec)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = l.Snapshots.Put(key, raw, at); err == nil || !errors.Is(err, ErrRaced) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("the snapshot of %s kept being written by other writers", key)
|
||||
}
|
||||
|
||||
// Apply applies one operation to one instance, stamped with author (ADR 0597 §3, §5, §6, §8; design 25 § 12.4 steps 1-5):
|
||||
// checked against its type, appended with the instance's last sequence expected, the state derived by its effect and
|
||||
// written at the revision read, and the snapshot written. A racing writer's append is refused; a full state bucket
|
||||
// answers taken with the state behind.
|
||||
func (l *Ledger) Apply(r buscalls.ApplyRequest, author buscalls.Author) (buscalls.Applied, error) {
|
||||
c, err := l.check(r)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
last, err := l.History.Last(c.key)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
held, rev, err := l.readState(c.key)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
base, err := l.derive(c.key, held, last)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
body := r.Body
|
||||
if len(body) == 0 {
|
||||
body = json.RawMessage("{}")
|
||||
}
|
||||
op, err := json.Marshal(buscalls.Operation{Instance: r.Instance, Operation: r.Operation, Body: body, Author: author, At: l.now()})
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
seq, err := l.History.Append(c.key, op, last)
|
||||
if errors.Is(err, ErrRaced) {
|
||||
return buscalls.Applied{}, raced{r.Instance}
|
||||
}
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
state, err := effect(base.State, c.op, c.body, seq, author)
|
||||
if err != nil {
|
||||
return buscalls.Applied{}, err
|
||||
}
|
||||
// From here the operation is taken: it is in the history, and a fault below is the state's to catch up, said here
|
||||
// and answered as the state behind, never as a refusal of what later takes effect (ADR 0597 §6).
|
||||
now, behind, err := l.settle(c.key, Record{Sequence: seq, State: state}, rev)
|
||||
if err != nil {
|
||||
l.say("[mesh-tools] %s's ledger %s took operation %d on %s and its state could not follow: %v; the next apply "+
|
||||
"or a rebuild catches it up", l.Module, l.Name, seq, r.Instance, err)
|
||||
behind, now = true, Record{Sequence: seq, State: state}
|
||||
}
|
||||
if behind {
|
||||
now = Record{Sequence: seq, State: state}
|
||||
}
|
||||
if err := l.writeSnapshot(c.key, now); err != nil {
|
||||
l.say("[mesh-tools] %s's ledger %s took operation %d on %s and its snapshot could not be written: %v; the next "+
|
||||
"apply or a snapshot repair writes it", l.Module, l.Name, seq, r.Instance, err)
|
||||
}
|
||||
return buscalls.Applied{Taken: true, Sequence: seq, StateBehind: behind}, nil
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
var owner = buscalls.Author{Account: "ledger.anchor.m"}
|
||||
|
||||
// 1966/AC13, 1966/AC14 and the checks of an operation against its type (ADR 0597 §2, §5, §6, §8; design 25 § 12.4).
|
||||
func TestApplyAppendsDerivesAndSnapshots(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
run func(t *testing.T, l *Ledger, f fakes)
|
||||
}{
|
||||
{"1966/AC13 an instance at sequence 4 is taken with 5, its state and snapshot the effect's result", func(t *testing.T, l *Ledger, f fakes) {
|
||||
for i := 1; i <= 4; i++ {
|
||||
applied(t, l, "decision/0290", "note", fmt.Sprintf(`{"says":"n%d"}`, i), owner)
|
||||
}
|
||||
if last, _ := f.history.Last("decision.0290"); last != 4 {
|
||||
t.Fatalf("the instance is at %d, want 4", last)
|
||||
}
|
||||
got := applied(t, l, "decision/0290", "title", `{"title":"A ledger","ignored":"x"}`, owner)
|
||||
if !got.Taken || got.Sequence != 5 || got.StateBehind {
|
||||
t.Fatalf("answered %+v, want taken with 5", got)
|
||||
}
|
||||
raw, _, _ := f.state.Get("decision.0290")
|
||||
state := record(t, raw)
|
||||
want := `{"author":{"account":"ledger.anchor.m"},"notes":[{"says":"n1"},{"says":"n2"},{"says":"n3"},{"says":"n4"}],"sequence":5,"title":"A ledger"}`
|
||||
if state.Sequence != 5 || string(state.State) != want {
|
||||
t.Fatalf("the state is %d %s, want 5 %s", state.Sequence, state.State, want)
|
||||
}
|
||||
snap, _, _ := f.snapshots.Get("decision.0290")
|
||||
if string(snap) != string(raw) {
|
||||
t.Fatalf("the snapshot is %s, the state %s", snap, raw)
|
||||
}
|
||||
}},
|
||||
{"1966/AC14 a full state bucket answers taken and state behind, and the next apply after room catches up", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/0290", "note", `{"says":"first"}`, owner)
|
||||
f.state.Full = true
|
||||
got := applied(t, l, "decision/0290", "note", `{"says":"while full"}`, owner)
|
||||
if !got.Taken || got.Sequence != 2 || !got.StateBehind {
|
||||
t.Fatalf("answered %+v, want taken with 2 and state behind", got)
|
||||
}
|
||||
if raw, _, _ := f.state.Get("decision.0290"); record(t, raw).Sequence != 1 {
|
||||
t.Fatalf("the full state moved: %s", raw)
|
||||
}
|
||||
f.state.Full = false
|
||||
got = applied(t, l, "decision/0290", "note", `{"says":"after room"}`, owner)
|
||||
raw, _, _ := f.state.Get("decision.0290")
|
||||
state := record(t, raw)
|
||||
if got.StateBehind || state.Sequence != 3 || string(state.State) != `{"notes":[{"says":"first"},{"says":"while full"},{"says":"after room"}]}` {
|
||||
t.Fatalf("after room: %+v, the state %d %s", got, state.Sequence, state.State)
|
||||
}
|
||||
}},
|
||||
{"each effect: set an entry, remove it, remove the instance", func(t *testing.T, l *Ledger, f fakes) {
|
||||
text := "the section's text"
|
||||
applied(t, l, "decision/1", "section", fmt.Sprintf(`{"id":"a","text":%q,"digest":%q}`, text, Digest(text)), owner)
|
||||
applied(t, l, "decision/1", "section", fmt.Sprintf(`{"id":"b","text":"b","digest":%q}`, Digest("b")), owner)
|
||||
applied(t, l, "decision/1", "drop-section", `{"id":"a"}`, owner)
|
||||
raw, _, _ := f.state.Get("decision.1")
|
||||
if got := string(record(t, raw).State); got != fmt.Sprintf(`{"sections":{"b":{"digest":%q}}}`, Digest("b")) {
|
||||
t.Fatalf("the state is %s", got)
|
||||
}
|
||||
applied(t, l, "decision/1", "withdraw", `{}`, owner)
|
||||
raw, _, _ = f.state.Get("decision.1")
|
||||
if got := record(t, raw); got.Sequence != 4 || string(got.State) != "null" {
|
||||
t.Fatalf("the removed instance is %d %s", got.Sequence, got.State)
|
||||
}
|
||||
}},
|
||||
{"an operation outside its type, a body off its schema and a digest off its text are refused", func(t *testing.T, l *Ledger, f fakes) {
|
||||
for _, c := range []struct{ instance, operation, body, refusal string }{
|
||||
{"listing/a17", "title", `{}`, `no type "listing"`},
|
||||
{"decision/0290", "accept", `{}`, `no operation "accept"`},
|
||||
{"decision/0290", "title", `{"title":7}`, "field title is a string"},
|
||||
{"decision/0290", "section", `{"id":"a","text":"x","digest":"sha256:00"}`, "is not the digest of its text"},
|
||||
{"decision/0290", "section", `{"text":"x"}`, "is not the digest"},
|
||||
{"decision/0290", "drop-section", `{}`, "names its entry by its field id"},
|
||||
{"decision", "title", `{}`, "is not an instance"},
|
||||
{"decision/a.b", "title", `{}`, "is not an instance"},
|
||||
} {
|
||||
_, err := l.Apply(buscalls.ApplyRequest{Instance: c.instance, Operation: c.operation, Body: json.RawMessage(c.body)}, owner)
|
||||
if err == nil || !strings.Contains(err.Error(), c.refusal) {
|
||||
t.Errorf("%s %s %s: %v, want %q", c.instance, c.operation, c.body, err, c.refusal)
|
||||
}
|
||||
}
|
||||
if last, _ := f.history.Last("decision.0290"); last != 0 {
|
||||
t.Fatalf("a refused operation was appended: %d", last)
|
||||
}
|
||||
}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
l, f := newLedger()
|
||||
c.run(t, l, f)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,204 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// The memory stores keep what the bus keeps, as the bus answers it, for a test of a ledger without a bus: a stream whose
|
||||
// sequence counts across every instance, a bucket with revisions that may be full, and snapshots written with an expected
|
||||
// sequence.
|
||||
|
||||
// MemoryHistory is a history in memory.
|
||||
type MemoryHistory struct {
|
||||
mu sync.Mutex
|
||||
seq uint64
|
||||
kept []struct {
|
||||
key string
|
||||
Kept
|
||||
}
|
||||
}
|
||||
|
||||
// Matches is whether a key pattern (`*` one name, `>` the rest) names a key.
|
||||
func Matches(pattern, key string) bool {
|
||||
p, k := strings.Split(pattern, "."), strings.Split(key, ".")
|
||||
for i, part := range p {
|
||||
if part == ">" {
|
||||
return len(k) > i
|
||||
}
|
||||
if i >= len(k) || (part != "*" && part != k[i]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return len(p) == len(k)
|
||||
}
|
||||
|
||||
func (h *MemoryHistory) Last(key string) (uint64, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
last := uint64(0)
|
||||
for _, k := range h.kept {
|
||||
if k.key == key {
|
||||
last = k.Sequence
|
||||
}
|
||||
}
|
||||
return last, nil
|
||||
}
|
||||
|
||||
func (h *MemoryHistory) Append(key string, op []byte, expect uint64) (uint64, error) {
|
||||
last, _ := h.Last(key)
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if last != expect {
|
||||
return 0, ErrRaced
|
||||
}
|
||||
h.seq++
|
||||
h.kept = append(h.kept, struct {
|
||||
key string
|
||||
Kept
|
||||
}{key, Kept{Sequence: h.seq, Data: op}})
|
||||
return h.seq, nil
|
||||
}
|
||||
|
||||
func (h *MemoryHistory) Read(keys string, from uint64, limit int) ([]Kept, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
var out []Kept
|
||||
for _, k := range h.kept {
|
||||
if k.Sequence >= from && Matches(keys, k.key) && (limit == 0 || len(out) < limit) {
|
||||
out = append(out, k.Kept)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Expire drops every operation of the history, as a ledger's keeping days end them.
|
||||
func (h *MemoryHistory) Expire() {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.kept = nil
|
||||
}
|
||||
|
||||
type memoryValue struct {
|
||||
value []byte
|
||||
rev uint64
|
||||
}
|
||||
|
||||
// MemoryState is a state bucket in memory.
|
||||
type MemoryState struct {
|
||||
mu sync.Mutex
|
||||
rev uint64
|
||||
// Full makes every write refused as a full bucket's.
|
||||
Full bool
|
||||
kv map[string]memoryValue
|
||||
}
|
||||
|
||||
func (s *MemoryState) Get(key string) ([]byte, uint64, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
v := s.kv[key]
|
||||
return v.value, v.rev, nil
|
||||
}
|
||||
|
||||
func (s *MemoryState) Write(key string, value []byte, revision uint64) (uint64, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.Full {
|
||||
return 0, ErrFull
|
||||
}
|
||||
if s.kv[key].rev != revision {
|
||||
return 0, ErrChanged
|
||||
}
|
||||
if s.kv == nil {
|
||||
s.kv = map[string]memoryValue{}
|
||||
}
|
||||
s.rev++
|
||||
s.kv[key] = memoryValue{value, s.rev}
|
||||
return s.rev, nil
|
||||
}
|
||||
|
||||
func (s *MemoryState) Keys() ([]string, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var out []string
|
||||
for k := range s.kv {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// MemorySnapshots are snapshots in memory.
|
||||
type MemorySnapshots struct {
|
||||
mu sync.Mutex
|
||||
at uint64
|
||||
kv map[string]memoryValue
|
||||
}
|
||||
|
||||
func (s *MemorySnapshots) Get(key string) ([]byte, uint64, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
v := s.kv[key]
|
||||
return v.value, v.rev, nil
|
||||
}
|
||||
|
||||
func (s *MemorySnapshots) Put(key string, value []byte, expect uint64) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.kv[key].rev != expect {
|
||||
return ErrRaced
|
||||
}
|
||||
if s.kv == nil {
|
||||
s.kv = map[string]memoryValue{}
|
||||
}
|
||||
s.at++
|
||||
s.kv[key] = memoryValue{value, s.at}
|
||||
return nil
|
||||
}
|
||||
|
||||
// MemoryIndex is an index's bucket in memory.
|
||||
type MemoryIndex struct {
|
||||
mu sync.Mutex
|
||||
kv map[string][]byte
|
||||
}
|
||||
|
||||
func (x *MemoryIndex) Entries(pattern string, limit int) ([]Indexed, error) {
|
||||
x.mu.Lock()
|
||||
defer x.mu.Unlock()
|
||||
if pattern == "" {
|
||||
pattern = ">"
|
||||
}
|
||||
var keys []string
|
||||
for k := range x.kv {
|
||||
if Matches(pattern, k) {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
}
|
||||
sort.Strings(keys)
|
||||
var out []Indexed
|
||||
for _, k := range keys {
|
||||
if limit > 0 && len(out) >= limit {
|
||||
break
|
||||
}
|
||||
out = append(out, Indexed{Key: k, Value: x.kv[k]})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (x *MemoryIndex) Put(key string, value []byte) error {
|
||||
x.mu.Lock()
|
||||
defer x.mu.Unlock()
|
||||
if x.kv == nil {
|
||||
x.kv = map[string][]byte{}
|
||||
}
|
||||
x.kv[key] = value
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *MemoryIndex) Delete(key string) error {
|
||||
x.mu.Lock()
|
||||
defer x.mu.Unlock()
|
||||
delete(x.kv, key)
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// Reading a ledger (ADR 0597 §10, design 25 § 12.5; issue 1966 T3): an instance's state, its history, a watch of its
|
||||
// states, and a declared index. Only the owning module reads its ledger: the runtime opens a ledger for the module whose
|
||||
// bundle asks, never one the request names.
|
||||
|
||||
// Index is one declared index's bucket, kept by a watcher in the owning module's process and rebuilt by reindex (ADR
|
||||
// 0597 §4).
|
||||
type Index interface {
|
||||
// Entries is the entries whose key a pattern names (`*` one name, `>` the rest; empty is every key), at most limit
|
||||
// (every one when 0), each as the bucket holds it.
|
||||
Entries(pattern string, limit int) ([]Indexed, error)
|
||||
// Put writes one entry; Delete removes one.
|
||||
Put(key string, value []byte) error
|
||||
Delete(key string) error
|
||||
}
|
||||
|
||||
// Indexed is one key of an index's bucket and its value, a buscalls.IndexEntry as JSON.
|
||||
type Indexed struct {
|
||||
Key string
|
||||
Value []byte
|
||||
}
|
||||
|
||||
// Watcher is a state bucket that hands each instance's record as it is and as it changes, under a key pattern.
|
||||
type Watcher interface {
|
||||
Watch(pattern string, deliver func(key string, value []byte, current bool) error) (func(), error)
|
||||
}
|
||||
|
||||
// HistoryMost is the most operations one history read answers.
|
||||
const HistoryMost = 1000
|
||||
|
||||
// Get is an instance's state as the state holds it, and the sequence of its last operation in it; null and 0 for an
|
||||
// instance with none.
|
||||
func (l *Ledger) Get(r buscalls.InstanceRequest) (buscalls.InstanceState, error) {
|
||||
key, err := KeyOf(r.Instance)
|
||||
if err != nil {
|
||||
return buscalls.InstanceState{}, err
|
||||
}
|
||||
rec, _, err := l.readState(key)
|
||||
if err != nil {
|
||||
return buscalls.InstanceState{}, err
|
||||
}
|
||||
state := rec.State
|
||||
if len(state) == 0 {
|
||||
state = json.RawMessage("null")
|
||||
}
|
||||
return buscalls.InstanceState{Instance: r.Instance, State: state, Sequence: rec.Sequence}, nil
|
||||
}
|
||||
|
||||
// operationsOf reads kept operations as the history answers them, each with its sequence.
|
||||
func operationsOf(kept []Kept) ([]buscalls.Operation, error) {
|
||||
out := make([]buscalls.Operation, 0, len(kept))
|
||||
for _, k := range kept {
|
||||
var o buscalls.Operation
|
||||
if err := json.Unmarshal(k.Data, &o); err != nil {
|
||||
return nil, fmt.Errorf("the history's operation %d is not one: %w", k.Sequence, err)
|
||||
}
|
||||
o.Sequence = k.Sequence
|
||||
out = append(out, o)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ReadHistory is an instance's operations from a sequence on, oldest first, each with its sequence and its proven author.
|
||||
func (l *Ledger) ReadHistory(r buscalls.HistoryRequest) (buscalls.HistoryPage, error) {
|
||||
key, err := KeyOf(r.Instance)
|
||||
if err != nil {
|
||||
return buscalls.HistoryPage{}, err
|
||||
}
|
||||
limit := r.Limit
|
||||
if limit <= 0 || limit > HistoryMost {
|
||||
limit = HistoryMost
|
||||
}
|
||||
last, err := l.History.Last(key)
|
||||
if err != nil {
|
||||
return buscalls.HistoryPage{}, err
|
||||
}
|
||||
kept, err := l.History.Read(key, r.From, limit+1)
|
||||
if err != nil {
|
||||
return buscalls.HistoryPage{}, err
|
||||
}
|
||||
page := buscalls.HistoryPage{Last: last}
|
||||
if len(kept) > limit {
|
||||
page.Next = kept[limit].Sequence
|
||||
kept = kept[:limit]
|
||||
}
|
||||
if page.Operations, err = operationsOf(kept); err != nil {
|
||||
return buscalls.HistoryPage{}, err
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
|
||||
// Watch hands deliver the state of every instance Instances names (`*` one name, `>` the rest, in an instance's path),
|
||||
// the current ones first, then each change.
|
||||
func (l *Ledger) Watch(r buscalls.LedgerWatchRequest, deliver func(buscalls.InstanceChange) error) (func(), error) {
|
||||
w, ok := l.State.(Watcher)
|
||||
if !ok {
|
||||
return nil, errors.New("this ledger's state cannot be watched")
|
||||
}
|
||||
pattern := ">"
|
||||
if r.Instances != "" {
|
||||
pattern = keyPattern(r.Instances)
|
||||
}
|
||||
return w.Watch(pattern, func(key string, value []byte, current bool) error {
|
||||
var rec Record
|
||||
if value != nil {
|
||||
if err := json.Unmarshal(value, &rec); err != nil {
|
||||
return fmt.Errorf("the state of %s is not a record: %w", key, err)
|
||||
}
|
||||
}
|
||||
state := rec.State
|
||||
if len(state) == 0 {
|
||||
state = json.RawMessage("null")
|
||||
}
|
||||
return deliver(buscalls.InstanceChange{Watch: r.Watch, Instance: InstanceOf(key), State: state, Sequence: rec.Sequence,
|
||||
Current: current})
|
||||
})
|
||||
}
|
||||
|
||||
// keyPattern is an instance pattern as a key pattern: its path's parts joined by dots.
|
||||
func keyPattern(instances string) string {
|
||||
out := []byte(instances)
|
||||
for i, c := range out {
|
||||
if c == '/' {
|
||||
out[i] = '.'
|
||||
}
|
||||
}
|
||||
return string(out)
|
||||
}
|
||||
|
||||
// index is a declared index by its name, or the reason there is none.
|
||||
func (l *Ledger) index(name string) (Index, error) {
|
||||
if ix, ok := l.Indexes[name]; ok {
|
||||
return ix, nil
|
||||
}
|
||||
return nil, fmt.Errorf("%s's ledger %s declares no index %q", l.Module, l.Name, name)
|
||||
}
|
||||
|
||||
// Query is a declared index's entries whose key Key names.
|
||||
func (l *Ledger) Query(r buscalls.QueryRequest) (buscalls.QueryAnswer, error) {
|
||||
ix, err := l.index(r.Index)
|
||||
if err != nil {
|
||||
return buscalls.QueryAnswer{}, err
|
||||
}
|
||||
held, err := ix.Entries(r.Key, r.Limit)
|
||||
if err != nil {
|
||||
return buscalls.QueryAnswer{}, err
|
||||
}
|
||||
entries := make([]buscalls.IndexEntry, 0, len(held))
|
||||
for _, h := range held {
|
||||
var e buscalls.IndexEntry
|
||||
if err := json.Unmarshal(h.Value, &e); err != nil {
|
||||
return buscalls.QueryAnswer{}, fmt.Errorf("the index %s's entry %s is not one: %w", r.Index, h.Key, err)
|
||||
}
|
||||
entries = append(entries, e)
|
||||
}
|
||||
return buscalls.QueryAnswer{Entries: entries}, nil
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// 1966/AC15, the other reads of a ledger (get, query) and its repairs (ADR 0597 §8, §10; design 25 § 12.5).
|
||||
func TestReadsAndRepairsOfALedger(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
run func(t *testing.T, l *Ledger, f fakes)
|
||||
}{
|
||||
{"1966/AC15 history answers an instance's operations oldest first with sequence and author", func(t *testing.T, l *Ledger, f fakes) {
|
||||
op := buscalls.Author{Account: "ledger.anchor.m", Caller: "op@shanks"}
|
||||
applied(t, l, "decision/1", "note", `{"says":"one"}`, owner)
|
||||
applied(t, l, "decision/2", "note", `{"says":"elsewhere"}`, owner)
|
||||
applied(t, l, "decision/1", "title", `{"title":"Two"}`, op)
|
||||
applied(t, l, "decision/1", "note", `{"says":"three"}`, owner)
|
||||
page, err := l.ReadHistory(buscalls.HistoryRequest{Instance: "decision/1", Limit: 2})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var got []string
|
||||
for _, o := range page.Operations {
|
||||
got = append(got, fmt.Sprintf("%s@%d %s %s", o.Operation, o.Sequence, o.Author.Account, o.Author.Caller))
|
||||
}
|
||||
if strings.Join(got, "; ") != "note@1 ledger.anchor.m ; title@3 ledger.anchor.m op@shanks" || page.Next != 4 || page.Last != 4 {
|
||||
t.Fatalf("the history is %q, next %d, last %d", got, page.Next, page.Last)
|
||||
}
|
||||
rest, _ := l.ReadHistory(buscalls.HistoryRequest{Instance: "decision/1", From: page.Next})
|
||||
if len(rest.Operations) != 1 || rest.Operations[0].Sequence != 4 || rest.Next != 0 {
|
||||
t.Fatalf("read on: %+v", rest)
|
||||
}
|
||||
}},
|
||||
{"get answers the state and its sequence, null for an instance with none", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
|
||||
got, err := l.Get(buscalls.InstanceRequest{Instance: "decision/1"})
|
||||
if err != nil || got.Sequence != 1 || !strings.Contains(string(got.State), `"title":"One"`) {
|
||||
t.Fatalf("got %+v, %v", got, err)
|
||||
}
|
||||
if none, err := l.Get(buscalls.InstanceRequest{Instance: "decision/9"}); err != nil || string(none.State) != "null" {
|
||||
t.Fatalf("none is %+v, %v", none, err)
|
||||
}
|
||||
}},
|
||||
{"query reads a declared index by key, and an undeclared one is refused", func(t *testing.T, l *Ledger, f fakes) {
|
||||
_ = f.index.Put("One.decision.1", []byte(`{"instance":"decision/1","sequence":1}`))
|
||||
_ = f.index.Put("Two.decision.2", []byte(`{"instance":"decision/2","sequence":2}`))
|
||||
got, err := l.Query(buscalls.QueryRequest{Index: "by-title", Key: "One.>"})
|
||||
if err != nil || len(got.Entries) != 1 || got.Entries[0] != (buscalls.IndexEntry{Instance: "decision/1", Sequence: 1}) {
|
||||
t.Fatalf("queried %+v, %v", got, err)
|
||||
}
|
||||
if _, err := l.Query(buscalls.QueryRequest{Index: "by-date"}); err == nil || !strings.Contains(err.Error(), `no index "by-date"`) {
|
||||
t.Fatalf("an undeclared index: %v", err)
|
||||
}
|
||||
}},
|
||||
{"a lost state rebuilds from the history, an expired one from its snapshot, and lag says what is behind", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
|
||||
applied(t, l, "decision/1", "note", `{"says":"kept"}`, owner)
|
||||
want, _, _ := f.state.Get("decision.1")
|
||||
f.state.kv = nil
|
||||
lag, _ := l.Lag(buscalls.LedgerRequest{})
|
||||
if len(lag.Instances) != 1 || lag.Instances[0].History != 2 || lag.Instances[0].State != 0 || lag.Instances[0].Snapshot != 2 {
|
||||
t.Fatalf("lag says %+v", lag)
|
||||
}
|
||||
if got, err := l.Rebuild(buscalls.RebuildRequest{}); err != nil || got.Written != 1 {
|
||||
t.Fatalf("rebuilt %+v, %v", got, err)
|
||||
}
|
||||
if got, _, _ := f.state.Get("decision.1"); string(record(t, got).State) != string(record(t, want).State) {
|
||||
t.Fatalf("rebuilt from the history %s, want %s", got, want)
|
||||
}
|
||||
f.state.kv = nil
|
||||
f.history.Expire()
|
||||
if got, err := l.Rebuild(buscalls.RebuildRequest{Instance: "decision/1", From: buscalls.RebuildFromSnapshot}); err != nil || got.Written != 1 {
|
||||
t.Fatalf("rebuilt %+v, %v", got, err)
|
||||
}
|
||||
if got, _, _ := f.state.Get("decision.1"); string(got) != string(want) {
|
||||
t.Fatalf("rebuilt from the snapshot %s, want %s", got, want)
|
||||
}
|
||||
}},
|
||||
{"reindex builds a declared index from the states, and lag says an index behind until it does", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
|
||||
applied(t, l, "decision/2", "note", `{"says":"untitled"}`, owner)
|
||||
_ = f.index.Put("stale.decision.9", []byte(`{}`))
|
||||
if lag, _ := l.Lag(buscalls.LedgerRequest{}); len(lag.Indexes) != 1 || lag.Indexes[0].Behind != 1 {
|
||||
t.Fatalf("lag says %+v", lag)
|
||||
}
|
||||
if got, err := l.Reindex(buscalls.ReindexRequest{Index: "by-title"}); err != nil || got.Written != 1 {
|
||||
t.Fatalf("reindexed %+v, %v", got, err)
|
||||
}
|
||||
entries, _ := f.index.Entries("", 0)
|
||||
if len(entries) != 1 || entries[0].Key != buscalls.IndexKey([]string{"One"}, "decision/1") || string(entries[0].Value) != `{"instance":"decision/1","sequence":1}` {
|
||||
t.Fatalf("the index holds %+v", entries)
|
||||
}
|
||||
if lag, _ := l.Lag(buscalls.LedgerRequest{}); len(lag.Indexes) != 0 || len(lag.Instances) != 0 {
|
||||
t.Fatalf("lag after reindex says %+v", lag)
|
||||
}
|
||||
}},
|
||||
{"snapshot rewrites a snapshot behind its state", func(t *testing.T, l *Ledger, f fakes) {
|
||||
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
|
||||
f.snapshots.kv = nil
|
||||
if got, err := l.Snapshot(buscalls.InstanceRequest{}); err != nil || got.Written != 1 {
|
||||
t.Fatalf("snapshot %+v, %v", got, err)
|
||||
}
|
||||
state, _, _ := f.state.Get("decision.1")
|
||||
if snap, _, _ := f.snapshots.Get("decision.1"); string(snap) != string(state) {
|
||||
t.Fatalf("the snapshot %s, the state %s", snap, state)
|
||||
}
|
||||
}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
l, f := newLedger()
|
||||
c.run(t, l, f)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,304 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// The ledger's repairs (ADR 0597 §8, §10; design 25 § 12.5; issue 1966 T4): rebuild an instance's state from its history
|
||||
// or its snapshot, rewrite a snapshot behind its state, build a declared index again, and say what is behind its history.
|
||||
// The tool runner serves them as tools of every module owning a ledger, its own ledgers only (the runtime's repairs.go).
|
||||
|
||||
// instances is every instance the ledger knows, by key: those its state holds and those its history names.
|
||||
func (l *Ledger) instances() ([]string, error) {
|
||||
seen := map[string]bool{}
|
||||
keys, err := l.State.Keys()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, k := range keys {
|
||||
seen[k] = true
|
||||
}
|
||||
kept, err := l.History.Read(">", 1, 0)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ops, err := operationsOf(kept)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, o := range ops {
|
||||
if k, err := KeyOf(o.Instance); err == nil {
|
||||
seen[k] = true
|
||||
}
|
||||
}
|
||||
out := make([]string, 0, len(seen))
|
||||
for k := range seen {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// keysOf is the one instance a request names, or every instance when it names none.
|
||||
func (l *Ledger) keysOf(instance string) ([]string, error) {
|
||||
if instance == "" {
|
||||
return l.instances()
|
||||
}
|
||||
k, err := KeyOf(instance)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return []string{k}, nil
|
||||
}
|
||||
|
||||
// Rebuild derives the state of an instance (every instance when it names none) again, from its whole history or from its
|
||||
// snapshot and the operations after it, and writes it, and its snapshot when that is behind. A derivation further back
|
||||
// than the state holds is not written: the state never goes back (ADR 0597 §5).
|
||||
func (l *Ledger) Rebuild(r buscalls.RebuildRequest) (buscalls.Repaired, error) {
|
||||
from := r.From
|
||||
if from == "" {
|
||||
from = buscalls.RebuildFromHistory
|
||||
}
|
||||
if from != buscalls.RebuildFromHistory && from != buscalls.RebuildFromSnapshot {
|
||||
return buscalls.Repaired{}, fmt.Errorf("a rebuild is from %q or %q, not %q", buscalls.RebuildFromHistory, buscalls.RebuildFromSnapshot, from)
|
||||
}
|
||||
keys, err := l.keysOf(r.Instance)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
var out buscalls.Repaired
|
||||
for _, key := range keys {
|
||||
base := Record{}
|
||||
if from == buscalls.RebuildFromSnapshot {
|
||||
if base, _, err = l.readSnapshot(key); err != nil {
|
||||
return out, err
|
||||
}
|
||||
}
|
||||
derived, err := l.replay(key, base, 0)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
held, rev, err := l.readState(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if derived.Sequence < held.Sequence {
|
||||
continue
|
||||
}
|
||||
now, _, err := l.settle(key, derived, rev)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if err := l.writeSnapshot(key, now); err != nil {
|
||||
return out, err
|
||||
}
|
||||
out.Written++
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Snapshot rewrites the snapshot of an instance (every instance when it names none) that is behind its state.
|
||||
func (l *Ledger) Snapshot(r buscalls.InstanceRequest) (buscalls.Repaired, error) {
|
||||
keys, err := l.keysOf(r.Instance)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
var out buscalls.Repaired
|
||||
for _, key := range keys {
|
||||
held, _, err := l.readState(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
snap, at, err := l.readSnapshot(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if held.Sequence == 0 || (at != 0 && snap.Sequence >= held.Sequence) {
|
||||
continue
|
||||
}
|
||||
if err := l.writeSnapshot(key, held); err != nil {
|
||||
return out, err
|
||||
}
|
||||
out.Written++
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// indexEntryOf is an instance's entry in a declared index, as reindex writes it and the owner's watcher keeps it
|
||||
// (mesh-sdk buscalls.IndexKey, buscalls.IndexEntry): its key the values of the fields the index is on, in their order,
|
||||
// then the instance; its value the instance and the sequence of the state it was taken from. False when the state holds
|
||||
// none of those fields. A value that is not text is written as its JSON.
|
||||
func indexEntryOf(on []string, key string, rec Record) (string, []byte, bool) {
|
||||
var state map[string]any
|
||||
if json.Unmarshal(rec.State, &state) != nil || state == nil {
|
||||
return "", nil, false
|
||||
}
|
||||
values := make([]string, 0, len(on))
|
||||
found := false
|
||||
for _, f := range on {
|
||||
v, has := state[f]
|
||||
found = found || has
|
||||
s, ok := v.(string)
|
||||
if !ok {
|
||||
raw, _ := json.Marshal(v)
|
||||
s = string(raw)
|
||||
}
|
||||
values = append(values, s)
|
||||
}
|
||||
if !found {
|
||||
return "", nil, false
|
||||
}
|
||||
instance := InstanceOf(key)
|
||||
value, _ := json.Marshal(buscalls.IndexEntry{Instance: instance, Sequence: rec.Sequence})
|
||||
return buscalls.IndexKey(values, instance), value, true
|
||||
}
|
||||
|
||||
// declared is the fields an index is on and the types declaring it.
|
||||
func (l *Ledger) declared(index string) ([]string, map[string]bool, error) {
|
||||
types := map[string]bool{}
|
||||
var on []string
|
||||
for _, t := range l.Types {
|
||||
for _, ix := range t.Indexes {
|
||||
if ix.Name == index {
|
||||
types[t.Name] = true
|
||||
on = ix.On
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(types) == 0 {
|
||||
return nil, nil, fmt.Errorf("%s's ledger %s declares no index %q", l.Module, l.Name, index)
|
||||
}
|
||||
return on, types, nil
|
||||
}
|
||||
|
||||
// wanted is every entry an index should hold, from the states of the types declaring it.
|
||||
func (l *Ledger) wanted(index string) (map[string][]byte, map[string]uint64, error) {
|
||||
on, types, err := l.declared(index)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
keys, err := l.instances()
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
want, seqs := map[string][]byte{}, map[string]uint64{}
|
||||
for _, key := range keys {
|
||||
name, _, _ := strings.Cut(key, ".")
|
||||
if !types[name] {
|
||||
continue
|
||||
}
|
||||
held, _, err := l.readState(key)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if k, v, ok := indexEntryOf(on, key, held); ok {
|
||||
want[k], seqs[k] = v, held.Sequence
|
||||
}
|
||||
}
|
||||
return want, seqs, nil
|
||||
}
|
||||
|
||||
// Reindex builds a declared index again from the ledger's states, derived from its history: every entry it should hold
|
||||
// written, every other removed.
|
||||
func (l *Ledger) Reindex(r buscalls.ReindexRequest) (buscalls.Repaired, error) {
|
||||
ix, err := l.index(r.Index)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
want, _, err := l.wanted(r.Index)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
held, err := ix.Entries("", 0)
|
||||
if err != nil {
|
||||
return buscalls.Repaired{}, err
|
||||
}
|
||||
var out buscalls.Repaired
|
||||
for _, e := range held {
|
||||
if _, keep := want[e.Key]; !keep {
|
||||
if err := ix.Delete(e.Key); err != nil {
|
||||
return out, err
|
||||
}
|
||||
}
|
||||
}
|
||||
keys := make([]string, 0, len(want))
|
||||
for k := range want {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
if err := ix.Put(k, want[k]); err != nil {
|
||||
return out, err
|
||||
}
|
||||
out.Written++
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Lag is every instance whose state or snapshot is behind its history, and how many entries each declared index lacks
|
||||
// or holds from an older state; nothing when nothing is behind.
|
||||
func (l *Ledger) Lag(buscalls.LedgerRequest) (buscalls.Lagged, error) {
|
||||
keys, err := l.instances()
|
||||
if err != nil {
|
||||
return buscalls.Lagged{}, err
|
||||
}
|
||||
var out buscalls.Lagged
|
||||
for _, key := range keys {
|
||||
last, err := l.History.Last(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
held, _, err := l.readState(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
snap, _, err := l.readSnapshot(key)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if held.Sequence < last || snap.Sequence < last {
|
||||
out.Instances = append(out.Instances, buscalls.InstanceLag{Instance: InstanceOf(key), History: last,
|
||||
State: held.Sequence, Snapshot: snap.Sequence})
|
||||
}
|
||||
}
|
||||
names := make([]string, 0, len(l.Indexes))
|
||||
for name := range l.Indexes {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, name := range names {
|
||||
if _, _, err := l.declared(name); err != nil {
|
||||
// A bucket bound for an index no type declares has nothing it should hold.
|
||||
continue
|
||||
}
|
||||
want, seqs, err := l.wanted(name)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
held, err := l.Indexes[name].Entries("", 0)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
at := map[string]uint64{}
|
||||
for _, e := range held {
|
||||
var v buscalls.IndexEntry
|
||||
_ = json.Unmarshal(e.Value, &v)
|
||||
at[e.Key] = v.Sequence
|
||||
}
|
||||
behind := uint64(0)
|
||||
for k := range want {
|
||||
if s, has := at[k]; !has || s < seqs[k] {
|
||||
behind++
|
||||
}
|
||||
}
|
||||
if behind > 0 {
|
||||
out.Indexes = append(out.Indexes, buscalls.IndexLag{Index: name, Behind: behind})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
package ledger
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// records is a ledger type of each effect: a title set, a section set as an entry, a note appended, a section dropped,
|
||||
// and the whole instance removed.
|
||||
var records = catalogue.LedgerType{
|
||||
Name: "decision",
|
||||
Schema: map[string]catalogue.LedgerField{
|
||||
"title": {Type: "string"}, "text": {Type: "string"}, "digest": {Type: "string", DigestOf: "text"},
|
||||
},
|
||||
Operations: []catalogue.LedgerOperation{
|
||||
{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}, Sequence: true, Author: true},
|
||||
{Name: "section", Effect: catalogue.EffectSetEntry, Into: "sections", Key: "id", Fields: []string{"digest"}},
|
||||
{Name: "note", Effect: catalogue.EffectAppend, Into: "notes", Fields: []string{"says"}},
|
||||
{Name: "drop-section", Effect: catalogue.EffectRemove, Into: "sections", Key: "id"},
|
||||
{Name: "withdraw", Effect: catalogue.EffectRemove},
|
||||
},
|
||||
Indexes: []catalogue.LedgerIndex{{Name: "by-title", On: []string{"title"}}},
|
||||
}
|
||||
|
||||
type fakes struct {
|
||||
history *MemoryHistory
|
||||
state *MemoryState
|
||||
snapshots *MemorySnapshots
|
||||
index *MemoryIndex
|
||||
}
|
||||
|
||||
func newLedger() (*Ledger, fakes) {
|
||||
f := fakes{&MemoryHistory{}, &MemoryState{}, &MemorySnapshots{}, &MemoryIndex{}}
|
||||
return &Ledger{Module: "m", Name: "records", Types: []catalogue.LedgerType{records}, History: f.history, State: f.state,
|
||||
Snapshots: f.snapshots, Indexes: map[string]Index{"by-title": f.index}}, f
|
||||
}
|
||||
|
||||
// applied applies an operation and fails the test on a refusal.
|
||||
func applied(t *testing.T, l *Ledger, instance, operation, body string, author buscalls.Author) buscalls.Applied {
|
||||
t.Helper()
|
||||
got, err := l.Apply(buscalls.ApplyRequest{Ledger: l.Name, Instance: instance, Operation: operation, Body: json.RawMessage(body)}, author)
|
||||
if err != nil {
|
||||
t.Fatalf("%s %s: %v", operation, body, err)
|
||||
}
|
||||
return got
|
||||
}
|
||||
|
||||
// record is the instance's record a store holds, read as a Record.
|
||||
func record(t *testing.T, raw []byte) Record {
|
||||
t.Helper()
|
||||
var r Record
|
||||
if raw != nil {
|
||||
if err := json.Unmarshal(raw, &r); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return r
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
// Package meshtest raises what the controller would, for tests against a real bus: the ASSIGNMENTS
|
||||
// Package meshtest raises what the mesh would, for tests against a real bus: the ASSIGNMENTS
|
||||
// and EVENTS streams, memberships issued by hand, and a fixture's path.
|
||||
//
|
||||
// **Every test has a bus of its own** (novox/hq ADR 0237, the controller's internal/testbus): a server
|
||||
// **Every test has a bus of its own** (novox/hq ADR 0237, the old testbus): a server
|
||||
// linked in at the release go.mod pins — the release the mesh runs, held so by a test beside this one —
|
||||
// started for the one test and gone after it. The packages shared one bus and had to run one at a time:
|
||||
// each test raised the streams afresh, and the console discovers every runtime that announces itself on
|
||||
@@ -11,7 +11,11 @@
|
||||
package meshtest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"git.novox.be/novox/mesh-sdk/go/buscompose"
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
@@ -22,6 +26,7 @@ import (
|
||||
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
@@ -67,7 +72,7 @@ func URL(t *testing.T) string {
|
||||
// buses are the running tests' buses, by test.
|
||||
var buses sync.Map
|
||||
|
||||
// Mesh is the controller's job, done by hand.
|
||||
// Mesh is the mesh's job, done by hand.
|
||||
type Mesh struct {
|
||||
nc *nats.Conn
|
||||
js nats.JetStreamContext
|
||||
@@ -81,11 +86,10 @@ func New(t *testing.T) *Mesh {
|
||||
t.Fatal(err)
|
||||
}
|
||||
js, _ := nc.JetStream()
|
||||
for _, s := range []string{"ASSIGNMENTS", "EVENTS"} {
|
||||
_ = js.DeleteStream(s)
|
||||
}
|
||||
if _, err := js.AddStream(&nats.StreamConfig{Name: "ASSIGNMENTS", Subjects: []string{"mesh.assignment.>"},
|
||||
MaxMsgsPerSubject: 1, AllowDirect: true}); err != nil {
|
||||
_ = js.DeleteStream("EVENTS")
|
||||
// The grants part's state memberships, where a runtime reads what is issued it (ADR 0571 clause 10).
|
||||
_ = js.DeleteKeyValue(buscompose.GrantsMembershipsBucket)
|
||||
if _, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: buscompose.GrantsMembershipsBucket, History: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := js.AddStream(&nats.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
|
||||
@@ -95,15 +99,119 @@ func New(t *testing.T) *Mesh {
|
||||
return &Mesh{nc: nc, js: js}
|
||||
}
|
||||
|
||||
// Issue publishes a membership.
|
||||
// Issue writes a membership where the grants part keeps it, its state memberships, key `<node>.<module>` (novox/hq ADR
|
||||
// 0571 clause 7 and 10; mesh-sdk buscompose.GrantsMembershipsBucket), the one place a runtime reads it. It writes with
|
||||
// mesh-sdk's encoder, the grants part's own, never json of this package's type: a value built here by hand agreed with
|
||||
// the reader's mistake and hid that the part wrote another shape (issue 774).
|
||||
func (m *Mesh) Issue(t *testing.T, mem bus.Membership) {
|
||||
t.Helper()
|
||||
body, _ := json.Marshal(mem)
|
||||
if _, err := m.js.Publish(bus.MembershipSubject(mem.Node, mem.Module), body); err != nil {
|
||||
m.putGrants(t, mem, membership.EncodeGrantsMembership(sdkMembership(t, mem)))
|
||||
}
|
||||
|
||||
// IssueInGrantsWrapped writes a membership as the grants part wrote it before mesh-sdk's codec (until 2026-10-30, what
|
||||
// the live mesh holds until the part writes again): its pass record, the membership inside, published saying what
|
||||
// became of it ("published", "unchanged", or why it was held back).
|
||||
func (m *Mesh) IssueInGrantsWrapped(t *testing.T, mem bus.Membership, published string) {
|
||||
t.Helper()
|
||||
body, _ := json.Marshal(map[string]any{"node": mem.Node, "module": mem.Module, "digest": "sha256:0",
|
||||
"membership": sdkMembership(t, mem), "published": published})
|
||||
m.putGrants(t, mem, body)
|
||||
}
|
||||
|
||||
func (m *Mesh) putGrants(t *testing.T, mem bus.Membership, body []byte) {
|
||||
t.Helper()
|
||||
kv, err := m.js.KeyValue(buscompose.GrantsMembershipsBucket)
|
||||
if err != nil {
|
||||
if kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: buscompose.GrantsMembershipsBucket, History: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if _, err := kv.Put(buscompose.MembershipKey(mem.Node, mem.Module), body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// DropGrants deletes the grants part's state memberships, as a bus that has not made it yet holds none; Issue makes it again.
|
||||
func (m *Mesh) DropGrants(t *testing.T) {
|
||||
t.Helper()
|
||||
if err := m.js.DeleteKeyValue(buscompose.GrantsMembershipsBucket); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// sdkMembership is a membership of this package's type as mesh-sdk's, which the grants part composes and writes.
|
||||
func sdkMembership(t *testing.T, mem bus.Membership) membership.Membership {
|
||||
t.Helper()
|
||||
raw, _ := json.Marshal(mem)
|
||||
var out membership.Membership
|
||||
if err := json.Unmarshal(raw, &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ConsoleReads are the states a console's runtime reads to say what the mesh holds (novox/hq ADR 0571 clauses 4, 5):
|
||||
// placement, the indexer's modules and manifests, and the seat holders.
|
||||
func ConsoleReads() []bus.StateIssued {
|
||||
return []bus.StateIssued{
|
||||
{Name: delivery.PlacementRead, Bucket: delivery.PlacementBucket},
|
||||
{Name: delivery.IndexerModule + ".modules", Bucket: delivery.ModulesBucket},
|
||||
{Name: delivery.IndexerModule + ".manifests", Bucket: delivery.ManifestsBucket},
|
||||
{Name: delivery.SeatHoldersRead, Bucket: delivery.SeatHoldersBucket},
|
||||
}
|
||||
}
|
||||
|
||||
// Placed writes what the mesh holds as its parts write it, with mesh-sdk's encoders: each module the indexer holds with
|
||||
// its manifest (tools, the tools it declares; none for a module without), and each node's placement of modules.
|
||||
func (m *Mesh) Placed(t *testing.T, tools map[string][]string, on map[string][]string) {
|
||||
t.Helper()
|
||||
kv := func(bucket string) nats.KeyValue {
|
||||
_ = m.js.DeleteKeyValue(bucket)
|
||||
b, err := m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: bucket, History: 1, MaxValueSize: 256 * 1024})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
modules, manifests, placement := kv(delivery.ModulesBucket), kv(delivery.ManifestsBucket), kv(delivery.PlacementBucket)
|
||||
kv(delivery.SeatHoldersBucket)
|
||||
for module, declared := range tools {
|
||||
manifest, _ := json.Marshal(map[string]any{"module": module, "tools": declared})
|
||||
digest := delivery.ManifestDigest(manifest)
|
||||
raw, err := delivery.EncodeManifest(delivery.Manifest{Module: module, JSON: string(manifest)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := manifests.Put(digest, raw); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, err = delivery.EncodeModule(delivery.Module{None: "not built here", Manifest: digest, Repository: "novox/" + module,
|
||||
Commit: strings.Repeat("a", 40)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := modules.Put(module, raw); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for node, placed := range on {
|
||||
p := delivery.Placement{V: delivery.Version, Node: node}
|
||||
for _, module := range placed {
|
||||
p.Modules = append(p.Modules, delivery.PlacedModule{Module: module})
|
||||
}
|
||||
// As the seat writes it: its node key and a key per module (research 072 row 37).
|
||||
values, err := delivery.PlacementValues(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for key, raw := range values {
|
||||
if _, err := placement.Put(key, raw); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// NextEvent is the subject the next event under a pattern lands on.
|
||||
func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
|
||||
t.Helper()
|
||||
@@ -122,7 +230,7 @@ func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
|
||||
return ch
|
||||
}
|
||||
|
||||
// MembershipOf is a membership as the controller issues one on a machine.
|
||||
// MembershipOf is a membership as the mesh issues one on a machine.
|
||||
func MembershipOf(module, node string, plain bool, seats map[string][]string) bus.Membership {
|
||||
own := "mesh.mod." + module
|
||||
serves := []bus.Served{{Subject: own + ".tool.{tool}." + node}}
|
||||
@@ -183,7 +291,7 @@ func (l *Logs) Has(fragments ...string) bool {
|
||||
// All is every line.
|
||||
func (l *Logs) All() string { return strings.Join(l.lines, "\n") }
|
||||
|
||||
// Consumer makes a module's durable consumer on a machine as the controller does: pull, on the
|
||||
// Consumer makes a module's durable consumer on a machine as the mesh does: pull, on the
|
||||
// EVENTS stream, filtered to what the module consumes, with a short ack wait so a test sees a
|
||||
// redelivery in seconds rather than the mesh's minutes.
|
||||
func (m *Mesh) Consumer(t *testing.T, node, module string, filters []string, ackWait time.Duration) {
|
||||
@@ -209,6 +317,18 @@ func (m *Mesh) Emit(t *testing.T, subject string, body any) {
|
||||
}
|
||||
}
|
||||
|
||||
// Publish says body on a plain subject, as a core publish with no stream behind it.
|
||||
func (m *Mesh) Publish(t *testing.T, subject string, body any) {
|
||||
t.Helper()
|
||||
data, _ := json.Marshal(body)
|
||||
if err := m.nc.Publish(subject, data); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := m.nc.Flush(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Pending is what a module's consumer still holds: delivered and not acknowledged, and not yet
|
||||
// delivered.
|
||||
func (m *Mesh) Pending(t *testing.T, node, module string) (ackPending, notDelivered uint64) {
|
||||
@@ -231,7 +351,7 @@ func (m *Mesh) BucketKeeping(t *testing.T, name string, history uint8) nats.KeyV
|
||||
return kv
|
||||
}
|
||||
|
||||
// Bucket makes a module's state afresh as the controller does from the catalogue (novox/hq ADR 0201),
|
||||
// Bucket makes a module's state afresh as the mesh does from the catalogue (novox/hq ADR 0201),
|
||||
// and answers it for writing what is there before a bundle starts.
|
||||
func (m *Mesh) Bucket(t *testing.T, name string) nats.KeyValue {
|
||||
t.Helper()
|
||||
@@ -243,10 +363,27 @@ func (m *Mesh) Bucket(t *testing.T, name string) nats.KeyValue {
|
||||
return kv
|
||||
}
|
||||
|
||||
// Log makes a module's log afresh as the controller does from the catalogue (novox/hq ADR 0297): a file
|
||||
// BucketKeyTTL makes a module's state afresh as the mesh does for one declared `key-ttl` (mesh-sdk buscompose:
|
||||
// a time to live per key allowed, its end leaving a marker kept a minute; issue 1952).
|
||||
func (m *Mesh) BucketKeyTTL(t *testing.T, name string) {
|
||||
t.Helper()
|
||||
_ = m.js.DeleteKeyValue(name)
|
||||
js, err := jetstream.New(m.nc)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: 1, MaxValueSize: 256 * 1024,
|
||||
LimitMarkerTTL: time.Minute}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Log makes a module's log afresh as the mesh does from the catalogue (novox/hq ADR 0297): a file
|
||||
// stream named LOG_<module>_<name> on mesh.log.<module>.<name>.>, limits retention, no age, no cap per
|
||||
// key, the declared cap, 260 KiB a message (a 256 KiB value and its headers), refusing new entries when full, direct get, no delete or
|
||||
// purge. It answers the membership entry the controller issues for it.
|
||||
// purge. It answers the membership entry the mesh issues for it.
|
||||
func (m *Mesh) Log(t *testing.T, module, name string, maxBytes int64) bus.LogIssued {
|
||||
t.Helper()
|
||||
stream := "LOG_" + module + "_" + name
|
||||
@@ -272,7 +409,7 @@ func (m *Mesh) Consumers(t *testing.T, stream string) int {
|
||||
return info.State.Consumers
|
||||
}
|
||||
|
||||
// ModuleStream makes a module's own stream afresh as the controller does from the catalogue (novox/hq ADR
|
||||
// ModuleStream makes a module's own stream afresh as the mesh does from the catalogue (novox/hq ADR
|
||||
// 0363 §1): named in capitals, taking `mesh.stream.<module>.event.>`, a subject space EVENTS does not take.
|
||||
func (m *Mesh) ModuleStream(t *testing.T, module, name string) {
|
||||
t.Helper()
|
||||
@@ -318,7 +455,7 @@ func (m *Mesh) Messages(t *testing.T, stream string) uint64 {
|
||||
return info.State.Msgs
|
||||
}
|
||||
|
||||
// RemoveConsumer deletes a module's consumer on a machine on a stream, as the controller does when a grant
|
||||
// RemoveConsumer deletes a module's consumer on a machine on a stream, as the mesh does when a grant
|
||||
// it cannot narrow is withdrawn (novox/hq ADR 0363 §6).
|
||||
func (m *Mesh) RemoveConsumer(t *testing.T, stream, node, module string) {
|
||||
t.Helper()
|
||||
|
||||
@@ -110,6 +110,13 @@ func TestTheRepositoryHoldsNoCheckScriptAndNodeToolsIsItsOnlyModule(t *testing.T
|
||||
if !d.IsDir() && d.Name() == "module.json" {
|
||||
found = append(found, filepath.ToSlash(p))
|
||||
}
|
||||
// ADR 0604 section 4: the TypeScript that called the old catalog module's tools is gone, and nothing
|
||||
// under node-tools names one (the needle is built so this file does not name it either).
|
||||
if !d.IsDir() && strings.Contains(filepath.ToSlash(p), "/node-tools/") {
|
||||
if body, rerr := os.ReadFile(p); rerr == nil && strings.Contains(string(body), "catalog"+"_modules") {
|
||||
t.Errorf("%s names a catalog tool of the retired module", p)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Package provide is the runtime's half of node-tools being a provider of `mcp-endpoint`: it runs the mesh SDK's
|
||||
// provider helper, so a consumer handed to the provisioner (claude-code's `${bound:mcp-endpoint:port}`) is answered
|
||||
// from an entry node-tools writes in its own bucket, not from the controller's binding file (novox/hq ADR 0349
|
||||
// from an entry node-tools writes in its own bucket, not from the old binding file (novox/hq ADR 0349
|
||||
// clause 12, issue 791).
|
||||
//
|
||||
// The offer carries no credential: the console's authority is the machine's login (ADR 0152, 0175), so the helper
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/provider"
|
||||
"git.novox.be/novox/mesh-sdk/go/provisions"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
@@ -38,15 +39,18 @@ func Offers() map[string]provider.Offer {
|
||||
}
|
||||
|
||||
// busOf is the helper's bus over the runtime's own state: the provider's bucket is its state `bound`, and the
|
||||
// controller's assignments are the state it reads, `mesh-controller.assignments` (both declared in the manifest).
|
||||
// provisions and plans of the seat mesh-provisioner are the states it reads, by mesh-sdk's names (novox/hq ADR 0570,
|
||||
// issue 1954 L6-T1). The old assignments are read no more (mesh-sdk 692dbf9, L6-P3).
|
||||
type busOf struct{ conn *bus.Conn }
|
||||
|
||||
func (b busOf) state(bucket string) (string, error) {
|
||||
switch bucket {
|
||||
case provider.BucketOf(Module):
|
||||
return "bound", nil
|
||||
case provider.AssignmentsBucket:
|
||||
return "mesh-controller.assignments", nil
|
||||
case provisions.ProvisionsBucket:
|
||||
return provisions.ProvisionsRead, nil
|
||||
case provisions.PlansBucket:
|
||||
return provisions.PlansRead, nil
|
||||
}
|
||||
return "", fmt.Errorf("%s is not a bucket this provider keeps or reads", bucket)
|
||||
}
|
||||
@@ -57,7 +61,8 @@ func (b busOf) Watch(ctx context.Context, bucket string, handle func(key string,
|
||||
return err
|
||||
}
|
||||
stop, err := b.conn.StateWatch(Module, name, "", func(ch bus.StateChange) error {
|
||||
if ch.Op == "delete" {
|
||||
if ch.Op != "put" {
|
||||
// A delete, or an expiry (issue 1952): the key is gone.
|
||||
handle(ch.Key, nil)
|
||||
} else {
|
||||
handle(ch.Key, ch.Value)
|
||||
|
||||
@@ -7,6 +7,10 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/provider"
|
||||
"git.novox.be/novox/mesh-sdk/go/provisions"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
@@ -41,19 +45,39 @@ func fill(t *testing.T, template string, entry map[string]any) string {
|
||||
func TestTheRuntimeAnswersAHandedConsumersMcpEndpointThroughTheHelper(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
bound := mesh.Bucket(t, "node-tools_bound")
|
||||
assignments := mesh.Bucket(t, "mesh-controller_assignments")
|
||||
plans := mesh.Bucket(t, provisions.PlansBucket)
|
||||
provs := mesh.Bucket(t, provisions.ProvisionsBucket)
|
||||
|
||||
mem := mt.MembershipOf("node-tools", "anchor", false, nil)
|
||||
mem.State = []bus.StateIssued{
|
||||
{Name: "bound", Bucket: "node-tools_bound", Writes: true},
|
||||
{Name: "mesh-controller.assignments", Bucket: "mesh-controller_assignments"},
|
||||
{Name: provisions.PlansRead, Bucket: provisions.PlansBucket},
|
||||
{Name: provisions.ProvisionsRead, Bucket: provisions.ProvisionsBucket},
|
||||
}
|
||||
mesh.Issue(t, mem)
|
||||
|
||||
// claude-code, handed on this machine, is bound to node-tools here for mcp-endpoint (issue 788's plan entry).
|
||||
plan := `{"node":"anchor","module":"claude-code","provisioning":{"provisions":[{"provision":"mcp-endpoint",` +
|
||||
`"provider":"anchor","provider-module":"node-tools","at":"127.0.0.1","from":"anchor","as":"mesh_anchor_claude_code"}]}}`
|
||||
if _, err := assignments.Put("anchor.claude-code", []byte(plan)); err != nil {
|
||||
// claude-code, handed on this machine, is bound to node-tools here for mcp-endpoint: its plan and its provision
|
||||
// key, written as mesh-provisioner writes them, with mesh-sdk's encoders (ADR 0570 clauses 1-3, issue 1954).
|
||||
plan, err := provisions.EncodePlan(provisions.Plan{Node: "anchor", Module: "claude-code", Plan: catalogue.Plan{
|
||||
Provisions: []catalogue.PlanProvision{{Provision: "mcp-endpoint", Provider: "anchor", ProviderModule: "node-tools",
|
||||
At: "127.0.0.1", From: "anchor", As: "mesh_anchor_claude_code"}}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := plans.Put("anchor.claude-code", plan); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key, err := provisions.Key("anchor", "claude-code", "mcp-endpoint", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
asked, err := provisions.Encode(provisions.Provision{Provision: "mcp-endpoint", State: provisions.Asked,
|
||||
Consumer: provisions.End{Node: "anchor", Module: "claude-code"}, Provider: provisions.End{Node: "anchor", Module: "node-tools"},
|
||||
At: time.Now()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := provs.Put(key, asked); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -102,6 +126,8 @@ func TestTheOfferServesThePortTheManifestServes(t *testing.T) {
|
||||
var m struct {
|
||||
Serves map[string]map[string]any `json:"serves"`
|
||||
Moved bool `json:"moved"`
|
||||
State []string `json:"state"`
|
||||
Reads []string `json:"reads"`
|
||||
Provides []struct {
|
||||
Name string `json:"name"`
|
||||
AnsweredBy string `json:"answered-by"`
|
||||
@@ -133,11 +159,32 @@ func TestTheOfferServesThePortTheManifestServes(t *testing.T) {
|
||||
t.Error("the manifest still carries a consumers file the helper does not need")
|
||||
}
|
||||
}
|
||||
for _, r := range m.Reads {
|
||||
if r == "mesh-placement.assignments" {
|
||||
t.Error("the manifest still reads mesh-placement.assignments, which the sdk's helper no longer reads (issue 1954)")
|
||||
}
|
||||
}
|
||||
for _, a := range m.Build.Artifacts {
|
||||
if a.Name == "files" || len(a.Env) > 0 {
|
||||
t.Errorf("the build still carries %+v, which only a moved module needs", a)
|
||||
}
|
||||
}
|
||||
// Every bucket the sdk's helper reaches is a state the manifest keeps or reads, by the sdk's names: the provider's
|
||||
// own and the seat mesh-provisioner's provisions and plans (issue 1954 L6-T1); the controller's assignments no more.
|
||||
declared := map[string]bool{}
|
||||
for _, n := range append(m.State, m.Reads...) {
|
||||
declared[n] = true
|
||||
}
|
||||
for bucket, want := range map[string]string{
|
||||
provider.BucketOf(Module): "bound",
|
||||
provisions.ProvisionsBucket: "mesh-provisioner.provisions",
|
||||
provisions.PlansBucket: "mesh-provisioner.plans",
|
||||
} {
|
||||
got, err := (busOf{}).state(bucket)
|
||||
if err != nil || got != want || !declared[got] {
|
||||
t.Errorf("the helper's bucket %s reaches state %q (%v), want %q declared in the manifest", bucket, got, err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Only the machine's runtime is the provider: a process on another credential, or with no machine, answers nobody.
|
||||
@@ -147,3 +194,24 @@ func TestOnlyTheMachinesRuntimeStartsTheHelper(t *testing.T) {
|
||||
Start(nil, c.module, c.node, time.Second, said.Logf)()
|
||||
}
|
||||
}
|
||||
|
||||
// 1126/B19 (ADR 0590): this manifest is not refused by the check that refuses any manifest naming the controller:
|
||||
// it reads, invokes, claims and requires nothing of it.
|
||||
func TestTheManifestNamesNothingOfTheController(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../module.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, err := catalogue.Decode(raw)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Read as text, as mesh-catalog's controllerdeps check reads its own: mesh-sdk names no module (issue 774), and the
|
||||
// manifest says it is the tool runner rather than being known by its name.
|
||||
if strings.Contains(string(raw), "mesh-controller") {
|
||||
t.Error("node-tools' manifest names the ended controller")
|
||||
}
|
||||
if !catalogue.IsToolRunner(m) {
|
||||
t.Error("node-tools' manifest does not say it is the tool runner (\"tool-runner\": true)")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
)
|
||||
|
||||
// BusState answers a bundle's calls on its module's bus state in the new form (mesh-sdk buscalls; novox/hq ADR 0595
|
||||
// §9, design 25 § 12.5; issue 1966): each as the module whose bundle asks, never a module the request names, so a
|
||||
// bundle reaches its own module's bindings alone.
|
||||
func (b *moduleBus) BusState(method string, params json.RawMessage) (json.RawMessage, error) {
|
||||
conn := b.all.conn
|
||||
var answer any
|
||||
var err error
|
||||
switch method {
|
||||
case buscalls.ObjectsPut:
|
||||
var r buscalls.ObjectsPutRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.ObjectsPut(b.module, r)
|
||||
}
|
||||
case buscalls.ObjectsGet:
|
||||
var r buscalls.ObjectsNameRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
var got *buscalls.ObjectData
|
||||
if got, err = conn.ObjectsGet(b.module, r); err == nil && got == nil {
|
||||
return json.RawMessage("null"), nil
|
||||
}
|
||||
answer = got
|
||||
}
|
||||
case buscalls.ObjectsDelete:
|
||||
var r buscalls.ObjectsNameRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
err = conn.ObjectsDelete(b.module, r)
|
||||
answer = buscalls.Done{}
|
||||
}
|
||||
case buscalls.ObjectsList:
|
||||
var r buscalls.ObjectsRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.ObjectsList(b.module, r)
|
||||
}
|
||||
case buscalls.ObjectsInfo:
|
||||
var r buscalls.ObjectsRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.ObjectsInfo(b.module, r)
|
||||
}
|
||||
case buscalls.StreamRead:
|
||||
var r buscalls.StreamReadRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.StreamRead(b.module, r)
|
||||
}
|
||||
case buscalls.StreamFill:
|
||||
var r buscalls.StreamRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.StreamFill(b.module, r)
|
||||
}
|
||||
case buscalls.WorkQueueSize:
|
||||
var r buscalls.WorkQueueRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.WorkQueueSize(b.module, r)
|
||||
}
|
||||
case buscalls.WorkQueueDead:
|
||||
var r buscalls.WorkQueueDeadRequest
|
||||
if err = decode(method, params, &r); err == nil {
|
||||
answer, err = conn.WorkQueueDead(b.module, r)
|
||||
}
|
||||
case buscalls.LedgerApply, buscalls.LedgerGet, buscalls.LedgerHistory, buscalls.LedgerQuery, buscalls.LedgerRebuild,
|
||||
buscalls.LedgerReindex, buscalls.LedgerSnapshot, buscalls.LedgerLag:
|
||||
answer, err = b.ledgerCall(method, params)
|
||||
default:
|
||||
return nil, fmt.Errorf("the runtime answers no %s", method)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return json.Marshal(answer)
|
||||
}
|
||||
|
||||
// BusStateWatch hands a bundle each change of its objects or its ledger, the current ones first.
|
||||
func (b *moduleBus) BusStateWatch(method string, params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
|
||||
switch method {
|
||||
case buscalls.ObjectsWatch:
|
||||
var r buscalls.ObjectsWatchRequest
|
||||
if err := decode(method, params, &r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return b.all.conn.ObjectsWatch(b.module, r, func(change buscalls.ObjectsChange) error {
|
||||
raw, err := json.Marshal(change)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return deliver(raw)
|
||||
})
|
||||
case buscalls.LedgerWatch:
|
||||
return b.ledgerWatch(params, deliver)
|
||||
}
|
||||
return nil, fmt.Errorf("the runtime answers no %s", method)
|
||||
}
|
||||
|
||||
// decode reads a bus-state call's request.
|
||||
func decode(method string, params json.RawMessage, into any) error {
|
||||
if err := json.Unmarshal(params, into); err != nil {
|
||||
return fmt.Errorf("%s's request is not of its shape (mesh-sdk buscalls): %w", method, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
|
||||
type saidCaller struct {
|
||||
Caller string `json:"caller"`
|
||||
Call string `json:"call"`
|
||||
Arguments map[string]any `json:"arguments"`
|
||||
}
|
||||
|
||||
@@ -36,6 +37,7 @@ func TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller(t *testing.T)
|
||||
asker.Logf = func(string, ...any) {}
|
||||
defer asker.Close()
|
||||
|
||||
calls := map[string]bool{}
|
||||
for _, key := range []string{"ledger.who", "ledger.who@anchor", "seat:node-desk.who@anchor"} {
|
||||
var got saidCaller
|
||||
mt.Until(t, func() error {
|
||||
@@ -49,6 +51,11 @@ func TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller(t *testing.T)
|
||||
if got.Caller != "shanks.node-tools" {
|
||||
t.Errorf("%s reached the module with caller %q, want the asker's bus user", key, got.Caller)
|
||||
}
|
||||
// And the id of the call in flight, one per call, which a ledger's apply names (novox/hq ADR 0597 §7).
|
||||
if got.Call == "" || calls[got.Call] {
|
||||
t.Errorf("%s reached the module with call id %q, want one of its own", key, got.Call)
|
||||
}
|
||||
calls[got.Call] = true
|
||||
if got.Arguments["caller"] != "novox.node-tools" {
|
||||
t.Errorf("%s: the arguments did not reach the module as sent: %v", key, got.Arguments)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"sync"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
)
|
||||
|
||||
// The ledger credentials (novox/hq ADR 0597 §9, design 25 § 12.4; issue 1966 T6): one per module assignment, held by
|
||||
// the machine's tool runner in its own secret `ledgers`, a map of module to credential (`{}` when none), which the vault
|
||||
// rewrites on each mint and retirement. **A module's ledger history is appended, and its snapshots written, only on
|
||||
// that module's own ledger connection**; everything else stays on the runner's own.
|
||||
|
||||
// LedgersFile is where the runner reads its own secret `ledgers`: MESH_LEDGERS_FILE when set, else beside its broker
|
||||
// credential, both in its directory mesh-state (`${dir:mesh-state}/ledgers`). Its manifest declares the secret only once
|
||||
// the vault holds it on every machine (B3 of issue 1966): until then a file not there means no ledger credentials, and
|
||||
// every ledger write is refused naming the missing one.
|
||||
const LedgersFile = "MESH_LEDGERS_FILE"
|
||||
|
||||
// LedgersFileOf is the path of the secret `ledgers` for a runtime whose broker credential is at broker.
|
||||
func LedgersFileOf(broker string) string {
|
||||
if file := os.Getenv(LedgersFile); file != "" {
|
||||
return file
|
||||
}
|
||||
if broker == "" {
|
||||
return ""
|
||||
}
|
||||
return filepath.Join(filepath.Dir(broker), "ledgers")
|
||||
}
|
||||
|
||||
// LedgerCredentials are the ledger connections of the modules on this machine, read from the secret `ledgers` and
|
||||
// followed as it changes, with no restart.
|
||||
type LedgerCredentials struct {
|
||||
file string
|
||||
dial func(bus.Credential) (*bus.Conn, error)
|
||||
logf func(string, ...any)
|
||||
|
||||
mu sync.Mutex
|
||||
creds map[string]bus.Credential
|
||||
conns map[string]*bus.Conn
|
||||
why string // why the file could not be read, when it could not
|
||||
stop func()
|
||||
}
|
||||
|
||||
// FollowLedgerCredentials reads the secret `ledgers` at file and follows it (WatchFile), dialling a module's connection
|
||||
// with dial the first time it is asked for.
|
||||
func FollowLedgerCredentials(file string, dial func(bus.Credential) (*bus.Conn, error), logf func(string, ...any)) *LedgerCredentials {
|
||||
l := &LedgerCredentials{file: file, dial: dial, logf: logf, creds: map[string]bus.Credential{}, conns: map[string]*bus.Conn{}}
|
||||
l.read()
|
||||
stop, err := WatchFile(file, l.read, logf, "a change of the ledger credentials is taken only after a restart")
|
||||
if err != nil {
|
||||
logf("[mesh-tools] cannot follow the ledger credentials at %s: %v; a change is taken only after a restart", file, err)
|
||||
} else {
|
||||
l.stop = stop
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
// read takes the file as it is now: a module whose credential changed or went has its connection closed, and is
|
||||
// dialled again from the new one when next asked for.
|
||||
func (l *LedgerCredentials) read() {
|
||||
creds, why := map[string]bus.Credential{}, ""
|
||||
raw, err := os.ReadFile(l.file)
|
||||
switch {
|
||||
case errors.Is(err, os.ErrNotExist):
|
||||
why = "the file " + l.file + " is not there yet"
|
||||
case err != nil:
|
||||
why = err.Error()
|
||||
case len(bytes.TrimSpace(raw)) == 0:
|
||||
why = "the file " + l.file + " is empty"
|
||||
default:
|
||||
if err := json.Unmarshal(raw, &creds); err != nil {
|
||||
why = "the file " + l.file + " is not a map of module to credential: " + err.Error()
|
||||
creds = map[string]bus.Credential{}
|
||||
}
|
||||
}
|
||||
l.mu.Lock()
|
||||
var closing []*bus.Conn
|
||||
for module, conn := range l.conns {
|
||||
if now, has := creds[module]; !has || !reflect.DeepEqual(now, l.creds[module]) {
|
||||
closing = append(closing, conn)
|
||||
delete(l.conns, module)
|
||||
}
|
||||
}
|
||||
saidBefore := l.why
|
||||
l.creds, l.why = creds, why
|
||||
l.mu.Unlock()
|
||||
for _, c := range closing {
|
||||
c.Close()
|
||||
}
|
||||
if why != "" && why != saidBefore {
|
||||
l.logf("[mesh-tools] the ledger credentials cannot be read (%s); no ledger is written until they can", why)
|
||||
}
|
||||
}
|
||||
|
||||
// Writer is a module's ledger connection, dialled from its credential the first time it is asked for; refused, naming
|
||||
// what is missing, when the secret holds none for it.
|
||||
func (l *LedgerCredentials) Writer(module string) (*bus.Conn, error) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if c := l.conns[module]; c != nil {
|
||||
return c, nil
|
||||
}
|
||||
cred, has := l.creds[module]
|
||||
if !has {
|
||||
why := ""
|
||||
if l.why != "" {
|
||||
why = " (" + l.why + ")"
|
||||
}
|
||||
return nil, fmt.Errorf("%s's ledger credential is missing from the tool runner's own secret ledgers%s: its history "+
|
||||
"is published only on its own ledger connection, which the grants part issues with its membership and the "+
|
||||
"vault writes there (novox/hq ADR 0597 §9)", module, why)
|
||||
}
|
||||
if cred.Module == "" {
|
||||
cred.Module = module
|
||||
}
|
||||
c, err := l.dial(cred)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s's ledger connection could not be opened: %w", module, err)
|
||||
}
|
||||
c.Logf = l.logf
|
||||
l.conns[module] = c
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// Stop ends the following and closes every ledger connection.
|
||||
func (l *LedgerCredentials) Stop() {
|
||||
if l.stop != nil {
|
||||
l.stop()
|
||||
}
|
||||
l.mu.Lock()
|
||||
conns := l.conns
|
||||
l.conns = map[string]*bus.Conn{}
|
||||
l.mu.Unlock()
|
||||
for _, c := range conns {
|
||||
c.Close()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/buscompose"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// 1966/AC22: given the secret ledgers holding m's credential, when m applies, then the append goes on m's connection
|
||||
// (the runner's own account may not publish on m's ledger, as the grants part grants it); given it missing, apply is
|
||||
// refused naming it; a change of the file is taken without a restart.
|
||||
func TestALedgerIsWrittenOnlyOnItsModulesLedgerConnection(t *testing.T) {
|
||||
ledgerSubjects := []string{buscompose.LedgerHistorySubject("m", "records") + ".>", buscompose.LedgerSnapshotSubject("m", "records") + ".>"}
|
||||
users := []*server.User{
|
||||
{Username: "node-tools", Password: "pw", Permissions: &server.Permissions{
|
||||
Publish: &server.SubjectPermission{Allow: []string{">"}, Deny: ledgerSubjects}}},
|
||||
// As the grants part composes a ledger account (mesh-grants buscompose ledgerPermissions): publish on its
|
||||
// module's ledger histories and snapshots, subscribe its own inbox, nothing else.
|
||||
{Username: "ledger.anchor.m", Password: "pw", Permissions: &server.Permissions{
|
||||
Publish: &server.SubjectPermission{Allow: ledgerSubjects},
|
||||
Subscribe: &server.SubjectPermission{Allow: []string{"_INBOX.ledger.anchor.m.>"}}}},
|
||||
{Username: "admin", Password: "pw"},
|
||||
}
|
||||
s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: t.TempDir(),
|
||||
NoLog: true, NoSigs: true, Users: users})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
go s.Start()
|
||||
if !s.ReadyForConnections(30 * time.Second) {
|
||||
t.Fatal("the bus did not come up")
|
||||
}
|
||||
t.Cleanup(s.Shutdown)
|
||||
url := s.ClientURL()
|
||||
|
||||
admin, err := nats.Connect(url, nats.UserInfo("admin", "pw"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(admin.Close)
|
||||
js, _ := jetstream.New(admin)
|
||||
ctx := context.Background()
|
||||
for _, cfg := range []jetstream.StreamConfig{
|
||||
{Name: "LEDGER_m_records", Subjects: []string{"mesh.ledger.m.records.history.>"}},
|
||||
{Name: "SNAPSHOT_m_records", Subjects: []string{"mesh.ledger.m.records.snapshot.>"}, MaxMsgsPerSubject: 1},
|
||||
} {
|
||||
if _, err := js.CreateStream(ctx, cfg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, b := range []string{"m_records", buscompose.GrantsMembershipsBucket} {
|
||||
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
mem := mt.MembershipOf("m", "anchor", false, nil)
|
||||
mem.Bindings = []membership.BusBinding{{Local: "records", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger,
|
||||
History: "LEDGER_m_records", Subject: "mesh.ledger.m.records.history", State: "m_records", Snapshot: "mesh.ledger.m.records.snapshot",
|
||||
Types: []catalogue.LedgerType{{Name: "decision", Operations: []catalogue.LedgerOperation{
|
||||
{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}}}}}}}
|
||||
raw, _ := json.Marshal(mem)
|
||||
var sdk membership.Membership
|
||||
_ = json.Unmarshal(raw, &sdk)
|
||||
grants, _ := js.KeyValue(ctx, buscompose.GrantsMembershipsBucket)
|
||||
if _, err := grants.Put(ctx, buscompose.MembershipKey("anchor", "m"), membership.EncodeGrantsMembership(sdk)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
conn, err := bus.Connect(bus.Credential{URL: url, Module: "node-tools", Node: "anchor", User: "node-tools", Password: "pw"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn.Logf = func(string, ...any) {}
|
||||
t.Cleanup(conn.Close)
|
||||
conn.Follow("m")
|
||||
mt.Until(t, func() error {
|
||||
if conn.Membership("m") == nil {
|
||||
return os.ErrNotExist
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
// Not there yet, as on a machine whose vault has not made it: no ledger credentials, and the runner serves on.
|
||||
file := filepath.Join(t.TempDir(), "ledgers")
|
||||
creds := FollowLedgerCredentials(file, bus.Dial, t.Logf)
|
||||
t.Cleanup(creds.Stop)
|
||||
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
|
||||
WithLedgerWriters(creds.Writer)(all)
|
||||
m := all.forModule("m").(*moduleBus)
|
||||
apply := func(instance string) error {
|
||||
_, err := m.BusState(buscalls.LedgerApply, json.RawMessage(`{"ledger":"records","instance":"`+instance+`","operation":"title","body":{"title":"T"}}`))
|
||||
return err
|
||||
}
|
||||
|
||||
// Missing: refused, naming it and why.
|
||||
if err := apply("decision/1"); err == nil || !strings.Contains(err.Error(), "m's ledger credential is missing from the tool runner's own secret ledgers (the file "+file+" is not there yet)") {
|
||||
t.Fatalf("an apply with no credential: %v", err)
|
||||
}
|
||||
// Written: taken without a restart, on m's connection (the runner's own account is denied m's ledger subjects, so an
|
||||
// append that lands went on m's), whose account stamps the operation.
|
||||
cred, _ := json.Marshal(map[string]bus.Credential{"m": {URL: url, Node: "anchor", Module: "m", User: "ledger.anchor.m", Password: "pw"}})
|
||||
if err := os.WriteFile(file, cred, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mt.Until(t, func() error { return apply("decision/1") })
|
||||
page, err := m.BusState(buscalls.LedgerHistory, json.RawMessage(`{"ledger":"records","instance":"decision/1"}`))
|
||||
if err != nil || !strings.Contains(string(page), `"author":{"account":"ledger.anchor.m"}`) {
|
||||
t.Fatalf("the history holds %s, %v", page, err)
|
||||
}
|
||||
// Gone again: refused again, the change taken without a restart.
|
||||
if err := os.WriteFile(file, []byte(`{}`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mt.Until(t, func() error {
|
||||
if err := apply("decision/2"); err == nil || !strings.Contains(err.Error(), "credential is missing") {
|
||||
return os.ErrExist
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,261 @@
|
||||
//go:build integration
|
||||
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/ledger"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// racing is a history whose append lets another writer apply first, once: before its own append, or after it.
|
||||
type racing struct {
|
||||
ledger.History
|
||||
before, after func()
|
||||
}
|
||||
|
||||
func (r *racing) Append(key string, op []byte, expect uint64) (uint64, error) {
|
||||
if f := r.before; f != nil {
|
||||
r.before = nil
|
||||
f()
|
||||
}
|
||||
seq, err := r.History.Append(key, op, expect)
|
||||
if f := r.after; f != nil && err == nil {
|
||||
r.after = nil
|
||||
f()
|
||||
}
|
||||
return seq, err
|
||||
}
|
||||
|
||||
// cut is a state whose next write is lost, as a writer stopped between its history and its state would lose it.
|
||||
type cut struct {
|
||||
ledger.State
|
||||
once bool
|
||||
}
|
||||
|
||||
func (c *cut) Write(key string, value []byte, revision uint64) (uint64, error) {
|
||||
if c.once {
|
||||
c.once = false
|
||||
return 0, errors.New("the writer stopped")
|
||||
}
|
||||
return c.State.Write(key, value, revision)
|
||||
}
|
||||
|
||||
// 1966/AC19: the ledger keeps its promises against the real bus (ADR 0597's How it is checked; design 25 § 12.7), one
|
||||
// case each, and every apply leaves the snapshot equal to the state.
|
||||
func TestTheLedgerKeepsItsPromisesOnTheBus(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
nc, err := nats.Connect(mt.URL(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(nc.Close)
|
||||
js, _ := jetstream.New(nc)
|
||||
ctx := context.Background()
|
||||
for _, cfg := range []jetstream.StreamConfig{
|
||||
{Name: "LEDGER_m_records", Subjects: []string{"mesh.ledger.m.records.history.>"}},
|
||||
{Name: "SNAPSHOT_m_records", Subjects: []string{"mesh.ledger.m.records.snapshot.>"}, MaxMsgsPerSubject: 1},
|
||||
} {
|
||||
if _, err := js.CreateStream(ctx, cfg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, b := range []string{"m_records", "m_records_by_title"} {
|
||||
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
types := []catalogue.LedgerType{{Name: "decision", Schema: map[string]catalogue.LedgerField{"title": {Type: "string"}},
|
||||
Operations: []catalogue.LedgerOperation{
|
||||
{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}, Author: true},
|
||||
{Name: "note", Effect: catalogue.EffectAppend, Into: "notes", Fields: []string{"says"}},
|
||||
},
|
||||
Indexes: []catalogue.LedgerIndex{{Name: "by-title", On: []string{"title"}}}}}
|
||||
binding := membership.BusBinding{Local: "records", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger,
|
||||
History: "LEDGER_m_records", Subject: "mesh.ledger.m.records.history", State: "m_records",
|
||||
Indexes: map[string]string{"by-title": "m_records_by_title"}, Snapshot: "mesh.ledger.m.records.snapshot"}
|
||||
mem := mt.MembershipOf("m", "anchor", false, nil)
|
||||
binding.Types = types
|
||||
mem.Bindings = []membership.BusBinding{binding}
|
||||
mesh.Issue(t, mem)
|
||||
conn := connect(t, "node-tools", "anchor")
|
||||
conn.Follow("m")
|
||||
mt.Until(t, func() error {
|
||||
if conn.Membership("m") == nil {
|
||||
return errors.New("not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
|
||||
WithLedgerWriters(func(string) (*bus.Conn, error) { return conn, nil })(all)
|
||||
m := all.forModule("m").(*moduleBus)
|
||||
|
||||
open := func(t *testing.T) *ledger.Ledger {
|
||||
t.Helper()
|
||||
l, err := all.ledger("m", "records")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return l
|
||||
}
|
||||
apply := func(t *testing.T, l *ledger.Ledger, instance, operation, body string) (buscalls.Applied, error) {
|
||||
t.Helper()
|
||||
return l.Apply(buscalls.ApplyRequest{Ledger: "records", Instance: instance, Operation: operation, Body: json.RawMessage(body)},
|
||||
buscalls.Author{Account: "m"})
|
||||
}
|
||||
must := func(t *testing.T, l *ledger.Ledger, instance, operation, body string) buscalls.Applied {
|
||||
t.Helper()
|
||||
got, err := apply(t, l, instance, operation, body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return got
|
||||
}
|
||||
held := func(t *testing.T, key string) ledger.Record {
|
||||
t.Helper()
|
||||
raw, _, err := open(t).State.Get(key)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var r ledger.Record
|
||||
if raw != nil {
|
||||
_ = json.Unmarshal(raw, &r)
|
||||
}
|
||||
return r
|
||||
}
|
||||
snapshotIsState := func(t *testing.T, key string) {
|
||||
t.Helper()
|
||||
l := open(t)
|
||||
state, _, _ := l.State.Get(key)
|
||||
snap, _, _ := l.Snapshots.Get(key)
|
||||
if string(snap) != string(state) {
|
||||
t.Fatalf("the snapshot of %s is %s, its state %s", key, snap, state)
|
||||
}
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
name, key string
|
||||
run func(t *testing.T)
|
||||
}{
|
||||
{"two writers race and one is refused", "decision.race", func(t *testing.T) {
|
||||
l := open(t)
|
||||
l.History = &racing{History: l.History, before: func() { must(t, open(t), "decision/race", "note", `{"says":"first"}`) }}
|
||||
_, err := apply(t, l, "decision/race", "note", `{"says":"second"}`)
|
||||
if !errors.Is(err, ledger.ErrRaced) || !strings.Contains(err.Error(), "not taken") {
|
||||
t.Fatalf("the racing writer was answered %v", err)
|
||||
}
|
||||
if got := held(t, "decision.race"); string(got.State) != `{"notes":[{"says":"first"}]}` {
|
||||
t.Fatalf("the state is %s", got.State)
|
||||
}
|
||||
}},
|
||||
{"two interleaved applies leave the state at the higher sequence", "decision.interleave", func(t *testing.T) {
|
||||
var inner buscalls.Applied
|
||||
l := open(t)
|
||||
l.History = &racing{History: l.History, after: func() { inner = must(t, open(t), "decision/interleave", "note", `{"says":"later"}`) }}
|
||||
outer := must(t, l, "decision/interleave", "note", `{"says":"earlier"}`)
|
||||
got := held(t, "decision.interleave")
|
||||
if inner.Sequence <= outer.Sequence || got.Sequence != inner.Sequence ||
|
||||
string(got.State) != `{"notes":[{"says":"earlier"},{"says":"later"}]}` {
|
||||
t.Fatalf("applied %d then %d, the state at %d: %s", outer.Sequence, inner.Sequence, got.Sequence, got.State)
|
||||
}
|
||||
}},
|
||||
{"a write cut between history and state is repaired by the next apply", "decision.cut", func(t *testing.T) {
|
||||
must(t, open(t), "decision/cut", "note", `{"says":"one"}`)
|
||||
l := open(t)
|
||||
l.State = &cut{State: l.State, once: true}
|
||||
if got := must(t, l, "decision/cut", "note", `{"says":"cut"}`); !got.Taken || !got.StateBehind {
|
||||
t.Fatalf("the cut apply was answered %+v", got)
|
||||
}
|
||||
must(t, open(t), "decision/cut", "note", `{"says":"three"}`)
|
||||
if got := held(t, "decision.cut"); string(got.State) != `{"notes":[{"says":"one"},{"says":"cut"},{"says":"three"}]}` {
|
||||
t.Fatalf("the state is %s", got.State)
|
||||
}
|
||||
}},
|
||||
{"a full state bucket answers taken with its sequence and catches up", "decision.full", func(t *testing.T) {
|
||||
must(t, open(t), "decision/full", "note", `{"says":"small"}`)
|
||||
kv, _ := js.KeyValue(ctx, "m_records")
|
||||
status, _ := kv.Status(ctx)
|
||||
if _, err := js.UpdateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: "m_records", MaxBytes: int64(status.Bytes()) + 64}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := must(t, open(t), "decision/full", "note", fmt.Sprintf(`{"says":%q}`, strings.Repeat("x", 512)))
|
||||
if !got.Taken || got.Sequence == 0 || !got.StateBehind {
|
||||
t.Fatalf("the full bucket's apply was answered %+v", got)
|
||||
}
|
||||
if _, err := js.UpdateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: "m_records", MaxBytes: -1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
after := must(t, open(t), "decision/full", "note", `{"says":"room"}`)
|
||||
if st := held(t, "decision.full"); after.StateBehind || st.Sequence != after.Sequence || !strings.Contains(string(st.State), strings.Repeat("x", 512)) {
|
||||
t.Fatalf("after room: %+v, the state at %d", after, st.Sequence)
|
||||
}
|
||||
}},
|
||||
{"an instance whose operations expired rebuilds from its snapshot", "decision.expired", func(t *testing.T) {
|
||||
must(t, open(t), "decision/expired", "title", `{"title":"Kept"}`)
|
||||
want := held(t, "decision.expired")
|
||||
s, _ := js.Stream(ctx, "LEDGER_m_records")
|
||||
if err := s.Purge(ctx, jetstream.WithPurgeSubject("mesh.ledger.m.records.history.decision.expired")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
kv, _ := js.KeyValue(ctx, "m_records")
|
||||
_ = kv.Purge(ctx, "decision.expired")
|
||||
got, err := open(t).Rebuild(buscalls.RebuildRequest{Instance: "decision/expired", From: buscalls.RebuildFromSnapshot})
|
||||
if st := held(t, "decision.expired"); err != nil || got.Written != 1 || string(st.State) != string(want.State) {
|
||||
t.Fatalf("rebuilt %+v, %v: the state %s, want %s", got, err, st.State, want.State)
|
||||
}
|
||||
}},
|
||||
{"rebuild, reindex and lag repair by hand", "decision.repair", func(t *testing.T) {
|
||||
must(t, open(t), "decision/repair", "title", `{"title":"Repaired"}`)
|
||||
kv, _ := js.KeyValue(ctx, "m_records")
|
||||
_ = kv.Purge(ctx, "decision.repair")
|
||||
l := open(t)
|
||||
lag, err := l.Lag(buscalls.LedgerRequest{})
|
||||
if err != nil || !strings.Contains(fmt.Sprint(lag), "decision/repair") || len(lag.Indexes) == 0 {
|
||||
t.Fatalf("lag says %+v, %v", lag, err)
|
||||
}
|
||||
if got, err := l.Rebuild(buscalls.RebuildRequest{Instance: "decision/repair"}); err != nil || got.Written != 1 {
|
||||
t.Fatalf("rebuilt %+v, %v", got, err)
|
||||
}
|
||||
if _, err := l.Reindex(buscalls.ReindexRequest{Index: "by-title"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if lag, err := l.Lag(buscalls.LedgerRequest{}); err != nil || len(lag.Instances) != 0 || len(lag.Indexes) != 0 {
|
||||
t.Fatalf("lag after the repairs says %+v, %v", lag, err)
|
||||
}
|
||||
q, err := l.Query(buscalls.QueryRequest{Index: "by-title", Key: "Repaired.>"})
|
||||
if err != nil || len(q.Entries) != 1 {
|
||||
t.Fatalf("queried %+v, %v", q, err)
|
||||
}
|
||||
}},
|
||||
{"a forged author is refused", "decision.forged", func(t *testing.T) {
|
||||
_, err := m.BusState(buscalls.LedgerApply, json.RawMessage(`{"ledger":"records","instance":"decision/forged","operation":"title","body":{"title":"T","author":"op@shanks"},"call":"forged"}`))
|
||||
if err == nil || !strings.Contains(err.Error(), "the call forged, which is not a tool call in flight") {
|
||||
t.Fatalf("a forged call was answered %v", err)
|
||||
}
|
||||
if last, _ := open(t).History.Last("decision.forged"); last != 0 {
|
||||
t.Fatalf("the forged apply was appended at %d", last)
|
||||
}
|
||||
}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
c.run(t)
|
||||
if c.key != "decision.forged" {
|
||||
snapshotIsState(t, c.key)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/launch"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/ledger"
|
||||
)
|
||||
|
||||
// A module's ledgers as the tool runner serves them (novox/hq ADR 0597, design 25 § 12.4, § 12.5; issue 1966): opened
|
||||
// for the module whose bundle or whose repair tool asks, from its own bindings alone, with its types and the connection
|
||||
// its history is published on.
|
||||
|
||||
// ledgerSource is how the runner opens a module's ledger beside its binding: the connection its history and snapshots
|
||||
// are published on (the module's ledger credential, ADR 0597 §9).
|
||||
type ledgerSource struct {
|
||||
writer func(module string) (*bus.Conn, error)
|
||||
}
|
||||
|
||||
// callsInFlight are the tool calls the runtime is serving, each with the module whose bundle serves it and the caller
|
||||
// the runtime proved from the subject the call arrived on (bus.CalledSubject, novox/hq issue 365).
|
||||
type callsInFlight struct {
|
||||
mu sync.Mutex
|
||||
of map[string]callInFlight
|
||||
}
|
||||
|
||||
type callInFlight struct{ module, caller string }
|
||||
|
||||
// begin keeps a call of module in flight for caller, under an id no bundle can guess, until end.
|
||||
func (f *callsInFlight) begin(module, caller string) (string, func()) {
|
||||
var raw [16]byte
|
||||
_, _ = rand.Read(raw[:])
|
||||
id := hex.EncodeToString(raw[:])
|
||||
f.mu.Lock()
|
||||
if f.of == nil {
|
||||
f.of = map[string]callInFlight{}
|
||||
}
|
||||
f.of[id] = callInFlight{module, caller}
|
||||
f.mu.Unlock()
|
||||
return id, func() {
|
||||
f.mu.Lock()
|
||||
delete(f.of, id)
|
||||
f.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// authorOf is the author an apply of module naming call is stamped with (ADR 0597 §7): the module's account, and the
|
||||
// caller the runtime proved for that call; naming no call, the account alone. A call not in flight, or another
|
||||
// module's, is refused, naming it. Nothing the bundle writes is taken as the author.
|
||||
func (f *callsInFlight) authorOf(module, account, call string) (buscalls.Author, error) {
|
||||
author := buscalls.Author{Account: account}
|
||||
if call == "" {
|
||||
return author, nil
|
||||
}
|
||||
f.mu.Lock()
|
||||
inFlight, ok := f.of[call]
|
||||
f.mu.Unlock()
|
||||
switch {
|
||||
case !ok:
|
||||
return author, fmt.Errorf("the apply names the call %s, which is not a tool call in flight here; an apply names the "+
|
||||
"call it serves as its tools/call's _meta %s gave it, or none (novox/hq ADR 0597 §7)", call, buscalls.MetaCall)
|
||||
case inFlight.module != module:
|
||||
return author, fmt.Errorf("the apply names the call %s, which is another module's tool call; %s names only its own "+
|
||||
"(novox/hq ADR 0597 §7)", call, module)
|
||||
}
|
||||
author.Caller = inFlight.caller
|
||||
return author, nil
|
||||
}
|
||||
|
||||
// Begin keeps one of the module's tool calls in flight for the caller the runtime proved (launch.Bus).
|
||||
func (b *moduleBus) Begin(caller string) (string, func()) { return b.all.calls.begin(b.module, caller) }
|
||||
|
||||
// ledger opens one of a module's ledgers by its local name.
|
||||
func (c *consumers) ledger(module, local string) (*ledger.Ledger, error) {
|
||||
b, err := c.conn.Binding(module, local, catalogue.StorageLedger)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Its types travel on its binding, copied by the grants part from the entry the manifest-accepted event carries
|
||||
// (mesh-sdk membership.BusBinding.Types): the runner reads no other part's state for them (ADR 0596).
|
||||
ts := b.Types
|
||||
if len(ts) == 0 {
|
||||
return nil, fmt.Errorf("%s's ledger %q is bound with no types, so no operation can be checked against one: its "+
|
||||
"binding carries the types its manifest asks for in `requires` (novox/hq ADR 0597 §1)", module, local)
|
||||
}
|
||||
var writer *bus.Conn
|
||||
if c.ledgers.writer != nil {
|
||||
if writer, err = c.ledgers.writer(module); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
history, state, snapshots, err := c.conn.LedgerStores(module, b, writer)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// The owning module's authenticated account: the user of the connection its history is published on.
|
||||
account := module
|
||||
if writer != nil && writer.Credential().User != "" {
|
||||
account = writer.Credential().User
|
||||
}
|
||||
return &ledger.Ledger{Module: module, Name: local, Account: account, Types: ts, History: history, State: state,
|
||||
Snapshots: snapshots, Indexes: c.conn.LedgerIndexes(b), Logf: c.logf}, nil
|
||||
}
|
||||
|
||||
// ledgerCall answers a bundle's ledger call, as the module whose bundle asks.
|
||||
func (b *moduleBus) ledgerCall(method string, params json.RawMessage) (any, error) {
|
||||
var named struct {
|
||||
Ledger string `json:"ledger"`
|
||||
}
|
||||
if err := decode(method, params, &named); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if method == buscalls.LedgerApply {
|
||||
var r buscalls.ApplyRequest
|
||||
if err := decode(method, params, &r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// The call is judged first: a forged one is refused before anything of the ledger is touched.
|
||||
if _, err := b.all.calls.authorOf(b.module, "", r.Call); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
l, err := b.all.ledger(b.module, r.Ledger)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
author, err := b.all.calls.authorOf(b.module, l.Account, r.Call)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return l.Apply(r, author)
|
||||
}
|
||||
l, err := b.all.ledger(b.module, named.Ledger)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
switch method {
|
||||
case buscalls.LedgerGet:
|
||||
var r buscalls.InstanceRequest
|
||||
_ = json.Unmarshal(params, &r)
|
||||
return l.Get(r)
|
||||
case buscalls.LedgerHistory:
|
||||
var r buscalls.HistoryRequest
|
||||
_ = json.Unmarshal(params, &r)
|
||||
return l.ReadHistory(r)
|
||||
case buscalls.LedgerQuery:
|
||||
var r buscalls.QueryRequest
|
||||
_ = json.Unmarshal(params, &r)
|
||||
return l.Query(r)
|
||||
case buscalls.LedgerRebuild:
|
||||
var r buscalls.RebuildRequest
|
||||
_ = json.Unmarshal(params, &r)
|
||||
return l.Rebuild(r)
|
||||
case buscalls.LedgerReindex:
|
||||
var r buscalls.ReindexRequest
|
||||
_ = json.Unmarshal(params, &r)
|
||||
return l.Reindex(r)
|
||||
case buscalls.LedgerSnapshot:
|
||||
var r buscalls.InstanceRequest
|
||||
_ = json.Unmarshal(params, &r)
|
||||
return l.Snapshot(r)
|
||||
case buscalls.LedgerLag:
|
||||
var r buscalls.LedgerRequest
|
||||
_ = json.Unmarshal(params, &r)
|
||||
return l.Lag(r)
|
||||
}
|
||||
return nil, fmt.Errorf("the runtime answers no %s", method)
|
||||
}
|
||||
|
||||
// ledgerWatch hands a bundle each change of its ledger's states.
|
||||
func (b *moduleBus) ledgerWatch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
|
||||
var r buscalls.LedgerWatchRequest
|
||||
if err := decode(buscalls.LedgerWatch, params, &r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
l, err := b.all.ledger(b.module, r.Ledger)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return l.Watch(r, func(change buscalls.InstanceChange) error {
|
||||
raw, err := json.Marshal(change)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return deliver(raw)
|
||||
})
|
||||
}
|
||||
|
||||
// hasLedger is whether a module's membership binds a ledger.
|
||||
func hasLedger(conn *bus.Conn, module string) bool {
|
||||
for _, b := range conn.Bindings(module) {
|
||||
if b.Storage == catalogue.StorageLedger {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// The repair tools the runner serves for every module owning a ledger (design 25 § 12.5), named so no bundle's tool is
|
||||
// taken for one.
|
||||
const (
|
||||
RepairRebuild = "ledger_rebuild"
|
||||
RepairReindex = "ledger_reindex"
|
||||
RepairSnapshot = "ledger_snapshot"
|
||||
RepairLag = "ledger_lag"
|
||||
)
|
||||
|
||||
// The effects a tool carries (novox/hq ADR 0387 §1).
|
||||
const (
|
||||
EffectReads = "reads"
|
||||
EffectActs = "acts"
|
||||
)
|
||||
|
||||
// repairTools are a module's ledger repairs as tools, each limited to its own ledgers: the request's ledger is a local
|
||||
// name of the module whose tool is called.
|
||||
func (c *consumers) repairTools(module string) []launch.Tool {
|
||||
schema := func(props string, required ...string) json.RawMessage {
|
||||
req, _ := json.Marshal(append([]string{"ledger"}, required...))
|
||||
return json.RawMessage(`{"type":"object","properties":{"ledger":{"type":"string","description":"the ledger's local name"}` +
|
||||
props + `},"required":` + string(req) + `}`)
|
||||
}
|
||||
instance := `,"instance":{"type":"string","description":"one instance, such as decision/0290; every instance when empty"}`
|
||||
run := func(do func(l *ledger.Ledger, args json.RawMessage) (any, error)) func(string, json.RawMessage) (json.RawMessage, error) {
|
||||
return func(_ string, args json.RawMessage) (json.RawMessage, error) {
|
||||
var named struct {
|
||||
Ledger string `json:"ledger"`
|
||||
}
|
||||
if err := json.Unmarshal(args, &named); err != nil || named.Ledger == "" {
|
||||
return nil, errors.New("a repair names the ledger, by its local name")
|
||||
}
|
||||
l, err := c.ledger(module, named.Ledger)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
answer, err := do(l, args)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return json.Marshal(answer)
|
||||
}
|
||||
}
|
||||
return []launch.Tool{
|
||||
{Name: RepairRebuild, Effect: EffectActs, Input: schema(instance + `,"from":{"type":"string","enum":["history","snapshot"],"description":"derive from the whole history (the default) or from the snapshot and the operations after it"}`),
|
||||
Description: "Derive a ledger instance's state again from its history or its snapshot, and write it and its snapshot (novox/hq ADR 0597 §10).",
|
||||
Run: run(func(l *ledger.Ledger, args json.RawMessage) (any, error) {
|
||||
var r buscalls.RebuildRequest
|
||||
_ = json.Unmarshal(args, &r)
|
||||
return l.Rebuild(r)
|
||||
})},
|
||||
{Name: RepairReindex, Effect: EffectActs, Input: schema(`,"index":{"type":"string","description":"the declared index"}`, "index"),
|
||||
Description: "Build a ledger's declared index again from its states, derived from its history (novox/hq ADR 0597 §4, §10).",
|
||||
Run: run(func(l *ledger.Ledger, args json.RawMessage) (any, error) {
|
||||
var r buscalls.ReindexRequest
|
||||
_ = json.Unmarshal(args, &r)
|
||||
return l.Reindex(r)
|
||||
})},
|
||||
{Name: RepairSnapshot, Effect: EffectActs, Input: schema(instance),
|
||||
Description: "Rewrite the snapshot of a ledger instance that is behind its state (novox/hq ADR 0597 §8, §10).",
|
||||
Run: run(func(l *ledger.Ledger, args json.RawMessage) (any, error) {
|
||||
var r buscalls.InstanceRequest
|
||||
_ = json.Unmarshal(args, &r)
|
||||
return l.Snapshot(r)
|
||||
})},
|
||||
{Name: RepairLag, Effect: EffectReads, Input: schema(""),
|
||||
Description: "Say every ledger instance whose state or snapshot is behind its history, and each declared index behind it (novox/hq ADR 0597 §10).",
|
||||
Run: run(func(l *ledger.Ledger, args json.RawMessage) (any, error) {
|
||||
var r buscalls.LedgerRequest
|
||||
_ = json.Unmarshal(args, &r)
|
||||
return l.Lag(r)
|
||||
})},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
package runtime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buscalls"
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/membership"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
|
||||
)
|
||||
|
||||
// 1966/AC16: given a module with a ledger binding, when the runner lists its tools, then the four repair tools are
|
||||
// among them, marked acts or reads; a module without one has none, and a binding written later brings them.
|
||||
func TestTheLedgerRepairsAreToolsOfEveryLedgerOwner(t *testing.T) {
|
||||
mesh := mt.New(t)
|
||||
owner := mt.MembershipOf("owner", "anchor", true, nil)
|
||||
owner.Bindings = []membership.BusBinding{{Local: "records", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger,
|
||||
History: "LEDGER_owner_records", Subject: "mesh.ledger.owner.records.history", State: "owner_records",
|
||||
Snapshot: "mesh.ledger.owner.records.snapshot"}}
|
||||
plain := mt.MembershipOf("plain", "anchor", true, nil)
|
||||
mesh.Issue(t, owner)
|
||||
mesh.Issue(t, plain)
|
||||
nodeTools := connect(t, "node-tools", "anchor")
|
||||
stop, err := Run(nodeTools, []Served{{"owner", []string{mt.Fixture("caller-echo.py")}},
|
||||
{"plain", []string{mt.Fixture("caller-echo.py")}}}, nil, func(string, ...any) {})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer stop()
|
||||
asker := connect(t, "console", "workstation")
|
||||
listed := func(module string) (string, error) {
|
||||
answered, err := asker.Ask(module+"."+ToolsVerb, map[string]any{}, "")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var got ToolsAnswer
|
||||
if err := json.Unmarshal(answered.Result, &got); err != nil {
|
||||
return "", err
|
||||
}
|
||||
var names []string
|
||||
for _, tool := range got.Tools {
|
||||
names = append(names, tool.Name+":"+tool.Effect)
|
||||
}
|
||||
sort.Strings(names)
|
||||
return strings.Join(names, " "), nil
|
||||
}
|
||||
for _, c := range []struct{ name, module, want string }{
|
||||
{"a ledger owner lists the four repairs", "owner", "ledger_lag:reads ledger_rebuild:acts ledger_reindex:acts ledger_snapshot:acts who:"},
|
||||
{"a module with no ledger has none", "plain", "who:"},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
var got string
|
||||
mt.Until(t, func() (err error) { got, err = listed(c.module); return })
|
||||
if got != c.want {
|
||||
t.Fatalf("%s lists %q, want %q", c.module, got, c.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
t.Run("a ledger bound later brings them", func(t *testing.T) {
|
||||
plain.Bindings = owner.Bindings
|
||||
mesh.Issue(t, plain)
|
||||
mt.Until(t, func() error {
|
||||
got, err := listed("plain")
|
||||
if err == nil && !strings.Contains(got, "ledger_rebuild:acts") {
|
||||
err = errors.New("listed " + got)
|
||||
}
|
||||
return err
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
// aLedger raises one ledger of module m on the bus as the bus module would create it, binds it in m's membership, and
|
||||
// answers the runtime's buses for m and for other; its types travel on its binding, and a second binding carries none.
|
||||
func aLedger(t *testing.T) (m, other *moduleBus, mesh *mt.Mesh) {
|
||||
t.Helper()
|
||||
mesh = mt.New(t)
|
||||
nc, err := nats.Connect(mt.URL(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(nc.Close)
|
||||
js, _ := jetstream.New(nc)
|
||||
ctx := context.Background()
|
||||
for _, cfg := range []jetstream.StreamConfig{
|
||||
{Name: "LEDGER_m_records", Subjects: []string{"mesh.ledger.m.records.history.>"}},
|
||||
{Name: "SNAPSHOT_m_records", Subjects: []string{"mesh.ledger.m.records.snapshot.>"}, MaxMsgsPerSubject: 1},
|
||||
} {
|
||||
if _, err := js.CreateStream(ctx, cfg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: "m_records"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mem := mt.MembershipOf("m", "anchor", false, nil)
|
||||
mem.Bindings = []membership.BusBinding{{Local: "records", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger,
|
||||
History: "LEDGER_m_records", Subject: "mesh.ledger.m.records.history", State: "m_records", Snapshot: "mesh.ledger.m.records.snapshot",
|
||||
Types: []catalogue.LedgerType{{Name: "decision", Schema: map[string]catalogue.LedgerField{"title": {Type: "string"}},
|
||||
Operations: []catalogue.LedgerOperation{{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}, Author: true}}}}},
|
||||
{Local: "untyped", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger, History: "LEDGER_m_records",
|
||||
State: "m_records"},
|
||||
{Local: "elsewhere", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger, History: "LEDGER_m_records",
|
||||
Subject: "mesh.ledger.m.records.history", State: "m_records", Types: []catalogue.LedgerType{{Name: "decision"}}}}
|
||||
mesh.Issue(t, mem)
|
||||
mesh.Issue(t, mt.MembershipOf("other", "anchor", false, nil))
|
||||
conn := connect(t, "node-tools", "anchor")
|
||||
conn.Follow("m")
|
||||
conn.Follow("other")
|
||||
mt.Until(t, func() error {
|
||||
if conn.Membership("m") == nil || conn.Membership("other") == nil {
|
||||
return errors.New("not issued yet")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}, ledgers: ledgerSource{
|
||||
writer: func(string) (*bus.Conn, error) { return conn, nil },
|
||||
}}
|
||||
return all.forModule("m").(*moduleBus), all.forModule("other").(*moduleBus), mesh
|
||||
}
|
||||
|
||||
// 1966/AC17, 1966/AC18: the author an apply is stamped with is the caller the runtime proved for the call in flight it
|
||||
// names, whatever author the bundle wrote; another module's call, or one not in flight, is refused naming it (ADR 0597 §7).
|
||||
func TestAnApplyIsStampedWithTheCallerTheRuntimeProved(t *testing.T) {
|
||||
m, other, _ := aLedger(t)
|
||||
call, end := m.Begin("op@shanks")
|
||||
defer end()
|
||||
theirs, endTheirs := other.Begin("op@shanks")
|
||||
defer endTheirs()
|
||||
forged := `"body":{"title":"T","author":{"account":"x","caller":"forged"}}`
|
||||
for _, c := range []struct{ name, params, caller, refusal string }{
|
||||
{"1966/AC17 a call in flight stamps its proven caller", `{"ledger":"records","instance":"decision/1","operation":"title",` + forged + `,"call":"` + call + `","author":"forged"}`, "op@shanks", ""},
|
||||
{"naming no call stamps the account alone", `{"ledger":"records","instance":"decision/2","operation":"title",` + forged + `}`, "", ""},
|
||||
{"1966/AC18 another module's call is refused, naming it", `{"ledger":"records","instance":"decision/3","operation":"title",` + forged + `,"call":"` + theirs + `"}`, "", "the call " + theirs + ", which is another module's"},
|
||||
{"a ledger bound with no types is refused, naming it", `{"ledger":"untyped","instance":"decision/3","operation":"title",` + forged + `}`, "", `m's ledger "untyped" is bound with no types`},
|
||||
{"a binding naming subjects its account is not granted is refused, naming both", `{"ledger":"elsewhere","instance":"decision/3","operation":"title",` + forged + `}`, "", "bound with the history subject mesh.ledger.m.records.history, and its ledger account is granted mesh.ledger.m.elsewhere.history alone"},
|
||||
{"a call not in flight is refused, naming it", `{"ledger":"records","instance":"decision/3","operation":"title",` + forged + `,"call":"c0ffee"}`, "", "the call c0ffee, which is not a tool call in flight"},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
var req struct {
|
||||
Instance string `json:"instance"`
|
||||
}
|
||||
_ = json.Unmarshal([]byte(c.params), &req)
|
||||
_, err := m.BusState(buscalls.LedgerApply, json.RawMessage(c.params))
|
||||
if c.refusal != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), c.refusal) {
|
||||
t.Fatalf("applied with %v, want refused with %q", err, c.refusal)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, err := m.BusState(buscalls.LedgerHistory, json.RawMessage(`{"ledger":"records","instance":"`+req.Instance+`"}`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var page buscalls.HistoryPage
|
||||
_ = json.Unmarshal(raw, &page)
|
||||
if len(page.Operations) != 1 || page.Operations[0].Author != (buscalls.Author{Account: "m", Caller: c.caller}) {
|
||||
t.Fatalf("the history holds %+v, want the author {m %s}", page.Operations, c.caller)
|
||||
}
|
||||
state, _ := m.BusState(buscalls.LedgerGet, json.RawMessage(`{"ledger":"records","instance":"`+req.Instance+`"}`))
|
||||
if !strings.Contains(string(state), `"author":{"account":"m"`) || strings.Contains(string(state), "forged") {
|
||||
t.Fatalf("the state took the bundle's author: %s", state)
|
||||
}
|
||||
})
|
||||
}
|
||||
end()
|
||||
if _, err := m.BusState(buscalls.LedgerApply, json.RawMessage(`{"ledger":"records","instance":"decision/4","operation":"title","call":"`+call+`"}`)); err == nil {
|
||||
t.Fatal("a call that ended was still taken as in flight")
|
||||
}
|
||||
}
|
||||
@@ -20,9 +20,13 @@ func logBuses(t *testing.T, maxBytes int64) (issues, other launch.Bus, mesh *mt.
|
||||
t.Helper()
|
||||
mesh = mt.New(t)
|
||||
m := mt.MembershipOf("issues", "anchor", false, nil)
|
||||
m.Logs = []bus.LogIssued{mesh.Log(t, "issues", "changes", maxBytes)}
|
||||
changes := mesh.Log(t, "issues", "changes", maxBytes)
|
||||
m.Logs = []bus.LogIssued{changes}
|
||||
mesh.Issue(t, m)
|
||||
mesh.Issue(t, mt.MembershipOf("other", "anchor", false, nil))
|
||||
// other reads issues' log, as its manifest's reads declare it (ADR 0571): `<module>.<log>`, without writing it.
|
||||
o := mt.MembershipOf("other", "anchor", false, nil)
|
||||
o.Logs = []bus.LogIssued{{Name: "issues.changes", Stream: changes.Stream, Subject: changes.Subject}}
|
||||
mesh.Issue(t, o)
|
||||
conn := connect(t, "node-tools", "anchor")
|
||||
conn.Follow("issues")
|
||||
conn.Follow("other")
|
||||
@@ -111,7 +115,7 @@ func infoOf(t *testing.T, b launch.Bus) map[string]uint64 {
|
||||
// RFC 3339; last is a key's last entry or null; info says entries, bytes, cap, first and last — and no
|
||||
// consumer is made for any of it.
|
||||
func TestALogAnswersEachVerbOnTheBus(t *testing.T) {
|
||||
b, _, mesh := logBuses(t, 1<<20)
|
||||
b, other, mesh := logBuses(t, 1<<20)
|
||||
before := time.Now().Add(-time.Second)
|
||||
|
||||
var seqs []uint64
|
||||
@@ -189,6 +193,64 @@ func TestALogAnswersEachVerbOnTheBus(t *testing.T) {
|
||||
if n := mesh.Consumers(t, "LOG_issues_changes"); n != 0 {
|
||||
t.Fatalf("the runtime made %d consumers of the log; it uses none", n)
|
||||
}
|
||||
|
||||
// From a time (ADR 0571): `since` in place of `from`, RFC 3339; both at once are refused.
|
||||
cut := all.Entries[2].At
|
||||
if p := readOf(t, b, fmt.Sprintf(`{"log":"changes","since":%q}`, cut)); len(p.Entries) == 0 || p.Entries[0].Sequence > seqs[2] ||
|
||||
p.Entries[len(p.Entries)-1].Sequence != seqs[4] {
|
||||
t.Fatalf("a read since %s: %+v", cut, p)
|
||||
}
|
||||
if p := readOf(t, b, fmt.Sprintf(`{"log":"changes","since":%q}`, time.Now().Add(time.Hour).Format(time.RFC3339))); len(p.Entries) != 0 {
|
||||
t.Fatalf("a read since an hour ahead: %+v", p)
|
||||
}
|
||||
if _, err := logAsk(b, "read", `{"log":"changes","from":2,"since":"2026-10-29T00:00:00Z"}`); err == nil {
|
||||
t.Fatal("a read from a sequence and a time was not refused")
|
||||
}
|
||||
// The newest of each key, in the log's order; `**` for the rest, as mesh-sdk says it.
|
||||
raw, err = logAsk(b, "newest", `{"log":"changes"}`)
|
||||
var newest []struct {
|
||||
Sequence uint64 `json:"sequence"`
|
||||
Key string `json:"key"`
|
||||
}
|
||||
if err != nil || json.Unmarshal(raw, &newest) != nil || len(newest) != 3 || newest[0].Key != "13" || newest[1].Key != "14.notes" ||
|
||||
newest[2].Sequence != seqs[4] {
|
||||
t.Fatalf("newest: %s %v", raw, err)
|
||||
}
|
||||
if raw, err := logAsk(b, "newest", `{"log":"changes","keys":"14.**"}`); err != nil || !strings.Contains(string(raw), `"14.notes"`) ||
|
||||
strings.Contains(string(raw), `"12"`) {
|
||||
t.Fatalf("newest of 14.**: %s %v", raw, err)
|
||||
}
|
||||
// Another module's log, read by the name its reads declare; never written.
|
||||
if p := readOf(t, other, `{"log":"issues.changes","from":1}`); len(p.Entries) != 5 {
|
||||
t.Fatalf("other's read of issues.changes: %+v", p)
|
||||
}
|
||||
if _, err := logAsk(other, "append", `{"log":"issues.changes","key":"1","value":{}}`); err == nil ||
|
||||
!strings.Contains(err.Error(), "without writing it") {
|
||||
t.Fatalf("other's append to the log it reads: %v", err)
|
||||
}
|
||||
// A watch hands each entry from where it is asked, then each new one, one at a time.
|
||||
heard := make(chan bus.LogEntry, 16)
|
||||
stop, err := b.LogWatch(json.RawMessage(fmt.Sprintf(`{"log":"changes","keys":"12","from":%d}`, seqs[2])), func(raw json.RawMessage) error {
|
||||
var e bus.LogEntry
|
||||
_ = json.Unmarshal(raw, &e)
|
||||
heard <- e
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer stop()
|
||||
next := appendOf(t, b, `{"log":"changes","key":"12","value":{"op":"late"}}`)
|
||||
for _, want := range []uint64{seqs[2], seqs[4], next} {
|
||||
select {
|
||||
case e := <-heard:
|
||||
if e.Sequence != want || e.Key != "12" {
|
||||
t.Fatalf("the watch handed %d %s, want %d", e.Sequence, e.Key, want)
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatalf("the watch did not hand entry %d", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A page holds about 4 MiB at most, whatever its limit, and says where to continue.
|
||||
@@ -267,7 +329,7 @@ func TestALogNotIssuedIsRefusedWithTheReason(t *testing.T) {
|
||||
} {
|
||||
_, err := logAsk(other, verb, params)
|
||||
if err == nil || !strings.Contains(err.Error(), `other keeps no log called "changes"`) ||
|
||||
!strings.Contains(err.Error(), "issued: none") {
|
||||
!strings.Contains(err.Error(), "issued: issues.changes") {
|
||||
t.Errorf("%s of a log not issued: %v", verb, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -52,6 +52,8 @@ type ListedTool struct {
|
||||
Description string `json:"description"`
|
||||
Input json.RawMessage `json:"input"`
|
||||
Subjects []string `json:"subjects,omitempty"`
|
||||
// Effect is the tool's mark where the runtime knows it (novox/hq ADR 0387 §1): its own tools, the ledger repairs.
|
||||
Effect string `json:"effect,omitempty"`
|
||||
}
|
||||
|
||||
// NodeRuntimeModule is the module that is a machine's runtime, serving every module on it.
|
||||
@@ -59,7 +61,7 @@ const NodeRuntimeModule = "node-tools"
|
||||
|
||||
// runtimeOnly are the words of the runtime's own environment no bundle is given.
|
||||
var runtimeOnly = map[string]bool{ToolEnv: true, ToolModules: true, "MESH_BROKER_FILE": true,
|
||||
"MESH_BROKER_URL": true, "MESH_CONSOLE_LISTEN": true}
|
||||
"MESH_BROKER_URL": true, "MESH_CONSOLE_LISTEN": true, LedgersFile: true}
|
||||
|
||||
// ServedModulesFrom reads MESH_TOOL_MODULES: `<module>=<entrypoint>` entries, comma-separated, several
|
||||
// per module. The one-module form — a bare path, or the runtime's own module — is the per-module
|
||||
@@ -133,6 +135,60 @@ type registration struct {
|
||||
module string // the module's own name, or a seat's
|
||||
owner string // the module whose bundle made it
|
||||
tools []launch.Tool
|
||||
// repairs says it is the runtime's own ledger repairs for the module, not a bundle's (ledgers.go).
|
||||
repairs bool
|
||||
}
|
||||
|
||||
// withRepairs is regs with the module's ledger repairs as its binding now says: added when it binds a ledger, removed
|
||||
// when it binds none.
|
||||
func withRepairs(regs []registration, repairs *registration) []registration {
|
||||
out := make([]registration, 0, len(regs)+1)
|
||||
for _, r := range regs {
|
||||
if !r.repairs {
|
||||
out = append(out, r)
|
||||
}
|
||||
}
|
||||
if repairs != nil {
|
||||
out = append(out, *repairs)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// repairsOf is a module's ledger repairs as a registration, nil when its membership binds no ledger.
|
||||
func (r *runner) repairsOf(module string) *registration {
|
||||
if !hasLedger(r.conn, module) {
|
||||
return nil
|
||||
}
|
||||
return ®istration{module: module, owner: module, tools: r.events.repairTools(module), repairs: true}
|
||||
}
|
||||
|
||||
// refreshRepairs serves each module's ledger repairs as its membership binds a ledger now, re-serving its tools where
|
||||
// that changed (a binding is written when the bus module created the ledger, design 25 § 12.2).
|
||||
func (r *runner) refreshRepairs() {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
changed := false
|
||||
for _, m := range r.modules {
|
||||
l := r.of[m]
|
||||
served := false
|
||||
for _, reg := range l.own {
|
||||
served = served || reg.repairs
|
||||
}
|
||||
repairs := r.repairsOf(m)
|
||||
if served == (repairs != nil) {
|
||||
continue
|
||||
}
|
||||
for i := len(l.serving) - 1; i >= 0; i-- {
|
||||
l.serving[i]()
|
||||
}
|
||||
l.serving, l.names = nil, nil
|
||||
l.own, l.regs = withRepairs(l.own, repairs), withRepairs(l.regs, repairs)
|
||||
r.serve(l)
|
||||
changed = true
|
||||
}
|
||||
if changed {
|
||||
r.bindVerbs()
|
||||
}
|
||||
}
|
||||
|
||||
// loadedModule is one served module as its bundles were last loaded: what they registered, what of it
|
||||
@@ -178,9 +234,17 @@ type runner struct {
|
||||
stops []func() // the watch, the announcement and the seats, in the order they began
|
||||
}
|
||||
|
||||
// Option sets how Run serves, beside what it is given.
|
||||
type Option func(*consumers)
|
||||
|
||||
// WithLedgerWriters gives the runtime the connection each module's ledger history is published on (ledger_credentials.go).
|
||||
func WithLedgerWriters(writer func(module string) (*bus.Conn, error)) Option {
|
||||
return func(c *consumers) { c.ledgers.writer = writer }
|
||||
}
|
||||
|
||||
// Run launches, binds and serves, and serves each module's bundle again when it appears, changes or
|
||||
// goes, with no restart. It answers a stop function.
|
||||
func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, logf func(string, ...any)) (func(), error) {
|
||||
func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, logf func(string, ...any), opts ...Option) (func(), error) {
|
||||
if account := os.Getenv(OperatorAccount); account != "" {
|
||||
home := ""
|
||||
if h := os.Getenv(OperatorHome); h != "" {
|
||||
@@ -203,6 +267,9 @@ func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, log
|
||||
// The module's events, for every child of it that subscribes (ADR 0198): one consumer per module,
|
||||
// bound the first time any of its children subscribes, each event handed to every child that did.
|
||||
r.events = &consumers{conn: conn, logf: logf, of: map[string]*moduleEvents{}}
|
||||
for _, opt := range opts {
|
||||
opt(r.events)
|
||||
}
|
||||
|
||||
for _, s := range served {
|
||||
r.of[s.Module] = r.load(s)
|
||||
@@ -230,6 +297,7 @@ func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, log
|
||||
line += fmt.Sprintf("; not serving %s, whose bundle(s) failed to load", strings.Join(failedNames, ", "))
|
||||
}
|
||||
logf("%s", line)
|
||||
conn.OnMembership(func(bus.Membership) { go func() { r.refreshRepairs(); conn.Flush() }() })
|
||||
stopSeats, again := serveSeats(conn, r.modules, r.registrations, logf)
|
||||
r.seatsAgain = again
|
||||
r.stops = append(r.stops, stopSeats)
|
||||
@@ -341,6 +409,9 @@ func (r *runner) load(s Served) *loadedModule {
|
||||
}
|
||||
}
|
||||
}
|
||||
if repairs := r.repairsOf(s.Module); repairs != nil {
|
||||
l.regs, l.own = append(l.regs, *repairs), append(l.own, *repairs)
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
@@ -411,7 +482,7 @@ func (r *runner) bindVerbs() {
|
||||
answer := ToolsAnswer{Module: module, Tools: []ListedTool{}, Failed: why}
|
||||
for _, t := range tools {
|
||||
answer.Tools = append(answer.Tools, ListedTool{Name: t.Name, Description: t.Description,
|
||||
Input: t.Input, Subjects: subjectsOf(r.conn, module, t.Name)})
|
||||
Input: t.Input, Subjects: subjectsOf(r.conn, module, t.Name), Effect: t.Effect})
|
||||
}
|
||||
return answer, nil
|
||||
})
|
||||
@@ -682,6 +753,10 @@ type consumers struct {
|
||||
seatBound map[string]func()
|
||||
// provisioned is every watch a module's code holds of its values at runtime (provision.go).
|
||||
provisioned provisionWatches
|
||||
// ledgers is how a module's ledger is opened (ledgers.go).
|
||||
ledgers ledgerSource
|
||||
// calls are the tool calls in flight, by the id each bundle was handed (ledgers.go, novox/hq ADR 0597 §7).
|
||||
calls callsInFlight
|
||||
}
|
||||
|
||||
type moduleEvents struct {
|
||||
@@ -868,10 +943,14 @@ type stateAsked struct {
|
||||
Key string `json:"key"`
|
||||
Value json.RawMessage `json:"value"`
|
||||
Revision uint64 `json:"revision"`
|
||||
// TTLSeconds gives a created or put key its own time to live (issue 1952): create makes it only when it has no
|
||||
// value, put renews it whole. Zero is none.
|
||||
TTLSeconds int `json:"ttl_seconds"`
|
||||
}
|
||||
|
||||
// State answers a bundle's `get`, `put`, `delete`, `keys`, `create`, `update` and `history` on its
|
||||
// module's state (ADR 0201, ADR 0259, ADR 0307). The runtime refuses, with the reason, a state the module was not issued and a write to one it only reads.
|
||||
// State answers a bundle's `get`, `put`, `delete`, `keys`, `create`, `update`, `history` and `info` on its
|
||||
// module's state (ADR 0201, ADR 0259, ADR 0307); `create` and `put` take `ttl_seconds`, a key's own time to live,
|
||||
// whose end every watch hears as an `expire` (issue 1952). The runtime refuses, with the reason, a state the module was not issued and a write to one it only reads.
|
||||
func (b *moduleBus) State(verb string, params json.RawMessage) (json.RawMessage, error) {
|
||||
var asked stateAsked
|
||||
if err := json.Unmarshal(params, &asked); err != nil || asked.State == "" {
|
||||
@@ -897,7 +976,13 @@ func (b *moduleBus) State(verb string, params json.RawMessage) (json.RawMessage,
|
||||
}
|
||||
answer = kept
|
||||
case "put":
|
||||
revision, err := conn.StatePut(b.module, asked.State, asked.Key, asked.Value)
|
||||
put := conn.StatePut
|
||||
if asked.TTLSeconds != 0 {
|
||||
put = func(module, name, key string, value json.RawMessage) (uint64, error) {
|
||||
return conn.StateRenewTTL(module, name, key, value, time.Duration(asked.TTLSeconds)*time.Second)
|
||||
}
|
||||
}
|
||||
revision, err := put(b.module, asked.State, asked.Key, asked.Value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -907,6 +992,13 @@ func (b *moduleBus) State(verb string, params json.RawMessage) (json.RawMessage,
|
||||
return nil, err
|
||||
}
|
||||
answer = map[string]any{}
|
||||
case "info":
|
||||
// The bucket's fill against its caps, so a bundle's tools can say how full it is.
|
||||
info, err := conn.StateInfo(b.module, asked.State)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
answer = info
|
||||
case "keys":
|
||||
keys, err := conn.StateKeys(b.module, asked.State)
|
||||
if err != nil {
|
||||
@@ -915,13 +1007,23 @@ func (b *moduleBus) State(verb string, params json.RawMessage) (json.RawMessage,
|
||||
answer = keys
|
||||
case "create":
|
||||
// Only when the key has no value: of two writers making it, one wins (novox/hq ADR 0259).
|
||||
revision, err := conn.StateCreate(b.module, asked.State, asked.Key, asked.Value)
|
||||
create := conn.StateCreate
|
||||
if asked.TTLSeconds != 0 {
|
||||
create = func(module, name, key string, value json.RawMessage) (uint64, error) {
|
||||
return conn.StateCreateTTL(module, name, key, value, time.Duration(asked.TTLSeconds)*time.Second)
|
||||
}
|
||||
}
|
||||
revision, err := create(b.module, asked.State, asked.Key, asked.Value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
answer = map[string]any{"revision": revision}
|
||||
case "update":
|
||||
// Only when the key is still at the revision read: a second answer to one ask loses (ADR 0259).
|
||||
if asked.TTLSeconds != 0 {
|
||||
return nil, fmt.Errorf("mesh/state.update gives no time to live: the bus sets one only when a key is " +
|
||||
"created; put the key with ttl_seconds to renew it whole (issue 1952)")
|
||||
}
|
||||
revision, err := conn.StateUpdate(b.module, asked.State, asked.Key, asked.Value, asked.Revision)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -942,9 +1044,24 @@ type logAsked struct {
|
||||
From uint64 `json:"from"`
|
||||
Keys string `json:"keys"`
|
||||
Limit int `json:"limit"`
|
||||
// Since is a read's or a watch's start as an RFC 3339 time, in place of From (novox/hq ADR 0571).
|
||||
Since string `json:"since"`
|
||||
}
|
||||
|
||||
// Log answers a bundle's `append`, `read`, `last` and `info` on its module's log (ADR 0297). The runtime
|
||||
// since is the asked start time, zero when none.
|
||||
func (a logAsked) since() (time.Time, error) {
|
||||
if a.Since == "" {
|
||||
return time.Time{}, nil
|
||||
}
|
||||
at, err := time.Parse(time.RFC3339Nano, a.Since)
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("since is an RFC 3339 time (2026-10-29T09:30:00Z), not %q", a.Since)
|
||||
}
|
||||
return at, nil
|
||||
}
|
||||
|
||||
// Log answers a bundle's `append`, `read`, `newest`, `last` and `info` on its module's log and on a log its module reads
|
||||
// as `<module>.<log>` (ADR 0297, ADR 0571). The runtime
|
||||
// refuses, with the reason, a log the module was not issued; an append whose `expect` is stale is
|
||||
// refused with the code of a lost state update.
|
||||
func (b *moduleBus) Log(verb string, params json.RawMessage) (json.RawMessage, error) {
|
||||
@@ -962,11 +1079,22 @@ func (b *moduleBus) Log(verb string, params json.RawMessage) (json.RawMessage, e
|
||||
}
|
||||
answer = map[string]any{"sequence": seq}
|
||||
case "read":
|
||||
page, err := conn.LogRead(b.module, asked.Log, asked.From, asked.Keys, asked.Limit)
|
||||
since, err := asked.since()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
page, err := conn.LogRead(b.module, asked.Log, asked.From, since, asked.Keys, asked.Limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
answer = page
|
||||
case "newest":
|
||||
// The newest entry of each key the pattern names (mesh-sdk process.LogReader.Newest).
|
||||
entries, err := conn.LogNewest(b.module, asked.Log, asked.Keys)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
answer = entries
|
||||
case "last":
|
||||
entry, err := conn.LogLast(b.module, asked.Log, asked.Key)
|
||||
if err != nil {
|
||||
@@ -988,6 +1116,26 @@ func (b *moduleBus) Log(verb string, params json.RawMessage) (json.RawMessage, e
|
||||
return json.Marshal(answer)
|
||||
}
|
||||
|
||||
// LogWatch hands a bundle each entry of a log, `{log, keys, from, since}`: from a sequence, from a time, or with neither
|
||||
// each new one, one at a time (novox/hq ADR 0571; mesh-sdk process.LogReader.Watch).
|
||||
func (b *moduleBus) LogWatch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
|
||||
var asked logAsked
|
||||
if err := json.Unmarshal(params, &asked); err != nil || asked.Log == "" {
|
||||
return nil, fmt.Errorf("mesh/log.watch names no log: {log, keys, from, since}")
|
||||
}
|
||||
since, err := asked.since()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return b.all.conn.LogWatch(b.module, asked.Log, asked.Keys, asked.From, since, func(e bus.LogEntry) error {
|
||||
raw, err := json.Marshal(e)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return deliver(raw)
|
||||
})
|
||||
}
|
||||
|
||||
// Watch hands a bundle its module's state as it is and as it changes (ADR 0201): `{state, key}`, the
|
||||
// key a pattern with `*` and `**`, empty for every key.
|
||||
func (b *moduleBus) Watch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
|
||||
|
||||
@@ -70,7 +70,7 @@ func TestABundlesSeatTrafficGoesThroughItsModulesBus(t *testing.T) {
|
||||
if !strings.Contains(string(first), `"duplicate":false`) || !strings.Contains(string(again), `"duplicate":true`) {
|
||||
t.Errorf("the same id twice: %s, then %s", first, again)
|
||||
}
|
||||
if _, err := asking.Seat("publish", json.RawMessage(`{"subject":"mesh.seat.operator-channel.accept.ask.mesh-controller","body":{}}`), nil); err == nil {
|
||||
if _, err := asking.Seat("publish", json.RawMessage(`{"subject":"mesh.seat.operator-channel.accept.ask.mesh-placement","body":{}}`), nil); err == nil {
|
||||
t.Error("a module submitted under another's name")
|
||||
}
|
||||
got, err := asking.Seat("record", json.RawMessage(`{"key":"d-1"}`), nil)
|
||||
|
||||
@@ -4,8 +4,10 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/novox/mesh-tools/node-tools/internal/bus"
|
||||
"github.com/novox/mesh-tools/node-tools/internal/launch"
|
||||
@@ -18,8 +20,10 @@ func counterBus(t *testing.T) launch.Bus {
|
||||
t.Helper()
|
||||
mesh := mt.New(t)
|
||||
mesh.Bucket(t, "issues_numbers")
|
||||
mesh.BucketKeyTTL(t, "issues_wakes")
|
||||
m := mt.MembershipOf("issues", "anchor", false, nil)
|
||||
m.State = []bus.StateIssued{{Name: "numbers", Bucket: "issues_numbers", Writes: true}}
|
||||
m.State = []bus.StateIssued{{Name: "numbers", Bucket: "issues_numbers", Writes: true},
|
||||
{Name: "wakes", Bucket: "issues_wakes", Writes: true}}
|
||||
mesh.Issue(t, m)
|
||||
conn := connect(t, "node-tools", "anchor")
|
||||
conn.Follow("issues")
|
||||
@@ -131,6 +135,9 @@ func TestTwoWritersIncrementingOneCounterOnTheBusLoseNoUpdate(t *testing.T) {
|
||||
// A write at a revision that is no longer the key's is refused with the lost-write code and writes
|
||||
// nothing; create is refused for a key with a value and taken for one deleted; an update names the
|
||||
// revision it read, never zero.
|
||||
// A key's own time to live (issue 1952, mesh-sdk process.State.CreateTTL and RenewTTL): created once, renewed whole by a
|
||||
// put, its end heard by a watch as an expiry; refused, writing nothing, on a bucket that takes none, under a second, and
|
||||
// on an update.
|
||||
func TestAStaleWriteIsRefusedAndWritesNothing(t *testing.T) {
|
||||
b := counterBus(t)
|
||||
state := func(verb string, extra map[string]any) (uint64, error) {
|
||||
@@ -200,4 +207,63 @@ func TestAStaleWriteIsRefusedAndWritesNothing(t *testing.T) {
|
||||
if v := value(); v != "1" {
|
||||
t.Fatalf("after delete and create: %s", v)
|
||||
}
|
||||
|
||||
// A time to live per key: refused where it cannot be kept, writing nothing.
|
||||
for _, c := range []struct {
|
||||
verb, state, why string
|
||||
ttl int
|
||||
}{
|
||||
{"create", "numbers", "does not take keys with their own time to live", 60},
|
||||
{"put", "numbers", "does not take keys with their own time to live", 60},
|
||||
{"create", "wakes", "shorter than the bus keeps", -1},
|
||||
{"update", "wakes", "gives no time to live", 60},
|
||||
} {
|
||||
_, err := ask(b, c.verb, map[string]any{"state": c.state, "key": "late", "value": 1, "revision": 1, "ttl_seconds": c.ttl})
|
||||
if err == nil || !strings.Contains(err.Error(), c.why) {
|
||||
t.Errorf("%s with ttl %d on %s: want refused for %q, got %v", c.verb, c.ttl, c.state, c.why, err)
|
||||
}
|
||||
if raw, _ := ask(b, "get", map[string]any{"state": c.state, "key": "late"}); string(raw) != "null" {
|
||||
t.Errorf("%s with ttl %d on %s wrote %s", c.verb, c.ttl, c.state, raw)
|
||||
}
|
||||
}
|
||||
|
||||
// The bucket's fill against its caps, for a bundle's tools.
|
||||
if raw, err := ask(b, "info", map[string]any{"state": "wakes"}); err != nil || !strings.Contains(string(raw),
|
||||
`"bucket":"issues_wakes","bytes":0,"keys":0,"max_bytes":-1,"max_value_bytes":262144,"history":1,"ttl_seconds":0,"key_ttl":true`) {
|
||||
t.Errorf("info of a key-ttl bucket answered %s, %v", raw, err)
|
||||
}
|
||||
|
||||
// Created once, renewed whole, and its end heard: a wake with no timer.
|
||||
heard := make(chan bus.StateChange, 8)
|
||||
stop, err := b.Watch(json.RawMessage(`{"state":"wakes"}`), func(raw json.RawMessage) error {
|
||||
var c bus.StateChange
|
||||
_ = json.Unmarshal(raw, &c)
|
||||
heard <- c
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer stop()
|
||||
if _, err := ask(b, "create", map[string]any{"state": "wakes", "key": "settle", "value": 1, "ttl_seconds": 60}); err != nil {
|
||||
t.Fatalf("create with a time to live: %v", err)
|
||||
}
|
||||
if _, err := ask(b, "create", map[string]any{"state": "wakes", "key": "settle", "value": 2, "ttl_seconds": 60}); !isLost(err) {
|
||||
t.Fatalf("a second create with a time to live: %v", err)
|
||||
}
|
||||
if _, err := ask(b, "put", map[string]any{"state": "wakes", "key": "settle", "value": 3, "ttl_seconds": 1}); err != nil {
|
||||
t.Fatalf("a put renewing the time to live: %v", err)
|
||||
}
|
||||
var ops []string
|
||||
for want := "put delete put expire"; strings.Join(ops, " ") != want; {
|
||||
select {
|
||||
case c := <-heard:
|
||||
ops = append(ops, c.Op)
|
||||
if c.Op == "put" && len(ops) == 3 && string(c.Value) != "3" {
|
||||
t.Fatalf("the renewed key holds %s", c.Value)
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatalf("the watch heard %q, want %q", strings.Join(ops, " "), want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+12
-3
@@ -1,7 +1,8 @@
|
||||
{
|
||||
"module": "node-tools",
|
||||
"version": "1",
|
||||
"version": "2",
|
||||
"slug": "node-tools",
|
||||
"tool-runner": true,
|
||||
"invokes": [
|
||||
"*"
|
||||
],
|
||||
@@ -9,13 +10,16 @@
|
||||
"mesh-knowledge.reviewed",
|
||||
"mesh-delivery-gate.transition",
|
||||
"gitea.forge.status",
|
||||
"mesh-controller.checked",
|
||||
"mesh-self-check.checked",
|
||||
"operator-channel.decided.claude-code"
|
||||
],
|
||||
"provides": [
|
||||
{
|
||||
"name": "mcp-endpoint",
|
||||
"scope": "node",
|
||||
"credential": {
|
||||
"form": "none"
|
||||
},
|
||||
"answered-by": "helper"
|
||||
}
|
||||
],
|
||||
@@ -65,6 +69,11 @@
|
||||
"bound"
|
||||
],
|
||||
"reads": [
|
||||
"mesh-controller.assignments"
|
||||
"mesh-provisioner.provisions",
|
||||
"mesh-provisioner.plans",
|
||||
"mesh-delivery-assigner.seat-holders",
|
||||
"mesh-delivery-assigner.placement",
|
||||
"mesh-delivery-indexer.modules",
|
||||
"mesh-delivery-indexer.manifests"
|
||||
]
|
||||
}
|
||||
|
||||
Generated
-4
@@ -11,10 +11,6 @@
|
||||
"@novox/mesh-sdk": "^0.1.6",
|
||||
"nats": "^2.29.0"
|
||||
},
|
||||
"bin": {
|
||||
"mesh": "dist/mesh.js",
|
||||
"mesh-tools": "dist/main.js"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.20.1",
|
||||
"esbuild": "^0.25.0",
|
||||
|
||||
+1
-10
@@ -1,17 +1,8 @@
|
||||
{
|
||||
"name": "@novox/mesh-tools",
|
||||
"version": "0.1.0",
|
||||
"description": "The Novox Mesh tool runtime \u2014 binds the mesh broker and serves the assigned modules' tools.",
|
||||
"description": "The Novox Mesh tool runtime — binds the mesh broker and serves the assigned modules' tools.",
|
||||
"type": "module",
|
||||
"bin": {
|
||||
"mesh-tools": "./dist/main.js",
|
||||
"mesh": "./dist/mesh.js"
|
||||
},
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"pretest": "tsc",
|
||||
"test": "node --test --test-concurrency=1 --experimental-strip-types 'test/*.test.ts'"
|
||||
},
|
||||
"dependencies": {
|
||||
"@novox/mesh-sdk": "^0.1.6",
|
||||
"nats": "^2.29.0"
|
||||
|
||||
@@ -1,139 +0,0 @@
|
||||
// What a runtime serves, it announces (novox/hq ADR 0197): the NATS services protocol's discovery —
|
||||
// `$SRV.PING`, `$SRV.INFO`, `$SRV.STATS`, and the same followed by the service's name and its id —
|
||||
// answered in the io.nats.micro.v1 format with what is served at the moment of the request. Serving is
|
||||
// unchanged; this only says what is served. One service per runtime process, because the bus admits
|
||||
// one reply per request from each responder: one endpoint per tool per subject, its metadata saying
|
||||
// which module, seat, scope and machine it is. The same shape the Go runtime answers.
|
||||
|
||||
import { StringCodec } from "nats";
|
||||
import { asSchema } from "@novox/mesh-sdk/stdio";
|
||||
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
|
||||
import type { RuntimeBroker } from "./broker-nats.js";
|
||||
|
||||
const sc = StringCodec();
|
||||
|
||||
export const VERSION = "0.1.0";
|
||||
export const INFO_RESPONSE = "io.nats.micro.v1.info_response";
|
||||
export const PING_RESPONSE = "io.nats.micro.v1.ping_response";
|
||||
export const STATS_RESPONSE = "io.nats.micro.v1.stats_response";
|
||||
|
||||
/** One tool served on one subject, as announced. */
|
||||
export interface Endpoint {
|
||||
kind: "tool" | "seat";
|
||||
module: string;
|
||||
tool: string;
|
||||
seat?: string;
|
||||
scope?: "mesh" | "node";
|
||||
node: string;
|
||||
description: string;
|
||||
schema: unknown;
|
||||
interchangeable: boolean;
|
||||
subject: string;
|
||||
queue?: string;
|
||||
}
|
||||
|
||||
/** A seat's verb as the runtime serves it, with the definition that answers it. */
|
||||
export interface ServedSeatVerb {
|
||||
seat: string;
|
||||
verb: string;
|
||||
subject: string;
|
||||
holder: string;
|
||||
tool: ToolDefinition;
|
||||
}
|
||||
|
||||
export interface Service {
|
||||
name: string;
|
||||
id: string;
|
||||
description: string;
|
||||
metadata: Record<string, string>;
|
||||
}
|
||||
|
||||
/** The endpoint's name as the protocol allows it; the metadata, not the name, identifies it. */
|
||||
function nameOf(e: Endpoint): string {
|
||||
const clean = (s: string) => s.replace(/[^A-Za-z0-9_-]/g, "_");
|
||||
return `${clean(e.kind === "seat" ? e.seat ?? "" : e.module)}__${clean(e.tool)}`;
|
||||
}
|
||||
|
||||
/** The info_response for these endpoints. */
|
||||
export function info(s: Service, endpoints: Endpoint[]): Record<string, unknown> {
|
||||
return {
|
||||
name: s.name, id: s.id, version: VERSION, metadata: s.metadata, type: INFO_RESPONSE, description: s.description,
|
||||
endpoints: endpoints.map((e) => {
|
||||
const metadata: Record<string, string> = {
|
||||
kind: e.kind, module: e.module, tool: e.tool, node: e.node, description: e.description,
|
||||
schema: JSON.stringify(e.schema ?? {}), interchangeable: e.interchangeable ? "true" : "false",
|
||||
};
|
||||
if (e.kind === "seat") {
|
||||
metadata.seat = e.seat ?? "";
|
||||
metadata.scope = e.scope ?? "mesh";
|
||||
}
|
||||
return { name: nameOf(e), subject: e.subject, queue_group: e.queue ?? "", metadata };
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
/** Everything served now: each module's tools on every subject issued for them, and each held
|
||||
* seat's verbs on the seat's subject. */
|
||||
export function endpointsOf(
|
||||
broker: RuntimeBroker,
|
||||
own: { module: string; tools: ToolDefinition[] }[],
|
||||
seats: ServedSeatVerb[],
|
||||
): Endpoint[] {
|
||||
const node = broker.node ?? "";
|
||||
const out: Endpoint[] = [];
|
||||
for (const { module, tools } of own) {
|
||||
for (const t of tools) {
|
||||
const served = broker.servedOn ? broker.servedOn(module, t.name) : [];
|
||||
const interchangeable = served.some((s) => s.subject === `mesh.mod.${module}.tool.${t.name}`);
|
||||
for (const s of served) {
|
||||
out.push({ kind: "tool", module, tool: t.name, node, description: t.description, schema: asSchema(t.input),
|
||||
interchangeable, subject: s.subject, queue: s.queue });
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const v of seats) {
|
||||
out.push({ kind: "seat", module: v.holder, tool: v.verb, seat: v.seat,
|
||||
scope: node && v.subject.endsWith(`.${node}`) ? "node" : "mesh", node, description: v.tool.description,
|
||||
schema: asSchema(v.tool.input), interchangeable: false, subject: v.subject });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Answer discovery for one service until stopped. */
|
||||
export function announce(broker: RuntimeBroker, s: Service, current: () => Endpoint[]): () => void {
|
||||
const started = new Date().toISOString();
|
||||
const identity = { name: s.name, id: s.id, version: VERSION, metadata: s.metadata };
|
||||
const answer = (subject: string): Uint8Array | undefined => {
|
||||
const parts = subject.split(".");
|
||||
if (parts[0] !== "$SRV" || parts.length < 2) return undefined;
|
||||
if (parts.length >= 3 && parts[2] !== s.name) return undefined; // another service's
|
||||
if (parts.length >= 4 && parts[3] !== s.id) return undefined; // another instance's
|
||||
let v: unknown;
|
||||
switch (parts[1]) {
|
||||
case "PING":
|
||||
v = { ...identity, type: PING_RESPONSE };
|
||||
break;
|
||||
case "INFO":
|
||||
v = info(s, current());
|
||||
break;
|
||||
case "STATS":
|
||||
v = { ...identity, type: STATS_RESPONSE, started, endpoints: current().map((e) => ({
|
||||
name: nameOf(e), subject: e.subject, queue_group: e.queue ?? "", num_requests: 0, num_errors: 0,
|
||||
last_error: "", processing_time: 0, average_processing_time: 0 })) };
|
||||
break;
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
return sc.encode(JSON.stringify(v));
|
||||
};
|
||||
const stops: (() => void)[] = [];
|
||||
// Exactly the questions asked of every service and of this one by name and instance — what the
|
||||
// grants allow (novox/hq ADR 0197). A wildcard is refused by the bus, and a refused subscription
|
||||
// ends a runtime: on 2026-10-03 it crash-looped every container that announced itself.
|
||||
for (const verb of ["PING", "INFO", "STATS"]) {
|
||||
for (const subject of [`$SRV.${verb}`, `$SRV.${verb}.${s.name}`, `$SRV.${verb}.${s.name}.${s.id}`]) {
|
||||
stops.push(broker.raw!(subject, (subj) => answer(subj)));
|
||||
}
|
||||
}
|
||||
return () => stops.forEach((stop) => stop());
|
||||
}
|
||||
@@ -1,680 +0,0 @@
|
||||
// The tool runtime's broker client, on NATS.
|
||||
//
|
||||
// **The sdk's contract does not change** (novox/hq ADR 0106, ADR 0039): a module is written
|
||||
// against `request`, `handle`, `publish`, `subscribe`, `close`, and the runtime implements them.
|
||||
// That is why a module built before any of this runs on the new runtime without a rebuild, and
|
||||
// why the sdk's own diff for the whole bus change is three comments.
|
||||
//
|
||||
// Underneath, everything is a subject and durability is JetStream (novox/hq design 25,
|
||||
// design 29).
|
||||
//
|
||||
// mesh.mod.<module>.event.<type> an event this module emits
|
||||
// mesh.mod.<module>.tool.<tool> a tool this module serves
|
||||
// mesh.seat.<seat>.accept.<verb> work submitted to a role
|
||||
//
|
||||
// The module never writes one of those: it names its events and tools locally and the mesh
|
||||
// derives the subject (design 29 §1), so reorganising the subject space leaves every module
|
||||
// correct.
|
||||
|
||||
import { AsyncLocalStorage } from "node:async_hooks";
|
||||
import { createHash } from "node:crypto";
|
||||
import net from "node:net";
|
||||
import tls from "node:tls";
|
||||
import { connect as natsConnect, headers as natsHeaders, StringCodec, type JsMsg, type Subscription, type TlsOptions } from "nats";
|
||||
import type { Broker, Envelope, EventHeaders } from "@novox/mesh-sdk/messaging";
|
||||
|
||||
const sc = StringCodec();
|
||||
|
||||
/** Requests wait this long for an answer before failing. Unchanged from what modules already
|
||||
* expect, so a module's timeout handling is not something the bus quietly redefines. */
|
||||
const REQUEST_TIMEOUT_MS = 30_000;
|
||||
|
||||
export class PinMismatchError extends Error {}
|
||||
|
||||
/** A broker credential as the mesh delivers it (novox/hq ADR 0120): the bus's address, the
|
||||
* fingerprint of the certificate it must present, and the node and module the account is scoped
|
||||
* to — the runtime derives its subjects from those rather than being told them. */
|
||||
export interface Credential {
|
||||
url: string;
|
||||
fingerprint?: string;
|
||||
node?: string;
|
||||
module?: string;
|
||||
user?: string;
|
||||
password?: string;
|
||||
/** The seats this module claims, with the verbs each promises (novox/hq ADR 0159). The runtime
|
||||
* serves each claimed seat's verbs with its tools of the same name; the bus admits only the
|
||||
* holder's subscription, so claiming and not holding costs a refused subscription and nothing else. */
|
||||
claims?: { seat: string; scope?: string; serves?: string[] }[];
|
||||
}
|
||||
|
||||
/**
|
||||
* What the mesh issued this assignment (novox/hq ADR 0160): where its tools are served, in which
|
||||
* queue, the verbs of the seats it holds, where its events land, what it may reach. Read from the
|
||||
* ASSIGNMENTS stream at `mesh.assignment.<node>.<module>` — the one subject a runtime derives for
|
||||
* itself — and followed live. Absent for a mesh older than the membership, and then the runtime
|
||||
* serves the shape it always derived, and says so.
|
||||
*/
|
||||
export interface Membership {
|
||||
node: string;
|
||||
module: string;
|
||||
/** Addresses a tool is answered on; `{tool}` stands for the tool's name. */
|
||||
serves: { subject: string; queue?: string }[];
|
||||
seats?: { seat: string; verb: string; subject: string }[];
|
||||
emits: string;
|
||||
reaches?: Record<string, string[]>;
|
||||
tools: string;
|
||||
}
|
||||
|
||||
/** What a tool call answers: the module's own result, and which machine answered it
|
||||
* (novox/hq ADR 0159) — a module on several machines is otherwise an answer from nowhere. */
|
||||
export interface Answered<Res> {
|
||||
result: Res;
|
||||
node?: string;
|
||||
}
|
||||
|
||||
/** The bus as the runtime sees it: the sdk's contract, and the few things only the runtime needs —
|
||||
* an answer that says which machine gave it, serving a subject that is not a module's own tool
|
||||
* (a seat's verb), and following the memberships of the modules it serves beside its own. */
|
||||
export interface RuntimeBroker extends Broker {
|
||||
/** Call a tool by key, or — when `on` names a subject the mesh listed for it (ADR 0160) — there. */
|
||||
ask<Req, Res>(key: string, body: Req, on?: string): Promise<Answered<Res>>;
|
||||
handleSubject<Req, Res>(subject: string, handler: (body: Req) => Promise<Res>): Promise<() => void>;
|
||||
/**
|
||||
* Serve another module's tools from this connection: read its membership on this machine and
|
||||
* follow it, so `handle("<module>.<tool>")` is served where the mesh issued that module (novox/hq
|
||||
* ADR 0175: one runtime per node, every assigned module's tools). The account must be allowed
|
||||
* to read that membership and to subscribe its subjects — the node's is; a module's own is not,
|
||||
* and is refused by the bus, not here.
|
||||
*/
|
||||
follow(module: string): Promise<void>;
|
||||
/** What the mesh issued this assignment — this module's when none is named — or undefined when
|
||||
* nothing has been issued yet. */
|
||||
membership(module?: string): Membership | undefined;
|
||||
/** Called when the mesh issues a new membership to any module this connection follows; the
|
||||
* runtime re-serves on it. The membership says which module it is for. */
|
||||
onMembership(handler: (m: Membership) => void): void;
|
||||
/** The modules whose memberships this connection follows: its own and every one `follow` added. */
|
||||
serving(): string[];
|
||||
/** The module this connection is: what its credential named, and what a bare key serves as. */
|
||||
readonly module: string;
|
||||
/** Where a served module's tool is answered right now (ADR 0197: what it serves, it announces). */
|
||||
servedOn?(module: string, tool: string): { subject: string; queue?: string }[];
|
||||
/** Answer a subject in a format of its own, not a tool's reply envelope — the NATS services
|
||||
* protocol's discovery (novox/hq ADR 0197). An undefined answer is no reply. */
|
||||
raw?(subject: string, answer: (subject: string, data: Uint8Array) => Uint8Array | undefined): () => void;
|
||||
/** The machine this connection serves on, when its credential names one. */
|
||||
readonly node?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Which module's tool is at work on this connection, when one is (novox/hq ADR 0175). One runtime
|
||||
* carries many modules' tools, and an event a tool emits must land on the emitting *module's*
|
||||
* subject, not the runtime's — so the runtime runs each tool inside this store, and `publish`
|
||||
* reads the module from it. Outside a tool the connection's own module stands.
|
||||
*/
|
||||
export const atWork = new AsyncLocalStorage<{ module: string }>();
|
||||
|
||||
const ASSIGNMENTS_STREAM = "ASSIGNMENTS";
|
||||
|
||||
/** The one address a runtime derives for itself (ADR 0160). */
|
||||
export function membershipSubject(node: string, module: string): string {
|
||||
return `mesh.assignment.${node}.${module}`;
|
||||
}
|
||||
|
||||
/** Whether a connection failure is worth retrying, or is a fact about this configuration that
|
||||
* retrying cannot change. The runtime's supervisor asks this and does not need to know what it
|
||||
* is connected to. */
|
||||
export function fatalBrokerReason(err: unknown): string | null {
|
||||
if (err instanceof PinMismatchError) return "the bus's certificate does not match the pin";
|
||||
const e = err as { code?: string; message?: string };
|
||||
const message = typeof e?.message === "string" ? e.message : String(err);
|
||||
if (e?.code === "ERR_INVALID_URL" || /invalid url/i.test(message)) {
|
||||
return "the bus address is not a usable URL";
|
||||
}
|
||||
if (/authorization violation|user authentication expired|permissions violation/i.test(message)) {
|
||||
return "the bus refused this account";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Connect to the mesh bus and return a Broker.
|
||||
*
|
||||
* **A module's subjects come from its credential, not from its calls.** `node` and `module` name
|
||||
* the account the mesh issued, and every subject this client publishes or subscribes is derived
|
||||
* from them — so a module cannot name another's namespace even by mistake, and what it emits
|
||||
* matches what the mesh authorised (ADR 0074's identity rule).
|
||||
*/
|
||||
export async function connectNats(
|
||||
target: string | Credential,
|
||||
opts: { module?: string } = {},
|
||||
): Promise<RuntimeBroker> {
|
||||
const cred: Credential = typeof target === "string" ? { url: target } : target;
|
||||
const self = cred.module ?? opts.module;
|
||||
if (!self) {
|
||||
throw new Error(
|
||||
"a broker credential with no module: the runtime derives its subjects from the account " +
|
||||
"the mesh issued, and cannot guess which module it is",
|
||||
);
|
||||
}
|
||||
|
||||
const conn = await natsConnect({
|
||||
servers: cred.url,
|
||||
user: cred.user,
|
||||
pass: cred.password,
|
||||
name: `${cred.node ?? "?"}.${self}`,
|
||||
tls: cred.fingerprint ? await pinnedTls(cred.url, cred.fingerprint) : undefined,
|
||||
// Its own inbox, not a random one: every user's inbox is private to it (design 25 §4), and the
|
||||
// grant names `_INBOX.<user>.>` — a reply space the client invented would be refused, and with
|
||||
// it every pull for the next message and every answer to a tool call.
|
||||
inboxPrefix: cred.user ? `_INBOX.${cred.user}` : undefined,
|
||||
// Reconnect forever: the bus being restarted is an upgrade, not a reason for every module on
|
||||
// the mesh to exit. `close()` stays the only thing that ends the connection.
|
||||
maxReconnectAttempts: -1,
|
||||
});
|
||||
const js = conn.jetstream();
|
||||
|
||||
const subs: Subscription[] = [];
|
||||
// Every registration, and the one reader that dispatches to them. A module has one durable
|
||||
// consumer; the loop belongs to the connection rather than to a subscription.
|
||||
const listeners: { pattern: string; handler: (env: Envelope<unknown>) => Promise<void> }[] = [];
|
||||
let reading: Awaited<ReturnType<Awaited<ReturnType<typeof js.consumers.get>>["consume"]>> | undefined;
|
||||
let closed = false;
|
||||
|
||||
const node = cred.node;
|
||||
|
||||
// The memberships, one per module this connection serves, each read once and followed. A
|
||||
// module's own runtime follows one, its own; the node's runtime follows one per assigned module
|
||||
// (novox/hq ADR 0175) — the same subject shape, the same stream, read as many times as there are
|
||||
// modules, and nothing on the bus learns a new shape for it. A direct get is one request on the
|
||||
// stream's API, which is the whole of what an account may ask JetStream for a subject it is
|
||||
// granted; a 404 is a mesh that has not issued one, which is a fact to say and not an error to
|
||||
// retry.
|
||||
const issued = new Map<string, Membership | undefined>();
|
||||
const issuedHandlers: ((m: Membership) => void)[] = [];
|
||||
const follow = async (module: string): Promise<void> => {
|
||||
if (!node || issued.has(module)) return;
|
||||
issued.set(module, undefined);
|
||||
const subjectOfTheirs = membershipSubject(node, module);
|
||||
try {
|
||||
// The subject-addressed form of a direct get — the stream, then the subject, nothing in
|
||||
// the body — because that is the one address the mesh grants this account on the
|
||||
// stream's API; the body form asks the stream's root, which it may not.
|
||||
const got = await conn.request(`$JS.API.DIRECT.GET.${ASSIGNMENTS_STREAM}.${subjectOfTheirs}`,
|
||||
new Uint8Array(0), { timeout: 5_000 });
|
||||
const status = got.headers?.code ?? 0;
|
||||
if (status === 0 && got.data.length > 0) {
|
||||
issued.set(module, JSON.parse(sc.decode(got.data)) as Membership);
|
||||
}
|
||||
} catch {
|
||||
// Not readable here: an older mesh, a stream not yet asserted, or no grant. Said below.
|
||||
}
|
||||
if (!issued.get(module)) {
|
||||
console.log(`[mesh-tools] no membership issued for ${module} on ${node} yet; serving the derived shape until one arrives`);
|
||||
}
|
||||
try {
|
||||
const live = conn.subscribe(subjectOfTheirs);
|
||||
subs.push(live);
|
||||
void (async () => {
|
||||
for await (const msg of live) {
|
||||
try {
|
||||
const m = JSON.parse(sc.decode(msg.data)) as Membership;
|
||||
issued.set(module, m);
|
||||
console.log(`[mesh-tools] ${module} on ${node} was issued a new membership; re-serving on it`);
|
||||
for (const h of issuedHandlers) h(m);
|
||||
} catch (err) {
|
||||
console.log(`[mesh-tools] a membership arrived that is not one: ${err}`);
|
||||
}
|
||||
}
|
||||
})();
|
||||
} catch {
|
||||
// A subscription this account may not make is a mesh older than the membership.
|
||||
}
|
||||
};
|
||||
await follow(self);
|
||||
|
||||
/** The subjects a tool of a served module is served on: from that module's membership when
|
||||
* issued, derived otherwise (the shape the mesh issues on day one, so the two agree). */
|
||||
const servedOn = (module: string, tool: string): { subject: string; queue?: string }[] => {
|
||||
const m = issued.get(module);
|
||||
if (m) {
|
||||
const out = m.serves.map((s) => ({ subject: s.subject.replace("{tool}", tool), queue: s.queue }));
|
||||
// The verb that lists what this module serves is answered on the mesh's plain address for it
|
||||
// whatever the placement — one answer suffices, so a queue — and on this machine's beside it.
|
||||
if (tool === "tools" && m.tools && !out.some((s) => s.subject === m.tools)) {
|
||||
out.unshift({ subject: m.tools, queue: `serve.${module}` });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
const base = `mesh.mod.${module}.tool.${tool}`;
|
||||
const out: { subject: string; queue?: string }[] = [{ subject: base, queue: `serve.${module}` }];
|
||||
if (node) out.push({ subject: `${base}.${node}` });
|
||||
return out;
|
||||
};
|
||||
|
||||
/** Where a call by key goes: a subject this connection's own membership says it reaches, when it
|
||||
* says one — the machine's when named — else the derived shape. */
|
||||
const reachedAt = (key: string): string => {
|
||||
const [name, wanted] = key.split("@", 2);
|
||||
const reach = issued.get(self)?.reaches?.[name];
|
||||
if (reach && reach.length > 0) {
|
||||
if (wanted) {
|
||||
const at = reach.find((s) => s.endsWith(`.${wanted}`));
|
||||
if (at) return at;
|
||||
} else {
|
||||
return reach[0];
|
||||
}
|
||||
}
|
||||
return toolSubject(key, self);
|
||||
};
|
||||
|
||||
/** Answer one subject with one handler, and say which machine answered (novox/hq ADR 0159). */
|
||||
const answerOn = <Req, Res>(
|
||||
subject: string,
|
||||
queue: string | undefined,
|
||||
handler: (body: Req) => Promise<Res>,
|
||||
): (() => void) => {
|
||||
const sub = conn.subscribe(subject, queue ? { queue } : {});
|
||||
subs.push(sub);
|
||||
void (async () => {
|
||||
// **A refused subscription is said, never fatal** (novox/hq 04-ISSUES/218, as 217 for the
|
||||
// announcements). The grants are the mesh's word on what this account may answer; a subject
|
||||
// they leave out — a seat claimed here and held elsewhere — costs that subject, never the
|
||||
// module's other tools, its handlers and its provisioning. Unhandled, the refusal ended the
|
||||
// process and a module's runtime crash-looped on 2026-10-04.
|
||||
try {
|
||||
for await (const msg of sub) {
|
||||
let reply: { result?: Res; error?: string; node?: string };
|
||||
try {
|
||||
reply = { result: await handler(JSON.parse(sc.decode(msg.data)) as Req) };
|
||||
} catch (err) {
|
||||
// The caller is told, rather than left to time out: a handler that threw is a
|
||||
// different failure from a tool nobody serves, and only one of them is worth retrying.
|
||||
reply = { error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
if (node) reply.node = node;
|
||||
msg.respond(sc.encode(JSON.stringify(reply)));
|
||||
}
|
||||
} catch (err) {
|
||||
console.log(`[mesh-tools] the bus refused ${subject}: ${err instanceof Error ? err.message : String(err)}; ` +
|
||||
"not served here, and the rest serves on");
|
||||
}
|
||||
})();
|
||||
return () => sub.unsubscribe();
|
||||
};
|
||||
|
||||
/**
|
||||
* Ask one question and await one answer, with the machine that gave it.
|
||||
*
|
||||
* Core NATS request/reply, not JetStream: a tool call must never be persisted (design 25 §3),
|
||||
* and a lost one is a timeout the caller already handles. The reply travels on the inbox the
|
||||
* request carries, which the responder may answer because its account has `allow_responses`
|
||||
* — one reply to a message it actually received, and nothing wider.
|
||||
*/
|
||||
const ask = async <Req, Res>(key: string, body: Req, on?: string): Promise<Answered<Res>> => {
|
||||
const msg = await conn.request(on ?? reachedAt(key), sc.encode(JSON.stringify(body)), {
|
||||
timeout: REQUEST_TIMEOUT_MS,
|
||||
});
|
||||
const reply = JSON.parse(sc.decode(msg.data)) as { result?: Res; error?: string; node?: string };
|
||||
if (reply.error) throw new Error(reply.error);
|
||||
return { result: reply.result as Res, node: reply.node };
|
||||
};
|
||||
|
||||
return {
|
||||
async request<Req, Res>(key: string, body: Req): Promise<Res> {
|
||||
return (await ask<Req, Res>(key, body)).result;
|
||||
},
|
||||
|
||||
ask,
|
||||
|
||||
/**
|
||||
* Answer a question, two ways (novox/hq ADR 0159): on the module's subject in a queue group,
|
||||
* so several machines may serve one tool and exactly one of them answers each call; and on the
|
||||
* same subject with this machine as its last token, so a caller that names the machine reaches
|
||||
* this instance and no other. A runtime that does not know its machine serves only the first,
|
||||
* which is how it always behaved.
|
||||
*/
|
||||
async handle<Req, Res>(key: string, handler: (body: Req) => Promise<Res>): Promise<() => void> {
|
||||
// A seat's verb named outright is served on the seat's subject as given, for a holder that
|
||||
// knows its role without a membership; everything else is a served module's own tool, served
|
||||
// where the mesh issued that module (ADR 0160). A key naming a module this connection does
|
||||
// not follow is not served here at all: a module serves its own tools, and the node's
|
||||
// runtime those of the modules it was given (ADR 0175) — never a stranger's.
|
||||
if (key.startsWith("seat:")) {
|
||||
const stop = answerOn(toolSubject(key, self), undefined, handler);
|
||||
return () => stop();
|
||||
}
|
||||
const dot = key.indexOf(".");
|
||||
const module = dot < 0 ? self : key.slice(0, dot);
|
||||
if (!issued.has(module) && module !== self) {
|
||||
throw new Error(
|
||||
`${self} cannot serve ${key}: a module serves its own tools, and a runtime those of the ` +
|
||||
"modules it follows",
|
||||
);
|
||||
}
|
||||
const tool = dot < 0 ? key : key.slice(dot + 1);
|
||||
let stops = servedOn(module, tool).map((s) => answerOn(s.subject, s.queue, handler));
|
||||
// When that module's new membership arrives, serve where it now says and stop serving where
|
||||
// it no longer does.
|
||||
issuedHandlers.push((m) => {
|
||||
if (m.module !== module) return;
|
||||
stops.forEach((stop) => stop());
|
||||
stops = servedOn(module, tool).map((s) => answerOn(s.subject, s.queue, handler));
|
||||
});
|
||||
return () => stops.forEach((stop) => stop());
|
||||
},
|
||||
|
||||
module: self,
|
||||
node,
|
||||
servedOn,
|
||||
raw(subject: string, answer: (subject: string, data: Uint8Array) => Uint8Array | undefined): () => void {
|
||||
const sub = conn.subscribe(subject);
|
||||
subs.push(sub);
|
||||
void (async () => {
|
||||
// **A refusal here is said, never fatal** (novox/hq 04-ISSUES/217). This serves discovery —
|
||||
// what the runtime says about itself — not the work; a bus that refuses it costs the mesh
|
||||
// seeing this runtime, not the runtime's tools and handlers. Unhandled, the refusal ended the
|
||||
// process and every per-module container crash-looped on 2026-10-03.
|
||||
try {
|
||||
for await (const msg of sub) {
|
||||
const body = answer(msg.subject, msg.data);
|
||||
if (body) msg.respond(body);
|
||||
}
|
||||
} catch (err) {
|
||||
console.log(`[mesh-tools] the bus refused ${subject}: ${err instanceof Error ? err.message : String(err)}; ` +
|
||||
"discovery will not see this runtime there, and it serves on");
|
||||
}
|
||||
})();
|
||||
return () => sub.unsubscribe();
|
||||
},
|
||||
follow,
|
||||
serving: () => [...issued.keys()],
|
||||
membership: (module?: string) => issued.get(module ?? self),
|
||||
onMembership: (handler: (m: Membership) => void) => {
|
||||
issuedHandlers.push(handler);
|
||||
},
|
||||
|
||||
async handleSubject<Req, Res>(subject: string, handler: (body: Req) => Promise<Res>): Promise<() => void> {
|
||||
return answerOn(subject, undefined, handler);
|
||||
},
|
||||
|
||||
/**
|
||||
* Emit an event.
|
||||
*
|
||||
* Published into JetStream and awaited, so a publish the bus never accepted fails the emit
|
||||
* rather than vanishing — at-least-once starts at the emitter, not only the consumer
|
||||
* (ADR 0042).
|
||||
*
|
||||
* `msgID` is the event's own id, so a redelivery after a crash between publishing and
|
||||
* acknowledging is de-duplicated by the server inside its window rather than seen twice.
|
||||
*/
|
||||
async publish<T>(env: Envelope<T>): Promise<void> {
|
||||
// **The body is the payload and the metadata rides as headers** (ADR 0042). That shape
|
||||
// is what the conformance suite pins: an implementation that nested the whole envelope in
|
||||
// the body would pass every one of its own tests and agree with nobody.
|
||||
const meta = (env.headers ?? {}) as Record<string, string>;
|
||||
const h = natsHeaders();
|
||||
for (const [k, v] of Object.entries(meta)) {
|
||||
if (v != null) h.set(k, String(v));
|
||||
}
|
||||
if (!meta["content-type"]) h.set("content-type", "application/json");
|
||||
if (env.node) h.set("x-node", env.node);
|
||||
|
||||
// The emitting module's subject: the tool at work's when a served module's tool emits from
|
||||
// the node's runtime (ADR 0175), this connection's own otherwise.
|
||||
await js.publish(eventSubject(env.key, atWork.getStore()?.module ?? self), sc.encode(JSON.stringify(env.body)), {
|
||||
headers: h,
|
||||
// De-duplicated by the server inside its window, so a redelivery after a crash between
|
||||
// publishing and acknowledging is not seen twice. Only the emitter can make this id.
|
||||
msgID: meta["x-event-id"],
|
||||
});
|
||||
},
|
||||
|
||||
/**
|
||||
* React to events.
|
||||
*
|
||||
* The durable consumer is the **controller's** to create, from what this module declared it
|
||||
* consumes (design 29 §3) — this binds to it and never creates one. A runtime that created
|
||||
* its own would be a module deciding its own delivery semantics, and its account cannot
|
||||
* reach the JetStream API to do it anyway.
|
||||
*
|
||||
* **One consumer, one loop, however many patterns a module registers.** A module has exactly one
|
||||
* durable consumer, so two loops reading it would each take half the messages — and a loop that
|
||||
* received one its own pattern does not match acknowledges it, which is the right answer for a
|
||||
* filter wider than anything registered and silent loss when it is another handler's. Every
|
||||
* registration is therefore dispatched from one reader, and a message is acknowledged once every
|
||||
* handler it is for has taken it.
|
||||
*/
|
||||
async subscribe<T>(
|
||||
pattern: string,
|
||||
handler: (env: Envelope<T>) => Promise<void>,
|
||||
): Promise<() => void> {
|
||||
const listener = { pattern, handler: handler as (env: Envelope<unknown>) => Promise<void> };
|
||||
listeners.push(listener);
|
||||
if (!reading) {
|
||||
const durable = `${cred.node ?? "?"}_${self}`;
|
||||
const consumer = await js.consumers.get("EVENTS", durable);
|
||||
const messages = await consumer.consume();
|
||||
reading = messages;
|
||||
void (async () => {
|
||||
for await (const msg of messages) {
|
||||
await deliver(msg, listeners);
|
||||
}
|
||||
})();
|
||||
}
|
||||
return () => {
|
||||
const at = listeners.indexOf(listener);
|
||||
if (at >= 0) listeners.splice(at, 1);
|
||||
if (listeners.length === 0 && reading) {
|
||||
void reading.close();
|
||||
reading = undefined;
|
||||
}
|
||||
};
|
||||
},
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (closed) return;
|
||||
closed = true;
|
||||
for (const sub of subs) sub.unsubscribe();
|
||||
// Drain rather than close: an in-flight reply is finished instead of dropped, which for a
|
||||
// tool call is the difference between an answer and an unexplained timeout at the caller.
|
||||
await conn.drain();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Deliver one event to every handler it is for, acknowledging only once each has taken it.
|
||||
*
|
||||
* Several registrations share one durable consumer, so matching happens here rather than by having
|
||||
* each registration read the stream: two readers of one consumer would split it between them, and a
|
||||
* message that reached the wrong one would be acknowledged as not-for-me and lost.
|
||||
*/
|
||||
async function deliver(
|
||||
msg: JsMsg,
|
||||
listeners: { pattern: string; handler: (env: Envelope<unknown>) => Promise<void> }[],
|
||||
): Promise<void> {
|
||||
let env: Envelope<unknown>;
|
||||
try {
|
||||
env = toEnvelope<unknown>(msg);
|
||||
} catch {
|
||||
// Unparseable: terminate it. Redelivering a message no version of this code can read is
|
||||
// an infinite loop, and what the controller keeps of a message given up on (DEAD_LETTERS,
|
||||
// novox/hq issue 330) is for handlers that fail, not for bytes that were never an envelope.
|
||||
msg.term();
|
||||
return;
|
||||
}
|
||||
const forThis = listeners.filter((l) => topicMatches(l.pattern, env.key));
|
||||
if (forThis.length === 0) {
|
||||
// The consumer's filters are the controller's, derived from what the module declared it
|
||||
// consumes, and may be wider than anything it registered a handler for. Acknowledge it, or it
|
||||
// would be redelivered until it expired.
|
||||
msg.ack();
|
||||
return;
|
||||
}
|
||||
try {
|
||||
for (const l of forThis) await l.handler(env);
|
||||
msg.ack();
|
||||
} catch {
|
||||
// Negative-acknowledge with a delay, so a handler failing on a transient cause gets another
|
||||
// attempt, and one failing permanently exhausts max-deliver rather than spinning: the consumer
|
||||
// gives the event up, and the controller keeps it in DEAD_LETTERS until a person delivers it
|
||||
// again or drops it (novox/hq issue 330). Delivered again, it arrives under
|
||||
// `mesh.again.<consumer>.…`, and keyFromSubject reads the same key from it. The consumer's
|
||||
// limits are the controller's; this only says "not done".
|
||||
msg.nak(5_000);
|
||||
}
|
||||
}
|
||||
|
||||
/** Rebuild the envelope a module sees, from the subject, the headers and the payload — the
|
||||
* mirror of publish, and the reason both live beside each other. */
|
||||
function toEnvelope<T>(msg: JsMsg): Envelope<T> {
|
||||
const headers: Record<string, string> = {};
|
||||
if (msg.headers) {
|
||||
for (const k of msg.headers.keys()) headers[k] = msg.headers.get(k);
|
||||
}
|
||||
return {
|
||||
// The event's own key, recovered from the subject: `mesh.mod.<module>.event.<key>`. The
|
||||
// module never sees the subject, only the key it declared.
|
||||
key: keyFromSubject(msg.subject),
|
||||
node: headers["x-node"] ?? "",
|
||||
body: JSON.parse(sc.decode(msg.data)) as T,
|
||||
headers: headers as EventHeaders,
|
||||
};
|
||||
}
|
||||
|
||||
/** The event key a module sees: the emitter and the event, which is exactly how its manifest names
|
||||
* what it consumes (novox/hq design 29 §1, 04-ISSUES/127).
|
||||
*
|
||||
* **One vocabulary for the declaration and the handler.** This returned the event name alone, so a
|
||||
* manifest declaring `consumes: builder.built` produced a handler pattern that could never match
|
||||
* the key it was compared against — and a module consuming the same event from two emitters could
|
||||
* not tell them apart except by reading a header. The subject already carries the emitter; naming it
|
||||
* here makes a mismatch between manifest and code a typo rather than a category error. */
|
||||
function keyFromSubject(subject: string): string {
|
||||
const marker = ".event.";
|
||||
const at = subject.indexOf(marker);
|
||||
if (at < 0) return subject;
|
||||
const event = subject.slice(at + marker.length);
|
||||
// `mesh.mod.<emitter>.event.…` — the emitter is the token before the marker.
|
||||
const before = subject.slice(0, at).split(".");
|
||||
const emitter = before[before.length - 1];
|
||||
return emitter ? `${emitter}.${event}` : event;
|
||||
}
|
||||
|
||||
/** A module's own event subject. Derived, never taken from the caller: the module names its
|
||||
* event and the mesh decides where it lands (design 29 §1). */
|
||||
function eventSubject(type: string, self: string): string {
|
||||
return `mesh.mod.${self}.event.${type}`;
|
||||
}
|
||||
|
||||
/** A tool's subject. A bare name is this module's own tool; `<module>.<tool>` addresses
|
||||
* another's, which is how a request reaches a module that is not this one; `seat:<seat>.<verb>`
|
||||
* addresses a role's tool, answered by whoever holds the seat (novox/hq ADR 0132) — with
|
||||
* `seat:<seat>.<verb>@<node>` for a node-scoped seat, whose tool carries the machine (design 33 §4). */
|
||||
function toolSubject(key: string, self: string): string {
|
||||
if (key.startsWith("seat:")) {
|
||||
const rest = key.slice("seat:".length);
|
||||
const dot = rest.indexOf(".");
|
||||
if (dot < 0) throw new Error(`"${key}" names a seat and no verb: seat:<seat>.<verb>`);
|
||||
const seat = rest.slice(0, dot);
|
||||
const [verb, node] = rest.slice(dot + 1).split("@", 2);
|
||||
return node ? `mesh.seat.${seat}.tool.${verb}.${node}` : `mesh.seat.${seat}.tool.${verb}`;
|
||||
}
|
||||
// `<module>.<tool>@<node>` names the machine (novox/hq ADR 0159): the same subject with the
|
||||
// machine as its last token, which is what that instance serves beside the queue.
|
||||
const [name, node] = key.split("@", 2);
|
||||
const dot = name.indexOf(".");
|
||||
const base = dot < 0
|
||||
? `mesh.mod.${self}.tool.${name}`
|
||||
: `mesh.mod.${name.slice(0, dot)}.tool.${name.slice(dot + 1)}`;
|
||||
return node ? `${base}.${node}` : base;
|
||||
}
|
||||
|
||||
/** A seat's verb, as its holder serves it: flat for a mesh seat, carrying the machine for a
|
||||
* node-scoped one (design 33 §4) — the same shape the controller grants. */
|
||||
export function seatToolSubject(seat: string, verb: string, scope: string | undefined, node: string | undefined): string {
|
||||
const base = `mesh.seat.${seat}.tool.${verb}`;
|
||||
return scope === "node" && node ? `${base}.${node}` : base;
|
||||
}
|
||||
|
||||
function normalizeFingerprint(fingerprint: string): string {
|
||||
return fingerprint.replace(/^sha256:/i, "").replace(/:/g, "").toLowerCase();
|
||||
}
|
||||
|
||||
/**
|
||||
* Dial once to see the certificate, and refuse unless it is exactly the one the mesh pinned.
|
||||
* A certificate authority is not consulted: the mesh issued this and knows its fingerprint,
|
||||
* which is stronger than trusting whoever a machine's trust store happens to contain.
|
||||
*
|
||||
* **The pin is the only check.** What comes back is handed to the client as its TLS options, and
|
||||
* the client's transport spreads them into Node's own `tls.connect` — so the pinned certificate
|
||||
* is the one authority the handshake accepts, and the hostname check beside it is replaced with
|
||||
* one that always passes. Pinning the exact certificate makes verifying its name redundant, and
|
||||
* the bus's certificate names the seat (`mesh-broker`), not the address a machine happens to
|
||||
* dial it by: every module on the mesh met "does not match certificate's altnames" the first time
|
||||
* it reached the handshake (2026-09-28).
|
||||
*/
|
||||
async function pinnedTls(rawUrl: string, fingerprint: string): Promise<TlsOptions> {
|
||||
const url = new URL(rawUrl.includes("://") ? rawUrl : `nats://${rawUrl}`);
|
||||
const port = url.port ? Number(url.port) : 4222;
|
||||
// **The bus speaks first, in the clear.** A NATS server sends its INFO line before TLS begins,
|
||||
// and only then expects the client to start the handshake; a raw TLS connect to that port reads
|
||||
// the INFO line as a TLS record and fails with "wrong version number" — which is what every
|
||||
// module met the first time it dialled the bus being built (2026-09-28). So: connect, wait for
|
||||
// INFO, then start TLS on the same socket, and read the certificate the server presents.
|
||||
const certificate = await new Promise<tls.DetailedPeerCertificate>((resolve, reject) => {
|
||||
const plain = net.connect({ host: url.hostname, port }, () => {});
|
||||
let seenInfo = false;
|
||||
let buffered = "";
|
||||
plain.on("error", reject);
|
||||
plain.on("data", (chunk: Buffer) => {
|
||||
if (seenInfo) return;
|
||||
buffered += chunk.toString("utf8");
|
||||
if (!buffered.includes("\r\n")) return;
|
||||
seenInfo = true;
|
||||
plain.removeAllListeners("data");
|
||||
const secure = tls.connect(
|
||||
{ socket: plain, rejectUnauthorized: false, servername: url.hostname },
|
||||
() => {
|
||||
const peer = secure.getPeerCertificate(true);
|
||||
secure.end();
|
||||
resolve(peer);
|
||||
},
|
||||
);
|
||||
secure.on("error", reject);
|
||||
});
|
||||
});
|
||||
const seen = createHash("sha256").update(certificate.raw).digest("hex");
|
||||
if (seen !== normalizeFingerprint(fingerprint)) {
|
||||
throw new PinMismatchError(
|
||||
`the bus at ${url.hostname}:${port} presented ${seen}, not the pinned ${normalizeFingerprint(fingerprint)}`,
|
||||
);
|
||||
}
|
||||
const pem = `-----BEGIN CERTIFICATE-----\n${certificate.raw.toString("base64").replace(/(.{64})/g, "$1\n")}\n-----END CERTIFICATE-----\n`;
|
||||
// Node's option, not the client's: the transport passes the whole object on. `undefined` from
|
||||
// checkServerIdentity is "the name is fine"; the pin above already decided the rest.
|
||||
return { ca: pem, checkServerIdentity: () => undefined } as TlsOptions;
|
||||
}
|
||||
|
||||
/** The mesh's topic matching: `*` is one token, `#` the rest. This is the module's vocabulary —
|
||||
* a module's `consumes` pattern is matched here, and the subject it becomes is the mesh's
|
||||
* business, not the module's. */
|
||||
export function topicMatches(pattern: string, key: string): boolean {
|
||||
return matchFrom(pattern.split("."), 0, key.split("."), 0);
|
||||
}
|
||||
|
||||
function matchFrom(p: string[], pi: number, k: string[], ki: number): boolean {
|
||||
if (pi === p.length) return ki === k.length;
|
||||
// `**` is the mesh's wildcard for the rest of a name; `#` is the old bus's, accepted so a pattern
|
||||
// written either way behaves the same while both buses ship (novox/hq design 29 §1).
|
||||
if (p[pi] === "#" || p[pi] === "**") {
|
||||
for (let skip = ki; skip <= k.length; skip++) {
|
||||
if (matchFrom(p, pi + 1, k, skip)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
if (ki === k.length) return false;
|
||||
if (p[pi] !== "*" && p[pi] !== k[ki]) return false;
|
||||
return matchFrom(p, pi + 1, k, ki + 1);
|
||||
}
|
||||
@@ -1,280 +0,0 @@
|
||||
/**
|
||||
* The mesh's tools, for whoever is on a machine (novox/hq design 25 §7, design 34).
|
||||
*
|
||||
* Two surfaces over one thing. A command line, for somebody at a terminal; an MCP server, for an
|
||||
* agent. Both are adapters over the same three calls — what tools are there, what does this one take,
|
||||
* call it — because a second way of reaching a tool is a second thing to keep correct.
|
||||
*
|
||||
* **It uses the same client a module's runtime uses.** Not a second protocol and not a bridge: the
|
||||
* caller connects as its own bus user — a person's, or the console's — publishes on the tool subjects
|
||||
* that account permits, and the server refuses anything else. So "what may this ask" is answered by the
|
||||
* same permission list that answers it for a module, and there is nothing here for an audit to read
|
||||
* separately.
|
||||
*
|
||||
* What the caller may NOT do is the more interesting half, and none of it is enforced here — it is the
|
||||
* account (design 25 §4): it cannot publish an event, so it cannot claim a module said something; it
|
||||
* has no consumer, so there is no delivery to acknowledge; and it cannot answer a request, so it cannot
|
||||
* impersonate a module on a bus where anyone may serve a tool.
|
||||
*/
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import type { Broker } from "@novox/mesh-sdk/messaging";
|
||||
|
||||
import { connectNats, type Answered, type Credential } from "./broker-nats.js";
|
||||
import { TOOLS_VERB, type ToolsAnswer } from "./runtime.js";
|
||||
|
||||
/** Where the catalogue answers which modules the mesh holds. */
|
||||
const CATALOGUE_MODULES = "mesh-catalog.catalog_modules";
|
||||
|
||||
/** Where the mesh answers every role's tools, from its records: the mesh-controller seat's own
|
||||
* `tools` verb (novox/hq ADR 0154, design 33 §5). */
|
||||
const SEAT_TOOLS = "seat:mesh-controller.tools";
|
||||
|
||||
/** A tool as its module describes it. */
|
||||
export interface Tool {
|
||||
/** The module that serves it — or, for a role's tool, the seat. */
|
||||
module: string;
|
||||
name: string;
|
||||
description?: string;
|
||||
/** The JSON schema of what it takes, as the module declared it. */
|
||||
input?: unknown;
|
||||
/** True for a role's tool: addressed to the seat, answered by whoever holds it (ADR 0132). */
|
||||
seat?: boolean;
|
||||
/** A role's scope: `node` for a seat held once per machine, whose verb is asked of one machine
|
||||
* (design 33 §4) and takes `node` for it; `mesh` or absent otherwise. */
|
||||
scope?: string;
|
||||
/** Where the tool is answered, as the mesh issued it (ADR 0160): the plain subject first when the
|
||||
* module answers for itself anywhere, then one per machine. Absent for a runtime older than this. */
|
||||
subjects?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* What the mesh could say about its tools when asked (design 34 §3).
|
||||
*
|
||||
* **Silence is named, never dropped.** A module the catalogue holds and nothing answered for is in
|
||||
* `notAnswering`, because a tool that is not offered looks exactly like a tool that does not exist,
|
||||
* and those need different people to fix them.
|
||||
*/
|
||||
export interface Listing {
|
||||
tools: Tool[];
|
||||
/** Modules the catalogue holds whose runtime did not answer `tools`: not assigned, not up, or built
|
||||
* before the runtime answered it. Each may still be called by name. The mesh's own records are
|
||||
* listed here as `mesh-controller (seat)` when the control plane did not answer. */
|
||||
notAnswering: string[];
|
||||
}
|
||||
|
||||
/** The seats and the verbs each declares, from the last listing, so a call can tell a role's tool
|
||||
* from a module's when the two share a prefix (a module and a seat may share a name). */
|
||||
export type Seats = Map<string, Set<string>>;
|
||||
|
||||
/** The roles' tools, keyed the way `toolKey` names them. */
|
||||
export function seatsIn(have: Listing): Seats {
|
||||
const seats: Seats = new Map();
|
||||
for (const t of have.tools) {
|
||||
if (!t.seat) continue;
|
||||
if (!seats.has(t.module)) seats.set(t.module, new Set());
|
||||
seats.get(t.module)!.add(t.name);
|
||||
}
|
||||
return seats;
|
||||
}
|
||||
|
||||
/** The key a call uses for `<prefix>.<name>`: a role's when the prefix is a seat declaring that
|
||||
* verb, a module's otherwise. Both names for one capability are deliberate and bounded (ADR 0132);
|
||||
* the seat wins only for a verb it actually declares, so a module's own tool is never shadowed. */
|
||||
export function toolKey(name: string, seats?: Seats): string {
|
||||
if (name.startsWith("seat:")) return name;
|
||||
const dot = name.indexOf(".");
|
||||
if (dot < 0) return name;
|
||||
const prefix = name.slice(0, dot);
|
||||
const verb = name.slice(dot + 1);
|
||||
if (seats?.get(prefix)?.has(verb)) return `seat:${prefix}.${verb}`;
|
||||
return name;
|
||||
}
|
||||
|
||||
/**
|
||||
* A person's credential, as `operator issue` prints it.
|
||||
*
|
||||
* The same shape a module is handed, minus the parts a module needs and a person does not: no node,
|
||||
* because a person is not on a machine, and no module, because they are not one.
|
||||
*/
|
||||
export interface PersonCredential extends Credential {
|
||||
person?: string;
|
||||
invokes?: string[];
|
||||
}
|
||||
|
||||
/** Read the credential from the file `operator issue` produced. */
|
||||
export async function credentialFrom(path: string): Promise<PersonCredential> {
|
||||
const raw = await readFile(path, "utf8");
|
||||
let held: PersonCredential;
|
||||
try {
|
||||
held = JSON.parse(raw) as PersonCredential;
|
||||
} catch (e) {
|
||||
throw new Error(
|
||||
`${path} is not a credential this mesh issued: ${(e as Error).message}. ` +
|
||||
"It is the JSON `operator issue` printed, saved verbatim.",
|
||||
);
|
||||
}
|
||||
if (!held.url || !held.user || !held.password) {
|
||||
throw new Error(
|
||||
`${path} names no bus, user or password. It is the JSON \`operator issue\` printed, saved ` +
|
||||
"verbatim — not an edited copy of it.",
|
||||
);
|
||||
}
|
||||
return held;
|
||||
}
|
||||
|
||||
/** Connect as this person. The module name the runtime wants is their own user, because every subject
|
||||
* it derives is for a tool somebody else serves. */
|
||||
export async function connectAs(held: PersonCredential): Promise<Broker> {
|
||||
return connectNats({ ...held, module: held.user });
|
||||
}
|
||||
|
||||
/**
|
||||
* Connect as the console: the module credential the mesh delivered (novox/hq ADR 0152), read from
|
||||
* the same variable every runtime reads. It names the node and the module, so the account's inbox
|
||||
* and subjects derive from what the mesh authorised and from nothing in this process's environment.
|
||||
*/
|
||||
export async function connectAsTheConsole(path: string): Promise<{ bus: Broker; who: string }> {
|
||||
const raw = await readFile(path, "utf8");
|
||||
let held: Credential;
|
||||
try {
|
||||
held = JSON.parse(raw) as Credential;
|
||||
} catch (e) {
|
||||
throw new Error(`${path} is not a broker credential: ${(e as Error).message}`);
|
||||
}
|
||||
if (!held.url || !held.module || !held.user) {
|
||||
throw new Error(
|
||||
`${path} names no bus, module or user: the console runs on the credential the mesh sealed to ` +
|
||||
"this machine for it, and nothing else",
|
||||
);
|
||||
}
|
||||
return { bus: await connectNats(held), who: `${held.node ?? "?"}.${held.module}` };
|
||||
}
|
||||
|
||||
/**
|
||||
* What tools the mesh has, asked of the modules (design 34 §3).
|
||||
*
|
||||
* The catalogue says which modules the mesh holds; each module says what it serves, through the one
|
||||
* verb its runtime answers for it. **Asked, not configured**: a client carrying its own list would be a
|
||||
* list that goes stale the first time a module is assigned. Every module is asked at once, and the bus
|
||||
* refuses at once a request nothing serves, so the cost is bounded by the modules that are up.
|
||||
*/
|
||||
export async function toolsOn(bus: Broker): Promise<Listing> {
|
||||
const [answered, roles] = await Promise.all([
|
||||
bus.request<Record<string, never>, { modules?: { module: string }[] }>(CATALOGUE_MODULES, {}),
|
||||
// The roles' tools, from the mesh's records (design 33 §5). Asked beside the modules rather
|
||||
// than first: a control plane that is restarting must not hide every module's tools with it.
|
||||
bus
|
||||
.request<Record<string, never>, { seats?: { seat: string; scope?: string; tools?: ToolsAnswer["tools"] }[] }>(
|
||||
SEAT_TOOLS,
|
||||
{},
|
||||
)
|
||||
.catch(() => undefined),
|
||||
]);
|
||||
const names = (answered.modules ?? []).map((m) => m.module).filter((m) => typeof m === "string");
|
||||
|
||||
const asked = await Promise.allSettled(
|
||||
names.map((module) => bus.request<Record<string, never>, ToolsAnswer>(`${module}.${TOOLS_VERB}`, {})),
|
||||
);
|
||||
const tools: Tool[] = [];
|
||||
const notAnswering: string[] = [];
|
||||
if (roles) {
|
||||
for (const s of roles.seats ?? []) {
|
||||
// A node-scoped seat's tool is asked of one machine (design 33 §4): listed with its scope, so
|
||||
// a caller names the machine and the call carries it — `seat:<seat>.<verb>@<node>`. Left out
|
||||
// of the listing, the verb never resolved as a seat's and nothing served it (ADR 0170).
|
||||
for (const t of s.tools ?? []) {
|
||||
tools.push({ module: s.seat, name: t.name, description: t.description, input: t.input, seat: true, scope: s.scope });
|
||||
}
|
||||
}
|
||||
} else {
|
||||
notAnswering.push("mesh-controller (seat)");
|
||||
}
|
||||
asked.forEach((outcome, i) => {
|
||||
const module = names[i]!;
|
||||
if (outcome.status === "fulfilled" && typeof outcome.value?.failed === "string") {
|
||||
// The runtime answered for it and serves nothing: the bundle failed to load (ADR 0175). Said
|
||||
// with the reason, because "not answering" would send somebody to check an assignment that
|
||||
// is fine.
|
||||
notAnswering.push(`${module} (its tools bundle failed to load: ${outcome.value.failed})`);
|
||||
} else if (outcome.status === "fulfilled" && Array.isArray(outcome.value?.tools)) {
|
||||
for (const t of outcome.value.tools) {
|
||||
tools.push({ module, name: t.name, description: t.description, input: t.input, subjects: t.subjects });
|
||||
}
|
||||
} else {
|
||||
notAnswering.push(module);
|
||||
}
|
||||
});
|
||||
tools.sort((a, b) => `${a.module}.${a.name}`.localeCompare(`${b.module}.${b.name}`));
|
||||
notAnswering.sort();
|
||||
return { tools, notAnswering };
|
||||
}
|
||||
|
||||
/** Call one tool. The key is `<module>.<tool>`, which is what a person types and what the account
|
||||
* permits — one vocabulary, so a refusal names the thing they asked for. */
|
||||
/** Call a tool and learn which machine answered (novox/hq ADR 0159). `<module>.<tool>@<node>` asks
|
||||
* the instance on one machine; without it, whichever instance answers first does, and the answer
|
||||
* says which. */
|
||||
export async function callTool(
|
||||
bus: Broker,
|
||||
key: string,
|
||||
args: unknown,
|
||||
seats?: Seats,
|
||||
listing?: Listing,
|
||||
): Promise<Answered<unknown>> {
|
||||
const at = key.indexOf("@");
|
||||
const name = at < 0 ? key : key.slice(0, at);
|
||||
const node = at < 0 ? "" : key.slice(at + 1);
|
||||
if (!name.includes(".")) {
|
||||
throw new Error(
|
||||
`"${key}" does not name a tool: write <module>.<tool>, as \`mesh tools\` lists them, ` +
|
||||
"or <module>.<tool>@<node> for the instance on one machine",
|
||||
);
|
||||
}
|
||||
const resolved = toolKey(name, seats) + (node ? `@${node}` : "");
|
||||
// Where the tool is answered is the module's to say and the mesh's to issue (ADR 0160): when the
|
||||
// listing carried subjects for it, the call goes to one of those and composes nothing.
|
||||
const on = subjectListed(name, node, listing);
|
||||
const asking = bus as Broker & { ask?: <Req, Res>(k: string, b: Req, on?: string) => Promise<Answered<Res>> };
|
||||
if (typeof asking.ask === "function") return asking.ask<unknown, unknown>(resolved, args ?? {}, on);
|
||||
return { result: await bus.request<unknown, unknown>(resolved, args ?? {}) };
|
||||
}
|
||||
|
||||
/** The subject the listing says answers `<module>.<tool>` — the machine's when one is named, else
|
||||
* the plain one — or undefined when the listing said none, and the key is composed as before. */
|
||||
export function subjectListed(name: string, node: string, listing?: Listing): string | undefined {
|
||||
if (!listing) return undefined;
|
||||
const dot = name.indexOf(".");
|
||||
const module = name.slice(0, dot);
|
||||
const tool = name.slice(dot + 1);
|
||||
const found = listing.tools.find((t) => t.module === module && t.name === tool && !t.seat);
|
||||
const subjects = found?.subjects ?? [];
|
||||
if (subjects.length === 0) return undefined;
|
||||
if (node) return subjects.find((s) => s.endsWith(`.${node}`));
|
||||
return subjects[0];
|
||||
}
|
||||
|
||||
/**
|
||||
* Why a call failed, said so that the remedy is in the words.
|
||||
*
|
||||
* Three answers a person actually gets, and they need different things done: nobody serves that tool,
|
||||
* the mesh refused this account, or the tool itself failed. Without this they are one timeout and a
|
||||
* stack trace.
|
||||
*/
|
||||
export function whyItFailed(key: string, err: unknown): string {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
if (/no responders|503/i.test(message)) {
|
||||
return `nothing serves ${key}. The module may not be assigned to any machine, or it is down` +
|
||||
(key.startsWith("seat:") ? ", or nothing holds that seat" : "") +
|
||||
" — `mesh tools` lists what answered.";
|
||||
}
|
||||
if (/permissions violation|authorization/i.test(message)) {
|
||||
return `this account may not call ${key}. What it may call was fixed when it was issued — a ` +
|
||||
"person's by `operator issue`, the console's by its manifest.";
|
||||
}
|
||||
if (/timeout/i.test(message)) {
|
||||
return `${key} did not answer in time. Something is serving it, so this is the tool being slow ` +
|
||||
"rather than absent.";
|
||||
}
|
||||
return `${key} failed: ${message}`;
|
||||
}
|
||||
@@ -1,165 +0,0 @@
|
||||
/**
|
||||
* The console's endpoint: MCP over HTTP, on a machine's loopback (novox/hq ADR 0152, design 34 §2).
|
||||
*
|
||||
* **Loopback is the authority boundary.** Whoever can connect is on the machine, and whoever is on the
|
||||
* machine is the account that owns the mesh there (ADR 0034, ADR 0144). So there is no token and no
|
||||
* login here, and the one thing this file enforces is that it binds nothing else: a console reachable
|
||||
* from another machine would be authority over the mesh handed to whoever finds the port.
|
||||
*
|
||||
* The transport is the streamable-HTTP shape an agent host speaks: `POST /mcp` with one JSON-RPC
|
||||
* message, answered with one JSON body. No session, because the surface holds nothing per caller; no
|
||||
* event stream, because nothing here has anything to say unasked.
|
||||
*/
|
||||
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
|
||||
|
||||
import type { Broker } from "@novox/mesh-sdk/messaging";
|
||||
|
||||
import { mcpSurface, type Reply, type Request } from "./mcp.js";
|
||||
|
||||
/** The most a request body may be. A tool's arguments are small; a megabyte is somebody else's file. */
|
||||
const BODY_LIMIT = 1 << 20;
|
||||
|
||||
export interface Listening {
|
||||
/** Where it listens, as `host:port`, with the port the machine actually gave. */
|
||||
address: string;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
/** Hosts that are this machine and no other. */
|
||||
const loopback = new Set(["127.0.0.1", "::1", "localhost", "[::1]"]);
|
||||
|
||||
/**
|
||||
* Listen on `host:port`. Refused unless the host is loopback — said before binding, so a manifest or
|
||||
* a flag that would open the console to a network is a startup failure rather than something
|
||||
* discovered by whoever finds it.
|
||||
*/
|
||||
export async function serveMcpHttp(bus: Broker, who: string, listen: string): Promise<Listening> {
|
||||
const at = listen.lastIndexOf(":");
|
||||
if (at < 0) {
|
||||
throw new Error(`"${listen}" is not host:port`);
|
||||
}
|
||||
const host = listen.slice(0, at);
|
||||
const port = Number(listen.slice(at + 1));
|
||||
if (!loopback.has(host)) {
|
||||
throw new Error(
|
||||
`the console listens on loopback and nowhere else (novox/hq ADR 0152): "${host}" is not this ` +
|
||||
"machine's own address — whoever is on the machine owns the mesh there, and nobody else may reach this",
|
||||
);
|
||||
}
|
||||
if (!Number.isInteger(port) || port < 0 || port > 65535) {
|
||||
throw new Error(`"${listen.slice(at + 1)}" is not a port`);
|
||||
}
|
||||
|
||||
const surface = mcpSurface(bus, who);
|
||||
const server = createServer((req, res) => {
|
||||
void route(req, res, surface.handle).catch((e) => {
|
||||
json(res, 500, { jsonrpc: "2.0", id: null, error: { code: -32603, message: String(e) } });
|
||||
});
|
||||
});
|
||||
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.once("error", reject);
|
||||
server.listen(port, host.replace(/^\[|\]$/g, ""), () => resolve());
|
||||
});
|
||||
const bound = server.address();
|
||||
const address = typeof bound === "object" && bound ? `${host}:${bound.port}` : listen;
|
||||
return {
|
||||
address,
|
||||
close: () =>
|
||||
new Promise<void>((resolve) => {
|
||||
server.close(() => resolve());
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
async function route(
|
||||
req: IncomingMessage,
|
||||
res: ServerResponse,
|
||||
handle: (r: Request) => Promise<Reply | undefined>,
|
||||
): Promise<void> {
|
||||
const path = (req.url ?? "/").split("?")[0];
|
||||
if (path === "/") {
|
||||
res.writeHead(200, { "content-type": "text/plain; charset=utf-8" });
|
||||
res.end("the mesh's console: MCP over HTTP at POST /mcp (novox/hq design 34)\n");
|
||||
return;
|
||||
}
|
||||
if (path !== "/mcp") {
|
||||
json(res, 404, { error: "the console serves /mcp and nothing else" });
|
||||
return;
|
||||
}
|
||||
switch (req.method) {
|
||||
case "POST":
|
||||
break;
|
||||
case "DELETE":
|
||||
// A host ending a session. There is no session to end; saying so is the truthful answer.
|
||||
res.writeHead(204).end();
|
||||
return;
|
||||
case "GET":
|
||||
// A host opening an event stream. The console has nothing to say unasked.
|
||||
res.writeHead(405, { allow: "POST, DELETE" }).end();
|
||||
return;
|
||||
default:
|
||||
res.writeHead(405, { allow: "POST, DELETE" }).end();
|
||||
return;
|
||||
}
|
||||
|
||||
let body: string;
|
||||
try {
|
||||
body = await read(req);
|
||||
} catch (e) {
|
||||
json(res, 413, { jsonrpc: "2.0", id: null, error: { code: -32600, message: String(e) } });
|
||||
return;
|
||||
}
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(body);
|
||||
} catch {
|
||||
json(res, 400, { jsonrpc: "2.0", id: null, error: { code: -32700, message: "the body is not JSON" } });
|
||||
return;
|
||||
}
|
||||
|
||||
// One message, or a batch of them; a batch is answered as a batch. A notification gets no reply
|
||||
// and, alone, no body: 202 is how the transport says "heard".
|
||||
if (Array.isArray(parsed)) {
|
||||
const replies = (await Promise.all(parsed.map((r) => handle(r as Request)))).filter(Boolean);
|
||||
if (replies.length === 0) {
|
||||
res.writeHead(202).end();
|
||||
} else {
|
||||
json(res, 200, replies);
|
||||
}
|
||||
return;
|
||||
}
|
||||
const reply = await handle(parsed as Request);
|
||||
if (!reply) {
|
||||
res.writeHead(202).end();
|
||||
return;
|
||||
}
|
||||
json(res, 200, reply);
|
||||
}
|
||||
|
||||
function read(req: IncomingMessage): Promise<string> {
|
||||
return new Promise((resolve, reject) => {
|
||||
let size = 0;
|
||||
const chunks: Buffer[] = [];
|
||||
req.on("data", (chunk: Buffer) => {
|
||||
size += chunk.length;
|
||||
if (size > BODY_LIMIT) {
|
||||
reject(new Error(`the request is larger than ${BODY_LIMIT} bytes`));
|
||||
req.destroy();
|
||||
return;
|
||||
}
|
||||
chunks.push(chunk);
|
||||
});
|
||||
req.on("end", () => resolve(Buffer.concat(chunks).toString("utf8")));
|
||||
req.on("error", reject);
|
||||
});
|
||||
}
|
||||
|
||||
function json(res: ServerResponse, status: number, body: unknown): void {
|
||||
const text = JSON.stringify(body);
|
||||
res.writeHead(status, {
|
||||
"content-type": "application/json; charset=utf-8",
|
||||
"content-length": Buffer.byteLength(text),
|
||||
});
|
||||
res.end(text);
|
||||
}
|
||||
@@ -1,192 +0,0 @@
|
||||
// A tools bundle as a process the runtime launches (novox/hq ADR 0188).
|
||||
//
|
||||
// The runtime does not run a tool's code itself when the bundle is not JavaScript: it starts the
|
||||
// bundle's executable as a child with the runtime's environment and speaks MCP over stdio to it —
|
||||
// `initialize`, `tools/list` once, `tools/call` per call. A Rust binary, a Go binary, a Python
|
||||
// script and a Node script are the same thing from here: a process that answers those. Everything
|
||||
// the mesh adds — the subjects from the membership, the held seats, the `tools` answer, a bundle
|
||||
// that failed named and the others serving — is the runtime's, outside this file.
|
||||
//
|
||||
// A tool the child lists as `<seat>.<verb>` is the module's implementation of that seat's verb;
|
||||
// any other name is the module's own tool. The same rule the in-process registration follows.
|
||||
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { accessSync, constants } from "node:fs";
|
||||
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
|
||||
import { broker, type Envelope } from "@novox/mesh-sdk/messaging";
|
||||
import { atWork } from "./broker-nats.js";
|
||||
|
||||
/** The protocol version this speaks; a bundle says the same. */
|
||||
export const PROTOCOL = "2025-03-26";
|
||||
|
||||
/** How long a child has to answer `initialize` and `tools/list` before it is a failed bundle, and
|
||||
* how long a call may take before the caller is told the tool is slow rather than absent. */
|
||||
const HANDSHAKE_MS = 10_000;
|
||||
const CALL_MS = 30_000;
|
||||
|
||||
/** Whether an entrypoint is launched as a process rather than imported: anything that is not a
|
||||
* plain JavaScript file, and a JavaScript file marked executable — a bundle written against the
|
||||
* protocol in TypeScript, served the same way as any other language. */
|
||||
export function launches(entry: string): boolean {
|
||||
const javascript = /\.(m|c)?js$/.test(entry);
|
||||
let executable = false;
|
||||
try {
|
||||
accessSync(entry, constants.X_OK);
|
||||
executable = true;
|
||||
} catch {
|
||||
// not executable, or not there — importing will say which
|
||||
}
|
||||
return !javascript || executable;
|
||||
}
|
||||
|
||||
/** What a launched bundle registers: the groups the in-process path would have, by name. */
|
||||
export interface Launched {
|
||||
registrations: { module: string; tools: ToolDefinition[] }[];
|
||||
stop(): void;
|
||||
}
|
||||
|
||||
interface Pending {
|
||||
resolve(v: any): void;
|
||||
reject(e: Error): void;
|
||||
timer: NodeJS.Timeout;
|
||||
}
|
||||
|
||||
/**
|
||||
* Launch a bundle and learn its tools. Rejects when the child cannot be started or does not complete
|
||||
* the handshake, which the runtime records as the bundle having failed. A child that exits later is
|
||||
* started again on the next call, once; a call in flight when it died is told so.
|
||||
*/
|
||||
export async function launch(module: string, entry: string, env: NodeJS.ProcessEnv = process.env): Promise<Launched> {
|
||||
let child: ChildProcess | undefined;
|
||||
let nextId = 1;
|
||||
const pending = new Map<number, Pending>();
|
||||
let stopped = false;
|
||||
|
||||
const start = async (): Promise<void> => {
|
||||
const proc = spawn(entry, [], { stdio: ["pipe", "pipe", "pipe"], env });
|
||||
child = proc;
|
||||
let buffered = "";
|
||||
proc.stdout!.on("data", (chunk: Buffer) => {
|
||||
buffered += chunk.toString("utf8");
|
||||
let at: number;
|
||||
while ((at = buffered.indexOf("\n")) >= 0) {
|
||||
const line = buffered.slice(0, at).trim();
|
||||
buffered = buffered.slice(at + 1);
|
||||
if (!line) continue;
|
||||
let reply: { id?: number | string; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
|
||||
try {
|
||||
reply = JSON.parse(line);
|
||||
} catch {
|
||||
console.log(`[mesh-tools] ${module}'s bundle said something that is not a reply: ${line.slice(0, 120)}`);
|
||||
continue;
|
||||
}
|
||||
// **The bundle asks the runtime to emit** (novox/hq ADR 0193): published on the bus as this
|
||||
// module, and answered once the bus has accepted it, so the tool's emit means what it means
|
||||
// in-process. Nothing else a bundle may ask.
|
||||
if (typeof reply.method === "string") {
|
||||
const id = reply.id;
|
||||
const answer = (m: Record<string, unknown>) => proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", id, ...m }) + "\n");
|
||||
if (reply.method !== "mesh/publish") {
|
||||
if (id !== undefined) answer({ error: { code: -32601, message: `the runtime answers no ${reply.method} from a bundle` } });
|
||||
continue;
|
||||
}
|
||||
atWork.run({ module }, () => broker().publish(reply.params as Envelope<unknown>))
|
||||
.then(() => { if (id !== undefined) answer({ result: {} }); })
|
||||
.catch((err: unknown) => { if (id !== undefined) answer({ error: { code: -32000, message: err instanceof Error ? err.message : String(err) } }); });
|
||||
continue;
|
||||
}
|
||||
const waiting = typeof reply.id === "number" ? pending.get(reply.id) : undefined;
|
||||
if (!waiting) continue;
|
||||
pending.delete(reply.id as number);
|
||||
clearTimeout(waiting.timer);
|
||||
if (reply.error) waiting.reject(new Error(reply.error.message ?? "the bundle refused the request"));
|
||||
else waiting.resolve(reply.result);
|
||||
}
|
||||
});
|
||||
// stderr is the bundle's log; kept under the module's name so a fault reads where it belongs.
|
||||
// The last thing it said is kept, so a bundle that dies says why in its own words, not by code.
|
||||
let lastSaid = "";
|
||||
proc.stderr!.on("data", (chunk: Buffer) => {
|
||||
for (const line of chunk.toString("utf8").split("\n")) {
|
||||
if (!line.trim()) continue;
|
||||
console.log(`[${module}] ${line}`);
|
||||
if (/\S/.test(line) && !/^\s+at\s/.test(line) && !/^Node\.js v/.test(line)) lastSaid = line.trim();
|
||||
}
|
||||
});
|
||||
const exited = new Promise<never>((_, reject) => {
|
||||
proc.once("error", (err) => reject(err));
|
||||
proc.once("exit", (code, signal) => {
|
||||
const why = `${module}'s bundle exited (${signal ?? code})` + (lastSaid ? `: ${lastSaid}` : "");
|
||||
for (const [id, p] of pending) {
|
||||
pending.delete(id);
|
||||
clearTimeout(p.timer);
|
||||
p.reject(new Error(why));
|
||||
}
|
||||
if (child === proc) child = undefined;
|
||||
if (!stopped) console.log(`[mesh-tools] ${why}; started again on its next call`);
|
||||
reject(new Error(why));
|
||||
});
|
||||
});
|
||||
const ask = (method: string, params: unknown, ms: number): Promise<any> =>
|
||||
Promise.race([
|
||||
new Promise<any>((resolve, reject) => {
|
||||
const id = nextId++;
|
||||
const timer = setTimeout(() => {
|
||||
pending.delete(id);
|
||||
reject(new Error(`${module}'s bundle did not answer ${method} in ${ms / 1000}s`));
|
||||
}, ms);
|
||||
pending.set(id, { resolve, reject, timer });
|
||||
proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n");
|
||||
}),
|
||||
exited,
|
||||
]);
|
||||
exited.catch(() => {}); // observed through the race; never unhandled
|
||||
(proc as ChildProcess & { ask?: typeof ask }).ask = ask;
|
||||
await ask("initialize", { protocolVersion: PROTOCOL, capabilities: {}, clientInfo: { name: "node-tools", version: "1" } }, HANDSHAKE_MS);
|
||||
proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");
|
||||
};
|
||||
|
||||
const asking = async (method: string, params: unknown, ms: number): Promise<any> => {
|
||||
if (!child) await start();
|
||||
return (child as ChildProcess & { ask: (m: string, p: unknown, ms: number) => Promise<any> }).ask(method, params, ms);
|
||||
};
|
||||
|
||||
await start();
|
||||
const listed = (await asking("tools/list", {}, HANDSHAKE_MS)) as { tools?: { name: string; description?: string; inputSchema?: unknown }[] };
|
||||
const groups = new Map<string, ToolDefinition[]>();
|
||||
for (const t of listed.tools ?? []) {
|
||||
const dot = t.name.indexOf(".");
|
||||
const under = dot < 0 ? module : t.name.slice(0, dot);
|
||||
const name = dot < 0 ? t.name : t.name.slice(dot + 1);
|
||||
const tools = groups.get(under) ?? [];
|
||||
tools.push({
|
||||
name,
|
||||
description: t.description ?? "",
|
||||
input: (t.inputSchema as Record<string, unknown> | undefined) ?? {},
|
||||
run: async (args) => {
|
||||
const result = (await asking("tools/call", { name: t.name, arguments: args ?? {} }, CALL_MS)) as {
|
||||
content?: { type: string; text?: string }[];
|
||||
isError?: boolean;
|
||||
};
|
||||
const text = result?.content?.find((c) => c.type === "text")?.text ?? "";
|
||||
if (result?.isError) throw new Error(text || `${module}.${t.name} failed`);
|
||||
// The bundle's answer is JSON as text (that is what every MCP host renders); handed back as
|
||||
// the value it encodes so a caller on the bus sees what an in-process tool would return.
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch {
|
||||
return text;
|
||||
}
|
||||
},
|
||||
});
|
||||
groups.set(under, tools);
|
||||
}
|
||||
return {
|
||||
registrations: [...groups].map(([under, tools]) => ({ module: under, tools })),
|
||||
stop: () => {
|
||||
stopped = true;
|
||||
child?.kill("SIGTERM");
|
||||
child = undefined;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -1,309 +0,0 @@
|
||||
// The runnable entrypoint. Three modes:
|
||||
//
|
||||
// mesh-tools serve — bind the broker and serve the assigned modules until
|
||||
// stopped; as the node-tools module, also the console on loopback. A module entrypoint that subscribes to events (on("#"))
|
||||
// starts consuming as it is imported, so this also runs consumers.
|
||||
// mesh-tools emit TYPE [JSON] emit one event onto the mesh and exit — an operable primitive,
|
||||
// and what an events test uses to put a message on the wire.
|
||||
// mesh-tools prepare bring this module's state to the shape this version needs and exit
|
||||
// — the runtime's answer to the word the mesh asks every module
|
||||
// (novox/hq ADR 0135). The entrypoints come from MESH_PREPARE, which
|
||||
// the module's own image names beside MESH_TOOL_MODULES.
|
||||
// mesh-tools run ENTRYPOINT run one compiled module entrypoint to completion and exit — the
|
||||
// runtime side of a run-once step (novox/hq ADR 0052). It imports
|
||||
// the given entrypoint, whose top-level code does its work — seed a
|
||||
// store, migrate, health-gate — and awaits it. It does NOT connect
|
||||
// to the broker: a first-boot step runs offline, before the module
|
||||
// has anything to talk to, and the host gates the container that
|
||||
// depends on it on this process exiting 0.
|
||||
//
|
||||
// The broker, in order of preference:
|
||||
// MESH_BROKER_FILE a sealed {url, fingerprint} the mesh delivered (novox/hq ADR 0043) — an
|
||||
// amqps account scoped to this module. Preferred: a module holds its own.
|
||||
// MESH_BROKER_URL a plain URL, for the bootstrap/admin case before a module has an account.
|
||||
// MESH_TOOL_MODULES <module>=/path/a,… the modules to serve and their compiled entrypoints (serve
|
||||
// mode; ADR 0175); a bare path is an entrypoint of the credential's own module
|
||||
// MESH_PREPARE /path/a,/path/b,… compiled entrypoints that prepare this module's state
|
||||
// MESH_MODULE / MESH_NODE the identity stamped onto emitted events (ADR 0042)
|
||||
|
||||
import { readFileSync } from "node:fs";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { connectNats, fatalBrokerReason as fatalNatsReason, type Credential } from "./broker-nats.js";
|
||||
import { takeToolEnvs, runTools, type ServedModule } from "./runtime.js";
|
||||
import { serveMcpHttp, type Listening } from "./http.js";
|
||||
|
||||
/** The credential this process connected with, for what it says beyond the connection (ADR 0159). */
|
||||
let lastCredential: Credential | undefined;
|
||||
import { invokeTool } from "@novox/mesh-sdk/tools";
|
||||
import { useBroker } from "@novox/mesh-sdk/messaging";
|
||||
import type { Broker } from "@novox/mesh-sdk/messaging";
|
||||
import { emit } from "@novox/mesh-sdk/events";
|
||||
|
||||
/**
|
||||
* Connect the way this process is meant to: with its sealed credential if the mesh gave it one, and
|
||||
* over the plain bootstrap URL otherwise. A scoped module assumes the foundation's exchanges exist —
|
||||
* its account may not declare them (ADR 0043).
|
||||
*/
|
||||
// fatalBrokerReasonFor is the reason a connection failure is final rather than "not yet", for
|
||||
// whichever bus this runtime is on — each transport knows its own refusals.
|
||||
function fatalBrokerReasonFor(err: unknown): string | null {
|
||||
return fatalNatsReason(err);
|
||||
}
|
||||
|
||||
async function connectBroker(): Promise<Broker> {
|
||||
const file = process.env.MESH_BROKER_FILE;
|
||||
if (file) {
|
||||
let credential: Credential;
|
||||
try {
|
||||
credential = JSON.parse(readFileSync(file, "utf8")) as Credential;
|
||||
lastCredential = credential;
|
||||
} catch (err) {
|
||||
console.error(`mesh-tools: cannot read the broker credential at ${file}: ${err}`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (!credential.url) {
|
||||
console.error(`mesh-tools: ${file} carries no url — it is not a broker credential`);
|
||||
process.exit(1);
|
||||
}
|
||||
// The mesh scoped this account to a node and module; take the runtime's identity from the
|
||||
// credential so its queue and the events it emits match what the mesh authorised, no matter
|
||||
// what the environment says.
|
||||
if (credential.node) process.env.MESH_NODE = credential.node;
|
||||
if (credential.module) process.env.MESH_MODULE = credential.module;
|
||||
// **The credential names the bus.** A module moved to the bus being built was handed a
|
||||
// credential for it — `nats://…` with user, password and fingerprint beside the address — and
|
||||
// nothing else in its environment changed (design 25; novox/hq design 28 task 5.2). The scheme
|
||||
// is enough to know which bus to speak; a runtime that always dialled the old one would keep
|
||||
// serving and answer nobody.
|
||||
// One bus (novox/hq ADR 0131, design 28 task 5.5): the credential names it, and it is this.
|
||||
return connectNats(credential);
|
||||
}
|
||||
const url = process.env.MESH_BROKER_URL;
|
||||
if (!url) {
|
||||
console.error(
|
||||
"mesh-tools: set MESH_BROKER_FILE (a sealed credential) or MESH_BROKER_URL — there is no broker to reach",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
return connectNats({ url });
|
||||
}
|
||||
|
||||
/**
|
||||
* Connect for serve mode, retrying while the broker is merely not reachable yet. At startup that
|
||||
* is the NORMAL case, not a failure: a container comes up in seconds and the overlay tunnel a
|
||||
* moment later (novox/hq issue 058). Exiting instead delegated the retry to the container
|
||||
* runtime, which read as a crash-loop to every restart-counting health check and every person
|
||||
* watching. Retried indefinitely, aloud: the dependency appears or somebody reads why not.
|
||||
*
|
||||
* A failure that waiting cannot fix (see fatalBrokerReason) is thrown at once rather than retried —
|
||||
* a permanent fault masquerading as "not reachable yet" is the silent non-progress this whole
|
||||
* change exists to remove. Missing-file and empty-URL configuration errors exit inside
|
||||
* connectBroker before they reach here; a malformed URL and a refused login are caught here.
|
||||
*/
|
||||
async function connectBrokerPatiently(): Promise<Broker> {
|
||||
for (let delay = 2_000; ; delay = Math.min(delay * 2, 30_000)) {
|
||||
try {
|
||||
return await connectBroker();
|
||||
} catch (err) {
|
||||
const fatal = fatalBrokerReasonFor(err);
|
||||
if (fatal !== null) {
|
||||
console.error(`mesh-tools: ${fatal} — waiting will not fix this; giving up`);
|
||||
throw err;
|
||||
}
|
||||
const why = err instanceof Error ? err.message : String(err);
|
||||
// A little jitter so every module that was up when the broker bounced does not retry in
|
||||
// lockstep and stampede it as it recovers.
|
||||
const wait = delay + Math.floor(Math.random() * 1_000);
|
||||
console.error(`mesh-tools: the broker is not reachable yet (${why}); retrying in ${Math.round(wait / 1000)}s`);
|
||||
await new Promise((r) => setTimeout(r, wait));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* What MESH_TOOL_MODULES names (novox/hq ADR 0175, to-be 38 WP1): `<module>=<entrypoint>` entries,
|
||||
* comma-separated, several per module allowed — the node's runtime serving every assigned module's
|
||||
* bundle. A bare path is the one-module form the per-module containers still set: an entrypoint of
|
||||
* the credential's own module. Both may appear; the result is one list of modules.
|
||||
*/
|
||||
export function servedModulesFrom(spec: string, own: string | undefined): { serves: ServedModule[]; moduleEntrypoints: string[] } {
|
||||
const serves = new Map<string, string[]>();
|
||||
const moduleEntrypoints: string[] = [];
|
||||
for (const raw of spec.split(",")) {
|
||||
const entry = raw.trim();
|
||||
if (!entry) continue;
|
||||
const eq = entry.indexOf("=");
|
||||
if (eq < 0) {
|
||||
moduleEntrypoints.push(entry);
|
||||
continue;
|
||||
}
|
||||
const module = entry.slice(0, eq).trim();
|
||||
const path = entry.slice(eq + 1).trim();
|
||||
if (!module || !path) {
|
||||
throw new Error(`MESH_TOOL_MODULES: "${entry}" is neither <module>=<entrypoint> nor an entrypoint of this module`);
|
||||
}
|
||||
if (module === own) {
|
||||
moduleEntrypoints.push(path);
|
||||
continue;
|
||||
}
|
||||
serves.set(module, [...(serves.get(module) ?? []), path]);
|
||||
}
|
||||
return { serves: [...serves].map(([module, entrypoints]) => ({ module, entrypoints })), moduleEntrypoints };
|
||||
}
|
||||
|
||||
/** The module that is the node's tool runtime (novox/hq ADR 0175, to-be 38 WP3): on its credential,
|
||||
* `serve` is also the console — MCP on the machine's loopback (design 34). */
|
||||
export const RUNTIME_MODULE = "node-tools";
|
||||
|
||||
/** Where the console listens when the runtime is node-tools and nothing says otherwise: the port
|
||||
* the module's manifest declares `from: machine`. MESH_CONSOLE_LISTEN overrides it either way. */
|
||||
const CONSOLE_LISTEN = "127.0.0.1:4270";
|
||||
|
||||
async function serve(): Promise<void> {
|
||||
const broker = await connectBrokerPatiently();
|
||||
// Parsed after connecting: a bare entrypoint belongs to the module the credential names.
|
||||
const { serves, moduleEntrypoints } = servedModulesFrom(process.env.MESH_TOOL_MODULES ?? "", lastCredential?.module);
|
||||
// Each module's environment, composed by the mesh (ADR 0192): taken before any bundle is imported.
|
||||
const envs = takeToolEnvs();
|
||||
const stop = await runTools({ broker, serves, moduleEntrypoints, credential: lastCredential, envs });
|
||||
|
||||
// The console is this runtime's serving mode (ADR 0175 §6): as node-tools, or wherever the
|
||||
// listen address is given, the same process answers MCP on loopback for whoever is on the
|
||||
// machine. A module's own runtime in a container on the machine's network does not — two of
|
||||
// them on one port would be the fault, and the console is one per machine.
|
||||
const listen = process.env.MESH_CONSOLE_LISTEN ?? (lastCredential?.module === RUNTIME_MODULE ? CONSOLE_LISTEN : "");
|
||||
let consoleUp: Listening | undefined;
|
||||
if (listen) {
|
||||
const who = `${lastCredential?.node ?? "?"}.${lastCredential?.module ?? RUNTIME_MODULE}`;
|
||||
consoleUp = await serveMcpHttp(broker, who, listen);
|
||||
console.log(`mesh console listening on http://${consoleUp.address}/mcp as ${who}`);
|
||||
}
|
||||
|
||||
const shutdown = async (): Promise<void> => {
|
||||
stop();
|
||||
await consoleUp?.close();
|
||||
await broker.close();
|
||||
process.exit(0);
|
||||
};
|
||||
process.on("SIGTERM", () => void shutdown());
|
||||
process.on("SIGINT", () => void shutdown());
|
||||
}
|
||||
|
||||
async function emitOnce(type: string, bodyJson: string): Promise<void> {
|
||||
let body: unknown = {};
|
||||
if (bodyJson) {
|
||||
try {
|
||||
body = JSON.parse(bodyJson);
|
||||
} catch {
|
||||
console.error(`mesh-tools emit: body is not JSON: ${bodyJson}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
const broker = await connectBroker();
|
||||
useBroker(() => broker);
|
||||
// emit awaits the broker's publish confirm (ADR 0042), so the event is accepted before we close.
|
||||
await emit(type, body);
|
||||
await broker.close();
|
||||
}
|
||||
|
||||
async function invokeOnce(module: string, tool: string, argsJson: string): Promise<void> {
|
||||
let args: Record<string, unknown> = {};
|
||||
if (argsJson) {
|
||||
try {
|
||||
args = JSON.parse(argsJson) as Record<string, unknown>;
|
||||
} catch {
|
||||
console.error(`mesh-tools invoke: args are not JSON: ${argsJson}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
const broker = await connectBroker();
|
||||
const result = await invokeTool(broker, module, tool, args);
|
||||
process.stdout.write(JSON.stringify(result) + "\n");
|
||||
await broker.close();
|
||||
}
|
||||
|
||||
/**
|
||||
* Run one compiled module entrypoint to completion — the runtime side of a run-once step
|
||||
* (novox/hq ADR 0052). Importing it runs its top-level code and awaits any top-level await, so this
|
||||
* returns only once the step's own code has finished; a step that throws rejects here and the
|
||||
* process exits non-zero, which is how the host knows the step did not complete and must not start
|
||||
* the container it gates. No broker is connected — a first-boot seed or migration runs offline.
|
||||
*/
|
||||
async function runEntry(entrypoint: string): Promise<void> {
|
||||
if (!entrypoint) {
|
||||
console.error("mesh-tools run <entrypoint> — a compiled module entrypoint path is required");
|
||||
process.exit(1);
|
||||
}
|
||||
// A file URL, not a bare path: dynamic import of an absolute path is not portable, and the
|
||||
// entrypoint the manifest names is an absolute path inside the image.
|
||||
await import(pathToFileURL(entrypoint).href);
|
||||
}
|
||||
|
||||
/**
|
||||
* Bring this module's state to the shape this version needs, and exit — the runtime's answer to the
|
||||
* one word the mesh asks every module (novox/hq ADR 0135).
|
||||
*
|
||||
* The entrypoints come from `MESH_PREPARE`, which a module's own image sets beside the entrypoints it
|
||||
* already lists there: the module knows which of its files prepares its state, and nothing else could.
|
||||
* Each is imported in the order given, to completion, with no broker — preparation runs before the
|
||||
* version that would use it, so there is nothing yet to talk to.
|
||||
*
|
||||
* **An empty list is a failure, not a no-op.** The mesh only asks this of a module whose manifest says
|
||||
* it prepares something; a module that says so and names nothing has been built wrong, and exiting 0
|
||||
* would let that version serve against a state nobody shaped.
|
||||
*/
|
||||
async function prepareState(): Promise<void> {
|
||||
const named = (process.env.MESH_PREPARE ?? "")
|
||||
.split(",")
|
||||
.map((entry) => entry.trim())
|
||||
.filter((entry) => entry !== "");
|
||||
if (named.length === 0) {
|
||||
console.error(
|
||||
"mesh-tools prepare: this module was asked to prepare its state and its image names nothing " +
|
||||
"to do it with — set MESH_PREPARE to the compiled entrypoint(s) that prepare it, the way " +
|
||||
"MESH_TOOL_MODULES names the ones it serves",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
for (const entrypoint of named) {
|
||||
console.log(`[mesh-tools] preparing with ${entrypoint}`);
|
||||
await runEntry(entrypoint);
|
||||
}
|
||||
console.log(`[mesh-tools] prepared: ${named.length} entrypoint(s) ran to completion`);
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const [command, ...rest] = process.argv.slice(2);
|
||||
if (command === "run") {
|
||||
await runEntry(rest[0] ?? "");
|
||||
return;
|
||||
}
|
||||
if (command === "prepare") {
|
||||
await prepareState();
|
||||
return;
|
||||
}
|
||||
if (command === "invoke") {
|
||||
const [module, tool] = rest;
|
||||
if (!module || !tool) {
|
||||
console.error("mesh-tools invoke <module> <tool> [json-args] — a module and tool are required");
|
||||
process.exit(1);
|
||||
}
|
||||
await invokeOnce(module, tool, rest[2] ?? "");
|
||||
return;
|
||||
}
|
||||
if (command === "emit") {
|
||||
const type = rest[0];
|
||||
if (!type) {
|
||||
console.error("mesh-tools emit <type> [json-body] — a routing key is required");
|
||||
process.exit(1);
|
||||
}
|
||||
await emitOnce(type, rest[1] ?? "");
|
||||
return;
|
||||
}
|
||||
await serve();
|
||||
}
|
||||
|
||||
// Only when run, so a test can import the pieces.
|
||||
if (process.argv[1] && import.meta.url === new URL(`file://${process.argv[1]}`).href) {
|
||||
void main();
|
||||
}
|
||||
@@ -1,258 +0,0 @@
|
||||
/**
|
||||
* The mesh's tools as an MCP server (novox/hq design 25 §7, design 34).
|
||||
*
|
||||
* **A thin adapter and nothing more.** Every tool an agent sees is one a module answered for and one
|
||||
* this account may call; the schema is the module's own; the answer is the module's own. Nothing here
|
||||
* decides anything, which is why it is short — an MCP surface that reshaped arguments or summarised
|
||||
* answers would be a second definition of what a tool is, and the module's code is the first.
|
||||
*
|
||||
* Implemented against the protocol directly rather than through a library: the surface is three
|
||||
* methods and one framing, and a dependency here would be a dependency on every machine.
|
||||
*
|
||||
* One handler, two transports. Over stdio for a program a person starts (`mesh mcp`), over HTTP on a
|
||||
* machine's loopback for the console the mesh assigns there (`mesh serve`, http.ts). The handler does
|
||||
* not know which asked.
|
||||
*/
|
||||
import type { Broker } from "@novox/mesh-sdk/messaging";
|
||||
|
||||
import { callTool, seatsIn, toolKey, toolsOn, whyItFailed, type Listing, type Seats } from "./client.js";
|
||||
|
||||
/** The protocol version this speaks. Stated, because a host that wants another should be told so
|
||||
* rather than discovering it through a shape it did not expect. */
|
||||
export const PROTOCOL = "2025-03-26";
|
||||
|
||||
export interface Request {
|
||||
jsonrpc: string;
|
||||
id?: number | string | null;
|
||||
method: string;
|
||||
params?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface Reply {
|
||||
jsonrpc: "2.0";
|
||||
id: Request["id"];
|
||||
result?: unknown;
|
||||
error?: { code: number; message: string };
|
||||
}
|
||||
|
||||
/** How long a fetched tool list is kept before the modules are asked again. An agent asks on every
|
||||
* turn; the mesh changes on the order of minutes. */
|
||||
export const LISTING_KEPT_MS = 30_000;
|
||||
|
||||
export interface Surface {
|
||||
/** Answer one request; undefined for a notification, which expects none. */
|
||||
handle(request: Request): Promise<Reply | undefined>;
|
||||
}
|
||||
|
||||
/**
|
||||
* The surface over one bus connection, as one account.
|
||||
*
|
||||
* The tool list is fetched when first asked and kept for a short while (design 34 §3): asking every
|
||||
* module on every `tools/list` would fan out on every agent turn for something nobody changed, and
|
||||
* never refreshing would hide a module assigned a moment ago.
|
||||
*/
|
||||
export function mcpSurface(bus: Broker, who: string): Surface {
|
||||
let known: { listing: Listing; at: number } | undefined;
|
||||
|
||||
const answer = (id: Request["id"], result: unknown): Reply => ({ jsonrpc: "2.0", id, result });
|
||||
const refuse = (id: Request["id"], code: number, message: string): Reply => ({
|
||||
jsonrpc: "2.0",
|
||||
id,
|
||||
error: { code, message },
|
||||
});
|
||||
|
||||
const listing = async (): Promise<Listing> => {
|
||||
if (!known || Date.now() - known.at > LISTING_KEPT_MS) {
|
||||
known = { listing: await toolsOn(bus), at: Date.now() };
|
||||
}
|
||||
return known.listing;
|
||||
};
|
||||
// The roles the last listing knew, so `<seat>.<verb>` resolves to the seat. Fetched once if a
|
||||
// call arrives before any list did; a listing that failed leaves no roles, and the name is then
|
||||
// a module's, which is the right fallback for a mesh whose control plane is away.
|
||||
const roles = async (): Promise<Seats | undefined> => {
|
||||
try {
|
||||
return seatsIn(await listing());
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
return {
|
||||
async handle(request) {
|
||||
// A notification has no id and expects no answer; `initialized` is the one every host sends.
|
||||
const notification = request.id === undefined || request.id === null;
|
||||
|
||||
switch (request.method) {
|
||||
case "initialize":
|
||||
return answer(request.id, {
|
||||
protocolVersion: PROTOCOL,
|
||||
capabilities: { tools: {} },
|
||||
serverInfo: { name: "mesh", version: "1" },
|
||||
// Said in the handshake, because an agent that knows whose authority it is acting under
|
||||
// can say so when a call is refused — and a refusal is the one thing here that is not
|
||||
// the mesh's fault or the tool's.
|
||||
instructions:
|
||||
`These are the tools of a Novox mesh, reached as ${who}. Every call goes to the module ` +
|
||||
`that serves it; what may be called was fixed when this account was issued, so a ` +
|
||||
`refusal means the account, not the tool. The list is what the running modules ` +
|
||||
`answered, plus every role's tools from the mesh's records — the mesh's own verbs ` +
|
||||
`(mesh-controller.status, .push, .assign …) among them; a module that did not answer ` +
|
||||
`is named in the list's _meta and can still be called by <module>.<tool>.`,
|
||||
});
|
||||
|
||||
case "notifications/initialized":
|
||||
return undefined;
|
||||
|
||||
case "ping":
|
||||
return notification ? undefined : answer(request.id, {});
|
||||
|
||||
case "tools/list": {
|
||||
let have: Listing;
|
||||
try {
|
||||
have = await listing();
|
||||
} catch (e) {
|
||||
return refuse(request.id, -32603, whyItFailed("mesh-catalog.catalog_modules", e));
|
||||
}
|
||||
return answer(request.id, {
|
||||
tools: have.tools.map((t) => ({
|
||||
name: `${t.module}.${t.name}`,
|
||||
description: t.description ?? `${t.name}, served by ${t.module}`,
|
||||
// The module's own schema, passed through — with `node`, the machine to ask when
|
||||
// the module runs on several (novox/hq ADR 0159); a seat's verb takes none, the
|
||||
// seat's scope decides. An empty object is a tool that takes nothing, which is a
|
||||
// real answer and not a missing one.
|
||||
inputSchema: t.seat && t.scope !== "node" ? asSchema(t.input)
|
||||
: t.seat ? withNode(asSchema(t.input), "the machine whose seat answers; required, the seat is held once per machine", true)
|
||||
: withNode(asSchema(t.input)),
|
||||
})),
|
||||
// Silence, named (design 34 §3): the modules the catalogue holds and nothing answered
|
||||
// for. Not a tool, so not in `tools`; not dropped either.
|
||||
_meta: { notAnswering: have.notAnswering },
|
||||
});
|
||||
}
|
||||
|
||||
case "tools/call": {
|
||||
const given = String(request.params?.name ?? "");
|
||||
const args = { ...((request.params?.arguments as Record<string, unknown> | undefined) ?? {}) };
|
||||
// The machine, when the caller names one, travels in the subject and never reaches the
|
||||
// module's arguments (novox/hq ADR 0159) — for a module's tool. A seat's verb takes no
|
||||
// machine from the console (the seat's scope decides), so a `node` among its arguments
|
||||
// is the verb's own, as `push` and `assign` take one, and is handed through untouched.
|
||||
const have = await listing().catch(() => undefined);
|
||||
const roles = have && seatsIn(have);
|
||||
const bare = given.split("@", 1)[0];
|
||||
const isSeatVerb = roles ? toolKey(bare, roles).startsWith("seat:") : false;
|
||||
// A node-scoped seat's verb is asked of one machine (design 33 §4, ADR 0170): `node`
|
||||
// names it and travels in the subject, as for a module's tool.
|
||||
const nodeScoped = isSeatVerb && (have?.tools.some((t) => t.seat && t.scope === "node" &&
|
||||
`${t.module}.${t.name}` === bare) ?? false);
|
||||
const takesNode = !isSeatVerb || nodeScoped;
|
||||
const node = takesNode && typeof args.node === "string" && args.node !== "" ? args.node : "";
|
||||
if (takesNode) delete args.node;
|
||||
if (nodeScoped && !node && !given.includes("@")) {
|
||||
return refuse(request.id, -32602, `${given} is a machine's seat's verb: name the machine with \`node\``);
|
||||
}
|
||||
const name = node && !given.includes("@") ? `${given}@${node}` : given;
|
||||
try {
|
||||
const { result, node: answeredBy } = await callTool(bus, name, args, roles, have);
|
||||
// Text, because that is what every host renders. The content is the module's answer
|
||||
// as JSON, unshaped: an adapter that flattened it would be deciding what matters in
|
||||
// somebody else's answer. Which machine answered follows it as its own line.
|
||||
const content: { type: string; text: string }[] = [
|
||||
{ type: "text", text: JSON.stringify(result, null, 2) },
|
||||
];
|
||||
if (answeredBy) content.push({ type: "text", text: `answered by ${answeredBy}` });
|
||||
return answer(request.id, { content });
|
||||
} catch (e) {
|
||||
// **An error the agent can act on, not a stack.** isError rather than a protocol
|
||||
// failure, because the call was well-formed and the mesh answered it — with a refusal,
|
||||
// an absence or a fault, and the words say which.
|
||||
return answer(request.id, {
|
||||
content: [{ type: "text", text: whyItFailed(name, e) }],
|
||||
isError: true,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
default:
|
||||
return notification ? undefined : refuse(request.id, -32601, `mesh's MCP surface has no ${request.method}`);
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A module's declared input as a JSON schema an agent can read.
|
||||
*
|
||||
* The sdk keeps a tool's input opaque, and the catalogue's modules write it as a bare map of
|
||||
* property to description — `{ module: { type, description } }` — which is the `properties` of a
|
||||
* schema rather than a schema. Wrapped here when that is what arrived; passed through when a module
|
||||
* already wrote a schema; an empty object when it declared nothing. The module's words are kept
|
||||
* either way.
|
||||
*/
|
||||
export function asSchema(input: unknown): Record<string, unknown> {
|
||||
if (!input || typeof input !== "object" || Array.isArray(input)) {
|
||||
return { type: "object", properties: {} };
|
||||
}
|
||||
const given = input as Record<string, unknown>;
|
||||
if (given.type === "object" || "properties" in given) return given;
|
||||
if (Object.keys(given).length === 0) return { type: "object", properties: {} };
|
||||
return { type: "object", properties: given };
|
||||
}
|
||||
|
||||
/**
|
||||
* Serve over stdio until stdin closes, which is how a host ends a session.
|
||||
*/
|
||||
export async function serveMcp(bus: Broker, who: string): Promise<void> {
|
||||
const surface = mcpSurface(bus, who);
|
||||
const say = (message: unknown) => {
|
||||
process.stdout.write(`${JSON.stringify(message)}\n`);
|
||||
};
|
||||
for await (const line of lines()) {
|
||||
let request: Request;
|
||||
try {
|
||||
request = JSON.parse(line) as Request;
|
||||
} catch {
|
||||
// Unparseable, and with no id there is nobody to tell. Skipped rather than answered, because a
|
||||
// reply to a request that was never framed is noise on the same channel.
|
||||
continue;
|
||||
}
|
||||
const reply = await surface.handle(request);
|
||||
if (reply) say(reply);
|
||||
}
|
||||
}
|
||||
|
||||
/** stdin as newline-framed messages, which is what MCP over stdio is. */
|
||||
async function* lines(): AsyncGenerator<string> {
|
||||
let buffered = "";
|
||||
for await (const chunk of process.stdin) {
|
||||
buffered += (chunk as Buffer).toString("utf8");
|
||||
let at: number;
|
||||
while ((at = buffered.indexOf("\n")) >= 0) {
|
||||
const line = buffered.slice(0, at).trim();
|
||||
buffered = buffered.slice(at + 1);
|
||||
if (line !== "") yield line;
|
||||
}
|
||||
}
|
||||
if (buffered.trim() !== "") yield buffered.trim();
|
||||
}
|
||||
|
||||
/** Every module tool takes an optional `node`: the machine to ask when the module runs on several
|
||||
* (novox/hq ADR 0159). Added to the listing, stripped before the call, never seen by the module. */
|
||||
function withNode(schema: Record<string, unknown>, description?: string, required = false): Record<string, unknown> {
|
||||
const properties = { ...((schema.properties as Record<string, unknown> | undefined) ?? {}) };
|
||||
if (!("node" in properties)) {
|
||||
properties.node = {
|
||||
type: "string",
|
||||
description: description ??
|
||||
"the machine to ask, when this module runs on several; else whichever answers, and the answer says which",
|
||||
};
|
||||
}
|
||||
const out: Record<string, unknown> = { ...schema, type: "object", properties };
|
||||
if (required) {
|
||||
const have = Array.isArray(schema.required) ? (schema.required as string[]) : [];
|
||||
out.required = have.includes("node") ? have : [...have, "node"];
|
||||
}
|
||||
return out;
|
||||
}
|
||||
@@ -1,278 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* `mesh` — the mesh's tools, for whoever is on a machine (novox/hq design 25 §7, design 34).
|
||||
*
|
||||
* Four verbs and nothing else. What tools are there, call one, serve the same two to an agent over
|
||||
* stdio, and serve them on a machine's loopback as the console the mesh assigns. Deliberately thin:
|
||||
* everything that could be a decision is one the mesh already made, and a client that grew opinions
|
||||
* would be a second place the mesh's behaviour is defined.
|
||||
*
|
||||
* mesh tools what the running modules answer
|
||||
* mesh call <module>.<tool> [json] call one, arguments as JSON on the command line or on stdin
|
||||
* mesh mcp the same two, as an MCP server over stdio
|
||||
* mesh serve [--listen host:port] the console: MCP over HTTP on this machine's loopback
|
||||
*
|
||||
* Who it speaks as, in order of preference:
|
||||
* MESH_BROKER_FILE the module credential the mesh delivered — the console's (ADR 0152)
|
||||
* MESH_CREDENTIAL / --credential <file> a person's, as `operator issue` printed it (design 25 §7)
|
||||
* MESH_CONSOLE / --console <url> no credential: `tools` and `call` go through a console
|
||||
* already running on this machine, over loopback HTTP
|
||||
*/
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import type { Broker } from "@novox/mesh-sdk/messaging";
|
||||
|
||||
import {
|
||||
callTool,
|
||||
connectAs,
|
||||
connectAsTheConsole,
|
||||
credentialFrom,
|
||||
seatsIn,
|
||||
toolsOn,
|
||||
whyItFailed,
|
||||
type Listing,
|
||||
} from "./client.js";
|
||||
import { serveMcpHttp } from "./http.js";
|
||||
import { serveMcp } from "./mcp.js";
|
||||
|
||||
const usage = `mesh tools
|
||||
mesh call <module>.<tool> [json]
|
||||
mesh mcp
|
||||
mesh serve [--listen host:port]
|
||||
|
||||
--credential <file> a person's credential, the JSON \`operator issue\` printed; default $MESH_CREDENTIAL
|
||||
--console <url> a console on this machine to ask through instead; default $MESH_CONSOLE
|
||||
--listen <host:port> where \`serve\` listens; loopback only; default $MESH_CONSOLE_LISTEN or 127.0.0.1:4270
|
||||
MESH_BROKER_FILE the module credential the mesh delivered, which \`serve\` runs on`;
|
||||
|
||||
/** What the console listens on when nothing says otherwise. */
|
||||
const DEFAULT_LISTEN = "127.0.0.1:4270";
|
||||
|
||||
async function main(argv: string[]): Promise<number> {
|
||||
const args = [...argv];
|
||||
let credentialPath = process.env.MESH_CREDENTIAL ?? "";
|
||||
let consoleUrl = process.env.MESH_CONSOLE ?? "";
|
||||
let listen = process.env.MESH_CONSOLE_LISTEN ?? DEFAULT_LISTEN;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const take = () => {
|
||||
const v = args[i + 1] ?? "";
|
||||
args.splice(i, 2);
|
||||
i--;
|
||||
return v;
|
||||
};
|
||||
if (args[i] === "--credential") credentialPath = take();
|
||||
else if (args[i] === "--console") consoleUrl = take();
|
||||
else if (args[i] === "--listen") listen = take();
|
||||
}
|
||||
const verb = args.shift();
|
||||
if (!verb || verb === "help" || verb === "--help") {
|
||||
console.log(usage);
|
||||
return verb ? 0 : 1;
|
||||
}
|
||||
|
||||
// Through a console already on this machine: no credential to hold, which is the point of one.
|
||||
if (consoleUrl && (verb === "tools" || verb === "call")) {
|
||||
return verb === "tools" ? listingVia(consoleUrl) : callingVia(consoleUrl, args);
|
||||
}
|
||||
|
||||
const { bus, who } = await connecting(credentialPath, verb);
|
||||
try {
|
||||
switch (verb) {
|
||||
case "tools":
|
||||
return await listing(bus, who);
|
||||
case "call":
|
||||
return await calling(bus, args);
|
||||
case "mcp":
|
||||
// Serves until stdin closes, which is how an MCP host ends a session.
|
||||
await serveMcp(bus, who);
|
||||
return 0;
|
||||
case "serve": {
|
||||
const up = await serveMcpHttp(bus, who, listen);
|
||||
console.log(`mesh console listening on http://${up.address}/mcp as ${who}`);
|
||||
await new Promise<void>((resolve) => {
|
||||
process.once("SIGTERM", () => resolve());
|
||||
process.once("SIGINT", () => resolve());
|
||||
});
|
||||
await up.close();
|
||||
return 0;
|
||||
}
|
||||
default:
|
||||
console.error(`mesh has no "${verb}".\n\n${usage}`);
|
||||
return 1;
|
||||
}
|
||||
} finally {
|
||||
await bus.close();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Who this process is on the bus. The module credential first: a console is started by the mesh with
|
||||
* MESH_BROKER_FILE and nothing else, and must not fall back to a person's file lying around.
|
||||
*/
|
||||
async function connecting(credentialPath: string, verb: string): Promise<{ bus: Broker; who: string }> {
|
||||
const delivered = process.env.MESH_BROKER_FILE;
|
||||
if (delivered) {
|
||||
return connectAsTheConsole(delivered);
|
||||
}
|
||||
if (!credentialPath) {
|
||||
throw new Error(
|
||||
verb === "serve"
|
||||
? "no credential: the console runs on MESH_BROKER_FILE, the module credential the mesh " +
|
||||
"delivered; to run it by hand, pass --credential <file> with a person's credential"
|
||||
: "no credential: set MESH_CREDENTIAL or pass --credential <file> (the JSON `operator issue` " +
|
||||
"printed, saved verbatim), or --console <url> to ask through a console on this machine",
|
||||
);
|
||||
}
|
||||
const held = await credentialFrom(credentialPath);
|
||||
return { bus: await connectAs(held), who: held.person ?? held.user ?? "somebody" };
|
||||
}
|
||||
|
||||
function printListing(have: Listing, who?: string): void {
|
||||
if (have.tools.length === 0) {
|
||||
console.log("no running module answered with any tool");
|
||||
}
|
||||
// **What the modules answered, not what this account may call.** The two differ and the
|
||||
// difference is the point: an account seeing only its own tools cannot tell "not installed" from
|
||||
// "not yours", and those need different people to fix them.
|
||||
for (const t of have.tools) {
|
||||
const name = `${t.module}.${t.name}`;
|
||||
const line = t.description ? `${name.padEnd(36)} ${t.description}` : name;
|
||||
console.log(t.seat ? `${line} (a role's tool: answered by whoever holds the ${t.module} seat)` : line);
|
||||
}
|
||||
if (have.notAnswering.length > 0) {
|
||||
console.log(
|
||||
`\nheld by the mesh and not answering: ${have.notAnswering.join(", ")} — not assigned, not up, ` +
|
||||
"or built before the runtime answered `tools`; each can still be called by name",
|
||||
);
|
||||
}
|
||||
if (who) {
|
||||
console.log(`\nasked as ${who}; what ${who} may call was fixed when the account was issued.`);
|
||||
}
|
||||
}
|
||||
|
||||
async function listing(bus: Broker, who?: string): Promise<number> {
|
||||
let have: Listing;
|
||||
try {
|
||||
have = await toolsOn(bus);
|
||||
} catch (e) {
|
||||
console.error(whyItFailed("mesh-catalog.catalog_modules", e));
|
||||
return 1;
|
||||
}
|
||||
printListing(have, who);
|
||||
return 0;
|
||||
}
|
||||
|
||||
async function calling(bus: Broker, args: string[]): Promise<number> {
|
||||
const key = args.shift();
|
||||
if (!key) {
|
||||
console.error("mesh call <module>.<tool> [json]");
|
||||
return 1;
|
||||
}
|
||||
const parsed = await argumentsFrom(args);
|
||||
if (parsed === undefined) return 1;
|
||||
try {
|
||||
// `<seat>.<verb>` reaches the role when the mesh lists that verb for the seat; `seat:` says so
|
||||
// outright and asks nothing first.
|
||||
const have = key.startsWith("seat:") ? undefined : await toolsOn(bus).catch(() => undefined);
|
||||
const { result, node } = await callTool(bus, key, parsed, have && seatsIn(have), have);
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
if (node) console.error(`answered by ${node}`);
|
||||
return 0;
|
||||
} catch (e) {
|
||||
console.error(whyItFailed(key, e));
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
/** The console's answer to one MCP request, over loopback HTTP. */
|
||||
async function viaConsole(consoleUrl: string, method: string, params?: unknown): Promise<any> {
|
||||
const endpoint = consoleUrl.endsWith("/mcp") ? consoleUrl : `${consoleUrl.replace(/\/$/, "")}/mcp`;
|
||||
const res = await fetch(endpoint, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json", accept: "application/json" },
|
||||
body: JSON.stringify({ jsonrpc: "2.0", id: 1, method, params }),
|
||||
});
|
||||
if (!res.ok) {
|
||||
throw new Error(`the console at ${endpoint} answered ${res.status}`);
|
||||
}
|
||||
const reply = (await res.json()) as { result?: any; error?: { message: string } };
|
||||
if (reply.error) throw new Error(reply.error.message);
|
||||
return reply.result;
|
||||
}
|
||||
|
||||
async function listingVia(consoleUrl: string): Promise<number> {
|
||||
try {
|
||||
const result = await viaConsole(consoleUrl, "tools/list");
|
||||
const have: Listing = {
|
||||
tools: (result.tools ?? []).map((t: { name: string; description?: string; inputSchema?: unknown }) => {
|
||||
const at = t.name.indexOf(".");
|
||||
return { module: t.name.slice(0, at), name: t.name.slice(at + 1), description: t.description, input: t.inputSchema };
|
||||
}),
|
||||
notAnswering: result._meta?.notAnswering ?? [],
|
||||
};
|
||||
printListing(have);
|
||||
return 0;
|
||||
} catch (e) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
async function callingVia(consoleUrl: string, args: string[]): Promise<number> {
|
||||
const key = args.shift();
|
||||
if (!key) {
|
||||
console.error("mesh call <module>.<tool> [json]");
|
||||
return 1;
|
||||
}
|
||||
const parsed = await argumentsFrom(args);
|
||||
if (parsed === undefined) return 1;
|
||||
try {
|
||||
const result = await viaConsole(consoleUrl, "tools/call", { name: key, arguments: parsed });
|
||||
const text = result?.content?.[0]?.text ?? JSON.stringify(result);
|
||||
if (result?.isError) {
|
||||
console.error(text);
|
||||
return 1;
|
||||
}
|
||||
console.log(text);
|
||||
return 0;
|
||||
} catch (e) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
/** A call's arguments: JSON on the command line, else on stdin, else nothing. Undefined when what
|
||||
* was given is not JSON, after saying so. */
|
||||
async function argumentsFrom(args: string[]): Promise<unknown> {
|
||||
const raw = args.length > 0 ? args.join(" ") : await maybeStdin();
|
||||
if (raw.trim() === "") return {};
|
||||
try {
|
||||
return JSON.parse(raw);
|
||||
} catch (e) {
|
||||
console.error(`the arguments are not JSON: ${(e as Error).message}`);
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/** Arguments on stdin, for a call whose JSON is too long or too quoted to type. Empty when stdin is a
|
||||
* terminal, so `mesh call x.y` with no arguments does not hang waiting for something nobody is
|
||||
* typing. */
|
||||
async function maybeStdin(): Promise<string> {
|
||||
if (process.stdin.isTTY) return "";
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of process.stdin) chunks.push(chunk as Buffer);
|
||||
return Buffer.concat(chunks).toString("utf8");
|
||||
}
|
||||
|
||||
// Only when run, so a test can import the pieces.
|
||||
if (process.argv[1] && import.meta.url === new URL(`file://${process.argv[1]}`).href) {
|
||||
main(process.argv.slice(2))
|
||||
.then((code) => process.exit(code))
|
||||
.catch((e) => {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
|
||||
export { main, usage };
|
||||
export const _readFile = readFile;
|
||||
@@ -1,384 +0,0 @@
|
||||
// The tool runtime — the per-node process that makes the mesh's tools actually serve (novox/hq
|
||||
// ADR 0175). It binds the mesh broker, loads the served modules' tool bundles (each of which calls
|
||||
// registerModuleTools as it imports), and serves every module's tools on that module's subjects and
|
||||
// every held seat's verbs on the seat's. Everything hard — dispatch, collection, duplicate-name
|
||||
// safety — is the sdk's; this is the wrapper.
|
||||
//
|
||||
// One runtime, many modules. It was written for one module per process and ran that way in a
|
||||
// container per module; it now serves a list, as the one process per node the host supervises,
|
||||
// and the per-module shape is the list with one entry. A bundle that fails to import is named —
|
||||
// in the log and in what `tools` answers for it — and the others serve.
|
||||
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { resolve } from "node:path";
|
||||
import { useBroker } from "@novox/mesh-sdk/messaging";
|
||||
import { collectTools, toolKey, type ToolDefinition } from "@novox/mesh-sdk/tools";
|
||||
import type { Broker } from "@novox/mesh-sdk/messaging";
|
||||
import { atWork, seatToolSubject, type Credential, type RuntimeBroker } from "./broker-nats.js";
|
||||
import { launch, launches } from "./launch.js";
|
||||
import { announce, endpointsOf, type ServedSeatVerb } from "./announce.js";
|
||||
|
||||
/**
|
||||
* The one verb every module's runtime answers for it (novox/hq ADR 0152, design 34 §3): the
|
||||
* module's tool names, descriptions and argument schemas, from the code that answers them and
|
||||
* from nowhere else. Discovery asks the module, because a copy kept anywhere else drifts.
|
||||
*/
|
||||
export const TOOLS_VERB = "tools";
|
||||
|
||||
/** What `tools` answers for one module. */
|
||||
export interface ToolsAnswer {
|
||||
module: string;
|
||||
tools: {
|
||||
name: string;
|
||||
description: string;
|
||||
input: Readonly<Record<string, unknown>>;
|
||||
/** Where this tool is answered, as the mesh issued it (ADR 0160): the module's plain subject
|
||||
* first when there is one, then this machine's. A caller composes nothing. */
|
||||
subjects?: string[];
|
||||
}[];
|
||||
/** Why this module serves nothing here, when its bundle failed to load (ADR 0175): said where
|
||||
* discovery looks, so a module that is silent and one that is broken are told apart. */
|
||||
failed?: string;
|
||||
}
|
||||
|
||||
/** One module this runtime serves: its name and its compiled tool entrypoints. */
|
||||
export interface ServedModule {
|
||||
module: string;
|
||||
/** Absolute paths to the module's compiled tool entrypoints (e.g. .../umami/tools/index.js). */
|
||||
entrypoints: string[];
|
||||
}
|
||||
|
||||
export interface RuntimeOptions {
|
||||
/** The mesh broker to serve over. */
|
||||
broker: Broker;
|
||||
/** The modules to serve, each with its entrypoints. */
|
||||
serves?: ServedModule[];
|
||||
/** The credential's own module's entrypoints — the one-module form, which the per-module
|
||||
* containers still use; the same as naming the credential's module in `serves`. */
|
||||
moduleEntrypoints?: string[];
|
||||
/** The credential the mesh delivered, for what it says about the seats this module claims
|
||||
* (novox/hq ADR 0159). Absent for a runtime started by hand, which then serves no seat its
|
||||
* memberships do not name. */
|
||||
credential?: Credential;
|
||||
/** What each served module's bundles are given (novox/hq ADR 0192): module → words, composed by
|
||||
* the mesh per machine. A module absent here is given the runtime's own words and nothing more. */
|
||||
envs?: ReadonlyMap<string, Readonly<Record<string, string>>>;
|
||||
}
|
||||
|
||||
/** The variable the mesh composes every served module's environment into, as JSON (ADR 0192). Read
|
||||
* once at start and removed from the process's environment, so no bundle finds another's there. */
|
||||
export const TOOL_ENV = "MESH_TOOL_ENV";
|
||||
|
||||
/** Read and remove the composed environments from an environment (the process's, by default). */
|
||||
export function takeToolEnvs(env: NodeJS.ProcessEnv = process.env): Map<string, Record<string, string>> {
|
||||
const raw = env[TOOL_ENV];
|
||||
delete env[TOOL_ENV];
|
||||
const out = new Map<string, Record<string, string>>();
|
||||
if (!raw) return out;
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(raw);
|
||||
} catch {
|
||||
throw new Error(`${TOOL_ENV} is not JSON; the mesh composes it as {"<module>": {"<word>": "<value>"}}`);
|
||||
}
|
||||
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
|
||||
throw new Error(`${TOOL_ENV} is not an object of modules`);
|
||||
}
|
||||
for (const [module, words] of Object.entries(parsed as Record<string, unknown>)) {
|
||||
if (!words || typeof words !== "object" || Array.isArray(words)) {
|
||||
throw new Error(`${TOOL_ENV}: ${module}'s environment is not an object of words`);
|
||||
}
|
||||
const own: Record<string, string> = {};
|
||||
for (const [k, v] of Object.entries(words as Record<string, unknown>)) own[k] = String(v);
|
||||
out.set(module, own);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Two environment words the mesh sets for the node's runtime and every tool reads from its
|
||||
* environment: whose machine this is (novox/hq to-be 37 §3, ADR 0175). */
|
||||
export const OPERATOR_ACCOUNT = "MESH_OPERATOR_ACCOUNT";
|
||||
export const OPERATOR_HOME = "MESH_OPERATOR_HOME";
|
||||
|
||||
/** Load the modules, bind the broker, and serve. Returns a stop function that unhooks serving. */
|
||||
export async function runTools(opts: RuntimeOptions): Promise<() => void> {
|
||||
useBroker(() => opts.broker);
|
||||
const runtime = opts.broker as RuntimeBroker;
|
||||
// Whose runtime this is: the credential's module, or the connection's own when a runtime is
|
||||
// started by hand without one — the broker was told its module when it connected.
|
||||
const self = opts.credential?.module ?? (typeof runtime.module === "string" ? runtime.module : undefined);
|
||||
|
||||
// What to serve: the list, with the one-module form folded in as the credential's own entry.
|
||||
const served = new Map<string, string[]>();
|
||||
for (const s of opts.serves ?? []) {
|
||||
served.set(s.module, [...(served.get(s.module) ?? []), ...s.entrypoints]);
|
||||
}
|
||||
const ownEntrypoints = opts.moduleEntrypoints ?? [];
|
||||
if (opts.moduleEntrypoints?.length) {
|
||||
if (!self) {
|
||||
throw new Error(
|
||||
"entrypoints were given with no module to serve them as: name the module (MESH_TOOL_MODULES " +
|
||||
"as <module>=<entrypoint>) or connect on a credential that names one",
|
||||
);
|
||||
}
|
||||
served.set(self, [...(served.get(self) ?? []), ...opts.moduleEntrypoints]);
|
||||
}
|
||||
|
||||
// The operator's machine, said once so a tool's behaviour under it can be read back from the
|
||||
// log. Tools read the two words from their own environment, which is this process's.
|
||||
const account = process.env[OPERATOR_ACCOUNT];
|
||||
if (account) {
|
||||
console.log(`[mesh-tools] the operator's account here is ${account}` +
|
||||
(process.env[OPERATOR_HOME] ? ` (home ${process.env[OPERATOR_HOME]})` : ""));
|
||||
}
|
||||
|
||||
// Follow every served module's membership before loading anything, so what each is issued is
|
||||
// known when its tools are bound. A module's own runtime already follows its own.
|
||||
if (typeof runtime.follow === "function") {
|
||||
for (const module of served.keys()) await runtime.follow(module);
|
||||
}
|
||||
|
||||
// What each module's bundles are given: the runtime's own words, and over them the module's own.
|
||||
const envs = opts.envs ?? new Map<string, Record<string, string>>();
|
||||
const envFor = (module: string): NodeJS.ProcessEnv => ({ ...process.env, ...(envs.get(module) ?? {}) });
|
||||
|
||||
// Every bundle this runtime serves is launched as a process and spoken to over MCP on stdio (ADR
|
||||
// 0188, ADR 0193): given the runtime's words and its module's own, and told the module it serves
|
||||
// it as. The runtime knows no language; an entrypoint that is not executable was not built to be
|
||||
// served, and is refused by name. A bundle that fails to start is named, and the others serve
|
||||
// (ADR 0175: one faulty bundle must not take the node's tools down).
|
||||
//
|
||||
// The one-module form — the credential's own module's entrypoints, which the per-module containers
|
||||
// still use for their event handlers and provisioners until they move (to-be 38 WP4c) — is imported
|
||||
// into this process as before: one module, one SDK, its container's own environment.
|
||||
// The machine this runtime serves, which an event a launched tool emits is stamped with.
|
||||
const node = opts.credential?.node ?? (typeof (runtime as unknown as { node?: unknown }).node === "string" ? (runtime as unknown as { node: string }).node : undefined);
|
||||
const failed = new Map<string, string>();
|
||||
const launched: { module: string; owner: string; tools: ToolDefinition[] }[] = [];
|
||||
const children: Array<() => void> = [];
|
||||
for (const [module, entrypoints] of served) {
|
||||
const imported = module === self && ownEntrypoints.length > 0;
|
||||
for (const entry of entrypoints) {
|
||||
const path = resolve(entry);
|
||||
try {
|
||||
if (imported) {
|
||||
await import(pathToFileURL(path).href);
|
||||
continue;
|
||||
}
|
||||
if (!launches(path)) {
|
||||
throw new Error(`${path} is not executable; a bundle the runtime serves is started, never imported, and its build makes it executable (novox/hq ADR 0193)`);
|
||||
}
|
||||
const child = await launch(module, path, {
|
||||
...envFor(module), MESH_SERVED_MODULE: module, MESH_MODULE: module,
|
||||
...(node ? { MESH_NODE: node } : {}),
|
||||
});
|
||||
children.push(child.stop);
|
||||
for (const r of child.registrations) launched.push({ ...r, owner: module });
|
||||
} catch (err) {
|
||||
const why = err instanceof Error ? err.message : String(err);
|
||||
failed.set(module, why);
|
||||
console.log(`[mesh-tools] ${module}'s bundle ${entry} failed to load: ${why}; its tools are not served here`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A registration under a served module's name is that module's tools, served on its subjects.
|
||||
// One under a seat's name is the module's implementation of that seat's verbs (ADR 0159, 0160):
|
||||
// served on the seat's subjects by serveClaimedSeats where some served module claims the seat,
|
||||
// never as a module's tools and never listed among them. A module named like its seat (the
|
||||
// catalogue is the mesh-catalog seat) registers once and is both. Anything else is said and left
|
||||
// out rather than fatal — on 2026-10-01 the credential of a module that had just learned to
|
||||
// implement a seat did not yet name the claim, and the whole runtime restarted for it.
|
||||
const claimed = seatsClaimed(served.keys(), self, opts.credential, runtime);
|
||||
const registrations = [
|
||||
...collectTools().map((r) => ({ ...r, owner: self ?? r.module })),
|
||||
...launched,
|
||||
];
|
||||
const ownRegistrations = registrations.filter(({ module, owner: by }) => {
|
||||
if (served.has(module)) return true;
|
||||
if (claimed.has(module)) return false;
|
||||
console.log(`[mesh-tools] ${by} registers tools under "${module}", which is neither a module served here nor a seat one of them claims; not served until the mesh issues the claim`);
|
||||
return false;
|
||||
});
|
||||
const stops: Array<() => void> = [...children];
|
||||
const stop = (): void => stops.splice(0).forEach((s) => s());
|
||||
|
||||
// Refused before anything is bound if a module named a tool of its own `tools`: one name
|
||||
// answering two things is the fault nobody can diagnose afterwards, and the runtime is the only
|
||||
// place that sees both. Likewise two tools of one module under one name.
|
||||
for (const { module, tools: own } of ownRegistrations) {
|
||||
if (own.some((t) => t.name === TOOLS_VERB)) {
|
||||
throw new Error(
|
||||
`${module} names a tool "${TOOLS_VERB}", which is the verb the runtime answers for every ` +
|
||||
"module with what it serves (novox/hq ADR 0152) — refused, rename it",
|
||||
);
|
||||
}
|
||||
const seen = new Set<string>();
|
||||
for (const t of own) {
|
||||
if (seen.has(t.name)) throw new Error(`${module} exposes two tools named ${t.name} — refused`);
|
||||
seen.add(t.name);
|
||||
}
|
||||
}
|
||||
|
||||
// Each tool on its own key, namespaced by its module (ADR 0047); where that key is answered is
|
||||
// the broker's to know from the module's membership (ADR 0160). A tool runs attributed to its
|
||||
// module, so what it emits lands on the module's subject and not the runtime's.
|
||||
const names: string[] = [];
|
||||
for (const { module, tools: own } of ownRegistrations) {
|
||||
for (const t of own) {
|
||||
names.push(toolKey(module, t.name));
|
||||
stops.push(await opts.broker.handle(toolKey(module, t.name), (args: Record<string, unknown> | undefined) =>
|
||||
atWork.run({ module }, () => t.run(args ?? {}))));
|
||||
}
|
||||
}
|
||||
|
||||
// And, for every served module, the verb that says what it serves — nothing, and why, for a
|
||||
// module whose bundle failed. A module that registered nothing and did not fail is a pure-events
|
||||
// module (the audit logger), whose scoped account may not declare the serve queue; it is left
|
||||
// silent as it always was.
|
||||
const byModule = new Map<string, ToolDefinition[]>();
|
||||
for (const { module, tools: own } of ownRegistrations) {
|
||||
byModule.set(module, [...(byModule.get(module) ?? []), ...own]);
|
||||
}
|
||||
for (const module of served.keys()) {
|
||||
const own = byModule.get(module) ?? [];
|
||||
const why = failed.get(module);
|
||||
if (own.length === 0 && !why) continue;
|
||||
const subjectsOf = (tool: string): string[] | undefined => {
|
||||
const m = typeof runtime.membership === "function" ? runtime.membership(module) : undefined;
|
||||
if (!m) return undefined;
|
||||
const plain = m.serves.filter((s) => s.queue).map((s) => s.subject.replace("{tool}", tool));
|
||||
const mine = m.serves.filter((s) => !s.queue).map((s) => s.subject.replace("{tool}", tool));
|
||||
return [...plain, ...mine];
|
||||
};
|
||||
stops.push(await opts.broker.handle(toolKey(module, TOOLS_VERB), async (): Promise<ToolsAnswer> => ({
|
||||
module,
|
||||
tools: own.map((t) => ({ name: t.name, description: t.description, input: t.input, subjects: subjectsOf(t.name) })),
|
||||
...(why ? { failed: why } : {}),
|
||||
})));
|
||||
}
|
||||
|
||||
console.log(`[mesh-tools] serving ${names.length} tool(s) for ${served.size} module(s): ${names.join(", ") || "(none)"}` +
|
||||
(failed.size ? `; not serving ${[...failed.keys()].join(", ")}, whose bundle(s) failed to load` : ""));
|
||||
const seats = await serveClaimedSeats(runtime, [...served.keys()], self, opts.credential, registrations);
|
||||
stops.push(seats.stop);
|
||||
// **What it serves, it announces** (novox/hq ADR 0197), asked at the moment of the request.
|
||||
if (typeof runtime.raw === "function") {
|
||||
stops.push(announce(runtime, {
|
||||
name: self ?? "runtime", id: runtime.node ?? self ?? "runtime",
|
||||
description: `the tool runtime of ${self ?? "a module"}${runtime.node ? ` on ${runtime.node}` : ""}`,
|
||||
metadata: runtime.node ? { node: runtime.node } : {},
|
||||
}, () => endpointsOf(runtime, ownRegistrations, seats.serving())));
|
||||
}
|
||||
return () => stop();
|
||||
}
|
||||
|
||||
/** The seats some served module claims: from the credential for its own module, and from every
|
||||
* served module's membership (ADR 0160) — the node's runtime holds no claims of its own. */
|
||||
function seatsClaimed(
|
||||
modules: Iterable<string>,
|
||||
self: string | undefined,
|
||||
credential: Credential | undefined,
|
||||
runtime: RuntimeBroker,
|
||||
): Set<string> {
|
||||
const out = new Set<string>();
|
||||
for (const c of credential?.claims ?? []) if (credential?.module === self) out.add(c.seat);
|
||||
for (const module of modules) {
|
||||
const m = typeof runtime.membership === "function" ? runtime.membership(module) : undefined;
|
||||
for (const s of m?.seats ?? []) out.add(s.seat);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** One seat's verb, where its callers ask, and which served module holds the seat. */
|
||||
interface SeatVerb {
|
||||
seat: string;
|
||||
verb: string;
|
||||
subject: string;
|
||||
holder: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Holding a seat means serving its tools (design 33 §3, novox/hq ADR 0159). What a served module
|
||||
* claims and promises comes from its membership (ADR 0160) — and, for a module's own runtime, from
|
||||
* its credential, which named the claims before memberships did. Each verb is served on the seat's
|
||||
* own subject by the tool of the same name registered under the seat's name. Whether this instance
|
||||
* *holds* the seat is the bus's to decide: only the holder's account may subscribe the seat's
|
||||
* subjects, so a claimant that does not hold it here is refused the subscription and serves nothing
|
||||
* — never a failure of its own tools.
|
||||
*/
|
||||
async function serveClaimedSeats(
|
||||
broker: RuntimeBroker,
|
||||
served: string[],
|
||||
self: string | undefined,
|
||||
credential: Credential | undefined,
|
||||
registrations: { module: string; owner: string; tools: ToolDefinition[] }[],
|
||||
): Promise<{ stop: () => void; serving: () => ServedSeatVerb[] }> {
|
||||
if (typeof broker.handleSubject !== "function") return { stop: () => {}, serving: () => [] };
|
||||
// A seat's verbs are the role's, not the software's (ADR 0159): implemented under the seat's
|
||||
// name — `registerModuleTools("mesh-store", …)` — and never confused with the module's own tools.
|
||||
const implementations = new Map<string, Map<string, (args: Record<string, unknown>) => Promise<unknown>>>();
|
||||
const definitions = new Map<string, Map<string, ToolDefinition>>();
|
||||
for (const { module, tools } of registrations) {
|
||||
const defs = definitions.get(module) ?? new Map<string, ToolDefinition>();
|
||||
for (const t of tools) defs.set(t.name, t);
|
||||
definitions.set(module, defs);
|
||||
}
|
||||
for (const { module, owner, tools } of registrations) {
|
||||
const verbs = implementations.get(module) ?? new Map<string, (args: Record<string, unknown>) => Promise<unknown>>();
|
||||
for (const t of tools) verbs.set(t.name, (args) => atWork.run({ module: owner }, () => t.run(args)));
|
||||
implementations.set(module, verbs);
|
||||
}
|
||||
|
||||
/** Every verb of every seat a served module claims, where the mesh issued it. */
|
||||
const wanted = (): SeatVerb[] => {
|
||||
const out: SeatVerb[] = [];
|
||||
const have = new Set<string>();
|
||||
const add = (v: SeatVerb): void => {
|
||||
if (have.has(v.subject)) return;
|
||||
have.add(v.subject);
|
||||
out.push(v);
|
||||
};
|
||||
for (const module of served) {
|
||||
const m = typeof broker.membership === "function" ? broker.membership(module) : undefined;
|
||||
for (const s of m?.seats ?? []) add({ seat: s.seat, verb: s.verb, subject: s.subject, holder: module });
|
||||
// The credential's claims, for the module's own runtime, ONLY until the mesh issues a
|
||||
// membership (novox/hq issue 218). The credential names what the module claims; the membership
|
||||
// names what it holds on this machine. A seat held once for the mesh is claimed by every
|
||||
// machine running the module and held by one, so once a membership exists it decides: a claim
|
||||
// it leaves out is not held here, and serving it anyway announced the seat from a machine the
|
||||
// bus then refused it on.
|
||||
if (module !== self || m) continue;
|
||||
for (const claim of credential?.claims ?? []) {
|
||||
for (const verb of claim.serves ?? []) {
|
||||
const subject = seatToolSubject(claim.seat, verb, claim.scope, credential?.node);
|
||||
add({ seat: claim.seat, verb, subject, holder: module });
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
};
|
||||
|
||||
let stops: (() => void)[] = [];
|
||||
let servingNow: ServedSeatVerb[] = [];
|
||||
const serve = async (): Promise<void> => {
|
||||
stops.forEach((s) => s());
|
||||
stops = [];
|
||||
servingNow = [];
|
||||
for (const v of wanted()) {
|
||||
const run = implementations.get(v.seat)?.get(v.verb);
|
||||
const tool = definitions.get(v.seat)?.get(v.verb);
|
||||
if (!run) {
|
||||
console.log(`[mesh-tools] ${v.holder} claims ${v.seat} and implements no ${v.verb}, which that seat promises; not served`);
|
||||
continue;
|
||||
}
|
||||
stops.push(await broker.handleSubject(v.subject, run));
|
||||
if (tool) servingNow.push({ ...v, tool });
|
||||
console.log(`[mesh-tools] serving ${v.seat}'s ${v.verb} on ${v.subject}, admitted where ${v.holder} holds the seat`);
|
||||
}
|
||||
};
|
||||
await serve();
|
||||
// A membership issued to any served module may add, move or withdraw a seat's verbs.
|
||||
if (typeof broker.onMembership === "function") broker.onMembership(() => void serve());
|
||||
return { stop: () => stops.forEach((s) => s()), serving: () => servingNow };
|
||||
}
|
||||
@@ -1,69 +0,0 @@
|
||||
/**
|
||||
* What a runtime serves, it announces (novox/hq ADR 0197): the TypeScript runtime the per-module
|
||||
* containers still run answers the NATS services protocol's discovery in the same shape as the Go
|
||||
* tool runtime — its module's tools on every subject issued, and the seat verbs it serves.
|
||||
*
|
||||
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/announce.test.ts
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { connect, StringCodec } from "nats";
|
||||
|
||||
import { resetTools } from "@novox/mesh-sdk/tools";
|
||||
|
||||
import { connectNats, membershipSubject } from "../dist/broker-nats.js";
|
||||
import { runTools } from "../dist/runtime.js";
|
||||
|
||||
const url = process.env.MESH_TEST_NATS;
|
||||
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
|
||||
const sc = StringCodec();
|
||||
|
||||
test("the runtime answers $SRV.INFO with what it serves, in the services protocol's format", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const nc = await connect({ servers: url });
|
||||
const jsm = await nc.jetstreamManager();
|
||||
try {
|
||||
await jsm.streams.delete("ASSIGNMENTS");
|
||||
} catch {
|
||||
// none yet
|
||||
}
|
||||
await jsm.streams.add({ name: "ASSIGNMENTS", subjects: ["mesh.assignment.>"], max_msgs_per_subject: 1, allow_direct: true } as never);
|
||||
await nc.jetstream().publish(membershipSubject("anchor", "shop"), sc.encode(JSON.stringify({
|
||||
node: "anchor", module: "shop",
|
||||
serves: [{ subject: "mesh.mod.shop.tool.{tool}.anchor" }, { subject: "mesh.mod.shop.tool.{tool}", queue: "serve.shop" }],
|
||||
emits: "mesh.mod.shop.event.{event}", tools: "mesh.mod.shop.tool.tools",
|
||||
})));
|
||||
const shop = await connectNats({ url, node: "anchor", module: "shop" });
|
||||
let stop = () => {};
|
||||
try {
|
||||
stop = await runTools({ broker: shop, moduleEntrypoints: [fixture("shop-tools.mjs")] });
|
||||
const msg = await nc.request("$SRV.INFO.shop.anchor", sc.encode(""), { timeout: 2000 });
|
||||
const info = JSON.parse(sc.decode(msg.data)) as {
|
||||
type: string; name: string; id: string; version: string;
|
||||
endpoints: { name: string; subject: string; queue_group: string; metadata: Record<string, string> }[];
|
||||
};
|
||||
assert.equal(info.type, "io.nats.micro.v1.info_response");
|
||||
assert.equal(info.name, "shop");
|
||||
assert.equal(info.id, "anchor");
|
||||
assert.ok(info.version);
|
||||
const price = info.endpoints.filter((e) => e.metadata.tool === "price").map((e) => `${e.subject}|${e.queue_group}`).sort();
|
||||
assert.deepEqual(price, ["mesh.mod.shop.tool.price.anchor|", "mesh.mod.shop.tool.price|serve.shop"]);
|
||||
const one = info.endpoints.find((e) => e.metadata.tool === "price")!;
|
||||
assert.equal(one.metadata.kind, "tool");
|
||||
assert.equal(one.metadata.module, "shop");
|
||||
assert.equal(one.metadata.node, "anchor");
|
||||
assert.equal(one.metadata.interchangeable, "true");
|
||||
assert.ok(JSON.parse(one.metadata.schema).type === "object");
|
||||
// Ping answers with the same identity; another service's request is not answered.
|
||||
const ping = JSON.parse(sc.decode((await nc.request("$SRV.PING", sc.encode(""), { timeout: 2000 })).data));
|
||||
assert.equal(ping.type, "io.nats.micro.v1.ping_response");
|
||||
await assert.rejects(nc.request("$SRV.INFO.somebody-else", sc.encode(""), { timeout: 300 }));
|
||||
} finally {
|
||||
stop();
|
||||
await shop.close();
|
||||
await nc.close();
|
||||
resetTools();
|
||||
}
|
||||
});
|
||||
@@ -1,188 +0,0 @@
|
||||
/**
|
||||
* A person's client, against a real bus.
|
||||
*
|
||||
* What is worth checking is not that a request/reply works — the runtime's own tests cover that — but
|
||||
* that the two surfaces are the same thing. An agent and a person must see the same tools and get the
|
||||
* same answers, or the MCP surface becomes a second definition of what a tool is.
|
||||
*
|
||||
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
|
||||
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/client.test.ts
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
|
||||
// The built output, not the source: the client imports its siblings as `.js`, which is what ships and
|
||||
// what every other file here does, and cannot be loaded as TypeScript directly. `pretest` builds.
|
||||
import { connectNats } from "../dist/broker-nats.js";
|
||||
import { callTool, seatsIn, toolKey, toolsOn, whyItFailed } from "../dist/client.js";
|
||||
|
||||
const url = process.env.MESH_TEST_NATS;
|
||||
|
||||
/** A mesh as discovery sees it (design 34 §3): the catalogue holding three modules, two of them up
|
||||
* and answering `tools`, one held and not running. Two connections, because a person and a module
|
||||
* are different users even in a test. */
|
||||
async function aMeshWithTools() {
|
||||
const catalogue = await connectNats({ url: url!, module: "mesh-catalog" });
|
||||
const shop = await connectNats({ url: url!, module: "shop" });
|
||||
await catalogue.handle("catalog_modules", async () => ({
|
||||
modules: [{ module: "shop" }, { module: "mesh-catalog" }, { module: "ghost" }],
|
||||
}));
|
||||
await catalogue.handle("tools", async () => ({
|
||||
module: "mesh-catalog",
|
||||
tools: [{ name: "catalog_modules", description: "what modules the mesh has", input: {} }],
|
||||
}));
|
||||
await shop.handle("tools", async () => ({
|
||||
module: "shop",
|
||||
tools: [{ name: "price", description: "what something costs", input: { type: "object" } }],
|
||||
}));
|
||||
await shop.handle("price", async (body: { of?: string }) => ({ of: body.of ?? "nothing", cost: 12 }));
|
||||
// And the mesh's own records, served by the holder of the mesh-controller seat (ADR 0154): one
|
||||
// role's tool, and the verb that lists every role's.
|
||||
const controller = await connectNats({ url: url!, module: "mesh-controller" });
|
||||
await controller.handle("seat:mesh-controller.tools", async () => ({
|
||||
seats: [
|
||||
{ seat: "mesh-controller", scope: "mesh", tools: [{ name: "status", description: "what is wrong", input: {} }] },
|
||||
{ seat: "node-dns-resolver", scope: "node", tools: [{ name: "lookup", description: "one machine's", input: {} }] },
|
||||
],
|
||||
}));
|
||||
await controller.handle("seat:mesh-controller.status", async () => ({ output: "all quiet", ok: true }));
|
||||
return {
|
||||
async close() {
|
||||
await catalogue.close();
|
||||
await shop.close();
|
||||
await controller.close();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
test("a person sees what the running modules answer, sorted, and who did not answer", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const mesh = await aMeshWithTools();
|
||||
const person = await connectNats({ url, module: "person.ada" });
|
||||
try {
|
||||
const began = Date.now();
|
||||
const have = await toolsOn(person);
|
||||
assert.deepEqual(
|
||||
have.tools.map((x) => `${x.module}.${x.name}`),
|
||||
["mesh-catalog.catalog_modules", "mesh-controller.status", "node-dns-resolver.lookup", "shop.price"],
|
||||
"the list is what the modules answered plus every role's tools, in a stable order",
|
||||
);
|
||||
// A role's tool is marked as one; a node-scoped seat's carries its scope, so a caller names
|
||||
// the machine and the verb resolves as the seat's (design 33 §4, ADR 0170).
|
||||
assert.ok(have.tools.find((x) => x.module === "mesh-controller")!.seat);
|
||||
const lookup = have.tools.find((x) => x.module === "node-dns-resolver")!;
|
||||
assert.ok(lookup.seat && lookup.scope === "node");
|
||||
assert.equal(toolKey("node-dns-resolver.lookup", seatsIn(have)), "seat:node-dns-resolver.lookup");
|
||||
// Silence is named, never dropped: a module the catalogue holds and nothing answered for.
|
||||
assert.deepEqual(have.notAnswering, ["ghost"]);
|
||||
// And at once: a module that is not running costs nothing, or the list is unusable.
|
||||
assert.ok(Date.now() - began < 5_000, "an absent module waited out the timeout");
|
||||
} finally {
|
||||
await person.close();
|
||||
await mesh.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("a person calls a tool and gets the module's own answer, unshaped", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const mesh = await aMeshWithTools();
|
||||
const person = await connectNats({ url, module: "person.ada" });
|
||||
try {
|
||||
const answer = await callTool(person, "shop.price", { of: "a hat" });
|
||||
assert.deepEqual(answer.result, { of: "a hat", cost: 12 });
|
||||
} finally {
|
||||
await person.close();
|
||||
await mesh.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("a tool nobody serves says so at once, and says what to do about it", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const person = await connectNats({ url: url!, module: "person.ada" });
|
||||
try {
|
||||
const began = Date.now();
|
||||
await assert.rejects(() => callTool(person, "ghost.missing", {}));
|
||||
// At once, not after the whole wait: "that module is down" and "that tool is slow" need
|
||||
// different things done, and a timeout cannot tell them apart.
|
||||
assert.ok(Date.now() - began < 5_000, "a tool nobody serves waited out the timeout");
|
||||
} finally {
|
||||
await person.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("a name that is not <module>.<tool> is refused before anything is sent", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const person = await connectNats({ url: url!, module: "person.ada" });
|
||||
try {
|
||||
await assert.rejects(() => callTool(person, "price", {}), /does not name a tool/);
|
||||
} finally {
|
||||
await person.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("each way a call fails says what to do about it", () => {
|
||||
// The three answers a person actually gets. Without this they are one timeout and a stack trace,
|
||||
// and the remedies are in three different places.
|
||||
assert.match(whyItFailed("shop.price", new Error("no responders")), /nothing serves shop\.price/);
|
||||
assert.match(
|
||||
whyItFailed("shop.price", new Error("Permissions Violation for Publish")),
|
||||
/may not call shop\.price/,
|
||||
);
|
||||
assert.match(whyItFailed("shop.price", new Error("timeout")), /did not answer in time/);
|
||||
assert.match(whyItFailed("shop.price", new Error("something else")), /something else/);
|
||||
});
|
||||
|
||||
test("a role's tool is reached through the seat, and a module's own name is never shadowed", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const { seatsIn, toolKey } = await import("../dist/client.js");
|
||||
const mesh = await aMeshWithTools();
|
||||
const person = await connectNats({ url, module: "person.ada" });
|
||||
try {
|
||||
const roles = seatsIn(await toolsOn(person));
|
||||
assert.equal(toolKey("mesh-controller.status", roles), "seat:mesh-controller.status");
|
||||
assert.equal(toolKey("mesh-controller.other", roles), "mesh-controller.other", "a verb the seat does not declare is a module's");
|
||||
assert.equal(toolKey("shop.price", roles), "shop.price");
|
||||
const answer = await callTool(person, "mesh-controller.status", {}, roles);
|
||||
assert.deepEqual(answer.result, { output: "all quiet", ok: true });
|
||||
const direct = await callTool(person, "seat:mesh-controller.status", {});
|
||||
assert.deepEqual(direct.result, { output: "all quiet", ok: true });
|
||||
} finally {
|
||||
await person.close();
|
||||
await mesh.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A module on two machines (novox/hq ADR 0159): a call names the machine and reaches that instance
|
||||
// and no other; a call that names none reaches one of them and says which; a seat's verb is served
|
||||
// by the claimant's tool of the same name on the seat's own subject.
|
||||
test("a tool call names the machine it is for, and every answer says which machine answered", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const onAnchor = await connectNats({ url: url!, module: "store", node: "anchor" });
|
||||
const onHome = await connectNats({ url: url!, module: "store", node: "home-server" });
|
||||
const asker = await connectNats({ url: url!, module: "console", node: "workstation" });
|
||||
try {
|
||||
await onAnchor.handle("databases", async () => ({ at: "anchor" }));
|
||||
await onHome.handle("databases", async () => ({ at: "home-server" }));
|
||||
|
||||
const home = await callTool(asker, "store.databases@home-server", {});
|
||||
assert.deepEqual(home.result, { at: "home-server" });
|
||||
assert.equal(home.node, "home-server");
|
||||
const anchor = await callTool(asker, "store.databases@anchor", {});
|
||||
assert.deepEqual(anchor.result, { at: "anchor" });
|
||||
assert.equal(anchor.node, "anchor");
|
||||
|
||||
const whichever = await callTool(asker, "store.databases", {});
|
||||
assert.ok(["anchor", "home-server"].includes(whichever.node ?? ""), `an unnamed call still says who answered: ${whichever.node}`);
|
||||
assert.deepEqual(whichever.result, { at: whichever.node });
|
||||
|
||||
// A seat's verb, served by the claimant on the seat's subject; a mesh seat's is flat.
|
||||
await onAnchor.handleSubject("mesh.seat.mesh-store.tool.databases", async () => ({ seat: "mesh-store", at: "anchor" }));
|
||||
const viaSeat = await callTool(asker, "seat:mesh-store.databases", {});
|
||||
assert.deepEqual(viaSeat.result, { seat: "mesh-store", at: "anchor" });
|
||||
assert.equal(viaSeat.node, "anchor");
|
||||
} finally {
|
||||
await onAnchor.close();
|
||||
await onHome.close();
|
||||
await asker.close();
|
||||
}
|
||||
});
|
||||
@@ -1,54 +0,0 @@
|
||||
// The TypeScript implementation, held to the shared fixtures (novox/hq ADR 0074, design 19).
|
||||
//
|
||||
// Run against a NATS server, because the question is what actually reaches the wire:
|
||||
//
|
||||
// docker run -d --rm --name c -p 14222:4222 nats:2.10-alpine -js
|
||||
// node test/conformance.mjs
|
||||
//
|
||||
// **The runner lives with the implementation it exercises; the fixture does not.** It is read
|
||||
// from the sdk's conformance directory by sibling path — the same file the Go suite reads. A
|
||||
// fixture copied into each implementation is two fixtures, and two fixtures drift, which is the
|
||||
// failure the suite exists to prevent.
|
||||
import { readFileSync } from "node:fs";
|
||||
import { connect } from "nats";
|
||||
|
||||
const clientPath = process.argv[2] ?? "../dist/broker-nats.js";
|
||||
const { connectNats } = await import(clientPath);
|
||||
const f = JSON.parse(readFileSync(new URL("../../mesh-sdk/conformance/events/module-event.json", import.meta.url)));
|
||||
const URL_ = process.env.MESH_TEST_NATS ?? "nats://127.0.0.1:14222";
|
||||
|
||||
let failed = 0;
|
||||
const check = (ok, what) => { console.log(` ${ok ? "ok " : "FAIL"} ${what}`); if (!ok) failed++; };
|
||||
|
||||
const admin = await connect({ servers: URL_ });
|
||||
const jsm = await admin.jetstreamManager();
|
||||
await jsm.streams.add({ name: "EVENTS", subjects: ["mesh.mod.*.event.>", "mesh.seat.*.event.>"] });
|
||||
|
||||
const shop = await connectNats({ url: URL_, node: f.given.node, module: f.given.module });
|
||||
await shop.publish({ key: f.given.key, node: f.given.node, body: f.given.body, headers: f.given.headers });
|
||||
|
||||
// What actually landed, read back from the stream rather than from the client that wrote it.
|
||||
const msg = await jsm.streams.getMessage("EVENTS", { last_by_subj: f.wire.subject });
|
||||
check(!!msg, `it lands on ${f.wire.subject}`);
|
||||
|
||||
if (msg) {
|
||||
const got = {};
|
||||
if (msg.header) for (const k of msg.header.keys()) got[k] = msg.header.get(k);
|
||||
for (const h of f.wire.requiredHeaders) {
|
||||
check(got[h] !== undefined && got[h] !== "", `${h} is set`);
|
||||
}
|
||||
check(got["content-type"] === f.wire.headerFormats["content-type"], "content-type is as pinned");
|
||||
check(new RegExp(f.wire.headerFormats["x-event-id"]).test(got["x-event-id"]), "x-event-id is as pinned");
|
||||
check(!Number.isNaN(Date.parse(got["x-time"])), "x-time parses as a date");
|
||||
check(got["x-source"] === f.given.module, "x-source agrees with the subject's module");
|
||||
|
||||
const payload = JSON.parse(new TextDecoder().decode(msg.data));
|
||||
check(payload.envelope === undefined && payload.key === undefined,
|
||||
"the payload is the body alone, not the envelope (the fixture refuses nesting)");
|
||||
check(JSON.stringify(payload) === JSON.stringify(f.given.body),
|
||||
"the body round-trips — semantic, not byte-exact, per the README");
|
||||
}
|
||||
|
||||
await shop.close(); await admin.close();
|
||||
console.log(failed ? `\n${failed} failed` : "\nall passed");
|
||||
process.exit(failed ? 1 : 0);
|
||||
+3
-2
@@ -1,6 +1,7 @@
|
||||
#!/usr/bin/env python3
|
||||
# A tools bundle that says who the runtime said called it (novox/hq issue 365): MCP over stdio, no SDK. Its tool
|
||||
# and its seat verb answer the caller the call's `_meta` named — "" when it named none — and the arguments as given.
|
||||
# and its seat verb answer the caller the call's `_meta` named — "" when it named none —, the id of the call in flight
|
||||
# it named (novox/hq ADR 0597 §7), and the arguments as given.
|
||||
import json, sys
|
||||
|
||||
def say(m):
|
||||
@@ -18,7 +19,7 @@ for line in sys.stdin:
|
||||
say({"id": rid, "result": {"tools": TOOLS}})
|
||||
elif m == "tools/call":
|
||||
meta = p.get("_meta") or {}
|
||||
said = {"caller": meta.get("mesh/caller", ""), "arguments": p.get("arguments") or {}}
|
||||
said = {"caller": meta.get("mesh/caller", ""), "call": meta.get("mesh/call", ""), "arguments": p.get("arguments") or {}}
|
||||
say({"id": rid, "result": {"content": [{"type": "text", "text": json.dumps(said)}]}})
|
||||
elif rid is not None:
|
||||
say({"id": rid, "result": {}})
|
||||
|
||||
@@ -1,113 +0,0 @@
|
||||
/**
|
||||
* The console: the same surface over HTTP on loopback, started the way the mesh starts it — on the
|
||||
* module credential in MESH_BROKER_FILE (novox/hq ADR 0152, design 34 §2).
|
||||
*
|
||||
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
|
||||
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/http.test.ts
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { mkdtemp, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { connectNats } from "../dist/broker-nats.js";
|
||||
import { serveMcpHttp } from "../dist/http.js";
|
||||
|
||||
const url = process.env.MESH_TEST_NATS;
|
||||
|
||||
async function aMesh(t: { after: (fn: () => Promise<void> | void) => void }) {
|
||||
const catalogue = await connectNats({ url: url!, module: "mesh-catalog" });
|
||||
const shop = await connectNats({ url: url!, module: "shop" });
|
||||
await catalogue.handle("catalog_modules", async () => ({ modules: [{ module: "shop" }] }));
|
||||
await shop.handle("tools", async () => ({
|
||||
module: "shop",
|
||||
tools: [{ name: "price", description: "what something costs", input: {} }],
|
||||
}));
|
||||
await shop.handle("price", async (body: { of?: string }) => ({ of: body.of ?? "nothing", cost: 12 }));
|
||||
t.after(async () => {
|
||||
await catalogue.close();
|
||||
await shop.close();
|
||||
});
|
||||
}
|
||||
|
||||
/** `mesh serve` as the mesh runs it: MESH_BROKER_FILE, a listen address, nothing else. */
|
||||
async function aConsole(t: { after: (fn: () => Promise<void> | void) => void }): Promise<string> {
|
||||
const dir = await mkdtemp("/tmp/mesh-console-");
|
||||
const credential = join(dir, "broker");
|
||||
await writeFile(credential, JSON.stringify({ url, node: "desk", module: "mesh-console", user: "desk.mesh-console", password: "x" }));
|
||||
const child: ChildProcess = spawn(process.execPath, ["dist/mesh.js", "serve", "--listen", "127.0.0.1:0"], {
|
||||
env: { ...process.env, MESH_BROKER_FILE: credential, MESH_CREDENTIAL: "" },
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
t.after(() => {
|
||||
child.kill("SIGTERM");
|
||||
});
|
||||
return new Promise((resolve, reject) => {
|
||||
let out = "";
|
||||
let err = "";
|
||||
child.stdout!.on("data", (d) => {
|
||||
out += d.toString();
|
||||
const m = /listening on (http:\/\/[^/]+\/mcp)/.exec(out);
|
||||
if (m) resolve(m[1]!);
|
||||
});
|
||||
child.stderr!.on("data", (d) => (err += d.toString()));
|
||||
child.on("exit", (code) => reject(new Error(`serve exited ${code}: ${err}`)));
|
||||
});
|
||||
}
|
||||
|
||||
async function post(endpoint: string, body: unknown): Promise<{ status: number; json?: any }> {
|
||||
const res = await fetch(endpoint, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json", accept: "application/json" },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
const text = await res.text();
|
||||
return { status: res.status, json: text ? JSON.parse(text) : undefined };
|
||||
}
|
||||
|
||||
test("the console answers a host on loopback, as the account the mesh gave it", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
await aMesh(t);
|
||||
const endpoint = await aConsole(t);
|
||||
|
||||
const hello = await post(endpoint, { jsonrpc: "2.0", id: 1, method: "initialize", params: {} });
|
||||
assert.equal(hello.status, 200);
|
||||
assert.match(hello.json.result.instructions, /desk\.mesh-console/, "the handshake names the console's account");
|
||||
|
||||
const heard = await post(endpoint, { jsonrpc: "2.0", method: "notifications/initialized" });
|
||||
assert.equal(heard.status, 202, "a notification is heard and not answered");
|
||||
|
||||
const listed = await post(endpoint, { jsonrpc: "2.0", id: 2, method: "tools/list" });
|
||||
assert.deepEqual(listed.json.result.tools.map((x: { name: string }) => x.name), ["shop.price"]);
|
||||
|
||||
const called = await post(endpoint, {
|
||||
jsonrpc: "2.0", id: 3, method: "tools/call", params: { name: "shop.price", arguments: { of: "a hat" } },
|
||||
});
|
||||
assert.deepEqual(JSON.parse(called.json.result.content[0].text), { of: "a hat", cost: 12 });
|
||||
|
||||
// A person's client through the same endpoint, with no credential of its own.
|
||||
const { main } = await import("../dist/mesh.js");
|
||||
const logged: string[] = [];
|
||||
const was = console.log;
|
||||
console.log = (line: string) => logged.push(String(line));
|
||||
try {
|
||||
assert.equal(await main(["tools", "--console", endpoint]), 0);
|
||||
} finally {
|
||||
console.log = was;
|
||||
}
|
||||
assert.ok(logged.some((l) => l.startsWith("shop.price")), `the client did not list through the console: ${logged}`);
|
||||
});
|
||||
|
||||
test("the console binds loopback and nowhere else", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const bus = await connectNats({ url, module: "mesh-console", node: "desk" });
|
||||
try {
|
||||
await assert.rejects(() => serveMcpHttp(bus, "desk.mesh-console", "0.0.0.0:0"), /loopback and nowhere else/);
|
||||
const up = await serveMcpHttp(bus, "desk.mesh-console", "127.0.0.1:0");
|
||||
assert.match(up.address, /^127\.0\.0\.1:\d+$/);
|
||||
await up.close();
|
||||
} finally {
|
||||
await bus.close();
|
||||
}
|
||||
});
|
||||
@@ -1,196 +0,0 @@
|
||||
/**
|
||||
* The MCP surface, driven the way a host drives it.
|
||||
*
|
||||
* **The claim worth checking is that it is the same thing the command line is.** An agent and a
|
||||
* person must see the same tools and get the same answers, or this becomes a second definition of what
|
||||
* a tool is — which is exactly what a thin adapter is supposed to avoid.
|
||||
*
|
||||
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
|
||||
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/mcp.test.ts
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { spawn } from "node:child_process";
|
||||
|
||||
import { connectNats } from "../dist/broker-nats.js";
|
||||
|
||||
const url = process.env.MESH_TEST_NATS;
|
||||
|
||||
/** A module answering the catalogue's list and one tool, plus a credential file the client reads. */
|
||||
async function aMeshAndACredential(t: { after: (fn: () => Promise<void> | void) => void }) {
|
||||
const catalogue = await connectNats({ url: url!, module: "mesh-catalog" });
|
||||
const shop = await connectNats({ url: url!, module: "shop" });
|
||||
await catalogue.handle("catalog_modules", async () => ({
|
||||
modules: [{ module: "shop" }, { module: "ghost" }],
|
||||
}));
|
||||
await shop.handle("tools", async () => ({
|
||||
module: "shop",
|
||||
tools: [{ name: "price", description: "what something costs", input: { of: { type: "string" } } }],
|
||||
}));
|
||||
await shop.handle("price", async (body: { of?: string }) => ({ of: body.of ?? "nothing", cost: 12 }));
|
||||
const controller = await connectNats({ url: url!, module: "mesh-controller" });
|
||||
await controller.handle("seat:mesh-controller.tools", async () => ({
|
||||
seats: [
|
||||
{ seat: "mesh-controller", scope: "mesh", tools: [
|
||||
{ name: "status", description: "what is wrong", input: {} },
|
||||
{ name: "push", description: "tell a machine", input: { node: { type: "string" } } },
|
||||
] },
|
||||
// A seat held once per machine (design 33 §4, ADR 0170): its verb is asked of one.
|
||||
{ seat: "node-dns-resolver", scope: "node", tools: [{ name: "lookup", description: "one machine's", input: {} }] },
|
||||
],
|
||||
}));
|
||||
await controller.handle("seat:node-dns-resolver.lookup@anchor", async () => ({ machine: "anchor", answered: true }));
|
||||
await controller.handle("seat:mesh-controller.status", async () => ({ output: "all quiet", ok: true }));
|
||||
await controller.handle("seat:mesh-controller.push", async (body: { node?: string }) => ({ told: body.node ?? "nobody" }));
|
||||
t.after(async () => {
|
||||
await catalogue.close();
|
||||
await shop.close();
|
||||
await controller.close();
|
||||
});
|
||||
|
||||
const { mkdtemp, writeFile } = await import("node:fs/promises");
|
||||
const { join } = await import("node:path");
|
||||
const dir = await mkdtemp("/tmp/mesh-client-");
|
||||
const path = join(dir, "credential.json");
|
||||
await writeFile(
|
||||
path,
|
||||
JSON.stringify({ url, user: "person.ada", password: "x", person: "ada", invokes: ["shop.price"] }),
|
||||
);
|
||||
return path;
|
||||
}
|
||||
|
||||
/** Drive `mesh mcp` over stdio and collect the replies, as a host would. */
|
||||
function driving(credential: string, requests: unknown[]): Promise<Record<string, any>[]> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const child = spawn(process.execPath, ["dist/mesh.js", "mcp", "--credential", credential], {
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
});
|
||||
let out = "";
|
||||
let err = "";
|
||||
child.stdout.on("data", (d) => (out += d.toString()));
|
||||
child.stderr.on("data", (d) => (err += d.toString()));
|
||||
child.on("error", reject);
|
||||
child.on("close", () => {
|
||||
const replies = out
|
||||
.split("\n")
|
||||
.filter((l) => l.trim() !== "")
|
||||
.map((l) => JSON.parse(l) as Record<string, any>);
|
||||
if (replies.length === 0 && err !== "") reject(new Error(err));
|
||||
else resolve(replies);
|
||||
});
|
||||
for (const r of requests) child.stdin.write(`${JSON.stringify(r)}\n`);
|
||||
child.stdin.end();
|
||||
});
|
||||
}
|
||||
|
||||
test("a host initialises, lists the mesh's tools and calls one", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const credential = await aMeshAndACredential(t);
|
||||
|
||||
const replies = await driving(credential, [
|
||||
{ jsonrpc: "2.0", id: 1, method: "initialize", params: {} },
|
||||
{ jsonrpc: "2.0", method: "notifications/initialized" },
|
||||
{ jsonrpc: "2.0", id: 2, method: "tools/list" },
|
||||
{ jsonrpc: "2.0", id: 3, method: "tools/call", params: { name: "shop.price", arguments: { of: "a hat" } } },
|
||||
]);
|
||||
|
||||
const byId = new Map(replies.map((r) => [r.id, r]));
|
||||
// A notification is answered with nothing, or a host waiting on ids sees a reply it cannot match.
|
||||
assert.equal(replies.length, 3, `expected three replies, got ${JSON.stringify(replies)}`);
|
||||
|
||||
const hello = byId.get(1)!.result;
|
||||
assert.equal(hello.protocolVersion, "2025-03-26");
|
||||
assert.ok(hello.capabilities.tools, "a server offering no tools is not this one");
|
||||
assert.match(hello.instructions, /ada/, "the handshake says whose authority a call is made under");
|
||||
|
||||
const listed = byId.get(2)!.result.tools;
|
||||
assert.deepEqual(listed.map((x: { name: string }) => x.name),
|
||||
["mesh-controller.push", "mesh-controller.status", "node-dns-resolver.lookup", "shop.price"],
|
||||
"the modules' tools and the roles', named the way a person names them");
|
||||
// A node-scoped seat's verb takes the machine, and requires it (ADR 0170).
|
||||
const lookup = listed[2];
|
||||
assert.equal(lookup.inputSchema.properties.node.type, "string");
|
||||
assert.deepEqual(lookup.inputSchema.required, ["node"]);
|
||||
const price = listed[3];
|
||||
assert.ok(price.inputSchema, "a tool with no schema is one an agent cannot call");
|
||||
// A module's bare property map arrives as a schema an agent can read, its words kept — and
|
||||
// `node`, the machine to ask when the module runs on several (novox/hq ADR 0159), beside them.
|
||||
assert.deepEqual(price.inputSchema.properties.of, { type: "string" });
|
||||
assert.equal(price.inputSchema.properties.node.type, "string", "a module's tool takes the machine to ask");
|
||||
assert.ok(!listed[1].inputSchema.properties?.node, "a seat's verb takes no machine; the seat's scope decides");
|
||||
assert.equal(listed[0].inputSchema.properties?.node?.type, "string", "a seat's verb that takes a node of its own keeps it");
|
||||
// Silence is named: the module the catalogue holds and nothing answered for.
|
||||
assert.deepEqual(byId.get(2)!.result._meta.notAnswering, ["ghost"]);
|
||||
|
||||
const called = byId.get(3)!.result;
|
||||
assert.ok(!called.isError, `the call failed: ${JSON.stringify(called)}`);
|
||||
// The module's own answer, unshaped. An adapter that summarised it would be deciding what matters
|
||||
// in somebody else's answer.
|
||||
assert.deepEqual(JSON.parse(called.content[0].text), { of: "a hat", cost: 12 });
|
||||
});
|
||||
|
||||
test("a tool nobody serves comes back as an error the agent can act on", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const credential = await aMeshAndACredential(t);
|
||||
|
||||
const replies = await driving(credential, [
|
||||
{ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "ghost.missing", arguments: {} } },
|
||||
]);
|
||||
const result = replies[0].result;
|
||||
// isError, not a protocol failure: the call was well-formed and the mesh answered it — with an
|
||||
// absence. A JSON-RPC error would tell the agent its request was malformed, which it was not.
|
||||
assert.ok(result?.isError, `expected a tool error, got ${JSON.stringify(replies[0])}`);
|
||||
assert.match(result.content[0].text, /nothing serves ghost\.missing/);
|
||||
});
|
||||
|
||||
test("a method this surface does not have is refused, and a notification is not", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const credential = await aMeshAndACredential(t);
|
||||
|
||||
const replies = await driving(credential, [
|
||||
{ jsonrpc: "2.0", id: 1, method: "resources/list" },
|
||||
{ jsonrpc: "2.0", method: "notifications/cancelled" },
|
||||
]);
|
||||
assert.equal(replies.length, 1, "a notification was answered");
|
||||
assert.equal(replies[0].error.code, -32601);
|
||||
assert.match(replies[0].error.message, /resources\/list/);
|
||||
});
|
||||
|
||||
// A seat's verb that takes a machine as its own argument — `push <node>` — keeps it: the console
|
||||
// moves `node` into the subject for a module's tool only (ADR 0159), never for a role's verb.
|
||||
test("a seat's verb keeps a node of its own; only a module's tool gives it to the subject", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const credential = await aMeshAndACredential(t);
|
||||
const replies = await driving(credential, [
|
||||
{ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "mesh-controller.push", arguments: { node: "anchor" } } },
|
||||
]);
|
||||
const result = replies[0].result;
|
||||
assert.ok(!result.isError, JSON.stringify(replies[0]));
|
||||
assert.deepEqual(JSON.parse(result.content[0].text), { told: "anchor" });
|
||||
});
|
||||
|
||||
test("a host calls the mesh's own verb through the seat", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const credential = await aMeshAndACredential(t);
|
||||
const replies = await driving(credential, [
|
||||
{ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "mesh-controller.status", arguments: {} } },
|
||||
]);
|
||||
const result = replies[0].result;
|
||||
assert.ok(!result.isError, JSON.stringify(replies[0]));
|
||||
assert.deepEqual(JSON.parse(result.content[0].text), { output: "all quiet", ok: true });
|
||||
});
|
||||
|
||||
test("a node-scoped seat's verb is asked of the machine named, and refused without one", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const credential = await aMeshAndACredential(t);
|
||||
const replies = await driving(credential, [
|
||||
{ jsonrpc: "2.0", id: 1, method: "initialize", params: {} },
|
||||
{ jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "node-dns-resolver.lookup", arguments: { node: "anchor" } } },
|
||||
{ jsonrpc: "2.0", id: 3, method: "tools/call", params: { name: "node-dns-resolver.lookup", arguments: {} } },
|
||||
]);
|
||||
const byId = new Map(replies.map((r) => [r.id, r]));
|
||||
const answered = byId.get(2)!.result;
|
||||
assert.ok(!answered.isError, JSON.stringify(answered));
|
||||
assert.match(answered.content[0].text, /"machine": "anchor"/, "the machine's holder answered");
|
||||
assert.match(byId.get(3)!.error?.message ?? JSON.stringify(byId.get(3)), /name the machine/);
|
||||
});
|
||||
@@ -1,255 +0,0 @@
|
||||
/**
|
||||
* The mesh issues an assignment's subjects, and a runtime serves what it is issued (novox/hq ADR
|
||||
* 0160). A runtime derives one address for itself — `mesh.assignment.<node>.<module>` — reads the
|
||||
* membership there, serves exactly what it says, and re-serves when a new one arrives. Against a
|
||||
* real bus with JetStream, because the membership is a direct get on a stream.
|
||||
*
|
||||
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
|
||||
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/membership.test.ts
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { connect, StringCodec } from "nats";
|
||||
|
||||
import { resetTools } from "@novox/mesh-sdk/tools";
|
||||
|
||||
import { connectNats, membershipSubject } from "../dist/broker-nats.js";
|
||||
import { callTool, subjectListed } from "../dist/client.js";
|
||||
import { runTools } from "../dist/runtime.js";
|
||||
|
||||
const url = process.env.MESH_TEST_NATS;
|
||||
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
|
||||
const sc = StringCodec();
|
||||
|
||||
/** The ASSIGNMENTS stream as the controller asserts it: last-per-subject, readable by direct get. */
|
||||
async function anAssignmentsStream() {
|
||||
const nc = await connect({ servers: url! });
|
||||
const jsm = await nc.jetstreamManager();
|
||||
try {
|
||||
await jsm.streams.delete("ASSIGNMENTS");
|
||||
} catch {
|
||||
// none yet
|
||||
}
|
||||
await jsm.streams.add({
|
||||
name: "ASSIGNMENTS",
|
||||
subjects: ["mesh.assignment.>"],
|
||||
max_msgs_per_subject: 1,
|
||||
allow_direct: true,
|
||||
} as never);
|
||||
return {
|
||||
async issue(m: object & { node: string; module: string }) {
|
||||
await nc.jetstream().publish(membershipSubject(m.node, m.module), sc.encode(JSON.stringify(m)));
|
||||
},
|
||||
async close() {
|
||||
await nc.close();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
test("a runtime serves exactly the subjects it is issued, and the listing carries them", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const stream = await anAssignmentsStream();
|
||||
// The mesh placed shop on two machines that are not interchangeable: each is served by name only.
|
||||
await stream.issue({
|
||||
node: "anchor",
|
||||
module: "shop",
|
||||
serves: [{ subject: "mesh.mod.shop.tool.{tool}.anchor" }],
|
||||
emits: "mesh.mod.shop.event.{event}",
|
||||
tools: "mesh.mod.shop.tool.tools",
|
||||
});
|
||||
const shop = await connectNats({ url, node: "anchor", module: "shop" });
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
let stop = () => {};
|
||||
try {
|
||||
stop = await runTools({ broker: shop, moduleEntrypoints: [fixture("shop-tools.mjs")] });
|
||||
assert.equal(shop.membership()?.module, "shop", "the runtime read what the mesh issued");
|
||||
// By name it answers; the plain subject was not issued, so nothing serves it.
|
||||
const named = await callTool(asker, "shop.price@anchor", { of: "a hat" });
|
||||
assert.deepEqual(named.result, { of: "a hat", cost: 12 });
|
||||
assert.equal(named.node, "anchor");
|
||||
await assert.rejects(callTool(asker, "shop.price", {}), /no responders|503/i, "a subject not issued is not served");
|
||||
// The tools answer says where each is served, and the client composes nothing.
|
||||
const tools = await asker.request<Record<string, never>, { tools: { name: string; subjects?: string[] }[] }>(
|
||||
"shop.tools",
|
||||
{},
|
||||
);
|
||||
assert.deepEqual(tools.tools.find((x) => x.name === "price")?.subjects, ["mesh.mod.shop.tool.price.anchor"]);
|
||||
const listing = { tools: [{ module: "shop", name: "price", subjects: ["mesh.mod.shop.tool.price.anchor"] }], notAnswering: [] };
|
||||
assert.equal(subjectListed("shop.price", "anchor", listing as never), "mesh.mod.shop.tool.price.anchor");
|
||||
assert.equal(subjectListed("shop.price", "elsewhere", listing as never), undefined);
|
||||
|
||||
// The mesh re-issues the membership with the plain subject too (the modules became
|
||||
// interchangeable); the runtime re-serves on it without a restart.
|
||||
await stream.issue({
|
||||
node: "anchor",
|
||||
module: "shop",
|
||||
serves: [{ subject: "mesh.mod.shop.tool.{tool}", queue: "serve.shop" }, { subject: "mesh.mod.shop.tool.{tool}.anchor" }],
|
||||
emits: "mesh.mod.shop.event.{event}",
|
||||
tools: "mesh.mod.shop.tool.tools",
|
||||
});
|
||||
await new Promise((r) => setTimeout(r, 300));
|
||||
const plain = await callTool(asker, "shop.price", { of: "a coat" });
|
||||
assert.deepEqual(plain.result, { of: "a coat", cost: 12 });
|
||||
assert.equal(plain.node, "anchor");
|
||||
} finally {
|
||||
stop();
|
||||
await asker.close();
|
||||
await shop.close();
|
||||
await stream.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("a seat's verbs are implemented under the seat's name, served where issued, never listed as the module's", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const stream = await anAssignmentsStream();
|
||||
await stream.issue({
|
||||
node: "anchor",
|
||||
module: "postgres",
|
||||
serves: [{ subject: "mesh.mod.postgres.tool.{tool}", queue: "serve.postgres" }, { subject: "mesh.mod.postgres.tool.{tool}.anchor" }],
|
||||
seats: [{ seat: "mesh-store", verb: "databases", subject: "mesh.seat.mesh-store.tool.databases" }],
|
||||
emits: "mesh.mod.postgres.event.{event}",
|
||||
tools: "mesh.mod.postgres.tool.tools",
|
||||
});
|
||||
const credential = { url, node: "anchor", module: "postgres", claims: [{ seat: "mesh-store", scope: "mesh", serves: ["databases", "query"] }] };
|
||||
const pg = await connectNats(credential);
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
let stop = () => {};
|
||||
try {
|
||||
stop = await runTools({ broker: pg, credential, moduleEntrypoints: [fixture("store-seat.mjs")] });
|
||||
// The module's own `databases` and the seat's are two tools: postgres's on its subject, the
|
||||
// store's on the seat's, each answering as itself.
|
||||
const own = await callTool(asker, "postgres.databases", {});
|
||||
assert.deepEqual(own.result, { software: "postgres" });
|
||||
const seat = await callTool(asker, "seat:mesh-store.databases", {});
|
||||
assert.deepEqual(seat.result, { seat: "mesh-store" });
|
||||
assert.equal(seat.node, "anchor");
|
||||
// What the seat does not promise — creating a database — is postgres's tool and not the store's.
|
||||
await assert.rejects(callTool(asker, "seat:mesh-store.postgres_create_database", {}), /no responders|503/i);
|
||||
// And the module's `tools` lists only postgres's own, never the seat's implementation.
|
||||
const tools = await asker.request<Record<string, never>, { module: string; tools: { name: string }[] }>("postgres.tools", {});
|
||||
assert.deepEqual(tools.tools.map((x) => x.name).sort(), ["databases", "postgres_create_database"]);
|
||||
await assert.rejects(asker.request("mesh-store.tools", {}), /no responders|503/i, "a seat is not a module with a tools verb");
|
||||
} finally {
|
||||
stop();
|
||||
await asker.close();
|
||||
await pg.close();
|
||||
await stream.close();
|
||||
}
|
||||
});
|
||||
|
||||
// novox/hq issue 218: the store seat is claimed by every machine running postgres and held by one.
|
||||
// Where the mesh issued a membership without the seat, the claim in the credential serves nothing:
|
||||
// the module's own tools answer, the seat's verbs do not, and the runtime does not announce them.
|
||||
test("a claimant the membership does not make the holder serves none of the seat's verbs", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const stream = await anAssignmentsStream();
|
||||
await stream.issue({
|
||||
node: "elsewhere",
|
||||
module: "postgres",
|
||||
serves: [{ subject: "mesh.mod.postgres.tool.{tool}.elsewhere" }],
|
||||
emits: "mesh.mod.postgres.event.{event}",
|
||||
tools: "mesh.mod.postgres.tool.tools",
|
||||
});
|
||||
const credential = { url, node: "elsewhere", module: "postgres", claims: [{ seat: "mesh-store", scope: "mesh", serves: ["databases", "query"] }] };
|
||||
const pg = await connectNats(credential);
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
let stop = () => {};
|
||||
try {
|
||||
stop = await runTools({ broker: pg, credential, moduleEntrypoints: [fixture("store-claimant.mjs")] });
|
||||
assert.deepEqual((await callTool(asker, "postgres.databases@elsewhere", {})).result, { software: "postgres" });
|
||||
await assert.rejects(callTool(asker, "seat:mesh-store.databases", {}), /no responders|503/i, "the store is not held here");
|
||||
} finally {
|
||||
stop();
|
||||
await asker.close();
|
||||
await pg.close();
|
||||
await stream.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("a module named like its seat registers once, and answers as the module and as the seat", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const stream = await anAssignmentsStream();
|
||||
await stream.issue({
|
||||
node: "anchor",
|
||||
module: "shop",
|
||||
serves: [{ subject: "mesh.mod.shop.tool.{tool}", queue: "serve.shop" }, { subject: "mesh.mod.shop.tool.{tool}.anchor" }],
|
||||
seats: [{ seat: "shop", verb: "price", subject: "mesh.seat.shop.tool.price" }],
|
||||
emits: "mesh.mod.shop.event.{event}",
|
||||
tools: "mesh.mod.shop.tool.tools",
|
||||
});
|
||||
const credential = { url, node: "anchor", module: "shop", claims: [{ seat: "shop", scope: "mesh", serves: ["price"] }] };
|
||||
const shop = await connectNats(credential);
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
let stop = () => {};
|
||||
try {
|
||||
stop = await runTools({ broker: shop, credential, moduleEntrypoints: [fixture("shop-seat.mjs")] });
|
||||
assert.deepEqual((await callTool(asker, "shop.price", { of: "a hat" })).result, { of: "a hat", cost: 12 });
|
||||
assert.deepEqual((await callTool(asker, "seat:shop.price", { of: "a hat" })).result, { of: "a hat", cost: 12 });
|
||||
const tools = await asker.request<Record<string, never>, { tools: { name: string }[] }>("shop.tools", {});
|
||||
assert.deepEqual(tools.tools.map((x) => x.name), ["price", "refund"]);
|
||||
} finally {
|
||||
stop();
|
||||
await asker.close();
|
||||
await shop.close();
|
||||
await stream.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("a mesh that has issued nothing yet gets the derived shape, and says so", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
const stream = await anAssignmentsStream();
|
||||
const said: string[] = [];
|
||||
const log = console.log;
|
||||
console.log = (...a: unknown[]) => said.push(a.join(" "));
|
||||
let lone;
|
||||
try {
|
||||
lone = await connectNats({ url, node: "home-server", module: "lone" });
|
||||
} finally {
|
||||
console.log = log;
|
||||
}
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
try {
|
||||
assert.equal(lone.membership(), undefined);
|
||||
assert.ok(said.some((s) => /no membership issued for lone on home-server/.test(s)), said.join("\n"));
|
||||
await lone.handle("ping", async () => ({ pong: true }));
|
||||
assert.deepEqual((await callTool(asker, "lone.ping", {})).result, { pong: true });
|
||||
assert.deepEqual((await callTool(asker, "lone.ping@home-server", {})).result, { pong: true });
|
||||
} finally {
|
||||
await asker.close();
|
||||
await lone.close();
|
||||
await stream.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A module that implements a seat its credential does not (yet) claim is not served for it and does
|
||||
// not fall over either: the runtime says so and serves the module's own tools.
|
||||
test("a registration under a seat the credential does not claim is said and skipped, not fatal", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const stream = await anAssignmentsStream();
|
||||
const credential = { url, node: "anchor", module: "postgres" };
|
||||
const pg = await connectNats(credential);
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
const said: string[] = [];
|
||||
const log = console.log;
|
||||
console.log = (...a: unknown[]) => said.push(a.join(" "));
|
||||
let stop = () => {};
|
||||
try {
|
||||
stop = await runTools({ broker: pg, credential, moduleEntrypoints: [fixture("store-seat-unclaimed.mjs")] });
|
||||
console.log = log;
|
||||
assert.ok(said.some((s) => /registers tools under "mesh-store".*not served/.test(s)), said.join("\n"));
|
||||
assert.deepEqual((await callTool(asker, "postgres.databases", {})).result, { software: "postgres" });
|
||||
await assert.rejects(callTool(asker, "seat:mesh-store.databases", {}), /no responders|503/i);
|
||||
} finally {
|
||||
console.log = log;
|
||||
stop();
|
||||
await asker.close();
|
||||
await pg.close();
|
||||
await stream.close();
|
||||
}
|
||||
});
|
||||
@@ -1,350 +0,0 @@
|
||||
/**
|
||||
* One runtime per node serves every assigned module's tools (novox/hq ADR 0175, to-be 38 WP1). The
|
||||
* node's runtime is handed a list of modules and their bundles on the node's credential; it reads
|
||||
* one membership per module, serves each module's tools on that module's subjects and each held
|
||||
* seat's verbs on the seat's, names a bundle that fails to load without dropping the others, and
|
||||
* re-serves a module whose membership is re-issued mid-run. Against a real bus with JetStream.
|
||||
*
|
||||
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
|
||||
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/node-runtime.test.ts
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { cpSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { connect, StringCodec } from "nats";
|
||||
|
||||
import { resetTools } from "@novox/mesh-sdk/tools";
|
||||
|
||||
import { connectNats, membershipSubject } from "../dist/broker-nats.js";
|
||||
import { callTool, toolsOn } from "../dist/client.js";
|
||||
import { servedModulesFrom } from "../dist/main.js";
|
||||
import { runTools, takeToolEnvs } from "../dist/runtime.js";
|
||||
|
||||
const url = process.env.MESH_TEST_NATS;
|
||||
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
|
||||
const sc = StringCodec();
|
||||
|
||||
/** The controller's job, done by hand: the ASSIGNMENTS stream (last-per-subject, direct get) and an
|
||||
* EVENTS stream for what a tool emits. */
|
||||
async function aMesh() {
|
||||
const nc = await connect({ servers: url! });
|
||||
const jsm = await nc.jetstreamManager();
|
||||
for (const name of ["ASSIGNMENTS", "EVENTS"]) {
|
||||
try {
|
||||
await jsm.streams.delete(name);
|
||||
} catch {
|
||||
// none yet
|
||||
}
|
||||
}
|
||||
await jsm.streams.add({ name: "ASSIGNMENTS", subjects: ["mesh.assignment.>"], max_msgs_per_subject: 1, allow_direct: true } as never);
|
||||
await jsm.streams.add({ name: "EVENTS", subjects: ["mesh.mod.*.event.>"] });
|
||||
return {
|
||||
async issue(m: object & { node: string; module: string }) {
|
||||
await nc.jetstream().publish(membershipSubject(m.node, m.module), sc.encode(JSON.stringify(m)));
|
||||
},
|
||||
/** The next subject an event lands on, under a pattern. */
|
||||
nextEvent(pattern: string): Promise<string> {
|
||||
const sub = nc.subscribe(pattern, { max: 1 });
|
||||
return (async () => {
|
||||
for await (const m of sub) return m.subject;
|
||||
throw new Error("no event");
|
||||
})();
|
||||
},
|
||||
async close() {
|
||||
await nc.close();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** A membership as the controller issues one on a machine, with the module's own subject when it
|
||||
* answers for the module anywhere, and the verbs of the node seats it holds. */
|
||||
function membershipOf(module: string, node: string, opts: { plain?: boolean; seats?: Record<string, string[]> } = {}) {
|
||||
const own = `mesh.mod.${module}`;
|
||||
const serves: { subject: string; queue?: string }[] = [{ subject: `${own}.tool.{tool}.${node}` }];
|
||||
if (opts.plain) serves.push({ subject: `${own}.tool.{tool}`, queue: `serve.${module}` });
|
||||
const seats = Object.entries(opts.seats ?? {}).flatMap(([seat, verbs]) =>
|
||||
verbs.map((verb) => ({ seat, verb, subject: `mesh.seat.${seat}.tool.${verb}.${node}` })));
|
||||
return { node, module, serves, seats, emits: `${own}.event.{event}`, tools: `${own}.tool.tools` };
|
||||
}
|
||||
|
||||
/** Wait for something to be served: the bus answers "no responders" at once until it is. */
|
||||
async function until<T>(attempt: () => Promise<T>, tries = 50): Promise<T> {
|
||||
for (let i = 0; ; i++) {
|
||||
try {
|
||||
return await attempt();
|
||||
} catch (e) {
|
||||
if (i >= tries) throw e;
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
test("MESH_TOOL_MODULES names modules and their entrypoints; a bare path is the credential's own module's", () => {
|
||||
const have = servedModulesFrom(" alpha=/a/tools/index.js, beta=/b/one.js ,beta=/b/two.js, /mine/index.js ,node-tools=/own/x.js", "node-tools");
|
||||
assert.deepEqual(have.serves, [
|
||||
{ module: "alpha", entrypoints: ["/a/tools/index.js"] },
|
||||
{ module: "beta", entrypoints: ["/b/one.js", "/b/two.js"] },
|
||||
]);
|
||||
// The credential's own module, named or bare, is the one-module form either way.
|
||||
assert.deepEqual(have.moduleEntrypoints, ["/mine/index.js", "/own/x.js"]);
|
||||
assert.throws(() => servedModulesFrom("=/nothing.js", undefined), /neither <module>=<entrypoint>/);
|
||||
assert.deepEqual(servedModulesFrom("", "x"), { serves: [], moduleEntrypoints: [] });
|
||||
});
|
||||
|
||||
test("the node's runtime serves five modules' bundles on one credential — two of them launched, one broken — and follows a re-issued membership", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const mesh = await aMesh();
|
||||
// Three modules assigned to the machine: alpha answers for itself anywhere, beta only here and
|
||||
// holds the node-shelf seat, gamma's bundle is broken.
|
||||
await mesh.issue(membershipOf("alpha", "anchor", { plain: true }));
|
||||
await mesh.issue(membershipOf("beta", "anchor", { seats: { "node-shelf": ["list", "clear"] } }));
|
||||
await mesh.issue(membershipOf("gamma", "anchor"));
|
||||
// Two more, launched rather than loaded (ADR 0188): delta is Python and holds the node-lamp seat;
|
||||
// epsilon is TypeScript written against the protocol and marked executable.
|
||||
await mesh.issue(membershipOf("delta", "anchor", { seats: { "node-lamp": ["on"] } }));
|
||||
await mesh.issue(membershipOf("epsilon", "anchor"));
|
||||
// The node's credential: the runtime module's name, no claims (seats come from the memberships).
|
||||
const credential = { url, node: "anchor", module: "node-tools" };
|
||||
const nodeTools = await connectNats(credential);
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
const said: string[] = [];
|
||||
const log = console.log;
|
||||
console.log = (...a: unknown[]) => said.push(a.join(" "));
|
||||
let stop = () => {};
|
||||
try {
|
||||
process.env.MESH_OPERATOR_ACCOUNT = "somebody";
|
||||
process.env.MESH_OPERATOR_HOME = "/home/somebody";
|
||||
stop = await runTools({
|
||||
broker: nodeTools,
|
||||
credential,
|
||||
serves: [
|
||||
{ module: "alpha", entrypoints: [fixture("many-alpha.serve.mjs")] },
|
||||
{ module: "beta", entrypoints: [fixture("many-beta.serve.mjs")] },
|
||||
{ module: "gamma", entrypoints: [fixture("many-broken.serve.mjs")] },
|
||||
{ module: "delta", entrypoints: [fixture("many-delta.py")] },
|
||||
{ module: "epsilon", entrypoints: [fixture("many-epsilon.mjs")] },
|
||||
],
|
||||
});
|
||||
console.log = log;
|
||||
assert.deepEqual(nodeTools.serving().sort(), ["alpha", "beta", "delta", "epsilon", "gamma", "node-tools"]);
|
||||
assert.ok(said.some((s) => /the operator's account here is somebody \(home \/home\/somebody\)/.test(s)), said.join("\n"));
|
||||
assert.ok(said.some((s) => /gamma's bundle .*many-broken\.serve\.mjs failed to load: gamma's bundle exited \(1\): Error: gamma's bundle cannot find its client; its tools are not served here/.test(s)), said.join("\n"));
|
||||
assert.ok(said.some((s) => /serving 8 tool\(s\) for 5 module\(s\): alpha\.one, alpha\.two, beta\.three, beta\.four, beta\.five, delta\.greet, delta\.die, epsilon\.seven; not serving gamma/.test(s)), said.join("\n"));
|
||||
|
||||
// Five tools answer, each where its module's membership says: alpha anywhere and here, beta here only.
|
||||
assert.deepEqual((await callTool(asker, "alpha.one", {})).result, { alpha: 1 });
|
||||
assert.deepEqual((await callTool(asker, "alpha.one@anchor", {})).result, { alpha: 1 });
|
||||
assert.deepEqual((await callTool(asker, "beta.three@anchor", {})).result, { beta: 3 });
|
||||
assert.deepEqual((await callTool(asker, "beta.four@anchor", {})).result, { beta: 4 });
|
||||
assert.deepEqual((await callTool(asker, "beta.five@anchor", {})).result, { beta: 5 });
|
||||
await assert.rejects(callTool(asker, "beta.three", {}), /no responders|503/i, "beta was not issued the module's plain subject");
|
||||
|
||||
// Two seat verbs answer on the seat's subjects, held by beta.
|
||||
assert.deepEqual((await callTool(asker, "seat:node-shelf.list@anchor", {})).result, { shelf: ["a", "b"] });
|
||||
assert.deepEqual((await callTool(asker, "seat:node-shelf.clear@anchor", {})).result, { cleared: true });
|
||||
|
||||
// A bundle in another language answers the same way, its seat verb among them; so does a
|
||||
// TypeScript bundle served through the protocol rather than imported.
|
||||
assert.deepEqual((await callTool(asker, "delta.greet@anchor", { who: "mesh" })).result, { greeting: "hello mesh", language: "python" });
|
||||
assert.deepEqual((await callTool(asker, "seat:node-lamp.on@anchor", {})).result, { on: true, language: "python" });
|
||||
assert.deepEqual((await callTool(asker, "epsilon.seven@anchor", {})).result, { epsilon: 7, via: "stdio" });
|
||||
// A child that exits mid-call tells the caller so and is started again on the next call.
|
||||
await assert.rejects(callTool(asker, "delta.die@anchor", {}), /delta's bundle exited \(3\)/);
|
||||
assert.deepEqual((await callTool(asker, "delta.greet@anchor", {})).result, { greeting: "hello world", language: "python" });
|
||||
|
||||
// A tool that emits does so as its module, not as the runtime.
|
||||
const landed = mesh.nextEvent("mesh.mod.*.event.>");
|
||||
assert.deepEqual((await callTool(asker, "alpha.two", {})).result, { alpha: 2 });
|
||||
assert.equal(await landed, "mesh.mod.alpha.event.happened");
|
||||
|
||||
// `tools` answers for each: what alpha and beta serve, and why gamma serves nothing.
|
||||
const gamma = await asker.request<Record<string, never>, { module: string; tools: unknown[]; failed?: string }>("gamma.tools@anchor", {});
|
||||
assert.deepEqual(gamma, { module: "gamma", tools: [], failed: "gamma's bundle exited (1): Error: gamma's bundle cannot find its client" });
|
||||
const beta = await asker.request<Record<string, never>, { tools: { name: string; subjects?: string[] }[] }>("beta.tools@anchor", {});
|
||||
assert.deepEqual(beta.tools.map((x) => x.name), ["three", "four", "five"]);
|
||||
assert.deepEqual(beta.tools[0]!.subjects, ["mesh.mod.beta.tool.three.anchor"]);
|
||||
|
||||
// And discovery says so, with the reason, beside the modules that answered.
|
||||
const catalogue = await connectNats({ url, module: "mesh-catalog" });
|
||||
await catalogue.handle("catalog_modules", async () => ({ modules: [{ module: "alpha" }, { module: "beta" }, { module: "gamma" }, { module: "delta" }, { module: "epsilon" }] }));
|
||||
try {
|
||||
const have = await toolsOn(asker);
|
||||
assert.deepEqual(have.tools.map((x) => `${x.module}.${x.name}`), ["alpha.one", "alpha.two", "beta.five", "beta.four", "beta.three", "delta.die", "delta.greet", "epsilon.seven"]);
|
||||
assert.deepEqual(have.notAnswering, ["gamma (its tools bundle failed to load: gamma's bundle exited (1): Error: gamma's bundle cannot find its client)", "mesh-controller (seat)"]);
|
||||
} finally {
|
||||
await catalogue.close();
|
||||
}
|
||||
|
||||
// The mesh re-issues beta's membership mid-run — now answering for the module anywhere — and
|
||||
// the runtime serves the new subject without a restart.
|
||||
await mesh.issue(membershipOf("beta", "anchor", { plain: true, seats: { "node-shelf": ["list", "clear"] } }));
|
||||
assert.deepEqual((await until(() => callTool(asker, "beta.three", {}))).result, { beta: 3 });
|
||||
assert.deepEqual((await callTool(asker, "seat:node-shelf.list@anchor", {})).result, { shelf: ["a", "b"] });
|
||||
} finally {
|
||||
console.log = log;
|
||||
delete process.env.MESH_OPERATOR_ACCOUNT;
|
||||
delete process.env.MESH_OPERATOR_HOME;
|
||||
stop();
|
||||
await asker.close();
|
||||
await nodeTools.close();
|
||||
await mesh.close();
|
||||
resetTools();
|
||||
}
|
||||
});
|
||||
|
||||
test("a bundle carrying its own copy of the SDK registers into the runtime's registry, and its tools are served (issue 209)", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
let dir = "";
|
||||
let stop = () => {};
|
||||
const closing: Array<() => Promise<void>> = [];
|
||||
const said: string[] = [];
|
||||
const log = console.log;
|
||||
try {
|
||||
// A bundle as the toolchain packs one: its compiled entrypoint, a package.json saying ES modules,
|
||||
// and its dependencies copied in — the SDK among them, a second copy beside the runtime's own,
|
||||
// and a dependency of the bundle's own that the runtime does not carry.
|
||||
dir = mkdtempSync(join(tmpdir(), "mesh-bundle-"));
|
||||
const sdk = realpathSync(fileURLToPath(new URL("../node_modules/@novox/mesh-sdk/", import.meta.url)));
|
||||
cpSync(sdk, join(dir, "node_modules", "@novox", "mesh-sdk"), { recursive: true });
|
||||
mkdirSync(join(dir, "node_modules", "zeta-flavour"), { recursive: true });
|
||||
writeFileSync(join(dir, "node_modules", "zeta-flavour", "package.json"), '{"name":"zeta-flavour","type":"module","main":"index.js"}\n');
|
||||
writeFileSync(join(dir, "node_modules", "zeta-flavour", "index.js"), 'export const flavour = "the bundle\'s own";\n');
|
||||
writeFileSync(join(dir, "package.json"), '{"type":"module","private":true}\n');
|
||||
writeFileSync(join(dir, "index.serve.mjs"),
|
||||
'#!/usr/bin/env node\nimport { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";\nawait import("./index.js");\nawait serveRegisteredOverStdio();\n', { mode: 0o755 });
|
||||
writeFileSync(join(dir, "index.js"),
|
||||
'import { registerModuleTools } from "@novox/mesh-sdk/tools";\n' +
|
||||
'import { flavour } from "zeta-flavour";\n' +
|
||||
'registerModuleTools("zeta", () => [{ name: "probe", description: "answers", input: {}, run: async () => ({ zeta: true, flavour }) }]);\n');
|
||||
const mesh = await aMesh();
|
||||
closing.push(() => mesh.close());
|
||||
await mesh.issue(membershipOf("zeta", "anchor"));
|
||||
const credential = { url, node: "anchor", module: "node-tools" };
|
||||
const nodeTools = await connectNats(credential);
|
||||
closing.push(() => nodeTools.close());
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
closing.push(() => asker.close());
|
||||
console.log = (...a: unknown[]) => said.push(a.join(" "));
|
||||
stop = await runTools({ broker: nodeTools, credential, serves: [{ module: "zeta", entrypoints: [join(dir, "index.serve.mjs")] }] });
|
||||
console.log = log;
|
||||
assert.ok(said.some((s) => /serving 1 tool\(s\) for 1 module\(s\): zeta\.probe/.test(s)), said.join("\n"));
|
||||
// The SDK is the runtime's (the registration arrived); the bundle's other dependency is its own.
|
||||
assert.deepEqual((await callTool(asker, "zeta.probe@anchor", {})).result, { zeta: true, flavour: "the bundle's own" });
|
||||
} finally {
|
||||
console.log = log;
|
||||
stop();
|
||||
for (const close of closing.reverse()) await close();
|
||||
if (dir) rmSync(dir, { recursive: true, force: true });
|
||||
resetTools();
|
||||
}
|
||||
});
|
||||
|
||||
test("each bundle is given its own environment and none of another's, imported or launched (ADR 0192)", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const mesh = await aMesh();
|
||||
for (const m of ["gamma", "delta", "zeta"]) await mesh.issue(membershipOf(m, "anchor"));
|
||||
const credential = { url, node: "anchor", module: "node-tools" };
|
||||
const nodeTools = await connectNats(credential);
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
const log = console.log;
|
||||
let stop = () => {};
|
||||
const before = process.env.MESH_TOOL_ENV;
|
||||
try {
|
||||
process.env.MESH_OPERATOR_ACCOUNT = "somebody";
|
||||
process.env.MESH_TOOL_ENV = JSON.stringify({
|
||||
gamma: { GAMMA_CONFIG_FILE: "/var/lib/mesh/gamma/config.json" },
|
||||
delta: { DELTA_TOKEN_FILE: "/var/lib/mesh/delta/token" },
|
||||
zeta: { ZETA_URL: "http://127.0.0.1:3000" },
|
||||
});
|
||||
const envs = takeToolEnvs();
|
||||
assert.equal(process.env.MESH_TOOL_ENV, undefined, "the composed environments were left in the process's");
|
||||
console.log = () => {};
|
||||
stop = await runTools({
|
||||
broker: nodeTools, credential, envs,
|
||||
serves: [
|
||||
{ module: "gamma", entrypoints: [fixture("env-gamma.serve.mjs")] },
|
||||
{ module: "delta", entrypoints: [fixture("env-delta.serve.mjs")] },
|
||||
{ module: "zeta", entrypoints: [fixture("env-zeta.mjs")] },
|
||||
],
|
||||
});
|
||||
console.log = log;
|
||||
assert.deepEqual((await callTool(asker, "gamma.given@anchor", {})).result,
|
||||
{ mine: "/var/lib/mesh/gamma/config.json", theirs: null, runtime: "somebody", composed: null });
|
||||
assert.deepEqual((await callTool(asker, "delta.given@anchor", {})).result,
|
||||
{ mine: "/var/lib/mesh/delta/token", theirs: null });
|
||||
assert.deepEqual((await callTool(asker, "zeta.given@anchor", {})).result,
|
||||
{ mine: "http://127.0.0.1:3000", theirs: null, composed: null });
|
||||
} finally {
|
||||
console.log = log;
|
||||
if (before === undefined) delete process.env.MESH_TOOL_ENV; else process.env.MESH_TOOL_ENV = before;
|
||||
delete process.env.MESH_OPERATOR_ACCOUNT;
|
||||
stop();
|
||||
await asker.close();
|
||||
await nodeTools.close();
|
||||
await mesh.close();
|
||||
resetTools();
|
||||
}
|
||||
});
|
||||
|
||||
test("a launched bundle is told the module it serves, so its seat's verbs stay the seat's (ADR 0193)", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const mesh = await aMesh();
|
||||
await mesh.issue(membershipOf("theta", "anchor", { seats: { "node-shelf": ["list"] } }));
|
||||
const credential = { url, node: "anchor", module: "node-tools" };
|
||||
const nodeTools = await connectNats(credential);
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
const log = console.log;
|
||||
let stop = () => {};
|
||||
try {
|
||||
console.log = () => {};
|
||||
stop = await runTools({ broker: nodeTools, credential, envs: new Map([["theta", { THETA_WORD: "given" }]]),
|
||||
serves: [{ module: "theta", entrypoints: [fixture("served-seat-first.mjs")] }] });
|
||||
console.log = log;
|
||||
assert.deepEqual((await callTool(asker, "theta.own@anchor", {})).result, { theta: "given" });
|
||||
assert.deepEqual((await callTool(asker, "seat:node-shelf.list@anchor", {})).result, { shelf: ["x"] });
|
||||
} finally {
|
||||
console.log = log;
|
||||
stop();
|
||||
await asker.close();
|
||||
await nodeTools.close();
|
||||
await mesh.close();
|
||||
resetTools();
|
||||
}
|
||||
});
|
||||
|
||||
test("an entrypoint that is not executable is refused by name, and the others serve (ADR 0193)", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const mesh = await aMesh();
|
||||
for (const m of ["alpha", "plain"]) await mesh.issue(membershipOf(m, "anchor"));
|
||||
const credential = { url, node: "anchor", module: "node-tools" };
|
||||
const nodeTools = await connectNats(credential);
|
||||
const asker = await connectNats({ url, module: "console", node: "workstation" });
|
||||
const said: string[] = [];
|
||||
const log = console.log;
|
||||
let stop = () => {};
|
||||
try {
|
||||
console.log = (...a: unknown[]) => said.push(a.join(" "));
|
||||
stop = await runTools({ broker: nodeTools, credential, serves: [
|
||||
{ module: "alpha", entrypoints: [fixture("many-alpha.serve.mjs")] },
|
||||
{ module: "plain", entrypoints: [fixture("many-alpha.mjs")] },
|
||||
] });
|
||||
console.log = log;
|
||||
assert.ok(said.some((s) => /plain's bundle .*many-alpha\.mjs failed to load: .* is not executable; a bundle the runtime serves is started, never imported/.test(s)), said.join("\n"));
|
||||
assert.deepEqual((await callTool(asker, "alpha.one@anchor", {})).result, { alpha: 1 });
|
||||
} finally {
|
||||
console.log = log;
|
||||
stop();
|
||||
await asker.close();
|
||||
await nodeTools.close();
|
||||
await mesh.close();
|
||||
resetTools();
|
||||
}
|
||||
});
|
||||
@@ -1,63 +0,0 @@
|
||||
/**
|
||||
* The runtime as the node-tools module (novox/hq ADR 0175 §6, to-be 38 WP3): started the way the
|
||||
* host starts it — `main.js` with the node's credential and MESH_TOOL_MODULES — it serves the bundles
|
||||
* AND answers MCP on loopback as the console, through which a tool it serves can be called.
|
||||
*
|
||||
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
|
||||
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/node-tools-serve.test.ts
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { mkdtemp, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import { connectNats } from "../dist/broker-nats.js";
|
||||
|
||||
const url = process.env.MESH_TEST_NATS;
|
||||
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
|
||||
|
||||
async function post(endpoint: string, body: unknown): Promise<any> {
|
||||
const res = await fetch(endpoint, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body) });
|
||||
return res.json();
|
||||
}
|
||||
|
||||
test("as node-tools, serve loads the bundles and is the console on loopback", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
// The catalogue, for discovery; the node credential names the runtime module and no claims.
|
||||
const catalogue = await connectNats({ url, module: "mesh-catalog" });
|
||||
await catalogue.handle("catalog_modules", async () => ({ modules: [{ module: "alpha" }] }));
|
||||
t.after(() => catalogue.close());
|
||||
const dir = await mkdtemp("/tmp/node-tools-");
|
||||
const credential = join(dir, "broker");
|
||||
await writeFile(credential, JSON.stringify({ url, node: "desk", module: "node-tools", user: "desk.node-tools", password: "x" }));
|
||||
const child: ChildProcess = spawn(process.execPath, ["dist/main.js"], {
|
||||
env: {
|
||||
...process.env,
|
||||
MESH_BROKER_FILE: credential,
|
||||
MESH_TOOL_MODULES: `alpha=${fixture("many-alpha.serve.mjs")}`,
|
||||
MESH_CONSOLE_LISTEN: "127.0.0.1:0",
|
||||
},
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
t.after(() => {
|
||||
child.kill("SIGTERM");
|
||||
});
|
||||
const endpoint = await new Promise<string>((resolve, reject) => {
|
||||
let out = "";
|
||||
let err = "";
|
||||
child.stdout!.on("data", (d) => {
|
||||
out += d.toString();
|
||||
const m = /listening on (http:\/\/[^/]+\/mcp) as desk\.node-tools/.exec(out);
|
||||
if (m) resolve(m[1]!);
|
||||
});
|
||||
child.stderr!.on("data", (d) => (err += d.toString()));
|
||||
child.on("exit", (code) => reject(new Error(`serve exited ${code}: ${err}`)));
|
||||
});
|
||||
const listed = await post(endpoint, { jsonrpc: "2.0", id: 1, method: "tools/list" });
|
||||
assert.deepEqual(listed.result.tools.map((x: any) => x.name).filter((n: string) => n.startsWith("alpha.")), ["alpha.one", "alpha.two"]);
|
||||
// A tool the same process serves on the bus, called through the console it also is.
|
||||
const called = await post(endpoint, { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "alpha.one", arguments: { node: "desk" } } });
|
||||
assert.deepEqual(JSON.parse(called.result.content[0].text), { alpha: 1 });
|
||||
});
|
||||
@@ -1,80 +0,0 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { fatalBrokerReason, PinMismatchError, topicMatches } from "../src/broker-nats.ts";
|
||||
|
||||
// novox/hq issue 058 (and its review): serve mode retries a broker that is not up yet, but must
|
||||
// give up at once on a failure waiting cannot fix — otherwise a permanent fault loops for ever
|
||||
// disguised as "not reachable". This is the classifier that draws the line; the bed cannot test it
|
||||
// (it starts the consumer only after the broker is up), so it is proven here.
|
||||
|
||||
test("a broker that is not up yet is retryable, not fatal", () => {
|
||||
for (const err of [
|
||||
Object.assign(new Error("connect ECONNREFUSED 10.42.0.1:5671"), { code: "ECONNREFUSED" }),
|
||||
Object.assign(new Error("connect ETIMEDOUT"), { code: "ETIMEDOUT" }),
|
||||
Object.assign(new Error("getaddrinfo EAI_AGAIN anchor.internal"), { code: "EAI_AGAIN" }),
|
||||
new Error("timed out fetching the broker's certificate"),
|
||||
]) {
|
||||
assert.equal(fatalBrokerReason(err), null, `should retry: ${(err as Error).message}`);
|
||||
}
|
||||
});
|
||||
|
||||
test("a certificate that does not match the pin is fatal, by type not by message", () => {
|
||||
// Typed, so rewording the message cannot turn an impostor back into an infinite retry.
|
||||
assert.notEqual(fatalBrokerReason(new PinMismatchError("anything at all")), null);
|
||||
// A plain Error with pin-ish words is NOT treated as the pin case — only the type is.
|
||||
assert.equal(fatalBrokerReason(new Error("the pinned value was fine")), null);
|
||||
});
|
||||
|
||||
test("a malformed broker URL is fatal — it never parses on the next try", () => {
|
||||
assert.notEqual(fatalBrokerReason(Object.assign(new Error("Invalid URL"), { code: "ERR_INVALID_URL" })), null);
|
||||
assert.notEqual(fatalBrokerReason(new Error("Invalid URL: not-a-url")), null);
|
||||
});
|
||||
|
||||
test("a refused login is fatal — a wrong or revoked credential, not an absent broker", () => {
|
||||
// The bus refuses a login in its own words; each is final, because the next try says the same.
|
||||
for (const msg of ["Authorization Violation", "nats: user authentication expired", "Permissions Violation for Subscription to \"x\""]) {
|
||||
assert.notEqual(fatalBrokerReason(new Error(msg)), null, `should be fatal: ${msg}`);
|
||||
}
|
||||
});
|
||||
|
||||
test("a non-Error value does not crash the classifier", () => {
|
||||
assert.equal(fatalBrokerReason("just a string"), null);
|
||||
assert.equal(fatalBrokerReason(undefined), null);
|
||||
});
|
||||
|
||||
// **A module asked to prepare its state and naming nothing is a failure, not a no-op** (novox/hq
|
||||
// ADR 0135). The mesh asks this only of a module whose manifest says it prepares something, so an
|
||||
// image that names nothing was built wrong, and exiting 0 would let that version serve against a
|
||||
// state nobody shaped.
|
||||
test("preparing with nothing named fails rather than passing quietly", async () => {
|
||||
const runtime = new URL("../dist/main.js", import.meta.url).pathname;
|
||||
const ran = await new Promise<{ code: number | null; said: string }>((resolve) => {
|
||||
const child = spawn(process.execPath, [runtime, "prepare"], {
|
||||
env: { ...process.env, MESH_PREPARE: "" },
|
||||
});
|
||||
let said = "";
|
||||
child.stderr.on("data", (chunk) => (said += String(chunk)));
|
||||
child.on("close", (code) => resolve({ code, said }));
|
||||
});
|
||||
assert.notEqual(ran.code, 0, "a module that prepares nothing exited 0, so its version would serve");
|
||||
assert.match(ran.said, /MESH_PREPARE/);
|
||||
});
|
||||
|
||||
// **One durable consumer feeds one reader, however many patterns a module registers.**
|
||||
//
|
||||
// A module has exactly one consumer, so two readers of it would each take half the messages — and a
|
||||
// reader that received one its own pattern does not match acknowledges it, which is right for a
|
||||
// filter wider than anything registered and silent loss when it is another handler's. The matching is
|
||||
// therefore pure and tested as such: what a message is for is decided by the patterns registered, not
|
||||
// by which reader happened to fetch it.
|
||||
test("a message is for every pattern that matches it, and nothing else", () => {
|
||||
const registered = ["mesh-build-machine.built", "mesh-controller.built-before"];
|
||||
const matched = (key: string) => registered.filter((p) => topicMatches(p, key));
|
||||
assert.deepEqual(matched("mesh-build-machine.built"), ["mesh-build-machine.built"]);
|
||||
assert.deepEqual(matched("mesh-controller.built-before"), ["mesh-controller.built-before"]);
|
||||
// Nothing registered for it: the consumer's filter is the controller's and may be wider.
|
||||
assert.deepEqual(matched("mesh-catalog.upgraded"), []);
|
||||
// And a handler that asked for everything gets both, which is what the audit logger does.
|
||||
assert.deepEqual(["#"].filter((p) => topicMatches(p, "mesh-controller.built-before")), ["#"]);
|
||||
});
|
||||
@@ -1,68 +0,0 @@
|
||||
/**
|
||||
* A refused announcement is said and never fatal (novox/hq 04-ISSUES/217): against a bus whose
|
||||
* permissions refuse one discovery subject, the runtime's raw subscription is refused, logged, and
|
||||
* the process keeps serving — its tools still answer.
|
||||
*
|
||||
* docker run -d --rm --name t -p 14233:4222 -v $PWD/test/fixtures/refusing-nats.conf:/c.conf nats:2.10-alpine -c /c.conf
|
||||
* MESH_TEST_REFUSING_NATS=nats://127.0.0.1:14233 node --test --experimental-strip-types test/refused.test.ts
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { connectNats } from "../dist/broker-nats.js";
|
||||
import { callTool } from "../dist/client.js";
|
||||
|
||||
const url = process.env.MESH_TEST_REFUSING_NATS;
|
||||
|
||||
test("a refused discovery subscription is logged and the runtime serves on", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_REFUSING_NATS unset");
|
||||
const bus = await connectNats({ url, user: "runtime", password: "runtime", module: "alpha", node: "anchor" });
|
||||
const said: string[] = [];
|
||||
const log = console.log;
|
||||
console.log = (...a: unknown[]) => said.push(a.join(" "));
|
||||
const crashed: unknown[] = [];
|
||||
const onRejection = (e: unknown) => crashed.push(e);
|
||||
process.on("unhandledRejection", onRejection);
|
||||
try {
|
||||
(bus as unknown as { raw: (s: string, f: () => Uint8Array | undefined) => () => void }).raw("$SRV.PING.>", () => undefined);
|
||||
const stop = await bus.handle("alpha.ping", async () => ({ pong: true }));
|
||||
for (let i = 0; i < 50 && !said.some((s) => s.includes("the bus refused $SRV.PING.>")); i++) await new Promise((r) => setTimeout(r, 50));
|
||||
console.log = log;
|
||||
assert.ok(said.some((s) => /the bus refused \$SRV\.PING\.>.*serves on/.test(s)), said.join("\n"));
|
||||
assert.equal(crashed.length, 0, `the refusal escaped: ${String(crashed[0])}`);
|
||||
stop();
|
||||
} finally {
|
||||
console.log = log;
|
||||
process.off("unhandledRejection", onRejection);
|
||||
await bus.close();
|
||||
}
|
||||
});
|
||||
|
||||
// novox/hq issue 218: a tool subject the grants leave out — a seat claimed here and held elsewhere —
|
||||
// is refused, said, and the module's other tools still answer.
|
||||
test("a refused tool subscription is logged and the module's other tools answer", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_REFUSING_NATS unset");
|
||||
const bus = await connectNats({ url, user: "runtime", password: "runtime", module: "alpha", node: "anchor" });
|
||||
const asker = await connectNats({ url, user: "runtime", password: "runtime", module: "console", node: "workstation" });
|
||||
const said: string[] = [];
|
||||
const log = console.log;
|
||||
console.log = (...a: unknown[]) => said.push(a.join(" "));
|
||||
const crashed: unknown[] = [];
|
||||
const onRejection = (e: unknown) => crashed.push(e);
|
||||
process.on("unhandledRejection", onRejection);
|
||||
try {
|
||||
const refused = await bus.handleSubject!("mesh.seat.held-elsewhere.tool.databases", async () => ({ seat: true }));
|
||||
const stop = await bus.handle("alpha.ping", async () => ({ pong: true }));
|
||||
for (let i = 0; i < 50 && !said.some((s) => s.includes("the bus refused mesh.seat.held-elsewhere")); i++) await new Promise((r) => setTimeout(r, 50));
|
||||
console.log = log;
|
||||
assert.ok(said.some((s) => /the bus refused mesh\.seat\.held-elsewhere\.tool\.databases.*serves on/.test(s)), said.join("\n"));
|
||||
assert.equal(crashed.length, 0, `the refusal escaped: ${String(crashed[0])}`);
|
||||
assert.deepEqual((await callTool(asker, "alpha.ping@anchor", {})).result, { pong: true });
|
||||
stop();
|
||||
refused();
|
||||
} finally {
|
||||
console.log = log;
|
||||
process.off("unhandledRejection", onRejection);
|
||||
await asker.close();
|
||||
await bus.close();
|
||||
}
|
||||
});
|
||||
@@ -1,57 +0,0 @@
|
||||
// Round-trip the runtime's NATS client against a real server: a tool call answered, and an
|
||||
// event emitted and received with its envelope intact.
|
||||
import { connect } from "nats";
|
||||
import { connectNats } from "../dist/broker-nats.js";
|
||||
|
||||
const URL = "nats://127.0.0.1:14222";
|
||||
|
||||
// The controller's job, done by hand here: the stream and the module's durable consumer.
|
||||
const admin = await connect({ servers: URL });
|
||||
const jsm = await admin.jetstreamManager();
|
||||
await jsm.streams.add({ name: "EVENTS", subjects: ["mesh.mod.*.event.>", "mesh.seat.*.event.>"] });
|
||||
await jsm.consumers.add("EVENTS", {
|
||||
durable_name: "one_audit", ack_policy: "explicit",
|
||||
filter_subjects: ["mesh.mod.shop.event.order.placed"],
|
||||
});
|
||||
|
||||
const shop = await connectNats({ url: URL, node: "one", module: "shop" });
|
||||
const audit = await connectNats({ url: URL, node: "one", module: "audit" });
|
||||
|
||||
let failures = 0;
|
||||
const check = (ok, what) => { console.log(` ${ok ? "ok " : "FAIL"} ${what}`); if (!ok) failures++; };
|
||||
|
||||
// A tool, served and called.
|
||||
await shop.handle("price", async (body) => ({ total: body.qty * 3 }));
|
||||
const answer = await audit.request("shop.price", { qty: 4 });
|
||||
check(answer.total === 12, "a tool call is answered across two connections");
|
||||
|
||||
// A handler that throws reaches the caller as an error, not a timeout.
|
||||
await shop.handle("boom", async () => { throw new Error("no"); });
|
||||
let threw = null;
|
||||
try { await audit.request("shop.boom", {}); } catch (e) { threw = e.message; }
|
||||
check(threw === "no", "a handler that throws answers the caller instead of timing out");
|
||||
|
||||
// An event, emitted and received with its envelope intact.
|
||||
const seen = [];
|
||||
await audit.subscribe("order.placed", async (env) => { seen.push(env); });
|
||||
await shop.publish({
|
||||
key: "order.placed", node: "one", body: { id: "a1" },
|
||||
headers: { "x-event-id": "e1", "x-node": "one", "content-type": "application/json" },
|
||||
});
|
||||
await new Promise((r) => setTimeout(r, 800));
|
||||
check(seen.length === 1, `exactly one delivery (saw ${seen.length})`);
|
||||
if (seen[0]) {
|
||||
check(seen[0].key === "order.placed", "the key survives the subject round trip");
|
||||
check(seen[0].body?.id === "a1", "the body is the payload, not the whole envelope");
|
||||
check(seen[0].node === "one", "the node comes back from the headers");
|
||||
check(seen[0].headers?.["x-event-id"] === "e1", "the event id survives as a header");
|
||||
}
|
||||
|
||||
// A module cannot reach into another's namespace by naming its own event oddly.
|
||||
await shop.publish({ key: "other", node: "one", body: {}, headers: { "x-event-id": "e2" } });
|
||||
const msg = await jsm.streams.getMessage("EVENTS", { last_by_subj: "mesh.mod.shop.event.other" });
|
||||
check(!!msg, "an event lands under the emitting module's own namespace");
|
||||
|
||||
await shop.close(); await audit.close(); await admin.close();
|
||||
console.log(failures ? `\n${failures} failed` : "\nall passed");
|
||||
process.exit(failures ? 1 : 0);
|
||||
@@ -1,60 +0,0 @@
|
||||
/**
|
||||
* Every module's runtime answers `tools` for it (novox/hq ADR 0152, design 34 §3): the names,
|
||||
* descriptions and schemas from the code that answers them. Against a real bus, because the claim is
|
||||
* what a second connection gets back.
|
||||
*
|
||||
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
|
||||
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/runtime-tools.test.ts
|
||||
*/
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import { resetTools } from "@novox/mesh-sdk/tools";
|
||||
|
||||
import { connectNats } from "../dist/broker-nats.js";
|
||||
import { runTools } from "../dist/runtime.js";
|
||||
|
||||
const url = process.env.MESH_TEST_NATS;
|
||||
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
|
||||
|
||||
test("a module registering two tools answers three names, the third being what it serves", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const shop = await connectNats({ url, node: "one", module: "shop" });
|
||||
const asker = await connectNats({ url, module: "person.ada" });
|
||||
const stop = await runTools({ broker: shop, moduleEntrypoints: [fixture("shop-tools.mjs")] });
|
||||
try {
|
||||
const answer = await asker.request<Record<string, never>, { module: string; tools: { name: string; input: unknown }[] }>(
|
||||
"shop.tools",
|
||||
{},
|
||||
);
|
||||
assert.equal(answer.module, "shop");
|
||||
assert.deepEqual(answer.tools.map((x) => x.name), ["price", "refund"]);
|
||||
// The schema travels with the name: a name alone is not callable by something that has never
|
||||
// seen the mesh before.
|
||||
assert.deepEqual(answer.tools[0].input, { of: { type: "string", description: "the thing" } });
|
||||
// And the tools themselves still answer beside it.
|
||||
const priced = await asker.request<{ of: string }, { cost: number }>("shop.price", { of: "a hat" });
|
||||
assert.equal(priced.cost, 12);
|
||||
} finally {
|
||||
stop();
|
||||
await asker.close();
|
||||
await shop.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("a module naming a tool of its own `tools` is refused at load", async (t) => {
|
||||
if (!url) return t.skip("MESH_TEST_NATS unset");
|
||||
resetTools();
|
||||
const clash = await connectNats({ url, node: "one", module: "clash" });
|
||||
try {
|
||||
await assert.rejects(
|
||||
() => runTools({ broker: clash, moduleEntrypoints: [fixture("clash-tools.mjs")] }),
|
||||
/names a tool "tools"/,
|
||||
);
|
||||
} finally {
|
||||
resetTools();
|
||||
await clash.close();
|
||||
}
|
||||
});
|
||||
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "NodeNext",
|
||||
"moduleResolution": "NodeNext",
|
||||
"declaration": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
}
|
||||
Reference in New Issue
Block a user