28 Commits
Author SHA1 Message Date
mesh-admin 7bd76275f9 Merge pull request 'A refused announcement is said, never fatal (hq issue 217)' (#43) from fix/a-refused-announcement-is-not-fatal into main 2026-10-03 21:24:50 +00:00
jochen 6ba0f4dc1e A refused announcement is said, never fatal (hq issue 217)
The raw subscription that answers discovery ran its loop unguarded, so a refusal escaped as an
unhandled rejection and ended the process: every per-module container crash-looped on 2026-10-03
over a subscription that only serves the mesh seeing the runtime. It is now caught, logged, and the
runtime serves on. Tested against a bus whose permissions refuse the subject; fails without it.
2026-10-03 23:24:38 +02:00
mesh-admin bc05658772 Merge pull request 'A mesh seat's holder answers for the machine it runs on (hq ADR 0197)' (#42) from fix/a-mesh-seats-holder-answers-for-its-machine into main 2026-10-03 21:23:06 +00:00
jochen 4af59636c3 A mesh seat's holder answers for the machine it runs on (hq ADR 0197)
The controller announces the mesh-controller seat without a machine — the seat is the mesh's — and
the console then reported it as not answering on the machine it is assigned to. An announcement that
names no machine now answers for wherever its module is assigned.
2026-10-03 23:22:56 +02:00
mesh-admin 6e425a000f Merge pull request 'The node's runtime is its modules' bus: it binds each module's consumer and hands its events to the bundle (hq ADR 0198)' (#41) from feat/0198-the-runtime-is-the-bus into main 2026-10-03 21:18:33 +00:00
jochen 14b6588839 Merge remote-tracking branch 'origin/main' into feat/0198-the-runtime-is-the-bus 2026-10-03 23:16:43 +02:00
mesh-admin 490aedfd36 Merge pull request 'Announce only on the subjects the grants allow (hq ADR 0197, issue 217)' (#40) from fix/announce-only-what-the-grants-allow into main 2026-10-03 21:11:24 +00:00
jochen f11ac6441c Announce only on the subjects the grants allow (hq ADR 0197)
Both runtimes subscribed $SRV.<verb>.> as a wildcard; the grants allow the bare question and the
service's own name and instance. The bus refused the wildcard, and the TypeScript runtime treats a
refused subscription as fatal, so every per-module container crash-looped after the image rolled.
They now subscribe exactly $SRV.<verb>, $SRV.<verb>.<name> and $SRV.<verb>.<name>.<id>.
2026-10-03 22:31:02 +02:00
jochen ffe229308c The node's runtime is its modules' bus: it binds each module's consumer and hands its events to the bundle (hq ADR 0198)
A launched bundle's mesh/subscribe binds the module's own durable consumer — EVENTS, <node>_<module>,
by name as the module's own runtime bound it, so nothing is lost or replayed in the move — and every
event goes to each child of the module that subscribed as mesh/event, acknowledged only when all
answered, negatively acknowledged after a short delay when one failed or died, terminated when it is
not an event. mesh/ask calls a tool as the module. Every launched bundle is started again when it
exits, with backoff, since long-running code waits for no call. Requires SDK 0.1.6.
2026-10-03 22:29:20 +02:00
mesh-admin df4f492a72 Merge pull request 'The mesh's tools are found by address, from what announces itself on the bus (hq ADR 0195, 0197)' (#39) from feat/0197-tools-announce-themselves into main 2026-10-03 20:19:53 +00:00
jochen 66e8be0e31 The TypeScript runtime announces what it serves too, in the same services format (hq ADR 0197)
The per-module containers still run this runtime; their tools and the seats they hold (the store's,
the catalogue's) must be found by the console the same way as the node runtime's. It answers
$SRV.PING, $SRV.INFO and $SRV.STATS with one service per process, one endpoint per tool per subject
and per seat verb served, the metadata as the Go runtime writes it.
2026-10-03 22:15:48 +02:00
jochen 7722668220 Every runtime announces what it serves in the NATS services protocol; the console discovers by asking the bus (hq ADR 0197)
The Go runtime answers $SRV.PING, $SRV.INFO and $SRV.STATS (and per name and id) in the
io.nats.micro.v1 format with what it serves at the moment it is asked: one service per runtime
process, since the bus admits one reply per request from each responder, and one endpoint per tool
per subject, its metadata saying module, seat, scope, machine, description, schema and whether the
module is interchangeable. Serving is unchanged.

The console gathers one $SRV.INFO request's answers instead of asking the catalogue's roster and
each module's tools, and reads the controller's records as JSON for what should have answered: an
assignment with tools that did not announce is named, a module without tools never is. The text
parsers of node list and module list are gone. Packages share the test bus: go test -p 1.
2026-10-03 22:14:16 +02:00
jochen e8989f3cf5 Merge remote-tracking branch 'origin/main' into feat/0197-tools-announce-themselves 2026-10-03 22:08:06 +02:00
mesh-admin 915f372a85 Merge pull request 'node-tools in Go: the node's runtime, launch-only, wire-compatible with the TypeScript (hq ADR 0193)' (#38) from feat/0193-node-tools-in-go into main 2026-10-03 20:02:15 +00:00
jochen 486dad99a5 Merge remote-tracking branch 'origin/main' into feat/0193-node-tools-in-go 2026-10-03 22:02:03 +02:00
mesh-admin 65f3b68076 Merge pull request 'node-tools launches every bundle it serves; a child's emit is published as its module (hq ADR 0193)' (#36) from feat/0193-the-runtime-launches-every-bundle into main 2026-10-03 20:01:58 +00:00
jochen a269a76c87 The console announces five tools and reaches everything by address (hq ADR 0195)
mesh_overview, mesh_machine, mesh_search, mesh_describe and mesh_call walk the mesh's structure;
every tool has one address per layer: <seat>.<verb>, <node>/<seat>.<verb>, <node>/<module>.<tool>,
and <module>.<tool> for a module the mesh issued a plain subject. A stateful module called without
its machine, a node seat without one, a mesh seat with one, or a module on the wrong machine is
refused naming what would work. Answers come from the mesh when asked, kept five seconds, so a tool
that arrives mid-session is found. The flat catalogue stays behind MESH_CONSOLE_FLAT=1 and old
<module>.<tool> names still answer.
2026-10-03 21:54:19 +02:00
jochen e6a33dd969 node-tools is built from Go (hq ADR 0193)
The node's runtime is the Go binary: one static executable the host runs as ./node-tools from its own
bundle. Node.js stays on the machine for the TypeScript bundles the runtime launches. The TypeScript
source stays in the repository: it is the runtime image the per-module containers run until WP4c.
2026-10-03 21:26:06 +02:00
mesh-admin 42e0987c64 Merge pull request 'Require SDK 0.1.5: a launched bundle names its module and emits through the runtime (hq ADR 0193)' (#37) from fix/the-toolchain-carries-sdk-0.1.5 into main 2026-10-03 19:23:59 +00:00
jochen ae6bdc9b06 Require SDK 0.1.5: a launched bundle names its module and emits through the runtime (hq ADR 0193)
The toolchain image installs from this package.json with a range; unchanged, Docker reused the
cached install and the image kept SDK 0.1.3 after 0.1.5 was published. Bundles copied that copy, so
a launched bundle registering its seat first served the seat's verbs as its own tools. Requiring
0.1.5 says what the runtime and its bundles need, and invalidates the cached layer.
2026-10-03 21:23:50 +02:00
jochen e45389f75a node-tools in Go: the node's runtime, launch-only, wire-compatible with the TypeScript (hq ADR 0193)
The runtime knows no language, so nothing ties it to Node.js. This ports its serve mode — the
pinned bus connection and patient connect, following memberships, launching every served bundle
over MCP on stdio with its own environment, a child's emit published as its module, each tool,
the tools verb and seat verbs served where the mesh issued them, and the console on loopback —
to one static binary. Same subjects, request and reply bodies, event headers and MCP answers.
The TypeScript stays: it is still the runtime inside the per-module containers until WP4c.
Tests run against a real bus and share the TypeScript fixtures.
2026-10-03 21:21:52 +02:00
jochen b182943c24 node-tools launches every bundle it serves; a child's emit is published as its module (hq ADR 0193)
Every served entrypoint is started as a process speaking MCP over stdio, told its module and node;
one that is not executable is refused by name. The import path, the SDK resolve hook (issue 209)
and the per-registration hand-off go. The one-module form the per-module containers use is still
imported until they move (to-be 38 WP4c). A child's mesh/publish is published as its module and
answered once accepted; a child that dies says why in its own last words. Fixtures are served
through launchers exactly as the builder writes them.
2026-10-03 21:12:15 +02:00
mesh-admin 26e7fcb4cb Merge pull request 'A launched bundle is told the module it serves (hq ADR 0193)' (#35) from feat/0193-a-launched-bundle-is-told-its-module into main 2026-10-03 19:08:06 +00:00
jochen 302645dd01 A launched bundle is told the module it serves (hq ADR 0193)
node-tools sets MESH_SERVED_MODULE for each child it launches, so the SDK's stdio loop lists that
module's tools by their names and a seat's verbs as the seat's, whichever was registered first.
2026-10-03 21:07:52 +02:00
mesh-admin 8e30ea9ff5 Merge pull request 'node-tools hands each bundle its own environment (hq ADR 0192)' (#33) from feat/0192-each-bundle-its-own-env into main 2026-10-03 13:34:05 +00:00
jochen 193ed0ac63 node-tools hands each bundle its own environment (hq ADR 0192)
The mesh composes every served module's words into MESH_TOOL_ENV; the runtime takes it at start
and removes it from its own environment, then gives each registration's contributor and each
launched child the runtime's words plus its own module's, never another's. Against an SDK
without collectToolsEach it says so and serves with the runtime's words only. The test serves
two imported bundles and one launched, each answering with its own words and none of the others'.
2026-10-03 15:28:49 +02:00
mesh-admin 4537494150 Merge pull request 'One SDK per runtime: a bundle's SDK import resolves to the runtime's copy (hq issue 209)' (#32) from fix/issue-209-one-sdk-per-runtime into main 2026-10-03 11:07:08 +00:00
jochen 7152148410 One SDK per runtime: a bundle's SDK import resolves to the runtime's copy (hq issue 209)
A bundle carries its dependencies, the SDK among them; imported in-process that copy was a
second SDK with its own tool registry, so a bundle registered its tools into a list the
runtime never read and served nothing, silently. A resolve hook (module.registerHooks, in
thread; module.register is deprecated from Node 26) now sends every import of
@novox/mesh-sdk, from whichever bundle, to the runtime's own copy: one registry, one broker.
The test loads a bundle from a directory holding its own SDK copy and sees its tool served.
2026-10-03 13:00:19 +02:00
49 changed files with 5303 additions and 39 deletions
+130
View File
@@ -0,0 +1,130 @@
// node-tools — the node's tool runtime, in Go (novox/hq ADR 0175, ADR 0193; to-be 38 WP4d).
//
// One process per machine: it connects to the bus on the node's credential, launches every assigned
// module's tools bundle and serves its tools and its seats' verbs; as the node-tools module — or
// wherever MESH_CONSOLE_LISTEN says — it is also the console, MCP over HTTP on loopback.
//
// MESH_BROKER_FILE the credential the mesh sealed to this machine for the runtime
// MESH_TOOL_MODULES <module>=<entrypoint>,… the bundles to launch
// MESH_TOOL_ENV {"<module>": {"<word>": "<value>"}} what each is given (ADR 0192)
// MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME their home
// MESH_CONSOLE_LISTEN where the console listens, overriding 127.0.0.1:4270
package main
import (
"encoding/json"
"fmt"
"log"
"math/rand/v2"
"os"
"os/signal"
"syscall"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/console"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
// runtimeModule is the module that is the node's tool runtime; on its credential, serving is also
// the console.
const runtimeModule = "node-tools"
// consoleListen is where the console listens when nothing says otherwise.
const consoleListen = "127.0.0.1:4270"
func main() {
log.SetFlags(0)
if len(os.Args) > 1 && os.Args[1] != "serve" {
fmt.Fprintf(os.Stderr, "node-tools: %q is not a mode; this runtime serves (novox/hq ADR 0193)\n", os.Args[1])
os.Exit(2)
}
cred := credential()
conn := connectPatiently(cred)
served, err := runtime.ServedModulesFrom(os.Getenv(runtime.ToolModules), cred.Module)
if err != nil {
log.Fatalf("node-tools: %v", err)
}
envs, err := runtime.TakeToolEnvs()
if err != nil {
log.Fatalf("node-tools: %v", err)
}
stop, err := runtime.Run(conn, served, envs, log.Printf)
if err != nil {
log.Fatalf("node-tools: %v", err)
}
listen := os.Getenv("MESH_CONSOLE_LISTEN")
if listen == "" && cred.Module == runtimeModule {
listen = consoleListen
}
var up *console.Listening
if listen != "" {
node := cred.Node
if node == "" {
node = "?"
}
who := node + "." + cred.Module
up, err = console.Serve(console.NewSurface(conn, who), listen)
if err != nil {
log.Fatalf("node-tools: %v", err)
}
log.Printf("mesh console listening on http://%s/mcp as %s", up.Address, who)
}
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGTERM, syscall.SIGINT)
<-signals
stop()
if up != nil {
_ = up.Close()
}
conn.Close()
}
// credential reads the credential the mesh delivered; a missing or unreadable one is a fault of
// configuration, said and final.
func credential() bus.Credential {
file := os.Getenv("MESH_BROKER_FILE")
if file == "" {
if url := os.Getenv("MESH_BROKER_URL"); url != "" {
return bus.Credential{URL: url, Module: runtimeModule}
}
fmt.Fprintln(os.Stderr, "node-tools: set MESH_BROKER_FILE (a sealed credential) or MESH_BROKER_URL — there is no broker to reach")
os.Exit(1)
}
raw, err := os.ReadFile(file)
if err != nil {
fmt.Fprintf(os.Stderr, "node-tools: cannot read the broker credential at %s: %v\n", file, err)
os.Exit(1)
}
var cred bus.Credential
if err := json.Unmarshal(raw, &cred); err != nil {
fmt.Fprintf(os.Stderr, "node-tools: cannot read the broker credential at %s: %v\n", file, err)
os.Exit(1)
}
if cred.URL == "" {
fmt.Fprintf(os.Stderr, "node-tools: %s carries no url — it is not a broker credential\n", file)
os.Exit(1)
}
return cred
}
// connectPatiently retries while the bus is merely not reachable yet — the normal case at startup —
// and gives up at once on what waiting cannot fix (issue 058).
func connectPatiently(cred bus.Credential) *bus.Conn {
for delay := 2 * time.Second; ; delay = min(delay*2, 30*time.Second) {
conn, err := bus.Connect(cred)
if err == nil {
return conn
}
if fatal := bus.Fatal(err); fatal != "" {
fmt.Fprintf(os.Stderr, "node-tools: %s — waiting will not fix this; giving up\n", fatal)
os.Exit(1)
}
wait := delay + time.Duration(rand.IntN(1000))*time.Millisecond
fmt.Fprintf(os.Stderr, "node-tools: the broker is not reachable yet (%v); retrying in %ds\n", err, int(wait.Round(time.Second)/time.Second))
time.Sleep(wait)
}
}
+15
View File
@@ -0,0 +1,15 @@
module github.com/novox/mesh-tools/node-tools
go 1.26.0
require (
github.com/nats-io/nats.go v1.54.0
golang.org/x/sys v0.48.0
)
require (
github.com/klauspost/compress v1.20.0 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
)
+12
View File
@@ -0,0 +1,12 @@
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+211
View File
@@ -0,0 +1,211 @@
// Package announce answers the NATS services protocol's discovery for what a runtime serves, and
// gathers the answers (novox/hq ADR 0197).
//
// A runtime does not re-serve its tools through a services library: serving is unchanged. It answers
// `$SRV.PING`, `$SRV.INFO` and `$SRV.STATS` — and the same followed by its service name, and by its
// name and id — in the format NATS's own tools read, with what it is serving at the moment it is
// asked. One service per runtime process: the bus admits one reply per request from each responder,
// so a runtime serving many modules and seats answers once, one endpoint per tool per subject, and
// says in each endpoint's metadata which module, seat, scope and machine it is.
package announce
import (
"encoding/json"
"strings"
"time"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// Version is the announced service version (semver, as the protocol requires).
const Version = "0.1.0"
// Window is how long the console gathers discovery answers: every instance answers one request, and
// how many will is what is being found out.
var Window = 750 * time.Millisecond
// Kinds of endpoint.
const (
KindTool = "tool" // a module's own tool
KindSeat = "seat" // a seat's verb, served by the module holding it
)
// Endpoint is one tool served on one subject, as it is announced.
type Endpoint struct {
Kind string
Module string // the module whose code answers
Tool string // the tool's or the verb's name
Seat string // for a seat's verb
Scope string // "mesh" or "node", for a seat's verb
Node string
Description string
Schema json.RawMessage
Interchangeable bool
Subject string
Queue string
}
// Name is the endpoint's name as the protocol allows it — letters, digits, `-` and `_` — the
// prefix and the tool joined by `__`; the metadata, not the name, is what identifies it.
func (e Endpoint) Name() string {
prefix := e.Module
if e.Kind == KindSeat {
prefix = e.Seat
}
return clean(prefix) + "__" + clean(e.Tool)
}
func clean(s string) string {
var b strings.Builder
for _, r := range s {
if r == '-' || r == '_' || (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') {
b.WriteRune(r)
} else {
b.WriteRune('_')
}
}
return b.String()
}
func (e Endpoint) info() micro.EndpointInfo {
md := map[string]string{
"kind": e.Kind, "module": e.Module, "tool": e.Tool, "node": e.Node,
"description": e.Description, "interchangeable": boolWord(e.Interchangeable),
}
schema := strings.TrimSpace(string(e.Schema))
if schema == "" || schema == "null" {
schema = "{}"
}
md["schema"] = schema
if e.Kind == KindSeat {
md["seat"] = e.Seat
md["scope"] = e.Scope
}
return micro.EndpointInfo{Name: e.Name(), Subject: e.Subject, QueueGroup: e.Queue, Metadata: md}
}
func boolWord(b bool) string {
if b {
return "true"
}
return "false"
}
// Service is who answers: the runtime's name, its instance, and a word about it.
type Service struct {
Name string
ID string
Description string
Metadata map[string]string
}
func (s Service) identity() micro.ServiceIdentity {
md := s.Metadata
if md == nil {
md = map[string]string{}
}
return micro.ServiceIdentity{Name: s.Name, ID: s.ID, Version: Version, Metadata: md}
}
// Info is the info_response for these endpoints.
func Info(s Service, endpoints []Endpoint) micro.Info {
out := micro.Info{ServiceIdentity: s.identity(), Type: micro.InfoResponseType,
Description: s.Description, Endpoints: []micro.EndpointInfo{}}
for _, e := range endpoints {
out.Endpoints = append(out.Endpoints, e.info())
}
return out
}
// Serve answers discovery for one service until stopped, asking `current` for its endpoints each
// time — so what is announced is what is served now, re-served memberships included.
func Serve(conn *bus.Conn, s Service, current func() []Endpoint) (func(), error) {
started := time.Now().UTC()
answer := func(subject string, _ []byte) []byte {
parts := strings.Split(subject, ".")
if len(parts) < 2 || parts[0] != "$SRV" {
return nil
}
if len(parts) >= 3 && parts[2] != s.Name {
return nil // another service's
}
if len(parts) >= 4 && parts[3] != s.ID {
return nil // another instance's
}
var v any
switch parts[1] {
case "PING":
v = micro.Ping{ServiceIdentity: s.identity(), Type: micro.PingResponseType}
case "INFO":
v = Info(s, current())
case "STATS":
st := micro.Stats{ServiceIdentity: s.identity(), Type: micro.StatsResponseType, Started: started,
Endpoints: []*micro.EndpointStats{}}
for _, e := range current() {
st.Endpoints = append(st.Endpoints, &micro.EndpointStats{Name: e.Name(), Subject: e.Subject, QueueGroup: e.Queue})
}
v = st
default:
return nil
}
body, err := json.Marshal(v)
if err != nil {
return nil
}
return body
}
var stops []func()
for _, verb := range []string{"PING", "INFO", "STATS"} {
// Exactly the questions asked of every service and of this one by name and instance — what the
// grants allow (novox/hq ADR 0197). A wildcard is refused by the bus.
for _, subject := range []string{"$SRV." + verb, "$SRV." + verb + "." + s.Name, "$SRV." + verb + "." + s.Name + "." + s.ID} {
stop, err := conn.Raw(subject, answer)
if err != nil {
for _, st := range stops {
st()
}
return func() {}, err
}
stops = append(stops, stop)
}
}
return func() {
for _, st := range stops {
st()
}
}, nil
}
// Gather asks every service on the bus what it serves and answers what came back within the window.
// An answer that is not an info_response is skipped.
func Gather(conn *bus.Conn) ([]micro.Info, error) {
raw, err := conn.Gather("$SRV.INFO", nil, Window)
if err != nil {
return nil, err
}
var out []micro.Info
for _, b := range raw {
var i micro.Info
if json.Unmarshal(b, &i) != nil || i.Type != micro.InfoResponseType {
continue
}
out = append(out, i)
}
return out, nil
}
// Endpoints reads an info_response's endpoints back into what they announce.
func Endpoints(i micro.Info) []Endpoint {
var out []Endpoint
for _, e := range i.Endpoints {
md := e.Metadata
out = append(out, Endpoint{
Kind: md["kind"], Module: md["module"], Tool: md["tool"], Seat: md["seat"], Scope: md["scope"],
Node: md["node"], Description: md["description"], Schema: json.RawMessage(md["schema"]),
Interchangeable: md["interchangeable"] == "true", Subject: e.Subject, Queue: e.QueueGroup,
})
}
return out
}
+735
View File
@@ -0,0 +1,735 @@
// Package bus is the node runtime's connection to the mesh bus, on NATS (novox/hq design 25, design
// 29, ADR 0160, ADR 0175). It is the Go port of node-tools' broker-nats.ts, and speaks the same
// wire: the same subjects, the same JSON request and reply bodies, the same event headers.
//
// mesh.mod.<module>.event.<type> an event a module emits
// mesh.mod.<module>.tool.<tool> a tool a module serves
// mesh.seat.<seat>.tool.<verb> a role's verb, answered by whoever holds the seat
package bus
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"log"
"regexp"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
// RequestTimeout is how long a call waits for its answer — what modules already expect.
const RequestTimeout = 30 * time.Second
const assignmentsStream = "ASSIGNMENTS"
// Credential is a broker credential as the mesh delivers it (novox/hq ADR 0120).
type Credential struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Node string `json:"node,omitempty"`
Module string `json:"module,omitempty"`
User string `json:"user,omitempty"`
Password string `json:"password,omitempty"`
Claims []Claim `json:"claims,omitempty"`
}
// Claim is a seat a module claims, with the verbs it promises (novox/hq ADR 0159).
type Claim struct {
Seat string `json:"seat"`
Scope string `json:"scope,omitempty"`
Serves []string `json:"serves,omitempty"`
}
// Membership is what the mesh issued one assignment (novox/hq ADR 0160).
type Membership struct {
Node string `json:"node"`
Module string `json:"module"`
Serves []Served `json:"serves"`
Seats []SeatVerb `json:"seats,omitempty"`
Emits string `json:"emits"`
Reaches map[string][]string `json:"reaches,omitempty"`
Tools string `json:"tools"`
}
// Served is an address a tool is answered on; `{tool}` stands for the tool's name.
type Served struct {
Subject string `json:"subject"`
Queue string `json:"queue,omitempty"`
}
// SeatVerb is one verb of a seat a module holds, where it is answered.
type SeatVerb struct {
Seat string `json:"seat"`
Verb string `json:"verb"`
Subject string `json:"subject"`
}
// Envelope is an event as a module emits it: the body is the payload, the metadata rides as headers
// (novox/hq ADR 0042).
type Envelope struct {
Key string `json:"key"`
Node string `json:"node,omitempty"`
Body json.RawMessage `json:"body"`
Headers map[string]string `json:"headers,omitempty"`
}
// Answered is a call's result and the machine that gave it (novox/hq ADR 0159).
type Answered struct {
Result json.RawMessage
Node string
}
// Handler answers one request body.
type Handler func(body json.RawMessage) (any, error)
// ErrPin is a bus whose certificate is not the one the mesh pinned: final, never retried.
var ErrPin = errors.New("the bus's certificate does not match the pin")
// Fatal is why a connection failure is final rather than "not yet", or "" when waiting may fix it —
// the same classification the TypeScript runtime makes.
func Fatal(err error) string {
if err == nil {
return ""
}
msg := err.Error()
if errors.Is(err, ErrPin) || strings.Contains(msg, ErrPin.Error()) {
return "the bus's certificate does not match the pin"
}
if regexp.MustCompile(`(?i)invalid url|no servers available for connection: .*url`).MatchString(msg) ||
strings.Contains(msg, "nats: invalid url") {
return "the bus address is not a usable URL"
}
if regexp.MustCompile(`(?i)authorization violation|user authentication expired|permissions violation`).MatchString(msg) {
return "the bus refused this account"
}
return ""
}
func normalizeFingerprint(f string) string {
f = strings.TrimSpace(f)
if len(f) > 7 && strings.EqualFold(f[:7], "sha256:") {
f = f[7:]
}
return strings.ToLower(strings.ReplaceAll(f, ":", ""))
}
// pinned accepts exactly the certificate with this SHA-256 and no other. The pin is the only check:
// the bus's certificate names the seat, not the address a machine dials it by.
func pinned(want string) *tls.Config {
want = normalizeFingerprint(want)
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(raw [][]byte, _ [][]*x509.Certificate) error {
if len(raw) == 0 {
return fmt.Errorf("%w: it presented none", ErrPin)
}
sum := sha256.Sum256(raw[0])
if got := hex.EncodeToString(sum[:]); got != want {
return fmt.Errorf("%w: it presented %s, not the pinned %s", ErrPin, got, want)
}
return nil
},
}
}
// Conn is the runtime's connection: the module it is, the memberships it follows, the subjects it
// answers.
type Conn struct {
nc *nats.Conn
js nats.JetStreamContext
self string
node string
cred Credential
mu sync.Mutex
issued map[string]*Membership // module → membership; present with nil = followed, none issued
onNew []func(Membership)
subs []*nats.Subscription
Logf func(format string, args ...any)
}
// Connect dials the bus as the credential's module. A module's subjects come from its credential,
// never from its calls (ADR 0074).
func Connect(cred Credential) (*Conn, error) {
if cred.Module == "" {
return nil, errors.New("a broker credential with no module: the runtime derives its subjects " +
"from the account the mesh issued, and cannot guess which module it is")
}
node := cred.Node
if node == "" {
node = "?"
}
opts := []nats.Option{
nats.Name(node + "." + cred.Module),
// Reconnect forever: the bus restarting is an upgrade, not a reason to exit.
nats.MaxReconnects(-1),
}
if cred.User != "" {
opts = append(opts, nats.UserInfo(cred.User, cred.Password),
// Its own inbox: every user's inbox is private to it (design 25 §4).
nats.CustomInboxPrefix("_INBOX."+cred.User))
}
if strings.TrimSpace(cred.Fingerprint) != "" {
opts = append(opts, nats.Secure(pinned(cred.Fingerprint)))
}
nc, err := nats.Connect(cred.URL, opts...)
if err != nil {
return nil, err
}
js, err := nc.JetStream()
if err != nil {
nc.Close()
return nil, err
}
c := &Conn{nc: nc, js: js, self: cred.Module, node: cred.Node, cred: cred,
issued: map[string]*Membership{}, Logf: log.Printf}
c.Follow(cred.Module)
return c, nil
}
// Module is what this connection is.
func (c *Conn) Module() string { return c.self }
// Node is the machine this connection's account is scoped to.
func (c *Conn) Node() string { return c.node }
// Credential is what this connection was opened with.
func (c *Conn) Credential() Credential { return c.cred }
// MembershipSubject is the one address a runtime derives for an assignment (ADR 0160).
func MembershipSubject(node, module string) string {
return "mesh.assignment." + node + "." + module
}
// Follow reads a module's membership on this machine once and follows it live, so its tools are
// served where the mesh issued them (ADR 0175).
func (c *Conn) Follow(module string) {
c.mu.Lock()
if c.node == "" {
c.mu.Unlock()
return
}
if _, has := c.issued[module]; has {
c.mu.Unlock()
return
}
c.issued[module] = nil
c.mu.Unlock()
subject := MembershipSubject(c.node, module)
// The subject-addressed direct get: the one address the mesh grants this account on the
// stream's API.
if got, err := c.nc.Request("$JS.API.DIRECT.GET."+assignmentsStream+"."+subject, nil, 5*time.Second); err == nil {
if got.Header.Get("Status") == "" && len(got.Data) > 0 {
var m Membership
if json.Unmarshal(got.Data, &m) == nil {
c.mu.Lock()
c.issued[module] = &m
c.mu.Unlock()
}
}
}
if c.Membership(module) == nil {
c.Logf("[mesh-tools] no membership issued for %s on %s yet; serving the derived shape until one arrives", module, c.node)
}
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
var m Membership
if err := json.Unmarshal(msg.Data, &m); err != nil {
c.Logf("[mesh-tools] a membership arrived that is not one: %v", err)
return
}
c.mu.Lock()
c.issued[module] = &m
handlers := append([]func(Membership){}, c.onNew...)
c.mu.Unlock()
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
for _, h := range handlers {
h(m)
}
_ = c.nc.Flush()
})
if err == nil {
c.track(sub)
}
}
// Membership is what the mesh issued a module here, or nil when nothing has been issued.
func (c *Conn) Membership(module string) *Membership {
c.mu.Lock()
defer c.mu.Unlock()
return c.issued[module]
}
// Following says whether this connection follows a module's membership.
func (c *Conn) Following(module string) bool {
c.mu.Lock()
defer c.mu.Unlock()
_, has := c.issued[module]
return has
}
// Serving is every module whose membership this connection follows.
func (c *Conn) Serving() []string {
c.mu.Lock()
defer c.mu.Unlock()
out := make([]string, 0, len(c.issued))
for m := range c.issued {
out = append(out, m)
}
return out
}
// OnMembership is called with every new membership any followed module is issued.
func (c *Conn) OnMembership(h func(Membership)) {
c.mu.Lock()
c.onNew = append(c.onNew, h)
c.mu.Unlock()
}
func (c *Conn) track(s *nats.Subscription) {
c.mu.Lock()
c.subs = append(c.subs, s)
c.mu.Unlock()
}
// servedOn is where a served module's tool is answered: its membership's subjects when issued, the
// derived shape otherwise (the shape the mesh issues on day one).
func (c *Conn) servedOn(module, tool string) []Served {
if m := c.Membership(module); m != nil {
out := make([]Served, 0, len(m.Serves)+1)
for _, s := range m.Serves {
out = append(out, Served{Subject: strings.ReplaceAll(s.Subject, "{tool}", tool), Queue: s.Queue})
}
if tool == "tools" && m.Tools != "" {
found := false
for _, s := range out {
found = found || s.Subject == m.Tools
}
if !found {
out = append([]Served{{Subject: m.Tools, Queue: "serve." + module}}, out...)
}
}
return out
}
base := "mesh.mod." + module + ".tool." + tool
out := []Served{{Subject: base, Queue: "serve." + module}}
if c.node != "" {
out = append(out, Served{Subject: base + "." + c.node})
}
return out
}
type reply struct {
Result any `json:"result,omitempty"`
Error string `json:"error,omitempty"`
Node string `json:"node,omitempty"`
}
// answerOn answers one subject with one handler, and says which machine answered (ADR 0159).
func (c *Conn) answerOn(subject, queue string, h Handler) (func(), error) {
cb := func(msg *nats.Msg) {
go func() {
var r reply
result, err := h(json.RawMessage(msg.Data))
if err != nil {
r.Error = err.Error()
} else {
r.Result = nullable(result)
}
r.Node = c.node
body, _ := wire.Marshal(r)
_ = msg.Respond(body)
}()
}
var sub *nats.Subscription
var err error
if queue != "" {
sub, err = c.nc.QueueSubscribe(subject, queue, cb)
} else {
sub, err = c.nc.Subscribe(subject, cb)
}
if err != nil {
return func() {}, err
}
c.track(sub)
return func() { _ = sub.Unsubscribe() }, nil
}
// nullable keeps a nil result as JSON null rather than dropping the key: the TypeScript reply always
// carries `result` when the handler did not throw.
func nullable(v any) any {
if v == nil {
return json.RawMessage("null")
}
return v
}
// HandleSubject answers a subject outright: a seat's verb where the mesh issued it.
func (c *Conn) HandleSubject(subject string, h Handler) (func(), error) {
return c.answerOn(subject, "", h)
}
// Handle serves `<module>.<tool>` where the mesh issued that module, and follows its membership:
// when a new one arrives, it serves where it now says and stops where it no longer does.
func (c *Conn) Handle(key string, h Handler) (func(), error) {
if strings.HasPrefix(key, "seat:") {
subject, err := ToolSubject(key, c.self)
if err != nil {
return func() {}, err
}
return c.answerOn(subject, "", h)
}
module, tool := c.self, key
if dot := strings.Index(key, "."); dot >= 0 {
module, tool = key[:dot], key[dot+1:]
}
if module != c.self && !c.Following(module) {
return func() {}, fmt.Errorf("%s cannot serve %s: a module serves its own tools, and a runtime "+
"those of the modules it follows", c.self, key)
}
var mu sync.Mutex
var stops []func()
serve := func() {
mu.Lock()
defer mu.Unlock()
for _, s := range stops {
s()
}
stops = nil
for _, s := range c.servedOn(module, tool) {
if stop, err := c.answerOn(s.Subject, s.Queue, h); err == nil {
stops = append(stops, stop)
} else {
c.Logf("[mesh-tools] cannot serve %s on %s: %v", key, s.Subject, err)
}
}
}
serve()
c.OnMembership(func(m Membership) {
if m.Module == module {
serve()
}
})
return func() {
mu.Lock()
defer mu.Unlock()
for _, s := range stops {
s()
}
stops = nil
}, nil
}
// reachedAt is where a call by key goes: a subject this connection's own membership says it
// reaches — the machine's when named — else the derived shape.
func (c *Conn) reachedAt(key string) (string, error) {
name, wanted, _ := strings.Cut(key, "@")
if m := c.Membership(c.self); m != nil {
if reach := m.Reaches[name]; len(reach) > 0 {
if wanted == "" {
return reach[0], nil
}
for _, s := range reach {
if strings.HasSuffix(s, "."+wanted) {
return s, nil
}
}
}
}
return ToolSubject(key, c.self)
}
// Ask calls a tool by key — or on a subject the mesh listed for it — and learns which machine
// answered. Core request/reply: a tool call is never persisted (design 25 §3).
func (c *Conn) Ask(key string, body any, on string) (Answered, error) {
subject := on
if subject == "" {
s, err := c.reachedAt(key)
if err != nil {
return Answered{}, err
}
subject = s
}
data, err := wire.Marshal(body)
if err != nil {
return Answered{}, err
}
msg, err := c.nc.Request(subject, data, RequestTimeout)
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return Answered{}, errors.New("503 no responders")
}
if errors.Is(err, nats.ErrTimeout) {
return Answered{}, errors.New("timeout")
}
return Answered{}, err
}
var r struct {
Result json.RawMessage `json:"result"`
Error string `json:"error"`
Node string `json:"node"`
}
if err := json.Unmarshal(msg.Data, &r); err != nil {
return Answered{}, err
}
if r.Error != "" {
return Answered{}, errors.New(r.Error)
}
return Answered{Result: r.Result, Node: r.Node}, nil
}
// PublishAs emits an event as a module: published into JetStream and awaited, de-duplicated by its
// own id (ADR 0042). The body is the payload; the metadata rides as headers.
func (c *Conn) PublishAs(module string, env Envelope) error {
msg := nats.NewMsg("mesh.mod." + module + ".event." + env.Key)
for k, v := range env.Headers {
msg.Header.Set(k, v)
}
if env.Headers["content-type"] == "" {
msg.Header.Set("content-type", "application/json")
}
if env.Node != "" {
msg.Header.Set("x-node", env.Node)
}
body := env.Body
if len(body) == 0 {
body = json.RawMessage("null")
}
msg.Data = body
var opts []nats.PubOpt
if id := env.Headers["x-event-id"]; id != "" {
opts = append(opts, nats.MsgId(id))
}
_, err := c.js.PublishMsg(msg, opts...)
return err
}
// ServedOn is where a served module's tool is answered right now: the membership's subjects when
// issued, the derived shape otherwise — what Handle subscribes, for what announces it (ADR 0197).
func (c *Conn) ServedOn(module, tool string) []Served { return c.servedOn(module, tool) }
// Raw answers one subject with a function of the request, not a tool's reply envelope: the NATS
// services protocol's discovery subjects answer in their own format (novox/hq ADR 0197). A nil
// answer is no reply — the request was for another service.
func (c *Conn) Raw(subject string, answer func(subject string, data []byte) []byte) (func(), error) {
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
if body := answer(msg.Subject, msg.Data); body != nil {
_ = msg.Respond(body)
}
})
if err != nil {
return func() {}, err
}
c.track(sub)
return func() { _ = sub.Unsubscribe() }, nil
}
// Gather publishes one request and collects every answer that arrives within the window: a
// discovery request every service instance answers (ADR 0197). It never stops early — how many will
// answer is what it is finding out.
func (c *Conn) Gather(subject string, body []byte, window time.Duration) ([][]byte, error) {
inbox := c.nc.NewRespInbox()
sub, err := c.nc.SubscribeSync(inbox)
if err != nil {
return nil, err
}
defer func() { _ = sub.Unsubscribe() }()
if err := c.nc.PublishRequest(subject, inbox, body); err != nil {
return nil, err
}
var out [][]byte
deadline := time.Now().Add(window)
for {
left := time.Until(deadline)
if left <= 0 {
return out, nil
}
msg, err := sub.NextMsg(left)
if err != nil {
if errors.Is(err, nats.ErrTimeout) {
return out, nil
}
return out, err
}
if len(msg.Data) > 0 {
out = append(out, msg.Data)
}
}
}
// Flush waits until the bus has every subscription made so far, so what is served is answerable
// when this returns.
func (c *Conn) Flush() { _ = c.nc.Flush() }
// Close unsubscribes everything and drains, so an in-flight reply is finished rather than dropped.
func (c *Conn) Close() {
c.mu.Lock()
subs := c.subs
c.subs = nil
c.mu.Unlock()
for _, s := range subs {
_ = s.Unsubscribe()
}
_ = c.nc.Drain()
}
// ToolSubject is a tool's subject. A bare name is this module's own; `<module>.<tool>` another's;
// `seat:<seat>.<verb>` a role's, with `@<node>` for a node-scoped seat (design 33 §4).
func ToolSubject(key, self string) (string, error) {
if strings.HasPrefix(key, "seat:") {
rest := strings.TrimPrefix(key, "seat:")
dot := strings.Index(rest, ".")
if dot < 0 {
return "", fmt.Errorf("%q names a seat and no verb: seat:<seat>.<verb>", key)
}
seat := rest[:dot]
verb, node, _ := strings.Cut(rest[dot+1:], "@")
if node != "" {
return "mesh.seat." + seat + ".tool." + verb + "." + node, nil
}
return "mesh.seat." + seat + ".tool." + verb, nil
}
name, node, _ := strings.Cut(key, "@")
var base string
if dot := strings.Index(name, "."); dot < 0 {
base = "mesh.mod." + self + ".tool." + name
} else {
base = "mesh.mod." + name[:dot] + ".tool." + name[dot+1:]
}
if node != "" {
return base + "." + node, nil
}
return base, nil
}
// SeatToolSubject is a seat's verb as its holder serves it: flat for a mesh seat, carrying the
// machine for a node-scoped one.
func SeatToolSubject(seat, verb, scope, node string) string {
base := "mesh.seat." + seat + ".tool." + verb
if scope == "node" && node != "" {
return base + "." + node
}
return base
}
// AskAs calls a tool on a module's behalf (novox/hq ADR 0198): a bare key is that module's own tool,
// `<module>.<tool>` another's, `seat:<seat>.<verb>[@<node>]` a role's — resolved through what the
// mesh issued that module to reach, as its own runtime resolved it, else the derived shape.
func (c *Conn) AskAs(module, key string, body any) (Answered, error) {
name, wanted, _ := strings.Cut(key, "@")
subject := ""
if m := c.Membership(module); m != nil {
if reach := m.Reaches[name]; len(reach) > 0 {
subject = reach[0]
if wanted != "" {
subject = ""
for _, s := range reach {
if strings.HasSuffix(s, "."+wanted) {
subject = s
}
}
}
}
}
if subject == "" {
s, err := ToolSubject(key, module)
if err != nil {
return Answered{}, err
}
subject = s
}
return c.Ask(key, body, subject)
}
// EventsStream is where every module's events land, and ConsumerOf the durable consumer the
// controller makes for a module on a machine: the same names the module's own runtime bound
// (node-tools/src/broker-nats.ts), so moving the module into the node's runtime neither loses an
// event nor sees one twice.
const EventsStream = "EVENTS"
// ConsumerOf is a module's durable consumer on a machine: `<node>_<module>`.
func ConsumerOf(node, module string) string {
if node == "" {
node = "?"
}
return node + "_" + module
}
// NakDelay is how long an event a handler failed waits before it is offered again: a transient cause
// gets another attempt, a permanent one exhausts the consumer's max-deliver rather than spinning.
var NakDelay = 5 * time.Second
// ConsumeAs reads a module's durable consumer and hands each event to deliver (novox/hq ADR 0198):
// acknowledged when deliver returns nil, negatively acknowledged after NakDelay when it returns an
// error, terminated when it is not an event at all. The consumer is the controller's to create; this
// binds to it and never makes one. Runs until stopped.
func (c *Conn) ConsumeAs(module string, deliver func(Envelope) error) (func(), error) {
durable := ConsumerOf(c.node, module)
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
// Bound by name, never created and never matched against a subject: the consumer and its
// filters are the controller's (design 29 §3), exactly as the module's own runtime bound it.
consumer, err := js.Consumer(ctx, EventsStream, durable)
if err != nil {
return nil, fmt.Errorf("binding %s's consumer %s: %w", module, durable, err)
}
reading, err := consumer.Consume(func(msg jetstream.Msg) {
env, ok := envelopeOf(msg.Subject(), msg.Headers(), msg.Data())
if !ok {
// Unparseable: redelivering bytes no version of a handler can read is a loop.
_ = msg.Term()
return
}
if err := deliver(env); err != nil {
_ = msg.NakWithDelay(NakDelay)
return
}
_ = msg.Ack()
})
if err != nil {
return nil, fmt.Errorf("reading %s's consumer %s: %w", module, durable, err)
}
var once sync.Once
return func() { once.Do(reading.Stop) }, nil
}
// envelopeOf rebuilds the envelope a module sees from a delivered event: its key is the emitter and
// the event, recovered from the subject; its metadata rides as headers (novox/hq ADR 0042).
func envelopeOf(subject string, header nats.Header, data []byte) (Envelope, bool) {
if !json.Valid(data) {
return Envelope{}, false
}
headers := map[string]string{}
for k := range header {
headers[k] = header.Get(k)
}
return Envelope{Key: keyOf(subject), Node: headers["x-node"], Body: json.RawMessage(data), Headers: headers}, true
}
// keyOf is the key a module sees for an event subject: `mesh.mod.<emitter>.event.<event>` is
// `<emitter>.<event>` — the vocabulary its manifest names what it consumes in.
func keyOf(subject string) string {
before, event, found := strings.Cut(subject, ".event.")
if !found {
return subject
}
parts := strings.Split(before, ".")
if emitter := parts[len(parts)-1]; emitter != "" {
return emitter + "." + event
}
return event
}
+48
View File
@@ -0,0 +1,48 @@
package bus
import (
"errors"
"fmt"
"testing"
)
func TestSubjectsAreTheOnesTheTypeScriptRuntimeUses(t *testing.T) {
for key, want := range map[string]string{
"status": "mesh.mod.self.tool.status",
"alpha.one": "mesh.mod.alpha.tool.one",
"alpha.one@anchor": "mesh.mod.alpha.tool.one.anchor",
"seat:node-shelf.list": "mesh.seat.node-shelf.tool.list",
"seat:node-shelf.list@anchor": "mesh.seat.node-shelf.tool.list.anchor",
} {
if got, err := ToolSubject(key, "self"); err != nil || got != want {
t.Errorf("%s: %s %v, want %s", key, got, err, want)
}
}
if _, err := ToolSubject("seat:nothing", "self"); err == nil {
t.Error("a seat with no verb was accepted")
}
if SeatToolSubject("s", "v", "node", "n") != "mesh.seat.s.tool.v.n" || SeatToolSubject("s", "v", "mesh", "n") != "mesh.seat.s.tool.v" {
t.Error("seat subjects")
}
}
func TestAFingerprintIsReadHoweverItIsWritten(t *testing.T) {
for _, f := range []string{"sha256:AB:CD:ef", "abcdef", "ABCDEF", "SHA256:abcdef"} {
if got := normalizeFingerprint(f); got != "abcdef" {
t.Errorf("%s → %s", f, got)
}
}
}
func TestWhatWaitingCannotFixIsFinal(t *testing.T) {
for err, want := range map[error]string{
fmt.Errorf("x509: %w: it presented aa", ErrPin): "the bus's certificate does not match the pin",
errors.New("nats: Authorization Violation"): "the bus refused this account",
errors.New("nats: invalid url"): "the bus address is not a usable URL",
errors.New("dial tcp 10.0.0.1:4222: connect: connection refused"): "",
} {
if got := Fatal(err); got != want {
t.Errorf("%v → %q, want %q", err, got, want)
}
}
}
+730
View File
@@ -0,0 +1,730 @@
package console
// The mesh's tools found by address, not announced whole (novox/hq ADR 0195, to-be 34 §3a).
//
// The console announces five tools. Everything the mesh answers is reached through them by one
// address per layer:
//
// <seat>.<verb> a seat held once for the mesh — its holder answers
// <node>/<seat>.<verb> a seat held once per machine — that machine's holder answers
// <node>/<module>.<tool> a module assigned to a machine — that assignment answers
// <module>.<tool> also, for a module whose instances are interchangeable (ADR 0160)
//
// A module that is not interchangeable is called with its machine or refused, naming the machines
// it runs on: "whichever answers" is no answer for state a machine holds.
//
// Every discovery verb asks the mesh when it is called — kept a few seconds at most, never for a
// session — so a tool that arrived a minute ago is found without the client reconnecting.
import (
"encoding/json"
"fmt"
"sort"
"strings"
"sync"
"time"
"github.com/novox/mesh-tools/node-tools/internal/announce"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// IndexKept is how long what the mesh answered is kept before it is asked again: long enough that
// one agent turn's search, describe and call ask once, short enough that nothing goes stale.
var IndexKept = 5 * time.Second
// The five tools the console announces. Names of the API's kind — letters, digits, `_`, `-`.
const (
verbOverview = "mesh_overview"
verbMachine = "mesh_machine"
verbSearch = "mesh_search"
verbDescribe = "mesh_describe"
verbCall = "mesh_call"
)
// searchCap is how many matches a search answers before it says how many more there were.
const searchCap = 25
const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e.g. `mesh-controller.nodes`); " +
"`<node>/<seat>.<verb>` for a seat every machine holds (e.g. `ace/node-packet-filter.rules`); " +
"`<node>/<module>.<tool>` for a module on one machine (e.g. `novox/postgres.postgres_list_databases`); " +
"and `<module>.<tool>` also for a module whose instances are interchangeable."
// discovery is the five tools as tools/list announces them.
func discovery() []map[string]any {
str := func(desc string) map[string]any { return map[string]any{"type": "string", "description": desc} }
obj := func(props map[string]any, required ...string) map[string]any {
s := map[string]any{"type": "object", "properties": props}
if len(required) > 0 {
s["required"] = required
}
return s
}
return []map[string]any{
{"name": verbOverview, "inputSchema": obj(map[string]any{}),
"description": "The mesh at a glance: the seats it holds once for the whole mesh with their verbs, the seats " +
"every machine holds, and its machines. Start here, then `mesh_machine` for one machine. " + grammar},
{"name": verbMachine, "inputSchema": obj(map[string]any{"node": str("the machine, as mesh_overview names it")}, "node"),
"description": "One machine: the seats it holds with their verbs, and the modules assigned to it with their tools — " +
"each with the address to describe or call it by. " + grammar},
{"name": verbSearch, "inputSchema": obj(map[string]any{"query": str("words to find in tool names and descriptions, e.g. `postgres databases`")}, "query"),
"description": "Find tools anywhere in the mesh by words: every match's address and a line of what it does, across the " +
"mesh's seats, the machines' seats and every module on every machine. " + grammar},
{"name": verbDescribe, "inputSchema": obj(map[string]any{"address": str("the tool's address")}, "address"),
"description": "What one tool does and the arguments it takes, as a JSON schema. The machine is in the address, " +
"never an argument. " + grammar},
{"name": verbCall, "inputSchema": obj(map[string]any{
"address": str("the tool's address"),
"arguments": map[string]any{"type": "object", "description": "the tool's arguments, as mesh_describe gives its schema"},
}, "address"),
"description": "Call one tool by its address with its arguments; the answer says which machine gave it. " + grammar},
}
}
func isDiscovery(name string) bool {
switch name {
case verbOverview, verbMachine, verbSearch, verbDescribe, verbCall:
return true
}
return false
}
// seatInfo is a seat as the mesh's records define it, and who holds it where.
type seatInfo struct {
Seat string
Scope string // "mesh" or "node"
Verbs []Tool
Holders []holder
}
type holder struct {
Module string `json:"module"`
Node string `json:"node"`
}
// moduleInfo is a module that answers tools: where it runs, whether any instance will do, and its
// tools as one of its instances described them.
type moduleInfo struct {
Module string
On []string
Interchangeable bool
Tools []Tool
}
// index is what the mesh answered about itself, at one moment.
type index struct {
Seats []seatInfo
Machines []string
Modules map[string]*moduleInfo
NotAnswering []string
Listing *Listing // the flat catalogue the call path resolves subjects with
}
func (x *index) seat(name string) *seatInfo {
for i := range x.Seats {
if x.Seats[i].Seat == name {
return &x.Seats[i]
}
}
return nil
}
func findTool(tools []Tool, name string) *Tool {
for i := range tools {
if tools[i].Name == name {
return &tools[i]
}
}
return nil
}
// controllerOutput is a controller seat verb's answer: the command's printed output.
func controllerOutput(conn *bus.Conn, verb string) (string, error) {
got, err := conn.Ask("seat:mesh-controller."+verb, map[string]any{}, "")
if err != nil {
return "", err
}
var r struct {
Output string `json:"output"`
OK *bool `json:"ok"`
}
if json.Unmarshal(got.Result, &r) != nil {
return "", fmt.Errorf("mesh-controller.%s answered something that is not its output", verb)
}
if r.OK != nil && !*r.OK {
return "", fmt.Errorf("mesh-controller.%s: %s", verb, strings.TrimSpace(r.Output))
}
return r.Output, nil
}
// jsonIn is the JSON document a command printed, after any lines it said first: a seat verb runs
// the controller's command, and a command may warn before it answers.
func jsonIn(output string) string {
t := strings.TrimSpace(output)
if strings.HasPrefix(t, "{") || strings.HasPrefix(t, "[") {
return t
}
for _, open := range []string{"\n{", "\n["} {
if i := strings.Index(output, open); i >= 0 {
return strings.TrimSpace(output[i+1:])
}
}
return ""
}
// recordedModule is a module as the controller's records hold it (`module list --json`): where the
// mesh assigned it, and whether it declares tools — what should announce itself, and where.
type recordedModule struct {
Module string `json:"module"`
On []string `json:"on"`
Tools bool `json:"tools"`
}
// recordedMachine is a machine as the controller's records hold it (`node list --json`).
type recordedMachine struct {
Name string `json:"name"`
}
// indexOn asks the mesh what it holds (novox/hq ADR 0197): what answers, from every runtime's own
// announcement on the bus — one `$SRV.INFO` request — and what should, from the controller's records
// read as JSON. Nothing is inferred from a roster and nothing is parsed from print.
func indexOn(conn *bus.Conn) (*index, error) {
var wg sync.WaitGroup
var nodesOut, modulesOut string
wg.Add(2)
go func() { defer wg.Done(); nodesOut, _ = controllerOutput(conn, "nodes") }()
go func() { defer wg.Done(); modulesOut, _ = controllerOutput(conn, "modules") }()
infos, err := announce.Gather(conn)
wg.Wait()
if err != nil {
return nil, err
}
l := &Listing{Tools: []Tool{}, NotAnswering: []string{}}
x := &index{Modules: map[string]*moduleInfo{}, Listing: l}
announced := map[string]map[string]bool{} // module → node → announced something
seats := map[string]*seatInfo{}
toolAt := map[string]int{} // <module>.<tool> or <seat>.<verb> → index in l.Tools
machines := map[string]bool{}
for _, info := range infos {
for _, e := range announce.Endpoints(info) {
if e.Node != "" {
machines[e.Node] = true
}
if announced[e.Module] == nil {
announced[e.Module] = map[string]bool{}
}
announced[e.Module][e.Node] = true
switch e.Kind {
case announce.KindSeat:
st := seats[e.Seat]
if st == nil {
st = &seatInfo{Seat: e.Seat, Scope: e.Scope}
seats[e.Seat] = st
}
if st.Scope != "node" && e.Scope == "node" {
st.Scope = "node"
}
h := holder{Module: e.Module, Node: e.Node}
if !containsHolder(st.Holders, h) {
st.Holders = append(st.Holders, h)
}
key := e.Seat + "." + e.Tool
if _, have := toolAt[key]; !have {
toolAt[key] = len(l.Tools)
t := Tool{Module: e.Seat, Name: e.Tool, Description: e.Description, Input: e.Schema, Seat: true, Scope: st.Scope}
l.Tools = append(l.Tools, t)
st.Verbs = append(st.Verbs, t)
}
case announce.KindTool:
m := x.Modules[e.Module]
if m == nil {
m = &moduleInfo{Module: e.Module}
x.Modules[e.Module] = m
}
if e.Node != "" && !contains(m.On, e.Node) {
m.On = append(m.On, e.Node)
}
m.Interchangeable = m.Interchangeable || e.Interchangeable
key := e.Module + "." + e.Tool
i, have := toolAt[key]
if !have {
i = len(l.Tools)
toolAt[key] = i
l.Tools = append(l.Tools, Tool{Module: e.Module, Name: e.Tool, Description: e.Description, Input: e.Schema})
}
if !contains(l.Tools[i].Subjects, e.Subject) {
l.Tools[i].Subjects = append(l.Tools[i].Subjects, e.Subject)
}
}
}
}
// A tool's subjects as a call looks them up: the plain one any instance answers first, then each
// machine's.
for i := range l.Tools {
t := &l.Tools[i]
if t.Seat {
continue
}
plain := "mesh.mod." + t.Module + ".tool." + t.Name
sort.SliceStable(t.Subjects, func(a, b int) bool {
if (t.Subjects[a] == plain) != (t.Subjects[b] == plain) {
return t.Subjects[a] == plain
}
return t.Subjects[a] < t.Subjects[b]
})
}
for _, t := range l.Tools {
if !t.Seat {
x.Modules[t.Module].Tools = append(x.Modules[t.Module].Tools, t)
}
}
for _, m := range x.Modules {
sort.Strings(m.On)
}
for _, st := range seats {
sort.Slice(st.Holders, func(a, b int) bool { return st.Holders[a].Node < st.Holders[b].Node })
x.Seats = append(x.Seats, *st)
}
sort.Slice(x.Seats, func(i, j int) bool { return x.Seats[i].Seat < x.Seats[j].Seat })
sort.SliceStable(l.Tools, func(i, j int) bool {
return l.Tools[i].Module+"."+l.Tools[i].Name < l.Tools[j].Module+"."+l.Tools[j].Name
})
// What should have answered: every assignment of a module that declares tools. Silence is named;
// a module with no tools is never a name here.
var recorded []recordedModule
if json.Unmarshal([]byte(jsonIn(modulesOut)), &recorded) == nil {
for _, m := range recorded {
if !m.Tools {
continue
}
for _, n := range m.On {
// A holder of a seat held once for the mesh announces no machine — the seat is the mesh's,
// not a machine's — so what it announced without one answers for wherever it is assigned.
if !announced[m.Module][n] && !announced[m.Module][""] {
l.NotAnswering = append(l.NotAnswering, m.Module+" on "+n)
}
}
}
} else {
l.NotAnswering = append(l.NotAnswering, "mesh-controller (its records of the modules did not answer, so what is missing cannot be said)")
}
sort.Strings(l.NotAnswering)
x.NotAnswering = l.NotAnswering
var known []recordedMachine
if json.Unmarshal([]byte(jsonIn(nodesOut)), &known) == nil {
for _, n := range known {
if n.Name != "" {
machines[n.Name] = true
}
}
}
for n := range machines {
x.Machines = append(x.Machines, n)
}
sort.Strings(x.Machines)
return x, nil
}
func containsHolder(hs []holder, h holder) bool {
for _, x := range hs {
if x == h {
return true
}
}
return false
}
func (s *Surface) index() (*index, error) {
s.mu.Lock()
if s.idx != nil && time.Since(s.idxAt) <= IndexKept {
x := s.idx
s.mu.Unlock()
return x, nil
}
s.mu.Unlock()
x, err := indexOn(s.conn)
if err != nil {
return nil, err
}
s.mu.Lock()
s.idx, s.idxAt = x, time.Now()
s.mu.Unlock()
return x, nil
}
// target is what an address resolves to.
type target struct {
Address string
Key string // the call key the existing path takes: seat:<s>.<v>[@node] or <m>.<t>[@node]
Name string // <seat>.<verb> or <module>.<tool>, for the listing's subject lookup
Node string
Tool Tool
Seat bool
}
// resolve turns an address into exactly one target, or says why it cannot.
func resolve(x *index, address string) (target, error) {
address = strings.TrimSpace(address)
node, rest, hasNode := strings.Cut(address, "/")
if !hasNode {
rest, node = address, ""
}
dot := strings.Index(rest, ".")
if dot <= 0 || dot == len(rest)-1 || strings.Contains(rest, "/") {
return target{}, fmt.Errorf("%q is not an address. %s", address, grammar)
}
prefix, name := rest[:dot], rest[dot+1:]
if s := x.seat(prefix); s != nil {
verb := findTool(s.Verbs, name)
if verb == nil {
return target{}, fmt.Errorf("the seat %s has no verb %s; it has %s", prefix, name, toolNames(s.Verbs))
}
if s.Scope == "node" {
if node == "" {
return target{}, fmt.Errorf("%s is held once per machine: write <node>/%s — it is held on %s",
prefix, rest, orNobody(nodesOf(s.Holders)))
}
return target{Address: node + "/" + rest, Key: "seat:" + rest + "@" + node, Name: rest, Node: node, Tool: *verb, Seat: true}, nil
}
if node != "" {
return target{}, fmt.Errorf("%s is held once for the whole mesh: write %s, without a machine", prefix, rest)
}
return target{Address: rest, Key: "seat:" + rest, Name: rest, Tool: *verb, Seat: true}, nil
}
m := x.Modules[prefix]
if m == nil {
return target{}, fmt.Errorf("nothing in the mesh is called %s: no seat, and no module that answers tools. "+
"mesh_search finds a tool by words", prefix)
}
tool := findTool(m.Tools, name)
if tool == nil {
return target{}, fmt.Errorf("%s has no tool %s; it has %s", prefix, name, toolNames(m.Tools))
}
if node == "" {
if !m.Interchangeable {
return target{}, fmt.Errorf("%s keeps state on each machine it runs on, so a call names the machine: "+
"write <node>/%s — it runs on %s", prefix, rest, orNobody(m.On))
}
return target{Address: rest, Key: rest, Name: rest, Tool: *tool}, nil
}
if len(m.On) > 0 && !contains(m.On, node) {
return target{}, fmt.Errorf("%s does not run on %s; it runs on %s", prefix, node, orNobody(m.On))
}
return target{Address: node + "/" + rest, Key: rest + "@" + node, Name: rest, Node: node, Tool: *tool}, nil
}
func contains(xs []string, s string) bool {
for _, x := range xs {
if x == s {
return true
}
}
return false
}
func toolNames(ts []Tool) string {
names := make([]string, 0, len(ts))
for _, t := range ts {
names = append(names, t.Name)
}
sort.Strings(names)
return strings.Join(names, ", ")
}
func nodesOf(hs []holder) []string {
var out []string
for _, h := range hs {
if h.Node != "" && !contains(out, h.Node) {
out = append(out, h.Node)
}
}
sort.Strings(out)
return out
}
func orNobody(nodes []string) string {
if len(nodes) == 0 {
return "no machine the mesh knows of"
}
return strings.Join(nodes, ", ")
}
// firstLine is a description's first line, for a list.
func firstLine(s string) string {
s = strings.TrimSpace(s)
if i := strings.IndexAny(s, "\n"); i >= 0 {
s = s[:i]
}
if len(s) > 160 {
s = s[:157] + "…"
}
return s
}
// schemaWithoutNode is a tool's schema as an agent passes it: the machine is in the address.
func schemaWithoutNode(raw json.RawMessage) map[string]any {
schema := asSchema(raw)
out := map[string]any{}
for k, v := range schema {
out[k] = v
}
if p, ok := schema["properties"].(map[string]any); ok {
props := map[string]any{}
for k, v := range p {
if k != "node" {
props[k] = v
}
}
out["properties"] = props
}
if r, ok := schema["required"].([]any); ok {
var keep []any
for _, k := range r {
if k != "node" {
keep = append(keep, k)
}
}
if len(keep) == 0 {
delete(out, "required")
} else {
out["required"] = keep
}
}
return out
}
// answerText is a discovery verb's answer as MCP content: JSON, indented.
func answerText(v any) map[string]any {
var b strings.Builder
enc := json.NewEncoder(&b)
enc.SetEscapeHTML(false) // `<node>/…` is read by an agent, not embedded in a page
enc.SetIndent("", " ")
_ = enc.Encode(v)
return map[string]any{"content": []map[string]any{{"type": "text", "text": strings.TrimRight(b.String(), "\n")}}}
}
func failure(text string) map[string]any {
return map[string]any{"content": []map[string]any{{"type": "text", "text": text}}, "isError": true}
}
// discover answers one of the five.
func (s *Surface) discover(name string, args map[string]any) map[string]any {
x, err := s.index()
if err != nil {
return failure("the mesh's discovery failed: " + err.Error())
}
str := func(k string) string { v, _ := args[k].(string); return strings.TrimSpace(v) }
switch name {
case verbOverview:
type verbLine struct {
Address string `json:"address"`
Description string `json:"description"`
}
var mesh, node []map[string]any
for _, st := range x.Seats {
var verbs []verbLine
for _, v := range st.Verbs {
addr := st.Seat + "." + v.Name
if st.Scope == "node" {
addr = "<node>/" + addr
}
verbs = append(verbs, verbLine{addr, firstLine(v.Description)})
}
entry := map[string]any{"seat": st.Seat, "verbs": verbs}
if st.Scope == "node" {
entry["held on"] = nodesOf(st.Holders)
node = append(node, entry)
} else {
if h := nodesOf(st.Holders); len(h) > 0 {
entry["held on"] = h
}
mesh = append(mesh, entry)
}
}
modules := 0
tools := 0
for _, m := range x.Modules {
modules++
tools += len(m.Tools)
}
return answerText(map[string]any{
"seats of the mesh": mesh,
"seats every machine": node,
"machines": x.Machines,
"modules with tools": fmt.Sprintf("%d modules, %d tools — mesh_machine lists a machine's, mesh_search finds one", modules, tools),
"not answering": x.NotAnswering,
})
case verbMachine:
node := str("node")
if node == "" {
return failure("mesh_machine needs `node`: one of " + orNobody(x.Machines))
}
if !contains(x.Machines, node) {
return failure(fmt.Sprintf("the mesh knows no machine %q; it has %s", node, orNobody(x.Machines)))
}
var seats []map[string]any
for _, st := range x.Seats {
if st.Scope != "node" || !contains(nodesOf(st.Holders), node) {
continue
}
var verbs []string
for _, v := range st.Verbs {
verbs = append(verbs, node+"/"+st.Seat+"."+v.Name)
}
var by string
for _, h := range st.Holders {
if h.Node == node {
by = h.Module
}
}
seats = append(seats, map[string]any{"seat": st.Seat, "held by": by, "verbs": verbs})
}
var modules []map[string]any
names := make([]string, 0, len(x.Modules))
for n := range x.Modules {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
m := x.Modules[n]
if !contains(m.On, node) {
continue
}
var tools []map[string]string
for _, t := range m.Tools {
tools = append(tools, map[string]string{"address": node + "/" + m.Module + "." + t.Name, "does": firstLine(t.Description)})
}
entry := map[string]any{"module": m.Module, "tools": tools}
if m.Interchangeable {
entry["interchangeable"] = "any instance answers <module>.<tool> as well"
}
modules = append(modules, entry)
}
return answerText(map[string]any{"machine": node, "seats": seats, "modules": modules})
case verbSearch:
query := strings.ToLower(str("query"))
if query == "" {
return failure("mesh_search needs `query`: words to find, e.g. `postgres databases`")
}
words := strings.Fields(query)
type hit struct {
Address string `json:"address"`
Does string `json:"does"`
Also string `json:"also,omitempty"`
}
var hits []hit
matches := func(parts ...string) bool {
hay := strings.ToLower(strings.Join(parts, " "))
for _, w := range words {
if !strings.Contains(hay, w) {
return false
}
}
return true
}
for _, st := range x.Seats {
for _, v := range st.Verbs {
if !matches(st.Seat, v.Name, v.Description) {
continue
}
if st.Scope == "node" {
on := nodesOf(st.Holders)
first := "<node>"
also := ""
if len(on) > 0 {
first = on[0]
if len(on) > 1 {
also = "also on " + strings.Join(on[1:], ", ")
}
}
hits = append(hits, hit{first + "/" + st.Seat + "." + v.Name, firstLine(v.Description), also})
} else {
hits = append(hits, hit{st.Seat + "." + v.Name, firstLine(v.Description), ""})
}
}
}
names := make([]string, 0, len(x.Modules))
for n := range x.Modules {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
m := x.Modules[n]
for _, t := range m.Tools {
if !matches(m.Module, t.Name, t.Description) {
continue
}
switch {
case m.Interchangeable:
hits = append(hits, hit{m.Module + "." + t.Name, firstLine(t.Description), "any instance; on " + orNobody(m.On)})
case len(m.On) == 0:
hits = append(hits, hit{"<node>/" + m.Module + "." + t.Name, firstLine(t.Description), "the mesh places it on no machine"})
default:
also := ""
if len(m.On) > 1 {
also = "also on " + strings.Join(m.On[1:], ", ")
}
hits = append(hits, hit{m.On[0] + "/" + m.Module + "." + t.Name, firstLine(t.Description), also})
}
}
}
out := map[string]any{"matches": hits}
if len(hits) > searchCap {
out["matches"] = hits[:searchCap]
out["more"] = fmt.Sprintf("%d more; narrow the words", len(hits)-searchCap)
}
if len(hits) == 0 {
out["matches"] = []hit{}
out["hint"] = "nothing matched every word; try fewer words, or mesh_overview and mesh_machine to browse"
}
return answerText(out)
case verbDescribe:
t, err := resolve(x, str("address"))
if err != nil {
return failure(err.Error())
}
description := t.Tool.Description
if description == "" {
description = t.Name
}
return answerText(map[string]any{"address": t.Address, "description": description,
"arguments": schemaWithoutNode(t.Tool.Input)})
case verbCall:
t, err := resolve(x, str("address"))
if err != nil {
return failure(err.Error())
}
callArgs := map[string]any{}
if a, ok := args["arguments"].(map[string]any); ok {
for k, v := range a {
if k != "node" {
callArgs[k] = v
}
}
}
var got bus.Answered
if t.Seat {
got, err = s.conn.Ask(t.Key, callArgs, "")
} else {
got, err = callTool(s.conn, t.Key, callArgs, nil, x.Listing)
}
if err != nil {
return failure(whyItFailed(t.Address, err))
}
content := []map[string]any{{"type": "text", "text": pretty(got.Result)}}
if got.Node != "" {
content = append(content, map[string]any{"type": "text", "text": "answered by " + got.Node})
}
return map[string]any{"content": content}
}
return failure("no discovery verb " + name)
}
+232
View File
@@ -0,0 +1,232 @@
package console
import (
"encoding/json"
"fmt"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/announce"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
// text is a tool result's first text, and whether it was an error.
func text(t *testing.T, reply map[string]any) (string, bool) {
t.Helper()
result, ok := reply["result"].(map[string]any)
if !ok {
t.Fatalf("no result: %v", reply)
}
content := result["content"].([]any)
isErr, _ := result["isError"].(bool)
var parts []string
for _, c := range content {
parts = append(parts, c.(map[string]any)["text"].(string))
}
return strings.Join(parts, "\n"), isErr
}
func call(t *testing.T, endpoint, tool string, args map[string]any) (string, bool) {
t.Helper()
body, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 9, "method": "tools/call",
"params": map[string]any{"name": tool, "arguments": args}})
return text(t, post(t, endpoint, string(body)))
}
// novox/hq ADR 0195: the console announces five tools, and everything the mesh answers is reached
// through them by one address per layer.
func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
was := IndexKept
IndexKept = 0 // every discovery asks the mesh, so a module arriving mid-test is found
t.Cleanup(func() { IndexKept = was })
mesh := mt.New(t)
// alpha: interchangeable (the mesh issued it a plain subject); beta: state on its machine, holds
// the node-shelf seat there.
mesh.Issue(t, mt.MembershipOf("alpha", "desk", true, nil))
mesh.Issue(t, mt.MembershipOf("beta", "desk", false, map[string][]string{"node-shelf": {"list", "clear"}}))
nodeTools := connect(t, "node-tools", "desk")
stop, err := runtime.Run(nodeTools, []runtime.Served{
{Module: "alpha", Entrypoints: []string{mt.Fixture("many-alpha.serve.mjs")}},
{Module: "beta", Entrypoints: []string{mt.Fixture("many-beta.serve.mjs")}},
}, nil, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
// Nothing may be asked for a roster or a module's `tools` any more (ADR 0197): counted.
var asked atomic.Int32
watcher := connect(t, "watcher", "")
for _, subject := range []string{"mesh.mod.*.tool.tools", "mesh.mod.*.tool.tools.*", "mesh.mod.mesh-catalog.>"} {
stop, err := watcher.Raw(subject, func(string, []byte) []byte { asked.Add(1); return nil })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
// The controller: its records as JSON, as its seat verbs answer them, and its own seat's verbs
// announced on the bus like every runtime's.
controller := connect(t, "mesh-controller", "bench")
out := func(s string) map[string]any { return map[string]any{"output": s, "ok": true} }
serve := func(verb string, answer func() any) {
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
return answer(), nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
serve("nodes", func() any {
return out("the bus's user list leaves out 2 user(s)\n" +
`[{"name":"bench","heard":"3m ago","mode":"converged","id":"1f2e"},{"name":"desk","heard":"here","mode":"converged","id":"9a8b"}]`)
})
var gammaOn atomic.Value
gammaOn.Store("[]")
serve("modules", func() any {
return out(fmt.Sprintf(`[{"module":"alpha","on":["desk"],"tools":true},{"module":"beta","on":["desk"],"tools":true},`+
`{"module":"gamma","on":%s,"tools":true},{"module":"delta","on":["desk"],"tools":false},`+
`{"module":"epsilon","on":["bench"],"tools":true},{"module":"mesh-controller","on":["bench"],"tools":true}]`, gammaOn.Load().(string)))
})
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-controller", ID: "bench"}, func() []announce.Endpoint {
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller", Scope: "mesh",
// As the live controller announces: a mesh seat's holder names no machine.
Tool: "nodes", Node: "", Description: "Every machine the mesh knows.", Schema: json.RawMessage(`{}`),
Subject: "mesh.seat.mesh-controller.tool.nodes"}}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopAnn)
controller.Flush()
watcher.Flush()
up, err := Serve(NewSurface(nodeTools, "desk.node-tools"), "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer up.Close()
endpoint := "http://" + up.Address + "/mcp"
// Five tools, nothing else.
listed := post(t, endpoint, `{"jsonrpc":"2.0","id":2,"method":"tools/list"}`)["result"].(map[string]any)
var names []string
for _, x := range listed["tools"].([]any) {
name := x.(map[string]any)["name"].(string)
names = append(names, name)
for _, r := range name {
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '_' || r == '-') || len(name) > 64 {
t.Errorf("%q is not a name the API takes", name)
}
}
}
if got := strings.Join(names, ","); got != "mesh_overview,mesh_machine,mesh_search,mesh_describe,mesh_call" {
t.Errorf("announced %s", got)
}
// The overview names the mesh's seats, the machines' seats and the machines.
overview, isErr := call(t, endpoint, "mesh_overview", nil)
if isErr || !strings.Contains(overview, "mesh-controller.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
!strings.Contains(overview, `"bench"`) || !strings.Contains(overview, `"desk"`) {
t.Errorf("overview: %s", overview)
}
// Silence is named only where tools should have answered: epsilon declares tools on bench and
// nothing there announced it; delta declares none and is never a name.
if strings.Contains(overview, "mesh-controller on bench") {
t.Errorf("a mesh seat's holder that announced no machine is called silent on its machine:\n%s", overview)
}
if !strings.Contains(overview, "epsilon on bench") || strings.Contains(overview, "delta") {
t.Errorf("not answering: %s", overview)
}
machine, isErr := call(t, endpoint, "mesh_machine", map[string]any{"node": "desk"})
if isErr || !strings.Contains(machine, "desk/node-shelf.list") || !strings.Contains(machine, "desk/beta.three") ||
!strings.Contains(machine, "desk/alpha.one") {
t.Errorf("machine: %s", machine)
}
// A mesh seat, a node seat, an assignment and an interchangeable module, each by address.
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-controller.nodes"}); isErr || !strings.Contains(got, "converged") {
t.Errorf("mesh seat: %s", got)
}
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "desk/node-shelf.list"}); isErr || !strings.Contains(got, `"a"`) {
t.Errorf("node seat: %s", got)
}
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "desk/beta.three"}); isErr ||
!strings.Contains(got, `"beta": 3`) || !strings.Contains(got, "answered by desk") {
t.Errorf("assignment: %s", got)
}
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "alpha.one"}); isErr || !strings.Contains(got, `"alpha": 1`) {
t.Errorf("interchangeable module: %s", got)
}
// Refused, by name, where the address does not say enough or says the wrong thing.
for address, want := range map[string]string{
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
"desk/mesh-controller.nodes": "held once for the whole mesh",
"bench/beta.three": "beta does not run on bench; it runs on desk",
"nonsense": "is not an address",
} {
got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": address})
if !isErr || !strings.Contains(got, want) {
t.Errorf("%s: %s (want %q)", address, got, want)
}
}
// Described without `node`: the address carries the machine.
described, isErr := call(t, endpoint, "mesh_describe", map[string]any{"address": "desk/beta.three"})
if isErr || strings.Contains(described, `"node"`) || !strings.Contains(described, `"address": "desk/beta.three"`) {
t.Errorf("describe: %s", described)
}
// Search finds across the layers; a module that starts serving after the first answer is found.
if got, _ := call(t, endpoint, "mesh_search", map[string]any{"query": "shelf"}); !strings.Contains(got, "desk/node-shelf.list") {
t.Errorf("search a node seat: %s", got)
}
if got, _ := call(t, endpoint, "mesh_search", map[string]any{"query": "gamma"}); strings.Contains(got, "gamma.given") {
t.Fatalf("gamma was found before it served: %s", got)
}
mesh.Issue(t, mt.MembershipOf("gamma", "desk", false, nil))
gammaOn.Store(`["desk"]`)
late := connect(t, "node-tools", "desk")
stopLate, err := runtime.Run(late, []runtime.Served{{Module: "gamma", Entrypoints: []string{mt.Fixture("env-gamma.serve.mjs")}}},
nil, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stopLate()
var found string
for i := 0; i < 30; i++ {
found, _ = call(t, endpoint, "mesh_search", map[string]any{"query": "gamma given"})
if strings.Contains(found, "desk/gamma.given") {
break
}
time.Sleep(100 * time.Millisecond)
}
if !strings.Contains(found, "desk/gamma.given") {
t.Errorf("a module that arrived later was not found: %s", found)
}
if n := asked.Load(); n != 0 {
t.Errorf("discovery asked a roster or a module's tools %d time(s); it asks the bus", n)
}
// The old names still answer, unannounced.
if got, isErr := call(t, endpoint, "alpha.one", nil); isErr || !strings.Contains(got, `"alpha": 1`) {
t.Errorf("an old name: %s", got)
}
}
func TestTheControllersRecordsAreReadAsJSON(t *testing.T) {
if got := jsonIn("a warning printed first\n[{\"name\":\"ace\"}]"); got != `[{"name":"ace"}]` {
t.Errorf("an array after a warning: %q", got)
}
if got := jsonIn(`{"seats":[]}`); got != `{"seats":[]}` {
t.Errorf("an object: %q", got)
}
}
+139
View File
@@ -0,0 +1,139 @@
package console
import (
"encoding/json"
"errors"
"fmt"
"regexp"
"strings"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// Tool is a tool as its module — or, for a role's tool, the mesh's records — describes it.
type Tool struct {
Module string
Name string
Description string
Input json.RawMessage
Seat bool
Scope string
Subjects []string
}
// Listing is what the mesh could say about its tools; silence is named, never dropped (design 34 §3).
type Listing struct {
Tools []Tool
NotAnswering []string
}
// Seats is the seats and their verbs, so `<seat>.<verb>` resolves to the role.
type Seats map[string]map[string]bool
func seatsIn(l *Listing) Seats {
out := Seats{}
for _, t := range l.Tools {
if !t.Seat {
continue
}
if out[t.Module] == nil {
out[t.Module] = map[string]bool{}
}
out[t.Module][t.Name] = true
}
return out
}
// toolKey is the key a call uses: a role's when the prefix is a seat declaring that verb.
func toolKey(name string, seats Seats) string {
if strings.HasPrefix(name, "seat:") {
return name
}
dot := strings.Index(name, ".")
if dot < 0 {
return name
}
if seats[name[:dot]][name[dot+1:]] {
return "seat:" + name
}
return name
}
// toolsOn is the flat catalogue (MESH_CONSOLE_FLAT=1): what announced itself on the bus, every
// tool and seat verb, and the assignments with tools that did not (novox/hq ADR 0197).
func toolsOn(conn *bus.Conn) (*Listing, error) {
x, err := indexOn(conn)
if err != nil {
return nil, err
}
return x.Listing, nil
}
// callTool calls `<module>.<tool>[@<node>]`, on the subject the listing names for it when it names one.
func callTool(conn *bus.Conn, key string, args any, seats Seats, l *Listing) (bus.Answered, error) {
name, node, _ := strings.Cut(key, "@")
if !strings.Contains(name, ".") {
return bus.Answered{}, fmt.Errorf("%q does not name a tool: write <module>.<tool>, as `mesh tools` lists "+
"them, or <module>.<tool>@<node> for the instance on one machine", key)
}
resolved := toolKey(name, seats)
if node != "" {
resolved += "@" + node
}
return conn.Ask(resolved, args, subjectListed(name, node, l))
}
func subjectListed(name, node string, l *Listing) string {
if l == nil {
return ""
}
dot := strings.Index(name, ".")
module, tool := name[:dot], name[dot+1:]
for _, t := range l.Tools {
if t.Module == module && t.Name == tool && !t.Seat {
if len(t.Subjects) == 0 {
return ""
}
if node == "" {
return t.Subjects[0]
}
for _, s := range t.Subjects {
if strings.HasSuffix(s, "."+node) {
return s
}
}
return ""
}
}
return ""
}
var (
noResponders = regexp.MustCompile(`(?i)no responders|503`)
refused = regexp.MustCompile(`(?i)permissions violation|authorization`)
timedOut = regexp.MustCompile(`(?i)timeout`)
)
// whyItFailed says why a call failed, so the remedy is in the words.
func whyItFailed(key string, err error) string {
if err == nil {
err = errors.New("failed")
}
msg := err.Error()
switch {
case noResponders.MatchString(msg):
extra := ""
if strings.HasPrefix(key, "seat:") {
extra = ", or nothing holds that seat"
}
return "nothing serves " + key + ". The module may not be assigned to any machine, or it is down" +
extra + " — `mesh tools` lists what answered."
case refused.MatchString(msg):
return "this account may not call " + key + ". What it may call was fixed when it was issued — a " +
"person's by `operator issue`, the console's by its manifest."
case timedOut.MatchString(msg):
return key + " did not answer in time. Something is serving it, so this is the tool being slow " +
"rather than absent."
}
return key + " failed: " + msg
}
+131
View File
@@ -0,0 +1,131 @@
package console
import (
"bytes"
"encoding/json"
"io"
"net/http"
"strings"
"testing"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
func connect(t *testing.T, module, node string) *bus.Conn {
t.Helper()
c, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: module, Node: node})
if err != nil {
t.Fatal(err)
}
c.Logf = func(string, ...any) {}
t.Cleanup(c.Close)
return c
}
func post(t *testing.T, endpoint string, body string) map[string]any {
t.Helper()
res, err := http.Post(endpoint, "application/json", bytes.NewBufferString(body))
if err != nil {
t.Fatal(err)
}
defer res.Body.Close()
raw, _ := io.ReadAll(res.Body)
var out map[string]any
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatalf("%d %s", res.StatusCode, raw)
}
return out
}
// As node-tools, the runtime serves the bundles and is the console on loopback: the listing is what
// the modules and the mesh's records answered, and a call reaches the module on the machine named.
func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "desk", true, nil))
mesh.Issue(t, mt.MembershipOf("beta", "desk", false, map[string][]string{"node-shelf": {"list", "clear"}}))
nodeTools := connect(t, "node-tools", "desk")
stop, err := runtime.Run(nodeTools, []runtime.Served{
{Module: "alpha", Entrypoints: []string{mt.Fixture("many-alpha.serve.mjs")}},
{Module: "beta", Entrypoints: []string{mt.Fixture("many-beta.serve.mjs")}},
}, nil, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
// The controller's records (ADR 0197): ghost declares tools on desk and nothing announces it.
controller := connect(t, "mesh-controller", "")
stopSeat, _ := controller.HandleSubject("mesh.seat.mesh-controller.tool.modules", func(json.RawMessage) (any, error) {
return map[string]any{"ok": true, "output": `[{"module":"alpha","on":["desk"],"tools":true},` +
`{"module":"beta","on":["desk"],"tools":true},{"module":"ghost","on":["desk"],"tools":true}]`}, nil
})
defer stopSeat()
controller.Flush()
flat := NewSurface(nodeTools, "desk.node-tools")
flat.Flat = true // the whole catalogue, as before ADR 0195: still reachable, no longer announced
up, err := Serve(flat, "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer up.Close()
endpoint := "http://" + up.Address + "/mcp"
init := post(t, endpoint, `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{}}}`)
if !strings.Contains(init["result"].(map[string]any)["instructions"].(string), "reached as desk.node-tools") {
t.Errorf("initialize: %v", init)
}
listed := post(t, endpoint, `{"jsonrpc":"2.0","id":2,"method":"tools/list"}`)["result"].(map[string]any)
var names []string
for _, x := range listed["tools"].([]any) {
names = append(names, x.(map[string]any)["name"].(string))
}
if got := strings.Join(names, ","); got != "alpha.one,alpha.two,beta.five,beta.four,beta.three,node-shelf.clear,node-shelf.list" {
t.Errorf("listed %s", got)
}
if got := listed["_meta"].(map[string]any)["notAnswering"]; len(got.([]any)) != 1 || got.([]any)[0] != "ghost on desk" {
t.Errorf("not answering: %v", got)
}
for _, x := range listed["tools"].([]any) {
tool := x.(map[string]any)
if tool["name"] == "node-shelf.list" {
schema := tool["inputSchema"].(map[string]any)
if req, _ := schema["required"].([]any); len(req) != 1 || req[0] != "node" {
t.Errorf("a node seat's verb does not require node: %v", schema)
}
}
}
called := post(t, endpoint, `{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"alpha.one","arguments":{"node":"desk"}}}`)
content := called["result"].(map[string]any)["content"].([]any)
var got map[string]any
_ = json.Unmarshal([]byte(content[0].(map[string]any)["text"].(string)), &got)
if got["alpha"] != float64(1) || content[1].(map[string]any)["text"] != "answered by desk" {
t.Errorf("called: %v", called)
}
seat := post(t, endpoint, `{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"node-shelf.list","arguments":{"node":"desk"}}}`)
if text := seat["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string); !strings.Contains(text, `"a"`) {
t.Errorf("seat verb: %v", seat)
}
refused := post(t, endpoint, `{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"node-shelf.list","arguments":{}}}`)
if refused["error"] == nil {
t.Errorf("a node seat's verb was called without its machine: %v", refused)
}
absent := post(t, endpoint, `{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"ghost.boo","arguments":{}}}`)
result := absent["result"].(map[string]any)
if result["isError"] != true || !strings.Contains(result["content"].([]any)[0].(map[string]any)["text"].(string), "nothing serves ghost.boo") {
t.Errorf("an absent tool: %v", absent)
}
res, err := http.Post(endpoint, "application/json", strings.NewReader(`{"jsonrpc":"2.0","method":"notifications/initialized"}`))
if err != nil || res.StatusCode != 202 {
t.Errorf("a notification: %v %v", res, err)
}
}
func TestTheConsoleListensOnLoopbackAndNowhereElse(t *testing.T) {
if _, err := Serve(NewSurface(nil, "x"), "0.0.0.0:0"); err == nil || !strings.Contains(err.Error(), "loopback and nowhere else") {
t.Errorf("a non-loopback console was not refused: %v", err)
}
}
+125
View File
@@ -0,0 +1,125 @@
package console
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"strconv"
"strings"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
// bodyLimit is the most a request body may be.
const bodyLimit = 1 << 20
var loopback = map[string]bool{"127.0.0.1": true, "::1": true, "localhost": true, "[::1]": true}
// Listening is a console that listens: where, with the port the machine gave, and how to stop it.
type Listening struct {
Address string
Close func() error
}
// Serve listens on host:port, refused unless the host is loopback — said before binding, so a
// console that would open to a network is a startup failure (ADR 0152).
func Serve(s *Surface, listen string) (*Listening, error) {
at := strings.LastIndex(listen, ":")
if at < 0 {
return nil, fmt.Errorf("%q is not host:port", listen)
}
host, portText := listen[:at], listen[at+1:]
if !loopback[host] {
return nil, fmt.Errorf(`the console listens on loopback and nowhere else (novox/hq ADR 0152): %q is not this `+
"machine's own address — whoever is on the machine owns the mesh there, and nobody else may reach this", host)
}
port, err := strconv.Atoi(portText)
if err != nil || port < 0 || port > 65535 {
return nil, fmt.Errorf("%q is not a port", portText)
}
ln, err := net.Listen("tcp", net.JoinHostPort(strings.Trim(host, "[]"), portText))
if err != nil {
return nil, err
}
server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { route(w, r, s) })}
go func() { _ = server.Serve(ln) }()
bound := ln.Addr().(*net.TCPAddr).Port
return &Listening{Address: host + ":" + strconv.Itoa(bound), Close: func() error { return server.Shutdown(context.Background()) }}, nil
}
func route(w http.ResponseWriter, r *http.Request, s *Surface) {
switch r.URL.Path {
case "/":
w.Header().Set("content-type", "text/plain; charset=utf-8")
_, _ = io.WriteString(w, "the mesh's console: MCP over HTTP at POST /mcp (novox/hq design 34)\n")
return
case "/mcp":
default:
writeJSON(w, 404, map[string]any{"error": "the console serves /mcp and nothing else"})
return
}
switch r.Method {
case http.MethodPost:
case http.MethodDelete:
w.WriteHeader(204) // no session to end
return
default:
w.Header().Set("allow", "POST, DELETE")
w.WriteHeader(405)
return
}
body, err := io.ReadAll(io.LimitReader(r.Body, bodyLimit+1))
if err != nil || len(body) > bodyLimit {
if err == nil {
err = fmt.Errorf("the request is larger than %d bytes", bodyLimit)
}
writeJSON(w, 413, map[string]any{"jsonrpc": "2.0", "id": nil, "error": map[string]any{"code": -32600, "message": err.Error()}})
return
}
trimmed := strings.TrimSpace(string(body))
if strings.HasPrefix(trimmed, "[") {
var batch []Request
if err := json.Unmarshal(body, &batch); err != nil {
writeJSON(w, 400, map[string]any{"jsonrpc": "2.0", "id": nil, "error": map[string]any{"code": -32700, "message": "the body is not JSON"}})
return
}
replies := []*Reply{}
for _, req := range batch {
if reply := s.Handle(req); reply != nil {
replies = append(replies, reply)
}
}
if len(replies) == 0 {
w.WriteHeader(202)
return
}
writeJSON(w, 200, replies)
return
}
var req Request
if err := json.Unmarshal(body, &req); err != nil {
writeJSON(w, 400, map[string]any{"jsonrpc": "2.0", "id": nil, "error": map[string]any{"code": -32700, "message": "the body is not JSON"}})
return
}
reply := s.Handle(req)
if reply == nil {
w.WriteHeader(202)
return
}
writeJSON(w, 200, reply)
}
func writeJSON(w http.ResponseWriter, status int, body any) {
text, err := wire.Marshal(body)
if err != nil {
text, _ = json.Marshal(map[string]any{"error": errors.New("unencodable answer").Error()})
}
w.Header().Set("content-type", "application/json; charset=utf-8")
w.Header().Set("content-length", strconv.Itoa(len(text)))
w.WriteHeader(status)
_, _ = w.Write(text)
}
+297
View File
@@ -0,0 +1,297 @@
// Package console is the mesh's tools as an MCP server on a machine's loopback (novox/hq design 34,
// ADR 0152, ADR 0175 §6): the Go port of node-tools' http.ts, mcp.ts and client.ts. A thin adapter:
// every tool listed is one a module answered for, the schema is the module's, the answer the module's.
package console
import (
"encoding/json"
"os"
"strings"
"sync"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// Protocol is the MCP version spoken to an agent host.
const Protocol = "2025-03-26"
// ListingKept is how long a fetched tool list is kept before the modules are asked again.
var ListingKept = 30 * time.Second
// Request is one JSON-RPC message from a host.
type Request struct {
JSONRPC string `json:"jsonrpc"`
ID json.RawMessage `json:"id,omitempty"`
Method string `json:"method"`
Params json.RawMessage `json:"params,omitempty"`
}
// Reply is one JSON-RPC answer.
type Reply struct {
JSONRPC string `json:"jsonrpc"`
ID json.RawMessage `json:"id"`
Result any `json:"result,omitempty"`
Error *RPCError `json:"error,omitempty"`
}
// RPCError is a protocol-level refusal.
type RPCError struct {
Code int `json:"code"`
Message string `json:"message"`
}
// Surface answers MCP requests over one bus connection, as one account.
type Surface struct {
conn *bus.Conn
who string
mu sync.Mutex
known *Listing
at time.Time
idx *index
idxAt time.Time
// Flat announces the whole catalogue, as the console did before ADR 0195: for a person reading
// it or a client that wants it. Off by default; MESH_CONSOLE_FLAT=1 turns it on.
Flat bool
}
// NewSurface is the surface over a connection, as `who`.
func NewSurface(conn *bus.Conn, who string) *Surface {
return &Surface{conn: conn, who: who, Flat: os.Getenv("MESH_CONSOLE_FLAT") == "1"}
}
func (s *Surface) listing() (*Listing, error) {
s.mu.Lock()
if s.known != nil && time.Since(s.at) <= ListingKept {
l := s.known
s.mu.Unlock()
return l, nil
}
s.mu.Unlock()
l, err := toolsOn(s.conn)
if err != nil {
return nil, err
}
s.mu.Lock()
s.known, s.at = l, time.Now()
s.mu.Unlock()
return l, nil
}
func isNotification(id json.RawMessage) bool {
t := strings.TrimSpace(string(id))
return t == "" || t == "null"
}
func answer(id json.RawMessage, result any) *Reply {
return &Reply{JSONRPC: "2.0", ID: idOrNull(id), Result: result}
}
func refuse(id json.RawMessage, code int, message string) *Reply {
return &Reply{JSONRPC: "2.0", ID: idOrNull(id), Error: &RPCError{Code: code, Message: message}}
}
func idOrNull(id json.RawMessage) json.RawMessage {
if isNotification(id) {
return json.RawMessage("null")
}
return id
}
// Handle answers one request; nil for a notification, which expects none.
func (s *Surface) Handle(r Request) *Reply {
notification := isNotification(r.ID)
switch r.Method {
case "initialize":
return answer(r.ID, map[string]any{
"protocolVersion": Protocol,
"capabilities": map[string]any{"tools": map[string]any{}},
"serverInfo": map[string]any{"name": "mesh", "version": "1"},
"instructions": s.instructions(),
})
case "notifications/initialized":
return nil
case "ping":
if notification {
return nil
}
return answer(r.ID, map[string]any{})
case "tools/list":
if !s.Flat {
return answer(r.ID, map[string]any{"tools": discovery()})
}
l, err := s.listing()
if err != nil {
return refuse(r.ID, -32603, "the mesh's discovery failed: "+err.Error())
}
tools := make([]map[string]any, 0, len(l.Tools))
for _, t := range l.Tools {
var schema map[string]any
switch {
case t.Seat && t.Scope != "node":
schema = asSchema(t.Input)
case t.Seat:
schema = withNode(asSchema(t.Input), "the machine whose seat answers; required, the seat is held once per machine", true)
default:
schema = withNode(asSchema(t.Input), "", false)
}
description := t.Description
if description == "" {
description = t.Name + ", served by " + t.Module
}
tools = append(tools, map[string]any{"name": t.Module + "." + t.Name, "description": description, "inputSchema": schema})
}
return answer(r.ID, map[string]any{"tools": tools, "_meta": map[string]any{"notAnswering": l.NotAnswering}})
case "tools/call":
var p struct {
Name string `json:"name"`
Arguments map[string]any `json:"arguments"`
}
_ = json.Unmarshal(r.Params, &p)
if isDiscovery(p.Name) {
if p.Arguments == nil {
p.Arguments = map[string]any{}
}
return answer(r.ID, s.discover(p.Name, p.Arguments))
}
args := map[string]any{}
for k, v := range p.Arguments {
args[k] = v
}
l, _ := s.listing()
var roles Seats
if l != nil {
roles = seatsIn(l)
}
bare, _, _ := strings.Cut(p.Name, "@")
isSeatVerb := roles != nil && strings.HasPrefix(toolKey(bare, roles), "seat:")
nodeScoped := false
if isSeatVerb && l != nil {
for _, t := range l.Tools {
if t.Seat && t.Scope == "node" && t.Module+"."+t.Name == bare {
nodeScoped = true
}
}
}
takesNode := !isSeatVerb || nodeScoped
node := ""
if takesNode {
if n, ok := args["node"].(string); ok {
node = n
}
delete(args, "node")
}
if nodeScoped && node == "" && !strings.Contains(p.Name, "@") {
return refuse(r.ID, -32602, p.Name+" is a machine's seat's verb: name the machine with `node`")
}
name := p.Name
if node != "" && !strings.Contains(p.Name, "@") {
name = p.Name + "@" + node
}
got, err := callTool(s.conn, name, args, roles, l)
if err != nil {
return answer(r.ID, map[string]any{
"content": []map[string]any{{"type": "text", "text": whyItFailed(name, err)}},
"isError": true,
})
}
content := []map[string]any{{"type": "text", "text": pretty(got.Result)}}
if got.Node != "" {
content = append(content, map[string]any{"type": "text", "text": "answered by " + got.Node})
}
return answer(r.ID, map[string]any{"content": content})
}
if notification {
return nil
}
return refuse(r.ID, -32601, "mesh's MCP surface has no "+r.Method)
}
// pretty is a module's answer as JSON text, indented as JSON.stringify(result, null, 2) writes it.
func pretty(raw json.RawMessage) string {
if len(raw) == 0 {
return "null"
}
var v any
if json.Unmarshal(raw, &v) != nil {
return string(raw)
}
b, err := json.MarshalIndent(v, "", " ")
if err != nil {
return string(raw)
}
return strings.NewReplacer(`<`, "<", `>`, ">", `&`, "&").Replace(string(b))
}
// asSchema is a module's declared input as a JSON schema: wrapped when it is a bare map of
// properties, passed through when it is a schema, empty when nothing was declared.
func asSchema(raw json.RawMessage) map[string]any {
var given map[string]any
if json.Unmarshal(raw, &given) != nil || given == nil {
return map[string]any{"type": "object", "properties": map[string]any{}}
}
if given["type"] == "object" {
return given
}
if _, has := given["properties"]; has {
return given
}
if len(given) == 0 {
return map[string]any{"type": "object", "properties": map[string]any{}}
}
return map[string]any{"type": "object", "properties": given}
}
// withNode adds the optional — or, for a node seat, required — `node` argument (ADR 0159).
func withNode(schema map[string]any, description string, required bool) map[string]any {
properties := map[string]any{}
if p, ok := schema["properties"].(map[string]any); ok {
for k, v := range p {
properties[k] = v
}
}
if _, has := properties["node"]; !has {
if description == "" {
description = "the machine to ask, when this module runs on several; else whichever answers, and the answer says which"
}
properties["node"] = map[string]any{"type": "string", "description": description}
}
out := map[string]any{}
for k, v := range schema {
out[k] = v
}
out["type"] = "object"
out["properties"] = properties
if required {
have := []any{}
if r, ok := schema["required"].([]any); ok {
have = r
}
hasNode := false
for _, x := range have {
hasNode = hasNode || x == "node"
}
if !hasNode {
have = append(have, "node")
}
out["required"] = have
}
return out
}
// instructions is what an agent host is told about this surface when it connects.
func (s *Surface) instructions() string {
if s.Flat {
return "These are the tools of a Novox mesh, reached as " + s.who + ". Every call goes to the module " +
"that serves it; what may be called was fixed when this account was issued, so a " +
"refusal means the account, not the tool. The list is what the running modules " +
"answered, plus every role's tools from the mesh's records — the mesh's own verbs " +
"(mesh-controller.status, .push, .assign …) among them; a module that did not answer " +
"is named in the list's _meta and can still be called by <module>.<tool>."
}
return "The tools of a Novox mesh, reached as " + s.who + ", found by address rather than listed " +
"whole (novox/hq ADR 0195). mesh_overview shows the mesh's seats and machines; mesh_machine one " +
"machine's seats and modules; mesh_search finds a tool by words; mesh_describe gives one tool's " +
"arguments; mesh_call calls it. " + grammar + " What may be called was fixed when this account " +
"was issued, so a refusal means the account, not the tool."
}
+511
View File
@@ -0,0 +1,511 @@
// Package launch starts a bundle the runtime serves and speaks MCP over stdio to it (novox/hq ADR
// 0188, ADR 0193): `initialize`, `tools/list` once, `tools/call` per call. A Go binary, a Python
// script and a Node launcher are the same thing here: an executable that answers those. The runtime
// knows no language; it starts the path it is given.
package launch
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"regexp"
"strings"
"sync"
"syscall"
"time"
"golang.org/x/sys/unix"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
// Protocol is the MCP version spoken to a bundle.
const Protocol = "2025-03-26"
// How long a child has for its handshake, and a call before the caller is told it is slow.
var (
HandshakeTimeout = 10 * time.Second
CallTimeout = 30 * time.Second
)
// Tool is one tool a launched bundle listed, and how to call it.
type Tool struct {
Name string
Description string
Input json.RawMessage
Run func(args json.RawMessage) (json.RawMessage, error)
}
// Registration is the tools a bundle listed under one name: its module's, or a seat's.
type Registration struct {
Module string
Tools []Tool
}
// Bus is what a launched bundle reaches the mesh through (novox/hq ADR 0193, ADR 0198): the runtime
// publishes, asks and subscribes on the module's behalf. Subscribe is asked each time the module's
// code subscribes; the runtime binds the module's consumer once and calls deliver for every event,
// acknowledging it on the bus when deliver returns nil.
type Bus interface {
Publish(params json.RawMessage) error
Ask(params json.RawMessage) (json.RawMessage, error)
Subscribe(deliver func(envelope json.RawMessage) error) error
}
// EventTimeout bounds how long a bundle has to handle one event before it is offered again.
var EventTimeout = 2 * time.Minute
// Restart backoff: a bundle that exits is started again at once, then after growing pauses while it
// keeps exiting, back to at once once it has run a while.
var (
RestartFirst = 500 * time.Millisecond
RestartMost = 30 * time.Second
RestartSettle = time.Minute
)
// Launched is a running bundle: what it registered, and how to stop it.
type Launched struct {
Registrations []Registration
Stop func()
}
// Executable says whether an entrypoint can be started: a bundle the runtime serves is executable,
// and one that is not was not built to be served (ADR 0193).
func Executable(path string) bool {
info, err := os.Stat(path)
if err != nil || info.IsDir() {
return false
}
return info.Mode().Perm()&0o111 != 0
}
type message struct {
JSONRPC string `json:"jsonrpc,omitempty"`
ID json.RawMessage `json:"id,omitempty"`
Method string `json:"method,omitempty"`
Params json.RawMessage `json:"params,omitempty"`
Result json.RawMessage `json:"result,omitempty"`
Error *struct {
Code int `json:"code"`
Message string `json:"message"`
} `json:"error,omitempty"`
}
type child struct {
cmd *exec.Cmd
stdin io.WriteCloser
writeMu sync.Mutex
mu sync.Mutex
pending map[int64]chan message
next int64
dead chan struct{}
why error
}
func (c *child) write(m any) error {
b, err := wire.Marshal(m)
if err != nil {
return err
}
c.writeMu.Lock()
defer c.writeMu.Unlock()
_, err = c.stdin.Write(append(b, '\n'))
return err
}
var stackLine = regexp.MustCompile(`^\s+at\s`)
// Start launches one bundle and learns its tools. It fails when the child cannot be started or does
// not complete the handshake. A child that exits later is started again on its next call.
func Start(module, entry string, env []string, mesh Bus, logf func(string, ...any)) (*Launched, error) {
var mu sync.Mutex
var current *child
// The child that last subscribed is the one events are handed to: a restarted child subscribes
// again as its code is imported, and from then on the module's events are its.
var subscriber *child
stopped := false
deliver := func(envelope json.RawMessage) error {
mu.Lock()
c := subscriber
mu.Unlock()
if c == nil {
return errors.New(module + "'s bundle is not running to take its events")
}
_, err := c.ask(module, "mesh/event", map[string]any{"envelope": envelope}, EventTimeout)
return err
}
var restart func(after time.Duration)
var startedAt time.Time
pause := RestartFirst
start := func() (*child, error) {
cmd := exec.Command(entry)
cmd.Env = env
stdin, err := cmd.StdinPipe()
if err != nil {
return nil, err
}
stdout, err := cmd.StdoutPipe()
if err != nil {
return nil, err
}
stderr, err := cmd.StderrPipe()
if err != nil {
return nil, err
}
if err := cmd.Start(); err != nil {
return nil, err
}
c := &child{cmd: cmd, stdin: stdin, pending: map[int64]chan message{}, next: 1, dead: make(chan struct{})}
var lastSaid string
var saidMu sync.Mutex
stderrDone := make(chan struct{})
go func() {
defer close(stderrDone)
scan := bufio.NewScanner(stderr)
scan.Buffer(make([]byte, 64*1024), 1<<20)
for scan.Scan() {
line := scan.Text()
if strings.TrimSpace(line) == "" {
continue
}
logf("[%s] %s", module, line)
if !stackLine.MatchString(line) && !strings.HasPrefix(line, "Node.js v") {
saidMu.Lock()
lastSaid = strings.TrimSpace(line)
saidMu.Unlock()
}
}
}()
go func() {
scan := bufio.NewScanner(stdout)
scan.Buffer(make([]byte, 64*1024), 16<<20)
for scan.Scan() {
line := strings.TrimSpace(scan.Text())
if line == "" {
continue
}
var m message
if err := json.Unmarshal([]byte(line), &m); err != nil {
if len(line) > 120 {
line = line[:120]
}
logf("[mesh-tools] %s's bundle said something that is not a reply: %s", module, line)
continue
}
// The bundle asks the runtime (ADR 0193, ADR 0198): to emit, to call a tool, to hand it the
// module's events. Each on the module's behalf, answered when done; nothing else.
if m.Method != "" {
go func(m message) {
var result any = map[string]any{}
var err error
switch m.Method {
case "mesh/publish":
err = mesh.Publish(m.Params)
case "mesh/ask":
var asked json.RawMessage
if asked, err = mesh.Ask(m.Params); err == nil {
result = asked
}
case "mesh/subscribe":
mu.Lock()
subscriber = c
mu.Unlock()
err = mesh.Subscribe(deliver)
default:
if len(m.ID) > 0 {
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID,
"error": map[string]any{"code": -32601, "message": "the runtime answers no " + m.Method + " from a bundle"}})
}
return
}
if len(m.ID) == 0 {
return
}
if err != nil {
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID,
"error": map[string]any{"code": -32000, "message": err.Error()}})
return
}
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID, "result": result})
}(m)
continue
}
var id int64
if json.Unmarshal(m.ID, &id) != nil {
continue
}
c.mu.Lock()
ch := c.pending[id]
delete(c.pending, id)
c.mu.Unlock()
if ch != nil {
ch <- m
}
}
<-stderrDone
err := cmd.Wait()
code := "0"
if exit := (*exec.ExitError)(nil); errors.As(err, &exit) {
if status, ok := exit.Sys().(syscall.WaitStatus); ok && status.Signaled() {
code = unix.SignalName(status.Signal())
} else {
code = fmt.Sprint(exit.ExitCode())
}
}
saidMu.Lock()
why := fmt.Sprintf("%s's bundle exited (%s)", module, code)
if lastSaid != "" {
why += ": " + lastSaid
}
saidMu.Unlock()
c.mu.Lock()
c.why = errors.New(why)
c.mu.Unlock()
close(c.dead)
mu.Lock()
// Only a child that was serving is brought back; one that died in its own handshake was
// never accepted, and whoever started it was told why.
wasServing := current == c
if current == c {
current = nil
}
if subscriber == c {
subscriber = nil
}
wasStopped := stopped
// Started again at once if it ran a while; after a growing pause while it keeps exiting.
wait := RestartFirst
if time.Since(startedAt) < RestartSettle {
wait = pause
if pause *= 2; pause > RestartMost {
pause = RestartMost
}
} else {
pause = RestartFirst
}
mu.Unlock()
if !wasStopped && wasServing {
// Every bundle stays up (ADR 0198): code that runs long — a handler, a provisioner —
// is not waiting for a call to bring it back, and a tool bundle back early costs nothing.
logf("[mesh-tools] %s; started again in %s", why, wait.Round(100*time.Millisecond))
restart(wait)
}
}()
if _, err := c.ask(module, "initialize", map[string]any{"protocolVersion": Protocol, "capabilities": map[string]any{},
"clientInfo": map[string]any{"name": "node-tools", "version": "1"}}, HandshakeTimeout); err != nil {
_ = cmd.Process.Kill()
return nil, err
}
_ = c.write(map[string]any{"jsonrpc": "2.0", "method": "notifications/initialized"})
return c, nil
}
asking := func(method string, params any, timeout time.Duration) (json.RawMessage, error) {
mu.Lock()
c := current
mu.Unlock()
if c == nil {
fresh, err := start()
if err != nil {
return nil, err
}
mu.Lock()
if current == nil {
current = fresh
startedAt = time.Now()
} else {
_ = fresh.cmd.Process.Signal(syscall.SIGTERM)
fresh = current
}
c = fresh
mu.Unlock()
}
return c.ask(module, method, params, timeout)
}
restart = func(after time.Duration) {
go func() {
time.Sleep(after)
mu.Lock()
if stopped || current != nil {
mu.Unlock()
return
}
mu.Unlock()
fresh, err := start()
mu.Lock()
defer mu.Unlock()
if err != nil {
if !stopped {
wait := pause
if pause *= 2; pause > RestartMost {
pause = RestartMost
}
logf("[mesh-tools] %s's bundle did not start again: %v; trying in %s", module, err, wait)
go restart(wait)
}
return
}
if stopped {
_ = fresh.cmd.Process.Signal(syscall.SIGTERM)
return
}
if current == nil {
current = fresh
startedAt = time.Now()
} else {
_ = fresh.cmd.Process.Signal(syscall.SIGTERM)
}
}()
}
first, err := start()
if err != nil {
return nil, err
}
mu.Lock()
current = first
startedAt = time.Now()
mu.Unlock()
raw, err := asking("tools/list", map[string]any{}, HandshakeTimeout)
if err != nil {
return nil, err
}
var listed struct {
Tools []struct {
Name string `json:"name"`
Description string `json:"description"`
InputSchema json.RawMessage `json:"inputSchema"`
} `json:"tools"`
}
if err := json.Unmarshal(raw, &listed); err != nil {
return nil, fmt.Errorf("%s's bundle listed its tools in a shape that is not MCP's: %w", module, err)
}
order := []string{}
groups := map[string][]Tool{}
for _, t := range listed.Tools {
under, name := module, t.Name
if dot := strings.Index(t.Name, "."); dot >= 0 {
under, name = t.Name[:dot], t.Name[dot+1:]
}
full := t.Name
input := t.InputSchema
if len(input) == 0 || string(input) == "null" {
input = json.RawMessage("{}")
}
tool := Tool{Name: name, Description: t.Description, Input: input,
Run: func(args json.RawMessage) (json.RawMessage, error) {
if len(args) == 0 || string(args) == "null" {
args = json.RawMessage("{}")
}
res, err := asking("tools/call", map[string]any{"name": full, "arguments": args}, CallTimeout)
if err != nil {
return nil, err
}
var called struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}
_ = json.Unmarshal(res, &called)
text := ""
for _, c := range called.Content {
if c.Type == "text" {
text = c.Text
break
}
}
if called.IsError {
if text == "" {
text = module + "." + full + " failed"
}
return nil, errors.New(text)
}
// The bundle's answer is JSON as text; handed back as the value it encodes.
if json.Valid([]byte(text)) && text != "" {
return json.RawMessage(text), nil
}
b, _ := json.Marshal(text)
return b, nil
}}
if _, seen := groups[under]; !seen {
order = append(order, under)
}
groups[under] = append(groups[under], tool)
}
out := &Launched{Stop: func() {
mu.Lock()
stopped = true
c := current
current = nil
mu.Unlock()
if c != nil && c.cmd.Process != nil {
_ = c.cmd.Process.Signal(syscall.SIGTERM)
}
}}
for _, under := range order {
out.Registrations = append(out.Registrations, Registration{Module: under, Tools: groups[under]})
}
return out, nil
}
func (c *child) ask(module, method string, params any, timeout time.Duration) (json.RawMessage, error) {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
return nil, c.why
}
id := c.next
c.next++
ch := make(chan message, 1)
c.pending[id] = ch
c.mu.Unlock()
if err := c.write(map[string]any{"jsonrpc": "2.0", "id": id, "method": method, "params": params}); err != nil {
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
select {
case <-c.dead:
return nil, c.deathReason()
case <-time.After(100 * time.Millisecond):
return nil, err
}
}
timer := time.NewTimer(timeout)
defer timer.Stop()
select {
case m := <-ch:
if m.Error != nil {
msg := m.Error.Message
if msg == "" {
msg = "the bundle refused the request"
}
return nil, errors.New(msg)
}
return m.Result, nil
case <-c.dead:
return nil, c.deathReason()
case <-timer.C:
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
return nil, fmt.Errorf("%s's bundle did not answer %s in %ds", module, method, int(timeout/time.Second))
}
}
func (c *child) deathReason() error {
c.mu.Lock()
defer c.mu.Unlock()
if c.why != nil {
return c.why
}
return errors.New("the bundle exited")
}
+184
View File
@@ -0,0 +1,184 @@
// Package meshtest raises what the controller would, for tests against a real bus: the ASSIGNMENTS
// and EVENTS streams, memberships issued by hand, and a fixture's path.
//
// **The packages share one bus, so run them one at a time: `go test -p 1 ./...`.** Each test raises
// the streams afresh, and the console discovers every runtime that announces itself on the bus
// (novox/hq ADR 0197) — a runtime from another package's test is, correctly, found.
package meshtest
import (
"encoding/json"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// URL is the test bus, or the test is skipped.
func URL(t *testing.T) string {
t.Helper()
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
return url
}
// Mesh is the controller's job, done by hand.
type Mesh struct {
nc *nats.Conn
js nats.JetStreamContext
}
// New raises the streams afresh.
func New(t *testing.T) *Mesh {
t.Helper()
nc, err := nats.Connect(URL(t))
if err != nil {
t.Fatal(err)
}
js, _ := nc.JetStream()
for _, s := range []string{"ASSIGNMENTS", "EVENTS"} {
_ = js.DeleteStream(s)
}
if _, err := js.AddStream(&nats.StreamConfig{Name: "ASSIGNMENTS", Subjects: []string{"mesh.assignment.>"},
MaxMsgsPerSubject: 1, AllowDirect: true}); err != nil {
t.Fatal(err)
}
if _, err := js.AddStream(&nats.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
return &Mesh{nc: nc, js: js}
}
// Issue publishes a membership.
func (m *Mesh) Issue(t *testing.T, mem bus.Membership) {
t.Helper()
body, _ := json.Marshal(mem)
if _, err := m.js.Publish(bus.MembershipSubject(mem.Node, mem.Module), body); err != nil {
t.Fatal(err)
}
}
// NextEvent is the subject the next event under a pattern lands on.
func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
t.Helper()
ch := make(chan string, 1)
sub, err := m.nc.Subscribe(pattern, func(msg *nats.Msg) {
select {
case ch <- msg.Subject:
default:
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
_ = m.nc.Flush()
return ch
}
// MembershipOf is a membership as the controller issues one on a machine.
func MembershipOf(module, node string, plain bool, seats map[string][]string) bus.Membership {
own := "mesh.mod." + module
serves := []bus.Served{{Subject: own + ".tool.{tool}." + node}}
if plain {
serves = append(serves, bus.Served{Subject: own + ".tool.{tool}", Queue: "serve." + module})
}
var verbs []bus.SeatVerb
for seat, vs := range seats {
for _, v := range vs {
verbs = append(verbs, bus.SeatVerb{Seat: seat, Verb: v, Subject: "mesh.seat." + seat + ".tool." + v + "." + node})
}
}
return bus.Membership{Node: node, Module: module, Serves: serves, Seats: verbs,
Emits: own + ".event.{event}", Tools: own + ".tool.tools"}
}
// Fixture is a path in node-tools/test/fixtures, which these tests share with the TypeScript ones.
func Fixture(name string) string {
_, here, _, _ := runtime.Caller(0)
return filepath.Join(filepath.Dir(here), "..", "..", "test", "fixtures", name)
}
// Until retries while the bus answers "no responders" — something not yet served.
func Until(t *testing.T, try func() error) {
t.Helper()
var err error
for i := 0; i < 50; i++ {
if err = try(); err == nil {
return
}
time.Sleep(100 * time.Millisecond)
}
t.Fatal(err)
}
// Logs collects what the runtime says.
type Logs struct{ lines []string }
// Logf is a logger that keeps the lines.
func (l *Logs) Logf(format string, args ...any) {
l.lines = append(l.lines, sprintf(format, args...))
}
// Has says whether a line contains every fragment.
func (l *Logs) Has(fragments ...string) bool {
for _, line := range l.lines {
all := true
for _, f := range fragments {
all = all && strings.Contains(line, f)
}
if all {
return true
}
}
return false
}
// All is every line.
func (l *Logs) All() string { return strings.Join(l.lines, "\n") }
// Consumer makes a module's durable consumer on a machine as the controller does: pull, on the
// EVENTS stream, filtered to what the module consumes, with a short ack wait so a test sees a
// redelivery in seconds rather than the mesh's minutes.
func (m *Mesh) Consumer(t *testing.T, node, module string, filters []string, ackWait time.Duration) {
t.Helper()
cfg := &nats.ConsumerConfig{Durable: node + "_" + module, AckPolicy: nats.AckExplicitPolicy,
AckWait: ackWait, MaxDeliver: 10, DeliverPolicy: nats.DeliverNewPolicy}
if len(filters) == 1 {
cfg.FilterSubject = filters[0]
} else {
cfg.FilterSubjects = filters
}
if _, err := m.js.AddConsumer("EVENTS", cfg); err != nil {
t.Fatal(err)
}
}
// Emit publishes an event as a module would, into the EVENTS stream.
func (m *Mesh) Emit(t *testing.T, subject string, body any) {
t.Helper()
data, _ := json.Marshal(body)
if _, err := m.js.Publish(subject, data); err != nil {
t.Fatal(err)
}
}
// Pending is what a module's consumer still holds: delivered and not acknowledged, and not yet
// delivered.
func (m *Mesh) Pending(t *testing.T, node, module string) (ackPending, notDelivered uint64) {
t.Helper()
info, err := m.js.ConsumerInfo("EVENTS", node+"_"+module)
if err != nil {
t.Fatal(err)
}
return uint64(info.NumAckPending), info.NumPending
}
+5
View File
@@ -0,0 +1,5 @@
package meshtest
import "fmt"
func sprintf(format string, args ...any) string { return fmt.Sprintf(format, args...) }
@@ -0,0 +1,113 @@
package runtime
import (
"encoding/json"
"sort"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq ADR 0197: what a runtime serves, it announces — the NATS services protocol's discovery,
// read here with NATS's own types, each endpoint a subject actually served — and the announcement
// follows a re-issued membership.
func TestTheRuntimeAnnouncesWhatItServesInTheServicesProtocol(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("beta", "anchor", false, map[string][]string{"node-shelf": {"list", "clear"}}))
nodeTools := connect(t, "node-tools", "anchor")
stop, err := Run(nodeTools, []Served{
{"alpha", []string{mt.Fixture("many-alpha.serve.mjs")}},
{"beta", []string{mt.Fixture("many-beta.serve.mjs")}},
}, nil, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
info := func() micro.Info {
t.Helper()
msg, err := nc.Request("$SRV.INFO.node-tools.anchor", nil, 2*time.Second)
if err != nil {
t.Fatal(err)
}
var i micro.Info
if err := json.Unmarshal(msg.Data, &i); err != nil {
t.Fatal(err)
}
return i
}
subjects := func(i micro.Info) string {
var out []string
for _, e := range i.Endpoints {
out = append(out, e.Subject+"|"+e.QueueGroup+"|"+e.Metadata["kind"]+"|"+e.Metadata["module"]+"|"+e.Metadata["seat"]+"|"+e.Metadata["scope"]+"|"+e.Metadata["interchangeable"])
}
sort.Strings(out)
return strings.Join(out, "\n")
}
got := info()
if got.Type != micro.InfoResponseType || got.Name != "node-tools" || got.ID != "anchor" || got.Version == "" {
t.Errorf("identity: %+v", got.ServiceIdentity)
}
want := strings.Join([]string{
"mesh.mod.alpha.tool.one.anchor||tool|alpha|||false",
"mesh.mod.alpha.tool.two.anchor||tool|alpha|||false",
"mesh.mod.beta.tool.five.anchor||tool|beta|||false",
"mesh.mod.beta.tool.four.anchor||tool|beta|||false",
"mesh.mod.beta.tool.three.anchor||tool|beta|||false",
"mesh.seat.node-shelf.tool.clear.anchor||seat|beta|node-shelf|node|false",
"mesh.seat.node-shelf.tool.list.anchor||seat|beta|node-shelf|node|false",
}, "\n")
if s := subjects(got); s != want {
t.Errorf("announced:\n%s\nwant:\n%s", s, want)
}
for _, e := range got.Endpoints {
if e.Metadata["node"] != "anchor" || e.Metadata["description"] == "" || !json.Valid([]byte(e.Metadata["schema"])) {
t.Errorf("endpoint metadata: %+v", e)
}
}
// Every subject announced is answered.
asker := connect(t, "console", "workstation")
for _, e := range got.Endpoints {
if _, err := asker.Ask("", map[string]any{}, e.Subject); err != nil {
t.Errorf("announced %s and does not answer it: %v", e.Subject, err)
}
}
// PING answers with the same identity, and a request for another service is not answered.
if msg, err := nc.Request("$SRV.PING", nil, 2*time.Second); err != nil || !strings.Contains(string(msg.Data), micro.PingResponseType) {
t.Errorf("ping: %v %v", msg, err)
}
if _, err := nc.Request("$SRV.INFO.somebody-else", nil, 300*time.Millisecond); err == nil {
t.Error("answered a request for another service")
}
// alpha is re-issued a plain subject: the announcement says so, and says it is interchangeable.
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", true, nil))
var after string
for i := 0; i < 40; i++ {
after = subjects(info())
if strings.Contains(after, "mesh.mod.alpha.tool.one|serve.alpha|tool|alpha|||true") {
break
}
time.Sleep(100 * time.Millisecond)
}
if !strings.Contains(after, "mesh.mod.alpha.tool.one|serve.alpha|tool|alpha|||true") ||
!strings.Contains(after, "mesh.mod.alpha.tool.one.anchor||tool|alpha|||true") {
t.Errorf("after a re-issued membership:\n%s", after)
}
_ = bus.Served{}
}
+157
View File
@@ -0,0 +1,157 @@
package runtime
import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// A logger safe to write from the runtime's goroutines.
type lines struct {
mu sync.Mutex
l []string
}
func (s *lines) logf(format string, args ...any) {
s.mu.Lock()
defer s.mu.Unlock()
s.l = append(s.l, strings.TrimSpace(sprintf(format, args...)))
}
func (s *lines) all() string {
s.mu.Lock()
defer s.mu.Unlock()
return strings.Join(s.l, "\n")
}
func sprintf(format string, args ...any) string { return fmtSprintf(format, args...) }
func countLines(t *testing.T, path, want string) int {
t.Helper()
b, _ := os.ReadFile(path)
n := 0
for _, l := range strings.Split(string(b), "\n") {
if l == want {
n++
}
}
return n
}
// novox/hq ADR 0198: a module's long-running code is launched by the node's runtime, and the runtime
// is its bus — it binds the module's own consumer, hands each event to the bundle, and acknowledges it
// only when the bundle has taken it; one the bundle failed or died on is offered again.
func TestTheRuntimeHandsAModulesEventsToItsBundleAndAcknowledgesThemOnlyWhenTaken(t *testing.T) {
mesh := mt.New(t)
restore := bus.NakDelay
bus.NakDelay = 300 * time.Millisecond
t.Cleanup(func() { bus.NakDelay = restore })
firstPause := launch.RestartFirst
launch.RestartFirst = 100 * time.Millisecond
t.Cleanup(func() { launch.RestartFirst = firstPause })
mesh.Issue(t, mt.MembershipOf("watcher", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("beta", "anchor", true, nil))
// The controller's consumer for the module, as its own runtime bound it: anchor_watcher on EVENTS.
mesh.Consumer(t, "anchor", "watcher", []string{"mesh.mod.alpha.event.>"}, 2*time.Second)
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
dir := t.TempDir()
log := filepath.Join(dir, "watch.log")
said := &lines{}
stop, err := Run(nodeTools, []Served{
{Module: "watcher", Entrypoints: []string{mt.Fixture("watcher.serve.mjs")}},
{Module: "beta", Entrypoints: []string{mt.Fixture("many-beta.serve.mjs")}},
}, map[string]map[string]string{"watcher": {"WATCH_LOG": log}}, said.logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
if !strings.Contains(said.all(), "watcher's events arrive on its consumer anchor_watcher") {
t.Fatalf("the module's consumer was not bound:\n%s", said.all())
}
// Handled once, acknowledged once.
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 1})
mt.Until(t, func() error {
if countLines(t, log, "handled 1") != 1 {
return errorf("event 1 not handled yet")
}
return nil
})
// The handler fails the first time: not acknowledged, offered again, then handled.
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 2, "fail": true})
// The bundle dies on the first offer: not acknowledged, the bundle is started again and the
// event is offered again once its ack wait has passed.
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 3, "die": true})
deadline := time.Now().Add(20 * time.Second)
for time.Now().Before(deadline) {
if countLines(t, log, "handled 2") == 1 && countLines(t, log, "handled 3") == 1 {
break
}
time.Sleep(200 * time.Millisecond)
}
if countLines(t, log, "handled 2") != 1 || countLines(t, log, "handled 3") != 1 {
b, _ := os.ReadFile(log)
t.Fatalf("a failed or interrupted event was not offered again and handled once:\n%s\nruntime:\n%s", b, said.all())
}
if countLines(t, log, "started") < 2 {
t.Errorf("the bundle that died was not started again: %s", said.all())
}
mt.Until(t, func() error {
ack, notYet := mesh.Pending(t, "anchor", "watcher")
if ack != 0 || notYet != 0 {
return errorf("still pending: %d unacknowledged, %d undelivered", ack, notYet)
}
return nil
})
if countLines(t, log, "handled 1") != 1 {
t.Errorf("event 1 was handled more than once")
}
// A tool asks another module's tool through the runtime, as the module.
got, err := call(t, asker, "watcher.relay@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"beta":3}`)
}
// A long-running bundle that exits is started again, without waiting for a call (ADR 0198).
func TestALongRunningBundleThatExitsIsStartedAgain(t *testing.T) {
mesh := mt.New(t)
firstPause := launch.RestartFirst
launch.RestartFirst = 100 * time.Millisecond
t.Cleanup(func() { launch.RestartFirst = firstPause })
mesh.Issue(t, mt.MembershipOf("flaky", "anchor", false, nil))
nodeTools := connect(t, "node-tools", "anchor")
log := filepath.Join(t.TempDir(), "flaky.log")
said := &lines{}
stop, err := Run(nodeTools, []Served{{Module: "flaky", Entrypoints: []string{mt.Fixture("flaky.serve.mjs")}}},
map[string]map[string]string{"flaky": {"FLAKY_LOG": log}}, said.logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
mt.Until(t, func() error {
if countLines(t, log, "started") < 3 {
return errorf("started %d time(s)", countLines(t, log, "started"))
}
return nil
})
if !strings.Contains(said.all(), "flaky's bundle exited (3)") || !strings.Contains(said.all(), "started again in") {
t.Errorf("the restart was not said:\n%s", said.all())
}
}
func fmtSprintf(format string, args ...any) string { return fmt.Sprintf(format, args...) }
func errorf(format string, args ...any) error { return fmt.Errorf(format, args...) }
+580
View File
@@ -0,0 +1,580 @@
// Package runtime is the node's tool runtime (novox/hq ADR 0175, ADR 0193), the Go port of
// node-tools' runtime.ts in its launch-only form: it launches every assigned module's bundle,
// serves each module's tools on that module's subjects and each held seat's verbs on the seat's,
// and answers for every module the verb that says what it serves. It imports nothing and knows no
// language.
package runtime
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"github.com/novox/mesh-tools/node-tools/internal/announce"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
)
// ToolsVerb is the verb every module's runtime answers for it (ADR 0152): its tools, from the code
// that answers them.
const ToolsVerb = "tools"
// Words the mesh sets for the runtime (ADR 0175, ADR 0192).
const (
ToolModules = "MESH_TOOL_MODULES"
ToolEnv = "MESH_TOOL_ENV"
OperatorAccount = "MESH_OPERATOR_ACCOUNT"
OperatorHome = "MESH_OPERATOR_HOME"
)
// Served is one module this runtime serves and its entrypoints.
type Served struct {
Module string
Entrypoints []string
}
// ToolsAnswer is what `tools` answers for one module.
type ToolsAnswer struct {
Module string `json:"module"`
Tools []ListedTool `json:"tools"`
Failed string `json:"failed,omitempty"`
}
// ListedTool is one tool as a module's `tools` answer lists it.
type ListedTool struct {
Name string `json:"name"`
Description string `json:"description"`
Input json.RawMessage `json:"input"`
Subjects []string `json:"subjects,omitempty"`
}
// ServedModulesFrom reads MESH_TOOL_MODULES: `<module>=<entrypoint>` entries, comma-separated, several
// per module. The one-module form — a bare path, or the runtime's own module — is the per-module
// containers' (to-be 38 WP4c) and refused here: the node's runtime imports nothing.
func ServedModulesFrom(spec, own string) ([]Served, error) {
order := []string{}
by := map[string][]string{}
for _, raw := range strings.Split(spec, ",") {
entry := strings.TrimSpace(raw)
if entry == "" {
continue
}
module, path, ok := strings.Cut(entry, "=")
module, path = strings.TrimSpace(module), strings.TrimSpace(path)
if !ok || module == "" || path == "" || module == own {
return nil, fmt.Errorf("%s: %q is not <module>=<entrypoint> of another module; the node's "+
"runtime launches the bundles it is given and imports nothing (novox/hq ADR 0193)", ToolModules, entry)
}
if _, seen := by[module]; !seen {
order = append(order, module)
}
by[module] = append(by[module], path)
}
out := make([]Served, 0, len(order))
for _, m := range order {
out = append(out, Served{Module: m, Entrypoints: by[m]})
}
return out, nil
}
// TakeToolEnvs reads the composed environments (ADR 0192) and removes them from the process's, so no
// bundle finds another's there.
func TakeToolEnvs() (map[string]map[string]string, error) {
raw := os.Getenv(ToolEnv)
os.Unsetenv(ToolEnv)
out := map[string]map[string]string{}
if raw == "" {
return out, nil
}
var parsed map[string]map[string]any
if err := json.Unmarshal([]byte(raw), &parsed); err != nil {
return nil, fmt.Errorf(`%s is not JSON of the shape {"<module>": {"<word>": "<value>"}}: %w`, ToolEnv, err)
}
for module, words := range parsed {
own := map[string]string{}
for k, v := range words {
if s, ok := v.(string); ok {
own[k] = s
} else {
b, _ := json.Marshal(v)
own[k] = string(b)
}
}
out[module] = own
}
return out, nil
}
type registration struct {
module string // the module's own name, or a seat's
owner string // the module whose bundle made it
tools []launch.Tool
}
// Run launches, binds and serves. It answers a stop function.
func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, logf func(string, ...any)) (func(), error) {
if account := os.Getenv(OperatorAccount); account != "" {
home := ""
if h := os.Getenv(OperatorHome); h != "" {
home = " (home " + h + ")"
}
logf("[mesh-tools] the operator's account here is %s%s", account, home)
}
modules := make([]string, 0, len(served))
isServed := map[string]bool{}
for _, s := range served {
modules = append(modules, s.Module)
isServed[s.Module] = true
conn.Follow(s.Module)
}
node := conn.Node()
base := os.Environ()
envFor := func(module string) []string {
words := map[string]string{}
for _, kv := range base {
if k, v, ok := strings.Cut(kv, "="); ok && k != ToolEnv {
words[k] = v
}
}
for k, v := range envs[module] {
words[k] = v
}
words["MESH_SERVED_MODULE"] = module
words["MESH_MODULE"] = module
if node != "" {
words["MESH_NODE"] = node
}
out := make([]string, 0, len(words))
for k, v := range words {
out = append(out, k+"="+v)
}
sort.Strings(out)
return out
}
// The module's events, for every child of it that subscribes (ADR 0198): one consumer per module,
// bound the first time any of its children subscribes, each event handed to every child that did.
events := &consumers{conn: conn, logf: logf, of: map[string]*moduleEvents{}}
failed := map[string]string{}
var registrations []registration
var stops []func()
for _, s := range served {
for _, entry := range s.Entrypoints {
path, _ := filepath.Abs(entry)
module := s.Module
fail := func(why string) {
failed[module] = why
logf("[mesh-tools] %s's bundle %s failed to load: %s; its tools are not served here", module, entry, why)
}
if !launch.Executable(path) {
fail(path + " is not executable; a bundle the runtime serves is started, never imported, and its build makes it executable (novox/hq ADR 0193)")
continue
}
child, err := launch.Start(module, path, envFor(module), events.forModule(module), logf)
if err != nil {
fail(err.Error())
continue
}
stops = append(stops, child.Stop)
for _, r := range child.Registrations {
registrations = append(registrations, registration{module: r.Module, owner: module, tools: r.Tools})
}
}
}
claimed := map[string]bool{}
for _, m := range modules {
if mem := conn.Membership(m); mem != nil {
for _, s := range mem.Seats {
claimed[s.Seat] = true
}
}
}
var own []registration
for _, r := range registrations {
switch {
case isServed[r.module]:
own = append(own, r)
case claimed[r.module]:
default:
logf(`[mesh-tools] %s registers tools under "%s", which is neither a module served here nor a seat one of them claims; not served until the mesh issues the claim`, r.owner, r.module)
}
}
stopAll := func() {
events.stopAll()
for i := len(stops) - 1; i >= 0; i-- {
stops[i]()
}
}
for _, r := range own {
seen := map[string]bool{}
for _, t := range r.tools {
if t.Name == ToolsVerb {
stopAll()
return nil, fmt.Errorf(`%s names a tool "%s", which is the verb the runtime answers for every module with what it serves (novox/hq ADR 0152) — refused, rename it`, r.module, ToolsVerb)
}
if seen[t.Name] {
stopAll()
return nil, fmt.Errorf("%s exposes two tools named %s — refused", r.module, t.Name)
}
seen[t.Name] = true
}
}
var names []string
byModule := map[string][]launch.Tool{}
for _, r := range own {
for _, t := range r.tools {
t := t
stop, err := conn.Handle(r.module+"."+t.Name, func(body json.RawMessage) (any, error) {
return t.Run(argsOf(body))
})
if err != nil {
logf("[mesh-tools] cannot serve %s.%s: %v", r.module, t.Name, err)
continue
}
names = append(names, r.module+"."+t.Name)
stops = append(stops, stop)
}
byModule[r.module] = append(byModule[r.module], r.tools...)
}
for _, module := range modules {
module := module
tools := byModule[module]
why := failed[module]
if len(tools) == 0 && why == "" {
continue // a pure-events module: silent, as it always was
}
stop, err := conn.Handle(module+"."+ToolsVerb, func(json.RawMessage) (any, error) {
answer := ToolsAnswer{Module: module, Tools: []ListedTool{}, Failed: why}
for _, t := range tools {
answer.Tools = append(answer.Tools, ListedTool{Name: t.Name, Description: t.Description,
Input: t.Input, Subjects: subjectsOf(conn, module, t.Name)})
}
return answer, nil
})
if err == nil {
stops = append(stops, stop)
}
}
failedNames := make([]string, 0, len(failed))
for _, m := range modules {
if _, f := failed[m]; f {
failedNames = append(failedNames, m)
}
}
line := fmt.Sprintf("[mesh-tools] serving %d tool(s) for %d module(s): %s", len(names), len(modules), orNone(names))
if len(failedNames) > 0 {
line += fmt.Sprintf("; not serving %s, whose bundle(s) failed to load", strings.Join(failedNames, ", "))
}
logf("%s", line)
stops = append(stops, serveSeats(conn, modules, registrations, logf))
// **What it serves, it announces** (novox/hq ADR 0197): the NATS services protocol's discovery,
// answered with what is served at the moment it is asked — re-served memberships included.
announced, err := announce.Serve(conn, announce.Service{
Name: conn.Module(), ID: instanceOf(conn),
Description: "the mesh's tool runtime on " + node + ": every assigned module's tools and the seats they hold",
Metadata: map[string]string{"node": node},
}, func() []announce.Endpoint { return endpointsOf(conn, own, registrations, modules) })
if err != nil {
logf("[mesh-tools] cannot announce what it serves: %v", err)
} else {
stops = append(stops, announced)
}
conn.Flush()
return stopAll, nil
}
// instanceOf is this runtime's instance on the bus: its machine, which is what tells two instances of
// one service apart; the connection's module where it has no machine.
func instanceOf(conn *bus.Conn) string {
if n := conn.Node(); n != "" {
return n
}
return conn.Module()
}
// endpointsOf is everything this runtime serves now: each served module's tools on every subject the
// mesh issued for them, and each held seat's verbs on the seat's subject (ADR 0197).
func endpointsOf(conn *bus.Conn, own []registration, registrations []registration, modules []string) []announce.Endpoint {
node := conn.Node()
var out []announce.Endpoint
for _, r := range own {
for _, t := range r.tools {
served := conn.ServedOn(r.module, t.Name)
interchangeable := false
for _, s := range served {
interchangeable = interchangeable || s.Subject == "mesh.mod."+r.module+".tool."+t.Name
}
for _, s := range served {
out = append(out, announce.Endpoint{Kind: announce.KindTool, Module: r.module, Tool: t.Name,
Node: node, Description: t.Description, Schema: t.Input, Interchangeable: interchangeable,
Subject: s.Subject, Queue: s.Queue})
}
}
}
impl := map[string]map[string]launch.Tool{}
for _, r := range registrations {
if impl[r.module] == nil {
impl[r.module] = map[string]launch.Tool{}
}
for _, t := range r.tools {
impl[r.module][t.Name] = t
}
}
have := map[string]bool{}
for _, module := range modules {
m := conn.Membership(module)
if m == nil {
continue
}
for _, v := range m.Seats {
t, ok := impl[v.Seat][v.Verb]
if !ok || have[v.Subject] {
continue
}
have[v.Subject] = true
scope := "mesh"
if node != "" && strings.HasSuffix(v.Subject, "."+node) {
scope = "node"
}
out = append(out, announce.Endpoint{Kind: announce.KindSeat, Module: module, Tool: v.Verb, Seat: v.Seat,
Scope: scope, Node: node, Description: t.Description, Schema: t.Input, Subject: v.Subject})
}
}
return out
}
func orNone(names []string) string {
if len(names) == 0 {
return "(none)"
}
return strings.Join(names, ", ")
}
// argsOf is a call's arguments as the tool receives them: an object, `{}` for none.
func argsOf(body json.RawMessage) json.RawMessage {
trimmed := strings.TrimSpace(string(body))
if trimmed == "" || trimmed == "null" {
return json.RawMessage("{}")
}
return body
}
// subjectsOf is where a tool is answered as the mesh issued it: the plain subject first, then this
// machine's; nothing before a membership is issued.
func subjectsOf(conn *bus.Conn, module, tool string) []string {
m := conn.Membership(module)
if m == nil {
return nil
}
var plain, mine []string
for _, s := range m.Serves {
subject := strings.ReplaceAll(s.Subject, "{tool}", tool)
if s.Queue != "" {
plain = append(plain, subject)
} else {
mine = append(mine, subject)
}
}
return append(plain, mine...)
}
// serveSeats serves every verb of every seat a served module holds, where the mesh issued it, by
// the tool of the same name registered under the seat's name (ADR 0159, 0160) — and serves again
// whenever a membership changes. Whether this machine holds the seat is the bus's to decide.
func serveSeats(conn *bus.Conn, modules []string, registrations []registration, logf func(string, ...any)) func() {
impl := map[string]map[string]launch.Tool{}
for _, r := range registrations {
if impl[r.module] == nil {
impl[r.module] = map[string]launch.Tool{}
}
for _, t := range r.tools {
impl[r.module][t.Name] = t
}
}
var mu sync.Mutex
var stops []func()
serve := func() {
mu.Lock()
defer mu.Unlock()
for _, s := range stops {
s()
}
stops = nil
have := map[string]bool{}
for _, module := range modules {
m := conn.Membership(module)
if m == nil {
continue
}
for _, v := range m.Seats {
if have[v.Subject] {
continue
}
have[v.Subject] = true
t, ok := impl[v.Seat][v.Verb]
if !ok {
logf("[mesh-tools] %s claims %s and implements no %s, which that seat promises; not served", module, v.Seat, v.Verb)
continue
}
stop, err := conn.HandleSubject(v.Subject, func(body json.RawMessage) (any, error) {
return t.Run(argsOf(body))
})
if err != nil {
logf("[mesh-tools] cannot serve %s's %s on %s: %v", v.Seat, v.Verb, v.Subject, err)
continue
}
stops = append(stops, stop)
logf("[mesh-tools] serving %s's %s on %s, admitted where %s holds the seat", v.Seat, v.Verb, v.Subject, module)
}
}
}
serve()
conn.OnMembership(func(bus.Membership) { go func() { serve(); conn.Flush() }() })
return func() {
mu.Lock()
defer mu.Unlock()
for _, s := range stops {
s()
}
stops = nil
}
}
// consumers holds, per module the runtime serves, the one durable consumer its events arrive on and
// the children its events are handed to (novox/hq ADR 0198).
type consumers struct {
conn *bus.Conn
logf func(string, ...any)
mu sync.Mutex
of map[string]*moduleEvents
}
type moduleEvents struct {
stop func()
delivers []*func(json.RawMessage) error
}
// stopAll unbinds every module's consumer.
func (c *consumers) stopAll() {
c.mu.Lock()
defer c.mu.Unlock()
for _, m := range c.of {
if m.stop != nil {
m.stop()
}
}
c.of = map[string]*moduleEvents{}
}
// forModule is the bus one launched bundle of a module reaches the mesh through.
func (c *consumers) forModule(module string) launch.Bus {
return &moduleBus{all: c, module: module}
}
type moduleBus struct {
all *consumers
module string
mu sync.Mutex
deliver *func(json.RawMessage) error
}
// Publish emits an event as the module (ADR 0193).
func (b *moduleBus) Publish(params json.RawMessage) error {
var env bus.Envelope
if err := json.Unmarshal(params, &env); err != nil {
return fmt.Errorf("not an event envelope: %w", err)
}
return b.all.conn.PublishAs(b.module, env)
}
// Ask calls a tool as the module: `{key, body}`, answered with the tool's result (ADR 0198).
func (b *moduleBus) Ask(params json.RawMessage) (json.RawMessage, error) {
var asked struct {
Key string `json:"key"`
Body json.RawMessage `json:"body"`
}
if err := json.Unmarshal(params, &asked); err != nil || asked.Key == "" {
return nil, fmt.Errorf("mesh/ask names no tool: {key, body}")
}
body := any(asked.Body)
if len(asked.Body) == 0 {
body = map[string]any{}
}
answered, err := b.all.conn.AskAs(b.module, asked.Key, body)
if err != nil {
return nil, err
}
if len(answered.Result) == 0 {
return json.RawMessage("null"), nil
}
return answered.Result, nil
}
// Subscribe hands this bundle the module's events. The consumer is bound once per module; each of
// the module's children that subscribed is handed every event, and the event is acknowledged only
// when all of them took it — one consumer split between two readers would give each half.
func (b *moduleBus) Subscribe(deliver func(json.RawMessage) error) error {
b.mu.Lock()
if b.deliver == nil {
d := deliver
b.deliver = &d
} else {
*b.deliver = deliver
}
mine := b.deliver
b.mu.Unlock()
c := b.all
c.mu.Lock()
defer c.mu.Unlock()
m := c.of[b.module]
if m == nil {
m = &moduleEvents{}
c.of[b.module] = m
}
listed := false
for _, d := range m.delivers {
if d == mine {
listed = true
}
}
if !listed {
m.delivers = append(m.delivers, mine)
}
if m.stop != nil {
return nil
}
module := b.module
stop, err := c.conn.ConsumeAs(module, func(env bus.Envelope) error {
raw, err := json.Marshal(env)
if err != nil {
return err
}
c.mu.Lock()
targets := append([]*func(json.RawMessage) error(nil), c.of[module].delivers...)
c.mu.Unlock()
for _, d := range targets {
if err := (*d)(raw); err != nil {
c.logf("[mesh-tools] %s did not take %s: %v; offered again", module, env.Key, err)
return err
}
}
return nil
})
if err != nil {
return err
}
m.stop = stop
c.logf("[mesh-tools] %s's events arrive on its consumer %s", module, bus.ConsumerOf(c.conn.Node(), module))
return nil
}
+251
View File
@@ -0,0 +1,251 @@
package runtime
import (
"encoding/json"
"os"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
func connect(t *testing.T, module, node string) *bus.Conn {
t.Helper()
c, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: module, Node: node})
if err != nil {
t.Fatal(err)
}
c.Logf = func(string, ...any) {}
t.Cleanup(c.Close)
return c
}
func call(t *testing.T, asker *bus.Conn, key string, args any) (string, error) {
t.Helper()
got, err := asker.Ask(key, args, "")
return string(got.Result), err
}
func same(t *testing.T, got, want string) {
t.Helper()
var a, b any
if json.Unmarshal([]byte(got), &a) != nil || json.Unmarshal([]byte(want), &b) != nil {
t.Fatalf("not JSON: got %s want %s", got, want)
}
ga, _ := json.Marshal(a)
gb, _ := json.Marshal(b)
if string(ga) != string(gb) {
t.Errorf("got %s, want %s", got, want)
}
}
// The node's runtime serves five modules' bundles on one credential — two TypeScript, one broken,
// one Python, one written against the protocol — and follows a re-issued membership (ADR 0175, 0193).
func TestTheNodesRuntimeServesFiveModulesAndFollowsAReissuedMembership(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", true, nil))
mesh.Issue(t, mt.MembershipOf("beta", "anchor", false, map[string][]string{"node-shelf": {"list", "clear"}}))
mesh.Issue(t, mt.MembershipOf("gamma", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("delta", "anchor", false, map[string][]string{"node-lamp": {"on"}}))
mesh.Issue(t, mt.MembershipOf("epsilon", "anchor", false, nil))
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
logs := &mt.Logs{}
t.Setenv("MESH_OPERATOR_ACCOUNT", "somebody")
t.Setenv("MESH_OPERATOR_HOME", "/home/somebody")
stop, err := Run(nodeTools, []Served{
{"alpha", []string{mt.Fixture("many-alpha.serve.mjs")}},
{"beta", []string{mt.Fixture("many-beta.serve.mjs")}},
{"gamma", []string{mt.Fixture("many-broken.serve.mjs")}},
{"delta", []string{mt.Fixture("many-delta.py")}},
{"epsilon", []string{mt.Fixture("many-epsilon.mjs")}},
}, nil, logs.Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
if !logs.Has("the operator's account here is somebody (home /home/somebody)") {
t.Errorf("the operator was not said:\n%s", logs.All())
}
if !logs.Has("gamma's bundle", "many-broken.serve.mjs failed to load: gamma's bundle exited (1): Error: gamma's bundle cannot find its client; its tools are not served here") {
t.Errorf("the broken bundle was not named with its own words:\n%s", logs.All())
}
if !logs.Has("serving 8 tool(s) for 5 module(s): alpha.one, alpha.two, beta.three, beta.four, beta.five, delta.greet, delta.die, epsilon.seven; not serving gamma") {
t.Errorf("not serving what it should:\n%s", logs.All())
}
for key, want := range map[string]string{
"alpha.one": `{"alpha":1}`, "alpha.one@anchor": `{"alpha":1}`, "beta.three@anchor": `{"beta":3}`,
"beta.four@anchor": `{"beta":4}`, "beta.five@anchor": `{"beta":5}`,
"seat:node-shelf.list@anchor": `{"shelf":["a","b"]}`, "seat:node-shelf.clear@anchor": `{"cleared":true}`,
"seat:node-lamp.on@anchor": `{"on":true,"language":"python"}`, "epsilon.seven@anchor": `{"epsilon":7,"via":"stdio"}`,
} {
got, err := call(t, asker, key, map[string]any{})
if err != nil {
t.Errorf("%s: %v", key, err)
continue
}
same(t, got, want)
}
if _, err := call(t, asker, "beta.three", map[string]any{}); err == nil || !strings.Contains(err.Error(), "no responders") {
t.Errorf("beta was not issued the plain subject, and answered on it: %v", err)
}
got, err := call(t, asker, "delta.greet@anchor", map[string]any{"who": "mesh"})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"greeting":"hello mesh","language":"python"}`)
// A launched tool that emits does so as its module, through the runtime.
landed := mesh.NextEvent(t, "mesh.mod.*.event.>")
got, err = call(t, asker, "alpha.two", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"alpha":2}`)
select {
case subject := <-landed:
if subject != "mesh.mod.alpha.event.happened" {
t.Errorf("the event landed on %s", subject)
}
case <-time.After(5 * time.Second):
t.Error("the event never landed")
}
// A bundle that dies mid-call is said, and started again on its next call.
if _, err := call(t, asker, "delta.die@anchor", map[string]any{}); err == nil {
t.Error("a bundle that died answered")
}
got, err = call(t, asker, "delta.greet@anchor", map[string]any{"who": "again"})
if err != nil {
t.Fatalf("not started again: %v", err)
}
same(t, got, `{"greeting":"hello again","language":"python"}`)
// `tools` answers for each, and why gamma serves nothing.
gamma, err := call(t, asker, "gamma.tools@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, gamma, `{"module":"gamma","tools":[],"failed":"gamma's bundle exited (1): Error: gamma's bundle cannot find its client"}`)
beta, _ := call(t, asker, "beta.tools@anchor", map[string]any{})
var answer ToolsAnswer
_ = json.Unmarshal([]byte(beta), &answer)
if len(answer.Tools) != 3 || answer.Tools[0].Name != "three" || strings.Join(answer.Tools[0].Subjects, ",") != "mesh.mod.beta.tool.three.anchor" {
t.Errorf("beta's tools answer: %s", beta)
}
// Re-issued mid-run, now answering for the module anywhere: served without a restart.
mesh.Issue(t, mt.MembershipOf("beta", "anchor", true, map[string][]string{"node-shelf": {"list", "clear"}}))
mt.Until(t, func() error { _, err := call(t, asker, "beta.three", map[string]any{}); return err })
got, err = call(t, asker, "seat:node-shelf.list@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"shelf":["a","b"]}`)
}
// Each bundle is given its own environment and none of another's (ADR 0192), and the composed
// environments are not left in the runtime's.
func TestEachBundleIsGivenItsOwnEnvironment(t *testing.T) {
mesh := mt.New(t)
for _, m := range []string{"gamma", "delta", "zeta"} {
mesh.Issue(t, mt.MembershipOf(m, "anchor", false, nil))
}
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
t.Setenv("MESH_OPERATOR_ACCOUNT", "somebody")
t.Setenv(ToolEnv, `{"gamma":{"GAMMA_CONFIG_FILE":"/var/lib/mesh/gamma/config.json"},"delta":{"DELTA_TOKEN_FILE":"/var/lib/mesh/delta/token"},"zeta":{"ZETA_URL":"http://127.0.0.1:3000"}}`)
envs, err := TakeToolEnvs()
if err != nil {
t.Fatal(err)
}
if _, left := os.LookupEnv(ToolEnv); left {
t.Error("the composed environments were left in the runtime's")
}
logs := &mt.Logs{}
stop, err := Run(nodeTools, []Served{
{"gamma", []string{mt.Fixture("env-gamma.serve.mjs")}},
{"delta", []string{mt.Fixture("env-delta.serve.mjs")}},
{"zeta", []string{mt.Fixture("env-zeta.mjs")}},
}, envs, logs.Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
for key, want := range map[string]string{
"gamma.given@anchor": `{"mine":"/var/lib/mesh/gamma/config.json","theirs":null,"runtime":"somebody","composed":null}`,
"delta.given@anchor": `{"mine":"/var/lib/mesh/delta/token","theirs":null}`,
"zeta.given@anchor": `{"mine":"http://127.0.0.1:3000","theirs":null,"composed":null}`,
} {
got, err := call(t, asker, key, map[string]any{})
if err != nil {
t.Fatalf("%s: %v\n%s", key, err, logs.All())
}
same(t, got, want)
}
}
// An entrypoint that is not executable is refused by name, and the others serve (ADR 0193).
func TestAnEntrypointThatIsNotExecutableIsRefused(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("plain", "anchor", false, nil))
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
logs := &mt.Logs{}
stop, err := Run(nodeTools, []Served{
{"alpha", []string{mt.Fixture("many-alpha.serve.mjs")}},
{"plain", []string{mt.Fixture("many-alpha.mjs")}},
}, nil, logs.Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
if !logs.Has("plain's bundle", "many-alpha.mjs failed to load:", "is not executable; a bundle the runtime serves is started, never imported") {
t.Errorf("the non-executable entrypoint was not refused by name:\n%s", logs.All())
}
got, err := call(t, asker, "alpha.one@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"alpha":1}`)
}
// A launched bundle is told the module it serves, so its seat's verbs stay the seat's (ADR 0193).
func TestALaunchedBundleRegisteringItsSeatFirstServesTheSeat(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("theta", "anchor", false, map[string][]string{"node-shelf": {"list"}}))
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
stop, err := Run(nodeTools, []Served{{"theta", []string{mt.Fixture("served-seat-first.mjs")}}},
map[string]map[string]string{"theta": {"THETA_WORD": "given"}}, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
got, err := call(t, asker, "theta.own@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"theta":"given"}`)
got, err = call(t, asker, "seat:node-shelf.list@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"shelf":["x"]}`)
}
func TestToolModulesNamesOtherModulesOnly(t *testing.T) {
got, err := ServedModulesFrom(" alpha=/a/tools/index.serve.mjs, beta=/b/one, beta=/b/two ", "node-tools")
if err != nil || len(got) != 2 || got[1].Module != "beta" || len(got[1].Entrypoints) != 2 {
t.Fatalf("%v %v", got, err)
}
for _, bad := range []string{"/mine/index.js", "node-tools=/own.js", "=/x"} {
if _, err := ServedModulesFrom(bad, "node-tools"); err == nil {
t.Errorf("%q was accepted", bad)
}
}
}
+18
View File
@@ -0,0 +1,18 @@
// Package wire is JSON as the TypeScript runtime writes it: no HTML escaping of <, > and &.
package wire
import (
"bytes"
"encoding/json"
)
// Marshal encodes v the way JSON.stringify does, without a trailing newline.
func Marshal(v any) ([]byte, error) {
var b bytes.Buffer
enc := json.NewEncoder(&b)
enc.SetEscapeHTML(false)
if err := enc.Encode(v); err != nil {
return nil, err
}
return bytes.TrimRight(b.Bytes(), "\n"), nil
}
+4 -4
View File
@@ -35,10 +35,10 @@
{ {
"name": "runtime", "name": "runtime",
"kind": "bundle", "kind": "bundle",
"language": "typescript", "language": "go",
"entrypoints": [ "system": "arch",
"src/main.js" "from": "cmd/node-tools",
] "binary": "node-tools"
} }
] ]
} }
+1 -1
View File
@@ -13,7 +13,7 @@
"test": "node --test --test-concurrency=1 --experimental-strip-types 'test/*.test.ts'" "test": "node --test --test-concurrency=1 --experimental-strip-types 'test/*.test.ts'"
}, },
"dependencies": { "dependencies": {
"@novox/mesh-sdk": "^0.1.0", "@novox/mesh-sdk": "^0.1.6",
"nats": "^2.29.0" "nats": "^2.29.0"
}, },
"devDependencies": { "devDependencies": {
+139
View File
@@ -0,0 +1,139 @@
// What a runtime serves, it announces (novox/hq ADR 0197): the NATS services protocol's discovery —
// `$SRV.PING`, `$SRV.INFO`, `$SRV.STATS`, and the same followed by the service's name and its id —
// answered in the io.nats.micro.v1 format with what is served at the moment of the request. Serving is
// unchanged; this only says what is served. One service per runtime process, because the bus admits
// one reply per request from each responder: one endpoint per tool per subject, its metadata saying
// which module, seat, scope and machine it is. The same shape the Go runtime answers.
import { StringCodec } from "nats";
import { asSchema } from "@novox/mesh-sdk/stdio";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import type { RuntimeBroker } from "./broker-nats.js";
const sc = StringCodec();
export const VERSION = "0.1.0";
export const INFO_RESPONSE = "io.nats.micro.v1.info_response";
export const PING_RESPONSE = "io.nats.micro.v1.ping_response";
export const STATS_RESPONSE = "io.nats.micro.v1.stats_response";
/** One tool served on one subject, as announced. */
export interface Endpoint {
kind: "tool" | "seat";
module: string;
tool: string;
seat?: string;
scope?: "mesh" | "node";
node: string;
description: string;
schema: unknown;
interchangeable: boolean;
subject: string;
queue?: string;
}
/** A seat's verb as the runtime serves it, with the definition that answers it. */
export interface ServedSeatVerb {
seat: string;
verb: string;
subject: string;
holder: string;
tool: ToolDefinition;
}
export interface Service {
name: string;
id: string;
description: string;
metadata: Record<string, string>;
}
/** The endpoint's name as the protocol allows it; the metadata, not the name, identifies it. */
function nameOf(e: Endpoint): string {
const clean = (s: string) => s.replace(/[^A-Za-z0-9_-]/g, "_");
return `${clean(e.kind === "seat" ? e.seat ?? "" : e.module)}__${clean(e.tool)}`;
}
/** The info_response for these endpoints. */
export function info(s: Service, endpoints: Endpoint[]): Record<string, unknown> {
return {
name: s.name, id: s.id, version: VERSION, metadata: s.metadata, type: INFO_RESPONSE, description: s.description,
endpoints: endpoints.map((e) => {
const metadata: Record<string, string> = {
kind: e.kind, module: e.module, tool: e.tool, node: e.node, description: e.description,
schema: JSON.stringify(e.schema ?? {}), interchangeable: e.interchangeable ? "true" : "false",
};
if (e.kind === "seat") {
metadata.seat = e.seat ?? "";
metadata.scope = e.scope ?? "mesh";
}
return { name: nameOf(e), subject: e.subject, queue_group: e.queue ?? "", metadata };
}),
};
}
/** Everything served now: each module's tools on every subject issued for them, and each held
* seat's verbs on the seat's subject. */
export function endpointsOf(
broker: RuntimeBroker,
own: { module: string; tools: ToolDefinition[] }[],
seats: ServedSeatVerb[],
): Endpoint[] {
const node = broker.node ?? "";
const out: Endpoint[] = [];
for (const { module, tools } of own) {
for (const t of tools) {
const served = broker.servedOn ? broker.servedOn(module, t.name) : [];
const interchangeable = served.some((s) => s.subject === `mesh.mod.${module}.tool.${t.name}`);
for (const s of served) {
out.push({ kind: "tool", module, tool: t.name, node, description: t.description, schema: asSchema(t.input),
interchangeable, subject: s.subject, queue: s.queue });
}
}
}
for (const v of seats) {
out.push({ kind: "seat", module: v.holder, tool: v.verb, seat: v.seat,
scope: node && v.subject.endsWith(`.${node}`) ? "node" : "mesh", node, description: v.tool.description,
schema: asSchema(v.tool.input), interchangeable: false, subject: v.subject });
}
return out;
}
/** Answer discovery for one service until stopped. */
export function announce(broker: RuntimeBroker, s: Service, current: () => Endpoint[]): () => void {
const started = new Date().toISOString();
const identity = { name: s.name, id: s.id, version: VERSION, metadata: s.metadata };
const answer = (subject: string): Uint8Array | undefined => {
const parts = subject.split(".");
if (parts[0] !== "$SRV" || parts.length < 2) return undefined;
if (parts.length >= 3 && parts[2] !== s.name) return undefined; // another service's
if (parts.length >= 4 && parts[3] !== s.id) return undefined; // another instance's
let v: unknown;
switch (parts[1]) {
case "PING":
v = { ...identity, type: PING_RESPONSE };
break;
case "INFO":
v = info(s, current());
break;
case "STATS":
v = { ...identity, type: STATS_RESPONSE, started, endpoints: current().map((e) => ({
name: nameOf(e), subject: e.subject, queue_group: e.queue ?? "", num_requests: 0, num_errors: 0,
last_error: "", processing_time: 0, average_processing_time: 0 })) };
break;
default:
return undefined;
}
return sc.encode(JSON.stringify(v));
};
const stops: (() => void)[] = [];
// Exactly the questions asked of every service and of this one by name and instance — what the
// grants allow (novox/hq ADR 0197). A wildcard is refused by the bus, and a refused subscription
// ends a runtime: on 2026-10-03 it crash-looped every container that announced itself.
for (const verb of ["PING", "INFO", "STATS"]) {
for (const subject of [`$SRV.${verb}`, `$SRV.${verb}.${s.name}`, `$SRV.${verb}.${s.name}.${s.id}`]) {
stops.push(broker.raw!(subject, (subj) => answer(subj)));
}
}
return () => stops.forEach((stop) => stop());
}
+29
View File
@@ -97,6 +97,13 @@ export interface RuntimeBroker extends Broker {
serving(): string[]; serving(): string[];
/** The module this connection is: what its credential named, and what a bare key serves as. */ /** The module this connection is: what its credential named, and what a bare key serves as. */
readonly module: string; readonly module: string;
/** Where a served module's tool is answered right now (ADR 0197: what it serves, it announces). */
servedOn?(module: string, tool: string): { subject: string; queue?: string }[];
/** Answer a subject in a format of its own, not a tool's reply envelope — the NATS services
* protocol's discovery (novox/hq ADR 0197). An undefined answer is no reply. */
raw?(subject: string, answer: (subject: string, data: Uint8Array) => Uint8Array | undefined): () => void;
/** The machine this connection serves on, when its credential names one. */
readonly node?: string;
} }
/** /**
@@ -348,6 +355,28 @@ export async function connectNats(
}, },
module: self, module: self,
node,
servedOn,
raw(subject: string, answer: (subject: string, data: Uint8Array) => Uint8Array | undefined): () => void {
const sub = conn.subscribe(subject);
subs.push(sub);
void (async () => {
// **A refusal here is said, never fatal** (novox/hq 04-ISSUES/217). This serves discovery —
// what the runtime says about itself — not the work; a bus that refuses it costs the mesh
// seeing this runtime, not the runtime's tools and handlers. Unhandled, the refusal ended the
// process and every per-module container crash-looped on 2026-10-03.
try {
for await (const msg of sub) {
const body = answer(msg.subject, msg.data);
if (body) msg.respond(body);
}
} catch (err) {
console.log(`[mesh-tools] the bus refused ${subject}: ${err instanceof Error ? err.message : String(err)}; ` +
"discovery will not see this runtime there, and it serves on");
}
})();
return () => sub.unsubscribe();
},
follow, follow,
serving: () => [...issued.keys()], serving: () => [...issued.keys()],
membership: (module?: string) => issued.get(module ?? self), membership: (module?: string) => issued.get(module ?? self),
+27 -4
View File
@@ -13,6 +13,8 @@
import { spawn, type ChildProcess } from "node:child_process"; import { spawn, type ChildProcess } from "node:child_process";
import { accessSync, constants } from "node:fs"; import { accessSync, constants } from "node:fs";
import type { ToolDefinition } from "@novox/mesh-sdk/tools"; import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import { broker, type Envelope } from "@novox/mesh-sdk/messaging";
import { atWork } from "./broker-nats.js";
/** The protocol version this speaks; a bundle says the same. */ /** The protocol version this speaks; a bundle says the same. */
export const PROTOCOL = "2025-03-26"; export const PROTOCOL = "2025-03-26";
@@ -71,29 +73,50 @@ export async function launch(module: string, entry: string, env: NodeJS.ProcessE
const line = buffered.slice(0, at).trim(); const line = buffered.slice(0, at).trim();
buffered = buffered.slice(at + 1); buffered = buffered.slice(at + 1);
if (!line) continue; if (!line) continue;
let reply: { id?: number; result?: unknown; error?: { message?: string } }; let reply: { id?: number | string; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
try { try {
reply = JSON.parse(line); reply = JSON.parse(line);
} catch { } catch {
console.log(`[mesh-tools] ${module}'s bundle said something that is not a reply: ${line.slice(0, 120)}`); console.log(`[mesh-tools] ${module}'s bundle said something that is not a reply: ${line.slice(0, 120)}`);
continue; continue;
} }
// **The bundle asks the runtime to emit** (novox/hq ADR 0193): published on the bus as this
// module, and answered once the bus has accepted it, so the tool's emit means what it means
// in-process. Nothing else a bundle may ask.
if (typeof reply.method === "string") {
const id = reply.id;
const answer = (m: Record<string, unknown>) => proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", id, ...m }) + "\n");
if (reply.method !== "mesh/publish") {
if (id !== undefined) answer({ error: { code: -32601, message: `the runtime answers no ${reply.method} from a bundle` } });
continue;
}
atWork.run({ module }, () => broker().publish(reply.params as Envelope<unknown>))
.then(() => { if (id !== undefined) answer({ result: {} }); })
.catch((err: unknown) => { if (id !== undefined) answer({ error: { code: -32000, message: err instanceof Error ? err.message : String(err) } }); });
continue;
}
const waiting = typeof reply.id === "number" ? pending.get(reply.id) : undefined; const waiting = typeof reply.id === "number" ? pending.get(reply.id) : undefined;
if (!waiting) continue; if (!waiting) continue;
pending.delete(reply.id!); pending.delete(reply.id as number);
clearTimeout(waiting.timer); clearTimeout(waiting.timer);
if (reply.error) waiting.reject(new Error(reply.error.message ?? "the bundle refused the request")); if (reply.error) waiting.reject(new Error(reply.error.message ?? "the bundle refused the request"));
else waiting.resolve(reply.result); else waiting.resolve(reply.result);
} }
}); });
// stderr is the bundle's log; kept under the module's name so a fault reads where it belongs. // stderr is the bundle's log; kept under the module's name so a fault reads where it belongs.
// The last thing it said is kept, so a bundle that dies says why in its own words, not by code.
let lastSaid = "";
proc.stderr!.on("data", (chunk: Buffer) => { proc.stderr!.on("data", (chunk: Buffer) => {
for (const line of chunk.toString("utf8").split("\n")) if (line.trim()) console.log(`[${module}] ${line}`); for (const line of chunk.toString("utf8").split("\n")) {
if (!line.trim()) continue;
console.log(`[${module}] ${line}`);
if (/\S/.test(line) && !/^\s+at\s/.test(line) && !/^Node\.js v/.test(line)) lastSaid = line.trim();
}
}); });
const exited = new Promise<never>((_, reject) => { const exited = new Promise<never>((_, reject) => {
proc.once("error", (err) => reject(err)); proc.once("error", (err) => reject(err));
proc.once("exit", (code, signal) => { proc.once("exit", (code, signal) => {
const why = `${module}'s bundle exited (${signal ?? code})`; const why = `${module}'s bundle exited (${signal ?? code})` + (lastSaid ? `: ${lastSaid}` : "");
for (const [id, p] of pending) { for (const [id, p] of pending) {
pending.delete(id); pending.delete(id);
clearTimeout(p.timer); clearTimeout(p.timer);
+4 -2
View File
@@ -29,7 +29,7 @@
import { readFileSync } from "node:fs"; import { readFileSync } from "node:fs";
import { pathToFileURL } from "node:url"; import { pathToFileURL } from "node:url";
import { connectNats, fatalBrokerReason as fatalNatsReason, type Credential } from "./broker-nats.js"; import { connectNats, fatalBrokerReason as fatalNatsReason, type Credential } from "./broker-nats.js";
import { runTools, type ServedModule } from "./runtime.js"; import { takeToolEnvs, runTools, type ServedModule } from "./runtime.js";
import { serveMcpHttp, type Listening } from "./http.js"; import { serveMcpHttp, type Listening } from "./http.js";
/** The credential this process connected with, for what it says beyond the connection (ADR 0159). */ /** The credential this process connected with, for what it says beyond the connection (ADR 0159). */
@@ -163,7 +163,9 @@ async function serve(): Promise<void> {
const broker = await connectBrokerPatiently(); const broker = await connectBrokerPatiently();
// Parsed after connecting: a bare entrypoint belongs to the module the credential names. // Parsed after connecting: a bare entrypoint belongs to the module the credential names.
const { serves, moduleEntrypoints } = servedModulesFrom(process.env.MESH_TOOL_MODULES ?? "", lastCredential?.module); const { serves, moduleEntrypoints } = servedModulesFrom(process.env.MESH_TOOL_MODULES ?? "", lastCredential?.module);
const stop = await runTools({ broker, serves, moduleEntrypoints, credential: lastCredential }); // Each module's environment, composed by the mesh (ADR 0192): taken before any bundle is imported.
const envs = takeToolEnvs();
const stop = await runTools({ broker, serves, moduleEntrypoints, credential: lastCredential, envs });
// The console is this runtime's serving mode (ADR 0175 §6): as node-tools, or wherever the // The console is this runtime's serving mode (ADR 0175 §6): as node-tools, or wherever the
// listen address is given, the same process answers MCP on loopback for whoever is on the // listen address is given, the same process answers MCP on loopback for whoever is on the
+86 -20
View File
@@ -16,6 +16,7 @@ import { collectTools, toolKey, type ToolDefinition } from "@novox/mesh-sdk/tool
import type { Broker } from "@novox/mesh-sdk/messaging"; import type { Broker } from "@novox/mesh-sdk/messaging";
import { atWork, seatToolSubject, type Credential, type RuntimeBroker } from "./broker-nats.js"; import { atWork, seatToolSubject, type Credential, type RuntimeBroker } from "./broker-nats.js";
import { launch, launches } from "./launch.js"; import { launch, launches } from "./launch.js";
import { announce, endpointsOf, type ServedSeatVerb } from "./announce.js";
/** /**
* The one verb every module's runtime answers for it (novox/hq ADR 0152, design 34 §3): the * The one verb every module's runtime answers for it (novox/hq ADR 0152, design 34 §3): the
@@ -59,6 +60,39 @@ export interface RuntimeOptions {
* (novox/hq ADR 0159). Absent for a runtime started by hand, which then serves no seat its * (novox/hq ADR 0159). Absent for a runtime started by hand, which then serves no seat its
* memberships do not name. */ * memberships do not name. */
credential?: Credential; credential?: Credential;
/** What each served module's bundles are given (novox/hq ADR 0192): module → words, composed by
* the mesh per machine. A module absent here is given the runtime's own words and nothing more. */
envs?: ReadonlyMap<string, Readonly<Record<string, string>>>;
}
/** The variable the mesh composes every served module's environment into, as JSON (ADR 0192). Read
* once at start and removed from the process's environment, so no bundle finds another's there. */
export const TOOL_ENV = "MESH_TOOL_ENV";
/** Read and remove the composed environments from an environment (the process's, by default). */
export function takeToolEnvs(env: NodeJS.ProcessEnv = process.env): Map<string, Record<string, string>> {
const raw = env[TOOL_ENV];
delete env[TOOL_ENV];
const out = new Map<string, Record<string, string>>();
if (!raw) return out;
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
throw new Error(`${TOOL_ENV} is not JSON; the mesh composes it as {"<module>": {"<word>": "<value>"}}`);
}
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
throw new Error(`${TOOL_ENV} is not an object of modules`);
}
for (const [module, words] of Object.entries(parsed as Record<string, unknown>)) {
if (!words || typeof words !== "object" || Array.isArray(words)) {
throw new Error(`${TOOL_ENV}: ${module}'s environment is not an object of words`);
}
const own: Record<string, string> = {};
for (const [k, v] of Object.entries(words as Record<string, unknown>)) own[k] = String(v);
out.set(module, own);
}
return out;
} }
/** Two environment words the mesh sets for the node's runtime and every tool reads from its /** Two environment words the mesh sets for the node's runtime and every tool reads from its
@@ -79,6 +113,7 @@ export async function runTools(opts: RuntimeOptions): Promise<() => void> {
for (const s of opts.serves ?? []) { for (const s of opts.serves ?? []) {
served.set(s.module, [...(served.get(s.module) ?? []), ...s.entrypoints]); served.set(s.module, [...(served.get(s.module) ?? []), ...s.entrypoints]);
} }
const ownEntrypoints = opts.moduleEntrypoints ?? [];
if (opts.moduleEntrypoints?.length) { if (opts.moduleEntrypoints?.length) {
if (!self) { if (!self) {
throw new Error( throw new Error(
@@ -103,30 +138,42 @@ export async function runTools(opts: RuntimeOptions): Promise<() => void> {
for (const module of served.keys()) await runtime.follow(module); for (const module of served.keys()) await runtime.follow(module);
} }
// Import each bundle, guarded (ADR 0175: one faulty bundle must not take the node's tools down). // What each module's bundles are given: the runtime's own words, and over them the module's own.
// Importing the entrypoint runs its registerModuleTools(...) — that is the whole handshake — and const envs = opts.envs ?? new Map<string, Record<string, string>>();
// the registrations it adds are the ones that appear after it, which is how each is attributed const envFor = (module: string): NodeJS.ProcessEnv => ({ ...process.env, ...(envs.get(module) ?? {}) });
// to the module whose bundle made it.
// A bundle that is not plain JavaScript — or is marked executable — is launched as a process // Every bundle this runtime serves is launched as a process and spoken to over MCP on stdio (ADR
// and spoken to over MCP on stdio instead (ADR 0188); what it lists is registered the same way. // 0188, ADR 0193): given the runtime's words and its module's own, and told the module it serves
// it as. The runtime knows no language; an entrypoint that is not executable was not built to be
// served, and is refused by name. A bundle that fails to start is named, and the others serve
// (ADR 0175: one faulty bundle must not take the node's tools down).
//
// The one-module form — the credential's own module's entrypoints, which the per-module containers
// still use for their event handlers and provisioners until they move (to-be 38 WP4c) — is imported
// into this process as before: one module, one SDK, its container's own environment.
// The machine this runtime serves, which an event a launched tool emits is stamped with.
const node = opts.credential?.node ?? (typeof (runtime as unknown as { node?: unknown }).node === "string" ? (runtime as unknown as { node: string }).node : undefined);
const failed = new Map<string, string>(); const failed = new Map<string, string>();
const owner: string[] = []; // registration index → the module whose bundle registered it
const launched: { module: string; owner: string; tools: ToolDefinition[] }[] = []; const launched: { module: string; owner: string; tools: ToolDefinition[] }[] = [];
const children: Array<() => void> = []; const children: Array<() => void> = [];
for (const [module, entrypoints] of served) { for (const [module, entrypoints] of served) {
const imported = module === self && ownEntrypoints.length > 0;
for (const entry of entrypoints) { for (const entry of entrypoints) {
const path = resolve(entry); const path = resolve(entry);
try { try {
if (launches(path)) { if (imported) {
const child = await launch(module, path); await import(pathToFileURL(path).href);
children.push(child.stop);
for (const r of child.registrations) launched.push({ ...r, owner: module });
continue; continue;
} }
const before = collectTools().length; if (!launches(path)) {
await import(pathToFileURL(path).href); throw new Error(`${path} is not executable; a bundle the runtime serves is started, never imported, and its build makes it executable (novox/hq ADR 0193)`);
const after = collectTools().length; }
for (let i = before; i < after; i++) owner[i] = module; const child = await launch(module, path, {
...envFor(module), MESH_SERVED_MODULE: module, MESH_MODULE: module,
...(node ? { MESH_NODE: node } : {}),
});
children.push(child.stop);
for (const r of child.registrations) launched.push({ ...r, owner: module });
} catch (err) { } catch (err) {
const why = err instanceof Error ? err.message : String(err); const why = err instanceof Error ? err.message : String(err);
failed.set(module, why); failed.set(module, why);
@@ -144,7 +191,7 @@ export async function runTools(opts: RuntimeOptions): Promise<() => void> {
// implement a seat did not yet name the claim, and the whole runtime restarted for it. // implement a seat did not yet name the claim, and the whole runtime restarted for it.
const claimed = seatsClaimed(served.keys(), self, opts.credential, runtime); const claimed = seatsClaimed(served.keys(), self, opts.credential, runtime);
const registrations = [ const registrations = [
...collectTools().map((r, i) => ({ ...r, owner: owner[i] ?? self ?? r.module })), ...collectTools().map((r) => ({ ...r, owner: self ?? r.module })),
...launched, ...launched,
]; ];
const ownRegistrations = registrations.filter(({ module, owner: by }) => { const ownRegistrations = registrations.filter(({ module, owner: by }) => {
@@ -213,7 +260,16 @@ export async function runTools(opts: RuntimeOptions): Promise<() => void> {
console.log(`[mesh-tools] serving ${names.length} tool(s) for ${served.size} module(s): ${names.join(", ") || "(none)"}` + console.log(`[mesh-tools] serving ${names.length} tool(s) for ${served.size} module(s): ${names.join(", ") || "(none)"}` +
(failed.size ? `; not serving ${[...failed.keys()].join(", ")}, whose bundle(s) failed to load` : "")); (failed.size ? `; not serving ${[...failed.keys()].join(", ")}, whose bundle(s) failed to load` : ""));
stops.push(await serveClaimedSeats(runtime, [...served.keys()], self, opts.credential, registrations)); const seats = await serveClaimedSeats(runtime, [...served.keys()], self, opts.credential, registrations);
stops.push(seats.stop);
// **What it serves, it announces** (novox/hq ADR 0197), asked at the moment of the request.
if (typeof runtime.raw === "function") {
stops.push(announce(runtime, {
name: self ?? "runtime", id: runtime.node ?? self ?? "runtime",
description: `the tool runtime of ${self ?? "a module"}${runtime.node ? ` on ${runtime.node}` : ""}`,
metadata: runtime.node ? { node: runtime.node } : {},
}, () => endpointsOf(runtime, ownRegistrations, seats.serving())));
}
return () => stop(); return () => stop();
} }
@@ -257,11 +313,17 @@ async function serveClaimedSeats(
self: string | undefined, self: string | undefined,
credential: Credential | undefined, credential: Credential | undefined,
registrations: { module: string; owner: string; tools: ToolDefinition[] }[], registrations: { module: string; owner: string; tools: ToolDefinition[] }[],
): Promise<() => void> { ): Promise<{ stop: () => void; serving: () => ServedSeatVerb[] }> {
if (typeof broker.handleSubject !== "function") return () => {}; if (typeof broker.handleSubject !== "function") return { stop: () => {}, serving: () => [] };
// A seat's verbs are the role's, not the software's (ADR 0159): implemented under the seat's // A seat's verbs are the role's, not the software's (ADR 0159): implemented under the seat's
// name — `registerModuleTools("mesh-store", …)` — and never confused with the module's own tools. // name — `registerModuleTools("mesh-store", …)` — and never confused with the module's own tools.
const implementations = new Map<string, Map<string, (args: Record<string, unknown>) => Promise<unknown>>>(); const implementations = new Map<string, Map<string, (args: Record<string, unknown>) => Promise<unknown>>>();
const definitions = new Map<string, Map<string, ToolDefinition>>();
for (const { module, tools } of registrations) {
const defs = definitions.get(module) ?? new Map<string, ToolDefinition>();
for (const t of tools) defs.set(t.name, t);
definitions.set(module, defs);
}
for (const { module, owner, tools } of registrations) { for (const { module, owner, tools } of registrations) {
const verbs = implementations.get(module) ?? new Map<string, (args: Record<string, unknown>) => Promise<unknown>>(); const verbs = implementations.get(module) ?? new Map<string, (args: Record<string, unknown>) => Promise<unknown>>();
for (const t of tools) verbs.set(t.name, (args) => atWork.run({ module: owner }, () => t.run(args))); for (const t of tools) verbs.set(t.name, (args) => atWork.run({ module: owner }, () => t.run(args)));
@@ -295,21 +357,25 @@ async function serveClaimedSeats(
}; };
let stops: (() => void)[] = []; let stops: (() => void)[] = [];
let servingNow: ServedSeatVerb[] = [];
const serve = async (): Promise<void> => { const serve = async (): Promise<void> => {
stops.forEach((s) => s()); stops.forEach((s) => s());
stops = []; stops = [];
servingNow = [];
for (const v of wanted()) { for (const v of wanted()) {
const run = implementations.get(v.seat)?.get(v.verb); const run = implementations.get(v.seat)?.get(v.verb);
const tool = definitions.get(v.seat)?.get(v.verb);
if (!run) { if (!run) {
console.log(`[mesh-tools] ${v.holder} claims ${v.seat} and implements no ${v.verb}, which that seat promises; not served`); console.log(`[mesh-tools] ${v.holder} claims ${v.seat} and implements no ${v.verb}, which that seat promises; not served`);
continue; continue;
} }
stops.push(await broker.handleSubject(v.subject, run)); stops.push(await broker.handleSubject(v.subject, run));
if (tool) servingNow.push({ ...v, tool });
console.log(`[mesh-tools] serving ${v.seat}'s ${v.verb} on ${v.subject}, admitted where ${v.holder} holds the seat`); console.log(`[mesh-tools] serving ${v.seat}'s ${v.verb} on ${v.subject}, admitted where ${v.holder} holds the seat`);
} }
}; };
await serve(); await serve();
// A membership issued to any served module may add, move or withdraw a seat's verbs. // A membership issued to any served module may add, move or withdraw a seat's verbs.
if (typeof broker.onMembership === "function") broker.onMembership(() => void serve()); if (typeof broker.onMembership === "function") broker.onMembership(() => void serve());
return () => stops.forEach((s) => s()); return { stop: () => stops.forEach((s) => s()), serving: () => servingNow };
} }
+69
View File
@@ -0,0 +1,69 @@
/**
* What a runtime serves, it announces (novox/hq ADR 0197): the TypeScript runtime the per-module
* containers still run answers the NATS services protocol's discovery in the same shape as the Go
* tool runtime — its module's tools on every subject issued, and the seat verbs it serves.
*
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/announce.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { fileURLToPath } from "node:url";
import { connect, StringCodec } from "nats";
import { resetTools } from "@novox/mesh-sdk/tools";
import { connectNats, membershipSubject } from "../dist/broker-nats.js";
import { runTools } from "../dist/runtime.js";
const url = process.env.MESH_TEST_NATS;
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
const sc = StringCodec();
test("the runtime answers $SRV.INFO with what it serves, in the services protocol's format", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const nc = await connect({ servers: url });
const jsm = await nc.jetstreamManager();
try {
await jsm.streams.delete("ASSIGNMENTS");
} catch {
// none yet
}
await jsm.streams.add({ name: "ASSIGNMENTS", subjects: ["mesh.assignment.>"], max_msgs_per_subject: 1, allow_direct: true } as never);
await nc.jetstream().publish(membershipSubject("anchor", "shop"), sc.encode(JSON.stringify({
node: "anchor", module: "shop",
serves: [{ subject: "mesh.mod.shop.tool.{tool}.anchor" }, { subject: "mesh.mod.shop.tool.{tool}", queue: "serve.shop" }],
emits: "mesh.mod.shop.event.{event}", tools: "mesh.mod.shop.tool.tools",
})));
const shop = await connectNats({ url, node: "anchor", module: "shop" });
let stop = () => {};
try {
stop = await runTools({ broker: shop, moduleEntrypoints: [fixture("shop-tools.mjs")] });
const msg = await nc.request("$SRV.INFO.shop.anchor", sc.encode(""), { timeout: 2000 });
const info = JSON.parse(sc.decode(msg.data)) as {
type: string; name: string; id: string; version: string;
endpoints: { name: string; subject: string; queue_group: string; metadata: Record<string, string> }[];
};
assert.equal(info.type, "io.nats.micro.v1.info_response");
assert.equal(info.name, "shop");
assert.equal(info.id, "anchor");
assert.ok(info.version);
const price = info.endpoints.filter((e) => e.metadata.tool === "price").map((e) => `${e.subject}|${e.queue_group}`).sort();
assert.deepEqual(price, ["mesh.mod.shop.tool.price.anchor|", "mesh.mod.shop.tool.price|serve.shop"]);
const one = info.endpoints.find((e) => e.metadata.tool === "price")!;
assert.equal(one.metadata.kind, "tool");
assert.equal(one.metadata.module, "shop");
assert.equal(one.metadata.node, "anchor");
assert.equal(one.metadata.interchangeable, "true");
assert.ok(JSON.parse(one.metadata.schema).type === "object");
// Ping answers with the same identity; another service's request is not answered.
const ping = JSON.parse(sc.decode((await nc.request("$SRV.PING", sc.encode(""), { timeout: 2000 })).data));
assert.equal(ping.type, "io.nats.micro.v1.ping_response");
await assert.rejects(nc.request("$SRV.INFO.somebody-else", sc.encode(""), { timeout: 300 }));
} finally {
stop();
await shop.close();
await nc.close();
resetTools();
}
});
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for clash-tools.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./clash-tools.mjs");
await serveRegisteredOverStdio();
+4
View File
@@ -0,0 +1,4 @@
import { registerModuleTools } from "@novox/mesh-sdk/tools";
registerModuleTools("delta", (env) => [
{ name: "given", description: "what delta was given", input: {}, run: async () => ({ mine: env.DELTA_TOKEN_FILE ?? null, theirs: env.GAMMA_CONFIG_FILE ?? null }) },
]);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for env-delta.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./env-delta.mjs");
await serveRegisteredOverStdio();
+5
View File
@@ -0,0 +1,5 @@
// A bundle that reads what it was given (novox/hq ADR 0192): its contributor's environment.
import { registerModuleTools } from "@novox/mesh-sdk/tools";
registerModuleTools("gamma", (env) => [
{ name: "given", description: "what gamma was given", input: {}, run: async () => ({ mine: env.GAMMA_CONFIG_FILE ?? null, theirs: env.DELTA_TOKEN_FILE ?? null, runtime: env.MESH_OPERATOR_ACCOUNT ?? null, composed: env.MESH_TOOL_ENV ?? null }) },
]);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for env-gamma.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./env-gamma.mjs");
await serveRegisteredOverStdio();
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env node
// Launched (ADR 0188): its environment is the child's own.
import { serveStdio } from "@novox/mesh-sdk/stdio";
await serveStdio("zeta", [
{ name: "given", description: "what zeta was given", input: {}, run: async () => ({ mine: process.env.ZETA_URL ?? null, theirs: process.env.GAMMA_CONFIG_FILE ?? null, composed: process.env.MESH_TOOL_ENV ?? null }) },
]);
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env node
// A long-running bundle that keeps exiting (novox/hq ADR 0198): the runtime starts it again each time.
import { appendFileSync } from "node:fs";
import { serveStdio } from "@novox/mesh-sdk/stdio";
appendFileSync(process.env.FLAKY_LOG, "started\n");
setTimeout(() => process.exit(3), 300);
await serveStdio("flaky", []);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for many-alpha.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./many-alpha.mjs");
await serveRegisteredOverStdio();
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for many-beta.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./many-beta.mjs");
await serveRegisteredOverStdio();
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for many-broken.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./many-broken.mjs");
await serveRegisteredOverStdio();
+10
View File
@@ -0,0 +1,10 @@
# A bus that refuses one subject: what the mesh's grants do to a subject they do not name (issue 217).
authorization {
users = [
{ user: "runtime", password: "runtime", permissions: {
publish: { allow: [">"] }
subscribe: { allow: [">"], deny: ["$SRV.PING.>"] }
} }
]
}
jetstream: enabled
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env node
// A launched bundle registering its seat before its own tools, served through the SDK's loop
// (novox/hq ADR 0193): which is the seat's and which the module's comes from the name it is given.
import { registerModuleTools } from "@novox/mesh-sdk/tools";
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
registerModuleTools("node-shelf", () => [{ name: "list", description: "the shelf", input: {}, run: async () => ({ shelf: ["x"] }) }]);
registerModuleTools("theta", (env) => [{ name: "own", description: "theta's", input: {}, run: async () => ({ theta: env.THETA_WORD ?? null }) }]);
await serveRegisteredOverStdio();
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for shop-seat.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./shop-seat.mjs");
await serveRegisteredOverStdio();
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for shop-tools.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./shop-tools.mjs");
await serveRegisteredOverStdio();
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for store-seat-unclaimed.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./store-seat-unclaimed.mjs");
await serveRegisteredOverStdio();
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for store-seat.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./store-seat.mjs");
await serveRegisteredOverStdio();
+24
View File
@@ -0,0 +1,24 @@
// A module whose code runs long (novox/hq ADR 0198): it subscribes as it is imported, handles each
// event once it can, fails the first time it is asked to, dies the first time it is told to, and one
// tool asks another module's tool through the runtime. What it did is written to WATCH_LOG.
import { appendFileSync, existsSync, writeFileSync } from "node:fs";
import { on } from "@novox/mesh-sdk/events";
import { broker } from "@novox/mesh-sdk/messaging";
import { registerModuleTools } from "@novox/mesh-sdk/tools";
const log = process.env.WATCH_LOG;
const once = (mark) => {
const file = `${log}.${mark}`;
if (existsSync(file)) return false;
writeFileSync(file, "1");
return true;
};
appendFileSync(log, "started\n");
await on("alpha.happened", async (e) => {
if (e.body.fail && once(`fail-${e.body.n}`)) throw new Error(`not yet ${e.body.n}`);
if (e.body.die && once(`die-${e.body.n}`)) process.exit(7);
appendFileSync(log, `handled ${e.body.n}\n`);
});
registerModuleTools("watcher", () => [
{ name: "relay", description: "asks beta", input: {}, run: async () => broker().request("beta.three", {}) },
]);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for watcher.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./watcher.mjs");
await serveRegisteredOverStdio();
+163 -7
View File
@@ -11,6 +11,9 @@
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { test } from "node:test"; import { test } from "node:test";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import { cpSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { connect, StringCodec } from "nats"; import { connect, StringCodec } from "nats";
import { resetTools } from "@novox/mesh-sdk/tools"; import { resetTools } from "@novox/mesh-sdk/tools";
@@ -18,7 +21,7 @@ import { resetTools } from "@novox/mesh-sdk/tools";
import { connectNats, membershipSubject } from "../dist/broker-nats.js"; import { connectNats, membershipSubject } from "../dist/broker-nats.js";
import { callTool, toolsOn } from "../dist/client.js"; import { callTool, toolsOn } from "../dist/client.js";
import { servedModulesFrom } from "../dist/main.js"; import { servedModulesFrom } from "../dist/main.js";
import { runTools } from "../dist/runtime.js"; import { runTools, takeToolEnvs } from "../dist/runtime.js";
const url = process.env.MESH_TEST_NATS; const url = process.env.MESH_TEST_NATS;
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url)); const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
@@ -119,9 +122,9 @@ test("the node's runtime serves five modules' bundles on one credential — two
broker: nodeTools, broker: nodeTools,
credential, credential,
serves: [ serves: [
{ module: "alpha", entrypoints: [fixture("many-alpha.mjs")] }, { module: "alpha", entrypoints: [fixture("many-alpha.serve.mjs")] },
{ module: "beta", entrypoints: [fixture("many-beta.mjs")] }, { module: "beta", entrypoints: [fixture("many-beta.serve.mjs")] },
{ module: "gamma", entrypoints: [fixture("many-broken.mjs")] }, { module: "gamma", entrypoints: [fixture("many-broken.serve.mjs")] },
{ module: "delta", entrypoints: [fixture("many-delta.py")] }, { module: "delta", entrypoints: [fixture("many-delta.py")] },
{ module: "epsilon", entrypoints: [fixture("many-epsilon.mjs")] }, { module: "epsilon", entrypoints: [fixture("many-epsilon.mjs")] },
], ],
@@ -129,7 +132,7 @@ test("the node's runtime serves five modules' bundles on one credential — two
console.log = log; console.log = log;
assert.deepEqual(nodeTools.serving().sort(), ["alpha", "beta", "delta", "epsilon", "gamma", "node-tools"]); assert.deepEqual(nodeTools.serving().sort(), ["alpha", "beta", "delta", "epsilon", "gamma", "node-tools"]);
assert.ok(said.some((s) => /the operator's account here is somebody \(home \/home\/somebody\)/.test(s)), said.join("\n")); assert.ok(said.some((s) => /the operator's account here is somebody \(home \/home\/somebody\)/.test(s)), said.join("\n"));
assert.ok(said.some((s) => /gamma's bundle .*many-broken\.mjs failed to load: gamma's bundle cannot find its client; its tools are not served here/.test(s)), said.join("\n")); assert.ok(said.some((s) => /gamma's bundle .*many-broken\.serve\.mjs failed to load: gamma's bundle exited \(1\): Error: gamma's bundle cannot find its client; its tools are not served here/.test(s)), said.join("\n"));
assert.ok(said.some((s) => /serving 8 tool\(s\) for 5 module\(s\): alpha\.one, alpha\.two, beta\.three, beta\.four, beta\.five, delta\.greet, delta\.die, epsilon\.seven; not serving gamma/.test(s)), said.join("\n")); assert.ok(said.some((s) => /serving 8 tool\(s\) for 5 module\(s\): alpha\.one, alpha\.two, beta\.three, beta\.four, beta\.five, delta\.greet, delta\.die, epsilon\.seven; not serving gamma/.test(s)), said.join("\n"));
// Five tools answer, each where its module's membership says: alpha anywhere and here, beta here only. // Five tools answer, each where its module's membership says: alpha anywhere and here, beta here only.
@@ -160,7 +163,7 @@ test("the node's runtime serves five modules' bundles on one credential — two
// `tools` answers for each: what alpha and beta serve, and why gamma serves nothing. // `tools` answers for each: what alpha and beta serve, and why gamma serves nothing.
const gamma = await asker.request<Record<string, never>, { module: string; tools: unknown[]; failed?: string }>("gamma.tools@anchor", {}); const gamma = await asker.request<Record<string, never>, { module: string; tools: unknown[]; failed?: string }>("gamma.tools@anchor", {});
assert.deepEqual(gamma, { module: "gamma", tools: [], failed: "gamma's bundle cannot find its client" }); assert.deepEqual(gamma, { module: "gamma", tools: [], failed: "gamma's bundle exited (1): Error: gamma's bundle cannot find its client" });
const beta = await asker.request<Record<string, never>, { tools: { name: string; subjects?: string[] }[] }>("beta.tools@anchor", {}); const beta = await asker.request<Record<string, never>, { tools: { name: string; subjects?: string[] }[] }>("beta.tools@anchor", {});
assert.deepEqual(beta.tools.map((x) => x.name), ["three", "four", "five"]); assert.deepEqual(beta.tools.map((x) => x.name), ["three", "four", "five"]);
assert.deepEqual(beta.tools[0]!.subjects, ["mesh.mod.beta.tool.three.anchor"]); assert.deepEqual(beta.tools[0]!.subjects, ["mesh.mod.beta.tool.three.anchor"]);
@@ -171,7 +174,7 @@ test("the node's runtime serves five modules' bundles on one credential — two
try { try {
const have = await toolsOn(asker); const have = await toolsOn(asker);
assert.deepEqual(have.tools.map((x) => `${x.module}.${x.name}`), ["alpha.one", "alpha.two", "beta.five", "beta.four", "beta.three", "delta.die", "delta.greet", "epsilon.seven"]); assert.deepEqual(have.tools.map((x) => `${x.module}.${x.name}`), ["alpha.one", "alpha.two", "beta.five", "beta.four", "beta.three", "delta.die", "delta.greet", "epsilon.seven"]);
assert.deepEqual(have.notAnswering, ["gamma (its tools bundle failed to load: gamma's bundle cannot find its client)", "mesh-controller (seat)"]); assert.deepEqual(have.notAnswering, ["gamma (its tools bundle failed to load: gamma's bundle exited (1): Error: gamma's bundle cannot find its client)", "mesh-controller (seat)"]);
} finally { } finally {
await catalogue.close(); await catalogue.close();
} }
@@ -192,3 +195,156 @@ test("the node's runtime serves five modules' bundles on one credential — two
resetTools(); resetTools();
} }
}); });
test("a bundle carrying its own copy of the SDK registers into the runtime's registry, and its tools are served (issue 209)", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
let dir = "";
let stop = () => {};
const closing: Array<() => Promise<void>> = [];
const said: string[] = [];
const log = console.log;
try {
// A bundle as the toolchain packs one: its compiled entrypoint, a package.json saying ES modules,
// and its dependencies copied in — the SDK among them, a second copy beside the runtime's own,
// and a dependency of the bundle's own that the runtime does not carry.
dir = mkdtempSync(join(tmpdir(), "mesh-bundle-"));
const sdk = realpathSync(fileURLToPath(new URL("../node_modules/@novox/mesh-sdk/", import.meta.url)));
cpSync(sdk, join(dir, "node_modules", "@novox", "mesh-sdk"), { recursive: true });
mkdirSync(join(dir, "node_modules", "zeta-flavour"), { recursive: true });
writeFileSync(join(dir, "node_modules", "zeta-flavour", "package.json"), '{"name":"zeta-flavour","type":"module","main":"index.js"}\n');
writeFileSync(join(dir, "node_modules", "zeta-flavour", "index.js"), 'export const flavour = "the bundle\'s own";\n');
writeFileSync(join(dir, "package.json"), '{"type":"module","private":true}\n');
writeFileSync(join(dir, "index.serve.mjs"),
'#!/usr/bin/env node\nimport { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";\nawait import("./index.js");\nawait serveRegisteredOverStdio();\n', { mode: 0o755 });
writeFileSync(join(dir, "index.js"),
'import { registerModuleTools } from "@novox/mesh-sdk/tools";\n' +
'import { flavour } from "zeta-flavour";\n' +
'registerModuleTools("zeta", () => [{ name: "probe", description: "answers", input: {}, run: async () => ({ zeta: true, flavour }) }]);\n');
const mesh = await aMesh();
closing.push(() => mesh.close());
await mesh.issue(membershipOf("zeta", "anchor"));
const credential = { url, node: "anchor", module: "node-tools" };
const nodeTools = await connectNats(credential);
closing.push(() => nodeTools.close());
const asker = await connectNats({ url, module: "console", node: "workstation" });
closing.push(() => asker.close());
console.log = (...a: unknown[]) => said.push(a.join(" "));
stop = await runTools({ broker: nodeTools, credential, serves: [{ module: "zeta", entrypoints: [join(dir, "index.serve.mjs")] }] });
console.log = log;
assert.ok(said.some((s) => /serving 1 tool\(s\) for 1 module\(s\): zeta\.probe/.test(s)), said.join("\n"));
// The SDK is the runtime's (the registration arrived); the bundle's other dependency is its own.
assert.deepEqual((await callTool(asker, "zeta.probe@anchor", {})).result, { zeta: true, flavour: "the bundle's own" });
} finally {
console.log = log;
stop();
for (const close of closing.reverse()) await close();
if (dir) rmSync(dir, { recursive: true, force: true });
resetTools();
}
});
test("each bundle is given its own environment and none of another's, imported or launched (ADR 0192)", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const mesh = await aMesh();
for (const m of ["gamma", "delta", "zeta"]) await mesh.issue(membershipOf(m, "anchor"));
const credential = { url, node: "anchor", module: "node-tools" };
const nodeTools = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
const log = console.log;
let stop = () => {};
const before = process.env.MESH_TOOL_ENV;
try {
process.env.MESH_OPERATOR_ACCOUNT = "somebody";
process.env.MESH_TOOL_ENV = JSON.stringify({
gamma: { GAMMA_CONFIG_FILE: "/var/lib/mesh/gamma/config.json" },
delta: { DELTA_TOKEN_FILE: "/var/lib/mesh/delta/token" },
zeta: { ZETA_URL: "http://127.0.0.1:3000" },
});
const envs = takeToolEnvs();
assert.equal(process.env.MESH_TOOL_ENV, undefined, "the composed environments were left in the process's");
console.log = () => {};
stop = await runTools({
broker: nodeTools, credential, envs,
serves: [
{ module: "gamma", entrypoints: [fixture("env-gamma.serve.mjs")] },
{ module: "delta", entrypoints: [fixture("env-delta.serve.mjs")] },
{ module: "zeta", entrypoints: [fixture("env-zeta.mjs")] },
],
});
console.log = log;
assert.deepEqual((await callTool(asker, "gamma.given@anchor", {})).result,
{ mine: "/var/lib/mesh/gamma/config.json", theirs: null, runtime: "somebody", composed: null });
assert.deepEqual((await callTool(asker, "delta.given@anchor", {})).result,
{ mine: "/var/lib/mesh/delta/token", theirs: null });
assert.deepEqual((await callTool(asker, "zeta.given@anchor", {})).result,
{ mine: "http://127.0.0.1:3000", theirs: null, composed: null });
} finally {
console.log = log;
if (before === undefined) delete process.env.MESH_TOOL_ENV; else process.env.MESH_TOOL_ENV = before;
delete process.env.MESH_OPERATOR_ACCOUNT;
stop();
await asker.close();
await nodeTools.close();
await mesh.close();
resetTools();
}
});
test("a launched bundle is told the module it serves, so its seat's verbs stay the seat's (ADR 0193)", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const mesh = await aMesh();
await mesh.issue(membershipOf("theta", "anchor", { seats: { "node-shelf": ["list"] } }));
const credential = { url, node: "anchor", module: "node-tools" };
const nodeTools = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
const log = console.log;
let stop = () => {};
try {
console.log = () => {};
stop = await runTools({ broker: nodeTools, credential, envs: new Map([["theta", { THETA_WORD: "given" }]]),
serves: [{ module: "theta", entrypoints: [fixture("served-seat-first.mjs")] }] });
console.log = log;
assert.deepEqual((await callTool(asker, "theta.own@anchor", {})).result, { theta: "given" });
assert.deepEqual((await callTool(asker, "seat:node-shelf.list@anchor", {})).result, { shelf: ["x"] });
} finally {
console.log = log;
stop();
await asker.close();
await nodeTools.close();
await mesh.close();
resetTools();
}
});
test("an entrypoint that is not executable is refused by name, and the others serve (ADR 0193)", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const mesh = await aMesh();
for (const m of ["alpha", "plain"]) await mesh.issue(membershipOf(m, "anchor"));
const credential = { url, node: "anchor", module: "node-tools" };
const nodeTools = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
const said: string[] = [];
const log = console.log;
let stop = () => {};
try {
console.log = (...a: unknown[]) => said.push(a.join(" "));
stop = await runTools({ broker: nodeTools, credential, serves: [
{ module: "alpha", entrypoints: [fixture("many-alpha.serve.mjs")] },
{ module: "plain", entrypoints: [fixture("many-alpha.mjs")] },
] });
console.log = log;
assert.ok(said.some((s) => /plain's bundle .*many-alpha\.mjs failed to load: .* is not executable; a bundle the runtime serves is started, never imported/.test(s)), said.join("\n"));
assert.deepEqual((await callTool(asker, "alpha.one@anchor", {})).result, { alpha: 1 });
} finally {
console.log = log;
stop();
await asker.close();
await nodeTools.close();
await mesh.close();
resetTools();
}
});
+1 -1
View File
@@ -36,7 +36,7 @@ test("as node-tools, serve loads the bundles and is the console on loopback", as
env: { env: {
...process.env, ...process.env,
MESH_BROKER_FILE: credential, MESH_BROKER_FILE: credential,
MESH_TOOL_MODULES: `alpha=${fixture("many-alpha.mjs")}`, MESH_TOOL_MODULES: `alpha=${fixture("many-alpha.serve.mjs")}`,
MESH_CONSOLE_LISTEN: "127.0.0.1:0", MESH_CONSOLE_LISTEN: "127.0.0.1:0",
}, },
stdio: ["ignore", "pipe", "pipe"], stdio: ["ignore", "pipe", "pipe"],
+37
View File
@@ -0,0 +1,37 @@
/**
* A refused announcement is said and never fatal (novox/hq 04-ISSUES/217): against a bus whose
* permissions refuse one discovery subject, the runtime's raw subscription is refused, logged, and
* the process keeps serving — its tools still answer.
*
* docker run -d --rm --name t -p 14233:4222 -v $PWD/test/fixtures/refusing-nats.conf:/c.conf nats:2.10-alpine -c /c.conf
* MESH_TEST_REFUSING_NATS=nats://127.0.0.1:14233 node --test --experimental-strip-types test/refused.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { connectNats } from "../dist/broker-nats.js";
const url = process.env.MESH_TEST_REFUSING_NATS;
test("a refused discovery subscription is logged and the runtime serves on", async (t) => {
if (!url) return t.skip("MESH_TEST_REFUSING_NATS unset");
const bus = await connectNats({ url, user: "runtime", password: "runtime", module: "alpha", node: "anchor" });
const said: string[] = [];
const log = console.log;
console.log = (...a: unknown[]) => said.push(a.join(" "));
const crashed: unknown[] = [];
const onRejection = (e: unknown) => crashed.push(e);
process.on("unhandledRejection", onRejection);
try {
(bus as unknown as { raw: (s: string, f: () => Uint8Array | undefined) => () => void }).raw("$SRV.PING.>", () => undefined);
const stop = await bus.handle("alpha.ping", async () => ({ pong: true }));
for (let i = 0; i < 50 && !said.some((s) => s.includes("the bus refused $SRV.PING.>")); i++) await new Promise((r) => setTimeout(r, 50));
console.log = log;
assert.ok(said.some((s) => /the bus refused \$SRV\.PING\.>.*serves on/.test(s)), said.join("\n"));
assert.equal(crashed.length, 0, `the refusal escaped: ${String(crashed[0])}`);
stop();
} finally {
console.log = log;
process.off("unhandledRejection", onRejection);
await bus.close();
}
});