Compare commits

...
149 Commits
Author SHA1 Message Date
mesh-admin 954e1a53ee Merge pull request 'Keep what discovery heard when a round hears nothing, and fetch only changed announcements (issue 554)' (#32) from fix/554-empty-discovery-round into main 2026-10-12 12:25:47 +00:00
jochen 424a7eec82 Keep what discovery heard when a round hears nothing, and fetch only changed announcements (issue 554)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
A round in which no PING reached the laptop's console within the 750 ms window
was taken as the mesh's answer that nothing is there, and replaced the index:
every seat and module call was refused for five seconds at a time. Discovery
now waits up to its patience for a first PING, a round that heard nothing is
not kept, a runtime not heard this round is resolved from what it announced
last (said with its age), and a round that misses runtimes is logged once.
A PING carries a digest of what the instance serves, so an unchanged
announcement is not pulled again every round (some 880 kB to the laptop).
2026-10-12 13:56:50 +02:00
mesh-admin 09c5f809cb Merge pull request 'The TypeScript toolchain holds pg, mongodb and mssql, so a catalogue check can check the modules that use them (issue 385)' (#31) from fix/385-an-untestable-module-fails-the-check into main 2026-10-12 10:46:07 +00:00
jochen 847055c758 Give the TypeScript toolchain pg, mongodb and mssql, so a catalogue check can type-check the modules that use them (issue 385)
mesh/merge-gate pass: builds mesh-tools, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/385-an-untestable-module-fails-the-check delivered: every member is delivered
A catalogue's TypeScript check fetches nothing and reads this image's
node_modules. Once a module it cannot check fails rather than passing with a
note, mesh-catalog, model-usage, mongodb and mssql could never merge again.
They are development dependencies, so the runtime image stays as it is.
2026-10-12 12:27:46 +02:00
mesh-admin aee834aeb6 Merge pull request 'Wait for a stopped bundle's exit instead of sleeping after it (issue 476)' (#30) from fix/476-wait-for-the-exit into main 2026-10-12 09:03:48 +00:00
jschoubben 494d0787e2 Wait for a stopped bundle's exit instead of sleeping after it
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed
Launched.Stop now returns once the bundle's exit is handled, bounded by
StopWait, and the launcher reads its restart pauses once at Start, so
the exit goroutine no longer races a test that changes them. The caller
tests drop their half-second pause (novox/hq issue 476).
2026-10-12 10:43:22 +02:00
mesh-admin 172b5af589 Merge pull request 'node-tools: serve mesh/state.history (ADR 0307, issue 502)' (#29) from adr-0307-state-history into main 2026-10-12 08:24:20 +00:00
jochen 3f0c1a6cec Serve a key's kept values so the issue tracker can carry them into its log (ADR 0307, issue 502)
mesh/delivery-group group adr-0307-state-history delivered: every member is delivered
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/state.history answers every value a bucket still keeps for a key,
oldest first, to whoever may get it, by the same watch the runtime already
uses, so no new grant is needed.
2026-10-12 10:18:34 +02:00
mesh-admin 4348b42844 Merge pull request 'node-tools: serve a module's log verbs (ADR 0297, issues 501, 502)' (#28) from adr-0297-log-verbs into main 2026-10-12 00:28:27 +00:00
jochen 82aefe1988 Refuse a credential in a log entry and take a full 256 KiB value with its expect header (ADR 0297 review)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
A log keeps what it is given for good, so the state guard against secrets
applies to it too; the stream holds 260 KiB a message so the headers an
append carries never push a 256 KiB value over.
2026-10-11 19:20:31 +02:00
jochen d535dd5322 Serve a module's log verbs so its record outlives a bucket's cap and the events' week (ADR 0297, issues 501, 502)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
The runtime answers mesh/log.append, read, last and info for a log issued
in the module's membership, by publish, batch direct get and stream info
alone. A stale expect is refused as a lost state update, so a module tells
it by the same code.
2026-10-11 19:18:46 +02:00
mesh-admin 336a2b1328 Merge pull request 'node-tools: a tool call names its caller, and the module is told it from the subject (hq issue 365)' (#27) from fix/365-a-tool-call-names-its-caller into main 2026-10-11 09:31:41 +00:00
jschoubben 541de48e04 node-tools: only a call pattern names a caller, and a fall-back for want of a responder is said (hq issue 365, review)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
callerOn reads a caller only under the call kind, so a tool subscription ending
in a wildcard never yields one. A caller-named call that reached nobody is said
once per subject before it is asked on the old subject, so a runtime or grant
that drifted behind is seen rather than a quiet unproven. The caller tests wait
for their bundle's exit, which reads the launcher's restart pauses another test
sets.
2026-10-11 11:22:29 +02:00
jschoubben ab0f888d67 node-tools: a tool call names its caller, and the module is told it from the subject (hq issue 365)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery-group group fix/365-a-tool-call-names-its-caller delivering: 0 of 4 delivered
mesh/delivery superseded: a newer head of the same pull request
Each tool and seat verb is also served on its caller-named subject, kind call
with the caller's bus user last, which the controller grants each user with its
own name alone; the runtime hands the module that caller in the call's _meta,
never from the body. Calls through the runtime (the console and modules'
invokes) ask the caller-named subject first and the old one only when the first
reached nobody: nothing serves it yet, or the bus refused it because the grant
is not delivered yet, which ends the call at once rather than after 30 s. The
old subjects stay served and say no caller for one release (hq issue 464).
2026-10-11 05:19:08 +02:00
mesh-admin 6a6ed725c0 Merge pull request 'Keep an event alive while its bundle handles it, and hold one at a time (hq issue 387)' (#26) from fix/387-an-event-is-kept-alive-while-it-is-handled into main 2026-10-10 15:26:49 +00:00
jochen 560876c074 No in-progress after an event is answered (review, hq issue 387)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-10 17:21:54 +02:00
jochen aef908b57b Keep an event alive while its bundle handles it, and hold one at a time (hq issue 387)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The consumer's ack wait is 30 seconds and the runtime gives a handler two minutes,
so an event handled longer was offered again while it was handled; events waiting
in the client's buffer ran out their ack wait too, so a burst reached max-deliver
with nothing failed. Say in-progress every 10 seconds, as a worker's work does,
and pull one event at a time.
2026-10-10 17:18:35 +02:00
mesh-admin c761c22c2a Merge pull request 'Prove a write by compare-and-set loses no update under contention, on a real bus (hq research 042)' (#25) from feat/a-counter-by-compare-and-set into main 2026-10-10 12:37:42 +00:00
jochen d0128adf5b Fail the contention test when the two writers never collided: it would prove nothing (the review of 2026-10-10)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
mesh/delivery-group group feat/a-counter-by-compare-and-set delivered: every member is delivered
2026-10-10 12:33:43 +02:00
jochen 1eaa1abbe9 Prove a write by compare-and-set loses no update under contention, on a real bus (hq research 042)
mesh/delivery-group group feat/a-counter-by-compare-and-set checking: 0 of 2 member(s) ready
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery superseded: a newer head of the same pull request
The issues module will hand out numbers through mesh/state.create and update. Two writers incrementing one
counter a thousand times each through the module's bus now end at two thousand with every number handed out
once; a stale, zero or future revision is refused with the lost-write code and writes nothing; a deleted key
can be created again; and the lost-write code reaches a bundle wrapped or not, and no other refusal carries it.
Both runtime tests run in the check's bundleless list, so mesh/repo-check runs them.

Part of the issue tracker decision, hq research 042.
2026-10-10 12:31:06 +02:00
mesh-admin 83c1994183 Merge pull request 'A bundle's seat traffic under its own name and kind, compare-and-set state, and a module's own account (hq ADR 0259)' (#23) from feat/asks-answered-on-any-channel into main 2026-10-09 14:17:37 +00:00
jochen 127047edd6 Serve a module from a runtime on its own account instead of switching users, keep bus words from bundles, and never give up a channel's work (hq ADR 0259 revision)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
mesh/delivery-group group feat/asks-answered-on-any-channel stopped: a member was stopped
2026-10-09 10:12:39 +02:00
jochen c6087129ce Tell a bundle the kinds its membership names 2026-10-09 10:12:39 +02:00
jochen 7b4428c510 Name the worker a piece of work came from, so a bundle taking two is handed each to its own handler 2026-10-09 10:12:39 +02:00
jochen 6d9906b6c9 Carry a bundle's seat traffic under its own name and kind, and start a module as its own account (hq ADR 0259) 2026-10-09 10:12:39 +02:00
mesh-admin 66de6774e0 Merge pull request 'Say what becomes of an event a handler gives up on (hq issue 330, ADR 0264)' (#24) from fix/330-a-message-given-up-on-is-kept into main 2026-10-08 19:24:27 +00:00
jochen c401bbff2d Say what becomes of an event a handler gives up on (hq issue 330)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/330-a-message-given-up-on-is-kept delivered: every member is delivered
The comments said a failing event dead-letters, which nothing did; the
controller now keeps it, and delivers it again under a subject this runtime
reads the same key from (ADR 0264).
2026-10-08 17:36:04 +02:00
mesh-admin d4845f7c8d Merge pull request 'Join a paged answer, and say one too large instead of timing out (hq issue 314)' (#22) from fix/314-a-large-answer-is-paged-not-lost into main 2026-10-08 11:06:23 +00:00
jochen 90c7871276 Join a paged answer, and say one too large instead of timing out (hq issue 314)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/314-a-large-answer-is-paged-not-lost delivered: every member is delivered
An answer larger than the bus's max_payload was refused by the client library
inside its holder, and the caller waited out its 30 s and read silence: the
controller's conditions and status for hours on 2026-10-08. The console now asks
for the parts of an answer the controller paged and joins them; a module's answer
too large for one message is answered as an error saying so, at once.
2026-10-08 11:25:50 +02:00
mesh-admin 9a4140b49b Merge pull request 'Find a tool by the shell command it replaces, best first (hq ADR 0245)' (#21) from feat/search-finds-what-a-verb-replaces into main 2026-10-07 19:04:13 +00:00
jochen aa209313c2 Rank a search the same whatever order discovery heard things in
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
A tie went to whichever tool discovery listed first, so docker logs found docker_secrets_in_logs
before docker_logs on some runs. A name the query says whole wins, then the shorter name.
2026-10-07 20:27:26 +02:00
jochen 49cb06f6c5 Find a tool by the shell command it replaces, best first (hq ADR 0245)
The agent searched with journalctl, logs and docker ps and found nothing, so it went over ssh. Search
now reads what each verb and tool replaces from the controller's records, matches a command line
against it, ranks the matches, and says the command it matched.
2026-10-07 19:21:13 +02:00
mesh-admin 26a58c7a79 Merge pull request 'Ask a seat once more when the controller refused as handing over (hq issue 289)' (#20) from fix/a-verb-survives-the-handover into main 2026-10-07 00:40:21 +00:00
mesh-admin 7e83940fcc Merge pull request 'Console: a seat and a module of one name are each reached (hq issue 287)' (#19) from fix/a-seat-and-a-module-of-one-name into main 2026-10-07 00:27:49 +00:00
jochen 2f045e042f Ask a seat once more when the controller refused as handing over (hq issue 289)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/a-verb-survives-the-handover delivered: every member is delivered
A controller being replaced refuses a call it cannot serve, marked retry:
handing-over; the one after it answers the same call.
2026-10-07 02:13:17 +02:00
jochen 41911f7f15 Check again, judged by the controller with the gate's fix (novox/hq issue 285)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-07 02:06:52 +02:00
jochen 0a5dbc7e9a Cite the hq issues by the numbers they were given: 285, 286, 287
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (0 of …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-07 02:00:11 +02:00
jochen dac4d423a8 Reach a seat and a module of one name each: list the seat's verbs, and route a machine's address to the module
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (0 of …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
mesh-delivery is the delivery's seat and the module holding it, which answers the seat's verbs with
tools of the same names. Discovery keyed the seat's verbs and the module's tools in one namespace, so
the module's tool took the key and the seat was listed with no verb; and every address with the name
resolved to the seat, so neither the seat's verbs nor the module's tools could be called. The keys are
apart, and `<node>/<module>.<tool>` reaches the module when the module serves it (novox/hq issue 284).
2026-10-07 01:37:46 +02:00
mesh-admin 01c98db0a4 Merge pull request 'A merge check of its own; every test on a bus of its own; git in the TypeScript toolchain (hq ADR 0238)' (#18) from feat/a-merge-check-of-its-own into main
mesh/delivery delivered
2026-10-06 20:55:58 +00:00
jochen b86a6ca228 Give the TypeScript toolchain git, for the merge checks that run in it (hq ADR 0237)
A repository's own merge-check.sh may declare this toolchain, and a suite that reads a
repository's history failed with spawnSync git ENOENT. Nothing compiled here calls it.
2026-10-06 22:06:16 +02:00
jochen 14bff5dfd7 Check the tool runtime's own code before it merges, every test on a bus of its own (hq ADR 0237)
A merge-check.sh, the repository's layer of the mesh's merge check (mesh/repo-check):
format, vet, and node-tools' suite under the race detector. The tests shared one bus and
had to run one package at a time; like the controller's (#97), each now starts a server
of its own at the release go.mod pins, held to the catalogue's bus image by a test. What
cannot run in the check — bundles that need @novox/mesh-sdk — is said as not tested.
2026-10-06 22:04:46 +02:00
mesh-admin c2a0683115 Merge pull request 'node-tools: say whose words a refused event is (hq issue 276)' (#17) from fix/an-event-handler-answers-what-it-did into main
mesh/delivery delivered
2026-10-06 16:17:05 +00:00
jochen 1792b0d9ba node-tools: say whose words a refused event is (hq issue 276)
"plex did not take radarr.download.completed: Unexpected end of JSON input"
read as the runtime failing to parse the bundle's answer. It was plex's own
handler error, relayed. A bundle's error answer is now a launch.Refused, and
the line says the handler answered an error, quotes it, names the event id
and the delay before the next offer; the runtime's own failures (no answer
in time, bundle exited) are said as before.

The rule, unchanged and now tested: any mesh/event answer that is not an
error takes the event, whatever its result says, including none.
2026-10-06 18:14:50 +02:00
mesh-admin f75780f1d0 Merge pull request 'Say the node tools are there, every minute (hq to-be 45 Phase 1, S11)' (#16) from feat/a-core-that-cannot-fail-silently-phase-1 into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-06 08:34:15 +00:00
jochen 0307df0850 Say the node tools are there, every minute (hq to-be 45 Phase 1, S11)
A machine whose node-engine is heard and whose runtime is gone is a
machine nobody can ask anything, and nothing said so. The runtime now
says on mesh.control.<node>.tools-alive, every minute, that it is there,
with its interval and build; the controller raises tools-silent after
three missed. Core NATS, like the host's heartbeat: a lost one is the
next one. A heartbeat the bus refuses is logged when that starts and
when it stops.
2026-10-06 10:06:29 +02:00
mesh-admin 730b4047f0 Merge pull request 'Say a timeout is not a failure, and where the controller keeps the answer (hq issue 265)' (#15) from fix/a-timeout-is-not-a-failure into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 23:33:12 +00:00
jochen fce3dcb98d Say a timeout is not a failure, and where the controller keeps the answer (hq issue 265) 2026-10-06 01:14:59 +02:00
mesh-admin 9730bd89c3 Merge pull request 'Console: keep a mesh seat's node argument, and refuse what a call would drop (hq issue 244)' (#14) from fix/console-keeps-a-mesh-seats-node into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 22:46:47 +00:00
jochen 3a7c9645e0 Keep a mesh seat's node argument, and refuse what a call would drop
The console took node out of every schema and every call, so the
controller's push, plan, assign, pin and settings were described without
the machine and called without it: a push naming one machine ran as a
push of every machine behind (hq issue 244). node is now taken out only
where the address names the machine, a different machine there is
refused, and an argument a seat's verb does not declare is refused.
2026-10-06 00:37:15 +02:00
mesh-admin 93c1ad8c4a Merge pull request 'Console: wait for every runtime that answered, name the ones it missed, add mesh_runtimes' (#13) from fix/console-sees-every-runtime into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-05 13:38:48 +00:00
jochen 52a0cc4fcb Forget a runtime's old instance once it answers under a new one, so a restart is not reported as a missed answer 2026-10-05 15:38:30 +02:00
jochen 379d19d907 Wait for every runtime discovery hears from, and name the ones it misses
The console gathered $SRV.INFO answers for a fixed 750 ms. The laptop's
runtime (48 modules, 341 endpoints, 164 kB) answers last every time: its
answer crosses to the broker on another machine and back, a median of
365 ms on a quiet link and 813 ms in one of 25 rounds measured. When it
missed the window the console said its modules ran nowhere ("nothing in
the mesh is called slack") or only on another machine.

Discovery now asks $SRV.PING alongside $SRV.INFO and waits, past the
window and up to 5 s, for every instance that answered PING. A runtime
that never sends what it serves, or answered before and not now, is
named in every answer that might concern it instead of the module being
called missing. An answer still too large after first-line descriptions
drops them, and says so in its metadata.

mesh_runtimes reports, per runtime, its machine, how long its answer
took, its size, its modules and tools, whether it was shortened and when
it was last heard, and the runtimes and machines not heard.
2026-10-05 15:36:10 +02:00
mesh-admin 8b789578c1 Merge pull request 'Keep a machine's discovery answer under the bus's message limit' (#52) from fix/discovery-fits-in-a-message into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 15:22:56 +00:00
jochen b3ebdd5edd Keep a machine's discovery answer under the bus's message limit
Every tool's schema in one $SRV.INFO reply outgrew max_payload on machines
serving 137-206 tools, and the refused reply was dropped silently, so search
and a machine's view went empty. Module tools now announce without schema;
describe and tools/list ask the module for it. An answer still too large has
its descriptions cut to a line, and a failed reply is logged.
2026-10-04 17:06:20 +02:00
mesh-admin 51c79d8461 Merge pull request 'The console says an account was refused only when the bus refused it' (#51) from fix/console-says-a-refusal-only-for-a-refusal into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 10:01:18 +00:00
jochen 8405e32efc The console says an account was refused only when the bus refused it
"authorization" alone matched a tool's own answer mentioning the word — the
runtime refusing a state value with an Authorization header read as
"this account may not call".
2026-10-04 11:31:42 +02:00
mesh-admin 670a7884ff Merge pull request 'Module state is hq ADR 0201 after all' (#50) from fix/module-state-is-0201 into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 09:03:18 +00:00
jochen 779ea67ea6 Module state is hq ADR 0201 after all: the derived-value record moved to 0202 on hq main 2026-10-04 11:02:43 +02:00
mesh-admin b149e9fcd6 Merge pull request 'node-tools provides its endpoint at node scope (hq to-be 40 WP1)' (#34) from feat/claude-code-agent into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 09:01:26 +00:00
mesh-admin 168cf02829 Merge pull request 'Module state is hq ADR 0202 (0201 landed first for a provider's derivations)' (#49) from fix/adr-0202-module-state into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 09:01:23 +00:00
jochen 70ff0f84af Module state is hq ADR 0202: 0201 landed first for a provider's derivations 2026-10-04 03:44:51 +02:00
mesh-admin 328550f920 Merge pull request 'The runtime serves a bundle its module's state (hq ADR 0201)' (#48) from feat/module-state-on-the-bus into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-04 01:43:37 +00:00
jochen 29c24a2367 The TypeScript SDK's state reaches the runtime, a narrowed watch only its keys (novox/hq ADR 0201) 2026-10-04 02:48:10 +02:00
jochen 1adfcad88e Name the watch a state change is for (novox/hq ADR 0201) 2026-10-04 02:45:31 +02:00
jochen dac8812968 gofmt 2026-10-04 02:45:10 +02:00
jochen 6cba894f01 The runtime serves a bundle its module's state (novox/hq ADR 0201)
mesh/state.get, put, delete, keys and watch on the stdio channel, from the
buckets the membership issues. A watch hands the current values without
deletions, then every change, and is answered once the current values are
delivered. Refused with the reason: state not issued, a reader's write, a
value with a credential-named field — the bus alone would answer a refused
write with a timeout.
2026-10-04 02:45:07 +02:00
mesh-admin 47cdfad754 Merge pull request 'Ask the bus again for a subscription it refused (hq issue 222)' (#47) from fix/issue-222-a-refused-subscription-is-asked-again into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 23:20:49 +00:00
jochen 85810ccc8c Ask the bus again for a subscription it refused (novox/hq issue 222)
A push sends the machine that needs a grant and the bus's machine in the same breath, and the
runtime can subscribe the moment before the bus reloads its user list. Refused once, the
subscription stayed dead until some later membership re-served it, and a newly assigned module ran
unreachable. A refused subject the runtime answers is now asked for again for about five minutes.
2026-10-04 01:20:44 +02:00
mesh-admin 2ba7451229 Merge pull request 'A refused tool subscription is said, never fatal (hq issue 218)' (#46) from fix/a-refused-seat-subscription-is-not-fatal into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 22:13:22 +00:00
jochen 6604d44372 A refused tool subscription is said, never fatal (novox/hq issue 218)
After the controller stopped granting a mesh seat to claimants that do not hold it, an image built
before the runtime followed its membership still subscribed the seat's subject, and the refusal
ended the process: ace's postgres runtime crash-looped. A subject the grants leave out now costs
that subject only, as an announcement's already did (issue 217).
2026-10-04 00:13:15 +02:00
mesh-admin 7b21440962 Merge pull request 'Serve a seat's verbs from the membership once one is issued (hq issue 218)' (#45) from fix/issue-218-the-membership-decides-the-seats into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 21:49:27 +00:00
jochen 0cea8d286e Serve a seat's verbs from the membership once one is issued (novox/hq issue 218)
The runtime added every seat its start-up credential claims even after the mesh issued a
membership without it. A seat held once for the mesh is claimed on every machine running the
module, so ace's postgres announced the store seat the bus then refused it on.
2026-10-03 23:48:15 +02:00
mesh-admin 094a7d0d7c Merge pull request 'The toolchain carries the SDK the mesh last published, and esbuild (hq issue 212, ADR 0193)' (#44) from feat/the-toolchain-follows-the-sdk-and-bundles into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 21:31:59 +00:00
jochen b52669577d The toolchain carries the SDK the mesh last published, and esbuild (hq issue 212, ADR 0193)
mesh-tools stands on mesh-sdk's published package: the build receives its exact version and
installs it after the package.json install, so a release is a new argument and the cached layer
cannot keep an older SDK; the planner orders the toolchain after the SDK, and every bundle after the
toolchain (issue 211). esbuild, a development dependency, is what the builder bundles each
TypeScript entrypoint and launcher into one file with.
2026-10-03 23:26:09 +02:00
mesh-admin 7bd76275f9 Merge pull request 'A refused announcement is said, never fatal (hq issue 217)' (#43) from fix/a-refused-announcement-is-not-fatal into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 21:24:50 +00:00
jochen 6ba0f4dc1e A refused announcement is said, never fatal (hq issue 217)
The raw subscription that answers discovery ran its loop unguarded, so a refusal escaped as an
unhandled rejection and ended the process: every per-module container crash-looped on 2026-10-03
over a subscription that only serves the mesh seeing the runtime. It is now caught, logged, and the
runtime serves on. Tested against a bus whose permissions refuse the subject; fails without it.
2026-10-03 23:24:38 +02:00
mesh-admin bc05658772 Merge pull request 'A mesh seat's holder answers for the machine it runs on (hq ADR 0197)' (#42) from fix/a-mesh-seats-holder-answers-for-its-machine into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 21:23:06 +00:00
jochen 4af59636c3 A mesh seat's holder answers for the machine it runs on (hq ADR 0197)
The controller announces the mesh-controller seat without a machine — the seat is the mesh's — and
the console then reported it as not answering on the machine it is assigned to. An announcement that
names no machine now answers for wherever its module is assigned.
2026-10-03 23:22:56 +02:00
mesh-admin 6e425a000f Merge pull request 'The node's runtime is its modules' bus: it binds each module's consumer and hands its events to the bundle (hq ADR 0198)' (#41) from feat/0198-the-runtime-is-the-bus into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 21:18:33 +00:00
jochen 14b6588839 Merge remote-tracking branch 'origin/main' into feat/0198-the-runtime-is-the-bus 2026-10-03 23:16:43 +02:00
mesh-admin 490aedfd36 Merge pull request 'Announce only on the subjects the grants allow (hq ADR 0197, issue 217)' (#40) from fix/announce-only-what-the-grants-allow into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 21:11:24 +00:00
jochen f11ac6441c Announce only on the subjects the grants allow (hq ADR 0197)
Both runtimes subscribed $SRV.<verb>.> as a wildcard; the grants allow the bare question and the
service's own name and instance. The bus refused the wildcard, and the TypeScript runtime treats a
refused subscription as fatal, so every per-module container crash-looped after the image rolled.
They now subscribe exactly $SRV.<verb>, $SRV.<verb>.<name> and $SRV.<verb>.<name>.<id>.
2026-10-03 22:31:02 +02:00
jochen ffe229308c The node's runtime is its modules' bus: it binds each module's consumer and hands its events to the bundle (hq ADR 0198)
A launched bundle's mesh/subscribe binds the module's own durable consumer — EVENTS, <node>_<module>,
by name as the module's own runtime bound it, so nothing is lost or replayed in the move — and every
event goes to each child of the module that subscribed as mesh/event, acknowledged only when all
answered, negatively acknowledged after a short delay when one failed or died, terminated when it is
not an event. mesh/ask calls a tool as the module. Every launched bundle is started again when it
exits, with backoff, since long-running code waits for no call. Requires SDK 0.1.6.
2026-10-03 22:29:20 +02:00
mesh-admin df4f492a72 Merge pull request 'The mesh's tools are found by address, from what announces itself on the bus (hq ADR 0195, 0197)' (#39) from feat/0197-tools-announce-themselves into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 20:19:53 +00:00
jochen 66e8be0e31 The TypeScript runtime announces what it serves too, in the same services format (hq ADR 0197)
The per-module containers still run this runtime; their tools and the seats they hold (the store's,
the catalogue's) must be found by the console the same way as the node runtime's. It answers
$SRV.PING, $SRV.INFO and $SRV.STATS with one service per process, one endpoint per tool per subject
and per seat verb served, the metadata as the Go runtime writes it.
2026-10-03 22:15:48 +02:00
jochen 7722668220 Every runtime announces what it serves in the NATS services protocol; the console discovers by asking the bus (hq ADR 0197)
The Go runtime answers $SRV.PING, $SRV.INFO and $SRV.STATS (and per name and id) in the
io.nats.micro.v1 format with what it serves at the moment it is asked: one service per runtime
process, since the bus admits one reply per request from each responder, and one endpoint per tool
per subject, its metadata saying module, seat, scope, machine, description, schema and whether the
module is interchangeable. Serving is unchanged.

The console gathers one $SRV.INFO request's answers instead of asking the catalogue's roster and
each module's tools, and reads the controller's records as JSON for what should have answered: an
assignment with tools that did not announce is named, a module without tools never is. The text
parsers of node list and module list are gone. Packages share the test bus: go test -p 1.
2026-10-03 22:14:16 +02:00
jochen e8989f3cf5 Merge remote-tracking branch 'origin/main' into feat/0197-tools-announce-themselves 2026-10-03 22:08:06 +02:00
mesh-admin 915f372a85 Merge pull request 'node-tools in Go: the node's runtime, launch-only, wire-compatible with the TypeScript (hq ADR 0193)' (#38) from feat/0193-node-tools-in-go into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 20:02:15 +00:00
jochen 486dad99a5 Merge remote-tracking branch 'origin/main' into feat/0193-node-tools-in-go 2026-10-03 22:02:03 +02:00
mesh-admin 65f3b68076 Merge pull request 'node-tools launches every bundle it serves; a child's emit is published as its module (hq ADR 0193)' (#36) from feat/0193-the-runtime-launches-every-bundle into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 20:01:58 +00:00
jochen a269a76c87 The console announces five tools and reaches everything by address (hq ADR 0195)
mesh_overview, mesh_machine, mesh_search, mesh_describe and mesh_call walk the mesh's structure;
every tool has one address per layer: <seat>.<verb>, <node>/<seat>.<verb>, <node>/<module>.<tool>,
and <module>.<tool> for a module the mesh issued a plain subject. A stateful module called without
its machine, a node seat without one, a mesh seat with one, or a module on the wrong machine is
refused naming what would work. Answers come from the mesh when asked, kept five seconds, so a tool
that arrives mid-session is found. The flat catalogue stays behind MESH_CONSOLE_FLAT=1 and old
<module>.<tool> names still answer.
2026-10-03 21:54:19 +02:00
jochen e6a33dd969 node-tools is built from Go (hq ADR 0193)
The node's runtime is the Go binary: one static executable the host runs as ./node-tools from its own
bundle. Node.js stays on the machine for the TypeScript bundles the runtime launches. The TypeScript
source stays in the repository: it is the runtime image the per-module containers run until WP4c.
2026-10-03 21:26:06 +02:00
mesh-admin 42e0987c64 Merge pull request 'Require SDK 0.1.5: a launched bundle names its module and emits through the runtime (hq ADR 0193)' (#37) from fix/the-toolchain-carries-sdk-0.1.5 into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 19:23:59 +00:00
jochen ae6bdc9b06 Require SDK 0.1.5: a launched bundle names its module and emits through the runtime (hq ADR 0193)
The toolchain image installs from this package.json with a range; unchanged, Docker reused the
cached install and the image kept SDK 0.1.3 after 0.1.5 was published. Bundles copied that copy, so
a launched bundle registering its seat first served the seat's verbs as its own tools. Requiring
0.1.5 says what the runtime and its bundles need, and invalidates the cached layer.
2026-10-03 21:23:50 +02:00
jochen e45389f75a node-tools in Go: the node's runtime, launch-only, wire-compatible with the TypeScript (hq ADR 0193)
The runtime knows no language, so nothing ties it to Node.js. This ports its serve mode — the
pinned bus connection and patient connect, following memberships, launching every served bundle
over MCP on stdio with its own environment, a child's emit published as its module, each tool,
the tools verb and seat verbs served where the mesh issued them, and the console on loopback —
to one static binary. Same subjects, request and reply bodies, event headers and MCP answers.
The TypeScript stays: it is still the runtime inside the per-module containers until WP4c.
Tests run against a real bus and share the TypeScript fixtures.
2026-10-03 21:21:52 +02:00
jochen b182943c24 node-tools launches every bundle it serves; a child's emit is published as its module (hq ADR 0193)
Every served entrypoint is started as a process speaking MCP over stdio, told its module and node;
one that is not executable is refused by name. The import path, the SDK resolve hook (issue 209)
and the per-registration hand-off go. The one-module form the per-module containers use is still
imported until they move (to-be 38 WP4c). A child's mesh/publish is published as its module and
answered once accepted; a child that dies says why in its own last words. Fixtures are served
through launchers exactly as the builder writes them.
2026-10-03 21:12:15 +02:00
mesh-admin 26e7fcb4cb Merge pull request 'A launched bundle is told the module it serves (hq ADR 0193)' (#35) from feat/0193-a-launched-bundle-is-told-its-module into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 19:08:06 +00:00
jochen 302645dd01 A launched bundle is told the module it serves (hq ADR 0193)
node-tools sets MESH_SERVED_MODULE for each child it launches, so the SDK's stdio loop lists that
module's tools by their names and a seat's verbs as the seat's, whichever was registered first.
2026-10-03 21:07:52 +02:00
jochen de4b63824e node-tools provides its endpoint at node scope, so a consumer on the machine is told the console's port rather than writing it (hq to-be 40 WP1) 2026-10-03 16:02:08 +02:00
mesh-admin 8e30ea9ff5 Merge pull request 'node-tools hands each bundle its own environment (hq ADR 0192)' (#33) from feat/0192-each-bundle-its-own-env into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 13:34:05 +00:00
jochen 193ed0ac63 node-tools hands each bundle its own environment (hq ADR 0192)
The mesh composes every served module's words into MESH_TOOL_ENV; the runtime takes it at start
and removes it from its own environment, then gives each registration's contributor and each
launched child the runtime's words plus its own module's, never another's. Against an SDK
without collectToolsEach it says so and serves with the runtime's words only. The test serves
two imported bundles and one launched, each answering with its own words and none of the others'.
2026-10-03 15:28:49 +02:00
mesh-admin 4537494150 Merge pull request 'One SDK per runtime: a bundle's SDK import resolves to the runtime's copy (hq issue 209)' (#32) from fix/issue-209-one-sdk-per-runtime into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-03 11:07:08 +00:00
jochen 7152148410 One SDK per runtime: a bundle's SDK import resolves to the runtime's copy (hq issue 209)
A bundle carries its dependencies, the SDK among them; imported in-process that copy was a
second SDK with its own tool registry, so a bundle registered its tools into a list the
runtime never read and served nothing, silently. A resolve hook (module.registerHooks, in
thread; module.register is deprecated from Node 26) now sends every import of
@novox/mesh-sdk, from whichever bundle, to the runtime's own copy: one registry, one broker.
The test loads a bundle from a directory holding its own SDK copy and sees its tool served.
2026-10-03 13:00:19 +02:00
mesh-admin 5d488a45f7 Merge pull request 'node-tools is a module beside mesh-tools: the runtime as a bundle, and serve is the console (hq ADR 0175, to-be 38 WP3)' (#31) from feat/wp3-node-tools into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 19:57:24 +00:00
jochen c46f9502ee node-tools is a module beside mesh-tools: the runtime as a bundle, and serve is the console (hq ADR 0175, to-be 38 WP3)
One repository, two modules (ADR 0069). `node-tools/` holds the runtime — its code, tests, package
and the manifest of the module the controller composes a process for on every machine it is
assigned to: a bundle of `src/main.js`, the interpreter as a package, a place for the node's
credential, the loopback port the console declared, and leave to call every tool. Nothing about
how it runs: which bundles to load, where the credential is and whose machine it is are the
controller's to compose (WP2). The root module `mesh-tools` keeps the two images TypeScript
bundles are compiled in and a module's own service may run in; it is no longer how tools reach a
node.

As node-tools, `serve` is also the console (ADR 0175 §6): the same process answers MCP on
loopback for whoever is on the machine, through which the tools it serves can be called. A
module's own runtime in a container keeps serving without a listener.

The toolchain image now carries /app/runtime — a package.json saying the compiled files are ES
modules and the production node_modules — for the builder to copy into every TypeScript bundle,
so a bundle unpacked on a machine starts (ADR 0188 §5; the builder's side is the controller's).
Proven here by compiling node-tools with the toolchain's exact flags and starting the result.
The AMQP probe script is gone with the bus it probed.
2026-10-02 21:43:37 +02:00
mesh-admin 2746fd31f0 Merge pull request 'Cite ADR 0188, not 0187: the record was renumbered before it merged' (#30) from fix/cite-adr-0188 into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 19:29:30 +00:00
jochen 82d7306ee0 Cite ADR 0188, not 0187: the record was renumbered before it merged (0187 is the dead-tracker record) 2026-10-02 21:29:04 +02:00
mesh-admin 2818f99b17 Merge pull request 'The runtime serves a list of modules on one credential, naming a bundle that fails to load (hq ADR 0175, to-be 38 WP1)' (#29) from feat/the-operators-machine into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 19:27:49 +00:00
jochen 1436b02755 A bundle that is not JavaScript is launched and spoken to over MCP on stdio (hq ADR 0187, to-be 38 WP1b)
The runtime imported a bundle into its own process, which only JavaScript can be. Now an entrypoint
that is not a plain JavaScript file — or is one marked executable — is started as a child with the
runtime's environment and asked `tools/list` once and `tools/call` per call; what it lists is
registered exactly as an imported bundle's registrations are, a `<seat>.<verb>` name as the seat's
implementation. So a tools bundle may be in any language, and the mesh's part — the subjects, the
seats, the `tools` answer, a failed bundle named — stays in the runtime and is shared by all of
them. A child that exits mid-call tells the caller so and is started again on its next call.

Proven against a real bus beside the three bundles already there: a Python bundle with no SDK at
all answers its tool and its seat verb; a TypeScript bundle written against the protocol and marked
executable is served through the launcher, shortcut off; a bundle told to exit is relaunched.
2026-10-02 21:24:20 +02:00
mesh-admin 4e0559c31a Merge pull request 'Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered' (#28) from fix/adr-0170-cited into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 16:44:07 +00:00
jochen 6390d1d7fb The runtime serves a list of modules on one credential, naming a bundle that fails to load (hq ADR 0175, to-be 38 WP1)
One tool runtime per node, host-side, is what the runtime was written to be; the catalogue built a
container per module around it instead. This lets `serve` take a list — MESH_TOOL_MODULES as
<module>=<entrypoint> entries — and do for every assigned module what it did for one: read that
module's membership and follow it, serve its tools where the membership says, serve each held
seat's verbs on the seat's subjects. The seats come from the memberships now, so the node's
credential carries no claims; a module's own runtime still reads its credential's, so nothing
built today changes behaviour. A bare path in MESH_TOOL_MODULES stays the one-module form.

A bundle that throws on import is said in the log and in what `tools` answers for its module
(`failed`), which discovery lists with the reason instead of as "not answering"; the other bundles
serve. The filter that dropped every registration under a name but the one module goes; what
stays is that a registration under a seat's name is served only where some served module claims
the seat. A tool runs attributed to its module, so an event it emits lands on the module's
subject and not the runtime's. MESH_OPERATOR_ACCOUNT and MESH_OPERATOR_HOME are read and said;
tools take them from their environment.

Proven against a real bus: three bundles, one broken; five tools and two seat verbs answer on
their subjects; `tools` names the failed bundle; a membership re-issued mid-run re-serves.
2026-10-02 18:22:30 +02:00
jschoubben 020003ea3f Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered after a collision on hq main 2026-10-02 14:52:28 +02:00
mesh-admin 809f07d085 Merge pull request 'A node-scoped seat's verb is callable through the console, naming its machine (hq ADR 0169)' (#27) from fix/a-node-scoped-seats-verb-is-callable-through-the-console into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 12:24:02 +00:00
jschoubben fb0dcd0cc1 A node-scoped seat's verb is callable through the console, naming its machine (hq ADR 0169)
The listing left node-scoped seats out, so <seat>.<verb> never resolved as a
seat's and the call went to a module subject nothing served. Listed now with
their scope: the schema requires the machine, the call carries it in the
subject, and a call without one is refused in words.
2026-10-02 14:23:26 +02:00
mesh-admin a3ace58362 Merge pull request 'A registration under a stranger's name is said and skipped, not fatal' (#26) from fix/a-registration-under-a-strangers-name-is-refused-not-fatal into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 14:47:31 +00:00
jschoubben feb13c1de6 A registration under a stranger's name is said and skipped, not fatal
A module implementing a seat its credential does not (yet) claim registered tools under the seat's
name; the runtime treated them as the module's own, refused to serve another module's key, and the
whole runtime restarted in a loop — postgres on the control node, 2026-10-01, whose credential predated
the claims. Such a registration is now named in the log and left out; the module's own tools serve.
2026-10-01 16:47:03 +02:00
mesh-admin e6676068c7 Merge pull request 'The membership is read by its subject' (#25) from fix/the-membership-is-read-by-its-subject into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 14:22:26 +00:00
jschoubben a01b5f5b78 The membership is read by its subject
The runtime asked the assignments stream's root for the last message by subject in the body, and the
mesh grants a module's account only the subject-addressed form of the direct get — the server refused
every read (Publish Violation on $JS.API.DIRECT.GET.ASSIGNMENTS for every module on the new runtime),
so every runtime kept the derived shape. The address is now the stream then the subject, nothing in the
body, which is the one address the account has.
2026-10-01 16:16:57 +02:00
mesh-admin ab8b13f512 Merge pull request 'A seat's verb keeps a node of its own; only a module's tool gives it to the subject' (#24) from fix/a-seat-verbs-node-is-its-own into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 13:44:22 +00:00
jschoubben ce8c37a7fc The listing test tells the seat's two verbs apart: one keeps its own node, the other takes none 2026-10-01 15:43:36 +02:00
jschoubben 809e2b11ec The listing test indexes the module's tool after the seat's two verbs 2026-10-01 15:42:58 +02:00
jschoubben 670b486ac0 A seat's verb keeps a node of its own; only a module's tool gives it to the subject
The console moved every call's node into the subject (ADR 0159), so mesh-controller.push {node: x}
became a call to the seat's verb on machine x, which nothing serves — the mesh's own verbs could not
be given a machine from the console at all. A role's verb takes no machine from the console; its
arguments are its own.
2026-10-01 15:42:35 +02:00
mesh-admin 56f82588af Merge pull request 'A runtime serves what the mesh issued it, and a seat's verbs are implemented under the seat's name (hq ADR 0160)' (#23) from feat/a-runtime-serves-what-it-is-issued into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 13:23:55 +00:00
jschoubben e7b98f1fbc A runtime serves what the mesh issued it, and a seat's verbs are implemented under the seat's name (hq ADR 0160)
The one address a runtime derives for itself is mesh.assignment.<node>.<module>. It reads the
membership there with a direct get on the ASSIGNMENTS stream, serves each tool exactly where the
membership says — the plain subject in the module's queue when the mesh issued one, this machine's
beside it — and follows the subject, re-serving when a new membership arrives. A mesh that has issued
nothing yet gets the shape it always derived, and the log says so.

A seat's verbs are the role's, not the software's (ADR 0159): a module implements them with
registerModuleTools("<seat>", …), the runtime serves that on the seat's subjects when the credential
claims the seat, and never lists it among the module's own tools. A module named like its seat
registers once and is both.

The tools answer carries each tool's subjects, and the console and CLI call the subject the listing
gave them instead of composing one.
2026-10-01 15:17:14 +02:00
mesh-admin 278a25b3e5 Merge pull request 'A tool call names the machine it is for, every answer says which machine answered, and a holder's runtime serves its seat's verbs (ADR 0159)' (#22) from feat/a-tool-call-names-the-machine into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 12:01:40 +00:00
jschoubben c65f1993ba A tool call names the machine it is for, every answer says which machine answered, and a holder's
runtime serves its seat's verbs (novox/hq ADR 0159)

A module on several machines served one subject in one queue group, so a call reached whichever
instance answered first and nobody could ask one machine's instance. Now each instance also serves
its subject with its machine as the last token, `<module>.<tool>@<node>` addresses it, and every
answer carries the machine that gave it: the console lists `node` on every module tool, strips it
into the subject, and appends "answered by <node>" to the answer; `mesh call` prints it.

And a seat's verbs are served by whoever claims the seat: the credential names the claimed seats
and the verbs each promises, the runtime serves each verb with the module's tool of the same name
on the seat's own subject, and the bus — which admits only the holder's subscription — decides where
that serving is real. Design 33 §3 and §4, built.
2026-10-01 13:58:22 +02:00
jschoubben 6a91d144b3 Merge pull request 'The console lists and calls a role's tools' (#21) from feat/the-mesh-answers-for-itself into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
Reviewed-on: http://git.novox.be/novox/mesh-tools/pulls/21
2026-09-30 15:54:31 +00:00
jschoubben 71965ef958 The console lists and calls a role's tools
seat:<seat>.<verb> addresses a role's tool (with @<node> for a node-scoped seat); the listing asks the
mesh-controller seat's tools verb beside the modules and marks a role's tools; <seat>.<verb> resolves
to the seat when the seat declares that verb, a module's own name otherwise (novox/hq ADR 0154).
2026-09-30 17:41:56 +02:00
jschoubben dea98e509a Merge pull request 'The console: mesh serve on loopback, and every runtime answers tools' (#20) from feat/the-console into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
Reviewed-on: http://git.novox.be/novox/mesh-tools/pulls/20
2026-09-30 14:46:54 +00:00
jschoubben 80b02740ab The console: mesh serve on loopback, and every runtime answers tools
The runtime serves a tools verb per module with names, descriptions and schemas (design 34 §3), and
refuses a module naming its own tool tools. Discovery asks catalog_modules then each module, naming
what did not answer. One MCP handler over two transports: stdio (mesh mcp) and loopback HTTP (mesh
serve, the mesh-console module, novox/hq ADR 0152); serve refuses any bind but loopback. tools/call
may go through a running console with --console and no credential.
2026-09-30 16:19:22 +02:00
mesh-admin 621d033d53 Merge pull request 'One consumer, one reader, however many patterns a module registers' (#19) from fix/one-consumer-one-loop into main 2026-09-28 14:25:47 +00:00
jschoubben 38831c5c56 One consumer, one reader, however many patterns a module registers
A module has exactly one durable consumer, and each subscribe() started its own reader of it. Two
readers split the stream between them, and a reader that receives a message its own pattern does not
match acknowledges it — which is the right answer for a filter wider than anything registered, and
silent loss when the message was another handler's. The first module to subscribe twice would have
dropped roughly half of each kind of event with nothing reporting it.

Every registration is now dispatched from one reader, and a message is acknowledged once every handler
it is for has taken it.
2026-09-28 16:25:42 +02:00
mesh-admin fdad2f3268 Merge pull request 'A module answers the word the mesh asks: prepare' (#18) from feat/a-module-answers-prepare into main 2026-09-28 10:45:02 +00:00
jschoubben 81972a4995 A module answers the word the mesh asks: prepare
The runtime gains `prepare`, which brings this module's state to the shape this version needs and
exits (novox/hq ADR 0135). The entrypoints come from MESH_PREPARE, which a module's own image names
beside the entrypoints it already lists there — the module knows which of its files prepares its
state and nothing else could. No broker is connected: preparation runs before the version that would
use it. An empty list fails rather than passing quietly, because the mesh asks this only of a module
whose manifest says it prepares something, and exiting 0 would let that version serve against a
state nobody shaped.
2026-09-28 12:45:00 +02:00
mesh-admin 10e8191717 Merge pull request 'The runtime hears on its own inbox' (#17) from fix/the-runtime-hears-on-its-own-inbox into main 2026-09-28 02:30:24 +00:00
jschoubben d703cebff4 The runtime hears on its own inbox
Every user's inbox is private to it and the grant names it; a reply space the client invented was
refused, and with it every pull for the next message and every answer to a tool call.
2026-09-28 04:30:22 +02:00
mesh-admin 46b56d53a6 Merge pull request 'The pin is the only check: the runtime stops verifying the bus's name' (#16) from fix/the-pin-is-the-only-check into main 2026-09-28 02:03:46 +00:00
jschoubben d4a2802342 The pin is the only check: the runtime stops verifying the bus's name
Every module on the new runtime reached the handshake and failed on 'does not match
certificate's altnames': the bus's certificate names the seat, not the address a machine dials
it by, and pinning the exact certificate already decides everything a name check could. The
client's transport spreads the TLS options into Node's tls.connect, so the hostname check is
replaced with one that passes and the pinned certificate is the one authority accepted.
2026-09-28 04:03:43 +02:00
mesh-admin 8acfa7a07d Merge pull request 'One bus: the runtime pins the certificate after the server speaks, and the old transport goes' (#15) from feat/one-bus into main 2026-09-28 01:09:01 +00:00
jschoubben f0104b7846 One bus: the runtime pins the certificate after the server speaks, and the old transport goes
Every module that dialled the new bus failed its handshake with "wrong version
number": the runtime pinned the server's certificate by a raw TLS connection to a
port on which the server speaks first, in the clear. The pin is taken after the
INFO line now, on the same socket, and then the real connection verifies against
exactly that certificate.

And the old transport is deleted — its client, its tests, its dependency — with
the wire-compatibility pins that only existed for the move (novox/hq ADR 0131,
design 28 task 5.5). A credential names the bus, and there is one.
2026-09-28 03:08:59 +02:00
mesh-admin cd26131c61 Merge pull request 'The runtime speaks the bus its credential names' (#14) from feat/the-runtime-speaks-the-bus-its-credential-names into main 2026-09-28 00:42:29 +00:00
jschoubben 9e3ff6fa45 The runtime speaks the bus its credential names
A module moved to the bus being built was handed a credential for it — nats://
with user, password and fingerprint beside the address — and nothing else in its
environment changed. The runtime always dialled the old bus, so every moved module
kept serving and answered nobody. The scheme in the credential is enough to know
which bus to speak; the nats broker was already written and never chosen.
2026-09-28 02:42:26 +02:00
jschoubben a4447f1251 Merge pull request 'A person's own client, and pins that the wire did not change' (#13) from feat/nats-genesis into main 2026-09-27 17:20:07 +00:00
jschoubben 3d55aeb1d8 The old bus's wire is pinned unchanged, because this has to merge to a running mesh
Every module's event names were converted from the old bus's routing keys to local
names, and this client maps them back. If that mapping is wrong anywhere a live mesh's
events stop being delivered — silently, because a binding that matches nothing is not an
error.

So the mapping is pinned against the literal routing keys the mesh published before,
taken from the manifests as they were: what each module now emits, what each now binds,
and that a handler still matches what the bus delivers. Including the audit logger's
"everything", which must stay `#` on this bus.

And a key already in the old form is left alone, so a module built from an older manifest
keeps working beside one built from a current manifest — which is the state the mesh will
actually be in between deployments.
2026-09-27 17:37:19 +02:00
jschoubben 9acc40145a A person's client: the mesh's tools from a workstation
Design 25 §7's second item. Two surfaces over one thing — a command line for somebody
at a terminal, an MCP server for an agent — and both are adapters over the same three
calls: what tools are there, what does this one take, call it. A second way of reaching
a tool would be a second thing to keep correct.

It uses the client a module's runtime uses. Not a bridge and not a second protocol: a
person connects as their own bus user and publishes on the tool subjects their account
permits, so "what may this person do" is answered by the same permission list that
answers it for a module, and an audit has nothing separate to read.

`mesh tools` lists what the *catalogue* has, not what this credential may call. The two
differ and the difference is the point: somebody seeing only their own tools cannot tell
"not installed" from "not yours", and those need different people to fix them.

A failed call says which of three things happened, because the remedies are in three
different places: nobody serves that tool, this credential may not call it, or the tool
itself was slow. Without that they are one timeout and a stack trace.

The MCP surface decides nothing. The tool names are the ones a person types, the schemas
are the modules' own, and an answer is passed through unshaped — an adapter that
summarised somebody else's answer would be deciding what matters in it. A tool that fails
comes back as a tool error rather than a protocol error, because the request was
well-formed and the mesh answered it.

Written against the protocol directly: it is three methods and one framing, and a
dependency here would be a dependency on every workstation.

Tests drive both surfaces against a real bus, including that a host's notification is
answered with nothing and an unknown method is refused. They run one file at a time,
because each stands up a module serving the same tool subjects and run together their
requests get split between them — which showed up as one test reading another's answer.
2026-09-27 17:03:13 +02:00
jschoubben fbeb373d1a Both clients map local event names to their own wire
A module names its events locally and each transport works out where they land.
That is what design 29 says and what neither client did: both passed the name
straight through, which happened to be right on the old bus because modules were
writing routing keys, and wrong on the new one (novox/hq 04-ISSUES/127).

The old bus's client now turns a local name into `module.<emitter>.<event>` on the
way out and back on the way in. Without that, converting the modules to local
names would have broken the mesh that is actually running.

**A handler and a manifest now say the same thing.** The key a module sees was the
event name alone, so a manifest declaring `consumes: builder.built` produced a
pattern that could never match what it was compared against — and a module
consuming one event from two emitters could only tell them apart by reading a
header. The subject already carries the emitter, so naming it in the key makes a
mismatch between manifest and code a typo instead of a category error.

Both matchers accept `**` for the rest of a name, which is how a manifest spells
it; the old bus's `#` still works, because both buses ship until the rollout.
2026-09-27 14:42:40 +02:00
jschoubben 2197c36fef Describe the client on its own terms
Same cleanup: the comments explained each decision by contrast with what
came before instead of stating it. The certificate constraint stays — it is
a fact about the mesh's certificates, not a comparison.
2026-09-26 23:51:00 +02:00
jschoubben 19560ca6a7 Hold the runtime's NATS client to the shared fixtures
Read back from the stream rather than from the client that wrote it, so the
check is what reached the wire. The runner lives with the implementation;
the fixture stays in one place.
2026-09-26 23:40:59 +02:00
jschoubben c1517c39a0 The tool runtime's client on NATS, behind the unchanged contract
Task 3.6 of novox/hq ADR 0116. A module is still written against request,
handle, publish, subscribe, close; only what is underneath changes. main.ts
still selects the AMQP client — steps 1 to 4 leave every node on AMQP, so
this ships beside it and is selected at the rollout.

Round-tripped against a real server (test/roundtrip.mjs): a tool answered
across two connections, a throwing handler reaching the caller as an error
rather than a timeout, an event delivered once with its key, body, node and
event id intact, and an event landing under its emitter's own namespace.

Three things the compiler and the server corrected:

- the envelope's field is `key`, not `type`, and the payload is `env.body`
  with metadata in headers — not the whole envelope re-encoded. An
  implementation that nested the envelope would pass all its own tests and
  agree with nobody, which is what the conformance suite exists to stop.
- the NATS client's TLS options are PEM strings with no verify hook, so the
  AMQP client's `checkServerIdentity: () => undefined` has no equivalent.
  The fingerprint check still happens and is still the guarantee, but the
  bus's certificate must now carry a SAN matching the address nodes dial.
  That is a constraint on the mesh's certificates, recorded where it bites.
- a durable consumer is bound, never created: a module's account cannot
  reach the JetStream API, and a runtime creating its own would be a module
  choosing its own delivery semantics.
2026-09-26 23:29:17 +02:00
128 changed files with 16482 additions and 772 deletions
+1
View File
@@ -1,2 +1,3 @@
node_modules/ node_modules/
dist/ dist/
.mesh-build/
+47 -16
View File
@@ -1,5 +1,8 @@
ARG NODE_BASE=node:22-bookworm-slim ARG NODE_BASE=node:22-bookworm-slim
# Three stages, two published images: the one modules are COMPILED in, and the one they RUN in. # The mesh-tools module: the two images every TypeScript module is COMPILED in and may RUN in. The
# runtime itself ships as the node-tools module's bundle (node-tools/, novox/hq ADR 0175, to-be 38
# WP3); these images are the toolchain for TypeScript bundles and the base a module's own service
# may still be built on. They are no longer how tools reach a node.
# #
# **They were the same image, and that was a mistake.** A module's recipe starts from this and # **They were the same image, and that was a mistake.** A module's recipe starts from this and
# invokes the compiler out of it, so the compiler had to be here — and because the same image was # invokes the compiler out of it, so the compiler had to be here — and because the same image was
@@ -19,36 +22,64 @@ RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates \ && apt-get install -y --no-install-recommends git ca-certificates \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
WORKDIR /app WORKDIR /app
COPY package.json ./ COPY node-tools/package.json ./
# The builder writes .npmrc into the build context; it authenticates to the mesh's package registry # The builder writes .npmrc into the build context; it authenticates to the mesh's package registry
# for the @novox scope, which is where @novox/mesh-sdk resolves. This stage is not published, so the # for the @novox scope, which is where @novox/mesh-sdk resolves. This stage is not published, so the
# credential travels no further than here. Development dependencies included: the compiler is one. # credential travels no further than here. Development dependencies included: the compiler is one.
COPY .npmrc ./.npmrc COPY .npmrc ./.npmrc
RUN npm install --no-audit --no-fund RUN npm install --no-audit --no-fund
# **The SDK this image carries is the one the mesh last published** (novox/hq issue 212). The range in
# package.json is resolved once and the layer above is cached, so a release reached no toolchain until
# that file changed. The exact version arrives as a build argument from the SDK module's published
# package (module.json `build.on`), so a new release is a new argument, this layer runs again — and
# the planner orders this module after the SDK, so a release rebuilds the toolchain and, after it,
# every bundle compiled in it (issue 211).
ARG MESH_SDK=@novox/mesh-sdk@latest
RUN npm install --no-audit --no-fund "${MESH_SDK}"
# ---- toolchain: what a module is compiled in, WITHOUT the credential -------------------------- # ---- compiling: the runtime's own code built, WITHOUT the credential -------------------------
FROM ${NODE_BASE} AS toolchain FROM ${NODE_BASE} AS compiling
WORKDIR /app WORKDIR /app
COPY package.json ./ COPY node-tools/package.json ./
# The resolved libraries, but not the .npmrc that resolved them. # The resolved libraries, but not the .npmrc that resolved them.
COPY --from=deps /app/node_modules ./node_modules COPY --from=deps /app/node_modules ./node_modules
# The toolkit arrives compiled. It used to arrive as sources, and this compiled it by hand — the COPY node-tools/tsconfig.json ./
# hook that builds it on install was running all along, and the result was then packed out of the COPY node-tools/src ./src
# package, because with no explicit file list npm falls back to .gitignore and that ignores the
# build output. Fixed where it belonged, in the toolkit.
COPY tsconfig.json ./
COPY src ./src
RUN npm run build RUN npm run build
# ---- what the running image needs, and nothing else ------------------------------------------- # ---- what a running bundle needs, and nothing else -------------------------------------------
# Its own stage so the toolchain image keeps its build tools while the runtime image does not. # Its own stage so the toolchain image keeps its build tools while the runtime image does not.
FROM toolchain AS lean FROM compiling AS lean
RUN npm prune --omit=dev RUN npm prune --omit=dev
# ---- runtime: what a module runs in ----------------------------------------------------------- # ---- toolchain: what a TypeScript bundle is compiled in ---------------------------------------
# The compiler, and esbuild (a development dependency) at /app/node_modules/esbuild: the builder
# bundles every entrypoint and launcher into one file with it (novox/hq ADR 0193), so a bundle
# carries what it imports and not this image's node_modules.
# Beside the compiler, at /app/runtime, what every TypeScript bundle runs with: the production
# dependencies the SDK and the runtime need, and a package.json saying the compiled files are ES
# modules. The builder copies this directory whole into a compiled bundle (novox/hq ADR 0188 §5),
# so a bundle unpacked on a machine starts — a `.js` without that package.json is read as
# CommonJS, and an import of `nats` without node_modules beside it resolves to nothing.
#
# **The client libraries a catalogue module depends on are here too** (novox/hq issue 385): pg, mongodb and
# mssql, development dependencies of node-tools, so the runtime stage prunes them. A catalogue's
# merge-check-typescript.sh fetches nothing and copies this image's node_modules beside a module; a module
# depending on a package that is not here cannot be type-checked or tested, and fails its check.
FROM compiling AS toolchain
COPY --from=lean /app/node_modules /app/runtime/node_modules
RUN printf '{"type":"module","private":true}\n' > /app/runtime/package.json
# **git, because a pull request's merge check runs in this image too** (novox/hq ADR 0237): a repository's
# own merge-check.sh that declares this toolchain runs here, and a suite that reads a repository's history
# (the lab's last-run record) ran into "spawnSync git ENOENT". Nothing compiled here calls it.
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/*
# ---- runtime: what a module's own service may run in ------------------------------------------
FROM ${NODE_BASE} AS runtime FROM ${NODE_BASE} AS runtime
WORKDIR /app WORKDIR /app
COPY package.json ./ COPY node-tools/package.json ./
COPY --from=lean /app/node_modules ./node_modules COPY --from=lean /app/node_modules ./node_modules
COPY --from=toolchain /app/dist ./dist COPY --from=compiling /app/dist ./dist
ENTRYPOINT ["node", "dist/main.js"] ENTRYPOINT ["node", "dist/main.js"]
+80 -17
View File
@@ -1,32 +1,95 @@
# mesh-tools # mesh-tools
The Novox Mesh **tool runtime** — the per-node process that makes a module's tools actually serve. Two modules in one repository (novox/hq ADR 0069), one piece of software:
A module ships its tools (built on [`@novox/mesh-sdk`](https://git.novox.be/novox/mesh-sdk)); this - **`node-tools`** (`node-tools/`) — the node's **tool runtime** as a module (ADR 0175, to-be 38
runtime is what loads them and puts them on the mesh. It: WP3): one process per machine the host runs from this bundle, serving every assigned module's tools
and every held seat's verbs on the bus, and answering MCP on the machine's loopback — the console
(design 34). The code, its tests and the `mesh` client all live there.
- **`mesh-tools`** (this directory) — the two images TypeScript bundles are compiled in and a module's
own *service* may still run in. Built from the same code; no longer how tools reach a node.
1. connects the mesh broker (novox/hq ADR 0001) — a concrete AMQP implementation of the sdk's The runtime:
`Broker` contract;
2. imports the assigned modules' compiled tool entrypoints, each of which registers its tools as it 1. connects the mesh bus on the node's credential — a concrete implementation of the sdk's `Broker`
loads; contract;
3. serves them through the sdk's `serveTools` harness, answering `tools.invoke` over the broker. 2. reads one membership per module it serves — what the mesh issued that module on this machine
(ADR 0160): where its tools are answered, which seats it holds — and follows each live;
3. imports each module's compiled tool entrypoints, each of which registers its tools as it loads,
guarded: a bundle that throws is named, in the log and in what `tools` answers for its module,
and the others serve;
4. serves every module's tools on that module's subjects and every held seat's verbs on the seat's.
A bundle that is not plain JavaScript — a Go or Rust binary, a Python script, or a JavaScript file
marked executable — is **launched** rather than imported (novox/hq ADR 0188): the runtime starts it
as a child with its own environment and speaks MCP over stdio to it, `tools/list` once and
`tools/call` per call. A tool it lists as `<seat>.<verb>` is the seat's implementation. A child that
exits is named in the log and started again on its next call. So a tools bundle may be written in
any language; the mesh's SDK for each is the stdio loop and nothing more (`node-tools/src/launch.ts`
is the runtime's side of it).
Everything hard — dispatch, collection, duplicate-name safety — is the sdk's. This is the thin Everything hard — dispatch, collection, duplicate-name safety — is the sdk's. This is the thin
wrapper that binds the broker and loads the modules. Keeping the AMQP client here, out of the sdk, wrapper that binds the bus and loads the modules. Keeping the bus client here, out of the sdk, is
is deliberate: a broker-client change never rebuilds a module (ADR 0039). deliberate: a bus-client change never rebuilds a module (ADR 0039). The runtime is module-agnostic:
it knows bundles and subjects, nothing of what any module does. A tool that needs root escalates
itself — root is the module's concern, not the runtime's.
## Running it ## Running it
``` ```
MESH_BROKER_URL amqp://… the mesh broker MESH_BROKER_FILE the node's sealed credential, as the mesh delivered it
MESH_TOOL_MODULES /a/tools/index.js,… the assigned modules' compiled tool entrypoints MESH_TOOL_MODULES alpha=/…/alpha/tools/index.js,beta=/…/beta/dist/index.js,…
the modules to serve and their compiled entrypoints; several entries may
name one module. A bare path is an entrypoint of the credential's own module
— the one-module form a per-module container still sets.
MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME where their home is; set by
the mesh when the node has an account, read by tools from their environment
MESH_BROKER_URL a plain URL instead of the credential, for the bootstrap case
``` ```
`node dist/main.js`, or the container (`Dockerfile`). On a node the host resolves both variables `node dist/main.js`. On a node the controller composes the variables and the host supervises the
and starts it like any other supervised workload. process like any other host-side workload (novox/hq to-be 38). As `node-tools` the same process is
the console: MCP on `127.0.0.1:4270` (or `MESH_CONSOLE_LISTEN`). The container (`Dockerfile`) is how
a module's own *service* may still be built; it is no longer how tools reach a node.
## `mesh` — the tools for whoever is on a machine
The same package carries the client (novox/hq design 25 §7, design 34): `mesh tools`, `mesh call
<module>.<tool> [json]`, `mesh mcp` (an MCP server over stdio for a program a person starts) and
`mesh serve` (the **console**: MCP over HTTP on a machine's loopback, started by the mesh as the
`mesh-console` module on the credential in `MESH_BROKER_FILE` — novox/hq ADR 0152). `mesh serve`
refuses to bind anything but loopback. With `--console <url>`, `tools` and `call` go through a console
already on the machine and need no credential.
The console announces six tools and reaches everything else by address (novox/hq ADR 0195):
`mesh_overview` (the mesh's seats and machines), `mesh_machine` (one machine's seats and modules),
`mesh_search` (a tool by words), `mesh_describe` (one tool's arguments), `mesh_call` (call one by
address) and `mesh_runtimes` (which runtimes answered discovery: per runtime its machine, how long its
answer took, its size in bytes, how many modules and tools it announced, whether it was shortened to
fit the bus, when it was last heard — and who was expected and not heard).
`mesh_search` also finds a tool by the shell command it replaces (novox/hq ADR 0245): each seat verb and
module tool says what it replaces in the controller's records (`replaces`, read from the controller's
`tools` and `modules` answers), so `journalctl -u x`, `systemctl status x` or `docker ps` puts the verb
for it first, and says the command it matched as `instead_of`. A word also matches through a short table
of the words agents use for the mesh's (`logs` finds the journal). Best first: a replaced command, then
words in a name or a replaced command, then words in a description; seats before modules on a tie.
Discovery asks the bus (ADR 0197): every runtime answers the NATS services protocol's `$SRV.PING` and
`$SRV.INFO` with what it serves at that moment, and the console reads where the controller's records
place each module. The console waits at least 750 ms, and up to 5 s for every runtime that answered
PING to send what it serves — a large runtime's answer crossing the bus to a broker on another machine
can arrive well after the rest. A runtime that said it is there and did not say what it serves, or
that answered earlier and not now, is named: the console never calls a module missing, or on another
machine, while a runtime that might serve it was not heard. A runtime whose answer outgrows the bus
announces first-line descriptions, then none, and says so. A module may not name a tool of its own
`tools`; the runtime refuses it at load.
## Verified ## Verified
`npm test` stands up LavinMQ (the mesh's broker) and proves the whole path over real AMQP: the `npm test` runs against a real NATS server with JetStream (`MESH_TEST_NATS`, see any test's header
runtime serves a registered tool, a separate connection invokes it by name and gets the result, and for the one-line `docker run`) and proves the whole path over the wire: the runtime serves a
an unknown tool is refused over the wire. registered tool, a separate connection invokes it by name and gets the result, an unknown tool is
refused, a runtime serves exactly the subjects it is issued and re-serves on a new membership, and
the node's runtime serves three modules' bundles on one credential — one of them broken, named and
not fatal — with every seat verb answering where the membership put it.
-48
View File
@@ -1,48 +0,0 @@
import amqp from "amqplib";
const PORT = process.argv[2];
const MPORT = process.argv[3];
const B = `http://127.0.0.1:${MPORT}`;
const AUTH = "Basic " + Buffer.from("guest:guest").toString("base64");
async function api(method, path, body) {
const r = await fetch(B + path, {
method,
headers: { "content-type": "application/json", authorization: AUTH },
body: body ? JSON.stringify(body) : undefined,
});
if (r.status >= 300 && r.status !== 404) throw new Error(`${method} ${path} -> ${r.status}`);
}
await api("PUT", "/api/exchanges/%2f/mesh.events.dead", { type: "topic", durable: true });
await api("PUT", "/api/users/al", { password: "s", tags: "" });
const Q = "anchor.al.events";
const D = "mesh.events.dead";
const q = Q.replace(/\./g, "\\.");
const d = D.replace(/\./g, "\\.");
// configure, write, read patterns per grant on the dead exchange
const combos = {
"none": { configure: `^${q}$`, write: `^${q}$`, read: `^${q}$` },
"read-dead": { configure: `^${q}$`, write: `^${q}$`, read: `^(${q}|${d})$` },
"write-dead": { configure: `^${q}$`, write: `^(${q}|${d})$`, read: `^${q}$` },
"configure-dead": { configure: `^(${q}|${d})$`, write: `^${q}$`, read: `^${q}$` },
"read+write-dead": { configure: `^${q}$`, write: `^(${q}|${d})$`, read: `^(${q}|${d})$` },
};
let i = 0;
for (const [label, perms] of Object.entries(combos)) {
await api("PUT", "/api/permissions/%2f/al", perms);
const queue = `${Q}.${i++}`; // fresh each time
try {
const c = await amqp.connect(`amqp://al:s@127.0.0.1:${PORT}/`);
const ch = await c.createChannel();
ch.on("error", () => {});
await ch.assertQueue(queue, { durable: true, deadLetterExchange: D });
console.log(`${label}: declare-with-DLX OK`);
await c.close();
} catch (e) {
console.log(`${label}: FAIL - ${String(e.message).slice(0, 70)}`);
}
}
+55
View File
@@ -0,0 +1,55 @@
#!/bin/sh
# mesh-check-toolchain: go
#
# The tool runtime's own check (novox/hq ADR 0237 as amended): the second layer of a pull request's merge
# check, `mesh/repo-check`, run by the build seat in the mesh's Go toolchain — and by hand.
#
# The gate — every machine of the facts snapshot composed with this change — is the build seat's first
# layer (`mesh/merge-gate`), run because the mesh's module graph builds node-tools and mesh-tools from
# here. This is the code's own: node-tools (Go) formatted, vetted and its suite, every test on a bus of its
# own at the release the mesh runs (internal/meshtest), packages in parallel, under the race detector when
# the toolchain has a C compiler.
#
# **Said, never passed silently**: the runtime's and the console's tests launch TypeScript and Python
# bundles that import the mesh's own package (@novox/mesh-sdk), which comes from the package registry — and
# a check of code nobody has approved is given no credential for it. Where node and that package are not
# present those two packages are not tested here, and the TypeScript in node-tools/src neither.
set -eu
cd node-tools
unformatted=$(gofmt -l cmd internal)
if [ -n "$unformatted" ]; then
echo "not gofmt'd:"
echo "$unformatted"
exit 1
fi
CGO_ENABLED=0 go vet ./...
packages=$(go list ./...)
bundleless=""
if ! command -v node >/dev/null 2>&1 || [ ! -d node_modules/@novox/mesh-sdk ]; then
echo "NOT TESTED HERE: internal/runtime and internal/console launch bundles that need node and @novox/mesh-sdk"
packages=$(printf '%s\n' "$packages" | grep -v -e '/internal/runtime$' -e '/internal/console$')
# What of the console launches no bundle runs anyway: how an address resolves (novox/hq issue 287),
# and how a search ranks (novox/hq ADR 0245), and what discovery heard and how a round that heard nothing is
# answered (novox/hq issue 554).
bundleless='^(TestASeatAndAModuleOfOneNameAreEachReached|TestSearch.*|TestASearchByCommandSaysWhatItReplaces|TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed|TestARuntimeHeardBeforeAndNotNowIsNamed|TestACallAfterARoundThatHeardNothingUsesWhatWasHeardBefore|TestAnAddressKnownOnlyFromAnEarlierAnswerSaysHowOld)$'
# And what of the runtime launches no bundle: how a bundle's seat traffic reaches the bus, what its
# environment holds, what a runtime on a module's own account serves (novox/hq ADR 0259), and that a write
# by compare-and-set loses no update under contention (novox/hq research 042), and a module's log verbs
# (novox/hq ADR 0297), and a key's history (novox/hq ADR 0307).
runtimeless='^(TestALogAnswersEachVerbOnTheBus|TestALogPageIsBoundedByItsBytes|TestAnAppendAtAStaleExpectIsRefusedAsALostUpdate|TestALogNotIssuedIsRefusedWithTheReason|TestWhatALogCannotHoldIsRefusedWithTheReason|TestALogRefusesAValueNamingACredential|TestAKeysHistoryIsEveryValueKeptOldestFirst|TestAKeysHistoryIsRefusedToAModuleNotIssuedTheState|TestARuntimeOnAModulesOwnAccountServesThatModuleAlone|TestNoBusWordReachesABundle|TestAToolCallNeverReachesASeatsEventAcceptOrProof|TestABundlesSeatTrafficGoesThroughItsModulesBus|TestALostCompareAndSetCarriesItsCode|TestTwoWritersIncrementingOneCounterOnTheBusLoseNoUpdate|TestAStaleWriteIsRefusedAndWritesNothing|TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller|TestTheSubjectThatNamesNoCallerStillWorksAndNamesNone)$'
fi
if command -v gcc >/dev/null 2>&1; then
CGO_ENABLED=1 go test -race -count=1 $packages
else
echo "NOT RACE-CHECKED: the toolchain holds no C compiler; the suite runs without the race detector"
CGO_ENABLED=0 go test -count=1 $packages
fi
if [ -n "$bundleless" ]; then
CGO_ENABLED=0 go test -count=1 -run "$bundleless" ./internal/console
fi
if [ -n "${runtimeless:-}" ]; then
CGO_ENABLED=0 go test -count=1 -run "$runtimeless" ./internal/runtime
fi
echo "NOT TESTED HERE: node-tools/src (TypeScript) needs @novox/mesh-sdk from the package registry"
+5
View File
@@ -21,6 +21,11 @@
{ {
"arg": "NODE_BASE", "arg": "NODE_BASE",
"image": "node@sha256:48e4b67d85f87bd551df43704e24d252f56cc5f8e9718841aace50f19948f0f9" "image": "node@sha256:48e4b67d85f87bd551df43704e24d252f56cc5f8e9718841aace50f19948f0f9"
},
{
"arg": "MESH_SDK",
"module": "mesh-sdk",
"artifact": "lib"
} }
] ]
}, },
+11
View File
@@ -0,0 +1,11 @@
# node-tools
The node's tool runtime as a module (novox/hq ADR 0175, to-be 38 WP3). Assigned to a machine, it is
one process the host runs from this bundle, as the operator's account: it serves every assigned
module's tools and every held seat's verbs on the bus, and answers MCP on the machine's loopback —
the console (design 34). The controller composes the process (which bundles to load, where the
credential is, whose machine it is); this manifest says only what the machine must have for it: the
interpreter, a place for the credential, the loopback port, and leave to call every tool.
The code is the `mesh-tools` package in this directory; the module at the repository root,
`mesh-tools`, builds the images TypeScript bundles are compiled in. See the repository README.
+138
View File
@@ -0,0 +1,138 @@
// node-tools — the node's tool runtime, in Go (novox/hq ADR 0175, ADR 0193; to-be 38 WP4d).
//
// One process per machine: it connects to the bus on the node's credential, launches every assigned
// module's tools bundle and serves its tools and its seats' verbs; as the node-tools module — or
// wherever MESH_CONSOLE_LISTEN says — it is also the console, MCP over HTTP on loopback.
//
// MESH_BROKER_FILE the credential the mesh sealed to this machine for the runtime
// MESH_TOOL_MODULES <module>=<entrypoint>,… the bundles to launch
// MESH_TOOL_ENV {"<module>": {"<word>": "<value>"}} what each is given (ADR 0192)
// MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME their home
// MESH_CONSOLE_LISTEN where the console listens, overriding 127.0.0.1:4270
package main
import (
"encoding/json"
"fmt"
"log"
"math/rand/v2"
"os"
"os/signal"
"syscall"
"time"
"github.com/novox/mesh-tools/node-tools/internal/alive"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/console"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
// runtimeModule is the module that is the node's tool runtime; on its credential, serving is also
// the console.
const runtimeModule = "node-tools"
// consoleListen is where the console listens when nothing says otherwise.
const consoleListen = "127.0.0.1:4270"
func main() {
log.SetFlags(0)
if len(os.Args) > 1 && os.Args[1] != "serve" {
fmt.Fprintf(os.Stderr, "node-tools: %q is not a mode; this runtime serves (novox/hq ADR 0193)\n", os.Args[1])
os.Exit(2)
}
cred := credential()
conn := connectPatiently(cred)
served, err := runtime.ServedModulesFrom(os.Getenv(runtime.ToolModules), cred.Module)
if err != nil {
log.Fatalf("node-tools: %v", err)
}
envs, err := runtime.TakeToolEnvs()
if err != nil {
log.Fatalf("node-tools: %v", err)
}
stop, err := runtime.Run(conn, served, envs, log.Printf)
if err != nil {
log.Fatalf("node-tools: %v", err)
}
// And says it is there (novox/hq to-be 45 S11): the machine's runtime, on its machine's name. A
// runtime on another credential — a module's own, in a test — is not the machine's.
stopSaying := func() {}
if cred.Module == runtimeModule && cred.Node != "" {
stopSaying = alive.Keep(conn, cred.Node, alive.Every, log.Printf)
}
listen := os.Getenv("MESH_CONSOLE_LISTEN")
if listen == "" && cred.Module == runtimeModule {
listen = consoleListen
}
var up *console.Listening
if listen != "" {
node := cred.Node
if node == "" {
node = "?"
}
who := node + "." + cred.Module
up, err = console.Serve(console.NewSurface(conn, who), listen)
if err != nil {
log.Fatalf("node-tools: %v", err)
}
log.Printf("mesh console listening on http://%s/mcp as %s", up.Address, who)
}
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGTERM, syscall.SIGINT)
<-signals
stopSaying()
stop()
if up != nil {
_ = up.Close()
}
conn.Close()
}
// credential reads the credential the mesh delivered; a missing or unreadable one is a fault of
// configuration, said and final.
func credential() bus.Credential {
file := os.Getenv("MESH_BROKER_FILE")
if file == "" {
if url := os.Getenv("MESH_BROKER_URL"); url != "" {
return bus.Credential{URL: url, Module: runtimeModule}
}
fmt.Fprintln(os.Stderr, "node-tools: set MESH_BROKER_FILE (a sealed credential) or MESH_BROKER_URL — there is no broker to reach")
os.Exit(1)
}
raw, err := os.ReadFile(file)
if err != nil {
fmt.Fprintf(os.Stderr, "node-tools: cannot read the broker credential at %s: %v\n", file, err)
os.Exit(1)
}
var cred bus.Credential
if err := json.Unmarshal(raw, &cred); err != nil {
fmt.Fprintf(os.Stderr, "node-tools: cannot read the broker credential at %s: %v\n", file, err)
os.Exit(1)
}
if cred.URL == "" {
fmt.Fprintf(os.Stderr, "node-tools: %s carries no url — it is not a broker credential\n", file)
os.Exit(1)
}
return cred
}
// connectPatiently retries while the bus is merely not reachable yet — the normal case at startup —
// and gives up at once on what waiting cannot fix (issue 058).
func connectPatiently(cred bus.Credential) *bus.Conn {
for delay := 2 * time.Second; ; delay = min(delay*2, 30*time.Second) {
conn, err := bus.Connect(cred)
if err == nil {
return conn
}
if fatal := bus.Fatal(err); fatal != "" {
fmt.Fprintf(os.Stderr, "node-tools: %s — waiting will not fix this; giving up\n", fatal)
os.Exit(1)
}
wait := delay + time.Duration(rand.IntN(1000))*time.Millisecond
fmt.Fprintf(os.Stderr, "node-tools: the broker is not reachable yet (%v); retrying in %ds\n", err, int(wait.Round(time.Second)/time.Second))
time.Sleep(wait)
}
}
+22
View File
@@ -0,0 +1,22 @@
module github.com/novox/mesh-tools/node-tools
go 1.26.0
require (
github.com/nats-io/nats.go v1.54.0
golang.org/x/sys v0.48.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nats-server/v2 v2.11.17 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
go.uber.org/automaxprocs v1.6.0 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+27
View File
@@ -0,0 +1,27 @@
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
go.uber.org/automaxprocs v1.6.0 h1:O3y2/QNTOdbF+e/dpXNNW7Rx2hZ4sTIPyybbxyNqTUs=
go.uber.org/automaxprocs v1.6.0/go.mod h1:ifeIMSnPZuznNm6jmdzmU3/bfk01Fe2fotchwEFJ8r8=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
+80
View File
@@ -0,0 +1,80 @@
// Package alive is the node tools saying they are there (novox/hq to-be 45 §3, S11).
//
// **A machine heard and its runtime gone is a machine nobody can ask anything.** The node-engine says
// every minute that the machine is there; nothing said whether the runtime serving every module's
// tools and every held seat's verbs was. Now it says so too, on its own subject, every interval, with
// the interval — the controller's watchdog raises `tools-silent` after three missed — and its build.
// Core NATS, kept by nobody, as the host's heartbeat: a lost one is the next one.
package alive
import (
"encoding/json"
"runtime/debug"
"sync"
"time"
)
// Every is how often the runtime says it is there.
const Every = 60 * time.Second
// Subject is where one machine's node tools say it.
func Subject(node string) string { return "mesh.control." + node + ".tools-alive" }
// Beat is what is said: the machine, the interval, and the build.
type Beat struct {
Node string `json:"node"`
IntervalSeconds int `json:"interval_seconds"`
Version string `json:"version,omitempty"`
}
// Sayer publishes one message, kept by nobody.
type Sayer interface {
Say(subject string, body []byte) error
}
// Version is this build's commit, as the toolchain stamped it; empty when it did not.
func Version() string {
info, ok := debug.ReadBuildInfo()
if !ok {
return ""
}
for _, s := range info.Settings {
if s.Key == "vcs.revision" {
return s.Value
}
}
return ""
}
// Keep says the runtime is there now and every interval until the returned function is called. A
// heartbeat that cannot be said is logged when that starts and when it stops — not every minute.
func Keep(bus Sayer, node string, every time.Duration, logf func(string, ...any)) func() {
body, _ := json.Marshal(Beat{Node: node, IntervalSeconds: int(every / time.Second), Version: Version()})
done := make(chan struct{})
var once sync.Once
go func() {
failing := ""
tick := time.NewTicker(every)
defer tick.Stop()
for {
why := ""
if err := bus.Say(Subject(node), body); err != nil {
why = err.Error()
}
if why != failing {
if why != "" {
logf("node-tools: could not say this runtime is there — the mesh will call it silent: %s", why)
} else {
logf("node-tools: says again that this runtime is there")
}
failing = why
}
select {
case <-done:
return
case <-tick.C:
}
}
}()
return func() { once.Do(func() { close(done) }) }
}
+74
View File
@@ -0,0 +1,74 @@
package alive
import (
"encoding/json"
"errors"
"sync"
"testing"
"time"
)
type said struct {
mu sync.Mutex
subs []string
body [][]byte
fail error
}
func (s *said) Say(subject string, body []byte) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.fail != nil {
return s.fail
}
s.subs, s.body = append(s.subs, subject), append(s.body, body)
return nil
}
func (s *said) count() int {
s.mu.Lock()
defer s.mu.Unlock()
return len(s.subs)
}
// **The runtime says it is there at once and every interval, as its machine, with the interval**: the
// controller's bound is three of them (to-be 45 S11).
func TestTheRuntimeSaysItIsThereEveryInterval(t *testing.T) {
s := &said{}
stop := Keep(s, "anchor", 20*time.Millisecond, t.Logf)
time.Sleep(70 * time.Millisecond)
stop()
stop()
if n := s.count(); n < 3 {
t.Fatalf("said %d times in three intervals", n)
}
var beat Beat
if err := json.Unmarshal(s.body[0], &beat); err != nil || beat.Node != "anchor" || beat.IntervalSeconds != 0 ||
s.subs[0] != "mesh.control.anchor.tools-alive" {
t.Fatalf("%s on %s: %v", s.body[0], s.subs[0], err)
}
n := s.count()
time.Sleep(50 * time.Millisecond)
if s.count() != n {
t.Fatal("said after it was stopped")
}
}
// **A heartbeat the bus will not take is said once, and again when it is taken**: not a line a minute.
func TestAHeartbeatThatCannotBeSaidIsLoggedOnce(t *testing.T) {
s := &said{fail: errors.New("permissions violation")}
var mu sync.Mutex
var lines []string
stop := Keep(s, "anchor", 10*time.Millisecond, func(f string, a ...any) {
mu.Lock()
lines = append(lines, f)
mu.Unlock()
})
time.Sleep(60 * time.Millisecond)
stop()
mu.Lock()
defer mu.Unlock()
if len(lines) != 1 {
t.Fatalf("logged %d lines for one failure", len(lines))
}
}
+472
View File
@@ -0,0 +1,472 @@
// Package announce answers the NATS services protocol's discovery for what a runtime serves, and
// gathers the answers (novox/hq ADR 0197).
//
// A runtime does not re-serve its tools through a services library: serving is unchanged. It answers
// `$SRV.PING`, `$SRV.INFO` and `$SRV.STATS` — and the same followed by its service name, and by its
// name and id — in the format NATS's own tools read, with what it is serving at the moment it is
// asked. One service per runtime process: the bus admits one reply per request from each responder,
// so a runtime serving many modules and seats answers once, one endpoint per tool per subject, and
// says in each endpoint's metadata which module, seat, scope and machine it is.
package announce
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"strings"
"time"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// Version is the announced service version (semver, as the protocol requires).
const Version = "0.1.0"
// Window is how long the console gathers discovery answers at the least: every instance answers one
// request, and how many will is what is being found out.
var Window = 750 * time.Millisecond
// Patience is how long the console waits, at the most, for the full answer of a service that has
// said it is there. **A fixed window lost the largest runtime** (2026-10-05): the laptop's answer —
// 48 modules, 341 endpoints, 164 kB — went up to the broker on another machine and back, arrived
// last every time (a median of 365 ms, one in 25 after 813 ms on a quiet link, later still while the
// runtime re-served), and when it missed the 750 ms window the console said its modules ran nowhere.
// A PING answer is a hundred bytes and arrives at once, so who is there is known early; what each
// serves is waited for until it arrives or this passes, and a service that never sends it is named.
var Patience = 5 * time.Second
// Digest is the PING answer's metadata key for a digest of what the instance serves now: a console
// that holds the answer of that digest does not ask for it again (novox/hq issue 554). Every round
// pulled every runtime's whole announcement — some 880 kB, every five seconds while a session works —
// to the laptop, and the hundred-byte PING answers queued behind it.
const Digest = "digest"
// Shortened is the service metadata key a runtime sets when its announcement was cut to fit the bus.
const Shortened = "shortened"
// Kinds of endpoint.
const (
KindTool = "tool" // a module's own tool
KindSeat = "seat" // a seat's verb, served by the module holding it
)
// Endpoint is one tool served on one subject, as it is announced.
type Endpoint struct {
Kind string
Module string // the module whose code answers
Tool string // the tool's or the verb's name
Seat string // for a seat's verb
Scope string // "mesh" or "node", for a seat's verb
Node string
Description string
Schema json.RawMessage
Interchangeable bool
Subject string
Queue string
}
// Name is the endpoint's name as the protocol allows it — letters, digits, `-` and `_` — the
// prefix and the tool joined by `__`; the metadata, not the name, is what identifies it.
func (e Endpoint) Name() string {
prefix := e.Module
if e.Kind == KindSeat {
prefix = e.Seat
}
return clean(prefix) + "__" + clean(e.Tool)
}
func clean(s string) string {
var b strings.Builder
for _, r := range s {
if r == '-' || r == '_' || (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') {
b.WriteRune(r)
} else {
b.WriteRune('_')
}
}
return b.String()
}
func (e Endpoint) info() micro.EndpointInfo {
md := map[string]string{
"kind": e.Kind, "module": e.Module, "tool": e.Tool, "node": e.Node,
"description": e.Description, "interchangeable": boolWord(e.Interchangeable),
}
// **A seat's verb carries its schema; a module's tool does not.** Every tool's schema, once per
// subject it answers on, made a runtime's one answer outgrow the bus's largest message (1 MiB) once
// machines served a couple of hundred tools — and the answer that cannot be sent is silence: the
// machine vanished from discovery (2026-10-04). A module's tool's schema is asked of the runtime that
// serves it when a person describes it (its `tools` verb); a seat's verbs are few, and a seat has no
// `tools` verb of its own to ask.
if e.Kind == KindSeat {
schema := strings.TrimSpace(string(e.Schema))
if schema == "" || schema == "null" {
schema = "{}"
}
md["schema"] = schema
}
if e.Kind == KindSeat {
md["seat"] = e.Seat
md["scope"] = e.Scope
}
return micro.EndpointInfo{Name: e.Name(), Subject: e.Subject, QueueGroup: e.Queue, Metadata: md}
}
func boolWord(b bool) string {
if b {
return "true"
}
return "false"
}
// Service is who answers: the runtime's name, its instance, and a word about it.
type Service struct {
Name string
ID string
Description string
Metadata map[string]string
}
func (s Service) identity() micro.ServiceIdentity {
md := s.Metadata
if md == nil {
md = map[string]string{}
}
return micro.ServiceIdentity{Name: s.Name, ID: s.ID, Version: Version, Metadata: md}
}
// Info is the info_response for these endpoints.
func Info(s Service, endpoints []Endpoint) micro.Info {
out := micro.Info{ServiceIdentity: s.identity(), Type: micro.InfoResponseType,
Description: s.Description, Endpoints: []micro.EndpointInfo{}}
for _, e := range endpoints {
out.Endpoints = append(out.Endpoints, e.info())
}
return out
}
// Serve answers discovery for one service until stopped, asking `current` for its endpoints each
// time — so what is announced is what is served now, re-served memberships included.
func Serve(conn *bus.Conn, s Service, current func() []Endpoint) (func(), error) {
started := time.Now().UTC()
answer := func(subject string, _ []byte) []byte {
parts := strings.Split(subject, ".")
if len(parts) < 2 || parts[0] != "$SRV" {
return nil
}
if len(parts) >= 3 && parts[2] != s.Name {
return nil // another service's
}
if len(parts) >= 4 && parts[3] != s.ID {
return nil // another instance's
}
var v any
switch parts[1] {
case "PING":
id := s.identity()
md := map[string]string{}
for k, v := range id.Metadata {
md[k] = v
}
md[Digest] = DigestOf(Info(s, current()))
id.Metadata = md
v = micro.Ping{ServiceIdentity: id, Type: micro.PingResponseType}
case "INFO":
v = Info(s, current())
case "STATS":
st := micro.Stats{ServiceIdentity: s.identity(), Type: micro.StatsResponseType, Started: started,
Endpoints: []*micro.EndpointStats{}}
for _, e := range current() {
st.Endpoints = append(st.Endpoints, &micro.EndpointStats{Name: e.Name(), Subject: e.Subject, QueueGroup: e.Queue})
}
v = st
default:
return nil
}
body, err := json.Marshal(v)
if err != nil {
return nil
}
// Too large for the bus is no answer at all, and was silent: shorten every description to its
// first line and say so, rather than vanish; still too large, leave descriptions out.
if limit := conn.MaxPayload(); limit > 0 && int64(len(body)) > limit {
if info, ok := v.(micro.Info); ok {
body = shorten(conn.Logf, info, body, limit)
}
}
return body
}
var stops []func()
for _, verb := range []string{"PING", "INFO", "STATS"} {
// Exactly the questions asked of every service and of this one by name and instance — what the
// grants allow (novox/hq ADR 0197). A wildcard is refused by the bus.
for _, subject := range []string{"$SRV." + verb, "$SRV." + verb + "." + s.Name, "$SRV." + verb + "." + s.Name + "." + s.ID} {
stop, err := conn.Raw(subject, answer)
if err != nil {
for _, st := range stops {
st()
}
return func() {}, err
}
stops = append(stops, stop)
}
}
return func() {
for _, st := range stops {
st()
}
}, nil
}
// DigestOf is a digest of an info_response: equal for equal announcements, whatever the moment.
func DigestOf(i micro.Info) string {
body, err := json.Marshal(i)
if err != nil {
return ""
}
sum := sha256.Sum256(body)
return hex.EncodeToString(sum[:12])
}
// shorten is an info_response cut to fit the bus: descriptions to their first line, then none, the
// service's metadata saying which (so discovery can tell a person what it was given).
func shorten(logf func(string, ...any), info micro.Info, body []byte, limit int64) []byte {
md := map[string]string{}
for k, v := range info.Metadata {
md[k] = v
}
info.Metadata = md
for _, step := range []string{"descriptions cut to their first line", "descriptions left out"} {
for i := range info.Endpoints {
e := info.Endpoints[i].Metadata
if step == "descriptions left out" {
delete(e, "description")
} else {
e["description"] = firstLine(e["description"])
}
}
md[Shortened] = step
shorter, err := json.Marshal(info)
if err != nil {
return body
}
logf("[mesh-tools] what this runtime serves is %d bytes, beyond the bus's %d; announced with %s (%d bytes)",
len(body), limit, step, len(shorter))
if int64(len(shorter)) <= limit {
return shorter
}
body = shorter
}
return body
}
// Heard is one service instance's answer to discovery: what it serves, how long it took to arrive,
// and how large it was.
type Heard struct {
Info micro.Info
Took time.Duration
Bytes int
Shortened string // why its descriptions are not whole, or ""
// Digest is the digest its PING gave for this answer, or "" when it gave none.
Digest string
// Reused is true when the instance's PING said it serves what the console already held, so its
// announcement was not asked for again; Took is then how long its PING took.
Reused bool
}
// Machine is the machine an instance runs on: its metadata's, or the one its endpoints name.
func (h Heard) Machine() string {
if n := h.Info.Metadata["node"]; n != "" {
return n
}
for _, e := range h.Info.Endpoints {
if n := e.Metadata["node"]; n != "" {
return n
}
}
return ""
}
// Instance names one service instance.
type Instance struct {
Name string
ID string
Machine string
}
// Discovery is what one round of discovery heard, and who said it was there but did not say what it
// serves in time.
type Discovery struct {
At time.Time
Waited time.Duration
Heard []Heard
Silent []Instance
// Pinged is how many PING answers arrived.
Pinged int
}
// Empty is a round in which nothing arrived at all, not even a PING: no answer about the mesh, but a
// round that failed (novox/hq issue 554).
func (d Discovery) Empty() bool { return d.Pinged == 0 && len(d.Heard) == 0 }
// Known is the announcement a console already holds for an instance at this digest, if any.
type Known func(name, id, digest string) (Heard, bool)
// Gather asks every service on the bus who it is (`$SRV.PING`), and each instance that answers what it
// serves (`$SRV.INFO.<name>.<id>`) unless its PING's digest is one `known` holds; it waits at least the
// window and at most the patience: until at least one instance answered PING and every instance that
// did has said what it serves. A round in which no PING arrives within the window is not taken as the
// mesh's answer that nothing is there (novox/hq issue 554): it waits on, up to the patience. An answer
// that is not the protocol's is skipped.
func Gather(conn *bus.Conn, known Known) (Discovery, error) {
start := time.Now()
d := Discovery{At: start.UTC()}
pings, stopPings, err := conn.Collect("$SRV.PING", nil)
if err != nil {
return d, err
}
defer stopPings()
infos, askInfo, stopInfos, err := conn.Collector("$SRV.INFO")
if err != nil {
return d, err
}
defer stopInfos()
type key struct{ name, id string }
pinged := map[key]int{}
answered := map[key]int{}
where := map[key]string{}
digest := map[key]string{}
asked := map[key]bool{}
reused := map[key]int{} // index in d.Heard of an answer held already
var order []key
complete := func() bool {
if len(pinged) == 0 {
return false // nobody said they are there: not an answer, a round that has not arrived
}
for k, n := range pinged {
if answered[k] < n {
return false
}
}
return true
}
ask := func(k key) {
if asked[k] {
return
}
asked[k] = true
if err := askInfo("$SRV.INFO."+k.name+"."+k.id, nil); err != nil {
conn.Logf("[mesh-tools] discovery could not ask %s/%s what it serves: %v", k.name, k.id, err)
}
}
least := time.NewTimer(Window)
defer least.Stop()
most := time.NewTimer(Patience)
defer most.Stop()
windowOver := false
for {
select {
case a := <-pings:
var p micro.Ping
if json.Unmarshal(a.Data, &p) != nil || p.Type != micro.PingResponseType {
continue
}
d.Pinged++
k := key{p.Name, p.ID}
_, seen := pinged[k]
if !seen {
order = append(order, k)
}
pinged[k]++
if where[k] == "" {
where[k] = p.Metadata["node"]
}
if seen {
// Two instances under one name and id: what each serves is asked of both, never assumed.
if i, ok := reused[k]; ok {
d.Heard = append(d.Heard[:i], d.Heard[i+1:]...)
for kk, j := range reused {
if j > i {
reused[kk] = j - 1
}
}
delete(reused, k)
answered[k]--
}
ask(k)
continue
}
digest[k] = p.Metadata[Digest]
if h, ok := heldAt(known, k.name, k.id, digest[k]); ok {
h.Took, h.Reused, h.Digest = a.At.Sub(start), true, digest[k]
reused[k] = len(d.Heard)
d.Heard = append(d.Heard, h)
answered[k]++
break
}
ask(k)
case a := <-infos:
var i micro.Info
if json.Unmarshal(a.Data, &i) != nil || i.Type != micro.InfoResponseType {
continue
}
k := key{i.Name, i.ID}
answered[k]++
d.Heard = append(d.Heard, Heard{Info: i, Took: a.At.Sub(start), Bytes: len(a.Data), Shortened: i.Metadata[Shortened],
Digest: digest[k]})
case <-least.C:
windowOver = true
case <-most.C:
d.Waited = time.Since(start)
for _, k := range order {
if answered[k] < pinged[k] {
d.Silent = append(d.Silent, Instance{Name: k.name, ID: k.id, Machine: where[k]})
}
}
return d, nil
}
if windowOver && complete() {
d.Waited = time.Since(start)
return d, nil
}
}
}
// heldAt is the answer `known` holds for an instance at this digest; never for a PING without one.
func heldAt(known Known, name, id, digest string) (Heard, bool) {
if known == nil || digest == "" {
return Heard{}, false
}
return known(name, id, digest)
}
// Endpoints reads an info_response's endpoints back into what they announce.
func Endpoints(i micro.Info) []Endpoint {
var out []Endpoint
for _, e := range i.Endpoints {
md := e.Metadata
out = append(out, Endpoint{
Kind: md["kind"], Module: md["module"], Tool: md["tool"], Seat: md["seat"], Scope: md["scope"],
Node: md["node"], Description: md["description"], Schema: json.RawMessage(md["schema"]),
Interchangeable: md["interchangeable"] == "true", Subject: e.Subject, Queue: e.QueueGroup,
})
}
return out
}
// firstLine is a description's first sentence or line, at most 160 characters.
func firstLine(s string) string {
if i := strings.IndexAny(s, "\n"); i >= 0 {
s = s[:i]
}
if i := strings.Index(s, ". "); i >= 0 {
s = s[:i+1]
}
if len(s) > 160 {
s = s[:157] + "..."
}
return s
}
@@ -0,0 +1,54 @@
package announce
import (
"encoding/json"
"fmt"
"strings"
"testing"
)
// What a machine serves fits the bus's largest message (1 MiB) with room to spare at several hundred
// tools: a module's tool is announced without its schema, which made a machine with ~200 tools outgrow it
// and vanish from discovery (2026-10-04). A seat's verb keeps its schema.
func TestAMachinesAnnouncementFitsTheBusAtSeveralHundredTools(t *testing.T) {
schema := json.RawMessage(`{"type":"object","properties":{` + strings.Repeat(`"field_with_a_long_name":{"type":"string","description":"a description of what this argument means, long enough to matter"},`, 12) + `"last":{"type":"string"}}}`)
description := strings.Repeat("A tool that does something useful, described at the length the catalogue's tools are. ", 3)
var endpoints []Endpoint
for i := 0; i < 400; i++ {
for _, subject := range []string{fmt.Sprintf("mesh.mod.m%d.tool.t", i), fmt.Sprintf("mesh.mod.m%d.tool.t.laptop", i)} {
endpoints = append(endpoints, Endpoint{Kind: KindTool, Module: fmt.Sprintf("m%d", i), Tool: "t", Node: "laptop",
Description: description, Schema: schema, Subject: subject})
}
}
endpoints = append(endpoints, Endpoint{Kind: KindSeat, Module: "m0", Tool: "verb", Seat: "a-seat", Scope: "mesh",
Description: "a verb", Schema: schema, Subject: "mesh.seat.a-seat.tool.verb"})
body, err := json.Marshal(Info(Service{Name: "node-tools", ID: "laptop"}, endpoints))
if err != nil {
t.Fatal(err)
}
if len(body) > 1<<20 {
t.Fatalf("400 tools announce %d bytes, beyond the bus's 1 MiB", len(body))
}
t.Logf("400 tools on two subjects each announce %d bytes", len(body))
back := Endpoints(Info(Service{Name: "node-tools", ID: "laptop"}, endpoints))
if len(back[0].Schema) != 0 {
t.Fatalf("a module's tool still carries its schema: %s", back[0].Schema)
}
if seat := back[len(back)-1]; !strings.Contains(string(seat.Schema), "field_with_a_long_name") {
t.Fatalf("a seat's verb lost its schema: %s", seat.Schema)
}
withSchemas := 0
for range endpoints {
withSchemas += len(schema) * 2 // escaped inside a string
}
t.Logf("with every schema it would have been over %d bytes", len(body)+withSchemas)
}
func TestFirstLineShortensADescription(t *testing.T) {
if got := firstLine("One thing. And more after it."); got != "One thing." {
t.Fatalf("%q", got)
}
if got := firstLine(strings.Repeat("x", 300)); len(got) != 160 {
t.Fatalf("%d", len(got))
}
}
@@ -0,0 +1,126 @@
package announce
import (
"encoding/json"
"sync/atomic"
"testing"
"time"
"github.com/nats-io/nats.go/micro"
)
// novox/hq issue 554: the laptop's console took a round in which not even a PING arrived within the
// 750 ms window as the mesh's answer that nothing is there, and refused every seat and module call. A
// PING that arrives after the window is waited for, up to the patience.
func TestARoundWaitsForAPingThatArrivesAfterTheWindow(t *testing.T) {
was := Patience
Patience = 2500 * time.Millisecond
t.Cleanup(func() { Patience = was })
late := connect(t, "node-tools", "laptop")
s := Service{Name: "node-tools", ID: "laptop", Metadata: map[string]string{"node": "laptop"}}
ping, _ := json.Marshal(micro.Ping{ServiceIdentity: s.identity(), Type: micro.PingResponseType})
stop, err := late.Raw("$SRV.PING", func(string, []byte) []byte { time.Sleep(Window + 300*time.Millisecond); return ping })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
body, _ := json.Marshal(Info(s, []Endpoint{{Kind: KindTool, Module: "slack", Tool: "slack_check", Node: "laptop", Subject: "mesh.mod.slack.tool.slack_check.laptop"}}))
for _, subject := range []string{"$SRV.INFO", "$SRV.INFO.node-tools.laptop"} {
stop, err := late.Raw(subject, func(string, []byte) []byte { return body })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
late.Flush()
d, err := Gather(connect(t, "console", "laptop"), nil)
if err != nil {
t.Fatal(err)
}
if d.Empty() || len(d.Heard) != 1 || d.Heard[0].Machine() != "laptop" {
t.Fatalf("a PING after the window was not waited for: waited %s, pinged %d, heard %+v", d.Waited, d.Pinged, d.Heard)
}
if d.Waited < Window || d.Waited >= Patience {
t.Errorf("waited %s; the PING came after %s and patience is %s", d.Waited, Window+300*time.Millisecond, Patience)
}
}
// A round in which nothing at all arrives is said to be empty, after the patience — not answered as an
// empty mesh at the window.
func TestARoundThatHearsNothingIsEmptyAfterThePatience(t *testing.T) {
was := Patience
Patience = 1500 * time.Millisecond
t.Cleanup(func() { Patience = was })
d, err := Gather(connect(t, "console", "laptop"), nil)
if err != nil {
t.Fatal(err)
}
if !d.Empty() || d.Waited < Patience {
t.Errorf("empty %v after %s (patience %s)", d.Empty(), d.Waited, Patience)
}
}
// What a runtime announces is fetched again only when its PING's digest changed: every round pulled
// some 880 kB of announcements to the laptop, and the PING answers queued behind them (issue 554).
func TestAnUnchangedAnnouncementIsNotFetchedAgain(t *testing.T) {
runtime := connect(t, "node-tools", "desk")
var endpoints atomic.Value
endpoints.Store([]Endpoint{{Kind: KindTool, Module: "alpha", Tool: "one", Node: "desk", Subject: "mesh.mod.alpha.tool.one.desk"}})
stop, err := Serve(runtime, Service{Name: "node-tools", ID: "desk", Metadata: map[string]string{"node": "desk"}},
func() []Endpoint { return endpoints.Load().([]Endpoint) })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
runtime.Flush()
// Every request for what desk serves, counted where it is answered.
var asked atomic.Int32
spy := connect(t, "node-tools", "spy")
for _, subject := range []string{"$SRV.INFO", "$SRV.INFO.node-tools", "$SRV.INFO.node-tools.desk"} {
stop, err := spy.Raw(subject, func(string, []byte) []byte { asked.Add(1); return nil })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
spy.Flush()
console := connect(t, "console", "laptop")
first, err := Gather(console, nil)
if err != nil || len(first.Heard) != 1 || first.Heard[0].Digest == "" || first.Heard[0].Reused {
t.Fatalf("first round: %v %+v", err, first.Heard)
}
if asked.Load() != 1 {
t.Errorf("the first round asked what desk serves %d time(s)", asked.Load())
}
held := first.Heard[0]
known := func(name, id, digest string) (Heard, bool) {
if name == held.Info.Name && id == held.Info.ID && digest == held.Digest {
return held, true
}
return Heard{}, false
}
asked.Store(0)
second, err := Gather(console, known)
if err != nil || len(second.Heard) != 1 || !second.Heard[0].Reused || len(Endpoints(second.Heard[0].Info)) != 1 {
t.Fatalf("an unchanged announcement: %v %+v", err, second.Heard)
}
if asked.Load() != 0 {
t.Errorf("an unchanged announcement was asked for %d time(s)", asked.Load())
}
endpoints.Store([]Endpoint{
{Kind: KindTool, Module: "alpha", Tool: "one", Node: "desk", Subject: "mesh.mod.alpha.tool.one.desk"},
{Kind: KindTool, Module: "alpha", Tool: "two", Node: "desk", Subject: "mesh.mod.alpha.tool.two.desk"},
})
third, err := Gather(console, known)
if err != nil || len(third.Heard) != 1 || third.Heard[0].Reused || len(Endpoints(third.Heard[0].Info)) != 2 ||
third.Heard[0].Digest == held.Digest {
t.Fatalf("a changed announcement was not fetched: %v %+v", err, third.Heard)
}
if asked.Load() != 1 {
t.Errorf("a changed announcement was asked for %d time(s)", asked.Load())
}
}
+160
View File
@@ -0,0 +1,160 @@
package announce
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
func connect(t *testing.T, module, node string) *bus.Conn {
t.Helper()
c, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: module, Node: node})
if err != nil {
t.Fatal(err)
}
c.Logf = func(string, ...any) {}
t.Cleanup(c.Close)
return c
}
// lagging answers PING at once and INFO after `lag` — or never, with a negative lag: a runtime whose
// one large answer is slow to cross the bus, as the laptop's was (2026-10-05).
func lagging(t *testing.T, conn *bus.Conn, s Service, lag time.Duration, endpoints []Endpoint) {
t.Helper()
ping, _ := json.Marshal(micro.Ping{ServiceIdentity: s.identity(), Type: micro.PingResponseType})
stop, err := conn.Raw("$SRV.PING", func(string, []byte) []byte { return ping })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
// Asked of every service, and of this instance by name and id — as discovery asks it (issue 554).
info := func(string, []byte) []byte {
if lag < 0 {
return nil
}
time.Sleep(lag)
body, _ := json.Marshal(Info(s, endpoints))
return body
}
for _, subject := range []string{"$SRV.INFO", "$SRV.INFO." + s.Name + "." + s.ID} {
stop, err := conn.Raw(subject, info)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
conn.Flush()
}
// A runtime whose answer arrives after the window is waited for, because it said it was there; one
// that never says what it serves is named, not dropped. With the fixed 750 ms window the laptop's
// modules were said to run nowhere whenever its answer was late (2026-10-05).
func TestDiscoveryWaitsForEveryRuntimeThatSaidItIsThere(t *testing.T) {
was := Patience
Patience = 2500 * time.Millisecond
t.Cleanup(func() { Patience = was })
fast := connect(t, "node-tools", "desk")
stopFast, err := Serve(fast, Service{Name: "node-tools", ID: "desk", Metadata: map[string]string{"node": "desk"}}, func() []Endpoint {
return []Endpoint{{Kind: KindTool, Module: "alpha", Tool: "one", Node: "desk", Subject: "mesh.mod.alpha.tool.one.desk"}}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopFast)
fast.Flush()
slow := connect(t, "node-tools", "laptop")
lagging(t, slow, Service{Name: "node-tools", ID: "laptop", Metadata: map[string]string{"node": "laptop"}}, Window+400*time.Millisecond,
[]Endpoint{{Kind: KindTool, Module: "slack", Tool: "slack_check", Node: "laptop", Subject: "mesh.mod.slack.tool.slack_check.laptop"}})
mute := connect(t, "node-tools", "sleeper")
lagging(t, mute, Service{Name: "node-tools", ID: "sleeper", Metadata: map[string]string{"node": "sleeper"}}, -1, nil)
asker := connect(t, "console", "desk")
start := time.Now()
d, err := Gather(asker, nil)
if err != nil {
t.Fatal(err)
}
took := time.Since(start)
heard := map[string]Heard{}
for _, h := range d.Heard {
heard[h.Info.ID] = h
}
if h, ok := heard["laptop"]; !ok {
t.Fatalf("the slow runtime was not heard: %+v", d.Heard)
} else if h.Took <= Window || h.Machine() != "laptop" || h.Bytes == 0 {
t.Errorf("the slow runtime's answer: took %s (window %s), machine %q, %d bytes", h.Took, Window, h.Machine(), h.Bytes)
}
if _, ok := heard["desk"]; !ok {
t.Errorf("the fast runtime was not heard: %+v", d.Heard)
}
if len(d.Silent) != 1 || d.Silent[0].ID != "sleeper" || d.Silent[0].Machine != "sleeper" {
t.Errorf("the runtime that never said what it serves is not named: %+v", d.Silent)
}
if took < Patience || took > Patience+time.Second {
t.Errorf("waited %s with a silent runtime; patience is %s", took, Patience)
}
}
// With nobody slow, discovery takes the window and no longer.
func TestDiscoveryDoesNotWaitWhenEveryoneAnswered(t *testing.T) {
fast := connect(t, "node-tools", "desk")
stop, err := Serve(fast, Service{Name: "node-tools", ID: "desk"}, func() []Endpoint { return nil })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
fast.Flush()
start := time.Now()
d, err := Gather(connect(t, "console", "desk"), nil)
if err != nil {
t.Fatal(err)
}
if took := time.Since(start); took > Window+500*time.Millisecond || len(d.Silent) != 0 || len(d.Heard) == 0 {
t.Errorf("took %s, heard %d, silent %+v", took, len(d.Heard), d.Silent)
}
}
// Too large even with first lines, descriptions are left out — and the answer says which, without
// touching the service's own metadata.
func TestAnAnswerTooLargeIsShortenedUntilItFits(t *testing.T) {
long := strings.Repeat("A sentence that is long. ", 20)
var endpoints []Endpoint
for i := 0; i < 50; i++ {
endpoints = append(endpoints, Endpoint{Kind: KindTool, Module: "m", Tool: "t" + strings.Repeat("x", i), Node: "laptop",
Description: long, Subject: "mesh.mod.m.tool.t"})
}
own := map[string]string{"node": "laptop"}
info := Info(Service{Name: "node-tools", ID: "laptop", Metadata: own}, endpoints)
full, _ := json.Marshal(info)
firstLines := 0
for range endpoints {
firstLines += len(firstLine(long))
}
limit := int64(len(full) - len(long)*len(endpoints) + firstLines/2) // first lines alone do not fit
var said []string
got := shorten(func(f string, a ...any) { said = append(said, f) }, info, full, limit)
if int64(len(got)) > limit {
t.Fatalf("still %d bytes, beyond %d", len(got), limit)
}
var back micro.Info
if err := json.Unmarshal(got, &back); err != nil {
t.Fatal(err)
}
if back.Metadata[Shortened] != "descriptions left out" || back.Metadata["node"] != "laptop" {
t.Errorf("metadata: %v", back.Metadata)
}
if _, has := own[Shortened]; has {
t.Error("the service's own metadata was changed")
}
if len(said) != 2 {
t.Errorf("each step is said: %v", said)
}
}
File diff suppressed because it is too large Load Diff
+62
View File
@@ -0,0 +1,62 @@
package bus
import (
"errors"
"fmt"
"testing"
)
func TestSubjectsAreTheOnesTheTypeScriptRuntimeUses(t *testing.T) {
for key, want := range map[string]string{
"status": "mesh.mod.self.tool.status",
"alpha.one": "mesh.mod.alpha.tool.one",
"alpha.one@anchor": "mesh.mod.alpha.tool.one.anchor",
"seat:node-shelf.list": "mesh.seat.node-shelf.tool.list",
"seat:node-shelf.list@anchor": "mesh.seat.node-shelf.tool.list.anchor",
} {
if got, err := ToolSubject(key, "self"); err != nil || got != want {
t.Errorf("%s: %s %v, want %s", key, got, err, want)
}
}
if _, err := ToolSubject("seat:nothing", "self"); err == nil {
t.Error("a seat with no verb was accepted")
}
if SeatToolSubject("s", "v", "node", "n") != "mesh.seat.s.tool.v.n" || SeatToolSubject("s", "v", "mesh", "n") != "mesh.seat.s.tool.v" {
t.Error("seat subjects")
}
}
func TestAFingerprintIsReadHoweverItIsWritten(t *testing.T) {
for _, f := range []string{"sha256:AB:CD:ef", "abcdef", "ABCDEF", "SHA256:abcdef"} {
if got := normalizeFingerprint(f); got != "abcdef" {
t.Errorf("%s → %s", f, got)
}
}
}
func TestWhatWaitingCannotFixIsFinal(t *testing.T) {
for err, want := range map[error]string{
fmt.Errorf("x509: %w: it presented aa", ErrPin): "the bus's certificate does not match the pin",
errors.New("nats: Authorization Violation"): "the bus refused this account",
errors.New("nats: invalid url"): "the bus address is not a usable URL",
errors.New("dial tcp 10.0.0.1:4222: connect: connection refused"): "",
} {
if got := Fatal(err); got != want {
t.Errorf("%v → %q, want %q", err, got, want)
}
}
}
// An event given up on and delivered again by the controller arrives under `mesh.again.<consumer>.` and
// the event's own subject without its `mesh.` (novox/hq issue 330): the handler sees the key it saw the
// first time, for a module's event and a seat's.
func TestAnEventDeliveredAgainHasItsOwnKey(t *testing.T) {
for original, again := range map[string]string{
"mesh.mod.gitea.event.pull.merged": "mesh.again.ace_sonarr.mod.gitea.event.pull.merged",
"mesh.seat.node-build-agent.event.built": "mesh.again.ace_sonarr.seat.node-build-agent.event.built",
} {
if keyOf(again) != keyOf(original) {
t.Errorf("%s reads as %q, the event as %q", again, keyOf(again), keyOf(original))
}
}
}
+212
View File
@@ -0,0 +1,212 @@
package bus
import (
"context"
"encoding/json"
"errors"
"regexp"
"strings"
"sync"
"github.com/nats-io/nats.go"
)
// A tool call names its caller (novox/hq issue 365, by ADR 0259 §3's precedent for asks).
//
// mesh.mod.<module>.call.<tool>.<caller> to whichever instance answers
// mesh.mod.<module>.call.<tool>.<node>.<caller> to the instance on one machine
// mesh.seat.<seat>.call.<verb>.<caller> a mesh seat's verb
// mesh.seat.<seat>.call.<verb>.<node>.<caller> a node seat's verb on one machine
//
// The last token is the bus user that published the call, and the controller grants each user the publish of
// these subjects with its own name there and no other — so the caller is a fact the bus server enforces, and
// the runtime hands it to the module from the subject a call arrived on, never from the payload.
//
// **A kind of its own, `call`, not the `tool` subject with a token appended.** Every grant to call a tool is a
// wildcard over the `tool` kind — `mesh.mod.<m>.tool.<t>.*` for the instance on any machine, `mesh.mod.*.tool.>`
// for every tool — and either matches a `tool` subject with any caller appended, so a caller could name
// another. Under `call` no grant but the caller-named ones reaches the caller token.
//
// The `tool` subjects are served beside these for one release, and say Unproven, so callers move without a
// gap; their retirement is hq issue 464.
const CallKind = "call"
// A bus user's name carries dots (`<node>.<module>`, `person.<name>`); a subject token cannot. In the token
// each dot is written `~`, which no part of a bus user's name may hold (the controller refuses any part that
// is not letters, digits, `_` or `-`), so the token is the user's name and nothing else.
const userDot = "~"
var userName = regexp.MustCompile(`^[A-Za-z0-9_-]+(\.[A-Za-z0-9_-]+)*$`)
// CallerToken is a bus user's name as the last token of a call it publishes, or "" for a name that is not a
// bus user's.
func CallerToken(user string) string {
if !userName.MatchString(user) {
return ""
}
return strings.ReplaceAll(user, ".", userDot)
}
// UserOfToken is the bus user a caller token names, or "" when it names none.
func UserOfToken(token string) string {
user := strings.ReplaceAll(token, userDot, ".")
if !userName.MatchString(user) {
return ""
}
return user
}
// toolShape splits a tool subject — `mesh.mod.<m>.tool.<rest>` or `mesh.seat.<s>.tool.<rest>` — into what
// comes before the kind and what after; ok is false for any other subject.
func toolShape(subject string) (head, rest string, ok bool) {
parts := strings.SplitN(subject, ".", 5)
if len(parts) != 5 || parts[0] != "mesh" || (parts[1] != "mod" && parts[1] != "seat") || parts[3] != "tool" ||
parts[2] == "" || parts[4] == "" {
return "", "", false
}
return parts[0] + "." + parts[1] + "." + parts[2], parts[4], true
}
// CalledSubject is where a call to a tool subject goes naming its caller: the same address under the `call`
// kind, the caller's token last. "" for a subject that is not a tool's, or a caller that is not a bus user.
func CalledSubject(subject, user string) string {
head, rest, ok := toolShape(subject)
token := CallerToken(user)
if !ok || token == "" || strings.ContainsAny(rest, "*>") {
return ""
}
return head + "." + CallKind + "." + rest + "." + token
}
// calledPattern is what a runtime answering a tool subject also subscribes, to hear the calls that name their
// caller: the same address under the `call` kind, with any caller. "" for a subject that is not a tool's.
func calledPattern(subject string) string {
head, rest, ok := toolShape(subject)
if !ok || strings.ContainsAny(rest, "*>") {
return ""
}
return head + "." + CallKind + "." + rest + ".*"
}
// callerOn is the bus user a call names, from the subject it arrived on: its last token, under a pattern
// calledPattern made. "" — said by the module as Unproven — for a call on the subject that names no caller.
func callerOn(pattern, subject string) string {
// Only the `call` kind names a caller: a `tool` pattern ending in a wildcard (a node seat's verb on any
// machine) is never read as one.
parts := strings.SplitN(pattern, ".", 5)
if len(parts) != 5 || parts[0] != "mesh" || (parts[1] != "mod" && parts[1] != "seat") || parts[3] != CallKind ||
!strings.HasSuffix(pattern, ".*") {
return ""
}
dot := strings.LastIndex(subject, ".")
if dot < 0 || subject[:dot] != strings.TrimSuffix(pattern, ".*") {
return ""
}
return UserOfToken(subject[dot+1:])
}
// CalledHandler answers one call, told who called: the bus user the subject named, or "" when it named none.
type CalledHandler func(caller string, body json.RawMessage) (any, error)
// callers is what a connection keeps of its caller-named calls: those in flight, by subject, so a refused
// publish ends its call at once instead of after RequestTimeout.
type callers struct {
mu sync.Mutex
// fellBack are the caller-named subjects already said to have reached nobody, so each is said once.
fellBack map[string]bool
next uint64
pending map[string]map[uint64]context.CancelCauseFunc
}
// errCallRefused is the bus refusing a caller-named publish: this user is not granted it yet (the controller's
// grant not delivered to the bus), so the call is asked again on the subject that names no caller.
var errCallRefused = errors.New("the bus refused the caller-named subject")
var refusedPublish = regexp.MustCompile(`(?i)Publish to "(\S+)"`)
// publishRefused ends every call in flight on a subject the bus refused this connection the publish of.
func (c *Conn) publishRefused(err error) bool {
if !errors.Is(err, nats.ErrPermissionViolation) {
return false
}
m := refusedPublish.FindStringSubmatch(err.Error())
if len(m) < 2 {
return false
}
c.calls.mu.Lock()
cancels := c.calls.pending[m[1]]
delete(c.calls.pending, m[1])
c.calls.mu.Unlock()
if len(cancels) > 0 {
c.Logf("[mesh-tools] the bus refused this account the caller-named %s; asked again on the subject that "+
"names no caller, until the mesh grants it (novox/hq issue 365)", m[1])
}
for _, cancel := range cancels {
cancel(errCallRefused)
}
return len(cancels) > 0
}
// askCalled asks once on a caller-named subject. fallBack says the call reached nobody — nothing serves that
// subject yet, or the bus refused this user its publish — so it may be asked on the subject that names no
// caller without being done twice.
func (c *Conn) askCalled(subject string, data []byte) (answered Answered, handingOver, fallBack bool, err error) {
ctx, cancel := context.WithCancelCause(context.Background())
ctx, stop := context.WithTimeout(ctx, RequestTimeout)
defer stop()
c.calls.mu.Lock()
if c.calls.pending == nil {
c.calls.pending = map[string]map[uint64]context.CancelCauseFunc{}
}
if c.calls.pending[subject] == nil {
c.calls.pending[subject] = map[uint64]context.CancelCauseFunc{}
}
c.calls.next++
id := c.calls.next
c.calls.pending[subject][id] = cancel
c.calls.mu.Unlock()
defer func() {
c.calls.mu.Lock()
if list := c.calls.pending[subject]; list != nil {
delete(list, id)
if len(list) == 0 {
delete(c.calls.pending, subject)
}
}
c.calls.mu.Unlock()
cancel(nil)
}()
msg, err := c.nc.RequestWithContext(ctx, subject, data)
if err != nil {
switch {
case errors.Is(context.Cause(ctx), errCallRefused):
return Answered{}, false, true, nil
case errors.Is(err, nats.ErrNoResponders):
c.saidFallBack(subject)
return Answered{}, false, true, nil
case errors.Is(err, context.DeadlineExceeded) || errors.Is(err, nats.ErrTimeout):
return Answered{}, false, false, errors.New("timeout")
}
return Answered{}, false, false, err
}
answered, handingOver, err = c.answerOf(subject, msg.Data)
return answered, handingOver, false, err
}
// saidFallBack says, once per subject, that a caller-named call reached nobody and is asked on the subject that
// names no caller: a runtime or holder not yet serving it, or a grant that drifted, is seen in the log rather
// than read later as a quiet "unproven" (novox/hq issue 365; expected for the controller's seat until hq issue
// 464).
func (c *Conn) saidFallBack(subject string) {
c.calls.mu.Lock()
if c.calls.fellBack == nil {
c.calls.fellBack = map[string]bool{}
}
first := !c.calls.fellBack[subject]
c.calls.fellBack[subject] = true
c.calls.mu.Unlock()
if first {
c.Logf("[mesh-tools] nothing answers the caller-named %s; asked on the subject that names no caller, where "+
"the module is told no caller (said once per subject; novox/hq issue 365)", subject)
}
}
+218
View File
@@ -0,0 +1,218 @@
package bus_test
import (
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// A bus whose users are granted as the controller grants them (mesh-controller internal/broker): each named
// user publishes the subjects listed for it.
func grantedBus(t *testing.T, publish map[string][]string) string {
t.Helper()
var users []*server.User
for user, allow := range publish {
// And the membership's direct get every connection asks once, answered "no responders" here.
allow = append(allow, "$JS.API.DIRECT.GET.>")
users = append(users, &server.User{Username: user, Password: "pw", Permissions: &server.Permissions{
Publish: &server.SubjectPermission{Allow: allow},
Subscribe: &server.SubjectPermission{Allow: []string{">"}},
}})
}
s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, NoLog: true, NoSigs: true,
Users: users})
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(s.Shutdown)
return s.ClientURL()
}
func dial(t *testing.T, url, module, node, user string) *bus.Conn {
t.Helper()
c, err := bus.Connect(bus.Credential{URL: url, Module: module, Node: node, User: user, Password: "pw"})
if err != nil {
t.Fatal(err)
}
c.Logf = func(string, ...any) {}
t.Cleanup(c.Close)
return c
}
type heard struct {
mu sync.Mutex
callers []string
}
func (h *heard) handler(caller string, _ json.RawMessage) (any, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.callers = append(h.callers, caller)
return map[string]any{"caller": caller}, nil
}
func (h *heard) all() []string {
h.mu.Lock()
defer h.mu.Unlock()
return append([]string(nil), h.callers...)
}
// A caller asks on the caller-named subject first; where nothing serves it yet — a runtime from before issue
// 365 — the same call is asked once on the subject that names no caller, at once, and done once.
func TestACallerFallsBackAtOnceWhereNothingServesTheCallerNamedSubject(t *testing.T) {
url := mt.URL(t)
server := dial(t, url, "ledger", "anchor", "")
h := &heard{}
// An older runtime: the subject that names no caller, alone.
stop, err := server.HandleSubject("mesh.mod.ledger.tool.who", func(body json.RawMessage) (any, error) {
return h.handler("", body)
})
if err != nil {
t.Fatal(err)
}
defer stop()
server.Flush()
asker := dial(t, url, "node-tools", "shanks", "shanks.node-tools")
began := time.Now()
answered, err := asker.Ask("ledger.who", map[string]any{}, "")
if err != nil {
t.Fatal(err)
}
if took := time.Since(began); took > 2*time.Second {
t.Errorf("the fall-back took %s", took)
}
if got := h.all(); len(got) != 1 || got[0] != "" || !strings.Contains(string(answered.Result), `"caller":""`) {
t.Errorf("the call was answered %v, heard %v; want once, on the old subject", string(answered.Result), got)
}
}
// Where the bus does not grant the caller-named subject yet — the controller's grants not yet delivered — the
// refusal ends the call at once and it is asked on the subject that names no caller: no caller waits out
// RequestTimeout for a grant that is on its way.
func TestACallerTheBusRefusesTheCallerNamedSubjectFallsBackAtOnce(t *testing.T) {
url := grantedBus(t, map[string][]string{
"anchor.node-tools": {"_INBOX.>"},
"shanks.node-tools": {"mesh.mod.*.tool.>"},
})
server := dial(t, url, "node-tools", "anchor", "anchor.node-tools")
h := &heard{}
stop, err := server.HandleSubjectCalled("mesh.mod.ledger.tool.who", h.handler)
if err != nil {
t.Fatal(err)
}
defer stop()
server.Flush()
asker := dial(t, url, "node-tools", "shanks", "shanks.node-tools")
var said []string
var mu sync.Mutex
asker.Logf = func(f string, a ...any) { mu.Lock(); said = append(said, fmt.Sprintf(f, a...)); mu.Unlock() }
began := time.Now()
if _, err := asker.Ask("ledger.who", map[string]any{}, ""); err != nil {
t.Fatal(err)
}
mu.Lock()
if !strings.Contains(strings.Join(said, "\n"), "the bus refused this account the caller-named mesh.mod.ledger.call.who.shanks~node-tools") {
t.Errorf("the refusal was not said: %v", said)
}
mu.Unlock()
if took := time.Since(began); took > 3*time.Second {
t.Errorf("the refused caller-named call held the caller %s", took)
}
if got := h.all(); len(got) != 1 || got[0] != "" {
t.Errorf("heard %v; want one call, on the subject that names no caller", got)
}
}
// With the controller's grant — the caller-named subjects with its own name last — a caller is heard as itself;
// and a caller cannot publish on a subject naming another, however it asks.
func TestAGrantedCallerIsHeardAsItselfAndCannotNameAnother(t *testing.T) {
url := grantedBus(t, map[string][]string{
"anchor.node-tools": {"_INBOX.>"},
"shanks.node-tools": {"mesh.mod.*.call.*.shanks~node-tools", "mesh.mod.*.call.*.*.shanks~node-tools"},
})
server := dial(t, url, "node-tools", "anchor", "anchor.node-tools")
h := &heard{}
stop, err := server.HandleSubjectCalled("mesh.mod.ledger.tool.who", h.handler)
if err != nil {
t.Fatal(err)
}
defer stop()
server.Flush()
asker := dial(t, url, "node-tools", "shanks", "shanks.node-tools")
if _, err := asker.Ask("ledger.who", map[string]any{}, ""); err != nil {
t.Fatal(err)
}
// Naming another on the subject itself: refused by the bus, so nothing is heard.
refusals := make(chan error, 4)
forger, err := nats.Connect(url, nats.UserInfo("shanks.node-tools", "pw"),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refusals <- err }))
if err != nil {
t.Fatal(err)
}
defer forger.Close()
if err := forger.Publish("mesh.mod.ledger.call.who.novox~node-tools", []byte(`{}`)); err != nil {
t.Fatal(err)
}
_ = forger.Flush()
select {
case err := <-refusals:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("the bus said %v, want a permissions violation", err)
}
case <-time.After(3 * time.Second):
t.Error("the bus did not refuse a call naming another caller")
}
if got := h.all(); len(got) != 1 || got[0] != "shanks.node-tools" {
t.Errorf("heard %v; want the asker alone, as itself", got)
}
}
// A fall-back for want of a responder on the caller-named subject is said, once per subject, so a runtime or
// grant that drifted behind is seen rather than read as a quiet "unproven".
func TestAFallBackForWantOfAResponderIsSaidOncePerSubject(t *testing.T) {
url := mt.URL(t)
// A runtime from before issue 365: the subject that names no caller, alone.
older, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer older.Close()
if _, err := older.Subscribe("mesh.mod.ledger.tool.who", func(m *nats.Msg) { _ = m.Respond([]byte(`{"result":{}}`)) }); err != nil {
t.Fatal(err)
}
_ = older.Flush()
asker := dial(t, url, "node-tools", "shanks", "shanks.node-tools")
var mu sync.Mutex
var said []string
asker.Logf = func(f string, a ...any) { mu.Lock(); said = append(said, fmt.Sprintf(f, a...)); mu.Unlock() }
for i := 0; i < 3; i++ {
if _, err := asker.Ask("ledger.who", map[string]any{}, ""); err != nil {
t.Fatal(err)
}
}
mu.Lock()
defer mu.Unlock()
n := 0
for _, s := range said {
if strings.Contains(s, "nothing answers the caller-named mesh.mod.ledger.call.who.shanks~node-tools") {
n++
}
}
if n != 1 {
t.Errorf("the fall-back was said %d times, want once: %v", n, said)
}
}
+72
View File
@@ -0,0 +1,72 @@
package bus
import "testing"
// A call names its caller under the `call` kind, the caller's bus user its last token (novox/hq issue 365).
func TestACallerNamedSubjectIsTheToolsAddressUnderCallWithTheCallerLast(t *testing.T) {
for subject, want := range map[string]string{
"mesh.mod.ledger.tool.who": "mesh.mod.ledger.call.who.shanks~node-tools",
"mesh.mod.ledger.tool.who.anchor": "mesh.mod.ledger.call.who.anchor.shanks~node-tools",
"mesh.seat.issue-tracker.tool.open": "mesh.seat.issue-tracker.call.open.shanks~node-tools",
"mesh.seat.node-desk.tool.who.anchor": "mesh.seat.node-desk.call.who.anchor.shanks~node-tools",
"mesh.mod.ledger.event.opened": "",
"mesh.seat.intake.accept.ask.x": "",
"mesh.mod.ledger.tool.>": "",
"mesh.mod.*.tool.who": "mesh.mod.*.call.who.shanks~node-tools",
"mesh.node.anchor.ask.report": "",
"mesh.seat.operator-channel.proof.x.y": "",
} {
if subject == "mesh.mod.*.tool.who" {
continue // a wildcard is never asked on
}
if got := CalledSubject(subject, "shanks.node-tools"); got != want {
t.Errorf("%s → %q, want %q", subject, got, want)
}
}
for _, user := range []string{"", "a b", "x.*", "x.>", "a~b", "a..b"} {
if got := CalledSubject("mesh.mod.ledger.tool.who", user); got != "" {
t.Errorf("a caller %q that is not a bus user was named: %s", user, got)
}
}
}
// The caller is read back from the subject a call arrived on, and only under the pattern that names one.
func TestTheCallerIsTheLastTokenOfTheSubjectTheCallArrivedOn(t *testing.T) {
pattern := calledPattern("mesh.mod.ledger.tool.who.anchor")
if pattern != "mesh.mod.ledger.call.who.anchor.*" {
t.Fatalf("pattern %q", pattern)
}
for subject, want := range map[string]string{
"mesh.mod.ledger.call.who.anchor.shanks~node-tools": "shanks.node-tools",
"mesh.mod.ledger.call.who.anchor.controller": "controller",
"mesh.mod.ledger.call.who.anchor.person~jochen": "person.jochen",
"mesh.mod.ledger.call.who.anchor.a~~b": "",
"mesh.mod.ledger.call.what.anchor.controller": "",
} {
if got := callerOn(pattern, subject); got != want {
t.Errorf("%s → %q, want %q", subject, got, want)
}
}
if got := callerOn("mesh.mod.ledger.tool.who.anchor", "mesh.mod.ledger.tool.who.anchor"); got != "" {
t.Errorf("a call on the subject that names no caller was handed %q", got)
}
for _, user := range []string{"controller", "shanks.node-tools", "person.jochen", "node.ace"} {
if back := UserOfToken(CallerToken(user)); back != user {
t.Errorf("%s → %s → %s", user, CallerToken(user), back)
}
}
}
// Only a pattern of the `call` kind names a caller: a `tool` subscription with a wildcard last — a node seat's
// verb on any machine, say — never yields one, whatever token a call arrived with.
func TestAToolPatternNeverYieldsACaller(t *testing.T) {
for pattern, subject := range map[string]string{
"mesh.mod.ledger.tool.who.*": "mesh.mod.ledger.tool.who.controller",
"mesh.seat.node-desk.tool.who.*": "mesh.seat.node-desk.tool.who.shanks~node-tools",
"mesh.mod.ledger.event.*": "mesh.mod.ledger.event.controller",
} {
if got := callerOn(pattern, subject); got != "" {
t.Errorf("%s on %s yielded caller %q", pattern, subject, got)
}
}
}
+62
View File
@@ -0,0 +1,62 @@
package bus_test
import (
"encoding/json"
"fmt"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq issue 387: an event a bundle handles for longer than the consumer's ack wait is kept alive while
// it is handled, and the events behind it wait on the bus rather than in the client, so a slow handler and a
// burst are each handed over once — never offered again with nothing failed, never toward max-deliver.
func TestASlowHandlerAndABurstAreEachHandedOverOnce(t *testing.T) {
mesh := mt.New(t)
restore := bus.EventHeartbeat
bus.EventHeartbeat = 300 * time.Millisecond
t.Cleanup(func() { bus.EventHeartbeat = restore })
mesh.Consumer(t, "anchor", "slow", []string{"mesh.mod.alpha.event.>"}, time.Second)
c, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
c.Logf = func(string, ...any) {}
t.Cleanup(c.Close)
var mu sync.Mutex
handled := map[int]int{}
stop, err := c.ConsumeAs("slow", func(env bus.Envelope) error {
var body struct{ N int }
_ = json.Unmarshal(env.Body, &body)
time.Sleep(1500 * time.Millisecond) // longer than the ack wait
mu.Lock()
handled[body.N]++
mu.Unlock()
return nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
for n := 1; n <= 3; n++ {
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": n})
}
mt.Until(t, func() error {
ack, notYet := mesh.Pending(t, "anchor", "slow")
if ack != 0 || notYet != 0 {
return fmt.Errorf("still pending: %d unacknowledged, %d undelivered", ack, notYet)
}
return nil
})
mu.Lock()
defer mu.Unlock()
for n := 1; n <= 3; n++ {
if handled[n] != 1 {
t.Errorf("event %d was handed over %d times: %v", n, handled[n], handled)
}
}
}
+73
View File
@@ -0,0 +1,73 @@
package bus_test
import (
"encoding/json"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// **A call the controller refused because it is handing over is asked once more** (novox/hq issue 289):
// the controller being replaced did nothing and marked its refusal, and the one after it answers. A
// refusal not so marked is final, and a second handover refusal is said, not asked a third time.
func TestAHandoverRefusalIsAskedOnceMore(t *testing.T) {
was := bus.HandoverPause
bus.HandoverPause = 10 * time.Millisecond
t.Cleanup(func() { bus.HandoverPause = was })
url := meshtest.URL(t)
seat, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer seat.Close()
c, err := bus.Connect(bus.Credential{URL: url, Module: "console", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
defer c.Close()
serve := func(t *testing.T, verb string, answers ...map[string]any) *atomic.Int32 {
t.Helper()
var asked atomic.Int32
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", verb, "", ""), func(m *nats.Msg) {
n := int(asked.Add(1)) - 1
if n >= len(answers) {
n = len(answers) - 1
}
body, _ := json.Marshal(answers[n])
_ = m.Respond(body)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
_ = seat.Flush()
return &asked
}
handingOver := map[string]any{"error": "the controller is handing over to the next one; nothing was done, ask again",
"retry": bus.RetryHandingOver}
asked := serve(t, "rotate", handingOver, map[string]any{"result": "rotated"})
got, err := c.Ask("seat:mesh-controller.rotate", map[string]any{}, "")
if err != nil || string(got.Result) != `"rotated"` || asked.Load() != 2 {
t.Fatalf("after a handover refusal: %s %v, asked %d times", got.Result, err, asked.Load())
}
asked = serve(t, "push", map[string]any{"error": "refused for its own reason"})
if _, err := c.Ask("seat:mesh-controller.push", map[string]any{}, ""); err == nil || asked.Load() != 1 {
t.Fatalf("an unmarked refusal was asked again: %v, asked %d times", err, asked.Load())
}
asked = serve(t, "status", handingOver)
_, err = c.Ask("seat:mesh-controller.status", map[string]any{}, "")
if err == nil || asked.Load() != 2 || !strings.Contains(err.Error(), "both times refused as a handover") {
t.Fatalf("two handover refusals: %v, asked %d times", err, asked.Load())
}
}
+367
View File
@@ -0,0 +1,367 @@
package bus
import (
"encoding/json"
"errors"
"fmt"
"regexp"
"sort"
"strconv"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// A module's log on the bus (novox/hq ADR 0297): a stream of operations, one subject per key, that the
// controller creates from what the module declared under `logs` and issues to the module's own instances
// in its membership. It mirrors a bucket (ADR 0201) in how it is reached: what a module may reach is
// decided here, from its membership, and refused with the reason before anything is sent.
//
// **The runtime uses no consumer of a log.** The bus grants it, per issued log, only the publish of an
// entry, the stream's info and the direct get: an append is a publish, a read is a batch direct get
// (NATS 2.11), the last entry of a key is a direct get by its subject.
// LogIssued is one log an assignment may reach (ADR 0297): its owner's alone, so there are no reads.
type LogIssued struct {
Name string `json:"name"`
Stream string `json:"stream"`
Subject string `json:"subject"`
Writes bool `json:"writes,omitempty"`
}
// LogEntry is one operation in a log: its sequence is its identity.
type LogEntry struct {
Sequence uint64 `json:"sequence"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
At time.Time `json:"at"`
}
// LogPage is one page of a read: Next is the sequence to continue from, 0 when nothing follows; Last
// is the stream's last sequence.
type LogPage struct {
Entries []LogEntry `json:"entries"`
Next uint64 `json:"next"`
Last uint64 `json:"last"`
}
// LogInfo is how full a log is.
type LogInfo struct {
Entries uint64 `json:"entries"`
Bytes uint64 `json:"bytes"`
MaxBytes uint64 `json:"max_bytes"`
First uint64 `json:"first"`
Last uint64 `json:"last"`
}
const (
// LogValueMax is the most one entry's value may hold (ADR 0297 §3); the stream's MaxMsgSize is 4 KiB
// more, for the headers an append carries.
LogValueMax = 256 * 1024
// LogKeyMax is the longest key.
LogKeyMax = 255
// LogReadDefault and LogReadMax are a page's entries when none is asked, and at most.
LogReadDefault = 100
LogReadMax = 1000
// LogPageBytes bounds the bytes of one page.
LogPageBytes = 4 * 1024 * 1024
)
// logToken is one name of a key: letters, digits, `_` and `-`.
var logToken = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
func checkLogKey(key string) error {
if key == "" || len(key) > LogKeyMax {
return fmt.Errorf("%q is not a log key: one or more names of letters, digits, _ and -, joined by "+
"dots, at most %d bytes (novox/hq ADR 0297)", key, LogKeyMax)
}
for _, t := range strings.Split(key, ".") {
if !logToken.MatchString(t) {
return fmt.Errorf("%q is not a log key: one or more names of letters, digits, _ and -, joined by "+
"dots, at most %d bytes (novox/hq ADR 0297)", key, LogKeyMax)
}
}
return nil
}
// logFilter is a read's key pattern as a subject filter under the log's subject: `*` for one name, `>`
// for the rest; empty is every key.
func logFilter(l LogIssued, pattern string) (string, error) {
if pattern == "" {
return l.Subject + ".>", nil
}
parts := strings.Split(pattern, ".")
for i, p := range parts {
switch {
case p == ">" && i == len(parts)-1, p == "*", logToken.MatchString(p):
default:
return "", fmt.Errorf("%q is not a key pattern: names, `*` for one and `>` for the rest, last "+
"(novox/hq ADR 0297)", pattern)
}
}
if len(pattern) > LogKeyMax {
return "", fmt.Errorf("%q is longer than a key can be (%d bytes)", pattern, LogKeyMax)
}
return l.Subject + "." + pattern, nil
}
// issuedLog is the log a module may reach by a name, or the reason it may not.
func (c *Conn) issuedLog(module, name string) (LogIssued, error) {
m := c.Membership(module)
if m == nil {
return LogIssued{}, fmt.Errorf("%s has no membership issued on %s yet, so no log of it is reachable "+
"until the mesh issues one (novox/hq ADR 0297)", module, c.node)
}
var names []string
for _, l := range m.Logs {
if l.Name == name {
return l, nil
}
names = append(names, l.Name)
}
sort.Strings(names)
issued := "none"
if len(names) > 0 {
issued = strings.Join(names, ", ")
}
return LogIssued{}, fmt.Errorf("%s keeps no log called %q: a module declares the logs it keeps under `logs`, "+
"and a log is its owner's alone; it was issued: %s (novox/hq ADR 0297)", module, name, issued)
}
// notOnTheBus says a log is issued and its stream is not there yet.
func notOnTheBus(l LogIssued) error {
return fmt.Errorf("the log %q is issued and its stream %s is not on the bus yet: the controller creates "+
"it from the catalogue on its next raise (novox/hq ADR 0297)", l.Name, l.Stream)
}
// LogAppend writes one entry for a key, as the module, and answers its sequence. With expect, it is
// written only when the key's last entry is still at that sequence (0: the key has none); otherwise it
// is refused as a lost compare-and-set, the same error a lost state update answers (ADR 0259).
func (c *Conn) LogAppend(module, name, key string, value json.RawMessage, expect *uint64) (uint64, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return 0, err
}
if !l.Writes {
return 0, fmt.Errorf("%s was issued the log %s without writing it (novox/hq ADR 0297)", module, name)
}
if err := checkLogKey(key); err != nil {
return 0, err
}
if len(value) == 0 || !json.Valid(value) {
return 0, fmt.Errorf("a log entry's value is JSON (novox/hq ADR 0297)")
}
if len(value) > LogValueMax {
return 0, fmt.Errorf("a log entry's value is at most %d bytes (256 KiB), and this one is %d (novox/hq ADR 0297)",
LogValueMax, len(value))
}
if field := credentialField(value); field != "" {
return 0, fmt.Errorf("%s's %s.%s carries a field %q, which names a credential: no secret is kept in a log, "+
"which keeps it for good. Name the secret and fetch it on request/reply (novox/hq ADR 0201 §13, ADR 0297)",
module, name, key, field)
}
msg := nats.NewMsg(l.Subject + "." + key)
msg.Data = value
if expect != nil {
msg.Header.Set(jetstream.ExpectedLastSubjSeqHeader, strconv.FormatUint(*expect, 10))
}
reply, err := c.nc.RequestMsg(msg, StateTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return 0, notOnTheBus(l)
}
if err != nil {
return 0, err
}
var ack struct {
Seq uint64 `json:"seq"`
Error *struct {
Code int `json:"code"`
ErrCode uint16 `json:"err_code"`
Description string `json:"description"`
} `json:"error"`
}
if err := json.Unmarshal(reply.Data, &ack); err != nil {
return 0, fmt.Errorf("the bus answered the append to %s with what is not an acknowledgement: %q", name, reply.Data)
}
if ack.Error != nil {
// The key's last entry is no longer the one expected: a lost compare-and-set, answered as a lost
// state update is (10164 is the same refusal from a replicated stream).
if code := jetstream.ErrorCode(ack.Error.ErrCode); code == jetstream.JSErrCodeStreamWrongLastSequence ||
code == jetstream.JSErrCodeStreamWrongLastSequenceConstant {
return 0, changed{name + "." + key}
}
return 0, fmt.Errorf("the bus refused the append to %s.%s: %s (%d)", name, key, ack.Error.Description, ack.Error.ErrCode)
}
if ack.Seq == 0 {
return 0, fmt.Errorf("the bus answered the append to %s.%s with no sequence", name, key)
}
return ack.Seq, nil
}
// LogRead is the entries from a sequence on, of the keys matching a pattern, at most limit of them and
// about 4 MiB, by one batch direct get.
func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int) (*LogPage, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
switch {
case limit == 0:
limit = LogReadDefault
case limit < 0 || limit > LogReadMax:
return nil, fmt.Errorf("a read's limit is 1 to %d entries, %d when none is given, not %d (novox/hq ADR 0297)",
LogReadMax, LogReadDefault, limit)
}
filter, err := logFilter(l, keys)
if err != nil {
return nil, err
}
if from == 0 {
from = 1
}
req, _ := json.Marshal(map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes})
inbox := c.nc.NewInbox()
sub, err := c.nc.SubscribeSync(inbox)
if err != nil {
return nil, err
}
defer func() { _ = sub.Unsubscribe() }()
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+l.Stream, inbox, req); err != nil {
return nil, err
}
page := &LogPage{Entries: []LogEntry{}}
deadline := time.Now().Add(StateTimeout)
for done := false; !done; {
msg, err := sub.NextMsg(time.Until(deadline))
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return nil, notOnTheBus(l)
}
if errors.Is(err, nats.ErrTimeout) {
return nil, fmt.Errorf("the read of %s did not end in %s", name, StateTimeout)
}
return nil, err
}
switch status := msg.Header.Get("Status"); status {
case "":
e, err := logEntryOf(l, msg)
if err != nil {
return nil, err
}
page.Entries = append(page.Entries, *e)
case "204": // the end of the batch: what is still pending after the last entry sent
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
if pending > 0 && len(page.Entries) > 0 {
page.Next = page.Entries[len(page.Entries)-1].Sequence + 1
}
done = true
case "404": // nothing from that sequence on
done = true
case "503":
return nil, notOnTheBus(l)
default:
return nil, fmt.Errorf("the read of %s was refused by the bus: %s %s", name, status, msg.Header.Get("Description"))
}
}
info, err := c.logInfo(l)
if err != nil {
return nil, err
}
page.Last = info.Last
return page, nil
}
// LogLast is a key's last entry, or nil when it has none.
func (c *Conn) LogLast(module, name, key string) (*LogEntry, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
if err := checkLogKey(key); err != nil {
return nil, err
}
reply, err := c.nc.Request("$JS.API.DIRECT.GET."+l.Stream+"."+l.Subject+"."+key, nil, StateTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return nil, notOnTheBus(l)
}
if err != nil {
return nil, err
}
switch status := reply.Header.Get("Status"); status {
case "":
return logEntryOf(l, reply)
case "404":
return nil, nil
default:
return nil, fmt.Errorf("the last entry of %s.%s could not be read: %s %s", name, key, status, reply.Header.Get("Description"))
}
}
// LogInfo is a log's entries, bytes, cap and first and last sequences.
func (c *Conn) LogInfo(module, name string) (*LogInfo, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
return c.logInfo(l)
}
func (c *Conn) logInfo(l LogIssued) (*LogInfo, error) {
reply, err := c.nc.Request("$JS.API.STREAM.INFO."+l.Stream, nil, StateTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return nil, notOnTheBus(l)
}
if err != nil {
return nil, err
}
var info struct {
Config struct {
MaxBytes int64 `json:"max_bytes"`
} `json:"config"`
State struct {
Msgs uint64 `json:"messages"`
Bytes uint64 `json:"bytes"`
FirstSeq uint64 `json:"first_seq"`
LastSeq uint64 `json:"last_seq"`
} `json:"state"`
Error *struct {
ErrCode uint16 `json:"err_code"`
Description string `json:"description"`
} `json:"error"`
}
if err := json.Unmarshal(reply.Data, &info); err != nil {
return nil, fmt.Errorf("the bus answered the info of %s with what is not one: %q", l.Name, reply.Data)
}
if info.Error != nil {
if info.Error.ErrCode == uint16(jetstream.JSErrCodeStreamNotFound) {
return nil, notOnTheBus(l)
}
return nil, fmt.Errorf("the info of the log %s was refused: %s (%d)", l.Name, info.Error.Description, info.Error.ErrCode)
}
var maxBytes uint64
if info.Config.MaxBytes > 0 {
maxBytes = uint64(info.Config.MaxBytes)
}
return &LogInfo{Entries: info.State.Msgs, Bytes: info.State.Bytes, MaxBytes: maxBytes,
First: info.State.FirstSeq, Last: info.State.LastSeq}, nil
}
// logEntryOf is an entry a direct get answered: its key is its subject under the log's.
func logEntryOf(l LogIssued, msg *nats.Msg) (*LogEntry, error) {
subject := msg.Header.Get("Nats-Subject")
key, ok := strings.CutPrefix(subject, l.Subject+".")
if !ok {
return nil, fmt.Errorf("the bus answered an entry of %s on %q, which is not under %s", l.Name, subject, l.Subject)
}
seq, err := strconv.ParseUint(msg.Header.Get("Nats-Sequence"), 10, 64)
if err != nil {
return nil, fmt.Errorf("the bus answered an entry of %s with no sequence", l.Name)
}
at, err := time.Parse(time.RFC3339Nano, msg.Header.Get("Nats-Time-Stamp"))
if err != nil {
return nil, fmt.Errorf("the bus answered entry %d of %s with no time: %v", seq, l.Name, err)
}
return &LogEntry{Sequence: seq, Key: key, Value: valueOf(msg.Data), At: at.UTC()}, nil
}
+123
View File
@@ -0,0 +1,123 @@
package bus
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
// An answer larger than the bus carries in one message (novox/hq issue 314).
//
// The bus carries one message of at most its max_payload, and the client library refuses a larger one
// before it leaves the process that sent it. Until this, a holder whose answer was too large wrote one
// line to its own log and its caller waited out RequestTimeout and read "did not answer" — the
// controller's `conditions` and `status` for hours on 2026-10-08, and the operator's channel with them.
//
// The controller pages such an answer (mesh-controller internal/link, pages.go): it answers an error
// carrying `paged` — the call it holds the answer under, its size and its parts — and a caller asks the
// same subject again once per part with PageHeader, each part answered raw with PartHeader. Asking the
// same subject means no grant beyond the one the caller already had. A module's answer too large is
// refused here, loudly, in a message that fits: its instances may be several, and a part asked again
// could reach another.
const (
// PageHeader asks for one part of a paged answer: `<call> <part>`, from zero.
PageHeader = "Mesh-Page"
// PartHeader is on a part: `<part>/<parts>`.
PartHeader = "Mesh-Part"
// partTries is how often one part is asked: during a handover a part can reach the controller that
// did not hold it.
partTries = 3
)
// PartTimeout bounds the asking of one part.
var PartTimeout = 10 * time.Second
// paged is what a too-large answer says instead of itself.
type paged struct {
Call string `json:"call"`
Bytes int `json:"bytes"`
Parts int `json:"parts"`
}
func pagedIn(data []byte) (paged, bool) {
if !bytes.Contains(data, []byte(`"paged"`)) {
return paged{}, false
}
var r struct {
Paged *paged `json:"paged"`
}
if json.Unmarshal(data, &r) != nil || r.Paged == nil || r.Paged.Call == "" || r.Paged.Parts < 1 {
return paged{}, false
}
return *r.Paged, true
}
// whole is a reply's whole answer: the reply itself, or every part of a paged one, joined. A part that
// cannot be had is an error naming it, never a shorter answer.
func (c *Conn) whole(subject string, data []byte) ([]byte, error) {
p, ok := pagedIn(data)
if !ok {
return data, nil
}
out := make([]byte, 0, p.Bytes)
for n := 0; n < p.Parts; n++ {
var part []byte
var err error
for try := 0; try < partTries; try++ {
if part, err = c.askPart(subject, p, n); err == nil {
break
}
}
if err != nil {
return nil, fmt.Errorf("the answer was %d bytes, more than the bus carries in one message, paged as %s "+
"in %d parts, and part %d could not be had: %w", p.Bytes, p.Call, p.Parts, n, err)
}
out = append(out, part...)
}
if len(out) != p.Bytes {
return nil, fmt.Errorf("the answer paged as %s was %d bytes, and its parts joined are %d", p.Call, p.Bytes, len(out))
}
return out, nil
}
func (c *Conn) askPart(subject string, p paged, n int) ([]byte, error) {
ask := nats.NewMsg(subject)
ask.Header.Set(PageHeader, fmt.Sprintf("%s %d", p.Call, n))
ask.Data = []byte(`{}`)
reply, err := c.nc.RequestMsg(ask, PartTimeout)
if err != nil {
return nil, err
}
if got := reply.Header.Get(PartHeader); got != fmt.Sprintf("%d/%d", n, p.Parts) {
var r struct {
Error string `json:"error"`
}
if json.Unmarshal(reply.Data, &r) == nil && r.Error != "" {
return nil, errors.New(r.Error)
}
return nil, fmt.Errorf("asked part %d/%d, answered %q", n, p.Parts, got)
}
return reply.Data, nil
}
// fits is an answer the bus can carry: the answer itself, or — when it is larger than one message —
// an error that says so, which is.
func (c *Conn) fits(subject string, body []byte) []byte {
limit := c.nc.MaxPayload()
if limit <= 0 || int64(len(body)) <= limit {
return body
}
c.Logf("[mesh-tools] the answer on %s is %d bytes, more than the bus carries in one message (%d): its caller "+
"is told so instead", subject, len(body), limit)
said, _ := wire.Marshal(reply{Node: c.node, Error: fmt.Sprintf("the answer is %d bytes, more than the bus "+
"carries in one message (%d), so it was not sent: ask for less — a narrower question, a shorter window, "+
"fewer lines", len(body), limit)})
return said
}
+80
View File
@@ -0,0 +1,80 @@
package bus_test
import (
"encoding/json"
"fmt"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq issue 314: an answer larger than the bus carries in one message arrives whole when the
// controller pages it, and a module's is said to be too large — neither is a 30-second silence.
func TestAnAnswerLargerThanOneMessage(t *testing.T) {
url := meshtest.URL(t)
seat, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer seat.Close()
c, err := bus.Connect(bus.Credential{URL: url, Module: "console", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
defer c.Close()
limit := int(seat.MaxPayload())
// The controller as it pages (mesh-controller internal/link, pages.go): the first answer an error
// with `paged`, each part asked again on the same subject with the page header.
large := strings.Repeat("a condition held past its bound; ", 3*limit/32)
whole, _ := json.Marshal(map[string]any{"result": large})
partSize := limit - 1024
parts := (len(whole) + partSize - 1) / partSize
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", "conditions", "", ""), func(m *nats.Msg) {
if page := m.Header.Get(bus.PageHeader); page != "" {
var id string
var n int
fmt.Sscanf(page, "%s %d", &id, &n)
end := min((n+1)*partSize, len(whole))
part := nats.NewMsg(m.Reply)
part.Header.Set(bus.PartHeader, fmt.Sprintf("%d/%d", n, parts))
part.Data = whole[n*partSize : end]
_ = m.RespondMsg(part)
return
}
body, _ := json.Marshal(map[string]any{"error": "too large for one message",
"paged": map[string]any{"call": "call-1-1", "bytes": len(whole), "parts": parts}})
_ = m.Respond(body)
})
if err != nil {
t.Fatal(err)
}
defer sub.Unsubscribe()
_ = seat.Flush()
got, err := c.Ask("seat:mesh-controller.conditions", map[string]any{}, "")
var said string
if err != nil || json.Unmarshal(got.Result, &said) != nil || said != large {
t.Fatalf("an answer of %d bytes in %d parts arrived as %d bytes: %v", len(whole), parts, len(got.Result), err)
}
// A module's answer too large is refused here in words, at once.
stop, err := c.HandleSubject("mesh.mod.journal.tool.lines", func(json.RawMessage) (any, error) { return large, nil })
if err != nil {
t.Fatal(err)
}
defer stop()
asked := time.Now()
_, err = c.Ask("journal.lines", map[string]any{}, "mesh.mod.journal.tool.lines")
if err == nil || !strings.Contains(err.Error(), "more than the bus carries") {
t.Fatalf("a module's answer too large for one message read as %v", err)
}
if took := time.Since(asked); took > 5*time.Second {
t.Fatalf("it took %s to say", took)
}
}
+86
View File
@@ -0,0 +1,86 @@
package bus
import (
"encoding/json"
"fmt"
"os"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
)
// novox/hq issue 222: a subscription the bus refused is asked for again. A push sends the machine
// that needs a grant and the bus's machine together; the runtime may subscribe the moment before the
// bus reloads, and a refusal must not leave the module unreachable until some later membership.
func TestARefusedSubscriptionIsAskedForAgain(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
c, err := Connect(Credential{URL: url, Module: "alpha", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
defer c.Close()
var mu sync.Mutex
var said []string
c.Logf = func(f string, a ...any) { mu.Lock(); said = append(said, fmt.Sprintf(f, a...)); mu.Unlock() }
c.RetryAfter = []time.Duration{50 * time.Millisecond, 50 * time.Millisecond}
subject := "mesh.mod.alpha.tool.ping.anchor"
stop, err := c.answerOn(subject, "", func(json.RawMessage) (any, error) { return "pong", nil })
if err != nil {
t.Fatal(err)
}
first := c.answering[answeringKey(subject, "")].sub
// What the bus says when the grant is not there yet.
c.refused(fmt.Errorf("%w: Permissions Violation for Subscription to %q", nats.ErrPermissionViolation, subject))
deadline := time.Now().Add(2 * time.Second)
for {
c.mu.Lock()
again := c.answering[answeringKey(subject, "")].sub
c.mu.Unlock()
if again != first {
break
}
if time.Now().After(deadline) {
t.Fatal("the refused subscription was not asked for again")
}
time.Sleep(10 * time.Millisecond)
}
asker, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer asker.Close()
reply, err := asker.Request(subject, []byte("{}"), 2*time.Second)
if err != nil {
t.Fatalf("the subject asked for again does not answer: %v", err)
}
if !strings.Contains(string(reply.Data), "pong") {
t.Fatalf("the subject asked for again answered %s", reply.Data)
}
// Past its attempts it is given up, and said.
for i := 0; i < 3; i++ {
c.refused(fmt.Errorf("%w: Permissions Violation for Subscription to %q", nats.ErrPermissionViolation, subject))
time.Sleep(80 * time.Millisecond)
}
mu.Lock()
gaveUp := strings.Contains(strings.Join(said, "\n"), "still refuses "+subject)
mu.Unlock()
if !gaveUp {
t.Errorf("never gave up on a subject that stays refused:\n%s", strings.Join(said, "\n"))
}
// A subject the runtime stopped answering is not asked for again.
stop()
c.refused(fmt.Errorf("%w: Permissions Violation for Subscription to %q", nats.ErrPermissionViolation, subject))
if _, still := c.answering[answeringKey(subject, "")]; still {
t.Error("a stopped subject is still tracked")
}
}
+439
View File
@@ -0,0 +1,439 @@
package bus
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// A bundle's traffic on seats that name their caller or their kind (novox/hq ADR 0259 §3): submitting an
// accept and saying an event under its own name or kind, taking a held seat's work from its worker, asking a
// proof and answering one, and reading a record kept for it.
//
// **The runtime keeps each module to its own name and kind.** One account per machine carries every module
// on it, so the bus enforces only the union; what one module may reach is its membership's `seat-traffic`,
// and anything else is refused here, with the reason, before it is sent. A bundle reaches this only through
// its own standard input and output: no tool call does.
// SeatTraffic is what a membership lists of a module's seat traffic (the controller's broker.SeatTraffic).
type SeatTraffic struct {
Publish []string `json:"publish,omitempty"`
Subscribe []string `json:"subscribe,omitempty"`
Answers []string `json:"answers,omitempty"`
Workers []SeatWorker `json:"workers,omitempty"`
Records []string `json:"records,omitempty"`
// Kinds are the holders of every kinded bench the module uses or watches, with what each promises: the
// controller's record of their claims (novox/hq ADR 0259 §5).
Kinds []KindHeld `json:"kinds,omitempty"`
}
// KindHeld is one kind of a kinded bench and who holds it.
type KindHeld struct {
Seat string `json:"seat"`
Kind string `json:"kind"`
Module string `json:"module"`
Node string `json:"node"`
Capabilities []string `json:"capabilities,omitempty"`
}
// SeatKinds is the kinds the module's membership names, as the controller issued them.
func (c *Conn) SeatKinds(module string) ([]KindHeld, error) {
t, err := c.trafficOf(module)
if err != nil {
return nil, err
}
return append([]KindHeld{}, t.Kinds...), nil
}
// SeatWorker is one work queue a holder takes from.
type SeatWorker struct {
Stream string `json:"Stream"`
Consumer string `json:"Consumer"`
Filter string `json:"Filter"`
}
// ProofTimeout bounds how long a proof waits for its answer.
var ProofTimeout = 15 * time.Second
// WorkHeartbeat is how often work a bundle is still doing is said to be in progress, well inside the
// worker's ack wait.
var WorkHeartbeat = 15 * time.Second
// WorkBackoff is how long a piece of work the bundle did not take waits before it is offered again: from
// five seconds, doubling, to ten minutes.
func WorkBackoff(delivered uint64) time.Duration {
wait := 5 * time.Second
for i := uint64(1); i < delivered && wait < 10*time.Minute; i++ {
wait *= 2
}
return min(wait, 10*time.Minute)
}
// SubjectMatches is a subject against a permission pattern, in the server's wildcards.
func SubjectMatches(pattern, subject string) bool {
p, s := strings.Split(pattern, "."), strings.Split(subject, ".")
for i, part := range p {
if part == ">" {
return len(s) > i
}
if i >= len(s) || (part != "*" && part != s[i]) {
return false
}
}
return len(p) == len(s)
}
func (c *Conn) trafficOf(module string) (*SeatTraffic, error) {
m := c.Membership(module)
if m == nil {
return nil, fmt.Errorf("%s has no membership issued on %s yet, so it reaches no seat (novox/hq ADR 0259)",
module, c.node)
}
if m.SeatTraffic == nil {
return nil, fmt.Errorf("%s holds, uses and watches no seat that names its caller or its kind (novox/hq "+
"ADR 0259): its manifest says so under `uses`, `claims` and `consumes`", module)
}
return m.SeatTraffic, nil
}
func listed(patterns []string, subject string) bool {
for _, p := range patterns {
if SubjectMatches(p, subject) {
return true
}
}
return false
}
// mayPublish refuses a subject the module's seat traffic does not list. A wildcard is never published.
func (c *Conn) mayPublish(module, subject string) error {
t, err := c.trafficOf(module)
if err != nil {
return err
}
if strings.ContainsAny(subject, "*>") || !listed(t.Publish, subject) {
return fmt.Errorf("%s may not publish %s: a module submits and says only under its own name or kind, "+
"and its membership lists %s (novox/hq ADR 0259)", module, subject, strings.Join(t.Publish, ", "))
}
return nil
}
func newID() string {
var b [16]byte
_, _ = rand.Read(b[:])
return hex.EncodeToString(b[:])
}
// SeatPublish submits an accept or says an event on a seat, as the module, into the stream that keeps it,
// awaited and de-duplicated by its id: the same id twice is one message. A proof is never published here.
func (c *Conn) SeatPublish(module, subject string, body json.RawMessage, id string) (uint64, error) {
seq, _, err := c.SeatPublishSaid(module, subject, body, id)
return seq, err
}
// SeatPublishSaid is SeatPublish, also saying whether the bus took it as a duplicate of one published before
// under the same id: the same message, kept once.
func (c *Conn) SeatPublishSaid(module, subject string, body json.RawMessage, id string) (uint64, bool, error) {
if strings.Contains(subject, ".proof.") {
return 0, false, fmt.Errorf("%s is a proof, asked and answered, never kept: ask it as one", subject)
}
if err := c.mayPublish(module, subject); err != nil {
return 0, false, err
}
if len(body) == 0 || !json.Valid(body) {
return 0, false, fmt.Errorf("what is said on %s is JSON", subject)
}
if id == "" {
id = newID()
}
// **The publisher's name is part of the id the bus de-duplicates by** (security review 2026-10-08): ids
// are the publisher's own, and one module could otherwise suppress another's warrant by publishing first
// under the same id.
id = module + "." + id
msg := nats.NewMsg(subject)
msg.Header.Set("x-event-id", id)
msg.Header.Set("x-source", module)
msg.Header.Set("x-node", c.node)
msg.Header.Set("x-time", time.Now().UTC().Format(time.RFC3339Nano))
msg.Header.Set("content-type", "application/json")
msg.Data = body
ack, err := c.js.PublishMsg(msg, nats.MsgId(id))
if err != nil {
return 0, false, err
}
return ack.Sequence, ack.Duplicate, nil
}
// SeatProve asks a proof and answers its reply: core request and reply, on no stream.
func (c *Conn) SeatProve(module, subject string, body json.RawMessage) (json.RawMessage, error) {
if !strings.Contains(subject, ".proof.") {
return nil, fmt.Errorf("%s is not a proof", subject)
}
if err := c.mayPublish(module, subject); err != nil {
return nil, err
}
reply, err := c.nc.Request(subject, body, ProofTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return nil, fmt.Errorf("nothing answers %s: the module watching the seat is not running", subject)
}
if err != nil {
return nil, err
}
if !json.Valid(reply.Data) {
return nil, fmt.Errorf("the answer to %s is not JSON", subject)
}
return json.RawMessage(reply.Data), nil
}
// SeatAnswer answers every proof on a subject the module's seat traffic lists among its answers, with what
// answer returns. A returned error is answered as `{"error": "<words>"}`, never as silence.
func (c *Conn) SeatAnswer(module, subject string, answer func(subject string, body json.RawMessage) (json.RawMessage, error)) (func(), error) {
t, err := c.trafficOf(module)
if err != nil {
return nil, err
}
allowed := false
for _, a := range t.Answers {
allowed = allowed || a == subject
}
if !allowed {
return nil, fmt.Errorf("%s does not answer %s: its membership lists %s (novox/hq ADR 0259)", module,
subject, strings.Join(t.Answers, ", "))
}
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
if msg.Reply == "" {
return
}
body := json.RawMessage(msg.Data)
if !json.Valid(body) {
_ = msg.Respond([]byte(`{"error":"a proof is JSON"}`))
return
}
out, err := answer(msg.Subject, body)
if err != nil {
b, _ := json.Marshal(map[string]string{"error": err.Error()})
_ = msg.Respond(b)
return
}
if len(out) == 0 {
out = json.RawMessage(`{}`)
}
if err := msg.Respond(out); err != nil {
c.Logf("[mesh-tools] %s could not answer the proof on %s: %v", module, msg.Subject, err)
}
})
if err != nil {
return nil, err
}
c.track(sub)
var once sync.Once
return func() { once.Do(func() { _ = sub.Unsubscribe() }) }, nil
}
// Work is one message a holder took from a seat's work queue.
type Work struct {
Worker string `json:"worker"`
Subject string `json:"subject"`
Body json.RawMessage `json:"body"`
Headers map[string]string `json:"headers,omitempty"`
}
// SeatTake reads a worker the module's seat traffic lists and hands each message to deliver: acknowledged
// when deliver returns nil, offered again after NakDelay when it returns an error, ended when it is not
// JSON. The worker is the controller's to create; this binds to it by name and never makes one.
func (c *Conn) SeatTake(module, consumer string, deliver func(Work) error) (func(), error) {
t, err := c.trafficOf(module)
if err != nil {
return nil, err
}
var w *SeatWorker
for i := range t.Workers {
if t.Workers[i].Consumer == consumer {
w = &t.Workers[i]
}
}
if w == nil {
var names []string
for _, x := range t.Workers {
names = append(names, x.Consumer)
}
return nil, fmt.Errorf("%s takes no work from %s: its membership lists %s (novox/hq ADR 0259)", module,
consumer, strings.Join(names, ", "))
}
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
bound, err := js.Consumer(ctx, w.Stream, w.Consumer)
if err != nil {
return nil, fmt.Errorf("binding %s's worker %s on %s: %w", module, w.Consumer, w.Stream, err)
}
reading, err := bound.Consume(func(msg jetstream.Msg) {
if !json.Valid(msg.Data()) {
_ = msg.Term()
return
}
headers := map[string]string{}
for k := range msg.Headers() {
headers[k] = msg.Headers().Get(k)
}
// Kept alive while the bundle works on it, however long that takes: the ack wait is for a holder
// that died, not one that is slow.
done := make(chan struct{})
go func() {
tick := time.NewTicker(WorkHeartbeat)
defer tick.Stop()
for {
select {
case <-done:
return
case <-tick.C:
_ = msg.InProgress()
}
}
}()
err := deliver(Work{Worker: w.Consumer, Subject: msg.Subject(), Body: json.RawMessage(msg.Data()), Headers: headers})
close(done)
if err != nil {
// Offered again later and later, never given up on: a channel that is away keeps its work
// (security and correctness reviews of 2026-10-08).
delivered := uint64(1)
if meta, merr := msg.Metadata(); merr == nil {
delivered = meta.NumDelivered
}
_ = msg.NakWithDelay(WorkBackoff(delivered))
return
}
_ = msg.Ack()
}, jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) {
// A worker deleted, or the bus refusing it, is said — never a holder that silently takes nothing.
c.Logf("[mesh-tools] %s's worker %s: %v", module, w.Consumer, err)
}))
if err != nil {
return nil, fmt.Errorf("reading %s's worker %s: %w", module, w.Consumer, err)
}
var once sync.Once
return func() { once.Do(reading.Stop) }, nil
}
// SeatRecord reads one key of a record a seat keeps for the module, under its own name: the bucket's
// current value of `<module>.<key>`, or nil when there is none.
func (c *Conn) SeatRecord(module, bucket, key string) (*StateEntry, error) {
t, err := c.trafficOf(module)
if err != nil {
return nil, err
}
if err := checkKey(key); err != nil {
return nil, err
}
if bucket == "" {
// The one record its membership lists, when it lists one: an asker reads the router's asks without
// knowing the router's name.
var named []string
for _, r := range t.Records {
if b, _, ok := strings.Cut(strings.TrimPrefix(r, "$JS.API.DIRECT.GET.KV_"), "."); ok {
named = append(named, b)
}
}
if len(named) != 1 {
return nil, fmt.Errorf("%s reads %d records under its name; name the one meant", module, len(named))
}
bucket = named[0]
}
subject := "$JS.API.DIRECT.GET.KV_" + bucket + ".$KV." + bucket + "." + module + "." + key
if !listed(t.Records, subject) {
return nil, fmt.Errorf("%s reads no record of %s under its name: its membership lists %s (novox/hq ADR 0259)",
module, bucket, strings.Join(t.Records, ", "))
}
reply, err := c.nc.Request(subject, nil, StateTimeout)
if err != nil {
return nil, err
}
if status := reply.Header.Get("Status"); status == "404" {
return nil, nil
} else if status != "" {
return nil, fmt.Errorf("the record %s.%s could not be read: %s %s", module, key, status, reply.Header.Get("Description"))
}
if op := reply.Header.Get("KV-Operation"); op == "DEL" || op == "PURGE" {
return nil, nil
}
var revision uint64
fmt.Sscan(reply.Header.Get("Nats-Sequence"), &revision)
return &StateEntry{Key: module + "." + key, Value: valueOf(reply.Data), Revision: revision}, nil
}
// StateCreate writes one key only when it has no value, and answers its revision; a key that has one is
// refused, naming it, so two writers racing to make it are one winner and one refusal.
func (c *Conn) StateCreate(module, name, key string, value json.RawMessage) (uint64, error) {
return c.stateWrite(module, name, key, value, func(ctx context.Context, kv jetstream.KeyValue) (uint64, error) {
return kv.Create(ctx, key, value)
})
}
// StateUpdate writes one key only when its revision is still the one given, and answers the new revision;
// a key changed meanwhile is refused, so of two writers that read one value only the first writes.
func (c *Conn) StateUpdate(module, name, key string, value json.RawMessage, revision uint64) (uint64, error) {
if revision == 0 {
// Revision zero would be read by the server as "the key is new": an update names the value it read.
return 0, fmt.Errorf("an update names the revision it read; zero is none — create the key instead")
}
return c.stateWrite(module, name, key, value, func(ctx context.Context, kv jetstream.KeyValue) (uint64, error) {
return kv.Update(ctx, key, value, revision)
})
}
// ErrStateChanged is a compare-and-set that lost: the key was made or changed by another writer first.
var ErrStateChanged = errors.New("the key was written by another writer first")
// changed is ErrStateChanged naming the key, with the code a bundle is answered it by.
type changed struct{ key string }
func (e changed) Error() string { return e.key + ": " + ErrStateChanged.Error() }
func (e changed) Is(target error) bool { return target == ErrStateChanged }
func (e changed) ErrorCode() int { return -32010 }
func (c *Conn) stateWrite(module, name, key string, value json.RawMessage,
write func(context.Context, jetstream.KeyValue) (uint64, error)) (uint64, error) {
s, err := c.writable(module, name)
if err != nil {
return 0, err
}
if err := checkKey(key); err != nil {
return 0, err
}
if len(value) == 0 || !json.Valid(value) {
return 0, fmt.Errorf("a state value is JSON")
}
if field := credentialField(value); field != "" {
return 0, fmt.Errorf("%s's %s.%s carries a field %q, which names a credential: no secret is kept in state "+
"(novox/hq ADR 0201)", module, name, key, field)
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return 0, err
}
rev, err := write(ctx, kv)
if errors.Is(err, jetstream.ErrKeyExists) || isWrongSequence(err) {
return 0, changed{name + "." + key}
}
return rev, err
}
// isWrongSequence is the server's refusal of an update whose revision is not the key's last.
func isWrongSequence(err error) bool {
var api *jetstream.APIError
return errors.As(err, &api) && api.ErrorCode == jetstream.JSErrCodeStreamWrongLastSequence
}
+239
View File
@@ -0,0 +1,239 @@
package bus_test
import (
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// The seats of novox/hq ADR 0259 §3 as the controller issues them: an asker, the router, and a channel of
// kind telegram, each with the seat traffic its membership lists.
func seatMesh(t *testing.T) (*mt.Mesh, *bus.Conn, nats.JetStreamContext) {
t.Helper()
mesh := mt.New(t)
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := nc.JetStream()
for _, s := range []*nats.StreamConfig{
{Name: "SEAT_OPERATOR_CHANNEL", Subjects: []string{"mesh.seat.operator-channel.accept.>"}, Retention: nats.WorkQueuePolicy},
{Name: "SEAT_CHANNEL", Subjects: []string{"mesh.seat.channel.accept.>"}, Retention: nats.WorkQueuePolicy},
{Name: "SEAT_EVENTS", Subjects: []string{"mesh.seat.*.event.>"}},
} {
if _, err := js.AddStream(s); err != nil {
t.Fatal(err)
}
}
for stream, c := range map[string]*nats.ConsumerConfig{
"SEAT_OPERATOR_CHANNEL": {Durable: "SEAT_OPERATOR_CHANNEL_worker", AckPolicy: nats.AckExplicitPolicy,
FilterSubject: "mesh.seat.operator-channel.accept.>", AckWait: 2 * time.Second},
"SEAT_CHANNEL": {Durable: "SEAT_CHANNEL_TELEGRAM_worker", AckPolicy: nats.AckExplicitPolicy,
FilterSubject: "mesh.seat.channel.accept.*.telegram", AckWait: 2 * time.Second},
} {
if _, err := js.AddConsumer(stream, c); err != nil {
t.Fatal(err)
}
}
if _, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: "messenger_asks"}); err != nil {
t.Fatal(err)
}
asker := mt.MembershipOf("mesh-delivery", "anchor", false, nil)
asker.SeatTraffic = &bus.SeatTraffic{
Publish: []string{"mesh.seat.operator-channel.accept.ask.mesh-delivery"},
Subscribe: []string{"mesh.seat.operator-channel.event.decided.mesh-delivery"},
Records: []string{"$JS.API.DIRECT.GET.KV_messenger_asks.$KV.messenger_asks.mesh-delivery.>"},
}
router := mt.MembershipOf("messenger", "anchor", false, nil)
router.SeatTraffic = &bus.SeatTraffic{
Publish: []string{"mesh.seat.operator-channel.event.decided.*", "mesh.seat.channel.accept.show.*"},
Answers: []string{"mesh.seat.intake.proof.code.*"},
Workers: []bus.SeatWorker{{Stream: "SEAT_OPERATOR_CHANNEL", Consumer: "SEAT_OPERATOR_CHANNEL_worker",
Filter: "mesh.seat.operator-channel.accept.>"}},
}
router.State = []bus.StateIssued{{Name: "asks", Bucket: "messenger_asks", Writes: true}}
telegram := mt.MembershipOf("telegram", "anchor", false, nil)
telegram.SeatTraffic = &bus.SeatTraffic{
Publish: []string{"mesh.seat.intake.event.choice.telegram", "mesh.seat.intake.proof.code.telegram"},
Workers: []bus.SeatWorker{{Stream: "SEAT_CHANNEL", Consumer: "SEAT_CHANNEL_TELEGRAM_worker",
Filter: "mesh.seat.channel.accept.*.telegram"}},
}
for _, m := range []bus.Membership{asker, router, telegram} {
mesh.Issue(t, m)
}
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
for _, m := range []string{"mesh-delivery", "messenger", "telegram"} {
conn.Follow(m)
}
mt.Until(t, func() error {
for _, m := range []string{"mesh-delivery", "messenger", "telegram"} {
if conn.Membership(m) == nil {
return errors.New(m + " not issued yet")
}
}
return nil
})
return mesh, conn, js
}
func TestAModuleSubmitsAndSaysOnlyUnderItsOwnNameAndKind(t *testing.T) {
_, conn, _ := seatMesh(t)
body := json.RawMessage(`{"id":"a1"}`)
if _, err := conn.SeatPublish("mesh-delivery", "mesh.seat.operator-channel.accept.ask.mesh-delivery", body, "a1"); err != nil {
t.Fatalf("an asker could not ask under its own name: %v", err)
}
for module, subject := range map[string]string{
"mesh-delivery": "mesh.seat.operator-channel.accept.ask.mesh-controller",
"telegram": "mesh.seat.intake.event.choice.desktop",
"messenger": "mesh.seat.intake.event.choice.telegram",
} {
if _, err := conn.SeatPublish(module, subject, body, ""); err == nil || !strings.Contains(err.Error(), "may not publish") {
t.Errorf("%s published %s: %v", module, subject, err)
}
}
if _, err := conn.SeatPublish("messenger", "mesh.seat.operator-channel.event.decided.*", body, ""); err == nil {
t.Error("a wildcard was published")
}
if _, err := conn.SeatPublish("telegram", "mesh.seat.intake.proof.code.telegram", body, ""); err == nil ||
!strings.Contains(err.Error(), "never kept") {
t.Errorf("a proof was published onto a stream: %v", err)
}
}
func TestTheHolderTakesItsWorkOnceAndAFailureComesBack(t *testing.T) {
_, conn, _ := seatMesh(t)
if _, err := conn.SeatPublish("mesh-delivery", "mesh.seat.operator-channel.accept.ask.mesh-delivery",
json.RawMessage(`{"id":"a1"}`), "a1"); err != nil {
t.Fatal(err)
}
// The same id again is one ask: a retry after a lost answer does not ask twice.
if _, err := conn.SeatPublish("mesh-delivery", "mesh.seat.operator-channel.accept.ask.mesh-delivery",
json.RawMessage(`{"id":"a1"}`), "a1"); err != nil {
t.Fatal(err)
}
var mu sync.Mutex
var got []string
fail := true
stop, err := conn.SeatTake("messenger", "SEAT_OPERATOR_CHANNEL_worker", func(w bus.Work) error {
mu.Lock()
defer mu.Unlock()
got = append(got, w.Subject+" from "+w.Headers["x-source"])
if fail {
fail = false
return errors.New("not recorded")
}
return nil
})
if err != nil {
t.Fatal(err)
}
defer stop()
deadline := time.Now().Add(15 * time.Second)
for {
mu.Lock()
n := len(got)
mu.Unlock()
if n >= 2 || time.Now().After(deadline) {
break
}
time.Sleep(50 * time.Millisecond)
}
time.Sleep(500 * time.Millisecond)
mu.Lock()
defer mu.Unlock()
if len(got) != 2 || got[0] != "mesh.seat.operator-channel.accept.ask.mesh-delivery from mesh-delivery" || got[0] != got[1] {
t.Fatalf("want the one ask, offered again after the failure: %v", got)
}
if _, err := conn.SeatTake("telegram", "SEAT_OPERATOR_CHANNEL_worker", func(bus.Work) error { return nil }); err == nil {
t.Error("a channel took the router's work")
}
}
func TestAProofIsAskedAndAnsweredAndNeverKept(t *testing.T) {
_, conn, js := seatMesh(t)
stop, err := conn.SeatAnswer("messenger", "mesh.seat.intake.proof.code.*", func(subject string, body json.RawMessage) (json.RawMessage, error) {
if !strings.HasSuffix(subject, ".telegram") {
return nil, errors.New("asked by another kind")
}
return json.RawMessage(`{"linked":true}`), nil
})
if err != nil {
t.Fatal(err)
}
defer stop()
var answer json.RawMessage
mt.Until(t, func() error {
answer, err = conn.SeatProve("telegram", "mesh.seat.intake.proof.code.telegram", json.RawMessage(`{"code":"123456"}`))
return err
})
if string(answer) != `{"linked":true}` {
t.Fatalf("the proof was answered %s", answer)
}
if _, err := conn.SeatProve("telegram", "mesh.seat.intake.proof.code.desktop", json.RawMessage(`{}`)); err == nil {
t.Error("a channel asked a proof as another kind")
}
if _, err := conn.SeatAnswer("telegram", "mesh.seat.intake.proof.code.*", nil); err == nil {
t.Error("a channel answers proofs")
}
for name := range js.StreamNames() {
info, _ := js.StreamInfo(name)
for _, s := range info.Config.Subjects {
if bus.SubjectMatches(s, "mesh.seat.intake.proof.code.telegram") {
t.Errorf("%s keeps proofs", name)
}
}
}
}
func TestAnAskerReadsItsOwnRecordAndNoOther(t *testing.T) {
_, conn, js := seatMesh(t)
kv, _ := js.KeyValue("messenger_asks")
if _, err := kv.Put("mesh-delivery.a1", []byte(`{"state":"open"}`)); err != nil {
t.Fatal(err)
}
if _, err := kv.Put("mesh-controller.a1", []byte(`{"state":"open"}`)); err != nil {
t.Fatal(err)
}
got, err := conn.SeatRecord("mesh-delivery", "messenger_asks", "a1")
if err != nil || got == nil || string(got.Value) != `{"state":"open"}` {
t.Fatalf("the asker could not read its own ask: %v %v", got, err)
}
if missing, err := conn.SeatRecord("mesh-delivery", "messenger_asks", "a2"); err != nil || missing != nil {
t.Fatalf("a missing ask read as %v, %v", missing, err)
}
if _, err := conn.SeatRecord("telegram", "messenger_asks", "a1"); err == nil {
t.Error("a module read records it is not given")
}
}
func TestOfTwoAnswersToOneAskOnlyTheFirstIsWritten(t *testing.T) {
_, conn, _ := seatMesh(t)
rev, err := conn.StateCreate("messenger", "asks", "mesh-delivery.a1", json.RawMessage(`{"state":"open"}`))
if err != nil {
t.Fatal(err)
}
if _, err := conn.StateCreate("messenger", "asks", "mesh-delivery.a1", json.RawMessage(`{"state":"open"}`)); !errors.Is(err, bus.ErrStateChanged) {
t.Errorf("a second create stood: %v", err)
}
if _, err := conn.StateUpdate("messenger", "asks", "mesh-delivery.a1", json.RawMessage(`{"state":"answered"}`), rev); err != nil {
t.Fatal(err)
}
if _, err := conn.StateUpdate("messenger", "asks", "mesh-delivery.a1", json.RawMessage(`{"state":"answered"}`), rev); !errors.Is(err, bus.ErrStateChanged) {
t.Errorf("a second answer from the same revision stood: %v", err)
}
}
+401
View File
@@ -0,0 +1,401 @@
package bus
import (
"context"
"encoding/json"
"errors"
"fmt"
"regexp"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
)
// A module's state on the bus (novox/hq ADR 0201): key-value buckets the controller creates from what
// the module declared, and issues to each assignment in its membership by the name the module uses —
// its own state by the local name, another module's as `<module>.<name>`.
//
// **The runtime keeps each module to its own buckets.** One account per machine carries every module
// on it, so the bus enforces only the union; and a write the bus refuses reaches the writer as a
// timeout, not a refusal (measured, novox/hq research 024). So what a module may reach is decided here,
// from its membership, and refused with the reason before anything is sent.
// StateIssued is one bucket an assignment may reach (ADR 0201).
type StateIssued struct {
Name string `json:"name"`
Bucket string `json:"bucket"`
Writes bool `json:"writes,omitempty"`
}
// StateEntry is one key's current value.
type StateEntry struct {
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Revision uint64 `json:"revision"`
}
// StateChange is one change a watch delivers: a key put or deleted.
type StateChange struct {
State string `json:"state"`
Key string `json:"key"`
Op string `json:"op"`
Value json.RawMessage `json:"value,omitempty"`
Revision uint64 `json:"revision"`
// Current is true for a value that was there when the watch began, false for a change since.
Current bool `json:"current"`
}
// StateTimeout bounds one state operation on the bus.
var StateTimeout = 10 * time.Second
// stateKey is a key the bus can hold: letters, digits and `-/_=.`, no leading or trailing dot. A
// module's convention for naming a machine in a key (`all.<server>`, `<machine>.<server>`) fits it.
var stateKey = regexp.MustCompile(`^[-/_=a-zA-Z0-9]+(\.[-/_=a-zA-Z0-9]+)*$`)
// issuedState is the bucket a module may reach by a name, and whether it may write it — or the reason
// it may not reach it at all.
func (c *Conn) issuedState(module, name string) (StateIssued, error) {
m := c.Membership(module)
if m == nil {
return StateIssued{}, fmt.Errorf("%s has no membership issued on %s yet, so no state of it is reachable "+
"until the mesh issues one (novox/hq ADR 0201)", module, c.node)
}
var names []string
for _, s := range m.State {
if s.Name == name {
return s, nil
}
names = append(names, s.Name)
}
sort.Strings(names)
issued := "none"
if len(names) > 0 {
issued = strings.Join(names, ", ")
}
return StateIssued{}, fmt.Errorf("%s keeps and reads no state called %q: it declares what it keeps under "+
"`state` and what it reads under `reads` as <module>.<name>, and was issued: %s (novox/hq ADR 0201)",
module, name, issued)
}
func (c *Conn) bucket(ctx context.Context, s StateIssued) (jetstream.KeyValue, error) {
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
kv, err := js.KeyValue(ctx, s.Bucket)
if err != nil {
if errors.Is(err, jetstream.ErrBucketNotFound) {
return nil, fmt.Errorf("the state %q is issued and its bucket is not on the bus yet: the controller "+
"creates it from the catalogue on its next raise", s.Name)
}
return nil, err
}
return kv, nil
}
func checkKey(key string) error {
if !stateKey.MatchString(key) {
return fmt.Errorf("%q is not a key the bus can hold: letters, digits and -/_=, in dot-separated "+
"names", key)
}
return nil
}
// StateGet is one key's current value, or nil when it has none.
func (c *Conn) StateGet(module, name, key string) (*StateEntry, error) {
s, err := c.issuedState(module, name)
if err != nil {
return nil, err
}
if err := checkKey(key); err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return nil, err
}
e, err := kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision()}, nil
}
// StateKept is one value a bucket still keeps for a key: a put, or a delete (a purge reads as one).
type StateKept struct {
Key string `json:"key"`
Value json.RawMessage `json:"value,omitempty"`
Revision uint64 `json:"revision"`
Op string `json:"op"`
At time.Time `json:"at"`
}
// StateHistory is every value the bucket still keeps for one key, oldest first, or none (novox/hq ADR
// 0307). It is a read like StateGet: served to the owner and to every reader, and writes nothing.
func (c *Conn) StateHistory(module, name, key string) ([]StateKept, error) {
s, err := c.issuedState(module, name)
if err != nil {
return nil, err
}
if err := checkKey(key); err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return nil, err
}
entries, err := kv.History(ctx, key)
kept := []StateKept{}
if errors.Is(err, jetstream.ErrKeyNotFound) {
return kept, nil
}
if err != nil {
return nil, err
}
for _, e := range entries {
k := StateKept{Key: e.Key(), Revision: e.Revision(), Op: "delete", At: e.Created().UTC()}
if e.Operation() == jetstream.KeyValuePut {
k.Op = "put"
k.Value = valueOf(e.Value())
}
kept = append(kept, k)
}
return kept, nil
}
// StatePut writes one key, as the module, where the module keeps the state. It answers the revision.
func (c *Conn) StatePut(module, name, key string, value json.RawMessage) (uint64, error) {
s, err := c.writable(module, name)
if err != nil {
return 0, err
}
if err := checkKey(key); err != nil {
return 0, err
}
if len(value) == 0 || !json.Valid(value) {
return 0, fmt.Errorf("a state value is JSON")
}
if field := credentialField(value); field != "" {
return 0, fmt.Errorf("%s's %s.%s carries a field %q, which names a credential: no secret is kept in state, "+
"sealed or not — a bucket is a stream, and a machine joining a year later reads it whole. Name the "+
"secret and fetch it on request/reply (novox/hq ADR 0201, design 32 §10)", module, name, key, field)
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return 0, err
}
return kv.Put(ctx, key, value)
}
// StateDelete removes one key, as the module, where the module keeps the state. A key that was not
// there is not an error: what is asked for is that it is gone.
func (c *Conn) StateDelete(module, name, key string) error {
s, err := c.writable(module, name)
if err != nil {
return err
}
if err := checkKey(key); err != nil {
return err
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return err
}
return kv.Delete(ctx, key)
}
// StateKeys is every key with a current value, sorted.
func (c *Conn) StateKeys(module, name string) ([]string, error) {
s, err := c.issuedState(module, name)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return nil, err
}
lister, err := kv.ListKeys(ctx)
if err != nil {
return nil, err
}
defer func() { _ = lister.Stop() }()
keys := []string{}
for k := range lister.Keys() {
keys = append(keys, k)
}
sort.Strings(keys)
return keys, nil
}
func (c *Conn) writable(module, name string) (StateIssued, error) {
s, err := c.issuedState(module, name)
if err != nil {
return s, err
}
if !s.Writes {
owner := name
if dot := strings.LastIndex(name, "."); dot > 0 {
owner = name[:dot]
}
return s, fmt.Errorf("%s reads %s and does not keep it: only %s's own instances write it (novox/hq ADR 0201)",
module, name, owner)
}
return s, nil
}
// StateWatch hands deliver the current value of every key matching the pattern — none that is
// deleted — and then every change, in order (ADR 0201). It returns once the current values are
// delivered; deliver is called from one goroutine, one change at a time, and an error from it is said
// and the watch goes on: state is not a queue, and the next change, or the next start, reads it again.
// The pattern is a key, with `*` for one name and `**` for the rest; empty is every key.
func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange) error) (stop func(), err error) {
s, err := c.issuedState(module, name)
if err != nil {
return nil, err
}
filter := ">"
if pattern != "" {
parts := strings.Split(pattern, ".")
for i, p := range parts {
if p == "**" {
if i != len(parts)-1 {
return nil, fmt.Errorf("%q: `**` stands for the rest of a key, so it comes last", pattern)
}
parts[i] = ">"
continue
}
if p != "*" && !stateKey.MatchString(p) {
return nil, fmt.Errorf("%q is not a key pattern: names, `*` for one and `**` for the rest", pattern)
}
}
filter = strings.Join(parts, ".")
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
kv, err := c.bucket(ctx, s)
cancel()
if err != nil {
return nil, err
}
watching, stopWatching := context.WithCancel(context.Background())
w, err := kv.Watch(watching, filter)
if err != nil {
stopWatching()
return nil, err
}
var once sync.Once
stop = func() {
once.Do(func() {
_ = w.Stop()
stopWatching()
})
}
current := make(chan struct{})
go func() {
initial := true
for e := range w.Updates() {
if e == nil {
// The end of what was there when the watch began.
if initial {
initial = false
close(current)
}
continue
}
change := StateChange{State: name, Key: e.Key(), Revision: e.Revision(), Current: initial}
switch e.Operation() {
case jetstream.KeyValuePut:
change.Op = "put"
change.Value = valueOf(e.Value())
default:
// A deletion among the current values is a key that is not there: not handed over
// (measured, research 024 — the server sends its marker among the initial values).
if initial {
continue
}
change.Op = "delete"
}
if err := deliver(change); err != nil {
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
module, change.Op, change.Key, name, err)
}
}
if initial {
close(current)
}
}()
select {
case <-current:
return stop, nil
case <-time.After(StateTimeout):
stop()
return nil, fmt.Errorf("the current values of %s did not arrive in %s", name, StateTimeout)
}
}
func valueOf(raw []byte) json.RawMessage {
if len(raw) == 0 || !json.Valid(raw) {
b, _ := json.Marshal(string(raw))
return b
}
return json.RawMessage(raw)
}
// credentialEndings are the ends of a field name that say its value is a credential. A guard against
// the ordinary mistake, not a determined one: a sealed value is plain text to anything inspecting it,
// so the rule is checked where it can be and said to be partial (ADR 0201).
var credentialEndings = []string{"password", "passwd", "secret", "token", "credential", "credentials",
"authorization", "apikey", "privatekey", "accesskey", "cookie"}
// credentialField is the first field anywhere in a JSON value whose name says it is a credential, or "".
func credentialField(value json.RawMessage) string {
var v any
if json.Unmarshal(value, &v) != nil {
return ""
}
var walk func(any) string
walk = func(v any) string {
switch t := v.(type) {
case map[string]any:
keys := make([]string, 0, len(t))
for k := range t {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
norm := strings.NewReplacer("-", "", "_", "", " ", "").Replace(strings.ToLower(k))
for _, end := range credentialEndings {
if strings.HasSuffix(norm, end) {
return k
}
}
if found := walk(t[k]); found != "" {
return found
}
}
case []any:
for _, x := range t {
if found := walk(x); found != "" {
return found
}
}
}
return ""
}
return walk(v)
}
+23
View File
@@ -0,0 +1,23 @@
package bus
import (
"encoding/json"
"testing"
)
// A value naming a credential anywhere in it is found (novox/hq ADR 0201) — the guard against the
// ordinary mistake — and a value that only mentions tokens as a count is not.
func TestACredentialNamedFieldIsFoundAnywhereInAValue(t *testing.T) {
for value, want := range map[string]string{
`{"url":"http://x","headers":{"Authorization":"Bearer s"}}`: "Authorization",
`[{"name":"a","env":{"API_KEY":"s"}}]`: "API_KEY",
`{"refresh_token":"s"}`: "refresh_token",
`{"clientSecret":"s"}`: "clientSecret",
`{"maxTokens":4096,"name":"a","generation":3}`: "",
`"just a string"`: "",
} {
if got := credentialField(json.RawMessage(value)); got != want {
t.Errorf("%s: found %q, want %q", value, got, want)
}
}
}
File diff suppressed because it is too large Load Diff
+243
View File
@@ -0,0 +1,243 @@
package console
import (
"encoding/json"
"fmt"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/announce"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
// text is a tool result's first text, and whether it was an error.
func text(t *testing.T, reply map[string]any) (string, bool) {
t.Helper()
result, ok := reply["result"].(map[string]any)
if !ok {
t.Fatalf("no result: %v", reply)
}
content := result["content"].([]any)
isErr, _ := result["isError"].(bool)
var parts []string
for _, c := range content {
parts = append(parts, c.(map[string]any)["text"].(string))
}
return strings.Join(parts, "\n"), isErr
}
func call(t *testing.T, endpoint, tool string, args map[string]any) (string, bool) {
t.Helper()
body, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 9, "method": "tools/call",
"params": map[string]any{"name": tool, "arguments": args}})
return text(t, post(t, endpoint, string(body)))
}
// novox/hq ADR 0195: the console announces six tools, and everything the mesh answers is reached
// through them by one address per layer.
func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
was := IndexKept
IndexKept = 0 // every discovery asks the mesh, so a module arriving mid-test is found
t.Cleanup(func() { IndexKept = was })
mesh := mt.New(t)
// alpha: interchangeable (the mesh issued it a plain subject); beta: state on its machine, holds
// the node-shelf seat there.
mesh.Issue(t, mt.MembershipOf("alpha", "desk", true, nil))
mesh.Issue(t, mt.MembershipOf("beta", "desk", false, map[string][]string{"node-shelf": {"list", "clear"}}))
nodeTools := connect(t, "node-tools", "desk")
stop, err := runtime.Run(nodeTools, []runtime.Served{
{Module: "alpha", Entrypoints: []string{mt.Fixture("many-alpha.serve.mjs")}},
{Module: "beta", Entrypoints: []string{mt.Fixture("many-beta.serve.mjs")}},
}, nil, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
// Nothing may be asked for a roster or a module's `tools` any more (ADR 0197): counted.
var asked atomic.Int32
watcher := connect(t, "watcher", "")
for _, subject := range []string{"mesh.mod.*.tool.tools", "mesh.mod.*.tool.tools.*", "mesh.mod.mesh-catalog.>"} {
stop, err := watcher.Raw(subject, func(string, []byte) []byte { asked.Add(1); return nil })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
// The controller: its records as JSON, as its seat verbs answer them, and its own seat's verbs
// announced on the bus like every runtime's.
controller := connect(t, "mesh-controller", "bench")
out := func(s string) map[string]any { return map[string]any{"output": s, "ok": true} }
serve := func(verb string, answer func() any) {
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
return answer(), nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
serve("nodes", func() any {
return out("the bus's user list leaves out 2 user(s)\n" +
`[{"name":"bench","heard":"3m ago","mode":"converged","id":"1f2e"},{"name":"desk","heard":"here","mode":"converged","id":"9a8b"}]`)
})
var gammaOn atomic.Value
gammaOn.Store("[]")
serve("modules", func() any {
return out(fmt.Sprintf(`[{"module":"alpha","on":["desk"],"tools":true},{"module":"beta","on":["desk"],"tools":true},`+
`{"module":"gamma","on":%s,"tools":true},{"module":"delta","on":["desk"],"tools":false},`+
`{"module":"epsilon","on":["bench"],"tools":true},{"module":"mesh-controller","on":["bench"],"tools":true}]`, gammaOn.Load().(string)))
})
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-controller", ID: "bench"}, func() []announce.Endpoint {
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller", Scope: "mesh",
// As the live controller announces: a mesh seat's holder names no machine.
Tool: "nodes", Node: "", Description: "Every machine the mesh knows.", Schema: json.RawMessage(`{}`),
Subject: "mesh.seat.mesh-controller.tool.nodes"}}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopAnn)
controller.Flush()
watcher.Flush()
up, err := Serve(NewSurface(nodeTools, "desk.node-tools"), "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer up.Close()
endpoint := "http://" + up.Address + "/mcp"
// Six tools, nothing else.
listed := post(t, endpoint, `{"jsonrpc":"2.0","id":2,"method":"tools/list"}`)["result"].(map[string]any)
var names []string
for _, x := range listed["tools"].([]any) {
name := x.(map[string]any)["name"].(string)
names = append(names, name)
for _, r := range name {
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '_' || r == '-') || len(name) > 64 {
t.Errorf("%q is not a name the API takes", name)
}
}
}
if got := strings.Join(names, ","); got != "mesh_overview,mesh_machine,mesh_search,mesh_describe,mesh_call,mesh_runtimes" {
t.Errorf("announced %s", got)
}
// The overview names the mesh's seats, the machines' seats and the machines.
overview, isErr := call(t, endpoint, "mesh_overview", nil)
if isErr || !strings.Contains(overview, "mesh-controller.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
!strings.Contains(overview, `"bench"`) || !strings.Contains(overview, `"desk"`) {
t.Errorf("overview: %s", overview)
}
// Silence is named only where tools should have answered: epsilon declares tools on bench and
// nothing there announced it; delta declares none and is never a name.
if strings.Contains(overview, "mesh-controller on bench") {
t.Errorf("a mesh seat's holder that announced no machine is called silent on its machine:\n%s", overview)
}
if !strings.Contains(overview, "epsilon on bench") || strings.Contains(overview, "delta") {
t.Errorf("not answering: %s", overview)
}
machine, isErr := call(t, endpoint, "mesh_machine", map[string]any{"node": "desk"})
if isErr || !strings.Contains(machine, "desk/node-shelf.list") || !strings.Contains(machine, "desk/beta.three") ||
!strings.Contains(machine, "desk/alpha.one") {
t.Errorf("machine: %s", machine)
}
// A mesh seat, a node seat, an assignment and an interchangeable module, each by address.
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-controller.nodes"}); isErr || !strings.Contains(got, "converged") {
t.Errorf("mesh seat: %s", got)
}
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "desk/node-shelf.list"}); isErr || !strings.Contains(got, `"a"`) {
t.Errorf("node seat: %s", got)
}
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "desk/beta.three"}); isErr ||
!strings.Contains(got, `"beta": 3`) || !strings.Contains(got, "answered by desk") {
t.Errorf("assignment: %s", got)
}
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "alpha.one"}); isErr || !strings.Contains(got, `"alpha": 1`) {
t.Errorf("interchangeable module: %s", got)
}
// Refused, by name, where the address does not say enough or says the wrong thing.
for address, want := range map[string]string{
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
"desk/mesh-controller.nodes": "held once for the whole mesh",
"bench/beta.three": "beta does not run on bench; it runs on desk",
"nonsense": "is not an address",
} {
got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": address})
if !isErr || !strings.Contains(got, want) {
t.Errorf("%s: %s (want %q)", address, got, want)
}
}
// Discovery itself asked nothing but the bus.
if n := asked.Load(); n != 0 {
t.Errorf("discovery asked a roster or a module's tools %d time(s); it asks the bus", n)
}
// Described without `node`: the address carries the machine.
described, isErr := call(t, endpoint, "mesh_describe", map[string]any{"address": "desk/beta.three"})
if isErr || strings.Contains(described, `"node"`) || !strings.Contains(described, `"address": "desk/beta.three"`) {
t.Errorf("describe: %s", described)
}
// Its schema, which discovery no longer carries for a module's tool, asked of the runtime serving it.
if !strings.Contains(described, `"verbose"`) {
t.Errorf("describe lost the tool's arguments: %s", described)
}
// Search finds across the layers; a module that starts serving after the first answer is found.
if got, _ := call(t, endpoint, "mesh_search", map[string]any{"query": "shelf"}); !strings.Contains(got, "desk/node-shelf.list") {
t.Errorf("search a node seat: %s", got)
}
if got, _ := call(t, endpoint, "mesh_search", map[string]any{"query": "gamma"}); strings.Contains(got, "gamma.given") {
t.Fatalf("gamma was found before it served: %s", got)
}
mesh.Issue(t, mt.MembershipOf("gamma", "desk", false, nil))
gammaOn.Store(`["desk"]`)
late := connect(t, "node-tools", "desk")
stopLate, err := runtime.Run(late, []runtime.Served{{Module: "gamma", Entrypoints: []string{mt.Fixture("env-gamma.serve.mjs")}}},
nil, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stopLate()
var found string
for i := 0; i < 30; i++ {
found, _ = call(t, endpoint, "mesh_search", map[string]any{"query": "gamma given"})
if strings.Contains(found, "desk/gamma.given") {
break
}
time.Sleep(100 * time.Millisecond)
}
if !strings.Contains(found, "desk/gamma.given") {
t.Errorf("a module that arrived later was not found: %s", found)
}
// Only describing a module's tool asks its runtime — once, for the schema discovery no longer carries
// (2026-10-04: every tool's schema made a machine's discovery answer outgrow the bus).
if n := asked.Load(); n != 1 {
t.Errorf("asked a module's tools %d time(s); describing one tool asks once, and discovery never", n)
}
// The old names still answer, unannounced.
if got, isErr := call(t, endpoint, "alpha.one", nil); isErr || !strings.Contains(got, `"alpha": 1`) {
t.Errorf("an old name: %s", got)
}
}
func TestTheControllersRecordsAreReadAsJSON(t *testing.T) {
if got := jsonIn("a warning printed first\n[{\"name\":\"ace\"}]"); got != `[{"name":"ace"}]` {
t.Errorf("an array after a warning: %q", got)
}
if got := jsonIn(`{"seats":[]}`); got != `{"seats":[]}` {
t.Errorf("an object: %q", got)
}
}
+155
View File
@@ -0,0 +1,155 @@
package console
import (
"encoding/json"
"errors"
"fmt"
"regexp"
"strings"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// Tool is a tool as its module — or, for a role's tool, the mesh's records — describes it.
type Tool struct {
Module string
Name string
Description string
Input json.RawMessage
Seat bool
Scope string
Subjects []string
}
// Listing is what the mesh could say about its tools; silence is named, never dropped (design 34 §3).
type Listing struct {
Tools []Tool
NotAnswering []string
}
// Seats is the seats and their verbs, so `<seat>.<verb>` resolves to the role.
type Seats map[string]map[string]bool
func seatsIn(l *Listing) Seats {
out := Seats{}
for _, t := range l.Tools {
if !t.Seat {
continue
}
if out[t.Module] == nil {
out[t.Module] = map[string]bool{}
}
out[t.Module][t.Name] = true
}
return out
}
// toolKey is the key a call uses: a role's when the prefix is a seat declaring that verb.
func toolKey(name string, seats Seats) string {
if strings.HasPrefix(name, "seat:") {
return name
}
dot := strings.Index(name, ".")
if dot < 0 {
return name
}
if seats[name[:dot]][name[dot+1:]] {
return "seat:" + name
}
return name
}
// toolsOn is the flat catalogue (MESH_CONSOLE_FLAT=1): what announced itself on the bus, every
// tool and seat verb, and the assignments with tools that did not (novox/hq ADR 0197).
func toolsOn(conn *bus.Conn) (*Listing, error) {
x, err := indexOn(conn, nil, nil)
if err != nil {
return nil, err
}
return x.Listing, nil
}
// callTool calls `<module>.<tool>[@<node>]`, on the subject the listing names for it when it names one.
func callTool(conn *bus.Conn, key string, args any, seats Seats, l *Listing) (bus.Answered, error) {
name, node, _ := strings.Cut(key, "@")
if !strings.Contains(name, ".") {
return bus.Answered{}, fmt.Errorf("%q does not name a tool: write <module>.<tool>, as `mesh tools` lists "+
"them, or <module>.<tool>@<node> for the instance on one machine", key)
}
resolved := toolKey(name, seats)
if node != "" {
resolved += "@" + node
}
return conn.Ask(resolved, args, subjectListed(name, node, l))
}
func subjectListed(name, node string, l *Listing) string {
if l == nil {
return ""
}
dot := strings.Index(name, ".")
module, tool := name[:dot], name[dot+1:]
for _, t := range l.Tools {
if t.Module == module && t.Name == tool && !t.Seat {
if len(t.Subjects) == 0 {
return ""
}
if node == "" {
return t.Subjects[0]
}
for _, s := range t.Subjects {
if strings.HasSuffix(s, "."+node) {
return s
}
}
return ""
}
}
return ""
}
var (
noResponders = regexp.MustCompile(`(?i)no responders|503`)
// The bus's own two refusals, by their whole phrase. "authorization" alone also matched a tool's own
// answer that merely mentions the word — the runtime refusing a state value with an Authorization
// header (novox/hq ADR 0201) read as "this account may not call", which sent the reader the wrong way.
refused = regexp.MustCompile(`(?i)permissions violation|authorization violation`)
timedOut = regexp.MustCompile(`(?i)timeout`)
)
// controllerSeat is the seat whose holder keeps what came of every call (novox/hq issue 265).
const controllerSeat = "mesh-controller"
// whyItFailed says why a call failed, so the remedy is in the words.
func whyItFailed(key string, err error) string {
if err == nil {
err = errors.New("failed")
}
msg := err.Error()
switch {
case noResponders.MatchString(msg):
extra := ""
if strings.HasPrefix(key, "seat:") {
extra = ", or nothing holds that seat"
}
return "nothing serves " + key + ". The module may not be assigned to any machine, or it is down" +
extra + " — `mesh tools` lists what answered."
case refused.MatchString(msg):
return "this account may not call " + key + ". What it may call was fixed when it was issued — a " +
"person's by `operator issue`, the console's by its manifest."
case timedOut.MatchString(msg):
// **A timeout is not a failure** (novox/hq issue 265). Something took the call, so it may
// still be running and may already have done what was asked: a push said this while it pushed.
said := key + " did not answer within " + bus.RequestTimeout.String() + ". That is not a failure: " +
"something is serving it, so it may still be running and may already have done what was " +
"asked — check before repeating a call that changes the mesh."
if strings.HasPrefix(strings.TrimPrefix(key, "seat:"), controllerSeat+".") {
// The controller answers every call within seconds, with an id while it runs, so silence
// from it is an answer lost on the way — which it keeps.
said += " The controller answers every call within seconds, so its answer was lost on the " +
"way; " + controllerSeat + ".calls lists its recent calls and what came of each."
}
return said
}
return key + " failed: " + msg
}
+131
View File
@@ -0,0 +1,131 @@
package console
import (
"bytes"
"encoding/json"
"io"
"net/http"
"strings"
"testing"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
func connect(t *testing.T, module, node string) *bus.Conn {
t.Helper()
c, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: module, Node: node})
if err != nil {
t.Fatal(err)
}
c.Logf = func(string, ...any) {}
t.Cleanup(c.Close)
return c
}
func post(t *testing.T, endpoint string, body string) map[string]any {
t.Helper()
res, err := http.Post(endpoint, "application/json", bytes.NewBufferString(body))
if err != nil {
t.Fatal(err)
}
defer res.Body.Close()
raw, _ := io.ReadAll(res.Body)
var out map[string]any
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatalf("%d %s", res.StatusCode, raw)
}
return out
}
// As node-tools, the runtime serves the bundles and is the console on loopback: the listing is what
// the modules and the mesh's records answered, and a call reaches the module on the machine named.
func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "desk", true, nil))
mesh.Issue(t, mt.MembershipOf("beta", "desk", false, map[string][]string{"node-shelf": {"list", "clear"}}))
nodeTools := connect(t, "node-tools", "desk")
stop, err := runtime.Run(nodeTools, []runtime.Served{
{Module: "alpha", Entrypoints: []string{mt.Fixture("many-alpha.serve.mjs")}},
{Module: "beta", Entrypoints: []string{mt.Fixture("many-beta.serve.mjs")}},
}, nil, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
// The controller's records (ADR 0197): ghost declares tools on desk and nothing announces it.
controller := connect(t, "mesh-controller", "")
stopSeat, _ := controller.HandleSubject("mesh.seat.mesh-controller.tool.modules", func(json.RawMessage) (any, error) {
return map[string]any{"ok": true, "output": `[{"module":"alpha","on":["desk"],"tools":true},` +
`{"module":"beta","on":["desk"],"tools":true},{"module":"ghost","on":["desk"],"tools":true}]`}, nil
})
defer stopSeat()
controller.Flush()
flat := NewSurface(nodeTools, "desk.node-tools")
flat.Flat = true // the whole catalogue, as before ADR 0195: still reachable, no longer announced
up, err := Serve(flat, "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer up.Close()
endpoint := "http://" + up.Address + "/mcp"
init := post(t, endpoint, `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{}}}`)
if !strings.Contains(init["result"].(map[string]any)["instructions"].(string), "reached as desk.node-tools") {
t.Errorf("initialize: %v", init)
}
listed := post(t, endpoint, `{"jsonrpc":"2.0","id":2,"method":"tools/list"}`)["result"].(map[string]any)
var names []string
for _, x := range listed["tools"].([]any) {
names = append(names, x.(map[string]any)["name"].(string))
}
if got := strings.Join(names, ","); got != "alpha.one,alpha.two,beta.five,beta.four,beta.three,node-shelf.clear,node-shelf.list" {
t.Errorf("listed %s", got)
}
if got := listed["_meta"].(map[string]any)["notAnswering"]; len(got.([]any)) != 1 || got.([]any)[0] != "ghost on desk" {
t.Errorf("not answering: %v", got)
}
for _, x := range listed["tools"].([]any) {
tool := x.(map[string]any)
if tool["name"] == "node-shelf.list" {
schema := tool["inputSchema"].(map[string]any)
if req, _ := schema["required"].([]any); len(req) != 1 || req[0] != "node" {
t.Errorf("a node seat's verb does not require node: %v", schema)
}
}
}
called := post(t, endpoint, `{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"alpha.one","arguments":{"node":"desk"}}}`)
content := called["result"].(map[string]any)["content"].([]any)
var got map[string]any
_ = json.Unmarshal([]byte(content[0].(map[string]any)["text"].(string)), &got)
if got["alpha"] != float64(1) || content[1].(map[string]any)["text"] != "answered by desk" {
t.Errorf("called: %v", called)
}
seat := post(t, endpoint, `{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"node-shelf.list","arguments":{"node":"desk"}}}`)
if text := seat["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string); !strings.Contains(text, `"a"`) {
t.Errorf("seat verb: %v", seat)
}
refused := post(t, endpoint, `{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"node-shelf.list","arguments":{}}}`)
if refused["error"] == nil {
t.Errorf("a node seat's verb was called without its machine: %v", refused)
}
absent := post(t, endpoint, `{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"ghost.boo","arguments":{}}}`)
result := absent["result"].(map[string]any)
if result["isError"] != true || !strings.Contains(result["content"].([]any)[0].(map[string]any)["text"].(string), "nothing serves ghost.boo") {
t.Errorf("an absent tool: %v", absent)
}
res, err := http.Post(endpoint, "application/json", strings.NewReader(`{"jsonrpc":"2.0","method":"notifications/initialized"}`))
if err != nil || res.StatusCode != 202 {
t.Errorf("a notification: %v %v", res, err)
}
}
func TestTheConsoleListensOnLoopbackAndNowhereElse(t *testing.T) {
if _, err := Serve(NewSurface(nil, "x"), "0.0.0.0:0"); err == nil || !strings.Contains(err.Error(), "loopback and nowhere else") {
t.Errorf("a non-loopback console was not refused: %v", err)
}
}
@@ -0,0 +1,151 @@
package console
import (
"encoding/json"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/announce"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq issue 554, reproduced: the laptop's console heard every runtime, then a round in which
// nothing answered discovery in time — and every seat and module call was refused ("it answered
// discovery 2s ago, not now") for the five seconds that round was kept, again and again. Now a call
// is resolved against what the runtime announced last, says how old that is, and reaches the tool; the
// empty round is logged once, is not kept, and the next question asks again.
func TestACallAfterARoundThatHeardNothingUsesWhatWasHeardBefore(t *testing.T) {
keptWas, patienceWas := IndexKept, announce.Patience
IndexKept, announce.Patience = 0, 1200*time.Millisecond
t.Cleanup(func() { IndexKept, announce.Patience = keptWas, patienceWas })
mt.New(t)
// The tools answer on their own connection; discovery is answered on another, which goes quiet.
tools := connect(t, "node-tools", "laptop")
stopTool, err := tools.HandleSubject("mesh.mod.slack.tool.slack_check.laptop", func(json.RawMessage) (any, error) {
return map[string]any{"ok": true}, nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopTool)
stopSeat, err := tools.HandleSubject("mesh.seat.issue-tracker.tool.list", func(json.RawMessage) (any, error) {
return map[string]any{"issues": []int{554}}, nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopSeat)
tools.Flush()
endpoints := []announce.Endpoint{
{Kind: announce.KindTool, Module: "slack", Tool: "slack_check", Node: "laptop", Subject: "mesh.mod.slack.tool.slack_check.laptop"},
{Kind: announce.KindSeat, Module: "issue-tracker", Seat: "issue-tracker", Tool: "list", Scope: "mesh", Node: "laptop",
Schema: json.RawMessage(`{}`), Subject: "mesh.seat.issue-tracker.tool.list"},
}
announcing := connect(t, "node-tools", "laptop")
stopAnnounce, err := announce.Serve(announcing, announce.Service{Name: "node-tools", ID: "laptop",
Metadata: map[string]string{"node": "laptop"}}, func() []announce.Endpoint { return endpoints })
if err != nil {
t.Fatal(err)
}
announcing.Flush()
console := connect(t, "node-tools", "laptop-console")
var mu sync.Mutex
var logged []string
console.Logf = func(f string, a ...any) {
mu.Lock()
defer mu.Unlock()
logged = append(logged, fmt.Sprintf(f, a...))
}
said := func(words string) int {
mu.Lock()
defer mu.Unlock()
n := 0
for _, l := range logged {
if strings.Contains(l, words) {
n++
}
}
return n
}
surface := NewSurface(console, "laptop.node-tools")
up, err := Serve(surface, "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer up.Close()
endpoint := "http://" + up.Address + "/mcp"
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "laptop/slack.slack_check"}); isErr || strings.Contains(got, "announced") {
t.Fatalf("while discovery hears the runtime: %s", got)
}
// Discovery goes quiet: no PING arrives at all.
stopAnnounce()
announcing.Flush()
for _, address := range []string{"laptop/slack.slack_check", "issue-tracker.list"} {
got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": address})
if isErr || !strings.Contains(got, "found from what node-tools on laptop announced") ||
!strings.Contains(got, "ago: it did not answer the latest round of discovery") {
t.Errorf("%s after a round that heard nothing: %s", address, got)
}
}
if got, isErr := call(t, endpoint, "mesh_describe", map[string]any{"address": "issue-tracker.list"}); isErr || !strings.Contains(got, `"found"`) {
t.Errorf("describe after a round that heard nothing: %s", got)
}
if n := said("discovery heard nothing at all"); n != 1 {
t.Errorf("the empty round was logged %d time(s), want once: %v", n, logged)
}
// A runtime really gone: the call is sent on what it announced, and fails with the bus's own refusal,
// saying where the address was found.
stopTool()
tools.Flush()
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "laptop/slack.slack_check"}); !isErr ||
!strings.Contains(got, "nothing serves") || !strings.Contains(got, "found from what node-tools on laptop announced") {
t.Errorf("a gone runtime's tool: %s", got)
}
surface.mu.Lock()
kept := surface.idx != nil && !surface.idx.Discovery.Empty() && time.Since(surface.idxAt) < time.Second
surface.mu.Unlock()
if kept {
t.Error("a round that heard nothing was kept as the answer")
}
// Discovery answers again: the calls are plain again, and that is logged once.
stopAgain, err := announce.Serve(announcing, announce.Service{Name: "node-tools", ID: "laptop",
Metadata: map[string]string{"node": "laptop"}}, func() []announce.Endpoint { return endpoints })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopAgain)
announcing.Flush()
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "issue-tracker.list"}); isErr || strings.Contains(got, "announced") {
t.Errorf("once discovery hears the runtime again: %s", got)
}
if n := said("discovery hears every runtime again"); n != 1 {
t.Errorf("the recovery was logged %d time(s): %v", n, logged)
}
}
// An address known only from an earlier answer says how old it is; one heard fresh on its machine is
// plain, even where the same module is known only from an earlier answer on another.
func TestAnAddressKnownOnlyFromAnEarlierAnswerSaysHowOld(t *testing.T) {
at := time.Now().Add(-20 * time.Second)
x := &index{Modules: map[string]*moduleInfo{"slack": {Module: "slack", On: []string{"laptop"},
Tools: []Tool{{Module: "slack", Name: "slack_check"}}}}, Recorded: map[string][]string{},
Stale: map[string]staleFrom{"mod:slack@laptop": {Runtime: "node-tools", Machine: "laptop", At: at}},
Discovery: announce.Discovery{Pinged: 0}}
got, err := resolve(x, "laptop/slack.slack_check")
if err != nil || got.Stale == nil || !strings.Contains(got.Stale.say(), "announced 20s ago") {
t.Fatalf("%+v %v", got, err)
}
// A module heard fresh on one machine and only earlier on another: the fresh one is plain.
x.Modules["slack"].On = []string{"desk", "laptop"}
if got, err := resolve(x, "desk/slack.slack_check"); err != nil || got.Stale != nil {
t.Errorf("a fresh machine's module marked stale: %+v %v", got, err)
}
}
+125
View File
@@ -0,0 +1,125 @@
package console
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"strconv"
"strings"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
// bodyLimit is the most a request body may be.
const bodyLimit = 1 << 20
var loopback = map[string]bool{"127.0.0.1": true, "::1": true, "localhost": true, "[::1]": true}
// Listening is a console that listens: where, with the port the machine gave, and how to stop it.
type Listening struct {
Address string
Close func() error
}
// Serve listens on host:port, refused unless the host is loopback — said before binding, so a
// console that would open to a network is a startup failure (ADR 0152).
func Serve(s *Surface, listen string) (*Listening, error) {
at := strings.LastIndex(listen, ":")
if at < 0 {
return nil, fmt.Errorf("%q is not host:port", listen)
}
host, portText := listen[:at], listen[at+1:]
if !loopback[host] {
return nil, fmt.Errorf(`the console listens on loopback and nowhere else (novox/hq ADR 0152): %q is not this `+
"machine's own address — whoever is on the machine owns the mesh there, and nobody else may reach this", host)
}
port, err := strconv.Atoi(portText)
if err != nil || port < 0 || port > 65535 {
return nil, fmt.Errorf("%q is not a port", portText)
}
ln, err := net.Listen("tcp", net.JoinHostPort(strings.Trim(host, "[]"), portText))
if err != nil {
return nil, err
}
server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { route(w, r, s) })}
go func() { _ = server.Serve(ln) }()
bound := ln.Addr().(*net.TCPAddr).Port
return &Listening{Address: host + ":" + strconv.Itoa(bound), Close: func() error { return server.Shutdown(context.Background()) }}, nil
}
func route(w http.ResponseWriter, r *http.Request, s *Surface) {
switch r.URL.Path {
case "/":
w.Header().Set("content-type", "text/plain; charset=utf-8")
_, _ = io.WriteString(w, "the mesh's console: MCP over HTTP at POST /mcp (novox/hq design 34)\n")
return
case "/mcp":
default:
writeJSON(w, 404, map[string]any{"error": "the console serves /mcp and nothing else"})
return
}
switch r.Method {
case http.MethodPost:
case http.MethodDelete:
w.WriteHeader(204) // no session to end
return
default:
w.Header().Set("allow", "POST, DELETE")
w.WriteHeader(405)
return
}
body, err := io.ReadAll(io.LimitReader(r.Body, bodyLimit+1))
if err != nil || len(body) > bodyLimit {
if err == nil {
err = fmt.Errorf("the request is larger than %d bytes", bodyLimit)
}
writeJSON(w, 413, map[string]any{"jsonrpc": "2.0", "id": nil, "error": map[string]any{"code": -32600, "message": err.Error()}})
return
}
trimmed := strings.TrimSpace(string(body))
if strings.HasPrefix(trimmed, "[") {
var batch []Request
if err := json.Unmarshal(body, &batch); err != nil {
writeJSON(w, 400, map[string]any{"jsonrpc": "2.0", "id": nil, "error": map[string]any{"code": -32700, "message": "the body is not JSON"}})
return
}
replies := []*Reply{}
for _, req := range batch {
if reply := s.Handle(req); reply != nil {
replies = append(replies, reply)
}
}
if len(replies) == 0 {
w.WriteHeader(202)
return
}
writeJSON(w, 200, replies)
return
}
var req Request
if err := json.Unmarshal(body, &req); err != nil {
writeJSON(w, 400, map[string]any{"jsonrpc": "2.0", "id": nil, "error": map[string]any{"code": -32700, "message": "the body is not JSON"}})
return
}
reply := s.Handle(req)
if reply == nil {
w.WriteHeader(202)
return
}
writeJSON(w, 200, reply)
}
func writeJSON(w http.ResponseWriter, status int, body any) {
text, err := wire.Marshal(body)
if err != nil {
text, _ = json.Marshal(map[string]any{"error": errors.New("unencodable answer").Error()})
}
w.Header().Set("content-type", "application/json; charset=utf-8")
w.Header().Set("content-length", strconv.Itoa(len(text)))
w.WriteHeader(status)
_, _ = w.Write(text)
}
+331
View File
@@ -0,0 +1,331 @@
// Package console is the mesh's tools as an MCP server on a machine's loopback (novox/hq design 34,
// ADR 0152, ADR 0175 §6): the Go port of node-tools' http.ts, mcp.ts and client.ts. A thin adapter:
// every tool listed is one a module answered for, the schema is the module's, the answer the module's.
package console
import (
"encoding/json"
"os"
"strings"
"sync"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// Protocol is the MCP version spoken to an agent host.
const Protocol = "2025-03-26"
// ListingKept is how long a fetched tool list is kept before the modules are asked again.
var ListingKept = 30 * time.Second
// Request is one JSON-RPC message from a host.
type Request struct {
JSONRPC string `json:"jsonrpc"`
ID json.RawMessage `json:"id,omitempty"`
Method string `json:"method"`
Params json.RawMessage `json:"params,omitempty"`
}
// Reply is one JSON-RPC answer.
type Reply struct {
JSONRPC string `json:"jsonrpc"`
ID json.RawMessage `json:"id"`
Result any `json:"result,omitempty"`
Error *RPCError `json:"error,omitempty"`
}
// RPCError is a protocol-level refusal.
type RPCError struct {
Code int `json:"code"`
Message string `json:"message"`
}
// Surface answers MCP requests over one bus connection, as one account.
type Surface struct {
conn *bus.Conn
who string
mu sync.Mutex
known *Listing
at time.Time
idx *index
idxAt time.Time
// runtimes is every runtime discovery has heard, as it last heard it (mesh_runtimes).
runtimes map[string]*runtimeSeen
// roundSaid is what the last round logged was missing, so a round is logged once (novox/hq issue 554).
roundSaid string
// Flat announces the whole catalogue, as the console did before ADR 0195: for a person reading
// it or a client that wants it. Off by default; MESH_CONSOLE_FLAT=1 turns it on.
Flat bool
}
// NewSurface is the surface over a connection, as `who`.
func NewSurface(conn *bus.Conn, who string) *Surface {
return &Surface{conn: conn, who: who, Flat: os.Getenv("MESH_CONSOLE_FLAT") == "1"}
}
func (s *Surface) listing() (*Listing, error) {
s.mu.Lock()
if s.known != nil && time.Since(s.at) <= ListingKept {
l := s.known
s.mu.Unlock()
return l, nil
}
s.mu.Unlock()
l, err := toolsOn(s.conn)
if err != nil {
return nil, err
}
s.mu.Lock()
s.known, s.at = l, time.Now()
s.mu.Unlock()
return l, nil
}
func isNotification(id json.RawMessage) bool {
t := strings.TrimSpace(string(id))
return t == "" || t == "null"
}
func answer(id json.RawMessage, result any) *Reply {
return &Reply{JSONRPC: "2.0", ID: idOrNull(id), Result: result}
}
func refuse(id json.RawMessage, code int, message string) *Reply {
return &Reply{JSONRPC: "2.0", ID: idOrNull(id), Error: &RPCError{Code: code, Message: message}}
}
func idOrNull(id json.RawMessage) json.RawMessage {
if isNotification(id) {
return json.RawMessage("null")
}
return id
}
// Handle answers one request; nil for a notification, which expects none.
func (s *Surface) Handle(r Request) *Reply {
notification := isNotification(r.ID)
switch r.Method {
case "initialize":
return answer(r.ID, map[string]any{
"protocolVersion": Protocol,
"capabilities": map[string]any{"tools": map[string]any{}},
"serverInfo": map[string]any{"name": "mesh", "version": "1"},
"instructions": s.instructions(),
})
case "notifications/initialized":
return nil
case "ping":
if notification {
return nil
}
return answer(r.ID, map[string]any{})
case "tools/list":
if !s.Flat {
return answer(r.ID, map[string]any{"tools": discovery()})
}
l, err := s.listing()
if err != nil {
return refuse(r.ID, -32603, "the mesh's discovery failed: "+err.Error())
}
tools := make([]map[string]any, 0, len(l.Tools))
inputs := map[string]map[string]json.RawMessage{} // module → tool → input, asked once per module
for i, t := range l.Tools {
if !t.Seat && len(t.Input) == 0 {
if inputs[t.Module] == nil {
inputs[t.Module] = toolInputs(s.conn, t.Module)
}
l.Tools[i].Input = inputs[t.Module][t.Name]
t = l.Tools[i]
}
var schema map[string]any
switch {
case t.Seat && t.Scope != "node":
schema = asSchema(t.Input)
case t.Seat:
schema = withNode(asSchema(t.Input), "the machine whose seat answers; required, the seat is held once per machine", true)
default:
schema = withNode(asSchema(t.Input), "", false)
}
description := t.Description
if description == "" {
description = t.Name + ", served by " + t.Module
}
tools = append(tools, map[string]any{"name": t.Module + "." + t.Name, "description": description, "inputSchema": schema})
}
return answer(r.ID, map[string]any{"tools": tools, "_meta": map[string]any{"notAnswering": l.NotAnswering}})
case "tools/call":
var p struct {
Name string `json:"name"`
Arguments map[string]any `json:"arguments"`
}
_ = json.Unmarshal(r.Params, &p)
if isDiscovery(p.Name) {
if p.Arguments == nil {
p.Arguments = map[string]any{}
}
return answer(r.ID, s.discover(p.Name, p.Arguments))
}
args := map[string]any{}
for k, v := range p.Arguments {
args[k] = v
}
l, _ := s.listing()
var roles Seats
if l != nil {
roles = seatsIn(l)
}
bare, _, _ := strings.Cut(p.Name, "@")
isSeatVerb := roles != nil && strings.HasPrefix(toolKey(bare, roles), "seat:")
nodeScoped := false
if isSeatVerb && l != nil {
for _, t := range l.Tools {
if t.Seat && t.Scope == "node" && t.Module+"."+t.Name == bare {
nodeScoped = true
}
}
}
takesNode := !isSeatVerb || nodeScoped
node := ""
if takesNode {
if n, ok := args["node"].(string); ok {
node = n
}
delete(args, "node")
}
if nodeScoped && node == "" && !strings.Contains(p.Name, "@") {
return refuse(r.ID, -32602, p.Name+" is a machine's seat's verb: name the machine with `node`")
}
name := p.Name
if node != "" && !strings.Contains(p.Name, "@") {
name = p.Name + "@" + node
}
got, err := callTool(s.conn, name, args, roles, l)
if err != nil {
return answer(r.ID, map[string]any{
"content": []map[string]any{{"type": "text", "text": whyItFailed(name, err)}},
"isError": true,
})
}
content := []map[string]any{{"type": "text", "text": pretty(got.Result)}}
if got.Node != "" {
content = append(content, map[string]any{"type": "text", "text": "answered by " + got.Node})
}
return answer(r.ID, map[string]any{"content": content})
}
if notification {
return nil
}
return refuse(r.ID, -32601, "mesh's MCP surface has no "+r.Method)
}
// pretty is a module's answer as JSON text, indented as JSON.stringify(result, null, 2) writes it.
func pretty(raw json.RawMessage) string {
if len(raw) == 0 {
return "null"
}
var v any
if json.Unmarshal(raw, &v) != nil {
return string(raw)
}
b, err := json.MarshalIndent(v, "", " ")
if err != nil {
return string(raw)
}
return strings.NewReplacer(`<`, "<", `>`, ">", `&`, "&").Replace(string(b))
}
// asSchema is a module's declared input as a JSON schema: wrapped when it is a bare map of
// properties, passed through when it is a schema, empty when nothing was declared.
func asSchema(raw json.RawMessage) map[string]any {
var given map[string]any
if json.Unmarshal(raw, &given) != nil || given == nil {
return map[string]any{"type": "object", "properties": map[string]any{}}
}
if given["type"] == "object" {
return given
}
if _, has := given["properties"]; has {
return given
}
if len(given) == 0 {
return map[string]any{"type": "object", "properties": map[string]any{}}
}
return map[string]any{"type": "object", "properties": given}
}
// withNode adds the optional — or, for a node seat, required — `node` argument (ADR 0159).
func withNode(schema map[string]any, description string, required bool) map[string]any {
properties := map[string]any{}
if p, ok := schema["properties"].(map[string]any); ok {
for k, v := range p {
properties[k] = v
}
}
if _, has := properties["node"]; !has {
if description == "" {
description = "the machine to ask, when this module runs on several; else whichever answers, and the answer says which"
}
properties["node"] = map[string]any{"type": "string", "description": description}
}
out := map[string]any{}
for k, v := range schema {
out[k] = v
}
out["type"] = "object"
out["properties"] = properties
if required {
have := []any{}
if r, ok := schema["required"].([]any); ok {
have = r
}
hasNode := false
for _, x := range have {
hasNode = hasNode || x == "node"
}
if !hasNode {
have = append(have, "node")
}
out["required"] = have
}
return out
}
// instructions is what an agent host is told about this surface when it connects.
func (s *Surface) instructions() string {
if s.Flat {
return "These are the tools of a Novox mesh, reached as " + s.who + ". Every call goes to the module " +
"that serves it; what may be called was fixed when this account was issued, so a " +
"refusal means the account, not the tool. The list is what the running modules " +
"answered, plus every role's tools from the mesh's records — the mesh's own verbs " +
"(mesh-controller.status, .push, .assign …) among them; a module that did not answer " +
"is named in the list's _meta and can still be called by <module>.<tool>."
}
return "The tools of a Novox mesh, reached as " + s.who + ", found by address rather than listed " +
"whole (novox/hq ADR 0195). mesh_overview shows the mesh's seats and machines; mesh_machine one " +
"machine's seats and modules; mesh_search finds a tool by words; mesh_describe gives one tool's " +
"arguments; mesh_call calls it; mesh_runtimes says which machines' runtimes answered discovery, how fast " +
"and how much — ask it when an address is said to be missing or on another machine. " + grammar + " What may be called was fixed when this account " +
"was issued, so a refusal means the account, not the tool."
}
// toolInputs is every tool's input schema of one module, asked of a runtime that serves it.
func toolInputs(conn *bus.Conn, module string) map[string]json.RawMessage {
out := map[string]json.RawMessage{}
got, err := conn.Ask(module+".tools", map[string]any{}, "")
if err != nil {
return out
}
var answer struct {
Tools []struct {
Name string `json:"name"`
Input json.RawMessage `json:"input"`
} `json:"tools"`
}
if json.Unmarshal(got.Result, &answer) == nil {
for _, t := range answer.Tools {
out[t.Name] = t.Input
}
}
return out
}
@@ -0,0 +1,52 @@
package console
import (
"encoding/json"
"strings"
"testing"
)
// novox/hq issue 244: `node` is taken out only where the address names the machine. A seat held once
// for the mesh takes the machine it acts on as `node`, and is described and called with it — on
// 2026-10-05 the controller's push was described as taking nothing, and a push naming one machine
// ran as a push of every machine behind.
func TestAMeshSeatsNodeIsItsOwnArgument(t *testing.T) {
push := Tool{Module: "mesh-controller", Name: "push", Seat: true, Scope: "mesh",
Input: json.RawMessage(`{"type":"object","properties":{"node":{"type":"string"},"behind":{"type":"string"}}}`)}
mesh := target{Address: "mesh-controller.push", Tool: push, Seat: true}
props := schemaAsPassed(mesh)["properties"].(map[string]any)
if _, has := props["node"]; !has {
t.Fatalf("a mesh seat's verb was described without its node: %v", props)
}
sent, err := argumentsAsSent(mesh, map[string]any{"node": "g1"})
if err != nil || sent["node"] != "g1" {
t.Fatalf("a mesh seat's verb was called without the machine it was given: %v %v", sent, err)
}
// An argument the seat's verb does not declare is refused, naming it, never sent to be dropped.
if _, err := argumentsAsSent(mesh, map[string]any{"machine": "g1"}); err == nil || !strings.Contains(err.Error(), "machine") {
t.Fatalf("an undeclared argument was sent: %v", err)
}
// A machine's seat: the address names the machine, so `node` is out of the schema; given the same
// machine again it is taken out, given another it is refused.
shelf := Tool{Module: "node-shelf", Name: "list", Seat: true, Scope: "node",
Input: json.RawMessage(`{"type":"object","properties":{"limit":{"type":"string"}}}`)}
onDesk := target{Address: "desk/node-shelf.list", Node: "desk", Tool: shelf, Seat: true}
if _, has := schemaAsPassed(onDesk)["properties"].(map[string]any)["node"]; has {
t.Fatal("a machine's seat was described with a node the address already names")
}
if sent, err := argumentsAsSent(onDesk, map[string]any{"node": "desk", "limit": "3"}); err != nil || sent["node"] != nil || sent["limit"] != "3" {
t.Fatalf("the same machine named twice: %v %v", sent, err)
}
if _, err := argumentsAsSent(onDesk, map[string]any{"node": "bench"}); err == nil || !strings.Contains(err.Error(), "bench") {
t.Fatalf("another machine in the arguments than in the address was dropped: %v", err)
}
// A module's tool is its own: what it takes beyond node is the module's to judge.
beta := target{Address: "desk/beta.three", Node: "desk", Tool: Tool{Module: "beta", Name: "three"}}
if sent, err := argumentsAsSent(beta, map[string]any{"node": "desk", "verbose": true}); err != nil || sent["verbose"] != true || sent["node"] != nil {
t.Fatalf("a module's tool: %v %v", sent, err)
}
}
@@ -0,0 +1,57 @@
package console
import (
"strings"
"testing"
"github.com/novox/mesh-tools/node-tools/internal/announce"
)
// **Issue 287**: mesh-delivery is the delivery's seat, held once for the mesh, and the module holding it,
// which answers the seat's verbs with tools of the same names and has tools of its own. Discovery put the
// module's tool and the seat's verb under one key, so the seat was listed with no verb; and every address
// with the name resolved to the seat — `mesh-delivery.deliveries` and `novox/mesh-delivery.delivery_status`
// alike were refused, the seat's verbs and the module's tools both out of reach.
func TestASeatAndAModuleOfOneNameAreEachReached(t *testing.T) {
var endpoints []announce.Endpoint
for _, tool := range []string{"deliveries", "delivery_status"} {
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindTool, Module: "mesh-delivery", Tool: tool,
Node: "anchor", Subject: "mesh.mod.mesh-delivery.tool." + tool + ".anchor"})
}
for _, verb := range []string{"deliveries", "stalled"} {
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-delivery", Seat: "mesh-delivery",
Tool: verb, Scope: "mesh", Subject: "mesh.seat.mesh-delivery.tool." + verb})
}
x, _, _ := indexOfEndpoints(endpoints)
s := x.seat("mesh-delivery")
if s == nil || len(s.Verbs) != 2 {
t.Fatalf("the seat is listed with verbs %v, not its two", s)
}
if m := x.Modules["mesh-delivery"]; m == nil || len(m.Tools) != 2 {
t.Fatalf("the module is listed with tools %v, not its two", m)
}
for address, want := range map[string]struct {
seat bool
key string
}{
"mesh-delivery.deliveries": {true, "seat:mesh-delivery.deliveries"},
"mesh-delivery.stalled": {true, "seat:mesh-delivery.stalled"},
"anchor/mesh-delivery.delivery_status": {false, "mesh-delivery.delivery_status@anchor"},
"anchor/mesh-delivery.deliveries": {false, "mesh-delivery.deliveries@anchor"},
} {
got, err := resolve(x, address)
if err != nil {
t.Errorf("%s: %v", address, err)
continue
}
if got.Seat != want.seat || got.Key != want.key {
t.Errorf("%s resolves to %+v, not the %s", address, got, want.key)
}
}
// A module tool of the name without a machine: the module keeps state on its machine, so it says so.
if _, err := resolve(x, "mesh-delivery.delivery_status"); err == nil || !strings.Contains(err.Error(), "<node>/") {
t.Errorf("the module's tool without a machine: %v", err)
}
}
@@ -0,0 +1,37 @@
package console
import (
"errors"
"strings"
"testing"
)
// A refusal is said only for the bus's own refusals; a tool's answer that mentions authorization is the
// tool's answer, not the account's.
func TestARefusalIsSaidOnlyForTheBussOwn(t *testing.T) {
for msg, refusal := range map[string]bool{
`nats: Permissions Violation for Publish to "mesh.mod.x.tool.y"`: true,
"nats: Authorization Violation": true,
`claude-code's servers.all.x carries a field "Authorization", which names a credential`: false,
} {
got := strings.HasPrefix(whyItFailed("x.y", errors.New(msg)), "this account may not call")
if got != refusal {
t.Errorf("%q read as a refusal: %v, want %v", msg, got, refusal)
}
}
}
// **A timeout says it is not a failure** (novox/hq issue 265): a push that outlasted the wait had
// pushed, and the caller read "did not answer in time". And from the controller, where its answer is.
func TestATimeoutIsNotAFailure(t *testing.T) {
got := whyItFailed("seat:mesh-controller.push", errors.New("timeout"))
for _, want := range []string{"not a failure", "may already have done", "mesh-controller.calls"} {
if !strings.Contains(got, want) {
t.Errorf("a controller timeout does not say %q: %s", want, got)
}
}
if other := whyItFailed("novox/postgres.postgres_list_databases", errors.New("timeout")); !strings.Contains(other, "not a failure") ||
strings.Contains(other, "calls") {
t.Errorf("a module's timeout: %s", other)
}
}
+322
View File
@@ -0,0 +1,322 @@
package console
// What discovery heard from each runtime (2026-10-05): the console said a module ran nowhere when the
// largest runtime's answer arrived after the window closed, and nothing showed that it had. Every
// round of discovery is remembered per runtime, so a runtime heard before and not now is named, and
// `mesh_runtimes` says who answered, how fast and how much.
import (
"fmt"
"log"
"sort"
"strings"
"time"
"github.com/novox/mesh-tools/node-tools/internal/announce"
)
// RuntimeForgotten is how long a runtime heard once is expected again: within it, its silence is said
// wherever an address it might answer is asked for; after it, it is listed only by mesh_runtimes.
var RuntimeForgotten = 15 * time.Minute
// runtimeSeen is one runtime instance as discovery last heard it.
type runtimeSeen struct {
Name string
ID string
Machine string
LastHeard time.Time
Took time.Duration
Bytes int
Modules int
Tools int
SeatVerbs int
Shortened string
Answered bool // in the latest round
Why string // why not, when it did not
// Reused is true when its PING said its announcement was unchanged, so it was not fetched again.
Reused bool
// Last is its last whole answer: what a round that does not hear it uses instead, and what an
// unchanged digest stands for (novox/hq issue 554).
Last announce.Heard
}
func runtimeKey(name, id string) string { return name + "/" + id }
// seenOf is what one answer announced, counted.
func seenOf(h announce.Heard, at time.Time) *runtimeSeen {
modules := map[string]bool{}
tools := map[string]bool{}
verbs := map[string]bool{}
for _, e := range announce.Endpoints(h.Info) {
switch e.Kind {
case announce.KindTool:
modules[e.Module] = true
tools[e.Module+"."+e.Tool] = true
case announce.KindSeat:
verbs[e.Seat+"."+e.Tool] = true
}
}
return &runtimeSeen{Name: h.Info.Name, ID: h.Info.ID, Machine: h.Machine(), LastHeard: at, Took: h.Took,
Bytes: h.Bytes, Modules: len(modules), Tools: len(tools), SeatVerbs: len(verbs), Shortened: h.Shortened, Answered: true,
Reused: h.Reused, Last: h}
}
// held is the answer this console holds for an instance at this digest (announce.Known): an instance
// whose announcement is unchanged is not asked for it again (novox/hq issue 554).
func (s *Surface) held(name, id, digest string) (announce.Heard, bool) {
s.mu.Lock()
defer s.mu.Unlock()
r := s.runtimes[runtimeKey(name, id)]
if r == nil || r.Last.Digest == "" || r.Last.Digest != digest || r.Last.Info.Name == "" {
return announce.Heard{}, false
}
h := r.Last
h.Took, h.Reused = 0, false
return h, true
}
// before is the last answer of every runtime this round did not hear, heard within RuntimeForgotten
// and not since restarted under another instance (novox/hq issue 554).
func (s *Surface) before(d announce.Discovery) []earlier {
heard := map[string]bool{}
running := map[string]bool{}
for _, h := range d.Heard {
heard[runtimeKey(h.Info.Name, h.Info.ID)] = true
if m := h.Machine(); m != "" {
running[h.Info.Name+"@"+m] = true
}
}
s.mu.Lock()
defer s.mu.Unlock()
var out []earlier
for k, r := range s.runtimes {
if heard[k] || r.Last.Info.Name == "" || r.LastHeard.IsZero() || time.Since(r.LastHeard) > RuntimeForgotten {
continue
}
if r.Machine != "" && running[r.Name+"@"+r.Machine] {
continue // restarted: the new instance answered
}
out = append(out, earlier{Heard: r.Last, At: r.LastHeard})
}
sort.Slice(out, func(i, j int) bool {
return runtimeKey(out[i].Heard.Info.Name, out[i].Heard.Info.ID) < runtimeKey(out[j].Heard.Info.Name, out[j].Heard.Info.ID)
})
return out
}
// sayRound logs a round of discovery that did not hear every runtime — once, when what it missed
// changes, and once when every runtime is heard again — so the next case leaves evidence in the
// journal (novox/hq issue 554). Called with s.mu held, after remember.
func (s *Surface) sayRound(x *index) {
d := x.Discovery
var missed []string
for _, u := range x.Unheard {
who := u.Runtime
if u.Machine != "" {
who += " on " + u.Machine
}
missed = append(missed, who)
}
sort.Strings(missed)
shape := strings.Join(missed, ", ")
if d.Empty() {
shape = "empty|" + shape
}
if shape == s.roundSaid {
return
}
s.roundSaid = shape
reused := 0
for _, h := range d.Heard {
if h.Reused {
reused++
}
}
logf := log.Printf
if s.conn != nil && s.conn.Logf != nil {
logf = s.conn.Logf
}
window := fmt.Sprintf("waited %s (window %s, patience %s)", d.Waited.Round(time.Millisecond), announce.Window, announce.Patience)
held := ""
if len(x.Earlier) > 0 {
held = fmt.Sprintf("; what %d of them announced earlier is used, with its age", len(x.Earlier))
}
switch {
case d.Empty():
logf("[mesh-tools] discovery heard nothing at all: no PING arrived; %s; not heard: %s%s; asking again at the next question",
window, orNone(missed), held)
case len(missed) > 0:
logf("[mesh-tools] discovery did not hear every runtime: %d PING answer(s), %d announcement(s) (%d unchanged, not fetched); %s; not heard: %s%s",
d.Pinged, len(d.Heard), reused, window, shape, held)
default:
logf("[mesh-tools] discovery hears every runtime again: %d PING answer(s), %d announcement(s) (%d unchanged, not fetched); %s",
d.Pinged, len(d.Heard), reused, window)
}
}
func orNone(xs []string) string {
if len(xs) == 0 {
return "none heard before either"
}
return strings.Join(xs, ", ")
}
// remember records a round of discovery and adds to the index every runtime heard lately that did not
// answer this time. Called with s.mu held.
func (s *Surface) remember(x *index) {
if s.runtimes == nil {
s.runtimes = map[string]*runtimeSeen{}
}
at := x.Discovery.At
if at.IsZero() {
at = time.Now().UTC()
}
heard := map[string]bool{}
for _, h := range x.Discovery.Heard {
k := runtimeKey(h.Info.Name, h.Info.ID)
seen := seenOf(h, at)
if heard[k] {
// Two instances under one name and id: both counted, the slower answer's time kept.
prev := s.runtimes[k]
seen.Modules += prev.Modules
seen.Tools += prev.Tools
seen.SeatVerbs += prev.SeatVerbs
seen.Bytes += prev.Bytes
if prev.Took > seen.Took {
seen.Took = prev.Took
}
}
heard[k] = true
s.runtimes[k] = seen
}
// A runtime restarted answers under a new instance id: the old one is gone, not silent, and saying
// it was missed would hide nothing but cry wolf after every restart.
now := map[string]bool{}
for k := range heard {
now[s.runtimes[k].Name+"@"+s.runtimes[k].Machine] = true
}
for k, r := range s.runtimes {
if !heard[k] && r.Machine != "" && now[r.Name+"@"+r.Machine] {
delete(s.runtimes, k)
}
}
silent := map[string]string{}
for _, u := range x.Discovery.Silent {
silent[runtimeKey(u.Name, u.ID)] = fmt.Sprintf("it answered PING; what it serves did not arrive within %s", announce.Patience)
}
for k, r := range s.runtimes {
if heard[k] {
continue
}
r.Answered = false
used := ""
if at, ok := x.Earlier[k]; ok {
used = fmt.Sprintf("; what it announced %s ago is used", time.Since(at).Round(time.Second))
}
if why, ok := silent[k]; ok {
r.Why = why
if used != "" {
for i := range x.Unheard {
if x.Unheard[i].Runtime == r.Name && x.Unheard[i].Machine == r.Machine && x.Unheard[i].Why == why {
x.Unheard[i].Why += used
}
}
}
continue // already in the index's unheard
}
r.Why = "it did not answer this round of discovery"
if time.Since(r.LastHeard) <= RuntimeForgotten {
x.Unheard = append(x.Unheard, unheard{Runtime: r.Name, Machine: r.Machine,
Why: fmt.Sprintf("it answered discovery %s ago, not now%s", time.Since(r.LastHeard).Round(time.Second), used)})
}
}
for _, u := range x.Discovery.Silent {
k := runtimeKey(u.Name, u.ID)
if _, known := s.runtimes[k]; !known {
s.runtimes[k] = &runtimeSeen{Name: u.Name, ID: u.ID, Machine: u.Machine, Why: silent[k]}
}
}
}
// runtimesAnswer is mesh_runtimes: discovery asked now, and every runtime as it was heard.
func (s *Surface) runtimesAnswer() map[string]any {
x, err := s.indexAsked(true)
if err != nil {
return failure("the mesh's discovery failed: " + err.Error())
}
s.mu.Lock()
var all []runtimeSeen
for _, r := range s.runtimes {
all = append(all, *r)
}
s.mu.Unlock()
sort.Slice(all, func(i, j int) bool {
if all[i].Machine != all[j].Machine {
return all[i].Machine < all[j].Machine
}
return runtimeKey(all[i].Name, all[i].ID) < runtimeKey(all[j].Name, all[j].ID)
})
now := time.Now()
var answered, notHeard []map[string]any
onMachine := map[string]bool{}
for _, r := range all {
entry := map[string]any{"runtime": r.Name, "instance": r.ID}
if r.Machine != "" {
entry["machine"] = r.Machine
}
if !r.LastHeard.IsZero() {
entry["last heard"] = r.LastHeard.Format(time.RFC3339)
entry["last heard, ago"] = now.Sub(r.LastHeard).Round(time.Second).String()
entry["took"] = r.Took.Round(time.Millisecond).String()
entry["bytes"] = r.Bytes
entry["modules"] = r.Modules
entry["tools"] = r.Tools
entry["seat verbs"] = r.SeatVerbs
entry["shortened"] = r.Shortened != ""
if r.Shortened != "" {
entry["shortened, how"] = r.Shortened
}
if r.Reused && r.Answered {
entry["announcement"] = "unchanged (its PING's digest), so not fetched again; took is its PING's"
}
}
if r.Answered {
onMachine[r.Machine] = true
answered = append(answered, entry)
} else {
entry["why"] = r.Why
if r.LastHeard.IsZero() {
entry["last heard"] = "never in full, since this console started"
}
notHeard = append(notHeard, entry)
}
}
for _, m := range x.Machines {
if !onMachine[m] && !machineListed(notHeard, m) {
notHeard = append(notHeard, map[string]any{"machine": m,
"why": "the mesh knows this machine and no runtime on it has answered discovery since this console started"})
}
}
if answered == nil {
answered = []map[string]any{}
}
if notHeard == nil {
notHeard = []map[string]any{}
}
return answerText(map[string]any{
"asked": x.Discovery.At.Format(time.RFC3339),
"waited": x.Discovery.Waited.Round(time.Millisecond).String(),
"window": fmt.Sprintf("at least %s; up to %s for a runtime that answered PING, or until one does", announce.Window, announce.Patience),
"answered": answered,
"not heard": notHeard,
})
}
func machineListed(entries []map[string]any, machine string) bool {
for _, e := range entries {
if e["machine"] == machine {
return true
}
}
return false
}
@@ -0,0 +1,223 @@
package console
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-tools/node-tools/internal/announce"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// slowRuntime answers PING at once and INFO after `lag`, or never with a negative lag.
func slowRuntime(t *testing.T, conn *bus.Conn, node string, lag time.Duration, endpoints []announce.Endpoint) {
t.Helper()
id := micro.ServiceIdentity{Name: "node-tools", ID: node, Version: announce.Version, Metadata: map[string]string{"node": node}}
ping, _ := json.Marshal(micro.Ping{ServiceIdentity: id, Type: micro.PingResponseType})
stop, err := conn.Raw("$SRV.PING", func(string, []byte) []byte { return ping })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
// Asked of every service, and of this instance by name and id — as discovery asks it (issue 554).
info := func(string, []byte) []byte {
if lag < 0 {
return nil
}
time.Sleep(lag)
body, _ := json.Marshal(announce.Info(announce.Service{Name: id.Name, ID: id.ID, Metadata: id.Metadata}, endpoints))
return body
}
for _, subject := range []string{"$SRV.INFO", "$SRV.INFO." + id.Name + "." + id.ID} {
stop, err := conn.Raw(subject, info)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
conn.Flush()
}
// The laptop's symptom (2026-10-05), reproduced: its runtime's one large answer reached the console
// after the 750 ms window, and the console said its modules ran nowhere — "nothing in the mesh is
// called slack" — or ran only on another machine. Now a runtime that said it is there is waited for,
// one that never says what it serves is named wherever an address it might answer is asked for, and
// mesh_runtimes shows who answered, how fast and how much.
func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T) {
keptWas, patienceWas := IndexKept, announce.Patience
// Kept for the test: one round of discovery answers every question but mesh_runtimes, which asks anew.
IndexKept, announce.Patience = time.Minute, 2500*time.Millisecond
t.Cleanup(func() { IndexKept, announce.Patience = keptWas, patienceWas })
mt.New(t)
laptop := connect(t, "node-tools", "laptop")
stopTool, err := laptop.HandleSubject("mesh.mod.slack.tool.slack_check.laptop", func(json.RawMessage) (any, error) {
return map[string]any{"ok": true}, nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopTool)
slowRuntime(t, laptop, "laptop", announce.Window+400*time.Millisecond, []announce.Endpoint{
{Kind: announce.KindTool, Module: "slack", Tool: "slack_check", Node: "laptop", Description: "Is Slack started once?",
Subject: "mesh.mod.slack.tool.slack_check.laptop"},
{Kind: announce.KindTool, Module: "nextcloud-client", Tool: "nextcloud_check", Node: "laptop",
Subject: "mesh.mod.nextcloud-client.tool.nextcloud_check.laptop"},
})
sleeper := connect(t, "node-tools", "sleeper")
slowRuntime(t, sleeper, "sleeper", -1, nil)
desktop := connect(t, "node-tools", "desktop")
slowRuntime(t, desktop, "desktop", 0, []announce.Endpoint{
{Kind: announce.KindTool, Module: "nextcloud-client", Tool: "nextcloud_check", Node: "desktop",
Subject: "mesh.mod.nextcloud-client.tool.nextcloud_check.desktop"},
})
controller := connect(t, "mesh-controller", "bench")
for verb, output := range map[string]string{
"nodes": `[{"name":"laptop"},{"name":"sleeper"},{"name":"desktop"},{"name":"bench"}]`,
"modules": `[{"module":"slack","on":["laptop"],"tools":true},{"module":"nap","on":["sleeper"],"tools":true},` +
`{"module":"nextcloud-client","on":["laptop","desktop","sleeper"],"tools":true}]`,
} {
output := output
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
return map[string]any{"output": output, "ok": true}, nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
controller.Flush()
console := connect(t, "node-tools", "laptop-console")
up, err := Serve(NewSurface(console, "laptop.node-tools"), "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer up.Close()
endpoint := "http://" + up.Address + "/mcp"
// The late answer is waited for: the laptop's module is found and called.
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "laptop/slack.slack_check"}); isErr || !strings.Contains(got, `"ok": true`) {
t.Errorf("a late runtime's module: %s", got)
}
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "laptop/nextcloud-client.nextcloud_check"}); isErr && strings.Contains(got, "does not run on laptop") {
t.Errorf("a late runtime's module was said to run elsewhere: %s", got)
}
machine, _ := call(t, endpoint, "mesh_machine", map[string]any{"node": "laptop"})
if !strings.Contains(machine, "laptop/slack.slack_check") {
t.Errorf("the late runtime's machine lists nothing: %s", machine)
}
// The silent one is named, never called missing.
for address, want := range map[string]string{
"sleeper/nap.doze": "nap is assigned to sleeper in the mesh's records, but nothing that answered discovery serves it: discovery did not hear in full from node-tools on sleeper (it answered PING",
"sleeper/nextcloud-client.nextcloud_check": "nextcloud-client on sleeper did not answer discovery: discovery did not hear in full from node-tools on sleeper",
} {
got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": address})
if !isErr || !strings.Contains(got, want) || strings.Contains(got, "nothing in the mesh is called") || strings.Contains(got, "does not run on") {
t.Errorf("%s: %s\nwant %q", address, got, want)
}
}
machine, _ = call(t, endpoint, "mesh_machine", map[string]any{"node": "sleeper"})
if !strings.Contains(machine, `"incomplete"`) || !strings.Contains(machine, `"modules": []`) {
t.Errorf("a machine whose runtime did not answer: %s", machine)
}
overview, _ := call(t, endpoint, "mesh_overview", nil)
if !strings.Contains(overview, "the runtime node-tools on sleeper") {
t.Errorf("overview does not name the silent runtime: %s", overview)
}
// mesh_runtimes: who answered, how fast, how much; who did not.
got, isErr := call(t, endpoint, "mesh_runtimes", nil)
if isErr {
t.Fatal(got)
}
var report struct {
Answered []map[string]any `json:"answered"`
NotHeard []map[string]any `json:"not heard"`
}
if err := json.Unmarshal([]byte(got), &report); err != nil {
t.Fatalf("%v: %s", err, got)
}
var lap map[string]any
for _, r := range report.Answered {
if r["machine"] == "laptop" {
lap = r
}
}
if lap == nil {
t.Fatalf("the laptop's runtime is not listed as answering: %s", got)
}
took, _ := time.ParseDuration(lap["took"].(string))
if took < announce.Window || lap["bytes"].(float64) <= 0 || lap["modules"].(float64) != 2 || lap["tools"].(float64) != 2 ||
lap["shortened"] != false || lap["last heard"] == nil {
t.Errorf("the laptop's runtime: %v", lap)
}
if !strings.Contains(got, `"machine": "sleeper"`) || !strings.Contains(got, "answered PING") {
t.Errorf("the silent runtime is not listed: %s", got)
}
silent := false
for _, r := range report.NotHeard {
silent = silent || r["machine"] == "sleeper"
}
if !silent {
t.Errorf("sleeper is not among the not heard: %s", got)
}
}
// A runtime heard once and silent now is named in the answers about its machine.
func TestARuntimeHeardBeforeAndNotNowIsNamed(t *testing.T) {
s := &Surface{}
before := &index{Modules: map[string]*moduleInfo{}, Recorded: map[string][]string{}, Discovery: announce.Discovery{At: time.Now().UTC(),
Heard: []announce.Heard{{Info: micro.Info{ServiceIdentity: micro.ServiceIdentity{Name: "node-tools", ID: "laptop",
Metadata: map[string]string{"node": "laptop"}}}, Took: 300 * time.Millisecond, Bytes: 1000}}}}
s.remember(before)
now := &index{Modules: map[string]*moduleInfo{"nextcloud-client": {Module: "nextcloud-client", On: []string{"desktop"},
Tools: []Tool{{Module: "nextcloud-client", Name: "nextcloud_check"}}}}, Recorded: map[string][]string{},
Discovery: announce.Discovery{At: time.Now().UTC(), Pinged: 1}} // desktop's runtime answered
s.remember(now)
if len(now.Unheard) != 1 || now.Unheard[0].Machine != "laptop" {
t.Fatalf("unheard: %+v", now.Unheard)
}
_, err := resolve(now, "laptop/nextcloud-client.nextcloud_check")
if err == nil || !strings.Contains(err.Error(), "nextcloud-client on laptop did not answer discovery") ||
!strings.Contains(err.Error(), "answered discovery") {
t.Errorf("%v", err)
}
_, err = resolve(now, "laptop/slack.slack_check")
if err == nil || strings.Contains(err.Error(), "nothing in the mesh is called") || !strings.Contains(err.Error(), "slack may be theirs") {
t.Errorf("%v", err)
}
// Nobody unheard: the plain answers stand.
quiet := &index{Modules: now.Modules, Recorded: map[string][]string{}, Discovery: now.Discovery}
if _, err := resolve(quiet, "laptop/nextcloud-client.nextcloud_check"); err == nil || !strings.Contains(err.Error(), "does not run on laptop; it runs on desktop") {
t.Errorf("%v", err)
}
if _, err := resolve(quiet, "laptop/slack.slack_check"); err == nil || !strings.Contains(err.Error(), "nothing in the mesh is called slack") {
t.Errorf("%v", err)
}
}
// A runtime restarted answers under a new instance id; its old instance is gone, not unheard.
func TestARestartedRuntimeIsNotNamedUnheard(t *testing.T) {
heard := func(id string) *index {
return &index{Modules: map[string]*moduleInfo{}, Recorded: map[string][]string{}, Discovery: announce.Discovery{At: time.Now().UTC(),
Heard: []announce.Heard{{Info: micro.Info{ServiceIdentity: micro.ServiceIdentity{Name: "node-tools", ID: id,
Metadata: map[string]string{"node": "laptop"}}}}}}}
}
s := &Surface{}
s.remember(heard("before-restart"))
after := heard("after-restart")
s.remember(after)
if len(after.Unheard) != 0 {
t.Errorf("a restarted runtime was named unheard: %+v", after.Unheard)
}
if len(s.runtimes) != 1 {
t.Errorf("the old instance is still kept: %d runtimes", len(s.runtimes))
}
}
+272
View File
@@ -0,0 +1,272 @@
package console
// Finding a tool by the words an agent already has (novox/hq ADR 0245).
//
// An agent that wants a unit's log reaches for `journalctl`, and one that wants a container's for `docker
// logs`; it searches with those words, or with `logs`. Matched against names and descriptions alone, none
// of them found the service manager's `journal` — whose description says "journal", never "logs" or
// "journalctl" — and the agent went round the mesh over ssh. So each verb and tool says what it replaces,
// and search matches three ways, best first:
//
// 1. **The query is a command a tool replaces** — `journalctl -u x`, `systemctl status x`, `docker ps`:
// every word of the replaced command, in order, among the query's words. The most specific command wins.
// 2. **Every word is in the tool's name, its replaced commands or its description**, a word also matching
// through a short table of the words agents use for the mesh's (logs ⇄ journal, ps ⇄ list). A word in
// the name or a replaced command weighs more than one in the description, and a name the query says
// whole more than one it says in part.
// 3. Nothing else: a search that matches nothing says so.
//
// Ties go to a seat — the place tools belong — then to the shorter name, then by address.
import (
"encoding/json"
"path"
"sort"
"strings"
"unicode"
)
// synonyms are the words an agent uses for the mesh's own, each matching the other's (a word, not part
// of one: `log` must not match `login`). Kept short: a command is said by the tool that replaces it.
var synonyms = map[string][]string{
"logs": {"log", "journal"},
"log": {"logs", "journal"},
"journal": {"logs", "log"},
"ps": {"list", "running"},
"ls": {"list"},
"list": {"ps", "ls"},
"restart": {"restarts"},
"service": {"unit", "units"},
"services": {"unit", "units"},
"unit": {"service"},
"container": {"containers", "docker"},
"firewall": {"filter", "nftables"},
"hosts": {"hostname"},
}
// hit is one match as mesh_search answers it.
type hit struct {
Address string `json:"address"`
Does string `json:"does"`
// InsteadOf is the command the query matched, when it matched one the tool replaces.
InsteadOf string `json:"instead_of,omitempty"`
Also string `json:"also,omitempty"`
score int
}
// words are a text's words, lower-cased, split at anything that is not a letter, a digit, `-` or `_`
// within a word — so `node-service-manager.journal` is its three names' parts and the whole.
func words(s string) []string {
var out []string
for _, w := range strings.FieldsFunc(strings.ToLower(s), func(r rune) bool {
return !(unicode.IsLetter(r) || unicode.IsDigit(r) || r == '-' || r == '_' || r == '/')
}) {
out = append(out, w)
for _, part := range strings.FieldsFunc(w, func(r rune) bool { return r == '-' || r == '_' || r == '/' }) {
if part != w {
out = append(out, part)
}
}
}
return out
}
// commandWords are a command line's words as matched against a replaced command: lower-cased, a
// program named by path reduced to its name, and what only runs it (`sudo`) left out.
func commandWords(line string) []string {
var out []string
for i, w := range strings.Fields(strings.ToLower(line)) {
if i == 0 || strings.HasPrefix(w, "/") {
w = path.Base(w)
}
if w == "sudo" {
continue
}
out = append(out, w)
}
return out
}
// inOrder says whether every one of want is among have, in the same order.
func inOrder(want, have []string) bool {
i := 0
for _, h := range have {
if i < len(want) && h == want[i] {
i++
}
}
return len(want) > 0 && i == len(want)
}
// replacedBy is the replaced command a query names, the most specific of those it matches, or "".
func replacedBy(query string, replaces []string) string {
q := commandWords(query)
best, bestLen := "", 0
for _, r := range replaces {
w := commandWords(r)
if inOrder(w, q) && len(w) > bestLen {
best, bestLen = r, len(w)
}
}
return best
}
// scoreOf is how well a tool matches the query: 0 for not at all.
func scoreOf(query string, qwords []string, name, description string, replaces []string) (int, string) {
if cmd := replacedBy(query, replaces); cmd != "" {
return 100 + 10*len(commandWords(cmd)), cmd
}
strong := strings.ToLower(name + " " + strings.Join(replaces, " "))
weak := strings.ToLower(description)
strongWords := map[string]bool{}
for _, w := range words(strong) {
strongWords[w] = true
}
weakWords := map[string]bool{}
for _, w := range words(weak) {
weakWords[w] = true
}
score := 10
for _, w := range qwords {
switch {
case strings.Contains(strong, w):
score += 5
case strings.Contains(weak, w):
score++
default:
found := 0
for _, s := range synonyms[w] {
if strongWords[s] {
found = 5
break
}
if weakWords[s] {
found = 1
}
}
if found == 0 {
return 0, ""
}
score += found
}
}
// A name the query says whole — `docker logs` for docker_logs, not docker_secrets_in_logs — first.
own := name[strings.LastIndex(name, ".")+1:]
whole := true
for _, part := range strings.FieldsFunc(strings.ToLower(own), func(r rune) bool { return r == '_' || r == '-' }) {
in := false
for _, w := range qwords {
in = in || w == part
}
whole = whole && in
}
if whole {
score += 2
}
return score, ""
}
// search is mesh_search's answer: every match, best first.
func (x *index) search(query string) []hit {
query = strings.ToLower(strings.TrimSpace(query))
qwords := strings.Fields(query)
var hits []hit
for _, st := range x.Seats {
for _, v := range st.Verbs {
replaces := x.Replaces["seat:"+st.Seat+"."+v.Name]
score, cmd := scoreOf(query, qwords, st.Seat+"."+v.Name, v.Description, replaces)
if score == 0 {
continue
}
h := hit{Does: firstLine(v.Description), InsteadOf: cmd, score: score + 1} // a seat first, on a tie
if st.Scope == "node" {
on := nodesOf(st.Holders)
first := "<node>"
if len(on) > 0 {
first = on[0]
if len(on) > 1 {
h.Also = "also on " + strings.Join(on[1:], ", ")
}
}
h.Address = first + "/" + st.Seat + "." + v.Name
} else {
h.Address = st.Seat + "." + v.Name
}
hits = append(hits, h)
}
}
names := make([]string, 0, len(x.Modules))
for n := range x.Modules {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
m := x.Modules[n]
for _, t := range m.Tools {
score, cmd := scoreOf(query, qwords, m.Module+"."+t.Name, t.Description, x.Replaces[m.Module+"."+t.Name])
if score == 0 {
continue
}
h := hit{Does: firstLine(t.Description), InsteadOf: cmd, score: score}
switch {
case m.Interchangeable:
h.Address, h.Also = m.Module+"."+t.Name, "any instance; on "+orNobody(m.On)
case len(m.On) == 0:
h.Address, h.Also = "<node>/"+m.Module+"."+t.Name, "the mesh places it on no machine"
default:
h.Address = m.On[0] + "/" + m.Module + "." + t.Name
if len(m.On) > 1 {
h.Also = "also on " + strings.Join(m.On[1:], ", ")
}
}
hits = append(hits, h)
}
}
// Best first; on a tie the shorter name — the plainer tool — then the address, so an answer never
// depends on the order discovery heard things in.
sort.SliceStable(hits, func(i, j int) bool {
if hits[i].score != hits[j].score {
return hits[i].score > hits[j].score
}
a, b := hits[i].Address[strings.LastIndex(hits[i].Address, ".")+1:], hits[j].Address[strings.LastIndex(hits[j].Address, ".")+1:]
if len(a) != len(b) {
return len(a) < len(b)
}
return hits[i].Address < hits[j].Address
})
return hits
}
// replacesOf is what every verb and tool replaces, from the controller's records (novox/hq ADR 0245):
// a seat's verbs from its `tools` answer, keyed `seat:<seat>.<verb>`, and a module's own tools from
// its `modules` answer, keyed `<module>.<tool>`. A controller older than the field answers neither,
// and search matches names and descriptions as before.
func replacesOf(toolsAnswer json.RawMessage, modules []recordedModule) map[string][]string {
out := map[string][]string{}
var seats struct {
Seats []struct {
Seat string `json:"seat"`
Tools []struct {
Name string `json:"name"`
Replaces []string `json:"replaces"`
} `json:"tools"`
} `json:"seats"`
}
if json.Unmarshal(toolsAnswer, &seats) == nil {
for _, s := range seats.Seats {
for _, t := range s.Tools {
if len(t.Replaces) > 0 {
out["seat:"+s.Seat+"."+t.Name] = t.Replaces
}
}
}
}
for _, m := range modules {
for tool, r := range m.Replaces {
if len(r) > 0 {
out[m.Module+"."+tool] = r
}
}
}
return out
}
+151
View File
@@ -0,0 +1,151 @@
package console
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-tools/node-tools/internal/announce"
)
// meshToSearch is a mesh as discovery and the controller's records describe it: the service manager's
// verbs on two machines, the controller's own, a docker module with its tools, and what each replaces
// as the records say it — the seats' from `tools`, the module's from `modules`.
func meshToSearch(t *testing.T) *index {
t.Helper()
var endpoints []announce.Endpoint
verbs := map[string]string{
"units": "The units the service manager knows in a scope, each with its load, active and sub state.",
"status": "One unit as the service manager sees it now: its states, whether it starts at boot.",
"restart": "Restart one unit.",
"journal": "The last lines of one unit's journal (at most 2000).",
}
for _, node := range []string{"anchor", "laptop"} {
for verb, does := range verbs {
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "systemd", Seat: "node-service-manager",
Tool: verb, Scope: "node", Node: node, Description: does,
Subject: "mesh.seat.node-service-manager.tool." + verb + "." + node})
}
}
for verb, does := range map[string]string{
"node": "What one machine reported it can do, what it is assigned, and why.",
"nodes": "Every machine the mesh knows, with whether it is converged or adopted.",
} {
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller",
Tool: verb, Scope: "mesh", Description: does, Subject: "mesh.seat.mesh-controller.tool." + verb})
}
for tool, does := range map[string]string{
"docker_list": "Every container on this machine: name, image, state, ports.",
"docker_logs": "The last lines a container wrote.",
"docker_secrets_in_logs": "Whether a container's logs hold a secret.",
"docker_restart": "Restart one container.",
} {
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindTool, Module: "docker", Tool: tool, Node: "anchor",
Description: does, Subject: "mesh.mod.docker.tool." + tool + ".anchor"})
}
x, _, _ := indexOfEndpoints(endpoints)
tools := json.RawMessage(`{"seats":[{"seat":"node-service-manager","scope":"node","tools":[
{"name":"units","replaces":["systemctl list-units","systemctl --failed"]},
{"name":"status","replaces":["systemctl status","systemctl is-active"]},
{"name":"restart","replaces":["systemctl restart"]},
{"name":"journal","replaces":["journalctl"]}]},
{"seat":"mesh-controller","scope":"mesh","tools":[{"name":"node","replaces":["hostnamectl","uptime"]},{"name":"nodes"}]}]}`)
x.Replaces = replacesOf(tools, []recordedModule{{Module: "docker", On: []string{"anchor"}, Tools: true,
Replaces: map[string][]string{"docker_list": {"docker ps"}, "docker_logs": {"docker logs"}, "docker_restart": {"docker restart"}}}})
return x
}
// **The words an agent has find the verb first** (novox/hq ADR 0245): the command it would have run, or
// the word it would have used, puts the mesh's tool for it at the top.
func TestSearchFindsTheToolThatReplacesACommandFirst(t *testing.T) {
x := meshToSearch(t)
for query, want := range map[string]string{
"journalctl": "anchor/node-service-manager.journal",
"journalctl -u mesh-controller -n 50": "anchor/node-service-manager.journal",
"sudo journalctl -fu sshd": "anchor/node-service-manager.journal",
"/usr/bin/journalctl": "anchor/node-service-manager.journal",
"logs": "anchor/node-service-manager.journal",
"systemctl status": "anchor/node-service-manager.status",
"systemctl status nats.service": "anchor/node-service-manager.status",
"systemctl restart mesh-controller": "anchor/node-service-manager.restart",
"systemctl --failed": "anchor/node-service-manager.units",
"docker ps": "anchor/docker.docker_list",
"docker ps -a": "anchor/docker.docker_list",
"docker logs": "anchor/docker.docker_logs",
"docker logs --tail 100 nats": "anchor/docker.docker_logs",
"docker restart nats": "anchor/docker.docker_restart",
"container logs": "anchor/docker.docker_logs",
"uptime": "mesh-controller.node",
} {
hits := x.search(query)
if len(hits) == 0 {
t.Errorf("%q found nothing; want %s first", query, want)
continue
}
if hits[0].Address != want {
var got []string
for _, h := range hits {
got = append(got, h.Address)
}
t.Errorf("%q found %s first; want %s (all: %s)", query, hits[0].Address, want, strings.Join(got, ", "))
}
}
}
// A match on a replaced command says which command it replaced, and the most specific one wins.
func TestASearchByCommandSaysWhatItReplaces(t *testing.T) {
x := meshToSearch(t)
hits := x.search("systemctl status sshd")
if hits[0].InsteadOf != "systemctl status" {
t.Errorf("the match does not say what it replaces: %+v", hits[0])
}
if hits[0].Also != "also on laptop" {
t.Errorf("a node seat's verb does not say its other machines: %+v", hits[0])
}
// `logs` also finds the container's logs, after the journal — both are answers.
var docker bool
for _, h := range x.search("logs")[:3] {
docker = docker || h.Address == "anchor/docker.docker_logs"
}
if !docker {
t.Errorf("`logs` does not find docker_logs among the first: %+v", x.search("logs"))
}
}
// What search did before stays: every word in a name or description, and nothing for a word nothing has.
// A synonym is a word, never part of one: `logs` matches through `log`, and finds no `login`.
func TestSearchStillMatchesEveryWord(t *testing.T) {
x := meshToSearch(t)
if hits := x.search("machine converged"); len(hits) != 1 || hits[0].Address != "mesh-controller.nodes" {
t.Errorf("words in a description: %+v", hits)
}
if hits := x.search("journalctl kubernetes"); len(hits) != 1 || hits[0].Address != "anchor/node-service-manager.journal" {
t.Errorf("a command with arguments nothing has is still that command: %+v", hits)
}
if hits := x.search("kubernetes"); len(hits) != 0 {
t.Errorf("a word nothing has matched: %+v", hits)
}
x.Modules["shell"] = &moduleInfo{Module: "shell", On: []string{"anchor"}, Tools: []Tool{{Module: "shell", Name: "shell_login",
Description: "The account's login shell."}}}
for _, h := range x.search("logs") {
if h.Address == "anchor/shell.shell_login" {
t.Errorf("`logs` found `login` through its synonym `log`: %+v", h)
}
}
}
// A controller older than `replaces` answers without it: search matches names and descriptions as before.
func TestSearchWithoutReplacesMatchesAsBefore(t *testing.T) {
x := meshToSearch(t)
x.Replaces = replacesOf(json.RawMessage(`{"seats":[{"seat":"node-service-manager","tools":[{"name":"journal"}]}]}`),
[]recordedModule{{Module: "docker"}})
if len(x.Replaces) != 0 {
t.Fatalf("replaces from records that have none: %v", x.Replaces)
}
if hits := x.search("journal"); len(hits) == 0 || hits[0].Address != "anchor/node-service-manager.journal" {
t.Errorf("a verb's own name: %+v", hits)
}
if hits := x.search("docker logs"); len(hits) == 0 || hits[0].Address != "anchor/docker.docker_logs" {
t.Errorf("a tool's own name: %+v", hits)
}
}
@@ -0,0 +1,30 @@
package launch
import (
"errors"
"fmt"
"testing"
)
type lostWrite struct{}
func (lostWrite) Error() string {
return "issues.numbers.issue: the key was written by another writer first"
}
func (lostWrite) ErrorCode() int { return CodeChanged }
// A lost compare-and-set is answered to a bundle with its own code, wrapped or not, and the SDK tells it by
// that code alone (mesh-sdk's stdio.CodeChanged); every other refusal is the general code, never that one.
func TestALostWriteIsAnsweredWithItsOwnCodeAndNothingElseIs(t *testing.T) {
if CodeChanged != -32010 {
t.Fatalf("the lost-write code is %d; mesh-sdk tells it by -32010", CodeChanged)
}
for _, err := range []error{lostWrite{}, fmt.Errorf("asked by issues: %w", lostWrite{})} {
if got := errorCode(err); got != CodeChanged {
t.Errorf("%v is answered %d", err, got)
}
}
if got := errorCode(errors.New("issues keeps no state numbers")); got == CodeChanged || got != -32000 {
t.Errorf("a plain refusal is answered %d", got)
}
}
@@ -0,0 +1,93 @@
package launch
import (
"encoding/json"
"errors"
"os"
"path/filepath"
"sync"
"testing"
"time"
)
// A bundle that answers `mesh/event` as $ANSWER says: an error, a result, or an answer with neither.
const answeringBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*)
printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"s1","method":"mesh/subscribe","params":{"pattern":"#"}}\n' ;;
*'"method":"mesh/event"'*)
case "$ANSWER" in
error) printf '{"jsonrpc":"2.0","id":%s,"error":{"code":-32000,"message":"Unexpected end of JSON input"}}\n' "$id" ;;
result) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
bare) printf '{"jsonrpc":"2.0","id":%s}\n' "$id" ;;
esac ;;
esac
done
`
type subscribing struct {
mu sync.Mutex
deliver func(json.RawMessage) error
}
func (b *subscribing) Publish(json.RawMessage) error { return nil }
func (b *subscribing) Ask(json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) State(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) Log(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) Watch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
func (b *subscribing) Seat(string, json.RawMessage, func(string, any) (json.RawMessage, error)) (json.RawMessage, error) {
return nil, nil
}
func (b *subscribing) Subscribe(d func(json.RawMessage) error) error {
b.mu.Lock()
defer b.mu.Unlock()
b.deliver = d
return nil
}
func deliverTo(t *testing.T, answer string) error {
t.Helper()
entry := filepath.Join(t.TempDir(), "bundle")
if err := os.WriteFile(entry, []byte(answeringBundle), 0o755); err != nil {
t.Fatal(err)
}
b := &subscribing{}
l, err := Start("plex", entry, append(os.Environ(), "ANSWER="+answer), b, t.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
for i := 0; ; i++ {
b.mu.Lock()
d := b.deliver
b.mu.Unlock()
if d != nil {
return d(json.RawMessage(`{"key":"radarr.download.completed","body":{"title":"x"}}`))
}
if i > 100 {
t.Fatal("the bundle never subscribed")
}
time.Sleep(20 * time.Millisecond)
}
}
// An event is taken by any answer that is not an error, and not taken by an error, which reaches the
// runtime as the bundle's own words — not as something the runtime failed to read (hq issue 276).
func TestAnEventIsTakenByAnyAnswerThatIsNotAnError(t *testing.T) {
for _, answer := range []string{"result", "bare"} {
if err := deliverTo(t, answer); err != nil {
t.Errorf("an answer %q did not take the event: %v", answer, err)
}
}
err := deliverTo(t, "error")
var refused *Refused
if !errors.As(err, &refused) || refused.Method != "mesh/event" || refused.Message != "Unexpected end of JSON input" {
t.Fatalf("an error answer was not the bundle's refusal of mesh/event: %#v", err)
}
}
+651
View File
@@ -0,0 +1,651 @@
// Package launch starts a bundle the runtime serves and speaks MCP over stdio to it (novox/hq ADR
// 0188, ADR 0193): `initialize`, `tools/list` once, `tools/call` per call. A Go binary, a Python
// script and a Node launcher are the same thing here: an executable that answers those. The runtime
// knows no language; it starts the path it is given.
package launch
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"regexp"
"strings"
"sync"
"syscall"
"time"
"golang.org/x/sys/unix"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
// Protocol is the MCP version spoken to a bundle.
const Protocol = "2025-03-26"
// How long a child has for its handshake, and a call before the caller is told it is slow.
var (
HandshakeTimeout = 10 * time.Second
CallTimeout = 30 * time.Second
)
// Tool is one tool a launched bundle listed, and how to call it.
type Tool struct {
Name string
Description string
Input json.RawMessage
// Run calls the tool for caller: the bus user the call's subject named, or "" when it named none. A
// caller is handed to the bundle in the call's `_meta` (MetaCaller), never among its arguments.
Run func(caller string, args json.RawMessage) (json.RawMessage, error)
}
// MetaCaller is where in a `tools/call`'s `_meta` the runtime names the caller it took from the subject the
// call arrived on (novox/hq issue 365): mesh-sdk's stdio.MetaCaller, which hands it to the tool as
// stdio.CallerOf, and says Unproven when it is absent.
const MetaCaller = "mesh/caller"
// callParams is a `tools/call`'s params: the tool, its arguments as the caller sent them, and — only when the
// subject named one — the caller, in `_meta`, which the runtime writes and no caller can.
func callParams(name, caller string, args json.RawMessage) map[string]any {
params := map[string]any{"name": name, "arguments": args}
if caller != "" {
params["_meta"] = map[string]any{MetaCaller: caller}
}
return params
}
// Registration is the tools a bundle listed under one name: its module's, or a seat's.
type Registration struct {
Module string
Tools []Tool
}
// Bus is what a launched bundle reaches the mesh through (novox/hq ADR 0193, ADR 0198): the runtime
// publishes, asks and subscribes on the module's behalf. Subscribe is asked each time the module's
// code subscribes; the runtime binds the module's consumer once and calls deliver for every event,
// acknowledging it on the bus when deliver returns nil.
//
// State and Watch reach the module's state (novox/hq ADR 0201): State answers `get`, `put`, `delete`
// and `keys`; Watch hands deliver the current values and then every change, and returns once the
// current values are delivered — the watch is the child's, and stops when the child does.
type Bus interface {
Publish(params json.RawMessage) error
Ask(params json.RawMessage) (json.RawMessage, error)
Subscribe(deliver func(envelope json.RawMessage) error) error
State(verb string, params json.RawMessage) (json.RawMessage, error)
Watch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
// Log answers a bundle's `append`, `read`, `last` and `info` on its module's log (novox/hq ADR 0297).
Log(verb string, params json.RawMessage) (json.RawMessage, error)
// Seat answers a bundle's seat traffic (novox/hq ADR 0259 §3): `publish`, `prove` and `record` at once;
// `take` and `answer` bind once and hand each message or proof on through handOn, to whichever child
// of the module asked last — a restarted child asks again as its code runs again.
Seat(verb string, params json.RawMessage, handOn func(method string, params any) (json.RawMessage, error)) (json.RawMessage, error)
}
// EventTimeout bounds how long a bundle has to handle one event before it is offered again.
var EventTimeout = 2 * time.Minute
// Refused is a bundle's error answer to what the runtime asked it: the bundle's own words, as opposed
// to the runtime not reaching it (not running, exited, no answer in time). For `mesh/event` it is the
// one way a bundle says it did not take an event: any answer that is not an error — `{}`, `null`, no
// result at all — takes it (mesh-sdk, "Taking an event"; novox/hq issue 276).
type Refused struct {
Method string
Message string
}
func (r *Refused) Error() string { return r.Message }
// Restart backoff: a bundle that exits is started again at once, then after growing pauses while it
// keeps exiting, back to at once once it has run a while.
var (
RestartFirst = 500 * time.Millisecond
RestartMost = 30 * time.Second
RestartSettle = time.Minute
)
// StopWait bounds how long Launched.Stop waits for its bundle's exit to be handled; a bundle that
// ignores SIGTERM longer than this is left to exit on its own (novox/hq issue 476).
var StopWait = 5 * time.Second
// Launched is a running bundle: what it registered, and how to stop it.
type Launched struct {
Registrations []Registration
Stop func()
}
// Executable says whether an entrypoint can be started: a bundle the runtime serves is executable,
// and one that is not was not built to be served (ADR 0193).
func Executable(path string) bool {
info, err := os.Stat(path)
if err != nil || info.IsDir() {
return false
}
return info.Mode().Perm()&0o111 != 0
}
type message struct {
JSONRPC string `json:"jsonrpc,omitempty"`
ID json.RawMessage `json:"id,omitempty"`
Method string `json:"method,omitempty"`
Params json.RawMessage `json:"params,omitempty"`
Result json.RawMessage `json:"result,omitempty"`
Error *struct {
Code int `json:"code"`
Message string `json:"message"`
} `json:"error,omitempty"`
}
type child struct {
cmd *exec.Cmd
stdin io.WriteCloser
writeMu sync.Mutex
mu sync.Mutex
pending map[int64]chan message
next int64
dead chan struct{}
// handled is closed once the child's exit has been handled whole — its state cleared and its
// restart decided — which Stop waits for (novox/hq issue 476).
handled chan struct{}
why error
// watches are the state watches this child asked for, stopped when it exits (ADR 0201): a child
// that starts again watches again, as its code runs again.
watches []func()
}
func (c *child) write(m any) error {
b, err := wire.Marshal(m)
if err != nil {
return err
}
c.writeMu.Lock()
defer c.writeMu.Unlock()
_, err = c.stdin.Write(append(b, '\n'))
return err
}
var stackLine = regexp.MustCompile(`^\s+at\s`)
// Start launches one bundle and learns its tools. It fails when the child cannot be started or does
// not complete the handshake. A child that exits later is started again on its next call.
func Start(module, entry string, env []string, mesh Bus, logf func(string, ...any)) (*Launched, error) {
var mu sync.Mutex
var current *child
// The child that last subscribed is the one events are handed to: a restarted child subscribes
// again as its code is imported, and from then on the module's events are its.
var subscriber *child
// The child that last asked to take a seat's work or answer its proofs is the one they are handed to.
var seatTaker *child
handOn := func(method string, params any) (json.RawMessage, error) {
mu.Lock()
c := seatTaker
mu.Unlock()
if c == nil {
return nil, errors.New(module + "'s bundle is not running to take it")
}
return c.ask(module, method, params, EventTimeout)
}
stopped := false
deliver := func(envelope json.RawMessage) error {
mu.Lock()
c := subscriber
mu.Unlock()
if c == nil {
return errors.New(module + "'s bundle is not running to take its events")
}
// Taken unless the answer is an error; what a result says is not read (see Refused).
_, err := c.ask(module, "mesh/event", map[string]any{"envelope": envelope}, EventTimeout)
return err
}
var restart func(after time.Duration)
var startedAt time.Time
// The restart pauses are read once, here: the exit handling runs on its own goroutine and must not
// read the package's settings while a caller (a test) changes them (novox/hq issue 476).
pauseFirst, pauseMost, settle := RestartFirst, RestartMost, RestartSettle
pause := pauseFirst
start := func() (*child, error) {
cmd := exec.Command(entry)
cmd.Env = env
stdin, err := cmd.StdinPipe()
if err != nil {
return nil, err
}
stdout, err := cmd.StdoutPipe()
if err != nil {
return nil, err
}
stderr, err := cmd.StderrPipe()
if err != nil {
return nil, err
}
if err := cmd.Start(); err != nil {
return nil, err
}
c := &child{cmd: cmd, stdin: stdin, pending: map[int64]chan message{}, next: 1, dead: make(chan struct{}),
handled: make(chan struct{})}
var lastSaid string
var saidMu sync.Mutex
stderrDone := make(chan struct{})
go func() {
defer close(stderrDone)
scan := bufio.NewScanner(stderr)
scan.Buffer(make([]byte, 64*1024), 1<<20)
for scan.Scan() {
line := scan.Text()
if strings.TrimSpace(line) == "" {
continue
}
logf("[%s] %s", module, line)
if !stackLine.MatchString(line) && !strings.HasPrefix(line, "Node.js v") {
saidMu.Lock()
lastSaid = strings.TrimSpace(line)
saidMu.Unlock()
}
}
}()
go func() {
defer close(c.handled)
scan := bufio.NewScanner(stdout)
scan.Buffer(make([]byte, 64*1024), 16<<20)
for scan.Scan() {
line := strings.TrimSpace(scan.Text())
if line == "" {
continue
}
var m message
if err := json.Unmarshal([]byte(line), &m); err != nil {
if len(line) > 120 {
line = line[:120]
}
logf("[mesh-tools] %s's bundle said something that is not a reply: %s", module, line)
continue
}
// The bundle asks the runtime (ADR 0193, ADR 0198): to emit, to call a tool, to hand it the
// module's events. Each on the module's behalf, answered when done; nothing else.
if m.Method != "" {
go func(m message) {
var result any = map[string]any{}
var err error
switch m.Method {
case "mesh/publish":
err = mesh.Publish(m.Params)
case "mesh/ask":
var asked json.RawMessage
if asked, err = mesh.Ask(m.Params); err == nil {
result = asked
}
case "mesh/subscribe":
mu.Lock()
subscriber = c
mu.Unlock()
err = mesh.Subscribe(deliver)
case "mesh/seat.publish", "mesh/seat.prove", "mesh/seat.record", "mesh/seat.kinds":
var answered json.RawMessage
if answered, err = mesh.Seat(strings.TrimPrefix(m.Method, "mesh/seat."), m.Params, nil); err == nil {
result = answered
}
case "mesh/seat.take", "mesh/seat.answer":
mu.Lock()
seatTaker = c
mu.Unlock()
var answered json.RawMessage
if answered, err = mesh.Seat(strings.TrimPrefix(m.Method, "mesh/seat."), m.Params, handOn); err == nil {
result = answered
}
case "mesh/state.get", "mesh/state.put", "mesh/state.delete", "mesh/state.keys",
"mesh/state.create", "mesh/state.update", "mesh/state.history":
var answered json.RawMessage
if answered, err = mesh.State(strings.TrimPrefix(m.Method, "mesh/state."), m.Params); err == nil {
result = answered
}
case "mesh/log.append", "mesh/log.read", "mesh/log.last", "mesh/log.info":
var answered json.RawMessage
if answered, err = mesh.Log(strings.TrimPrefix(m.Method, "mesh/log."), m.Params); err == nil {
result = answered
}
case "mesh/state.watch":
// Each change is asked of this child, in order, naming the watch it is for by
// the id the child asked it with — two watches of one state are two handlers;
// the watch is answered once the current values have been handed over, so a
// bundle that awaits it has the whole of the state before it goes on (ADR 0201).
var stop func()
watch := m.ID
stop, err = mesh.Watch(m.Params, func(change json.RawMessage) error {
_, err := c.ask(module, "mesh/state", map[string]any{"watch": watch, "change": change}, EventTimeout)
return err
})
if err == nil {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
stop()
} else {
c.watches = append(c.watches, stop)
c.mu.Unlock()
}
}
default:
if len(m.ID) > 0 {
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID,
"error": map[string]any{"code": -32601, "message": "the runtime answers no " + m.Method + " from a bundle"}})
}
return
}
if len(m.ID) == 0 {
return
}
if err != nil {
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID,
"error": map[string]any{"code": errorCode(err), "message": err.Error()}})
return
}
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID, "result": result})
}(m)
continue
}
var id int64
if json.Unmarshal(m.ID, &id) != nil {
continue
}
c.mu.Lock()
ch := c.pending[id]
delete(c.pending, id)
c.mu.Unlock()
if ch != nil {
ch <- m
}
}
<-stderrDone
err := cmd.Wait()
code := "0"
if exit := (*exec.ExitError)(nil); errors.As(err, &exit) {
if status, ok := exit.Sys().(syscall.WaitStatus); ok && status.Signaled() {
code = unix.SignalName(status.Signal())
} else {
code = fmt.Sprint(exit.ExitCode())
}
}
saidMu.Lock()
why := fmt.Sprintf("%s's bundle exited (%s)", module, code)
if lastSaid != "" {
why += ": " + lastSaid
}
saidMu.Unlock()
c.mu.Lock()
c.why = errors.New(why)
watches := c.watches
c.watches = nil
c.mu.Unlock()
for _, stop := range watches {
stop()
}
close(c.dead)
mu.Lock()
// Only a child that was serving is brought back; one that died in its own handshake was
// never accepted, and whoever started it was told why.
wasServing := current == c
if current == c {
current = nil
}
if subscriber == c {
subscriber = nil
}
if seatTaker == c {
seatTaker = nil
}
wasStopped := stopped
// Started again at once if it ran a while; after a growing pause while it keeps exiting.
wait := pauseFirst
if time.Since(startedAt) < settle {
wait = pause
if pause *= 2; pause > pauseMost {
pause = pauseMost
}
} else {
pause = pauseFirst
}
mu.Unlock()
if !wasStopped && wasServing {
// Every bundle stays up (ADR 0198): code that runs long — a handler, a provisioner —
// is not waiting for a call to bring it back, and a tool bundle back early costs nothing.
logf("[mesh-tools] %s; started again in %s", why, wait.Round(100*time.Millisecond))
restart(wait)
}
}()
if _, err := c.ask(module, "initialize", map[string]any{"protocolVersion": Protocol, "capabilities": map[string]any{},
"clientInfo": map[string]any{"name": "node-tools", "version": "1"}}, HandshakeTimeout); err != nil {
_ = cmd.Process.Kill()
return nil, err
}
_ = c.write(map[string]any{"jsonrpc": "2.0", "method": "notifications/initialized"})
return c, nil
}
asking := func(method string, params any, timeout time.Duration) (json.RawMessage, error) {
mu.Lock()
c := current
mu.Unlock()
if c == nil {
fresh, err := start()
if err != nil {
return nil, err
}
mu.Lock()
if current == nil {
current = fresh
startedAt = time.Now()
} else {
_ = fresh.cmd.Process.Signal(syscall.SIGTERM)
fresh = current
}
c = fresh
mu.Unlock()
}
return c.ask(module, method, params, timeout)
}
restart = func(after time.Duration) {
go func() {
time.Sleep(after)
mu.Lock()
if stopped || current != nil {
mu.Unlock()
return
}
mu.Unlock()
fresh, err := start()
mu.Lock()
defer mu.Unlock()
if err != nil {
if !stopped {
wait := pause
if pause *= 2; pause > pauseMost {
pause = pauseMost
}
logf("[mesh-tools] %s's bundle did not start again: %v; trying in %s", module, err, wait)
go restart(wait)
}
return
}
if stopped {
_ = fresh.cmd.Process.Signal(syscall.SIGTERM)
return
}
if current == nil {
current = fresh
startedAt = time.Now()
} else {
_ = fresh.cmd.Process.Signal(syscall.SIGTERM)
}
}()
}
first, err := start()
if err != nil {
return nil, err
}
mu.Lock()
current = first
startedAt = time.Now()
mu.Unlock()
raw, err := asking("tools/list", map[string]any{}, HandshakeTimeout)
if err != nil {
return nil, err
}
var listed struct {
Tools []struct {
Name string `json:"name"`
Description string `json:"description"`
InputSchema json.RawMessage `json:"inputSchema"`
} `json:"tools"`
}
if err := json.Unmarshal(raw, &listed); err != nil {
return nil, fmt.Errorf("%s's bundle listed its tools in a shape that is not MCP's: %w", module, err)
}
order := []string{}
groups := map[string][]Tool{}
for _, t := range listed.Tools {
under, name := module, t.Name
if dot := strings.Index(t.Name, "."); dot >= 0 {
under, name = t.Name[:dot], t.Name[dot+1:]
}
full := t.Name
input := t.InputSchema
if len(input) == 0 || string(input) == "null" {
input = json.RawMessage("{}")
}
tool := Tool{Name: name, Description: t.Description, Input: input,
Run: func(caller string, args json.RawMessage) (json.RawMessage, error) {
if len(args) == 0 || string(args) == "null" {
args = json.RawMessage("{}")
}
res, err := asking("tools/call", callParams(full, caller, args), CallTimeout)
if err != nil {
return nil, err
}
var called struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
IsError bool `json:"isError"`
}
_ = json.Unmarshal(res, &called)
text := ""
for _, c := range called.Content {
if c.Type == "text" {
text = c.Text
break
}
}
if called.IsError {
if text == "" {
text = module + "." + full + " failed"
}
return nil, errors.New(text)
}
// The bundle's answer is JSON as text; handed back as the value it encodes.
if json.Valid([]byte(text)) && text != "" {
return json.RawMessage(text), nil
}
b, _ := json.Marshal(text)
return b, nil
}}
if _, seen := groups[under]; !seen {
order = append(order, under)
}
groups[under] = append(groups[under], tool)
}
out := &Launched{Stop: func() {
mu.Lock()
stopped = true
c := current
current = nil
mu.Unlock()
if c != nil && c.cmd.Process != nil {
_ = c.cmd.Process.Signal(syscall.SIGTERM)
// Returns once the bundle's exit is handled, so nothing of it runs on after Stop
// (novox/hq issue 476); bounded, for a bundle that does not exit on SIGTERM.
select {
case <-c.handled:
case <-time.After(StopWait):
}
}
}}
for _, under := range order {
out.Registrations = append(out.Registrations, Registration{Module: under, Tools: groups[under]})
}
return out, nil
}
func (c *child) ask(module, method string, params any, timeout time.Duration) (json.RawMessage, error) {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
return nil, c.why
}
id := c.next
c.next++
ch := make(chan message, 1)
c.pending[id] = ch
c.mu.Unlock()
if err := c.write(map[string]any{"jsonrpc": "2.0", "id": id, "method": method, "params": params}); err != nil {
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
select {
case <-c.dead:
return nil, c.deathReason()
case <-time.After(100 * time.Millisecond):
return nil, err
}
}
timer := time.NewTimer(timeout)
defer timer.Stop()
select {
case m := <-ch:
if m.Error != nil {
msg := m.Error.Message
if msg == "" {
msg = "the bundle refused the request"
}
return nil, &Refused{Method: method, Message: msg}
}
return m.Result, nil
case <-c.dead:
return nil, c.deathReason()
case <-timer.C:
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
return nil, fmt.Errorf("%s's bundle did not answer %s in %ds", module, method, int(timeout/time.Second))
}
}
func (c *child) deathReason() error {
c.mu.Lock()
defer c.mu.Unlock()
if c.why != nil {
return c.why
}
return errors.New("the bundle exited")
}
// CodeChanged is the error code a bundle is answered a lost compare-and-set with (mesh-sdk's
// stdio.CodeChanged): told by its code, never by its words (novox/hq ADR 0259).
const CodeChanged = -32010
// errorCode is the code an error is answered to a bundle with: a lost compare-and-set's own, else the
// general one.
func errorCode(err error) int {
var coded interface{ ErrorCode() int }
if errors.As(err, &coded) {
return coded.ErrorCode()
}
return -32000
}
+71
View File
@@ -0,0 +1,71 @@
package launch
import (
"encoding/json"
"os"
"path/filepath"
"sync"
"testing"
"time"
)
// A bundle that asks to take a worker's work and answers each piece it is handed.
const takingBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*)
printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"s1","method":"mesh/seat.take","params":{"worker":"SEAT_CHANNEL_TELEGRAM_worker"}}\n' ;;
*'"method":"mesh/work"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"taken":true}}\n' "$id" ;;
esac
done
`
type seating struct {
subscribing
mu sync.Mutex
verb string
params string
handOn func(string, any) (json.RawMessage, error)
}
func (b *seating) Seat(verb string, params json.RawMessage, handOn func(string, any) (json.RawMessage, error)) (json.RawMessage, error) {
b.mu.Lock()
defer b.mu.Unlock()
b.verb, b.params, b.handOn = verb, string(params), handOn
return json.RawMessage(`{}`), nil
}
// novox/hq ADR 0259 §3: a bundle's seat traffic reaches the runtime's bus from its own channel, and the work
// it takes is handed back to it.
func TestABundleTakesASeatsWorkThroughItsOwnChannel(t *testing.T) {
entry := filepath.Join(t.TempDir(), "bundle")
if err := os.WriteFile(entry, []byte(takingBundle), 0o755); err != nil {
t.Fatal(err)
}
b := &seating{}
l, err := Start("telegram", entry, os.Environ(), b, t.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
var handOn func(string, any) (json.RawMessage, error)
for i := 0; handOn == nil; i++ {
if i > 100 {
t.Fatal("the bundle never asked to take its work")
}
time.Sleep(20 * time.Millisecond)
b.mu.Lock()
handOn = b.handOn
b.mu.Unlock()
}
if b.verb != "take" || b.params != `{"worker":"SEAT_CHANNEL_TELEGRAM_worker"}` {
t.Fatalf("asked %s %s", b.verb, b.params)
}
got, err := handOn("mesh/work", map[string]any{"work": map[string]any{"subject": "mesh.seat.channel.accept.show.telegram"}})
if err != nil || string(got) != `{"taken":true}` {
t.Fatalf("the work was not handed to the bundle: %s %v", got, err)
}
}
+273
View File
@@ -0,0 +1,273 @@
// Package meshtest raises what the controller would, for tests against a real bus: the ASSIGNMENTS
// and EVENTS streams, memberships issued by hand, and a fixture's path.
//
// **Every test has a bus of its own** (novox/hq ADR 0237, the controller's internal/testbus): a server
// linked in at the release go.mod pins — the release the mesh runs, held so by a test beside this one —
// started for the one test and gone after it. The packages shared one bus and had to run one at a time:
// each test raised the streams afresh, and the console discovers every runtime that announces itself on
// the bus (ADR 0197), so a runtime from another package's test was, correctly, found. Now the suite runs
// as Go runs it, in parallel and under the race detector. A person may still point a run at a bus of
// their own with MESH_TEST_NATS_EXTERNAL=1 and MESH_TEST_NATS; then the run is theirs to serialise.
package meshtest
import (
"encoding/json"
"os"
"path/filepath"
"runtime"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// Version is the release of the server the tests run.
const Version = server.VERSION
// URL is the bus of this test alone: started at its first ask, the same one at every ask after — a test
// that dials twice, or hands the address to the code it tests, reaches one bus — and shut down when the
// test ends.
func URL(t *testing.T) string {
t.Helper()
if os.Getenv("MESH_TEST_NATS_EXTERNAL") == "1" {
if url := os.Getenv("MESH_TEST_NATS"); url != "" {
return url
}
t.Fatal("MESH_TEST_NATS_EXTERNAL=1 and MESH_TEST_NATS names no bus")
}
if url, ok := buses.Load(t); ok {
return url.(string)
}
opts := &server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: t.TempDir(),
NoLog: true, NoSigs: true}
s, err := server.NewServer(opts)
if err != nil {
t.Fatalf("a bus for this test could not be made: %v", err)
}
go s.Start()
if !s.ReadyForConnections(30 * time.Second) {
s.Shutdown()
t.Fatal("a bus for this test did not come up within 30s")
}
url := s.ClientURL()
buses.Store(t, url)
t.Cleanup(func() {
buses.Delete(t)
s.Shutdown()
s.WaitForShutdown()
})
return url
}
// buses are the running tests' buses, by test.
var buses sync.Map
// Mesh is the controller's job, done by hand.
type Mesh struct {
nc *nats.Conn
js nats.JetStreamContext
}
// New raises the streams afresh.
func New(t *testing.T) *Mesh {
t.Helper()
nc, err := nats.Connect(URL(t))
if err != nil {
t.Fatal(err)
}
js, _ := nc.JetStream()
for _, s := range []string{"ASSIGNMENTS", "EVENTS"} {
_ = js.DeleteStream(s)
}
if _, err := js.AddStream(&nats.StreamConfig{Name: "ASSIGNMENTS", Subjects: []string{"mesh.assignment.>"},
MaxMsgsPerSubject: 1, AllowDirect: true}); err != nil {
t.Fatal(err)
}
if _, err := js.AddStream(&nats.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
return &Mesh{nc: nc, js: js}
}
// Issue publishes a membership.
func (m *Mesh) Issue(t *testing.T, mem bus.Membership) {
t.Helper()
body, _ := json.Marshal(mem)
if _, err := m.js.Publish(bus.MembershipSubject(mem.Node, mem.Module), body); err != nil {
t.Fatal(err)
}
}
// NextEvent is the subject the next event under a pattern lands on.
func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
t.Helper()
ch := make(chan string, 1)
sub, err := m.nc.Subscribe(pattern, func(msg *nats.Msg) {
select {
case ch <- msg.Subject:
default:
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
_ = m.nc.Flush()
return ch
}
// MembershipOf is a membership as the controller issues one on a machine.
func MembershipOf(module, node string, plain bool, seats map[string][]string) bus.Membership {
own := "mesh.mod." + module
serves := []bus.Served{{Subject: own + ".tool.{tool}." + node}}
if plain {
serves = append(serves, bus.Served{Subject: own + ".tool.{tool}", Queue: "serve." + module})
}
var verbs []bus.SeatVerb
for seat, vs := range seats {
for _, v := range vs {
verbs = append(verbs, bus.SeatVerb{Seat: seat, Verb: v, Subject: "mesh.seat." + seat + ".tool." + v + "." + node})
}
}
return bus.Membership{Node: node, Module: module, Serves: serves, Seats: verbs,
Emits: own + ".event.{event}", Tools: own + ".tool.tools"}
}
// Fixture is a path in node-tools/test/fixtures, which these tests share with the TypeScript ones.
func Fixture(name string) string {
_, here, _, _ := runtime.Caller(0)
return filepath.Join(filepath.Dir(here), "..", "..", "test", "fixtures", name)
}
// Until retries while the bus answers "no responders" — something not yet served.
func Until(t *testing.T, try func() error) {
t.Helper()
var err error
for i := 0; i < 50; i++ {
if err = try(); err == nil {
return
}
time.Sleep(100 * time.Millisecond)
}
t.Fatal(err)
}
// Logs collects what the runtime says.
type Logs struct{ lines []string }
// Logf is a logger that keeps the lines.
func (l *Logs) Logf(format string, args ...any) {
l.lines = append(l.lines, sprintf(format, args...))
}
// Has says whether a line contains every fragment.
func (l *Logs) Has(fragments ...string) bool {
for _, line := range l.lines {
all := true
for _, f := range fragments {
all = all && strings.Contains(line, f)
}
if all {
return true
}
}
return false
}
// All is every line.
func (l *Logs) All() string { return strings.Join(l.lines, "\n") }
// Consumer makes a module's durable consumer on a machine as the controller does: pull, on the
// EVENTS stream, filtered to what the module consumes, with a short ack wait so a test sees a
// redelivery in seconds rather than the mesh's minutes.
func (m *Mesh) Consumer(t *testing.T, node, module string, filters []string, ackWait time.Duration) {
t.Helper()
cfg := &nats.ConsumerConfig{Durable: node + "_" + module, AckPolicy: nats.AckExplicitPolicy,
AckWait: ackWait, MaxDeliver: 10, DeliverPolicy: nats.DeliverNewPolicy}
if len(filters) == 1 {
cfg.FilterSubject = filters[0]
} else {
cfg.FilterSubjects = filters
}
if _, err := m.js.AddConsumer("EVENTS", cfg); err != nil {
t.Fatal(err)
}
}
// Emit publishes an event as a module would, into the EVENTS stream.
func (m *Mesh) Emit(t *testing.T, subject string, body any) {
t.Helper()
data, _ := json.Marshal(body)
if _, err := m.js.Publish(subject, data); err != nil {
t.Fatal(err)
}
}
// Pending is what a module's consumer still holds: delivered and not acknowledged, and not yet
// delivered.
func (m *Mesh) Pending(t *testing.T, node, module string) (ackPending, notDelivered uint64) {
t.Helper()
info, err := m.js.ConsumerInfo("EVENTS", node+"_"+module)
if err != nil {
t.Fatal(err)
}
return uint64(info.NumAckPending), info.NumPending
}
// BucketKeeping makes a module's state afresh as Bucket does, keeping up to history values a key.
func (m *Mesh) BucketKeeping(t *testing.T, name string, history uint8) nats.KeyValue {
t.Helper()
_ = m.js.DeleteKeyValue(name)
kv, err := m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: name, History: history, MaxValueSize: 256 * 1024})
if err != nil {
t.Fatal(err)
}
return kv
}
// Bucket makes a module's state afresh as the controller does from the catalogue (novox/hq ADR 0201),
// and answers it for writing what is there before a bundle starts.
func (m *Mesh) Bucket(t *testing.T, name string) nats.KeyValue {
t.Helper()
_ = m.js.DeleteKeyValue(name)
kv, err := m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: name, History: 1, MaxValueSize: 256 * 1024})
if err != nil {
t.Fatal(err)
}
return kv
}
// Log makes a module's log afresh as the controller does from the catalogue (novox/hq ADR 0297): a file
// stream named LOG_<module>_<name> on mesh.log.<module>.<name>.>, limits retention, no age, no cap per
// key, the declared cap, 260 KiB a message (a 256 KiB value and its headers), refusing new entries when full, direct get, no delete or
// purge. It answers the membership entry the controller issues for it.
func (m *Mesh) Log(t *testing.T, module, name string, maxBytes int64) bus.LogIssued {
t.Helper()
stream := "LOG_" + module + "_" + name
subject := "mesh.log." + module + "." + name
_ = m.js.DeleteStream(stream)
if _, err := m.js.AddStream(&nats.StreamConfig{Name: stream, Subjects: []string{subject + ".>"},
Storage: nats.FileStorage, Retention: nats.LimitsPolicy, MaxAge: 0, MaxMsgsPerSubject: -1,
MaxBytes: maxBytes, MaxMsgSize: 260 * 1024, Discard: nats.DiscardNew, AllowDirect: true,
DenyDelete: true, DenyPurge: true, Replicas: 1,
Description: module + "'s log " + name + ", kept as its record (novox/hq ADR 0297)"}); err != nil {
t.Fatal(err)
}
return bus.LogIssued{Name: name, Stream: stream, Subject: subject, Writes: true}
}
// Consumers is how many consumers a stream has.
func (m *Mesh) Consumers(t *testing.T, stream string) int {
t.Helper()
info, err := m.js.StreamInfo(stream)
if err != nil {
t.Fatal(err)
}
return info.State.Consumers
}
@@ -0,0 +1,75 @@
package meshtest
import (
"encoding/json"
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// **The bus the tests run is the bus the mesh runs** (novox/hq ADR 0227 rule 9, ADR 0237): the server linked
// into the tests is held to the release the catalogue's bus image is — read from beside this checkout, as
// the build seat clones it for a merge check — and, given the facts snapshot, to the release the mesh's bus
// server says it runs. A bus upgrade moves the catalogue's pin; this then fails until go.mod's moves with it.
func TestTheBusTheTestsRunIsTheBusTheMeshRuns(t *testing.T) {
beside := os.Getenv("MESH_CHECK_BESIDE")
if beside == "" {
beside = filepath.Join("..", "..", "..", "..")
}
dockerfile := filepath.Join(beside, "mesh-catalog", "modules", "nats", "Dockerfile")
raw, err := os.ReadFile(dockerfile)
switch {
case err == nil:
pinned := regexp.MustCompile(`upstream: nats ([0-9.]+)-alpine`).FindSubmatch(raw)
if pinned == nil {
t.Fatalf("%s says no release its digest is", dockerfile)
}
if string(pinned[1]) != Version {
t.Errorf("the tests run bus %s and the catalogue's bus image is %s: move go.mod's nats-server pin with the image",
Version, pinned[1])
}
case os.IsNotExist(err):
t.Logf("the catalogue is not beside this checkout, so its bus image is not compared")
default:
t.Fatal(err)
}
path := os.Getenv("MESH_FACTS")
if path == "" {
t.Logf("no MESH_FACTS: the bus the mesh runs is compared in a merge check, which has it")
return
}
body, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
var f struct {
Taken string `json:"taken"`
Versions struct {
Bus string `json:"bus"`
} `json:"versions"`
}
if err := json.Unmarshal(body, &f); err != nil {
t.Fatal(err)
}
if f.Versions.Bus != "" && strings.TrimPrefix(f.Versions.Bus, "v") != Version {
t.Errorf("the tests run bus %s and the mesh's bus runs %s (facts of %s)", Version, f.Versions.Bus, f.Taken)
}
}
// Each test is given a bus of its own, and the same one at every ask.
func TestEachTestHasABusOfItsOwn(t *testing.T) {
var first string
t.Run("one", func(t *testing.T) {
first = URL(t)
if URL(t) != first {
t.Fatal("two buses for one test")
}
})
t.Run("two", func(t *testing.T) {
if URL(t) == first {
t.Fatal("two tests were given one bus")
}
})
}
+5
View File
@@ -0,0 +1,5 @@
package meshtest
import "fmt"
func sprintf(format string, args ...any) string { return fmt.Sprintf(format, args...) }
@@ -0,0 +1,121 @@
package runtime
import (
"encoding/json"
"sort"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq ADR 0197: what a runtime serves, it announces — the NATS services protocol's discovery,
// read here with NATS's own types, each endpoint a subject actually served — and the announcement
// follows a re-issued membership.
func TestTheRuntimeAnnouncesWhatItServesInTheServicesProtocol(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("beta", "anchor", false, map[string][]string{"node-shelf": {"list", "clear"}}))
nodeTools := connect(t, "node-tools", "anchor")
stop, err := Run(nodeTools, []Served{
{"alpha", []string{mt.Fixture("many-alpha.serve.mjs")}},
{"beta", []string{mt.Fixture("many-beta.serve.mjs")}},
}, nil, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
info := func() micro.Info {
t.Helper()
msg, err := nc.Request("$SRV.INFO.node-tools.anchor", nil, 2*time.Second)
if err != nil {
t.Fatal(err)
}
var i micro.Info
if err := json.Unmarshal(msg.Data, &i); err != nil {
t.Fatal(err)
}
return i
}
subjects := func(i micro.Info) string {
var out []string
for _, e := range i.Endpoints {
out = append(out, e.Subject+"|"+e.QueueGroup+"|"+e.Metadata["kind"]+"|"+e.Metadata["module"]+"|"+e.Metadata["seat"]+"|"+e.Metadata["scope"]+"|"+e.Metadata["interchangeable"])
}
sort.Strings(out)
return strings.Join(out, "\n")
}
got := info()
if got.Type != micro.InfoResponseType || got.Name != "node-tools" || got.ID != "anchor" || got.Version == "" {
t.Errorf("identity: %+v", got.ServiceIdentity)
}
want := strings.Join([]string{
"mesh.mod.alpha.tool.one.anchor||tool|alpha|||false",
"mesh.mod.alpha.tool.two.anchor||tool|alpha|||false",
"mesh.mod.beta.tool.five.anchor||tool|beta|||false",
"mesh.mod.beta.tool.four.anchor||tool|beta|||false",
"mesh.mod.beta.tool.three.anchor||tool|beta|||false",
"mesh.seat.node-shelf.tool.clear.anchor||seat|beta|node-shelf|node|false",
"mesh.seat.node-shelf.tool.list.anchor||seat|beta|node-shelf|node|false",
}, "\n")
if s := subjects(got); s != want {
t.Errorf("announced:\n%s\nwant:\n%s", s, want)
}
for _, e := range got.Endpoints {
if e.Metadata["node"] != "anchor" || e.Metadata["description"] == "" {
t.Errorf("endpoint metadata: %+v", e)
}
// A seat's verb carries its schema; a module's tool does not — it outgrew the bus (2026-10-04).
_, hasSchema := e.Metadata["schema"]
if e.Metadata["kind"] == "seat" && !json.Valid([]byte(e.Metadata["schema"])) {
t.Errorf("a seat's verb without a schema: %+v", e)
}
if e.Metadata["kind"] == "tool" && hasSchema {
t.Errorf("a module's tool still announces its schema: %+v", e)
}
}
// Every subject announced is answered.
asker := connect(t, "console", "workstation")
for _, e := range got.Endpoints {
if _, err := asker.Ask("", map[string]any{}, e.Subject); err != nil {
t.Errorf("announced %s and does not answer it: %v", e.Subject, err)
}
}
// PING answers with the same identity, and a request for another service is not answered.
if msg, err := nc.Request("$SRV.PING", nil, 2*time.Second); err != nil || !strings.Contains(string(msg.Data), micro.PingResponseType) {
t.Errorf("ping: %v %v", msg, err)
}
if _, err := nc.Request("$SRV.INFO.somebody-else", nil, 300*time.Millisecond); err == nil {
t.Error("answered a request for another service")
}
// alpha is re-issued a plain subject: the announcement says so, and says it is interchangeable.
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", true, nil))
var after string
for i := 0; i < 40; i++ {
after = subjects(info())
if strings.Contains(after, "mesh.mod.alpha.tool.one|serve.alpha|tool|alpha|||true") {
break
}
time.Sleep(100 * time.Millisecond)
}
if !strings.Contains(after, "mesh.mod.alpha.tool.one|serve.alpha|tool|alpha|||true") ||
!strings.Contains(after, "mesh.mod.alpha.tool.one.anchor||tool|alpha|||true") {
t.Errorf("after a re-issued membership:\n%s", after)
}
_ = bus.Served{}
}
@@ -0,0 +1,91 @@
package runtime
import (
"encoding/json"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
type saidCaller struct {
Caller string `json:"caller"`
Arguments map[string]any `json:"arguments"`
}
// **A module is told who called it, from the subject the call arrived on** (novox/hq issue 365): a call on the
// caller-named subject reaches the bundle with that caller in the call's `_meta`, for a module's tool, on its
// machine's address, and for a seat's verb; a caller named in the arguments is only an argument.
func TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("ledger", "anchor", true, map[string][]string{"node-desk": {"who"}}))
nodeTools := connect(t, "node-tools", "anchor")
stop, err := Run(nodeTools, []Served{{"ledger", []string{mt.Fixture("caller-echo.py")}}}, nil, func(string, ...any) {})
if err != nil {
t.Fatal(err)
}
defer stop() // returns once the bundle's exit is handled (novox/hq issue 476)
asker, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "shanks",
User: "shanks.node-tools", Password: "unused"})
if err != nil {
t.Fatal(err)
}
asker.Logf = func(string, ...any) {}
defer asker.Close()
for _, key := range []string{"ledger.who", "ledger.who@anchor", "seat:node-desk.who@anchor"} {
var got saidCaller
mt.Until(t, func() error {
answered, err := asker.Ask(key, map[string]any{"caller": "novox.node-tools",
"_meta": map[string]any{"mesh/caller": "novox.controller"}}, "")
if err == nil {
err = json.Unmarshal(answered.Result, &got)
}
return err
})
if got.Caller != "shanks.node-tools" {
t.Errorf("%s reached the module with caller %q, want the asker's bus user", key, got.Caller)
}
if got.Arguments["caller"] != "novox.node-tools" {
t.Errorf("%s: the arguments did not reach the module as sent: %v", key, got.Arguments)
}
}
}
// The subject that names no caller is still served, for one release (retired by hq issue 464), and says so:
// the module is named no caller, whatever the body claims.
func TestTheSubjectThatNamesNoCallerStillWorksAndNamesNone(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("ledger", "anchor", true, map[string][]string{"node-desk": {"who"}}))
nodeTools := connect(t, "node-tools", "anchor")
stop, err := Run(nodeTools, []Served{{"ledger", []string{mt.Fixture("caller-echo.py")}}}, nil, func(string, ...any) {})
if err != nil {
t.Fatal(err)
}
defer stop() // returns once the bundle's exit is handled (novox/hq issue 476)
old, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
defer old.Close()
body, _ := json.Marshal(map[string]any{"_meta": map[string]any{"mesh/caller": "novox.controller"}})
for _, subject := range []string{"mesh.mod.ledger.tool.who", "mesh.mod.ledger.tool.who.anchor", "mesh.seat.node-desk.tool.who.anchor"} {
var reply struct {
Result saidCaller `json:"result"`
Error string `json:"error"`
}
mt.Until(t, func() error {
msg, err := old.Request(subject, body, 5*time.Second)
if err == nil {
err = json.Unmarshal(msg.Data, &reply)
}
return err
})
if reply.Error != "" || reply.Result.Caller != "" {
t.Errorf("%s: the module was named caller %q (error %q), want none", subject, reply.Result.Caller, reply.Error)
}
}
}
+157
View File
@@ -0,0 +1,157 @@
package runtime
import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// A logger safe to write from the runtime's goroutines.
type lines struct {
mu sync.Mutex
l []string
}
func (s *lines) logf(format string, args ...any) {
s.mu.Lock()
defer s.mu.Unlock()
s.l = append(s.l, strings.TrimSpace(sprintf(format, args...)))
}
func (s *lines) all() string {
s.mu.Lock()
defer s.mu.Unlock()
return strings.Join(s.l, "\n")
}
func sprintf(format string, args ...any) string { return fmtSprintf(format, args...) }
func countLines(t *testing.T, path, want string) int {
t.Helper()
b, _ := os.ReadFile(path)
n := 0
for _, l := range strings.Split(string(b), "\n") {
if l == want {
n++
}
}
return n
}
// novox/hq ADR 0198: a module's long-running code is launched by the node's runtime, and the runtime
// is its bus — it binds the module's own consumer, hands each event to the bundle, and acknowledges it
// only when the bundle has taken it; one the bundle failed or died on is offered again.
func TestTheRuntimeHandsAModulesEventsToItsBundleAndAcknowledgesThemOnlyWhenTaken(t *testing.T) {
mesh := mt.New(t)
restore := bus.NakDelay
bus.NakDelay = 300 * time.Millisecond
t.Cleanup(func() { bus.NakDelay = restore })
firstPause := launch.RestartFirst
launch.RestartFirst = 100 * time.Millisecond
t.Cleanup(func() { launch.RestartFirst = firstPause })
mesh.Issue(t, mt.MembershipOf("watcher", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("beta", "anchor", true, nil))
// The controller's consumer for the module, as its own runtime bound it: anchor_watcher on EVENTS.
mesh.Consumer(t, "anchor", "watcher", []string{"mesh.mod.alpha.event.>"}, 2*time.Second)
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
dir := t.TempDir()
log := filepath.Join(dir, "watch.log")
said := &lines{}
stop, err := Run(nodeTools, []Served{
{Module: "watcher", Entrypoints: []string{mt.Fixture("watcher.serve.mjs")}},
{Module: "beta", Entrypoints: []string{mt.Fixture("many-beta.serve.mjs")}},
}, map[string]map[string]string{"watcher": {"WATCH_LOG": log}}, said.logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
if !strings.Contains(said.all(), "watcher's events arrive on its consumer anchor_watcher") {
t.Fatalf("the module's consumer was not bound:\n%s", said.all())
}
// Handled once, acknowledged once.
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 1})
mt.Until(t, func() error {
if countLines(t, log, "handled 1") != 1 {
return errorf("event 1 not handled yet")
}
return nil
})
// The handler fails the first time: not acknowledged, offered again, then handled.
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 2, "fail": true})
// The bundle dies on the first offer: not acknowledged, the bundle is started again and the
// event is offered again once its ack wait has passed.
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 3, "die": true})
deadline := time.Now().Add(20 * time.Second)
for time.Now().Before(deadline) {
if countLines(t, log, "handled 2") == 1 && countLines(t, log, "handled 3") == 1 {
break
}
time.Sleep(200 * time.Millisecond)
}
if countLines(t, log, "handled 2") != 1 || countLines(t, log, "handled 3") != 1 {
b, _ := os.ReadFile(log)
t.Fatalf("a failed or interrupted event was not offered again and handled once:\n%s\nruntime:\n%s", b, said.all())
}
if countLines(t, log, "started") < 2 {
t.Errorf("the bundle that died was not started again: %s", said.all())
}
mt.Until(t, func() error {
ack, notYet := mesh.Pending(t, "anchor", "watcher")
if ack != 0 || notYet != 0 {
return errorf("still pending: %d unacknowledged, %d undelivered", ack, notYet)
}
return nil
})
if countLines(t, log, "handled 1") != 1 {
t.Errorf("event 1 was handled more than once")
}
// A tool asks another module's tool through the runtime, as the module.
got, err := call(t, asker, "watcher.relay@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"beta":3}`)
}
// A long-running bundle that exits is started again, without waiting for a call (ADR 0198).
func TestALongRunningBundleThatExitsIsStartedAgain(t *testing.T) {
mesh := mt.New(t)
firstPause := launch.RestartFirst
launch.RestartFirst = 100 * time.Millisecond
t.Cleanup(func() { launch.RestartFirst = firstPause })
mesh.Issue(t, mt.MembershipOf("flaky", "anchor", false, nil))
nodeTools := connect(t, "node-tools", "anchor")
log := filepath.Join(t.TempDir(), "flaky.log")
said := &lines{}
stop, err := Run(nodeTools, []Served{{Module: "flaky", Entrypoints: []string{mt.Fixture("flaky.serve.mjs")}}},
map[string]map[string]string{"flaky": {"FLAKY_LOG": log}}, said.logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
mt.Until(t, func() error {
if countLines(t, log, "started") < 3 {
return errorf("started %d time(s)", countLines(t, log, "started"))
}
return nil
})
if !strings.Contains(said.all(), "flaky's bundle exited (3)") || !strings.Contains(said.all(), "started again in") {
t.Errorf("the restart was not said:\n%s", said.all())
}
}
func fmtSprintf(format string, args ...any) string { return fmt.Sprintf(format, args...) }
func errorf(format string, args ...any) error { return fmt.Errorf(format, args...) }
+336
View File
@@ -0,0 +1,336 @@
package runtime
import (
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// logBuses is a module, issues, that keeps a log `changes` on a real bus, and a module, other, on the
// same runtime that was issued no log — each reached as a bundle reaches it: mesh/log.<verb> with its
// JSON, through its module's bus (novox/hq ADR 0297).
func logBuses(t *testing.T, maxBytes int64) (issues, other launch.Bus, mesh *mt.Mesh) {
t.Helper()
mesh = mt.New(t)
m := mt.MembershipOf("issues", "anchor", false, nil)
m.Logs = []bus.LogIssued{mesh.Log(t, "issues", "changes", maxBytes)}
mesh.Issue(t, m)
mesh.Issue(t, mt.MembershipOf("other", "anchor", false, nil))
conn := connect(t, "node-tools", "anchor")
conn.Follow("issues")
conn.Follow("other")
mt.Until(t, func() error {
if conn.Membership("issues") == nil || conn.Membership("other") == nil {
return errors.New("not issued yet")
}
return nil
})
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
return all.forModule("issues"), all.forModule("other"), mesh
}
// logAsk is a bundle's mesh/log.<verb>, with its params exactly as they go over the wire.
func logAsk(b launch.Bus, verb, params string) (json.RawMessage, error) {
return b.Log(verb, json.RawMessage(params))
}
func appendOf(t *testing.T, b launch.Bus, params string) uint64 {
t.Helper()
raw, err := logAsk(b, "append", params)
if err != nil {
t.Fatalf("append %s: %v", params, err)
}
var r struct {
Sequence *uint64 `json:"sequence"`
}
if err := json.Unmarshal(raw, &r); err != nil || r.Sequence == nil {
t.Fatalf("append answered %s, not {sequence}", raw)
}
return *r.Sequence
}
// wirePage is a read's answer with exactly the contract's field names.
type wirePage struct {
Entries []struct {
Sequence uint64 `json:"sequence"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
At string `json:"at"`
} `json:"entries"`
Next uint64 `json:"next"`
Last uint64 `json:"last"`
}
func readOf(t *testing.T, b launch.Bus, params string) wirePage {
t.Helper()
raw, err := logAsk(b, "read", params)
if err != nil {
t.Fatalf("read %s: %v", params, err)
}
var fields map[string]json.RawMessage
_ = json.Unmarshal(raw, &fields)
for _, f := range []string{"entries", "next", "last"} {
if _, ok := fields[f]; !ok {
t.Fatalf("read answered %s, without %q", raw, f)
}
}
var p wirePage
if err := json.Unmarshal(raw, &p); err != nil {
t.Fatal(err)
}
return p
}
func infoOf(t *testing.T, b launch.Bus) map[string]uint64 {
t.Helper()
raw, err := logAsk(b, "info", `{"log":"changes"}`)
if err != nil {
t.Fatal(err)
}
var info map[string]uint64
if err := json.Unmarshal(raw, &info); err != nil {
t.Fatalf("info answered %s: %v", raw, err)
}
for _, f := range []string{"entries", "bytes", "max_bytes", "first", "last"} {
if _, ok := info[f]; !ok {
t.Fatalf("info answered %s, without %q", raw, f)
}
}
return info
}
// Each verb against a real bus: append answers the sequence, read pages from a sequence on, of a key
// pattern, with the sequence to continue from (0 at the end) and the log's last, each entry's time in
// RFC 3339; last is a key's last entry or null; info says entries, bytes, cap, first and last — and no
// consumer is made for any of it.
func TestALogAnswersEachVerbOnTheBus(t *testing.T) {
b, _, mesh := logBuses(t, 1<<20)
before := time.Now().Add(-time.Second)
var seqs []uint64
for i, key := range []string{"12", "13", "12", "14.notes", "12"} {
seqs = append(seqs, appendOf(t, b, fmt.Sprintf(`{"log":"changes","key":%q,"value":{"op":"move","n":%d}}`, key, i)))
}
for i := 1; i < len(seqs); i++ {
if seqs[i] != seqs[i-1]+1 {
t.Fatalf("sequences %v are not the log's order", seqs)
}
}
// Everything, no keys and no limit: the default page holds all five.
all := readOf(t, b, `{"log":"changes","from":0}`)
if len(all.Entries) != 5 || all.Next != 0 || all.Last != seqs[4] {
t.Fatalf("a read of everything: %d entries, next %d, last %d", len(all.Entries), all.Next, all.Last)
}
for i, e := range all.Entries {
at, err := time.Parse(time.RFC3339, e.At)
if err != nil || at.Before(before) || at.After(time.Now().Add(time.Second)) {
t.Errorf("entry %d's time %q is not RFC 3339 and now: %v", e.Sequence, e.At, err)
}
if e.Sequence != seqs[i] || string(e.Value) != fmt.Sprintf(`{"op":"move","n":%d}`, i) {
t.Errorf("entry %d: %d %s %s", i, e.Sequence, e.Key, e.Value)
}
}
if all.Entries[3].Key != "14.notes" {
t.Errorf("a key of two names read back as %q", all.Entries[3].Key)
}
// One key's entries, a page of two at a time.
first := readOf(t, b, `{"log":"changes","from":1,"keys":"12","limit":2}`)
if len(first.Entries) != 2 || first.Entries[0].Sequence != seqs[0] || first.Entries[1].Sequence != seqs[2] ||
first.Next != seqs[2]+1 {
t.Fatalf("the first page of key 12: %+v", first)
}
second := readOf(t, b, fmt.Sprintf(`{"log":"changes","from":%d,"keys":"12","limit":2}`, first.Next))
if len(second.Entries) != 1 || second.Entries[0].Sequence != seqs[4] || second.Next != 0 {
t.Fatalf("the second page of key 12: %+v", second)
}
// A pattern: `*` one name, `>` the rest.
if p := readOf(t, b, `{"log":"changes","from":1,"keys":"14.*"}`); len(p.Entries) != 1 || p.Entries[0].Key != "14.notes" {
t.Fatalf("14.*: %+v", p)
}
if p := readOf(t, b, `{"log":"changes","from":1,"keys":"14.>"}`); len(p.Entries) != 1 {
t.Fatalf("14.>: %+v", p)
}
// Past the end: nothing, nothing follows.
if p := readOf(t, b, fmt.Sprintf(`{"log":"changes","from":%d}`, seqs[4]+1)); len(p.Entries) != 0 || p.Next != 0 || p.Last != seqs[4] {
t.Fatalf("a read past the end: %+v", p)
}
if p := readOf(t, b, `{"log":"changes","from":1,"keys":"99"}`); len(p.Entries) != 0 || p.Next != 0 {
t.Fatalf("a key with no entries: %+v", p)
}
raw, err := logAsk(b, "last", `{"log":"changes","key":"12"}`)
if err != nil {
t.Fatal(err)
}
var last map[string]json.RawMessage
_ = json.Unmarshal(raw, &last)
if string(last["sequence"]) != fmt.Sprint(seqs[4]) || string(last["key"]) != `"12"` ||
string(last["value"]) != `{"op":"move","n":4}` || len(last["at"]) == 0 {
t.Fatalf("the last entry of 12: %s", raw)
}
if raw, err := logAsk(b, "last", `{"log":"changes","key":"99"}`); err != nil || string(raw) != "null" {
t.Fatalf("the last entry of a key with none: %s %v", raw, err)
}
info := infoOf(t, b)
if info["entries"] != 5 || info["first"] != seqs[0] || info["last"] != seqs[4] || info["max_bytes"] != 1<<20 ||
info["bytes"] == 0 {
t.Fatalf("info: %v", info)
}
if n := mesh.Consumers(t, "LOG_issues_changes"); n != 0 {
t.Fatalf("the runtime made %d consumers of the log; it uses none", n)
}
}
// A page holds about 4 MiB at most, whatever its limit, and says where to continue.
func TestALogPageIsBoundedByItsBytes(t *testing.T) {
b, _, _ := logBuses(t, 64<<20)
big := `"` + strings.Repeat("x", 250*1024) + `"`
for i := 0; i < 20; i++ {
appendOf(t, b, fmt.Sprintf(`{"log":"changes","key":"k%d","value":%s}`, i, big))
}
p := readOf(t, b, `{"log":"changes","from":1,"limit":1000}`)
if len(p.Entries) == 0 || len(p.Entries) >= 20 || p.Next != p.Entries[len(p.Entries)-1].Sequence+1 {
t.Fatalf("a page of 20 entries of 250 KiB: %d entries, next %d", len(p.Entries), p.Next)
}
got := len(p.Entries)
for p.Next != 0 {
p = readOf(t, b, fmt.Sprintf(`{"log":"changes","from":%d,"limit":1000}`, p.Next))
got += len(p.Entries)
}
if got != 20 {
t.Fatalf("paging read %d of 20 entries", got)
}
}
// An append with `expect` is written only when the key's last entry is the one expected — 0 when it
// has none — and is otherwise refused with the error a lost mesh/state.update answers: the same code,
// the same error, and nothing written. An append without `expect` is not checked.
func TestAnAppendAtAStaleExpectIsRefusedAsALostUpdate(t *testing.T) {
b, _, _ := logBuses(t, 1<<20)
isLost := func(err error) bool {
var coded interface{ ErrorCode() int }
return errors.As(err, &coded) && coded.ErrorCode() == launch.CodeChanged && errors.Is(err, bus.ErrStateChanged)
}
first := appendOf(t, b, `{"log":"changes","key":"7","value":{"op":"open"},"expect":0}`)
if _, err := logAsk(b, "append", `{"log":"changes","key":"7","value":{"op":"open"},"expect":0}`); !isLost(err) {
t.Fatalf("a second open of one key at expect 0: %v", err)
}
second := appendOf(t, b, fmt.Sprintf(`{"log":"changes","key":"7","value":{"op":"move"},"expect":%d}`, first))
if _, err := logAsk(b, "append", fmt.Sprintf(`{"log":"changes","key":"7","value":{"op":"note"},"expect":%d}`, first)); !isLost(err) {
t.Fatalf("an append at a stale expect: %v", err)
}
if _, err := logAsk(b, "append", fmt.Sprintf(`{"log":"changes","key":"7","value":{"op":"note"},"expect":%d}`, second+100)); !isLost(err) {
t.Fatalf("an append at an expect not yet written: %v", err)
}
// Another key's entries do not make this key's expect stale.
appendOf(t, b, `{"log":"changes","key":"8","value":{"op":"open"}}`)
third := appendOf(t, b, fmt.Sprintf(`{"log":"changes","key":"7","value":{"op":"note"},"expect":%d}`, second))
// No expect: no check.
appendOf(t, b, `{"log":"changes","key":"7","value":{"op":"set"}}`)
if info := infoOf(t, b); info["entries"] != 5 {
t.Fatalf("refused appends wrote: %d entries, not 5", info["entries"])
}
p := readOf(t, b, `{"log":"changes","from":1,"keys":"7"}`)
if len(p.Entries) != 4 || p.Entries[2].Sequence != third {
t.Fatalf("key 7 holds %+v", p.Entries)
}
// The refusal is the very error a lost state update is: its code reaches a bundle as CodeChanged.
_, err := logAsk(b, "append", `{"log":"changes","key":"7","value":{},"expect":0}`)
var coded interface{ ErrorCode() int }
if !errors.As(err, &coded) || coded.ErrorCode() != -32010 {
t.Fatalf("the refusal carries no lost-update code: %v", err)
}
}
// A log is its owner's alone: another module on the same runtime is refused it with the reason, and so
// is a log of a name the owner was not issued — before anything is sent.
func TestALogNotIssuedIsRefusedWithTheReason(t *testing.T) {
issues, other, _ := logBuses(t, 1<<20)
for verb, params := range map[string]string{
"append": `{"log":"changes","key":"1","value":{}}`,
"read": `{"log":"changes","from":1}`,
"last": `{"log":"changes","key":"1"}`,
"info": `{"log":"changes"}`,
} {
_, err := logAsk(other, verb, params)
if err == nil || !strings.Contains(err.Error(), `other keeps no log called "changes"`) ||
!strings.Contains(err.Error(), "issued: none") {
t.Errorf("%s of a log not issued: %v", verb, err)
}
}
_, err := logAsk(issues, "append", `{"log":"moves","key":"1","value":{}}`)
if err == nil || !strings.Contains(err.Error(), `issues keeps no log called "moves"`) ||
!strings.Contains(err.Error(), "issued: changes") {
t.Errorf("an append to a log of a name not issued: %v", err)
}
if info := infoOf(t, issues); info["entries"] != 0 {
t.Fatalf("a refused append wrote: %v", info)
}
}
// What the bus cannot hold, or the contract does not allow, is refused with the reason and not sent:
// a value that is not JSON or over 256 KiB, a key that is not names joined by dots or over 255 bytes, a
// key pattern with `>` not last, a limit over 1000.
func TestWhatALogCannotHoldIsRefusedWithTheReason(t *testing.T) {
b, _, _ := logBuses(t, 1<<20)
over := `"` + strings.Repeat("x", 256*1024) + `"`
for params, reason := range map[string]string{
`{"log":"changes","key":"1","value":nope}`: "names no log",
`{"log":"changes","key":"1"}`: "is JSON",
`{"log":"changes","key":"1","value":` + over + `}`: "at most 262144 bytes",
`{"log":"changes","key":"a b","value":1}`: "is not a log key",
`{"log":"changes","key":"a..b","value":1}`: "is not a log key",
`{"log":"changes","key":"a.*","value":1}`: "is not a log key",
`{"log":"changes","key":"` + strings.Repeat("k", 256) + `","value":1}`: "is not a log key",
`{"key":"1","value":1}`: "names no log",
} {
if _, err := logAsk(b, "append", params); err == nil || !strings.Contains(err.Error(), reason) {
t.Errorf("append %.80s: %v, want %q", params, err, reason)
}
}
for params, reason := range map[string]string{
`{"log":"changes","from":1,"keys":"a.>.b"}`: "is not a key pattern",
`{"log":"changes","from":1,"limit":1001}`: "1 to 1000",
`{"log":"changes","from":1,"limit":-1}`: "1 to 1000",
} {
if _, err := logAsk(b, "read", params); err == nil || !strings.Contains(err.Error(), reason) {
t.Errorf("read %s: %v, want %q", params, err, reason)
}
}
if info := infoOf(t, b); info["entries"] != 0 {
t.Fatalf("a refused append wrote: %v", info)
}
// The largest value allowed is taken, also with the header an expect carries.
largest := `"` + strings.Repeat("x", 256*1024-2) + `"`
appendOf(t, b, `{"log":"changes","key":"1","value":`+largest+`,"expect":0}`)
appendOf(t, b, `{"log":"changes","key":"2","value":`+largest+`}`)
}
// A value naming a credential anywhere in it is refused, as mesh/state.put refuses one (novox/hq ADR
// 0201 §13): a log keeps what it is given for good. A value that only counts tokens is taken.
func TestALogRefusesAValueNamingACredential(t *testing.T) {
b, _, _ := logBuses(t, 1<<20)
for _, value := range []string{`{"op":"set","api_key":"s"}`, `{"op":"note","with":{"headers":{"Authorization":"Bearer s"}}}`} {
_, err := logAsk(b, "append", `{"log":"changes","key":"1","value":`+value+`}`)
if err == nil || !strings.Contains(err.Error(), "names a credential") {
t.Errorf("append of %s: %v", value, err)
}
}
if info := infoOf(t, b); info["entries"] != 0 {
t.Fatalf("a refused append wrote: %v", info)
}
appendOf(t, b, `{"log":"changes","key":"1","value":{"op":"note","maxTokens":4096}}`)
}
@@ -0,0 +1,56 @@
package runtime
import (
"strings"
"testing"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// novox/hq ADR 0259 §8: a runtime on a module's own account serves that module alone, and the machine's
// runtime never serves itself.
func TestARuntimeOnAModulesOwnAccountServesThatModuleAlone(t *testing.T) {
if got, err := ServedModulesFrom("telegram=/b/telegram/tools/telegram", "telegram"); err != nil || len(got) != 1 {
t.Fatalf("a module's own runtime could not serve it: %v", err)
}
if _, err := ServedModulesFrom("telegram=/b/t,dunst=/b/d", "telegram"); err == nil || !strings.Contains(err.Error(), "serves telegram alone") {
t.Errorf("a module's own runtime served another: %v", err)
}
if _, err := ServedModulesFrom("node-tools=/b/n", "node-tools"); err == nil {
t.Error("the machine's runtime served itself")
}
}
// No bus word reaches a bundle (security review of 2026-10-08), not even one its module's words name.
func TestNoBusWordReachesABundle(t *testing.T) {
base := []string{"MESH_BROKER_FILE=/etc/mesh/broker", "MESH_BROKER_URL=nats://x", "MESH_TOOL_ENV={}",
"MESH_TOOL_MODULES=a=/b", "MESH_CONSOLE_LISTEN=127.0.0.1:1", "PATH=/usr/bin"}
env := strings.Join(BundleEnv(base, map[string]string{"MESH_BROKER_FILE": "/again", "OWN": "1"}, "telegram", "anchor"), "\n")
for _, never := range []string{"MESH_BROKER_FILE", "MESH_BROKER_URL", "MESH_TOOL_ENV", "MESH_TOOL_MODULES", "MESH_CONSOLE_LISTEN"} {
if strings.Contains(env, never+"=") {
t.Errorf("%s reached the bundle", never)
}
}
for _, want := range []string{"PATH=/usr/bin", "OWN=1", "MESH_MODULE=telegram", "MESH_NODE=anchor"} {
if !strings.Contains(env, want) {
t.Errorf("%s did not reach the bundle", want)
}
}
}
// A tool call reaches only a tool's subject: whatever key a caller names, it is never a seat's event, accept
// or proof — so no tool call says a choice, a link or a code, or submits an ask, in anybody's name.
func TestAToolCallNeverReachesASeatsEventAcceptOrProof(t *testing.T) {
for _, key := range []string{"seat:intake.event.choice.telegram", "seat:intake.proof.code.telegram",
"seat:operator-channel.accept.ask.mesh-delivery", "intake.event", "seat:operator-channel.event.decided.x@anchor",
"telegram.anything", "x"} {
subject, err := bus.ToolSubject(key, "node-tools")
if err != nil {
continue
}
tokens := strings.Split(subject, ".")
if len(tokens) < 4 || tokens[3] != "tool" {
t.Errorf("%q reaches %s, which is not a tool's subject", key, subject)
}
}
}
+870
View File
@@ -0,0 +1,870 @@
// Package runtime is the node's tool runtime (novox/hq ADR 0175, ADR 0193), the Go port of
// node-tools' runtime.ts in its launch-only form: it launches every assigned module's bundle,
// serves each module's tools on that module's subjects and each held seat's verbs on the seat's,
// and answers for every module the verb that says what it serves. It imports nothing and knows no
// language.
package runtime
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"github.com/novox/mesh-tools/node-tools/internal/announce"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
)
// ToolsVerb is the verb every module's runtime answers for it (ADR 0152): its tools, from the code
// that answers them.
const ToolsVerb = "tools"
// Words the mesh sets for the runtime (ADR 0175, ADR 0192).
const (
ToolModules = "MESH_TOOL_MODULES"
ToolEnv = "MESH_TOOL_ENV"
OperatorAccount = "MESH_OPERATOR_ACCOUNT"
OperatorHome = "MESH_OPERATOR_HOME"
)
// Served is one module this runtime serves and its entrypoints.
type Served struct {
Module string
Entrypoints []string
}
// ToolsAnswer is what `tools` answers for one module.
type ToolsAnswer struct {
Module string `json:"module"`
Tools []ListedTool `json:"tools"`
Failed string `json:"failed,omitempty"`
}
// ListedTool is one tool as a module's `tools` answer lists it.
type ListedTool struct {
Name string `json:"name"`
Description string `json:"description"`
Input json.RawMessage `json:"input"`
Subjects []string `json:"subjects,omitempty"`
}
// NodeRuntimeModule is the module that is a machine's runtime, serving every module on it.
const NodeRuntimeModule = "node-tools"
// runtimeOnly are the words of the runtime's own environment no bundle is given.
var runtimeOnly = map[string]bool{ToolEnv: true, ToolModules: true, "MESH_BROKER_FILE": true,
"MESH_BROKER_URL": true, "MESH_CONSOLE_LISTEN": true}
// ServedModulesFrom reads MESH_TOOL_MODULES: `<module>=<entrypoint>` entries, comma-separated, several
// per module. The one-module form — a bare path, or the runtime's own module — is the per-module
// containers' (to-be 38 WP4c) and refused here: the node's runtime imports nothing.
func ServedModulesFrom(spec, own string) ([]Served, error) {
order := []string{}
by := map[string][]string{}
for _, raw := range strings.Split(spec, ",") {
entry := strings.TrimSpace(raw)
if entry == "" {
continue
}
module, path, ok := strings.Cut(entry, "=")
module, path = strings.TrimSpace(module), strings.TrimSpace(path)
if !ok || module == "" || path == "" {
return nil, fmt.Errorf("%s: %q is not <module>=<entrypoint>; the runtime launches the bundles it "+
"is given and imports nothing (novox/hq ADR 0193)", ToolModules, entry)
}
// The node's runtime serves the machine's modules and never itself. **A runtime on a module's own
// account serves that module and nothing else** (novox/hq ADR 0259 §8): the router and a channel that
// proves its sender reach the bus on an account of their own, never the machine's runtime.
switch {
case own == NodeRuntimeModule && module == own:
return nil, fmt.Errorf("%s: %q is the runtime itself; it launches the bundles it is given and "+
"imports nothing (novox/hq ADR 0193)", ToolModules, entry)
case own != NodeRuntimeModule && module != own:
return nil, fmt.Errorf("%s: %q is another module's; a runtime on %s's own account serves %s alone "+
"(novox/hq ADR 0259)", ToolModules, entry, own, own)
}
if _, seen := by[module]; !seen {
order = append(order, module)
}
by[module] = append(by[module], path)
}
out := make([]Served, 0, len(order))
for _, m := range order {
out = append(out, Served{Module: m, Entrypoints: by[m]})
}
return out, nil
}
// TakeToolEnvs reads the composed environments (ADR 0192) and removes them from the process's, so no
// bundle finds another's there.
func TakeToolEnvs() (map[string]map[string]string, error) {
raw := os.Getenv(ToolEnv)
os.Unsetenv(ToolEnv)
out := map[string]map[string]string{}
if raw == "" {
return out, nil
}
var parsed map[string]map[string]any
if err := json.Unmarshal([]byte(raw), &parsed); err != nil {
return nil, fmt.Errorf(`%s is not JSON of the shape {"<module>": {"<word>": "<value>"}}: %w`, ToolEnv, err)
}
for module, words := range parsed {
own := map[string]string{}
for k, v := range words {
if s, ok := v.(string); ok {
own[k] = s
} else {
b, _ := json.Marshal(v)
own[k] = string(b)
}
}
out[module] = own
}
return out, nil
}
type registration struct {
module string // the module's own name, or a seat's
owner string // the module whose bundle made it
tools []launch.Tool
}
// Run launches, binds and serves. It answers a stop function.
func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, logf func(string, ...any)) (func(), error) {
if account := os.Getenv(OperatorAccount); account != "" {
home := ""
if h := os.Getenv(OperatorHome); h != "" {
home = " (home " + h + ")"
}
logf("[mesh-tools] the operator's account here is %s%s", account, home)
}
modules := make([]string, 0, len(served))
isServed := map[string]bool{}
for _, s := range served {
modules = append(modules, s.Module)
isServed[s.Module] = true
conn.Follow(s.Module)
}
node := conn.Node()
base := os.Environ()
envFor := func(module string) []string { return BundleEnv(base, envs[module], module, node) }
// The module's events, for every child of it that subscribes (ADR 0198): one consumer per module,
// bound the first time any of its children subscribes, each event handed to every child that did.
events := &consumers{conn: conn, logf: logf, of: map[string]*moduleEvents{}}
failed := map[string]string{}
var registrations []registration
var stops []func()
for _, s := range served {
for _, entry := range s.Entrypoints {
path, _ := filepath.Abs(entry)
module := s.Module
fail := func(why string) {
failed[module] = why
logf("[mesh-tools] %s's bundle %s failed to load: %s; its tools are not served here", module, entry, why)
}
if !launch.Executable(path) {
fail(path + " is not executable; a bundle the runtime serves is started, never imported, and its build makes it executable (novox/hq ADR 0193)")
continue
}
child, err := launch.Start(module, path, envFor(module), events.forModule(module), logf)
if err != nil {
fail(err.Error())
continue
}
stops = append(stops, child.Stop)
for _, r := range child.Registrations {
registrations = append(registrations, registration{module: r.Module, owner: module, tools: r.Tools})
}
}
}
claimed := map[string]bool{}
for _, m := range modules {
if mem := conn.Membership(m); mem != nil {
for _, s := range mem.Seats {
claimed[s.Seat] = true
}
}
}
var own []registration
for _, r := range registrations {
switch {
case isServed[r.module]:
own = append(own, r)
case claimed[r.module]:
default:
logf(`[mesh-tools] %s registers tools under "%s", which is neither a module served here nor a seat one of them claims; not served until the mesh issues the claim`, r.owner, r.module)
}
}
stopAll := func() {
events.stopAll()
for i := len(stops) - 1; i >= 0; i-- {
stops[i]()
}
}
for _, r := range own {
seen := map[string]bool{}
for _, t := range r.tools {
if t.Name == ToolsVerb {
stopAll()
return nil, fmt.Errorf(`%s names a tool "%s", which is the verb the runtime answers for every module with what it serves (novox/hq ADR 0152) — refused, rename it`, r.module, ToolsVerb)
}
if seen[t.Name] {
stopAll()
return nil, fmt.Errorf("%s exposes two tools named %s — refused", r.module, t.Name)
}
seen[t.Name] = true
}
}
var names []string
byModule := map[string][]launch.Tool{}
for _, r := range own {
for _, t := range r.tools {
t := t
// Told who called, from the subject the call arrived on, never from its body (novox/hq issue 365).
stop, err := conn.HandleCalled(r.module+"."+t.Name, func(caller string, body json.RawMessage) (any, error) {
return t.Run(caller, argsOf(body))
})
if err != nil {
logf("[mesh-tools] cannot serve %s.%s: %v", r.module, t.Name, err)
continue
}
names = append(names, r.module+"."+t.Name)
stops = append(stops, stop)
}
byModule[r.module] = append(byModule[r.module], r.tools...)
}
for _, module := range modules {
module := module
tools := byModule[module]
why := failed[module]
if len(tools) == 0 && why == "" {
continue // a pure-events module: silent, as it always was
}
stop, err := conn.Handle(module+"."+ToolsVerb, func(json.RawMessage) (any, error) {
answer := ToolsAnswer{Module: module, Tools: []ListedTool{}, Failed: why}
for _, t := range tools {
answer.Tools = append(answer.Tools, ListedTool{Name: t.Name, Description: t.Description,
Input: t.Input, Subjects: subjectsOf(conn, module, t.Name)})
}
return answer, nil
})
if err == nil {
stops = append(stops, stop)
}
}
failedNames := make([]string, 0, len(failed))
for _, m := range modules {
if _, f := failed[m]; f {
failedNames = append(failedNames, m)
}
}
line := fmt.Sprintf("[mesh-tools] serving %d tool(s) for %d module(s): %s", len(names), len(modules), orNone(names))
if len(failedNames) > 0 {
line += fmt.Sprintf("; not serving %s, whose bundle(s) failed to load", strings.Join(failedNames, ", "))
}
logf("%s", line)
stops = append(stops, serveSeats(conn, modules, registrations, logf))
// **What it serves, it announces** (novox/hq ADR 0197): the NATS services protocol's discovery,
// answered with what is served at the moment it is asked — re-served memberships included.
announced, err := announce.Serve(conn, announce.Service{
Name: conn.Module(), ID: instanceOf(conn),
Description: "the mesh's tool runtime on " + node + ": every assigned module's tools and the seats they hold",
Metadata: map[string]string{"node": node},
}, func() []announce.Endpoint { return endpointsOf(conn, own, registrations, modules) })
if err != nil {
logf("[mesh-tools] cannot announce what it serves: %v", err)
} else {
stops = append(stops, announced)
}
conn.Flush()
return stopAll, nil
}
// instanceOf is this runtime's instance on the bus: its machine, which is what tells two instances of
// one service apart; the connection's module where it has no machine.
func instanceOf(conn *bus.Conn) string {
if n := conn.Node(); n != "" {
return n
}
return conn.Module()
}
// endpointsOf is everything this runtime serves now: each served module's tools on every subject the
// mesh issued for them, and each held seat's verbs on the seat's subject (ADR 0197).
func endpointsOf(conn *bus.Conn, own []registration, registrations []registration, modules []string) []announce.Endpoint {
node := conn.Node()
var out []announce.Endpoint
for _, r := range own {
for _, t := range r.tools {
served := conn.ServedOn(r.module, t.Name)
interchangeable := false
for _, s := range served {
interchangeable = interchangeable || s.Subject == "mesh.mod."+r.module+".tool."+t.Name
}
for _, s := range served {
out = append(out, announce.Endpoint{Kind: announce.KindTool, Module: r.module, Tool: t.Name,
Node: node, Description: t.Description, Schema: t.Input, Interchangeable: interchangeable,
Subject: s.Subject, Queue: s.Queue})
}
}
}
impl := map[string]map[string]launch.Tool{}
for _, r := range registrations {
if impl[r.module] == nil {
impl[r.module] = map[string]launch.Tool{}
}
for _, t := range r.tools {
impl[r.module][t.Name] = t
}
}
have := map[string]bool{}
for _, module := range modules {
m := conn.Membership(module)
if m == nil {
continue
}
for _, v := range m.Seats {
t, ok := impl[v.Seat][v.Verb]
if !ok || have[v.Subject] {
continue
}
have[v.Subject] = true
scope := "mesh"
if node != "" && strings.HasSuffix(v.Subject, "."+node) {
scope = "node"
}
out = append(out, announce.Endpoint{Kind: announce.KindSeat, Module: module, Tool: v.Verb, Seat: v.Seat,
Scope: scope, Node: node, Description: t.Description, Schema: t.Input, Subject: v.Subject})
}
}
return out
}
func orNone(names []string) string {
if len(names) == 0 {
return "(none)"
}
return strings.Join(names, ", ")
}
// argsOf is a call's arguments as the tool receives them: an object, `{}` for none.
func argsOf(body json.RawMessage) json.RawMessage {
trimmed := strings.TrimSpace(string(body))
if trimmed == "" || trimmed == "null" {
return json.RawMessage("{}")
}
return body
}
// subjectsOf is where a tool is answered as the mesh issued it: the plain subject first, then this
// machine's; nothing before a membership is issued.
func subjectsOf(conn *bus.Conn, module, tool string) []string {
m := conn.Membership(module)
if m == nil {
return nil
}
var plain, mine []string
for _, s := range m.Serves {
subject := strings.ReplaceAll(s.Subject, "{tool}", tool)
if s.Queue != "" {
plain = append(plain, subject)
} else {
mine = append(mine, subject)
}
}
return append(plain, mine...)
}
// serveSeats serves every verb of every seat a served module holds, where the mesh issued it, by
// the tool of the same name registered under the seat's name (ADR 0159, 0160) — and serves again
// whenever a membership changes. Whether this machine holds the seat is the bus's to decide.
func serveSeats(conn *bus.Conn, modules []string, registrations []registration, logf func(string, ...any)) func() {
impl := map[string]map[string]launch.Tool{}
for _, r := range registrations {
if impl[r.module] == nil {
impl[r.module] = map[string]launch.Tool{}
}
for _, t := range r.tools {
impl[r.module][t.Name] = t
}
}
var mu sync.Mutex
var stops []func()
serve := func() {
mu.Lock()
defer mu.Unlock()
for _, s := range stops {
s()
}
stops = nil
have := map[string]bool{}
for _, module := range modules {
m := conn.Membership(module)
if m == nil {
continue
}
for _, v := range m.Seats {
if have[v.Subject] {
continue
}
have[v.Subject] = true
t, ok := impl[v.Seat][v.Verb]
if !ok {
logf("[mesh-tools] %s claims %s and implements no %s, which that seat promises; not served", module, v.Seat, v.Verb)
continue
}
stop, err := conn.HandleSubjectCalled(v.Subject, func(caller string, body json.RawMessage) (any, error) {
return t.Run(caller, argsOf(body))
})
if err != nil {
logf("[mesh-tools] cannot serve %s's %s on %s: %v", v.Seat, v.Verb, v.Subject, err)
continue
}
stops = append(stops, stop)
logf("[mesh-tools] serving %s's %s on %s, admitted where %s holds the seat", v.Seat, v.Verb, v.Subject, module)
}
}
}
serve()
conn.OnMembership(func(bus.Membership) { go func() { serve(); conn.Flush() }() })
return func() {
mu.Lock()
defer mu.Unlock()
for _, s := range stops {
s()
}
stops = nil
}
}
// consumers holds, per module the runtime serves, the one durable consumer its events arrive on and
// the children its events are handed to (novox/hq ADR 0198).
type consumers struct {
conn *bus.Conn
logf func(string, ...any)
mu sync.Mutex
of map[string]*moduleEvents
// seatBound is each module's seat traffic bound once: a worker taken, a proof subject answered.
seatBound map[string]func()
}
type moduleEvents struct {
stop func()
delivers []*func(json.RawMessage) error
}
// stopAll unbinds every module's consumer.
func (c *consumers) stopAll() {
c.mu.Lock()
defer c.mu.Unlock()
for _, m := range c.of {
if m.stop != nil {
m.stop()
}
}
c.of = map[string]*moduleEvents{}
for _, stop := range c.seatBound {
stop()
}
c.seatBound = map[string]func(){}
}
// forModule is the bus one launched bundle of a module reaches the mesh through.
func (c *consumers) forModule(module string) launch.Bus {
return &moduleBus{all: c, module: module}
}
type moduleBus struct {
all *consumers
module string
mu sync.Mutex
deliver *func(json.RawMessage) error
}
// Publish emits an event as the module (ADR 0193).
func (b *moduleBus) Publish(params json.RawMessage) error {
var env bus.Envelope
if err := json.Unmarshal(params, &env); err != nil {
return fmt.Errorf("not an event envelope: %w", err)
}
return b.all.conn.PublishAs(b.module, env)
}
// Ask calls a tool as the module: `{key, body}`, answered with the tool's result (ADR 0198).
func (b *moduleBus) Ask(params json.RawMessage) (json.RawMessage, error) {
var asked struct {
Key string `json:"key"`
Body json.RawMessage `json:"body"`
}
if err := json.Unmarshal(params, &asked); err != nil || asked.Key == "" {
return nil, fmt.Errorf("mesh/ask names no tool: {key, body}")
}
body := any(asked.Body)
if len(asked.Body) == 0 {
body = map[string]any{}
}
answered, err := b.all.conn.AskAs(b.module, asked.Key, body)
if err != nil {
return nil, err
}
if len(answered.Result) == 0 {
return json.RawMessage("null"), nil
}
return answered.Result, nil
}
// Subscribe hands this bundle the module's events. The consumer is bound once per module; each of
// the module's children that subscribed is handed every event, and the event is acknowledged only
// when all of them took it — one consumer split between two readers would give each half.
func (b *moduleBus) Subscribe(deliver func(json.RawMessage) error) error {
b.mu.Lock()
if b.deliver == nil {
d := deliver
b.deliver = &d
} else {
*b.deliver = deliver
}
mine := b.deliver
b.mu.Unlock()
c := b.all
c.mu.Lock()
defer c.mu.Unlock()
m := c.of[b.module]
if m == nil {
m = &moduleEvents{}
c.of[b.module] = m
}
listed := false
for _, d := range m.delivers {
if d == mine {
listed = true
}
}
if !listed {
m.delivers = append(m.delivers, mine)
}
if m.stop != nil {
return nil
}
module := b.module
stop, err := c.conn.ConsumeAs(module, func(env bus.Envelope) error {
raw, err := json.Marshal(env)
if err != nil {
return err
}
c.mu.Lock()
targets := append([]*func(json.RawMessage) error(nil), c.of[module].delivers...)
c.mu.Unlock()
for _, d := range targets {
if err := (*d)(raw); err != nil {
c.logf("[mesh-tools] %s did not take %s%s: %s; offered again in %s", module, env.Key,
eventRef(env), whyNotTaken(err), bus.NakDelay)
return err
}
}
return nil
})
if err != nil {
return err
}
m.stop = stop
c.logf("[mesh-tools] %s's events arrive on its consumer %s", module, bus.ConsumerOf(c.conn.Node(), module))
return nil
}
// whyNotTaken says why a bundle did not take an event, so the line names who failed: the module's
// handler, in its own words — an error it answered — or the runtime not reaching it. A bundle's
// handler error read bare ("Unexpected end of JSON input") looked like the runtime's (issue 276).
func whyNotTaken(err error) string {
var refused *launch.Refused
if errors.As(err, &refused) {
return fmt.Sprintf("its handler answered an error, in its own words: %q (an event is taken by any answer that is not an error)", refused.Message)
}
return err.Error()
}
// eventRef names the event a line is about by its id, when it has one, so its offers can be told
// apart from another event's.
func eventRef(env bus.Envelope) string {
if id := env.Headers["x-event-id"]; id != "" {
return " (event " + id + ")"
}
return ""
}
// stateAsked is what a bundle names when it reaches its state (ADR 0201): the state by the name its
// module uses, a key, and for a put the value.
type stateAsked struct {
State string `json:"state"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Revision uint64 `json:"revision"`
}
// State answers a bundle's `get`, `put`, `delete`, `keys`, `create`, `update` and `history` on its
// module's state (ADR 0201, ADR 0259, ADR 0307). The runtime refuses, with the reason, a state the module was not issued and a write to one it only reads.
func (b *moduleBus) State(verb string, params json.RawMessage) (json.RawMessage, error) {
var asked stateAsked
if err := json.Unmarshal(params, &asked); err != nil || asked.State == "" {
return nil, fmt.Errorf("mesh/state.%s names no state: {state, key, value}", verb)
}
conn := b.all.conn
var answer any
switch verb {
case "get":
entry, err := conn.StateGet(b.module, asked.State, asked.Key)
if err != nil {
return nil, err
}
if entry == nil {
return json.RawMessage("null"), nil
}
answer = entry
case "history":
// Every value the bucket still keeps for the key, oldest first (novox/hq ADR 0307).
kept, err := conn.StateHistory(b.module, asked.State, asked.Key)
if err != nil {
return nil, err
}
answer = kept
case "put":
revision, err := conn.StatePut(b.module, asked.State, asked.Key, asked.Value)
if err != nil {
return nil, err
}
answer = map[string]any{"revision": revision}
case "delete":
if err := conn.StateDelete(b.module, asked.State, asked.Key); err != nil {
return nil, err
}
answer = map[string]any{}
case "keys":
keys, err := conn.StateKeys(b.module, asked.State)
if err != nil {
return nil, err
}
answer = keys
case "create":
// Only when the key has no value: of two writers making it, one wins (novox/hq ADR 0259).
revision, err := conn.StateCreate(b.module, asked.State, asked.Key, asked.Value)
if err != nil {
return nil, err
}
answer = map[string]any{"revision": revision}
case "update":
// Only when the key is still at the revision read: a second answer to one ask loses (ADR 0259).
revision, err := conn.StateUpdate(b.module, asked.State, asked.Key, asked.Value, asked.Revision)
if err != nil {
return nil, err
}
answer = map[string]any{"revision": revision}
default:
return nil, fmt.Errorf("the runtime answers no mesh/state.%s", verb)
}
return json.Marshal(answer)
}
// logAsked is what a bundle names when it reaches its log (novox/hq ADR 0297).
type logAsked struct {
Log string `json:"log"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Expect *uint64 `json:"expect"`
From uint64 `json:"from"`
Keys string `json:"keys"`
Limit int `json:"limit"`
}
// Log answers a bundle's `append`, `read`, `last` and `info` on its module's log (ADR 0297). The runtime
// refuses, with the reason, a log the module was not issued; an append whose `expect` is stale is
// refused with the code of a lost state update.
func (b *moduleBus) Log(verb string, params json.RawMessage) (json.RawMessage, error) {
var asked logAsked
if err := json.Unmarshal(params, &asked); err != nil || asked.Log == "" {
return nil, fmt.Errorf("mesh/log.%s names no log: {log, ...}", verb)
}
conn := b.all.conn
var answer any
switch verb {
case "append":
seq, err := conn.LogAppend(b.module, asked.Log, asked.Key, asked.Value, asked.Expect)
if err != nil {
return nil, err
}
answer = map[string]any{"sequence": seq}
case "read":
page, err := conn.LogRead(b.module, asked.Log, asked.From, asked.Keys, asked.Limit)
if err != nil {
return nil, err
}
answer = page
case "last":
entry, err := conn.LogLast(b.module, asked.Log, asked.Key)
if err != nil {
return nil, err
}
if entry == nil {
return json.RawMessage("null"), nil
}
answer = entry
case "info":
info, err := conn.LogInfo(b.module, asked.Log)
if err != nil {
return nil, err
}
answer = info
default:
return nil, fmt.Errorf("the runtime answers no mesh/log.%s", verb)
}
return json.Marshal(answer)
}
// Watch hands a bundle its module's state as it is and as it changes (ADR 0201): `{state, key}`, the
// key a pattern with `*` and `**`, empty for every key.
func (b *moduleBus) Watch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
var asked stateAsked
if err := json.Unmarshal(params, &asked); err != nil || asked.State == "" {
return nil, fmt.Errorf("mesh/state.watch names no state: {state, key}")
}
return b.all.conn.StateWatch(b.module, asked.State, asked.Key, func(change bus.StateChange) error {
raw, err := json.Marshal(change)
if err != nil {
return err
}
return deliver(raw)
})
}
// seatAsked is what a bundle names for its seat traffic (novox/hq ADR 0259 §3).
type seatAsked struct {
Subject string `json:"subject"`
Body json.RawMessage `json:"body"`
ID string `json:"id"`
Worker string `json:"worker"`
Bucket string `json:"bucket"`
Key string `json:"key"`
}
// Seat answers a bundle's seat traffic, each as the module and only as far as its membership lists:
// `publish` an accept or an event, `prove` (ask a proof), `record` (read a record kept for it), `take` a
// worker's work and `answer` a proof subject — the last two bound once per module and subject, each
// message or proof handed to the module's child that asked last (novox/hq ADR 0259 §3).
func (b *moduleBus) Seat(verb string, params json.RawMessage, handOn func(string, any) (json.RawMessage, error)) (json.RawMessage, error) {
var asked seatAsked
if err := json.Unmarshal(params, &asked); err != nil {
return nil, fmt.Errorf("mesh/seat.%s: %w", verb, err)
}
conn := b.all.conn
switch verb {
case "publish":
seq, duplicate, err := conn.SeatPublishSaid(b.module, asked.Subject, asked.Body, asked.ID)
if err != nil {
return nil, err
}
return json.Marshal(map[string]any{"sequence": seq, "duplicate": duplicate})
case "prove":
return conn.SeatProve(b.module, asked.Subject, asked.Body)
case "kinds":
kinds, err := conn.SeatKinds(b.module)
if err != nil {
return nil, err
}
return json.Marshal(kinds)
case "record":
entry, err := conn.SeatRecord(b.module, asked.Bucket, asked.Key)
if err != nil || entry == nil {
return json.RawMessage("null"), err
}
return json.Marshal(entry)
case "take":
key := "take " + asked.Worker
if b.bound(key) {
return json.RawMessage(`{}`), nil
}
stop, err := conn.SeatTake(b.module, asked.Worker, func(w bus.Work) error {
_, err := handOn("mesh/work", map[string]any{"work": w})
return err
})
if err != nil {
return nil, err
}
b.bind(key, stop)
b.all.logf("[mesh-tools] %s takes its work from %s", b.module, asked.Worker)
return json.RawMessage(`{}`), nil
case "answer":
key := "answer " + asked.Subject
if b.bound(key) {
return json.RawMessage(`{}`), nil
}
stop, err := conn.SeatAnswer(b.module, asked.Subject, func(subject string, body json.RawMessage) (json.RawMessage, error) {
return handOn("mesh/proof", map[string]any{"subject": subject, "body": body})
})
if err != nil {
return nil, err
}
b.bind(key, stop)
b.all.logf("[mesh-tools] %s answers the proofs on %s", b.module, asked.Subject)
return json.RawMessage(`{}`), nil
}
return nil, fmt.Errorf("the runtime answers no mesh/seat.%s", verb)
}
// bound and bind keep, per module, what its seat traffic is bound to, so a child that starts again and
// asks again is handed what is already bound instead of a second reader of it.
func (b *moduleBus) bound(key string) bool {
b.all.mu.Lock()
defer b.all.mu.Unlock()
_, ok := b.all.seatBound[b.module+" "+key]
return ok
}
func (b *moduleBus) bind(key string, stop func()) {
b.all.mu.Lock()
defer b.all.mu.Unlock()
if b.all.seatBound == nil {
b.all.seatBound = map[string]func(){}
}
b.all.seatBound[b.module+" "+key] = stop
}
// BundleEnv is what one module's bundle is started with: the runtime's environment without its own words,
// the module's composed words over it, and the module's and machine's names. **No bus word reaches a
// bundle** (security review of 2026-10-08): the credential and the bus's address are the runtime's, and a
// bundle reaches the bus only through the runtime.
func BundleEnv(base []string, given map[string]string, module, node string) []string {
words := map[string]string{}
for _, kv := range base {
if k, v, ok := strings.Cut(kv, "="); ok && !runtimeOnly[k] {
words[k] = v
}
}
for k, v := range given {
if !runtimeOnly[k] {
words[k] = v
}
}
words["MESH_SERVED_MODULE"] = module
words["MESH_MODULE"] = module
if node != "" {
words["MESH_NODE"] = node
}
out := make([]string, 0, len(words))
for k, v := range words {
out = append(out, k+"="+v)
}
sort.Strings(out)
return out
}
+268
View File
@@ -0,0 +1,268 @@
package runtime
import (
"encoding/json"
"errors"
"os"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
func connect(t *testing.T, module, node string) *bus.Conn {
t.Helper()
c, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: module, Node: node})
if err != nil {
t.Fatal(err)
}
c.Logf = func(string, ...any) {}
t.Cleanup(c.Close)
return c
}
func call(t *testing.T, asker *bus.Conn, key string, args any) (string, error) {
t.Helper()
got, err := asker.Ask(key, args, "")
return string(got.Result), err
}
func same(t *testing.T, got, want string) {
t.Helper()
var a, b any
if json.Unmarshal([]byte(got), &a) != nil || json.Unmarshal([]byte(want), &b) != nil {
t.Fatalf("not JSON: got %s want %s", got, want)
}
ga, _ := json.Marshal(a)
gb, _ := json.Marshal(b)
if string(ga) != string(gb) {
t.Errorf("got %s, want %s", got, want)
}
}
// The node's runtime serves five modules' bundles on one credential — two TypeScript, one broken,
// one Python, one written against the protocol — and follows a re-issued membership (ADR 0175, 0193).
func TestTheNodesRuntimeServesFiveModulesAndFollowsAReissuedMembership(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", true, nil))
mesh.Issue(t, mt.MembershipOf("beta", "anchor", false, map[string][]string{"node-shelf": {"list", "clear"}}))
mesh.Issue(t, mt.MembershipOf("gamma", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("delta", "anchor", false, map[string][]string{"node-lamp": {"on"}}))
mesh.Issue(t, mt.MembershipOf("epsilon", "anchor", false, nil))
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
logs := &mt.Logs{}
t.Setenv("MESH_OPERATOR_ACCOUNT", "somebody")
t.Setenv("MESH_OPERATOR_HOME", "/home/somebody")
stop, err := Run(nodeTools, []Served{
{"alpha", []string{mt.Fixture("many-alpha.serve.mjs")}},
{"beta", []string{mt.Fixture("many-beta.serve.mjs")}},
{"gamma", []string{mt.Fixture("many-broken.serve.mjs")}},
{"delta", []string{mt.Fixture("many-delta.py")}},
{"epsilon", []string{mt.Fixture("many-epsilon.mjs")}},
}, nil, logs.Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
if !logs.Has("the operator's account here is somebody (home /home/somebody)") {
t.Errorf("the operator was not said:\n%s", logs.All())
}
if !logs.Has("gamma's bundle", "many-broken.serve.mjs failed to load: gamma's bundle exited (1): Error: gamma's bundle cannot find its client; its tools are not served here") {
t.Errorf("the broken bundle was not named with its own words:\n%s", logs.All())
}
if !logs.Has("serving 8 tool(s) for 5 module(s): alpha.one, alpha.two, beta.three, beta.four, beta.five, delta.greet, delta.die, epsilon.seven; not serving gamma") {
t.Errorf("not serving what it should:\n%s", logs.All())
}
for key, want := range map[string]string{
"alpha.one": `{"alpha":1}`, "alpha.one@anchor": `{"alpha":1}`, "beta.three@anchor": `{"beta":3}`,
"beta.four@anchor": `{"beta":4}`, "beta.five@anchor": `{"beta":5}`,
"seat:node-shelf.list@anchor": `{"shelf":["a","b"]}`, "seat:node-shelf.clear@anchor": `{"cleared":true}`,
"seat:node-lamp.on@anchor": `{"on":true,"language":"python"}`, "epsilon.seven@anchor": `{"epsilon":7,"via":"stdio"}`,
} {
got, err := call(t, asker, key, map[string]any{})
if err != nil {
t.Errorf("%s: %v", key, err)
continue
}
same(t, got, want)
}
if _, err := call(t, asker, "beta.three", map[string]any{}); err == nil || !strings.Contains(err.Error(), "no responders") {
t.Errorf("beta was not issued the plain subject, and answered on it: %v", err)
}
got, err := call(t, asker, "delta.greet@anchor", map[string]any{"who": "mesh"})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"greeting":"hello mesh","language":"python"}`)
// A launched tool that emits does so as its module, through the runtime.
landed := mesh.NextEvent(t, "mesh.mod.*.event.>")
got, err = call(t, asker, "alpha.two", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"alpha":2}`)
select {
case subject := <-landed:
if subject != "mesh.mod.alpha.event.happened" {
t.Errorf("the event landed on %s", subject)
}
case <-time.After(5 * time.Second):
t.Error("the event never landed")
}
// A bundle that dies mid-call is said, and started again on its next call.
if _, err := call(t, asker, "delta.die@anchor", map[string]any{}); err == nil {
t.Error("a bundle that died answered")
}
got, err = call(t, asker, "delta.greet@anchor", map[string]any{"who": "again"})
if err != nil {
t.Fatalf("not started again: %v", err)
}
same(t, got, `{"greeting":"hello again","language":"python"}`)
// `tools` answers for each, and why gamma serves nothing.
gamma, err := call(t, asker, "gamma.tools@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, gamma, `{"module":"gamma","tools":[],"failed":"gamma's bundle exited (1): Error: gamma's bundle cannot find its client"}`)
beta, _ := call(t, asker, "beta.tools@anchor", map[string]any{})
var answer ToolsAnswer
_ = json.Unmarshal([]byte(beta), &answer)
if len(answer.Tools) != 3 || answer.Tools[0].Name != "three" || strings.Join(answer.Tools[0].Subjects, ",") != "mesh.mod.beta.tool.three.anchor" {
t.Errorf("beta's tools answer: %s", beta)
}
// Re-issued mid-run, now answering for the module anywhere: served without a restart.
mesh.Issue(t, mt.MembershipOf("beta", "anchor", true, map[string][]string{"node-shelf": {"list", "clear"}}))
mt.Until(t, func() error { _, err := call(t, asker, "beta.three", map[string]any{}); return err })
got, err = call(t, asker, "seat:node-shelf.list@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"shelf":["a","b"]}`)
}
// Each bundle is given its own environment and none of another's (ADR 0192), and the composed
// environments are not left in the runtime's.
func TestEachBundleIsGivenItsOwnEnvironment(t *testing.T) {
mesh := mt.New(t)
for _, m := range []string{"gamma", "delta", "zeta"} {
mesh.Issue(t, mt.MembershipOf(m, "anchor", false, nil))
}
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
t.Setenv("MESH_OPERATOR_ACCOUNT", "somebody")
t.Setenv(ToolEnv, `{"gamma":{"GAMMA_CONFIG_FILE":"/var/lib/mesh/gamma/config.json"},"delta":{"DELTA_TOKEN_FILE":"/var/lib/mesh/delta/token"},"zeta":{"ZETA_URL":"http://127.0.0.1:3000"}}`)
envs, err := TakeToolEnvs()
if err != nil {
t.Fatal(err)
}
if _, left := os.LookupEnv(ToolEnv); left {
t.Error("the composed environments were left in the runtime's")
}
logs := &mt.Logs{}
stop, err := Run(nodeTools, []Served{
{"gamma", []string{mt.Fixture("env-gamma.serve.mjs")}},
{"delta", []string{mt.Fixture("env-delta.serve.mjs")}},
{"zeta", []string{mt.Fixture("env-zeta.mjs")}},
}, envs, logs.Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
for key, want := range map[string]string{
"gamma.given@anchor": `{"mine":"/var/lib/mesh/gamma/config.json","theirs":null,"runtime":"somebody","composed":null}`,
"delta.given@anchor": `{"mine":"/var/lib/mesh/delta/token","theirs":null}`,
"zeta.given@anchor": `{"mine":"http://127.0.0.1:3000","theirs":null,"composed":null}`,
} {
got, err := call(t, asker, key, map[string]any{})
if err != nil {
t.Fatalf("%s: %v\n%s", key, err, logs.All())
}
same(t, got, want)
}
}
// An entrypoint that is not executable is refused by name, and the others serve (ADR 0193).
func TestAnEntrypointThatIsNotExecutableIsRefused(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("plain", "anchor", false, nil))
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
logs := &mt.Logs{}
stop, err := Run(nodeTools, []Served{
{"alpha", []string{mt.Fixture("many-alpha.serve.mjs")}},
{"plain", []string{mt.Fixture("many-alpha.mjs")}},
}, nil, logs.Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
if !logs.Has("plain's bundle", "many-alpha.mjs failed to load:", "is not executable; a bundle the runtime serves is started, never imported") {
t.Errorf("the non-executable entrypoint was not refused by name:\n%s", logs.All())
}
got, err := call(t, asker, "alpha.one@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"alpha":1}`)
}
// A launched bundle is told the module it serves, so its seat's verbs stay the seat's (ADR 0193).
func TestALaunchedBundleRegisteringItsSeatFirstServesTheSeat(t *testing.T) {
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("theta", "anchor", false, map[string][]string{"node-shelf": {"list"}}))
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
stop, err := Run(nodeTools, []Served{{"theta", []string{mt.Fixture("served-seat-first.mjs")}}},
map[string]map[string]string{"theta": {"THETA_WORD": "given"}}, (&mt.Logs{}).Logf)
if err != nil {
t.Fatal(err)
}
defer stop()
got, err := call(t, asker, "theta.own@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"theta":"given"}`)
got, err = call(t, asker, "seat:node-shelf.list@anchor", map[string]any{})
if err != nil {
t.Fatal(err)
}
same(t, got, `{"shelf":["x"]}`)
}
func TestToolModulesNamesOtherModulesOnly(t *testing.T) {
got, err := ServedModulesFrom(" alpha=/a/tools/index.serve.mjs, beta=/b/one, beta=/b/two ", "node-tools")
if err != nil || len(got) != 2 || got[1].Module != "beta" || len(got[1].Entrypoints) != 2 {
t.Fatalf("%v %v", got, err)
}
for _, bad := range []string{"/mine/index.js", "node-tools=/own.js", "=/x"} {
if _, err := ServedModulesFrom(bad, "node-tools"); err == nil {
t.Errorf("%q was accepted", bad)
}
}
}
// A line about an event a bundle did not take says who failed: the handler in its own words, or the
// runtime not reaching it (hq issue 276).
func TestAnUntakenEventSaysWhoFailed(t *testing.T) {
handler := whyNotTaken(&launch.Refused{Method: "mesh/event", Message: "Unexpected end of JSON input"})
if !strings.Contains(handler, `its handler answered an error, in its own words: "Unexpected end of JSON input"`) {
t.Errorf("a handler's error is not said as the handler's: %s", handler)
}
if got := whyNotTaken(errors.New("plex's bundle did not answer mesh/event in 120s")); got != "plex's bundle did not answer mesh/event in 120s" {
t.Errorf("the runtime's own failure is not said as it is: %s", got)
}
if got := eventRef(bus.Envelope{Headers: map[string]string{"x-event-id": "e1"}}); got != " (event e1)" {
t.Errorf("the event is not named by its id: %q", got)
}
}
@@ -0,0 +1,149 @@
package runtime
import (
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// A bundle's seat traffic through moduleBus.Seat (novox/hq ADR 0259 §3), against a real bus: publish under
// its own name with the id prefixed and a duplicate said; the record read without naming the bucket; a
// worker and a proof subject bound once however often a restarted child asks; and nothing beyond what the
// membership lists.
func TestABundlesSeatTrafficGoesThroughItsModulesBus(t *testing.T) {
mesh := mt.New(t)
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := nc.JetStream()
if _, err := js.AddStream(&nats.StreamConfig{Name: "SEAT_OPERATOR_CHANNEL", Retention: nats.WorkQueuePolicy,
Subjects: []string{"mesh.seat.operator-channel.accept.>"}}); err != nil {
t.Fatal(err)
}
if _, err := js.AddConsumer("SEAT_OPERATOR_CHANNEL", &nats.ConsumerConfig{Durable: "SEAT_OPERATOR_CHANNEL_worker",
AckPolicy: nats.AckExplicitPolicy, FilterSubject: "mesh.seat.operator-channel.accept.>"}); err != nil {
t.Fatal(err)
}
kv, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: "messenger_asks"})
if err != nil {
t.Fatal(err)
}
_, _ = kv.Put("mesh-delivery.d-1", []byte(`{"state":"open"}`))
asker := mt.MembershipOf("mesh-delivery", "anchor", false, nil)
asker.SeatTraffic = &bus.SeatTraffic{Publish: []string{"mesh.seat.operator-channel.accept.ask.mesh-delivery"},
Records: []string{"$JS.API.DIRECT.GET.KV_messenger_asks.$KV.messenger_asks.mesh-delivery.>"}}
router := mt.MembershipOf("messenger", "anchor", false, nil)
router.SeatTraffic = &bus.SeatTraffic{Answers: []string{"mesh.seat.intake.proof.code.*"},
Workers: []bus.SeatWorker{{Stream: "SEAT_OPERATOR_CHANNEL", Consumer: "SEAT_OPERATOR_CHANNEL_worker",
Filter: "mesh.seat.operator-channel.accept.>"}}}
mesh.Issue(t, asker)
mesh.Issue(t, router)
conn := connect(t, "node-tools", "anchor")
conn.Follow("mesh-delivery")
conn.Follow("messenger")
mt.Until(t, func() error {
if conn.Membership("mesh-delivery") == nil || conn.Membership("messenger") == nil {
return errors.New("not issued yet")
}
return nil
})
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
asking := all.forModule("mesh-delivery").(*moduleBus)
routing := all.forModule("messenger").(*moduleBus)
publish := json.RawMessage(`{"subject":"mesh.seat.operator-channel.accept.ask.mesh-delivery","body":{"id":"d-1"},"id":"d-1"}`)
first, err := asking.Seat("publish", publish, nil)
if err != nil {
t.Fatal(err)
}
again, _ := asking.Seat("publish", publish, nil)
if !strings.Contains(string(first), `"duplicate":false`) || !strings.Contains(string(again), `"duplicate":true`) {
t.Errorf("the same id twice: %s, then %s", first, again)
}
if _, err := asking.Seat("publish", json.RawMessage(`{"subject":"mesh.seat.operator-channel.accept.ask.mesh-controller","body":{}}`), nil); err == nil {
t.Error("a module submitted under another's name")
}
got, err := asking.Seat("record", json.RawMessage(`{"key":"d-1"}`), nil)
if err != nil || !strings.Contains(string(got), `"state":"open"`) {
t.Errorf("the record read without its bucket: %s %v", got, err)
}
var mu sync.Mutex
var taken []string
handOn := func(method string, params any) (json.RawMessage, error) {
mu.Lock()
defer mu.Unlock()
raw, _ := json.Marshal(params)
taken = append(taken, method+" "+string(raw))
return json.RawMessage(`{}`), nil
}
for i := 0; i < 2; i++ { // a child that started again asks again
if _, err := routing.Seat("take", json.RawMessage(`{"worker":"SEAT_OPERATOR_CHANNEL_worker"}`), handOn); err != nil {
t.Fatal(err)
}
}
deadline := time.Now().Add(10 * time.Second)
for {
mu.Lock()
n := len(taken)
mu.Unlock()
if n >= 1 || time.Now().After(deadline) {
break
}
time.Sleep(50 * time.Millisecond)
}
time.Sleep(300 * time.Millisecond)
mu.Lock()
if len(taken) != 1 || !strings.Contains(taken[0], "mesh/work") || !strings.Contains(taken[0], `"x-event-id":"mesh-delivery.d-1"`) {
t.Errorf("taken %v", taken)
}
mu.Unlock()
if _, err := asking.Seat("take", json.RawMessage(`{"worker":"SEAT_OPERATOR_CHANNEL_worker"}`), handOn); err == nil {
t.Error("an asker took the router's work")
}
if _, err := asking.Seat("answer", json.RawMessage(`{"subject":"mesh.seat.intake.proof.code.*"}`), handOn); err == nil {
t.Error("an asker answered proofs")
}
if _, err := asking.Seat("anything", json.RawMessage(`{}`), nil); err == nil {
t.Error("an unknown verb was answered")
}
}
// A lost compare-and-set is answered with its code, so the SDK knows it without reading words.
func TestALostCompareAndSetCarriesItsCode(t *testing.T) {
mesh := mt.New(t)
mesh.Bucket(t, "messenger_asks")
m := mt.MembershipOf("messenger", "anchor", false, nil)
m.State = []bus.StateIssued{{Name: "asks", Bucket: "messenger_asks", Writes: true}}
mesh.Issue(t, m)
conn := connect(t, "node-tools", "anchor")
conn.Follow("messenger")
mt.Until(t, func() error {
if conn.Membership("messenger") == nil {
return errors.New("not issued yet")
}
return nil
})
b := (&consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}).forModule("messenger")
if _, err := b.State("create", json.RawMessage(`{"state":"asks","key":"a","value":{}}`)); err != nil {
t.Fatal(err)
}
_, err := b.State("create", json.RawMessage(`{"state":"asks","key":"a","value":{}}`))
var coded interface{ ErrorCode() int }
if !errors.As(err, &coded) || coded.ErrorCode() != -32010 || !errors.Is(err, bus.ErrStateChanged) {
t.Errorf("a second create: %v", err)
}
if _, err := b.State("update", json.RawMessage(`{"state":"asks","key":"a","value":{},"revision":0}`)); err == nil {
t.Error("an update of revision zero was taken")
}
}
@@ -0,0 +1,203 @@
package runtime
import (
"encoding/json"
"errors"
"fmt"
"sync"
"testing"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// counterBus is a module, issues, that keeps a state numbers in a bucket of a real bus, reached as a bundle
// reaches it: mesh/state.<verb> with its JSON, through the module's bus.
func counterBus(t *testing.T) launch.Bus {
t.Helper()
mesh := mt.New(t)
mesh.Bucket(t, "issues_numbers")
m := mt.MembershipOf("issues", "anchor", false, nil)
m.State = []bus.StateIssued{{Name: "numbers", Bucket: "issues_numbers", Writes: true}}
mesh.Issue(t, m)
conn := connect(t, "node-tools", "anchor")
conn.Follow("issues")
mt.Until(t, func() error {
if conn.Membership("issues") == nil {
return errors.New("not issued yet")
}
return nil
})
return (&consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}).forModule("issues")
}
func ask(b launch.Bus, verb string, params map[string]any) (json.RawMessage, error) {
raw, _ := json.Marshal(params)
return b.State(verb, raw)
}
// increment is what a bundle's counter does over the wire: get, then create (no value) or update at the
// revision read; a lost write — told by its code alone — is read again and tried again.
func increment(b launch.Bus, key string, attempts int) (n uint64, lost int, err error) {
for i := 0; i < attempts; i++ {
raw, err := ask(b, "get", map[string]any{"state": "numbers", "key": key})
if err != nil {
return 0, lost, err
}
var e *bus.StateEntry
if err := json.Unmarshal(raw, &e); err != nil {
return 0, lost, err
}
if e == nil {
n = 1
_, err = ask(b, "create", map[string]any{"state": "numbers", "key": key, "value": n})
} else {
if err := json.Unmarshal(e.Value, &n); err != nil {
return 0, lost, err
}
n++
_, err = ask(b, "update", map[string]any{"state": "numbers", "key": key, "value": n, "revision": e.Revision})
}
var coded interface{ ErrorCode() int }
switch {
case err == nil:
return n, lost, nil
case errors.As(err, &coded) && coded.ErrorCode() == launch.CodeChanged:
lost++
default:
return 0, lost, err
}
}
return 0, lost, fmt.Errorf("%s lost %d times", key, attempts)
}
// Two writers incrementing one counter a thousand times each, against a real bus, end at two thousand:
// every number handed out once and none lost, because the runtime writes only at the revision read
// (novox/hq research 042: an issue's number is handed out once).
func TestTwoWritersIncrementingOneCounterOnTheBusLoseNoUpdate(t *testing.T) {
b := counterBus(t)
const each = 1000
var wg sync.WaitGroup
numbers := make([][]uint64, 2)
errs := make([]error, 2)
lost := make([]int, 2)
for w := range numbers {
wg.Add(1)
go func(w int) {
defer wg.Done()
for i := 0; i < each; i++ {
n, l, err := increment(b, "issue", 1000)
lost[w] += l
if err != nil {
errs[w] = err
return
}
numbers[w] = append(numbers[w], n)
}
}(w)
}
wg.Wait()
if err := errors.Join(errs...); err != nil {
t.Fatal(err)
}
seen := map[uint64]bool{}
for w, ns := range numbers {
for i, n := range ns {
if seen[n] {
t.Errorf("the number %d was handed out twice", n)
}
seen[n] = true
if i > 0 && n <= ns[i-1] {
t.Errorf("writer %d was handed %d after %d", w, n, ns[i-1])
}
}
}
raw, err := ask(b, "get", map[string]any{"state": "numbers", "key": "issue"})
if err != nil {
t.Fatal(err)
}
var e bus.StateEntry
_ = json.Unmarshal(raw, &e)
if string(e.Value) != "2000" || len(seen) != 2*each {
t.Fatalf("the counter ended at %s with %d numbers handed out, not 2000: an increment was lost", e.Value, len(seen))
}
if lost[0]+lost[1] == 0 {
t.Fatal("the two writers never collided: the test proved nothing about contention")
}
t.Logf("2000 increments, %d writes lost to the other writer and tried again", lost[0]+lost[1])
}
// A write at a revision that is no longer the key's is refused with the lost-write code and writes
// nothing; create is refused for a key with a value and taken for one deleted; an update names the
// revision it read, never zero.
func TestAStaleWriteIsRefusedAndWritesNothing(t *testing.T) {
b := counterBus(t)
state := func(verb string, extra map[string]any) (uint64, error) {
p := map[string]any{"state": "numbers", "key": "issue"}
for k, v := range extra {
p[k] = v
}
raw, err := ask(b, verb, p)
if err != nil {
return 0, err
}
var r struct {
Revision uint64 `json:"revision"`
}
_ = json.Unmarshal(raw, &r)
return r.Revision, nil
}
isLost := func(err error) bool {
var coded interface{ ErrorCode() int }
return errors.As(err, &coded) && coded.ErrorCode() == launch.CodeChanged && errors.Is(err, bus.ErrStateChanged)
}
value := func() string {
raw, err := ask(b, "get", map[string]any{"state": "numbers", "key": "issue"})
if err != nil {
t.Fatal(err)
}
var e *bus.StateEntry
_ = json.Unmarshal(raw, &e)
if e == nil {
return "none"
}
return string(e.Value)
}
if _, err := state("update", map[string]any{"value": 1, "revision": 0}); err == nil || isLost(err) {
t.Fatalf("an update at revision zero: %v", err)
}
if value() != "none" {
t.Fatal("an update at revision zero wrote")
}
first, err := state("create", map[string]any{"value": 1})
if err != nil || first == 0 {
t.Fatalf("create of a key with no value: %d %v", first, err)
}
if _, err := state("create", map[string]any{"value": 9}); !isLost(err) {
t.Fatalf("a second create: %v", err)
}
second, err := state("update", map[string]any{"value": 2, "revision": first})
if err != nil || second <= first {
t.Fatalf("an update at the revision read: %d %v", second, err)
}
if _, err := state("update", map[string]any{"value": 9, "revision": first}); !isLost(err) {
t.Fatalf("an update at a stale revision: %v", err)
}
if _, err := state("update", map[string]any{"value": 9, "revision": second + 100}); !isLost(err) {
t.Fatalf("an update at a revision not yet written: %v", err)
}
if v := value(); v != "2" {
t.Fatalf("refused writes changed the value to %s", v)
}
if _, err := ask(b, "delete", map[string]any{"state": "numbers", "key": "issue"}); err != nil {
t.Fatal(err)
}
if _, err := state("create", map[string]any{"value": 1}); err != nil {
t.Fatalf("create of a deleted key: %v", err)
}
if v := value(); v != "1" {
t.Fatalf("after delete and create: %s", v)
}
}
@@ -0,0 +1,152 @@
package runtime
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// historyBuses are three modules on one runtime, each reached as a bundle reaches it (novox/hq ADR
// 0307): issues keeps a state `issues` in a bucket keeping 8 values a key, triage reads it as
// `issues.issues`, and other was issued no state.
func historyBuses(t *testing.T) (owner, reader, other launch.Bus) {
t.Helper()
mesh := mt.New(t)
mesh.BucketKeeping(t, "issues_issues", 8)
m := mt.MembershipOf("issues", "anchor", false, nil)
m.State = []bus.StateIssued{{Name: "issues", Bucket: "issues_issues", Writes: true}}
mesh.Issue(t, m)
r := mt.MembershipOf("triage", "anchor", false, nil)
r.State = []bus.StateIssued{{Name: "issues.issues", Bucket: "issues_issues"}}
mesh.Issue(t, r)
mesh.Issue(t, mt.MembershipOf("other", "anchor", false, nil))
conn := connect(t, "node-tools", "anchor")
for _, module := range []string{"issues", "triage", "other"} {
conn.Follow(module)
}
mt.Until(t, func() error {
for _, module := range []string{"issues", "triage", "other"} {
if conn.Membership(module) == nil {
return errors.New(module + " not issued yet")
}
}
return nil
})
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
return all.forModule("issues"), all.forModule("triage"), all.forModule("other")
}
// kept is one value of a history answer, by the field names on the wire.
type kept struct {
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Revision uint64 `json:"revision"`
Op string `json:"op"`
At string `json:"at"`
}
func historyOf(t *testing.T, b launch.Bus, state, key string) []kept {
t.Helper()
raw, err := ask(b, "history", map[string]any{"state": state, "key": key})
if err != nil {
t.Fatalf("history of %s %s: %v", state, key, err)
}
var h []kept
if err := json.Unmarshal(raw, &h); err != nil || h == nil {
t.Fatalf("history answered %s, not an array", raw)
}
return h
}
// A key's history is every value the bucket keeps, oldest first, each with its revision, op and time
// in RFC 3339 — a delete with no value; an unknown key's is []. The owner and a reader are served it,
// and reading it writes nothing.
func TestAKeysHistoryIsEveryValueKeptOldestFirst(t *testing.T) {
owner, reader, _ := historyBuses(t)
before := time.Now().Add(-time.Second)
var revisions []uint64
for _, v := range []string{`{"state":"open"}`, `{"state":"fixing"}`, `{"state":"resolved"}`} {
raw, err := ask(owner, "put", map[string]any{"state": "issues", "key": "502", "value": json.RawMessage(v)})
if err != nil {
t.Fatal(err)
}
var r struct {
Revision uint64 `json:"revision"`
}
_ = json.Unmarshal(raw, &r)
revisions = append(revisions, r.Revision)
}
if _, err := ask(owner, "put", map[string]any{"state": "issues", "key": "503", "value": 1}); err != nil {
t.Fatal(err)
}
if _, err := ask(owner, "delete", map[string]any{"state": "issues", "key": "502"}); err != nil {
t.Fatal(err)
}
for who, b := range map[string]launch.Bus{"the owner": owner, "a reader": reader} {
name := "issues"
if b == reader {
name = "issues.issues"
}
h := historyOf(t, b, name, "502")
if len(h) != 4 {
t.Fatalf("%s read %d values of 502, not 4: %+v", who, len(h), h)
}
for i, v := range []string{`{"state":"open"}`, `{"state":"fixing"}`, `{"state":"resolved"}`} {
if h[i].Op != "put" || string(h[i].Value) != v || h[i].Revision != revisions[i] || h[i].Key != "502" {
t.Errorf("%s: value %d is %+v", who, i, h[i])
}
}
if h[3].Op != "delete" || len(h[3].Value) != 0 && string(h[3].Value) != "null" || h[3].Revision <= revisions[2] {
t.Errorf("%s: the delete reads %+v", who, h[3])
}
for i, k := range h {
at, err := time.Parse(time.RFC3339, k.At)
if err != nil || at.Before(before) || at.After(time.Now().Add(time.Second)) {
t.Errorf("%s: value %d's time %q is not RFC 3339 and now: %v", who, i, k.At, err)
}
if i > 0 && k.Revision <= h[i-1].Revision {
t.Errorf("%s: not oldest first: %+v", who, h)
}
}
if h := historyOf(t, b, name, "999"); len(h) != 0 {
t.Errorf("%s: an unknown key's history is %+v, not []", who, h)
}
}
// The history is read, never written: the key is still deleted and keeps the same four values.
if raw, err := ask(owner, "get", map[string]any{"state": "issues", "key": "502"}); err != nil || string(raw) != "null" {
t.Fatalf("after history, 502 reads %s %v", raw, err)
}
if h := historyOf(t, owner, "issues", "502"); len(h) != 4 {
t.Fatalf("reading the history changed it: %+v", h)
}
// A key the bucket holds nothing for answers [] on the wire, not null.
if raw, _ := ask(owner, "history", map[string]any{"state": "issues", "key": "999"}); string(raw) != "[]" {
t.Fatalf("an unknown key's history is %s on the wire", raw)
}
}
// A module the bucket was not issued to is refused its history with the reason, as it is a get.
func TestAKeysHistoryIsRefusedToAModuleNotIssuedTheState(t *testing.T) {
owner, _, other := historyBuses(t)
if _, err := ask(owner, "put", map[string]any{"state": "issues", "key": "1", "value": 1}); err != nil {
t.Fatal(err)
}
for _, name := range []string{"issues", "issues.issues"} {
_, err := ask(other, "history", map[string]any{"state": name, "key": "1"})
if err == nil || !strings.Contains(err.Error(), "keeps and reads no state called") ||
!strings.Contains(err.Error(), "issued: none") {
t.Errorf("history of %s to a module not issued it: %v", name, err)
}
}
if _, err := ask(owner, "history", map[string]any{"state": "issues", "key": "a b"}); err == nil ||
!strings.Contains(err.Error(), "not a key") {
t.Errorf("history of a key the bus cannot hold: %v", err)
}
}
+178
View File
@@ -0,0 +1,178 @@
package runtime
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq ADR 0201: a module keeps its current state in buckets it declares, and its code reaches
// them through the runtime. The owner's instances write and read; a reader only reads; a watch hands
// the current values — none that is deleted — and then every change; the runtime refuses, with the
// reason, what the module was not issued, a write to state it only reads, and a value naming a
// credential.
func TestABundleKeepsAndWatchesItsStateThroughTheRuntime(t *testing.T) {
mesh := mt.New(t)
kv := mesh.Bucket(t, "keeper_servers")
// What was there before anything started: one value, and one key since deleted.
if _, err := kv.Put("all.one", []byte(`{"url":"http://one"}`)); err != nil {
t.Fatal(err)
}
if _, err := kv.Put("all.gone", []byte(`{}`)); err != nil {
t.Fatal(err)
}
if err := kv.Delete("all.gone"); err != nil {
t.Fatal(err)
}
keeper := mt.MembershipOf("keeper", "anchor", false, nil)
keeper.State = []bus.StateIssued{{Name: "servers", Bucket: "keeper_servers", Writes: true}}
peeker := mt.MembershipOf("peeker", "anchor", false, nil)
peeker.State = []bus.StateIssued{{Name: "keeper.servers", Bucket: "keeper_servers"}}
mesh.Issue(t, keeper)
mesh.Issue(t, peeker)
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
dir := t.TempDir()
keeperLog, peekerLog := filepath.Join(dir, "keeper.log"), filepath.Join(dir, "peeker.log")
said := &lines{}
stop, err := Run(nodeTools, []Served{
{Module: "keeper", Entrypoints: []string{mt.Fixture("state-keeper.mjs")}},
{Module: "peeker", Entrypoints: []string{mt.Fixture("state-keeper.mjs")}},
}, map[string]map[string]string{
"keeper": {"STATE_NAME": "servers", "STATE_LOG": keeperLog},
"peeker": {"STATE_NAME": "keeper.servers", "STATE_LOG": peekerLog},
}, said.logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
// Both read the whole current state at start: the value, not the deleted key, then the end of it.
for _, log := range []string{keeperLog, peekerLog} {
waitFor(t, log, "current")
lines := read(t, log)
if lines[0] != `was put all.one {"url":"http://one"}` || lines[1] != "current" || strings.Contains(strings.Join(lines, "\n"), "all.gone") {
t.Fatalf("the current state was not handed over as it is:\n%s\nruntime:\n%s", strings.Join(lines, "\n"), said.all())
}
}
// The owner writes; every watch sees the change.
got, err := call(t, asker, "keeper.put@anchor", map[string]any{"state": "servers", "key": "anchor.two", "value": map[string]any{"url": "http://two"}})
if err != nil {
t.Fatal(err)
}
if !strings.Contains(got, `"revision"`) {
t.Fatalf("a put answered %s", got)
}
waitFor(t, keeperLog, `now put anchor.two {"url":"http://two"}`)
waitFor(t, peekerLog, `now put anchor.two {"url":"http://two"}`)
// Get and keys, by the owner and by the reader.
got, err = call(t, asker, "peeker.get@anchor", map[string]any{"state": "keeper.servers", "key": "anchor.two"})
if err != nil {
t.Fatal(err)
}
if !strings.Contains(got, `"value":{"url":"http://two"}`) {
t.Fatalf("the reader's get answered %s", got)
}
got, err = call(t, asker, "peeker.get@anchor", map[string]any{"state": "keeper.servers", "key": "nothing.here"})
if err != nil || got != "null" {
t.Fatalf("a key with no value answered %s, %v", got, err)
}
got, err = call(t, asker, "keeper.keys@anchor", map[string]any{"state": "servers"})
if err != nil {
t.Fatal(err)
}
same(t, got, `["all.one","anchor.two"]`)
// Refused, with the reason, before anything is sent.
for _, c := range []struct {
key, why string
args map[string]any
}{
{"peeker.put@anchor", "peeker reads keeper.servers and does not keep it",
map[string]any{"state": "keeper.servers", "key": "x", "value": 1}},
{"keeper.get@anchor", `keeper keeps and reads no state called "other"`,
map[string]any{"state": "other", "key": "x"}},
{"keeper.put@anchor", `carries a field "accessToken", which names a credential`,
map[string]any{"state": "servers", "key": "x", "value": map[string]any{"headers": map[string]any{"accessToken": "s"}}}},
{"keeper.put@anchor", "is not a key the bus can hold",
map[string]any{"state": "servers", "key": "a..b", "value": 1}},
} {
if _, err := call(t, asker, c.key, c.args); err == nil || !strings.Contains(err.Error(), c.why) {
t.Errorf("%s %v: want refused for %q, got %v", c.key, c.args, c.why, err)
}
}
// A delete reaches every watch.
if _, err := call(t, asker, "keeper.del@anchor", map[string]any{"state": "servers", "key": "all.one"}); err != nil {
t.Fatal(err)
}
waitFor(t, peekerLog, "now delete all.one")
waitFor(t, keeperLog, "now delete all.one")
}
func read(t *testing.T, path string) []string {
t.Helper()
b, _ := os.ReadFile(path)
return strings.Split(strings.TrimSpace(string(b)), "\n")
}
func waitFor(t *testing.T, path, line string) {
t.Helper()
mt.Until(t, func() error {
for _, l := range read(t, path) {
if l == line {
return nil
}
}
return errorf("%s has no %q yet: %s", filepath.Base(path), line, strings.Join(read(t, path), " | "))
})
}
// The TypeScript SDK against the runtime (ADR 0201): two watches of one state, each handed only its
// own changes, the current values handled before the module's import goes on.
func TestTheSDKsStateReachesTheRuntime(t *testing.T) {
mesh := mt.New(t)
kv := mesh.Bucket(t, "sdkkeeper_servers")
if _, err := kv.Put("all.one", []byte(`{"url":"http://one"}`)); err != nil {
t.Fatal(err)
}
m := mt.MembershipOf("sdkkeeper", "anchor", false, nil)
m.State = []bus.StateIssued{{Name: "servers", Bucket: "sdkkeeper_servers", Writes: true}}
mesh.Issue(t, m)
nodeTools := connect(t, "node-tools", "anchor")
asker := connect(t, "console", "workstation")
log := filepath.Join(t.TempDir(), "sdk.log")
said := &lines{}
stop, err := Run(nodeTools, []Served{{Module: "sdkkeeper", Entrypoints: []string{mt.Fixture("state-sdk.serve.mjs")}}},
map[string]map[string]string{"sdkkeeper": {"STATE_LOG": log}}, said.logf)
if err != nil {
t.Fatalf("%v\n%s", err, said.all())
}
t.Cleanup(stop)
waitFor(t, log, "current")
if got := strings.Join(read(t, log), " | "); got != "was put all.one | current" {
t.Fatalf("the import went on with %q", got)
}
if _, err := call(t, asker, "sdkkeeper.register@anchor", map[string]any{"key": "anchor.two", "url": "http://two"}); err != nil {
t.Fatal(err)
}
waitFor(t, log, "now put anchor.two")
waitFor(t, log, "narrow put anchor.two")
if _, err := call(t, asker, "sdkkeeper.register@anchor", map[string]any{"key": "all.three", "url": "http://three"}); err != nil {
t.Fatal(err)
}
waitFor(t, log, "now put all.three")
for _, l := range read(t, log) {
if l == "narrow put all.three" {
t.Fatalf("a narrowed watch was handed a key outside it: %s", strings.Join(read(t, log), " | "))
}
}
}
+18
View File
@@ -0,0 +1,18 @@
// Package wire is JSON as the TypeScript runtime writes it: no HTML escaping of <, > and &.
package wire
import (
"bytes"
"encoding/json"
)
// Marshal encodes v the way JSON.stringify does, without a trailing newline.
func Marshal(v any) ([]byte, error) {
var b bytes.Buffer
enc := json.NewEncoder(&b)
enc.SetEscapeHTML(false)
if err := enc.Encode(v); err != nil {
return nil, err
}
return bytes.TrimRight(b.Bytes(), "\n"), nil
}
+56
View File
@@ -0,0 +1,56 @@
{
"module": "node-tools",
"version": "1",
"slug": "node-tools",
"invokes": [
"*"
],
"provides": [
{
"name": "mcp-endpoint",
"scope": "node"
}
],
"serves": {
"mcp-endpoint": {
"port": 4270
}
},
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"listens": [
{
"name": "mcp",
"port": 4270,
"protocol": "tcp",
"from": "machine",
"why": "the mesh's tools for whoever is on this machine, over MCP on loopback; the machine's login is the authority (novox/hq ADR 0152, 0175)"
}
],
"resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0755",
"place": "mesh"
},
{
"id": "interpreter",
"type": "package",
"package": "nodejs"
}
],
"build": {
"artifacts": [
{
"name": "runtime",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/node-tools",
"binary": "node-tools"
}
]
}
}
+76
View File
@@ -0,0 +1,76 @@
{
"name": "@novox/mesh-tools",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@novox/mesh-tools",
"version": "0.1.0",
"dependencies": {
"@novox/mesh-sdk": "^0.1.0",
"nats": "^2.29.0"
},
"bin": {
"mesh": "dist/mesh.js",
"mesh-tools": "dist/main.js"
},
"devDependencies": {
"@types/node": "^22.20.1",
"typescript": "^5.9.3"
}
},
"node_modules/@novox/mesh-sdk": {
"version": "0.1.1"
},
"node_modules/@types/node": {
"version": "22.20.1",
"dev": true,
"license": "MIT",
"dependencies": {
"undici-types": "~6.21.0"
}
},
"node_modules/nats": {
"version": "2.29.3",
"license": "Apache-2.0",
"dependencies": {
"nkeys.js": "1.1.0"
},
"engines": {
"node": ">= 14.0.0"
}
},
"node_modules/nkeys.js": {
"version": "1.1.0",
"license": "Apache-2.0",
"dependencies": {
"tweetnacl": "1.0.3"
},
"engines": {
"node": ">=10.0.0"
}
},
"node_modules/tweetnacl": {
"version": "1.0.3",
"license": "Unlicense"
},
"node_modules/typescript": {
"version": "5.9.3",
"dev": true,
"license": "Apache-2.0",
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
},
"engines": {
"node": ">=14.17"
}
},
"node_modules/undici-types": {
"version": "6.21.0",
"dev": true,
"license": "MIT"
}
}
}
+27
View File
@@ -0,0 +1,27 @@
{
"name": "@novox/mesh-tools",
"version": "0.1.0",
"description": "The Novox Mesh tool runtime \u2014 binds the mesh broker and serves the assigned modules' tools.",
"type": "module",
"bin": {
"mesh-tools": "./dist/main.js",
"mesh": "./dist/mesh.js"
},
"scripts": {
"build": "tsc",
"pretest": "tsc",
"test": "node --test --test-concurrency=1 --experimental-strip-types 'test/*.test.ts'"
},
"dependencies": {
"@novox/mesh-sdk": "^0.1.6",
"nats": "^2.29.0"
},
"devDependencies": {
"@types/node": "^22.20.1",
"typescript": "^5.9.3",
"esbuild": "^0.25.0",
"mongodb": "^6.21.0",
"mssql": "^11.0.2",
"pg": "^8"
}
}
+139
View File
@@ -0,0 +1,139 @@
// What a runtime serves, it announces (novox/hq ADR 0197): the NATS services protocol's discovery —
// `$SRV.PING`, `$SRV.INFO`, `$SRV.STATS`, and the same followed by the service's name and its id —
// answered in the io.nats.micro.v1 format with what is served at the moment of the request. Serving is
// unchanged; this only says what is served. One service per runtime process, because the bus admits
// one reply per request from each responder: one endpoint per tool per subject, its metadata saying
// which module, seat, scope and machine it is. The same shape the Go runtime answers.
import { StringCodec } from "nats";
import { asSchema } from "@novox/mesh-sdk/stdio";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import type { RuntimeBroker } from "./broker-nats.js";
const sc = StringCodec();
export const VERSION = "0.1.0";
export const INFO_RESPONSE = "io.nats.micro.v1.info_response";
export const PING_RESPONSE = "io.nats.micro.v1.ping_response";
export const STATS_RESPONSE = "io.nats.micro.v1.stats_response";
/** One tool served on one subject, as announced. */
export interface Endpoint {
kind: "tool" | "seat";
module: string;
tool: string;
seat?: string;
scope?: "mesh" | "node";
node: string;
description: string;
schema: unknown;
interchangeable: boolean;
subject: string;
queue?: string;
}
/** A seat's verb as the runtime serves it, with the definition that answers it. */
export interface ServedSeatVerb {
seat: string;
verb: string;
subject: string;
holder: string;
tool: ToolDefinition;
}
export interface Service {
name: string;
id: string;
description: string;
metadata: Record<string, string>;
}
/** The endpoint's name as the protocol allows it; the metadata, not the name, identifies it. */
function nameOf(e: Endpoint): string {
const clean = (s: string) => s.replace(/[^A-Za-z0-9_-]/g, "_");
return `${clean(e.kind === "seat" ? e.seat ?? "" : e.module)}__${clean(e.tool)}`;
}
/** The info_response for these endpoints. */
export function info(s: Service, endpoints: Endpoint[]): Record<string, unknown> {
return {
name: s.name, id: s.id, version: VERSION, metadata: s.metadata, type: INFO_RESPONSE, description: s.description,
endpoints: endpoints.map((e) => {
const metadata: Record<string, string> = {
kind: e.kind, module: e.module, tool: e.tool, node: e.node, description: e.description,
schema: JSON.stringify(e.schema ?? {}), interchangeable: e.interchangeable ? "true" : "false",
};
if (e.kind === "seat") {
metadata.seat = e.seat ?? "";
metadata.scope = e.scope ?? "mesh";
}
return { name: nameOf(e), subject: e.subject, queue_group: e.queue ?? "", metadata };
}),
};
}
/** Everything served now: each module's tools on every subject issued for them, and each held
* seat's verbs on the seat's subject. */
export function endpointsOf(
broker: RuntimeBroker,
own: { module: string; tools: ToolDefinition[] }[],
seats: ServedSeatVerb[],
): Endpoint[] {
const node = broker.node ?? "";
const out: Endpoint[] = [];
for (const { module, tools } of own) {
for (const t of tools) {
const served = broker.servedOn ? broker.servedOn(module, t.name) : [];
const interchangeable = served.some((s) => s.subject === `mesh.mod.${module}.tool.${t.name}`);
for (const s of served) {
out.push({ kind: "tool", module, tool: t.name, node, description: t.description, schema: asSchema(t.input),
interchangeable, subject: s.subject, queue: s.queue });
}
}
}
for (const v of seats) {
out.push({ kind: "seat", module: v.holder, tool: v.verb, seat: v.seat,
scope: node && v.subject.endsWith(`.${node}`) ? "node" : "mesh", node, description: v.tool.description,
schema: asSchema(v.tool.input), interchangeable: false, subject: v.subject });
}
return out;
}
/** Answer discovery for one service until stopped. */
export function announce(broker: RuntimeBroker, s: Service, current: () => Endpoint[]): () => void {
const started = new Date().toISOString();
const identity = { name: s.name, id: s.id, version: VERSION, metadata: s.metadata };
const answer = (subject: string): Uint8Array | undefined => {
const parts = subject.split(".");
if (parts[0] !== "$SRV" || parts.length < 2) return undefined;
if (parts.length >= 3 && parts[2] !== s.name) return undefined; // another service's
if (parts.length >= 4 && parts[3] !== s.id) return undefined; // another instance's
let v: unknown;
switch (parts[1]) {
case "PING":
v = { ...identity, type: PING_RESPONSE };
break;
case "INFO":
v = info(s, current());
break;
case "STATS":
v = { ...identity, type: STATS_RESPONSE, started, endpoints: current().map((e) => ({
name: nameOf(e), subject: e.subject, queue_group: e.queue ?? "", num_requests: 0, num_errors: 0,
last_error: "", processing_time: 0, average_processing_time: 0 })) };
break;
default:
return undefined;
}
return sc.encode(JSON.stringify(v));
};
const stops: (() => void)[] = [];
// Exactly the questions asked of every service and of this one by name and instance — what the
// grants allow (novox/hq ADR 0197). A wildcard is refused by the bus, and a refused subscription
// ends a runtime: on 2026-10-03 it crash-looped every container that announced itself.
for (const verb of ["PING", "INFO", "STATS"]) {
for (const subject of [`$SRV.${verb}`, `$SRV.${verb}.${s.name}`, `$SRV.${verb}.${s.name}.${s.id}`]) {
stops.push(broker.raw!(subject, (subj) => answer(subj)));
}
}
return () => stops.forEach((stop) => stop());
}
+680
View File
@@ -0,0 +1,680 @@
// The tool runtime's broker client, on NATS.
//
// **The sdk's contract does not change** (novox/hq ADR 0106, ADR 0039): a module is written
// against `request`, `handle`, `publish`, `subscribe`, `close`, and the runtime implements them.
// That is why a module built before any of this runs on the new runtime without a rebuild, and
// why the sdk's own diff for the whole bus change is three comments.
//
// Underneath, everything is a subject and durability is JetStream (novox/hq design 25,
// design 29).
//
// mesh.mod.<module>.event.<type> an event this module emits
// mesh.mod.<module>.tool.<tool> a tool this module serves
// mesh.seat.<seat>.accept.<verb> work submitted to a role
//
// The module never writes one of those: it names its events and tools locally and the mesh
// derives the subject (design 29 §1), so reorganising the subject space leaves every module
// correct.
import { AsyncLocalStorage } from "node:async_hooks";
import { createHash } from "node:crypto";
import net from "node:net";
import tls from "node:tls";
import { connect as natsConnect, headers as natsHeaders, StringCodec, type JsMsg, type Subscription, type TlsOptions } from "nats";
import type { Broker, Envelope, EventHeaders } from "@novox/mesh-sdk/messaging";
const sc = StringCodec();
/** Requests wait this long for an answer before failing. Unchanged from what modules already
* expect, so a module's timeout handling is not something the bus quietly redefines. */
const REQUEST_TIMEOUT_MS = 30_000;
export class PinMismatchError extends Error {}
/** A broker credential as the mesh delivers it (novox/hq ADR 0120): the bus's address, the
* fingerprint of the certificate it must present, and the node and module the account is scoped
* to — the runtime derives its subjects from those rather than being told them. */
export interface Credential {
url: string;
fingerprint?: string;
node?: string;
module?: string;
user?: string;
password?: string;
/** The seats this module claims, with the verbs each promises (novox/hq ADR 0159). The runtime
* serves each claimed seat's verbs with its tools of the same name; the bus admits only the
* holder's subscription, so claiming and not holding costs a refused subscription and nothing else. */
claims?: { seat: string; scope?: string; serves?: string[] }[];
}
/**
* What the mesh issued this assignment (novox/hq ADR 0160): where its tools are served, in which
* queue, the verbs of the seats it holds, where its events land, what it may reach. Read from the
* ASSIGNMENTS stream at `mesh.assignment.<node>.<module>` — the one subject a runtime derives for
* itself — and followed live. Absent for a mesh older than the membership, and then the runtime
* serves the shape it always derived, and says so.
*/
export interface Membership {
node: string;
module: string;
/** Addresses a tool is answered on; `{tool}` stands for the tool's name. */
serves: { subject: string; queue?: string }[];
seats?: { seat: string; verb: string; subject: string }[];
emits: string;
reaches?: Record<string, string[]>;
tools: string;
}
/** What a tool call answers: the module's own result, and which machine answered it
* (novox/hq ADR 0159) — a module on several machines is otherwise an answer from nowhere. */
export interface Answered<Res> {
result: Res;
node?: string;
}
/** The bus as the runtime sees it: the sdk's contract, and the few things only the runtime needs —
* an answer that says which machine gave it, serving a subject that is not a module's own tool
* (a seat's verb), and following the memberships of the modules it serves beside its own. */
export interface RuntimeBroker extends Broker {
/** Call a tool by key, or — when `on` names a subject the mesh listed for it (ADR 0160) — there. */
ask<Req, Res>(key: string, body: Req, on?: string): Promise<Answered<Res>>;
handleSubject<Req, Res>(subject: string, handler: (body: Req) => Promise<Res>): Promise<() => void>;
/**
* Serve another module's tools from this connection: read its membership on this machine and
* follow it, so `handle("<module>.<tool>")` is served where the mesh issued that module (novox/hq
* ADR 0175: one runtime per node, every assigned module's tools). The account must be allowed
* to read that membership and to subscribe its subjects — the node's is; a module's own is not,
* and is refused by the bus, not here.
*/
follow(module: string): Promise<void>;
/** What the mesh issued this assignment — this module's when none is named — or undefined when
* nothing has been issued yet. */
membership(module?: string): Membership | undefined;
/** Called when the mesh issues a new membership to any module this connection follows; the
* runtime re-serves on it. The membership says which module it is for. */
onMembership(handler: (m: Membership) => void): void;
/** The modules whose memberships this connection follows: its own and every one `follow` added. */
serving(): string[];
/** The module this connection is: what its credential named, and what a bare key serves as. */
readonly module: string;
/** Where a served module's tool is answered right now (ADR 0197: what it serves, it announces). */
servedOn?(module: string, tool: string): { subject: string; queue?: string }[];
/** Answer a subject in a format of its own, not a tool's reply envelope — the NATS services
* protocol's discovery (novox/hq ADR 0197). An undefined answer is no reply. */
raw?(subject: string, answer: (subject: string, data: Uint8Array) => Uint8Array | undefined): () => void;
/** The machine this connection serves on, when its credential names one. */
readonly node?: string;
}
/**
* Which module's tool is at work on this connection, when one is (novox/hq ADR 0175). One runtime
* carries many modules' tools, and an event a tool emits must land on the emitting *module's*
* subject, not the runtime's — so the runtime runs each tool inside this store, and `publish`
* reads the module from it. Outside a tool the connection's own module stands.
*/
export const atWork = new AsyncLocalStorage<{ module: string }>();
const ASSIGNMENTS_STREAM = "ASSIGNMENTS";
/** The one address a runtime derives for itself (ADR 0160). */
export function membershipSubject(node: string, module: string): string {
return `mesh.assignment.${node}.${module}`;
}
/** Whether a connection failure is worth retrying, or is a fact about this configuration that
* retrying cannot change. The runtime's supervisor asks this and does not need to know what it
* is connected to. */
export function fatalBrokerReason(err: unknown): string | null {
if (err instanceof PinMismatchError) return "the bus's certificate does not match the pin";
const e = err as { code?: string; message?: string };
const message = typeof e?.message === "string" ? e.message : String(err);
if (e?.code === "ERR_INVALID_URL" || /invalid url/i.test(message)) {
return "the bus address is not a usable URL";
}
if (/authorization violation|user authentication expired|permissions violation/i.test(message)) {
return "the bus refused this account";
}
return null;
}
/**
* Connect to the mesh bus and return a Broker.
*
* **A module's subjects come from its credential, not from its calls.** `node` and `module` name
* the account the mesh issued, and every subject this client publishes or subscribes is derived
* from them — so a module cannot name another's namespace even by mistake, and what it emits
* matches what the mesh authorised (ADR 0074's identity rule).
*/
export async function connectNats(
target: string | Credential,
opts: { module?: string } = {},
): Promise<RuntimeBroker> {
const cred: Credential = typeof target === "string" ? { url: target } : target;
const self = cred.module ?? opts.module;
if (!self) {
throw new Error(
"a broker credential with no module: the runtime derives its subjects from the account " +
"the mesh issued, and cannot guess which module it is",
);
}
const conn = await natsConnect({
servers: cred.url,
user: cred.user,
pass: cred.password,
name: `${cred.node ?? "?"}.${self}`,
tls: cred.fingerprint ? await pinnedTls(cred.url, cred.fingerprint) : undefined,
// Its own inbox, not a random one: every user's inbox is private to it (design 25 §4), and the
// grant names `_INBOX.<user>.>` — a reply space the client invented would be refused, and with
// it every pull for the next message and every answer to a tool call.
inboxPrefix: cred.user ? `_INBOX.${cred.user}` : undefined,
// Reconnect forever: the bus being restarted is an upgrade, not a reason for every module on
// the mesh to exit. `close()` stays the only thing that ends the connection.
maxReconnectAttempts: -1,
});
const js = conn.jetstream();
const subs: Subscription[] = [];
// Every registration, and the one reader that dispatches to them. A module has one durable
// consumer; the loop belongs to the connection rather than to a subscription.
const listeners: { pattern: string; handler: (env: Envelope<unknown>) => Promise<void> }[] = [];
let reading: Awaited<ReturnType<Awaited<ReturnType<typeof js.consumers.get>>["consume"]>> | undefined;
let closed = false;
const node = cred.node;
// The memberships, one per module this connection serves, each read once and followed. A
// module's own runtime follows one, its own; the node's runtime follows one per assigned module
// (novox/hq ADR 0175) — the same subject shape, the same stream, read as many times as there are
// modules, and nothing on the bus learns a new shape for it. A direct get is one request on the
// stream's API, which is the whole of what an account may ask JetStream for a subject it is
// granted; a 404 is a mesh that has not issued one, which is a fact to say and not an error to
// retry.
const issued = new Map<string, Membership | undefined>();
const issuedHandlers: ((m: Membership) => void)[] = [];
const follow = async (module: string): Promise<void> => {
if (!node || issued.has(module)) return;
issued.set(module, undefined);
const subjectOfTheirs = membershipSubject(node, module);
try {
// The subject-addressed form of a direct get — the stream, then the subject, nothing in
// the body — because that is the one address the mesh grants this account on the
// stream's API; the body form asks the stream's root, which it may not.
const got = await conn.request(`$JS.API.DIRECT.GET.${ASSIGNMENTS_STREAM}.${subjectOfTheirs}`,
new Uint8Array(0), { timeout: 5_000 });
const status = got.headers?.code ?? 0;
if (status === 0 && got.data.length > 0) {
issued.set(module, JSON.parse(sc.decode(got.data)) as Membership);
}
} catch {
// Not readable here: an older mesh, a stream not yet asserted, or no grant. Said below.
}
if (!issued.get(module)) {
console.log(`[mesh-tools] no membership issued for ${module} on ${node} yet; serving the derived shape until one arrives`);
}
try {
const live = conn.subscribe(subjectOfTheirs);
subs.push(live);
void (async () => {
for await (const msg of live) {
try {
const m = JSON.parse(sc.decode(msg.data)) as Membership;
issued.set(module, m);
console.log(`[mesh-tools] ${module} on ${node} was issued a new membership; re-serving on it`);
for (const h of issuedHandlers) h(m);
} catch (err) {
console.log(`[mesh-tools] a membership arrived that is not one: ${err}`);
}
}
})();
} catch {
// A subscription this account may not make is a mesh older than the membership.
}
};
await follow(self);
/** The subjects a tool of a served module is served on: from that module's membership when
* issued, derived otherwise (the shape the mesh issues on day one, so the two agree). */
const servedOn = (module: string, tool: string): { subject: string; queue?: string }[] => {
const m = issued.get(module);
if (m) {
const out = m.serves.map((s) => ({ subject: s.subject.replace("{tool}", tool), queue: s.queue }));
// The verb that lists what this module serves is answered on the mesh's plain address for it
// whatever the placement — one answer suffices, so a queue — and on this machine's beside it.
if (tool === "tools" && m.tools && !out.some((s) => s.subject === m.tools)) {
out.unshift({ subject: m.tools, queue: `serve.${module}` });
}
return out;
}
const base = `mesh.mod.${module}.tool.${tool}`;
const out: { subject: string; queue?: string }[] = [{ subject: base, queue: `serve.${module}` }];
if (node) out.push({ subject: `${base}.${node}` });
return out;
};
/** Where a call by key goes: a subject this connection's own membership says it reaches, when it
* says one — the machine's when named — else the derived shape. */
const reachedAt = (key: string): string => {
const [name, wanted] = key.split("@", 2);
const reach = issued.get(self)?.reaches?.[name];
if (reach && reach.length > 0) {
if (wanted) {
const at = reach.find((s) => s.endsWith(`.${wanted}`));
if (at) return at;
} else {
return reach[0];
}
}
return toolSubject(key, self);
};
/** Answer one subject with one handler, and say which machine answered (novox/hq ADR 0159). */
const answerOn = <Req, Res>(
subject: string,
queue: string | undefined,
handler: (body: Req) => Promise<Res>,
): (() => void) => {
const sub = conn.subscribe(subject, queue ? { queue } : {});
subs.push(sub);
void (async () => {
// **A refused subscription is said, never fatal** (novox/hq 04-ISSUES/218, as 217 for the
// announcements). The grants are the mesh's word on what this account may answer; a subject
// they leave out — a seat claimed here and held elsewhere — costs that subject, never the
// module's other tools, its handlers and its provisioning. Unhandled, the refusal ended the
// process and a module's runtime crash-looped on 2026-10-04.
try {
for await (const msg of sub) {
let reply: { result?: Res; error?: string; node?: string };
try {
reply = { result: await handler(JSON.parse(sc.decode(msg.data)) as Req) };
} catch (err) {
// The caller is told, rather than left to time out: a handler that threw is a
// different failure from a tool nobody serves, and only one of them is worth retrying.
reply = { error: err instanceof Error ? err.message : String(err) };
}
if (node) reply.node = node;
msg.respond(sc.encode(JSON.stringify(reply)));
}
} catch (err) {
console.log(`[mesh-tools] the bus refused ${subject}: ${err instanceof Error ? err.message : String(err)}; ` +
"not served here, and the rest serves on");
}
})();
return () => sub.unsubscribe();
};
/**
* Ask one question and await one answer, with the machine that gave it.
*
* Core NATS request/reply, not JetStream: a tool call must never be persisted (design 25 §3),
* and a lost one is a timeout the caller already handles. The reply travels on the inbox the
* request carries, which the responder may answer because its account has `allow_responses`
* — one reply to a message it actually received, and nothing wider.
*/
const ask = async <Req, Res>(key: string, body: Req, on?: string): Promise<Answered<Res>> => {
const msg = await conn.request(on ?? reachedAt(key), sc.encode(JSON.stringify(body)), {
timeout: REQUEST_TIMEOUT_MS,
});
const reply = JSON.parse(sc.decode(msg.data)) as { result?: Res; error?: string; node?: string };
if (reply.error) throw new Error(reply.error);
return { result: reply.result as Res, node: reply.node };
};
return {
async request<Req, Res>(key: string, body: Req): Promise<Res> {
return (await ask<Req, Res>(key, body)).result;
},
ask,
/**
* Answer a question, two ways (novox/hq ADR 0159): on the module's subject in a queue group,
* so several machines may serve one tool and exactly one of them answers each call; and on the
* same subject with this machine as its last token, so a caller that names the machine reaches
* this instance and no other. A runtime that does not know its machine serves only the first,
* which is how it always behaved.
*/
async handle<Req, Res>(key: string, handler: (body: Req) => Promise<Res>): Promise<() => void> {
// A seat's verb named outright is served on the seat's subject as given, for a holder that
// knows its role without a membership; everything else is a served module's own tool, served
// where the mesh issued that module (ADR 0160). A key naming a module this connection does
// not follow is not served here at all: a module serves its own tools, and the node's
// runtime those of the modules it was given (ADR 0175) — never a stranger's.
if (key.startsWith("seat:")) {
const stop = answerOn(toolSubject(key, self), undefined, handler);
return () => stop();
}
const dot = key.indexOf(".");
const module = dot < 0 ? self : key.slice(0, dot);
if (!issued.has(module) && module !== self) {
throw new Error(
`${self} cannot serve ${key}: a module serves its own tools, and a runtime those of the ` +
"modules it follows",
);
}
const tool = dot < 0 ? key : key.slice(dot + 1);
let stops = servedOn(module, tool).map((s) => answerOn(s.subject, s.queue, handler));
// When that module's new membership arrives, serve where it now says and stop serving where
// it no longer does.
issuedHandlers.push((m) => {
if (m.module !== module) return;
stops.forEach((stop) => stop());
stops = servedOn(module, tool).map((s) => answerOn(s.subject, s.queue, handler));
});
return () => stops.forEach((stop) => stop());
},
module: self,
node,
servedOn,
raw(subject: string, answer: (subject: string, data: Uint8Array) => Uint8Array | undefined): () => void {
const sub = conn.subscribe(subject);
subs.push(sub);
void (async () => {
// **A refusal here is said, never fatal** (novox/hq 04-ISSUES/217). This serves discovery —
// what the runtime says about itself — not the work; a bus that refuses it costs the mesh
// seeing this runtime, not the runtime's tools and handlers. Unhandled, the refusal ended the
// process and every per-module container crash-looped on 2026-10-03.
try {
for await (const msg of sub) {
const body = answer(msg.subject, msg.data);
if (body) msg.respond(body);
}
} catch (err) {
console.log(`[mesh-tools] the bus refused ${subject}: ${err instanceof Error ? err.message : String(err)}; ` +
"discovery will not see this runtime there, and it serves on");
}
})();
return () => sub.unsubscribe();
},
follow,
serving: () => [...issued.keys()],
membership: (module?: string) => issued.get(module ?? self),
onMembership: (handler: (m: Membership) => void) => {
issuedHandlers.push(handler);
},
async handleSubject<Req, Res>(subject: string, handler: (body: Req) => Promise<Res>): Promise<() => void> {
return answerOn(subject, undefined, handler);
},
/**
* Emit an event.
*
* Published into JetStream and awaited, so a publish the bus never accepted fails the emit
* rather than vanishing — at-least-once starts at the emitter, not only the consumer
* (ADR 0042).
*
* `msgID` is the event's own id, so a redelivery after a crash between publishing and
* acknowledging is de-duplicated by the server inside its window rather than seen twice.
*/
async publish<T>(env: Envelope<T>): Promise<void> {
// **The body is the payload and the metadata rides as headers** (ADR 0042). That shape
// is what the conformance suite pins: an implementation that nested the whole envelope in
// the body would pass every one of its own tests and agree with nobody.
const meta = (env.headers ?? {}) as Record<string, string>;
const h = natsHeaders();
for (const [k, v] of Object.entries(meta)) {
if (v != null) h.set(k, String(v));
}
if (!meta["content-type"]) h.set("content-type", "application/json");
if (env.node) h.set("x-node", env.node);
// The emitting module's subject: the tool at work's when a served module's tool emits from
// the node's runtime (ADR 0175), this connection's own otherwise.
await js.publish(eventSubject(env.key, atWork.getStore()?.module ?? self), sc.encode(JSON.stringify(env.body)), {
headers: h,
// De-duplicated by the server inside its window, so a redelivery after a crash between
// publishing and acknowledging is not seen twice. Only the emitter can make this id.
msgID: meta["x-event-id"],
});
},
/**
* React to events.
*
* The durable consumer is the **controller's** to create, from what this module declared it
* consumes (design 29 §3) — this binds to it and never creates one. A runtime that created
* its own would be a module deciding its own delivery semantics, and its account cannot
* reach the JetStream API to do it anyway.
*
* **One consumer, one loop, however many patterns a module registers.** A module has exactly one
* durable consumer, so two loops reading it would each take half the messages — and a loop that
* received one its own pattern does not match acknowledges it, which is the right answer for a
* filter wider than anything registered and silent loss when it is another handler's. Every
* registration is therefore dispatched from one reader, and a message is acknowledged once every
* handler it is for has taken it.
*/
async subscribe<T>(
pattern: string,
handler: (env: Envelope<T>) => Promise<void>,
): Promise<() => void> {
const listener = { pattern, handler: handler as (env: Envelope<unknown>) => Promise<void> };
listeners.push(listener);
if (!reading) {
const durable = `${cred.node ?? "?"}_${self}`;
const consumer = await js.consumers.get("EVENTS", durable);
const messages = await consumer.consume();
reading = messages;
void (async () => {
for await (const msg of messages) {
await deliver(msg, listeners);
}
})();
}
return () => {
const at = listeners.indexOf(listener);
if (at >= 0) listeners.splice(at, 1);
if (listeners.length === 0 && reading) {
void reading.close();
reading = undefined;
}
};
},
async close(): Promise<void> {
if (closed) return;
closed = true;
for (const sub of subs) sub.unsubscribe();
// Drain rather than close: an in-flight reply is finished instead of dropped, which for a
// tool call is the difference between an answer and an unexplained timeout at the caller.
await conn.drain();
},
};
}
/**
* Deliver one event to every handler it is for, acknowledging only once each has taken it.
*
* Several registrations share one durable consumer, so matching happens here rather than by having
* each registration read the stream: two readers of one consumer would split it between them, and a
* message that reached the wrong one would be acknowledged as not-for-me and lost.
*/
async function deliver(
msg: JsMsg,
listeners: { pattern: string; handler: (env: Envelope<unknown>) => Promise<void> }[],
): Promise<void> {
let env: Envelope<unknown>;
try {
env = toEnvelope<unknown>(msg);
} catch {
// Unparseable: terminate it. Redelivering a message no version of this code can read is
// an infinite loop, and what the controller keeps of a message given up on (DEAD_LETTERS,
// novox/hq issue 330) is for handlers that fail, not for bytes that were never an envelope.
msg.term();
return;
}
const forThis = listeners.filter((l) => topicMatches(l.pattern, env.key));
if (forThis.length === 0) {
// The consumer's filters are the controller's, derived from what the module declared it
// consumes, and may be wider than anything it registered a handler for. Acknowledge it, or it
// would be redelivered until it expired.
msg.ack();
return;
}
try {
for (const l of forThis) await l.handler(env);
msg.ack();
} catch {
// Negative-acknowledge with a delay, so a handler failing on a transient cause gets another
// attempt, and one failing permanently exhausts max-deliver rather than spinning: the consumer
// gives the event up, and the controller keeps it in DEAD_LETTERS until a person delivers it
// again or drops it (novox/hq issue 330). Delivered again, it arrives under
// `mesh.again.<consumer>.…`, and keyFromSubject reads the same key from it. The consumer's
// limits are the controller's; this only says "not done".
msg.nak(5_000);
}
}
/** Rebuild the envelope a module sees, from the subject, the headers and the payload — the
* mirror of publish, and the reason both live beside each other. */
function toEnvelope<T>(msg: JsMsg): Envelope<T> {
const headers: Record<string, string> = {};
if (msg.headers) {
for (const k of msg.headers.keys()) headers[k] = msg.headers.get(k);
}
return {
// The event's own key, recovered from the subject: `mesh.mod.<module>.event.<key>`. The
// module never sees the subject, only the key it declared.
key: keyFromSubject(msg.subject),
node: headers["x-node"] ?? "",
body: JSON.parse(sc.decode(msg.data)) as T,
headers: headers as EventHeaders,
};
}
/** The event key a module sees: the emitter and the event, which is exactly how its manifest names
* what it consumes (novox/hq design 29 §1, 04-ISSUES/127).
*
* **One vocabulary for the declaration and the handler.** This returned the event name alone, so a
* manifest declaring `consumes: builder.built` produced a handler pattern that could never match
* the key it was compared against — and a module consuming the same event from two emitters could
* not tell them apart except by reading a header. The subject already carries the emitter; naming it
* here makes a mismatch between manifest and code a typo rather than a category error. */
function keyFromSubject(subject: string): string {
const marker = ".event.";
const at = subject.indexOf(marker);
if (at < 0) return subject;
const event = subject.slice(at + marker.length);
// `mesh.mod.<emitter>.event.…` — the emitter is the token before the marker.
const before = subject.slice(0, at).split(".");
const emitter = before[before.length - 1];
return emitter ? `${emitter}.${event}` : event;
}
/** A module's own event subject. Derived, never taken from the caller: the module names its
* event and the mesh decides where it lands (design 29 §1). */
function eventSubject(type: string, self: string): string {
return `mesh.mod.${self}.event.${type}`;
}
/** A tool's subject. A bare name is this module's own tool; `<module>.<tool>` addresses
* another's, which is how a request reaches a module that is not this one; `seat:<seat>.<verb>`
* addresses a role's tool, answered by whoever holds the seat (novox/hq ADR 0132) — with
* `seat:<seat>.<verb>@<node>` for a node-scoped seat, whose tool carries the machine (design 33 §4). */
function toolSubject(key: string, self: string): string {
if (key.startsWith("seat:")) {
const rest = key.slice("seat:".length);
const dot = rest.indexOf(".");
if (dot < 0) throw new Error(`"${key}" names a seat and no verb: seat:<seat>.<verb>`);
const seat = rest.slice(0, dot);
const [verb, node] = rest.slice(dot + 1).split("@", 2);
return node ? `mesh.seat.${seat}.tool.${verb}.${node}` : `mesh.seat.${seat}.tool.${verb}`;
}
// `<module>.<tool>@<node>` names the machine (novox/hq ADR 0159): the same subject with the
// machine as its last token, which is what that instance serves beside the queue.
const [name, node] = key.split("@", 2);
const dot = name.indexOf(".");
const base = dot < 0
? `mesh.mod.${self}.tool.${name}`
: `mesh.mod.${name.slice(0, dot)}.tool.${name.slice(dot + 1)}`;
return node ? `${base}.${node}` : base;
}
/** A seat's verb, as its holder serves it: flat for a mesh seat, carrying the machine for a
* node-scoped one (design 33 §4) — the same shape the controller grants. */
export function seatToolSubject(seat: string, verb: string, scope: string | undefined, node: string | undefined): string {
const base = `mesh.seat.${seat}.tool.${verb}`;
return scope === "node" && node ? `${base}.${node}` : base;
}
function normalizeFingerprint(fingerprint: string): string {
return fingerprint.replace(/^sha256:/i, "").replace(/:/g, "").toLowerCase();
}
/**
* Dial once to see the certificate, and refuse unless it is exactly the one the mesh pinned.
* A certificate authority is not consulted: the mesh issued this and knows its fingerprint,
* which is stronger than trusting whoever a machine's trust store happens to contain.
*
* **The pin is the only check.** What comes back is handed to the client as its TLS options, and
* the client's transport spreads them into Node's own `tls.connect` — so the pinned certificate
* is the one authority the handshake accepts, and the hostname check beside it is replaced with
* one that always passes. Pinning the exact certificate makes verifying its name redundant, and
* the bus's certificate names the seat (`mesh-broker`), not the address a machine happens to
* dial it by: every module on the mesh met "does not match certificate's altnames" the first time
* it reached the handshake (2026-09-28).
*/
async function pinnedTls(rawUrl: string, fingerprint: string): Promise<TlsOptions> {
const url = new URL(rawUrl.includes("://") ? rawUrl : `nats://${rawUrl}`);
const port = url.port ? Number(url.port) : 4222;
// **The bus speaks first, in the clear.** A NATS server sends its INFO line before TLS begins,
// and only then expects the client to start the handshake; a raw TLS connect to that port reads
// the INFO line as a TLS record and fails with "wrong version number" — which is what every
// module met the first time it dialled the bus being built (2026-09-28). So: connect, wait for
// INFO, then start TLS on the same socket, and read the certificate the server presents.
const certificate = await new Promise<tls.DetailedPeerCertificate>((resolve, reject) => {
const plain = net.connect({ host: url.hostname, port }, () => {});
let seenInfo = false;
let buffered = "";
plain.on("error", reject);
plain.on("data", (chunk: Buffer) => {
if (seenInfo) return;
buffered += chunk.toString("utf8");
if (!buffered.includes("\r\n")) return;
seenInfo = true;
plain.removeAllListeners("data");
const secure = tls.connect(
{ socket: plain, rejectUnauthorized: false, servername: url.hostname },
() => {
const peer = secure.getPeerCertificate(true);
secure.end();
resolve(peer);
},
);
secure.on("error", reject);
});
});
const seen = createHash("sha256").update(certificate.raw).digest("hex");
if (seen !== normalizeFingerprint(fingerprint)) {
throw new PinMismatchError(
`the bus at ${url.hostname}:${port} presented ${seen}, not the pinned ${normalizeFingerprint(fingerprint)}`,
);
}
const pem = `-----BEGIN CERTIFICATE-----\n${certificate.raw.toString("base64").replace(/(.{64})/g, "$1\n")}\n-----END CERTIFICATE-----\n`;
// Node's option, not the client's: the transport passes the whole object on. `undefined` from
// checkServerIdentity is "the name is fine"; the pin above already decided the rest.
return { ca: pem, checkServerIdentity: () => undefined } as TlsOptions;
}
/** The mesh's topic matching: `*` is one token, `#` the rest. This is the module's vocabulary —
* a module's `consumes` pattern is matched here, and the subject it becomes is the mesh's
* business, not the module's. */
export function topicMatches(pattern: string, key: string): boolean {
return matchFrom(pattern.split("."), 0, key.split("."), 0);
}
function matchFrom(p: string[], pi: number, k: string[], ki: number): boolean {
if (pi === p.length) return ki === k.length;
// `**` is the mesh's wildcard for the rest of a name; `#` is the old bus's, accepted so a pattern
// written either way behaves the same while both buses ship (novox/hq design 29 §1).
if (p[pi] === "#" || p[pi] === "**") {
for (let skip = ki; skip <= k.length; skip++) {
if (matchFrom(p, pi + 1, k, skip)) return true;
}
return false;
}
if (ki === k.length) return false;
if (p[pi] !== "*" && p[pi] !== k[ki]) return false;
return matchFrom(p, pi + 1, k, ki + 1);
}
+280
View File
@@ -0,0 +1,280 @@
/**
* The mesh's tools, for whoever is on a machine (novox/hq design 25 §7, design 34).
*
* Two surfaces over one thing. A command line, for somebody at a terminal; an MCP server, for an
* agent. Both are adapters over the same three calls — what tools are there, what does this one take,
* call it — because a second way of reaching a tool is a second thing to keep correct.
*
* **It uses the same client a module's runtime uses.** Not a second protocol and not a bridge: the
* caller connects as its own bus user — a person's, or the console's — publishes on the tool subjects
* that account permits, and the server refuses anything else. So "what may this ask" is answered by the
* same permission list that answers it for a module, and there is nothing here for an audit to read
* separately.
*
* What the caller may NOT do is the more interesting half, and none of it is enforced here — it is the
* account (design 25 §4): it cannot publish an event, so it cannot claim a module said something; it
* has no consumer, so there is no delivery to acknowledge; and it cannot answer a request, so it cannot
* impersonate a module on a bus where anyone may serve a tool.
*/
import { readFile } from "node:fs/promises";
import type { Broker } from "@novox/mesh-sdk/messaging";
import { connectNats, type Answered, type Credential } from "./broker-nats.js";
import { TOOLS_VERB, type ToolsAnswer } from "./runtime.js";
/** Where the catalogue answers which modules the mesh holds. */
const CATALOGUE_MODULES = "mesh-catalog.catalog_modules";
/** Where the mesh answers every role's tools, from its records: the mesh-controller seat's own
* `tools` verb (novox/hq ADR 0154, design 33 §5). */
const SEAT_TOOLS = "seat:mesh-controller.tools";
/** A tool as its module describes it. */
export interface Tool {
/** The module that serves it — or, for a role's tool, the seat. */
module: string;
name: string;
description?: string;
/** The JSON schema of what it takes, as the module declared it. */
input?: unknown;
/** True for a role's tool: addressed to the seat, answered by whoever holds it (ADR 0132). */
seat?: boolean;
/** A role's scope: `node` for a seat held once per machine, whose verb is asked of one machine
* (design 33 §4) and takes `node` for it; `mesh` or absent otherwise. */
scope?: string;
/** Where the tool is answered, as the mesh issued it (ADR 0160): the plain subject first when the
* module answers for itself anywhere, then one per machine. Absent for a runtime older than this. */
subjects?: string[];
}
/**
* What the mesh could say about its tools when asked (design 34 §3).
*
* **Silence is named, never dropped.** A module the catalogue holds and nothing answered for is in
* `notAnswering`, because a tool that is not offered looks exactly like a tool that does not exist,
* and those need different people to fix them.
*/
export interface Listing {
tools: Tool[];
/** Modules the catalogue holds whose runtime did not answer `tools`: not assigned, not up, or built
* before the runtime answered it. Each may still be called by name. The mesh's own records are
* listed here as `mesh-controller (seat)` when the control plane did not answer. */
notAnswering: string[];
}
/** The seats and the verbs each declares, from the last listing, so a call can tell a role's tool
* from a module's when the two share a prefix (a module and a seat may share a name). */
export type Seats = Map<string, Set<string>>;
/** The roles' tools, keyed the way `toolKey` names them. */
export function seatsIn(have: Listing): Seats {
const seats: Seats = new Map();
for (const t of have.tools) {
if (!t.seat) continue;
if (!seats.has(t.module)) seats.set(t.module, new Set());
seats.get(t.module)!.add(t.name);
}
return seats;
}
/** The key a call uses for `<prefix>.<name>`: a role's when the prefix is a seat declaring that
* verb, a module's otherwise. Both names for one capability are deliberate and bounded (ADR 0132);
* the seat wins only for a verb it actually declares, so a module's own tool is never shadowed. */
export function toolKey(name: string, seats?: Seats): string {
if (name.startsWith("seat:")) return name;
const dot = name.indexOf(".");
if (dot < 0) return name;
const prefix = name.slice(0, dot);
const verb = name.slice(dot + 1);
if (seats?.get(prefix)?.has(verb)) return `seat:${prefix}.${verb}`;
return name;
}
/**
* A person's credential, as `operator issue` prints it.
*
* The same shape a module is handed, minus the parts a module needs and a person does not: no node,
* because a person is not on a machine, and no module, because they are not one.
*/
export interface PersonCredential extends Credential {
person?: string;
invokes?: string[];
}
/** Read the credential from the file `operator issue` produced. */
export async function credentialFrom(path: string): Promise<PersonCredential> {
const raw = await readFile(path, "utf8");
let held: PersonCredential;
try {
held = JSON.parse(raw) as PersonCredential;
} catch (e) {
throw new Error(
`${path} is not a credential this mesh issued: ${(e as Error).message}. ` +
"It is the JSON `operator issue` printed, saved verbatim.",
);
}
if (!held.url || !held.user || !held.password) {
throw new Error(
`${path} names no bus, user or password. It is the JSON \`operator issue\` printed, saved ` +
"verbatim — not an edited copy of it.",
);
}
return held;
}
/** Connect as this person. The module name the runtime wants is their own user, because every subject
* it derives is for a tool somebody else serves. */
export async function connectAs(held: PersonCredential): Promise<Broker> {
return connectNats({ ...held, module: held.user });
}
/**
* Connect as the console: the module credential the mesh delivered (novox/hq ADR 0152), read from
* the same variable every runtime reads. It names the node and the module, so the account's inbox
* and subjects derive from what the mesh authorised and from nothing in this process's environment.
*/
export async function connectAsTheConsole(path: string): Promise<{ bus: Broker; who: string }> {
const raw = await readFile(path, "utf8");
let held: Credential;
try {
held = JSON.parse(raw) as Credential;
} catch (e) {
throw new Error(`${path} is not a broker credential: ${(e as Error).message}`);
}
if (!held.url || !held.module || !held.user) {
throw new Error(
`${path} names no bus, module or user: the console runs on the credential the mesh sealed to ` +
"this machine for it, and nothing else",
);
}
return { bus: await connectNats(held), who: `${held.node ?? "?"}.${held.module}` };
}
/**
* What tools the mesh has, asked of the modules (design 34 §3).
*
* The catalogue says which modules the mesh holds; each module says what it serves, through the one
* verb its runtime answers for it. **Asked, not configured**: a client carrying its own list would be a
* list that goes stale the first time a module is assigned. Every module is asked at once, and the bus
* refuses at once a request nothing serves, so the cost is bounded by the modules that are up.
*/
export async function toolsOn(bus: Broker): Promise<Listing> {
const [answered, roles] = await Promise.all([
bus.request<Record<string, never>, { modules?: { module: string }[] }>(CATALOGUE_MODULES, {}),
// The roles' tools, from the mesh's records (design 33 §5). Asked beside the modules rather
// than first: a control plane that is restarting must not hide every module's tools with it.
bus
.request<Record<string, never>, { seats?: { seat: string; scope?: string; tools?: ToolsAnswer["tools"] }[] }>(
SEAT_TOOLS,
{},
)
.catch(() => undefined),
]);
const names = (answered.modules ?? []).map((m) => m.module).filter((m) => typeof m === "string");
const asked = await Promise.allSettled(
names.map((module) => bus.request<Record<string, never>, ToolsAnswer>(`${module}.${TOOLS_VERB}`, {})),
);
const tools: Tool[] = [];
const notAnswering: string[] = [];
if (roles) {
for (const s of roles.seats ?? []) {
// A node-scoped seat's tool is asked of one machine (design 33 §4): listed with its scope, so
// a caller names the machine and the call carries it — `seat:<seat>.<verb>@<node>`. Left out
// of the listing, the verb never resolved as a seat's and nothing served it (ADR 0170).
for (const t of s.tools ?? []) {
tools.push({ module: s.seat, name: t.name, description: t.description, input: t.input, seat: true, scope: s.scope });
}
}
} else {
notAnswering.push("mesh-controller (seat)");
}
asked.forEach((outcome, i) => {
const module = names[i]!;
if (outcome.status === "fulfilled" && typeof outcome.value?.failed === "string") {
// The runtime answered for it and serves nothing: the bundle failed to load (ADR 0175). Said
// with the reason, because "not answering" would send somebody to check an assignment that
// is fine.
notAnswering.push(`${module} (its tools bundle failed to load: ${outcome.value.failed})`);
} else if (outcome.status === "fulfilled" && Array.isArray(outcome.value?.tools)) {
for (const t of outcome.value.tools) {
tools.push({ module, name: t.name, description: t.description, input: t.input, subjects: t.subjects });
}
} else {
notAnswering.push(module);
}
});
tools.sort((a, b) => `${a.module}.${a.name}`.localeCompare(`${b.module}.${b.name}`));
notAnswering.sort();
return { tools, notAnswering };
}
/** Call one tool. The key is `<module>.<tool>`, which is what a person types and what the account
* permits — one vocabulary, so a refusal names the thing they asked for. */
/** Call a tool and learn which machine answered (novox/hq ADR 0159). `<module>.<tool>@<node>` asks
* the instance on one machine; without it, whichever instance answers first does, and the answer
* says which. */
export async function callTool(
bus: Broker,
key: string,
args: unknown,
seats?: Seats,
listing?: Listing,
): Promise<Answered<unknown>> {
const at = key.indexOf("@");
const name = at < 0 ? key : key.slice(0, at);
const node = at < 0 ? "" : key.slice(at + 1);
if (!name.includes(".")) {
throw new Error(
`"${key}" does not name a tool: write <module>.<tool>, as \`mesh tools\` lists them, ` +
"or <module>.<tool>@<node> for the instance on one machine",
);
}
const resolved = toolKey(name, seats) + (node ? `@${node}` : "");
// Where the tool is answered is the module's to say and the mesh's to issue (ADR 0160): when the
// listing carried subjects for it, the call goes to one of those and composes nothing.
const on = subjectListed(name, node, listing);
const asking = bus as Broker & { ask?: <Req, Res>(k: string, b: Req, on?: string) => Promise<Answered<Res>> };
if (typeof asking.ask === "function") return asking.ask<unknown, unknown>(resolved, args ?? {}, on);
return { result: await bus.request<unknown, unknown>(resolved, args ?? {}) };
}
/** The subject the listing says answers `<module>.<tool>` — the machine's when one is named, else
* the plain one — or undefined when the listing said none, and the key is composed as before. */
export function subjectListed(name: string, node: string, listing?: Listing): string | undefined {
if (!listing) return undefined;
const dot = name.indexOf(".");
const module = name.slice(0, dot);
const tool = name.slice(dot + 1);
const found = listing.tools.find((t) => t.module === module && t.name === tool && !t.seat);
const subjects = found?.subjects ?? [];
if (subjects.length === 0) return undefined;
if (node) return subjects.find((s) => s.endsWith(`.${node}`));
return subjects[0];
}
/**
* Why a call failed, said so that the remedy is in the words.
*
* Three answers a person actually gets, and they need different things done: nobody serves that tool,
* the mesh refused this account, or the tool itself failed. Without this they are one timeout and a
* stack trace.
*/
export function whyItFailed(key: string, err: unknown): string {
const message = err instanceof Error ? err.message : String(err);
if (/no responders|503/i.test(message)) {
return `nothing serves ${key}. The module may not be assigned to any machine, or it is down` +
(key.startsWith("seat:") ? ", or nothing holds that seat" : "") +
" — `mesh tools` lists what answered.";
}
if (/permissions violation|authorization/i.test(message)) {
return `this account may not call ${key}. What it may call was fixed when it was issued — a ` +
"person's by `operator issue`, the console's by its manifest.";
}
if (/timeout/i.test(message)) {
return `${key} did not answer in time. Something is serving it, so this is the tool being slow ` +
"rather than absent.";
}
return `${key} failed: ${message}`;
}
+165
View File
@@ -0,0 +1,165 @@
/**
* The console's endpoint: MCP over HTTP, on a machine's loopback (novox/hq ADR 0152, design 34 §2).
*
* **Loopback is the authority boundary.** Whoever can connect is on the machine, and whoever is on the
* machine is the account that owns the mesh there (ADR 0034, ADR 0144). So there is no token and no
* login here, and the one thing this file enforces is that it binds nothing else: a console reachable
* from another machine would be authority over the mesh handed to whoever finds the port.
*
* The transport is the streamable-HTTP shape an agent host speaks: `POST /mcp` with one JSON-RPC
* message, answered with one JSON body. No session, because the surface holds nothing per caller; no
* event stream, because nothing here has anything to say unasked.
*/
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import type { Broker } from "@novox/mesh-sdk/messaging";
import { mcpSurface, type Reply, type Request } from "./mcp.js";
/** The most a request body may be. A tool's arguments are small; a megabyte is somebody else's file. */
const BODY_LIMIT = 1 << 20;
export interface Listening {
/** Where it listens, as `host:port`, with the port the machine actually gave. */
address: string;
close(): Promise<void>;
}
/** Hosts that are this machine and no other. */
const loopback = new Set(["127.0.0.1", "::1", "localhost", "[::1]"]);
/**
* Listen on `host:port`. Refused unless the host is loopback — said before binding, so a manifest or
* a flag that would open the console to a network is a startup failure rather than something
* discovered by whoever finds it.
*/
export async function serveMcpHttp(bus: Broker, who: string, listen: string): Promise<Listening> {
const at = listen.lastIndexOf(":");
if (at < 0) {
throw new Error(`"${listen}" is not host:port`);
}
const host = listen.slice(0, at);
const port = Number(listen.slice(at + 1));
if (!loopback.has(host)) {
throw new Error(
`the console listens on loopback and nowhere else (novox/hq ADR 0152): "${host}" is not this ` +
"machine's own address — whoever is on the machine owns the mesh there, and nobody else may reach this",
);
}
if (!Number.isInteger(port) || port < 0 || port > 65535) {
throw new Error(`"${listen.slice(at + 1)}" is not a port`);
}
const surface = mcpSurface(bus, who);
const server = createServer((req, res) => {
void route(req, res, surface.handle).catch((e) => {
json(res, 500, { jsonrpc: "2.0", id: null, error: { code: -32603, message: String(e) } });
});
});
await new Promise<void>((resolve, reject) => {
server.once("error", reject);
server.listen(port, host.replace(/^\[|\]$/g, ""), () => resolve());
});
const bound = server.address();
const address = typeof bound === "object" && bound ? `${host}:${bound.port}` : listen;
return {
address,
close: () =>
new Promise<void>((resolve) => {
server.close(() => resolve());
}),
};
}
async function route(
req: IncomingMessage,
res: ServerResponse,
handle: (r: Request) => Promise<Reply | undefined>,
): Promise<void> {
const path = (req.url ?? "/").split("?")[0];
if (path === "/") {
res.writeHead(200, { "content-type": "text/plain; charset=utf-8" });
res.end("the mesh's console: MCP over HTTP at POST /mcp (novox/hq design 34)\n");
return;
}
if (path !== "/mcp") {
json(res, 404, { error: "the console serves /mcp and nothing else" });
return;
}
switch (req.method) {
case "POST":
break;
case "DELETE":
// A host ending a session. There is no session to end; saying so is the truthful answer.
res.writeHead(204).end();
return;
case "GET":
// A host opening an event stream. The console has nothing to say unasked.
res.writeHead(405, { allow: "POST, DELETE" }).end();
return;
default:
res.writeHead(405, { allow: "POST, DELETE" }).end();
return;
}
let body: string;
try {
body = await read(req);
} catch (e) {
json(res, 413, { jsonrpc: "2.0", id: null, error: { code: -32600, message: String(e) } });
return;
}
let parsed: unknown;
try {
parsed = JSON.parse(body);
} catch {
json(res, 400, { jsonrpc: "2.0", id: null, error: { code: -32700, message: "the body is not JSON" } });
return;
}
// One message, or a batch of them; a batch is answered as a batch. A notification gets no reply
// and, alone, no body: 202 is how the transport says "heard".
if (Array.isArray(parsed)) {
const replies = (await Promise.all(parsed.map((r) => handle(r as Request)))).filter(Boolean);
if (replies.length === 0) {
res.writeHead(202).end();
} else {
json(res, 200, replies);
}
return;
}
const reply = await handle(parsed as Request);
if (!reply) {
res.writeHead(202).end();
return;
}
json(res, 200, reply);
}
function read(req: IncomingMessage): Promise<string> {
return new Promise((resolve, reject) => {
let size = 0;
const chunks: Buffer[] = [];
req.on("data", (chunk: Buffer) => {
size += chunk.length;
if (size > BODY_LIMIT) {
reject(new Error(`the request is larger than ${BODY_LIMIT} bytes`));
req.destroy();
return;
}
chunks.push(chunk);
});
req.on("end", () => resolve(Buffer.concat(chunks).toString("utf8")));
req.on("error", reject);
});
}
function json(res: ServerResponse, status: number, body: unknown): void {
const text = JSON.stringify(body);
res.writeHead(status, {
"content-type": "application/json; charset=utf-8",
"content-length": Buffer.byteLength(text),
});
res.end(text);
}
+192
View File
@@ -0,0 +1,192 @@
// A tools bundle as a process the runtime launches (novox/hq ADR 0188).
//
// The runtime does not run a tool's code itself when the bundle is not JavaScript: it starts the
// bundle's executable as a child with the runtime's environment and speaks MCP over stdio to it —
// `initialize`, `tools/list` once, `tools/call` per call. A Rust binary, a Go binary, a Python
// script and a Node script are the same thing from here: a process that answers those. Everything
// the mesh adds — the subjects from the membership, the held seats, the `tools` answer, a bundle
// that failed named and the others serving — is the runtime's, outside this file.
//
// A tool the child lists as `<seat>.<verb>` is the module's implementation of that seat's verb;
// any other name is the module's own tool. The same rule the in-process registration follows.
import { spawn, type ChildProcess } from "node:child_process";
import { accessSync, constants } from "node:fs";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import { broker, type Envelope } from "@novox/mesh-sdk/messaging";
import { atWork } from "./broker-nats.js";
/** The protocol version this speaks; a bundle says the same. */
export const PROTOCOL = "2025-03-26";
/** How long a child has to answer `initialize` and `tools/list` before it is a failed bundle, and
* how long a call may take before the caller is told the tool is slow rather than absent. */
const HANDSHAKE_MS = 10_000;
const CALL_MS = 30_000;
/** Whether an entrypoint is launched as a process rather than imported: anything that is not a
* plain JavaScript file, and a JavaScript file marked executable — a bundle written against the
* protocol in TypeScript, served the same way as any other language. */
export function launches(entry: string): boolean {
const javascript = /\.(m|c)?js$/.test(entry);
let executable = false;
try {
accessSync(entry, constants.X_OK);
executable = true;
} catch {
// not executable, or not there — importing will say which
}
return !javascript || executable;
}
/** What a launched bundle registers: the groups the in-process path would have, by name. */
export interface Launched {
registrations: { module: string; tools: ToolDefinition[] }[];
stop(): void;
}
interface Pending {
resolve(v: any): void;
reject(e: Error): void;
timer: NodeJS.Timeout;
}
/**
* Launch a bundle and learn its tools. Rejects when the child cannot be started or does not complete
* the handshake, which the runtime records as the bundle having failed. A child that exits later is
* started again on the next call, once; a call in flight when it died is told so.
*/
export async function launch(module: string, entry: string, env: NodeJS.ProcessEnv = process.env): Promise<Launched> {
let child: ChildProcess | undefined;
let nextId = 1;
const pending = new Map<number, Pending>();
let stopped = false;
const start = async (): Promise<void> => {
const proc = spawn(entry, [], { stdio: ["pipe", "pipe", "pipe"], env });
child = proc;
let buffered = "";
proc.stdout!.on("data", (chunk: Buffer) => {
buffered += chunk.toString("utf8");
let at: number;
while ((at = buffered.indexOf("\n")) >= 0) {
const line = buffered.slice(0, at).trim();
buffered = buffered.slice(at + 1);
if (!line) continue;
let reply: { id?: number | string; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
try {
reply = JSON.parse(line);
} catch {
console.log(`[mesh-tools] ${module}'s bundle said something that is not a reply: ${line.slice(0, 120)}`);
continue;
}
// **The bundle asks the runtime to emit** (novox/hq ADR 0193): published on the bus as this
// module, and answered once the bus has accepted it, so the tool's emit means what it means
// in-process. Nothing else a bundle may ask.
if (typeof reply.method === "string") {
const id = reply.id;
const answer = (m: Record<string, unknown>) => proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", id, ...m }) + "\n");
if (reply.method !== "mesh/publish") {
if (id !== undefined) answer({ error: { code: -32601, message: `the runtime answers no ${reply.method} from a bundle` } });
continue;
}
atWork.run({ module }, () => broker().publish(reply.params as Envelope<unknown>))
.then(() => { if (id !== undefined) answer({ result: {} }); })
.catch((err: unknown) => { if (id !== undefined) answer({ error: { code: -32000, message: err instanceof Error ? err.message : String(err) } }); });
continue;
}
const waiting = typeof reply.id === "number" ? pending.get(reply.id) : undefined;
if (!waiting) continue;
pending.delete(reply.id as number);
clearTimeout(waiting.timer);
if (reply.error) waiting.reject(new Error(reply.error.message ?? "the bundle refused the request"));
else waiting.resolve(reply.result);
}
});
// stderr is the bundle's log; kept under the module's name so a fault reads where it belongs.
// The last thing it said is kept, so a bundle that dies says why in its own words, not by code.
let lastSaid = "";
proc.stderr!.on("data", (chunk: Buffer) => {
for (const line of chunk.toString("utf8").split("\n")) {
if (!line.trim()) continue;
console.log(`[${module}] ${line}`);
if (/\S/.test(line) && !/^\s+at\s/.test(line) && !/^Node\.js v/.test(line)) lastSaid = line.trim();
}
});
const exited = new Promise<never>((_, reject) => {
proc.once("error", (err) => reject(err));
proc.once("exit", (code, signal) => {
const why = `${module}'s bundle exited (${signal ?? code})` + (lastSaid ? `: ${lastSaid}` : "");
for (const [id, p] of pending) {
pending.delete(id);
clearTimeout(p.timer);
p.reject(new Error(why));
}
if (child === proc) child = undefined;
if (!stopped) console.log(`[mesh-tools] ${why}; started again on its next call`);
reject(new Error(why));
});
});
const ask = (method: string, params: unknown, ms: number): Promise<any> =>
Promise.race([
new Promise<any>((resolve, reject) => {
const id = nextId++;
const timer = setTimeout(() => {
pending.delete(id);
reject(new Error(`${module}'s bundle did not answer ${method} in ${ms / 1000}s`));
}, ms);
pending.set(id, { resolve, reject, timer });
proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n");
}),
exited,
]);
exited.catch(() => {}); // observed through the race; never unhandled
(proc as ChildProcess & { ask?: typeof ask }).ask = ask;
await ask("initialize", { protocolVersion: PROTOCOL, capabilities: {}, clientInfo: { name: "node-tools", version: "1" } }, HANDSHAKE_MS);
proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");
};
const asking = async (method: string, params: unknown, ms: number): Promise<any> => {
if (!child) await start();
return (child as ChildProcess & { ask: (m: string, p: unknown, ms: number) => Promise<any> }).ask(method, params, ms);
};
await start();
const listed = (await asking("tools/list", {}, HANDSHAKE_MS)) as { tools?: { name: string; description?: string; inputSchema?: unknown }[] };
const groups = new Map<string, ToolDefinition[]>();
for (const t of listed.tools ?? []) {
const dot = t.name.indexOf(".");
const under = dot < 0 ? module : t.name.slice(0, dot);
const name = dot < 0 ? t.name : t.name.slice(dot + 1);
const tools = groups.get(under) ?? [];
tools.push({
name,
description: t.description ?? "",
input: (t.inputSchema as Record<string, unknown> | undefined) ?? {},
run: async (args) => {
const result = (await asking("tools/call", { name: t.name, arguments: args ?? {} }, CALL_MS)) as {
content?: { type: string; text?: string }[];
isError?: boolean;
};
const text = result?.content?.find((c) => c.type === "text")?.text ?? "";
if (result?.isError) throw new Error(text || `${module}.${t.name} failed`);
// The bundle's answer is JSON as text (that is what every MCP host renders); handed back as
// the value it encodes so a caller on the bus sees what an in-process tool would return.
try {
return JSON.parse(text);
} catch {
return text;
}
},
});
groups.set(under, tools);
}
return {
registrations: [...groups].map(([under, tools]) => ({ module: under, tools })),
stop: () => {
stopped = true;
child?.kill("SIGTERM");
child = undefined;
},
};
}
+128 -15
View File
@@ -1,10 +1,14 @@
// The runnable entrypoint. Three modes: // The runnable entrypoint. Three modes:
// //
// mesh-tools serve — bind the broker and serve the assigned modules until // mesh-tools serve — bind the broker and serve the assigned modules until
// stopped. A module entrypoint that subscribes to events (on("#")) // stopped; as the node-tools module, also the console on loopback. A module entrypoint that subscribes to events (on("#"))
// starts consuming as it is imported, so this also runs consumers. // starts consuming as it is imported, so this also runs consumers.
// mesh-tools emit TYPE [JSON] emit one event onto the mesh and exit — an operable primitive, // mesh-tools emit TYPE [JSON] emit one event onto the mesh and exit — an operable primitive,
// and what an events test uses to put a message on the wire. // and what an events test uses to put a message on the wire.
// mesh-tools prepare bring this module's state to the shape this version needs and exit
// — the runtime's answer to the word the mesh asks every module
// (novox/hq ADR 0135). The entrypoints come from MESH_PREPARE, which
// the module's own image names beside MESH_TOOL_MODULES.
// mesh-tools run ENTRYPOINT run one compiled module entrypoint to completion and exit — the // mesh-tools run ENTRYPOINT run one compiled module entrypoint to completion and exit — the
// runtime side of a run-once step (novox/hq ADR 0052). It imports // runtime side of a run-once step (novox/hq ADR 0052). It imports
// the given entrypoint, whose top-level code does its work — seed a // the given entrypoint, whose top-level code does its work — seed a
@@ -17,14 +21,19 @@
// MESH_BROKER_FILE a sealed {url, fingerprint} the mesh delivered (novox/hq ADR 0043) — an // MESH_BROKER_FILE a sealed {url, fingerprint} the mesh delivered (novox/hq ADR 0043) — an
// amqps account scoped to this module. Preferred: a module holds its own. // amqps account scoped to this module. Preferred: a module holds its own.
// MESH_BROKER_URL a plain URL, for the bootstrap/admin case before a module has an account. // MESH_BROKER_URL a plain URL, for the bootstrap/admin case before a module has an account.
// MESH_TOOL_MODULES /path/a,/path/b,… compiled module entrypoints (serve mode) // MESH_TOOL_MODULES <module>=/path/a,… the modules to serve and their compiled entrypoints (serve
// mode; ADR 0175); a bare path is an entrypoint of the credential's own module
// MESH_PREPARE /path/a,/path/b,… compiled entrypoints that prepare this module's state
// MESH_MODULE / MESH_NODE the identity stamped onto emitted events (ADR 0042) // MESH_MODULE / MESH_NODE the identity stamped onto emitted events (ADR 0042)
import { readFileSync } from "node:fs"; import { readFileSync } from "node:fs";
import { pathToFileURL } from "node:url"; import { pathToFileURL } from "node:url";
import { connectAmqp, fatalBrokerReason } from "./broker-amqp.js"; import { connectNats, fatalBrokerReason as fatalNatsReason, type Credential } from "./broker-nats.js";
import type { Credential } from "./broker-amqp.js"; import { takeToolEnvs, runTools, type ServedModule } from "./runtime.js";
import { runTools } from "./runtime.js"; import { serveMcpHttp, type Listening } from "./http.js";
/** The credential this process connected with, for what it says beyond the connection (ADR 0159). */
let lastCredential: Credential | undefined;
import { invokeTool } from "@novox/mesh-sdk/tools"; import { invokeTool } from "@novox/mesh-sdk/tools";
import { useBroker } from "@novox/mesh-sdk/messaging"; import { useBroker } from "@novox/mesh-sdk/messaging";
import type { Broker } from "@novox/mesh-sdk/messaging"; import type { Broker } from "@novox/mesh-sdk/messaging";
@@ -35,12 +44,19 @@ import { emit } from "@novox/mesh-sdk/events";
* over the plain bootstrap URL otherwise. A scoped module assumes the foundation's exchanges exist — * over the plain bootstrap URL otherwise. A scoped module assumes the foundation's exchanges exist —
* its account may not declare them (ADR 0043). * its account may not declare them (ADR 0043).
*/ */
// fatalBrokerReasonFor is the reason a connection failure is final rather than "not yet", for
// whichever bus this runtime is on — each transport knows its own refusals.
function fatalBrokerReasonFor(err: unknown): string | null {
return fatalNatsReason(err);
}
async function connectBroker(): Promise<Broker> { async function connectBroker(): Promise<Broker> {
const file = process.env.MESH_BROKER_FILE; const file = process.env.MESH_BROKER_FILE;
if (file) { if (file) {
let credential: Credential; let credential: Credential;
try { try {
credential = JSON.parse(readFileSync(file, "utf8")) as Credential; credential = JSON.parse(readFileSync(file, "utf8")) as Credential;
lastCredential = credential;
} catch (err) { } catch (err) {
console.error(`mesh-tools: cannot read the broker credential at ${file}: ${err}`); console.error(`mesh-tools: cannot read the broker credential at ${file}: ${err}`);
process.exit(1); process.exit(1);
@@ -54,7 +70,13 @@ async function connectBroker(): Promise<Broker> {
// what the environment says. // what the environment says.
if (credential.node) process.env.MESH_NODE = credential.node; if (credential.node) process.env.MESH_NODE = credential.node;
if (credential.module) process.env.MESH_MODULE = credential.module; if (credential.module) process.env.MESH_MODULE = credential.module;
return connectAmqp(credential, { assumeExchanges: true }); // **The credential names the bus.** A module moved to the bus being built was handed a
// credential for it — `nats://…` with user, password and fingerprint beside the address — and
// nothing else in its environment changed (design 25; novox/hq design 28 task 5.2). The scheme
// is enough to know which bus to speak; a runtime that always dialled the old one would keep
// serving and answer nobody.
// One bus (novox/hq ADR 0131, design 28 task 5.5): the credential names it, and it is this.
return connectNats(credential);
} }
const url = process.env.MESH_BROKER_URL; const url = process.env.MESH_BROKER_URL;
if (!url) { if (!url) {
@@ -63,7 +85,7 @@ async function connectBroker(): Promise<Broker> {
); );
process.exit(1); process.exit(1);
} }
return connectAmqp(url); return connectNats({ url });
} }
/** /**
@@ -83,7 +105,7 @@ async function connectBrokerPatiently(): Promise<Broker> {
try { try {
return await connectBroker(); return await connectBroker();
} catch (err) { } catch (err) {
const fatal = fatalBrokerReason(err); const fatal = fatalBrokerReasonFor(err);
if (fatal !== null) { if (fatal !== null) {
console.error(`mesh-tools: ${fatal} — waiting will not fix this; giving up`); console.error(`mesh-tools: ${fatal} — waiting will not fix this; giving up`);
throw err; throw err;
@@ -98,17 +120,68 @@ async function connectBrokerPatiently(): Promise<Broker> {
} }
} }
async function serve(): Promise<void> { /**
const moduleEntrypoints = (process.env.MESH_TOOL_MODULES ?? "") * What MESH_TOOL_MODULES names (novox/hq ADR 0175, to-be 38 WP1): `<module>=<entrypoint>` entries,
.split(",") * comma-separated, several per module allowed — the node's runtime serving every assigned module's
.map((s) => s.trim()) * bundle. A bare path is the one-module form the per-module containers still set: an entrypoint of
.filter(Boolean); * the credential's own module. Both may appear; the result is one list of modules.
*/
export function servedModulesFrom(spec: string, own: string | undefined): { serves: ServedModule[]; moduleEntrypoints: string[] } {
const serves = new Map<string, string[]>();
const moduleEntrypoints: string[] = [];
for (const raw of spec.split(",")) {
const entry = raw.trim();
if (!entry) continue;
const eq = entry.indexOf("=");
if (eq < 0) {
moduleEntrypoints.push(entry);
continue;
}
const module = entry.slice(0, eq).trim();
const path = entry.slice(eq + 1).trim();
if (!module || !path) {
throw new Error(`MESH_TOOL_MODULES: "${entry}" is neither <module>=<entrypoint> nor an entrypoint of this module`);
}
if (module === own) {
moduleEntrypoints.push(path);
continue;
}
serves.set(module, [...(serves.get(module) ?? []), path]);
}
return { serves: [...serves].map(([module, entrypoints]) => ({ module, entrypoints })), moduleEntrypoints };
}
/** The module that is the node's tool runtime (novox/hq ADR 0175, to-be 38 WP3): on its credential,
* `serve` is also the console — MCP on the machine's loopback (design 34). */
export const RUNTIME_MODULE = "node-tools";
/** Where the console listens when the runtime is node-tools and nothing says otherwise: the port
* the module's manifest declares `from: machine`. MESH_CONSOLE_LISTEN overrides it either way. */
const CONSOLE_LISTEN = "127.0.0.1:4270";
async function serve(): Promise<void> {
const broker = await connectBrokerPatiently(); const broker = await connectBrokerPatiently();
const stop = await runTools({ broker, moduleEntrypoints }); // Parsed after connecting: a bare entrypoint belongs to the module the credential names.
const { serves, moduleEntrypoints } = servedModulesFrom(process.env.MESH_TOOL_MODULES ?? "", lastCredential?.module);
// Each module's environment, composed by the mesh (ADR 0192): taken before any bundle is imported.
const envs = takeToolEnvs();
const stop = await runTools({ broker, serves, moduleEntrypoints, credential: lastCredential, envs });
// The console is this runtime's serving mode (ADR 0175 §6): as node-tools, or wherever the
// listen address is given, the same process answers MCP on loopback for whoever is on the
// machine. A module's own runtime in a container on the machine's network does not — two of
// them on one port would be the fault, and the console is one per machine.
const listen = process.env.MESH_CONSOLE_LISTEN ?? (lastCredential?.module === RUNTIME_MODULE ? CONSOLE_LISTEN : "");
let consoleUp: Listening | undefined;
if (listen) {
const who = `${lastCredential?.node ?? "?"}.${lastCredential?.module ?? RUNTIME_MODULE}`;
consoleUp = await serveMcpHttp(broker, who, listen);
console.log(`mesh console listening on http://${consoleUp.address}/mcp as ${who}`);
}
const shutdown = async (): Promise<void> => { const shutdown = async (): Promise<void> => {
stop(); stop();
await consoleUp?.close();
await broker.close(); await broker.close();
process.exit(0); process.exit(0);
}; };
@@ -166,12 +239,49 @@ async function runEntry(entrypoint: string): Promise<void> {
await import(pathToFileURL(entrypoint).href); await import(pathToFileURL(entrypoint).href);
} }
/**
* Bring this module's state to the shape this version needs, and exit — the runtime's answer to the
* one word the mesh asks every module (novox/hq ADR 0135).
*
* The entrypoints come from `MESH_PREPARE`, which a module's own image sets beside the entrypoints it
* already lists there: the module knows which of its files prepares its state, and nothing else could.
* Each is imported in the order given, to completion, with no broker — preparation runs before the
* version that would use it, so there is nothing yet to talk to.
*
* **An empty list is a failure, not a no-op.** The mesh only asks this of a module whose manifest says
* it prepares something; a module that says so and names nothing has been built wrong, and exiting 0
* would let that version serve against a state nobody shaped.
*/
async function prepareState(): Promise<void> {
const named = (process.env.MESH_PREPARE ?? "")
.split(",")
.map((entry) => entry.trim())
.filter((entry) => entry !== "");
if (named.length === 0) {
console.error(
"mesh-tools prepare: this module was asked to prepare its state and its image names nothing " +
"to do it with — set MESH_PREPARE to the compiled entrypoint(s) that prepare it, the way " +
"MESH_TOOL_MODULES names the ones it serves",
);
process.exit(1);
}
for (const entrypoint of named) {
console.log(`[mesh-tools] preparing with ${entrypoint}`);
await runEntry(entrypoint);
}
console.log(`[mesh-tools] prepared: ${named.length} entrypoint(s) ran to completion`);
}
async function main(): Promise<void> { async function main(): Promise<void> {
const [command, ...rest] = process.argv.slice(2); const [command, ...rest] = process.argv.slice(2);
if (command === "run") { if (command === "run") {
await runEntry(rest[0] ?? ""); await runEntry(rest[0] ?? "");
return; return;
} }
if (command === "prepare") {
await prepareState();
return;
}
if (command === "invoke") { if (command === "invoke") {
const [module, tool] = rest; const [module, tool] = rest;
if (!module || !tool) { if (!module || !tool) {
@@ -193,4 +303,7 @@ async function main(): Promise<void> {
await serve(); await serve();
} }
void main(); // Only when run, so a test can import the pieces.
if (process.argv[1] && import.meta.url === new URL(`file://${process.argv[1]}`).href) {
void main();
}
+258
View File
@@ -0,0 +1,258 @@
/**
* The mesh's tools as an MCP server (novox/hq design 25 §7, design 34).
*
* **A thin adapter and nothing more.** Every tool an agent sees is one a module answered for and one
* this account may call; the schema is the module's own; the answer is the module's own. Nothing here
* decides anything, which is why it is short — an MCP surface that reshaped arguments or summarised
* answers would be a second definition of what a tool is, and the module's code is the first.
*
* Implemented against the protocol directly rather than through a library: the surface is three
* methods and one framing, and a dependency here would be a dependency on every machine.
*
* One handler, two transports. Over stdio for a program a person starts (`mesh mcp`), over HTTP on a
* machine's loopback for the console the mesh assigns there (`mesh serve`, http.ts). The handler does
* not know which asked.
*/
import type { Broker } from "@novox/mesh-sdk/messaging";
import { callTool, seatsIn, toolKey, toolsOn, whyItFailed, type Listing, type Seats } from "./client.js";
/** The protocol version this speaks. Stated, because a host that wants another should be told so
* rather than discovering it through a shape it did not expect. */
export const PROTOCOL = "2025-03-26";
export interface Request {
jsonrpc: string;
id?: number | string | null;
method: string;
params?: Record<string, unknown>;
}
export interface Reply {
jsonrpc: "2.0";
id: Request["id"];
result?: unknown;
error?: { code: number; message: string };
}
/** How long a fetched tool list is kept before the modules are asked again. An agent asks on every
* turn; the mesh changes on the order of minutes. */
export const LISTING_KEPT_MS = 30_000;
export interface Surface {
/** Answer one request; undefined for a notification, which expects none. */
handle(request: Request): Promise<Reply | undefined>;
}
/**
* The surface over one bus connection, as one account.
*
* The tool list is fetched when first asked and kept for a short while (design 34 §3): asking every
* module on every `tools/list` would fan out on every agent turn for something nobody changed, and
* never refreshing would hide a module assigned a moment ago.
*/
export function mcpSurface(bus: Broker, who: string): Surface {
let known: { listing: Listing; at: number } | undefined;
const answer = (id: Request["id"], result: unknown): Reply => ({ jsonrpc: "2.0", id, result });
const refuse = (id: Request["id"], code: number, message: string): Reply => ({
jsonrpc: "2.0",
id,
error: { code, message },
});
const listing = async (): Promise<Listing> => {
if (!known || Date.now() - known.at > LISTING_KEPT_MS) {
known = { listing: await toolsOn(bus), at: Date.now() };
}
return known.listing;
};
// The roles the last listing knew, so `<seat>.<verb>` resolves to the seat. Fetched once if a
// call arrives before any list did; a listing that failed leaves no roles, and the name is then
// a module's, which is the right fallback for a mesh whose control plane is away.
const roles = async (): Promise<Seats | undefined> => {
try {
return seatsIn(await listing());
} catch {
return undefined;
}
};
return {
async handle(request) {
// A notification has no id and expects no answer; `initialized` is the one every host sends.
const notification = request.id === undefined || request.id === null;
switch (request.method) {
case "initialize":
return answer(request.id, {
protocolVersion: PROTOCOL,
capabilities: { tools: {} },
serverInfo: { name: "mesh", version: "1" },
// Said in the handshake, because an agent that knows whose authority it is acting under
// can say so when a call is refused — and a refusal is the one thing here that is not
// the mesh's fault or the tool's.
instructions:
`These are the tools of a Novox mesh, reached as ${who}. Every call goes to the module ` +
`that serves it; what may be called was fixed when this account was issued, so a ` +
`refusal means the account, not the tool. The list is what the running modules ` +
`answered, plus every role's tools from the mesh's records — the mesh's own verbs ` +
`(mesh-controller.status, .push, .assign …) among them; a module that did not answer ` +
`is named in the list's _meta and can still be called by <module>.<tool>.`,
});
case "notifications/initialized":
return undefined;
case "ping":
return notification ? undefined : answer(request.id, {});
case "tools/list": {
let have: Listing;
try {
have = await listing();
} catch (e) {
return refuse(request.id, -32603, whyItFailed("mesh-catalog.catalog_modules", e));
}
return answer(request.id, {
tools: have.tools.map((t) => ({
name: `${t.module}.${t.name}`,
description: t.description ?? `${t.name}, served by ${t.module}`,
// The module's own schema, passed through — with `node`, the machine to ask when
// the module runs on several (novox/hq ADR 0159); a seat's verb takes none, the
// seat's scope decides. An empty object is a tool that takes nothing, which is a
// real answer and not a missing one.
inputSchema: t.seat && t.scope !== "node" ? asSchema(t.input)
: t.seat ? withNode(asSchema(t.input), "the machine whose seat answers; required, the seat is held once per machine", true)
: withNode(asSchema(t.input)),
})),
// Silence, named (design 34 §3): the modules the catalogue holds and nothing answered
// for. Not a tool, so not in `tools`; not dropped either.
_meta: { notAnswering: have.notAnswering },
});
}
case "tools/call": {
const given = String(request.params?.name ?? "");
const args = { ...((request.params?.arguments as Record<string, unknown> | undefined) ?? {}) };
// The machine, when the caller names one, travels in the subject and never reaches the
// module's arguments (novox/hq ADR 0159) — for a module's tool. A seat's verb takes no
// machine from the console (the seat's scope decides), so a `node` among its arguments
// is the verb's own, as `push` and `assign` take one, and is handed through untouched.
const have = await listing().catch(() => undefined);
const roles = have && seatsIn(have);
const bare = given.split("@", 1)[0];
const isSeatVerb = roles ? toolKey(bare, roles).startsWith("seat:") : false;
// A node-scoped seat's verb is asked of one machine (design 33 §4, ADR 0170): `node`
// names it and travels in the subject, as for a module's tool.
const nodeScoped = isSeatVerb && (have?.tools.some((t) => t.seat && t.scope === "node" &&
`${t.module}.${t.name}` === bare) ?? false);
const takesNode = !isSeatVerb || nodeScoped;
const node = takesNode && typeof args.node === "string" && args.node !== "" ? args.node : "";
if (takesNode) delete args.node;
if (nodeScoped && !node && !given.includes("@")) {
return refuse(request.id, -32602, `${given} is a machine's seat's verb: name the machine with \`node\``);
}
const name = node && !given.includes("@") ? `${given}@${node}` : given;
try {
const { result, node: answeredBy } = await callTool(bus, name, args, roles, have);
// Text, because that is what every host renders. The content is the module's answer
// as JSON, unshaped: an adapter that flattened it would be deciding what matters in
// somebody else's answer. Which machine answered follows it as its own line.
const content: { type: string; text: string }[] = [
{ type: "text", text: JSON.stringify(result, null, 2) },
];
if (answeredBy) content.push({ type: "text", text: `answered by ${answeredBy}` });
return answer(request.id, { content });
} catch (e) {
// **An error the agent can act on, not a stack.** isError rather than a protocol
// failure, because the call was well-formed and the mesh answered it — with a refusal,
// an absence or a fault, and the words say which.
return answer(request.id, {
content: [{ type: "text", text: whyItFailed(name, e) }],
isError: true,
});
}
}
default:
return notification ? undefined : refuse(request.id, -32601, `mesh's MCP surface has no ${request.method}`);
}
},
};
}
/**
* A module's declared input as a JSON schema an agent can read.
*
* The sdk keeps a tool's input opaque, and the catalogue's modules write it as a bare map of
* property to description — `{ module: { type, description } }` — which is the `properties` of a
* schema rather than a schema. Wrapped here when that is what arrived; passed through when a module
* already wrote a schema; an empty object when it declared nothing. The module's words are kept
* either way.
*/
export function asSchema(input: unknown): Record<string, unknown> {
if (!input || typeof input !== "object" || Array.isArray(input)) {
return { type: "object", properties: {} };
}
const given = input as Record<string, unknown>;
if (given.type === "object" || "properties" in given) return given;
if (Object.keys(given).length === 0) return { type: "object", properties: {} };
return { type: "object", properties: given };
}
/**
* Serve over stdio until stdin closes, which is how a host ends a session.
*/
export async function serveMcp(bus: Broker, who: string): Promise<void> {
const surface = mcpSurface(bus, who);
const say = (message: unknown) => {
process.stdout.write(`${JSON.stringify(message)}\n`);
};
for await (const line of lines()) {
let request: Request;
try {
request = JSON.parse(line) as Request;
} catch {
// Unparseable, and with no id there is nobody to tell. Skipped rather than answered, because a
// reply to a request that was never framed is noise on the same channel.
continue;
}
const reply = await surface.handle(request);
if (reply) say(reply);
}
}
/** stdin as newline-framed messages, which is what MCP over stdio is. */
async function* lines(): AsyncGenerator<string> {
let buffered = "";
for await (const chunk of process.stdin) {
buffered += (chunk as Buffer).toString("utf8");
let at: number;
while ((at = buffered.indexOf("\n")) >= 0) {
const line = buffered.slice(0, at).trim();
buffered = buffered.slice(at + 1);
if (line !== "") yield line;
}
}
if (buffered.trim() !== "") yield buffered.trim();
}
/** Every module tool takes an optional `node`: the machine to ask when the module runs on several
* (novox/hq ADR 0159). Added to the listing, stripped before the call, never seen by the module. */
function withNode(schema: Record<string, unknown>, description?: string, required = false): Record<string, unknown> {
const properties = { ...((schema.properties as Record<string, unknown> | undefined) ?? {}) };
if (!("node" in properties)) {
properties.node = {
type: "string",
description: description ??
"the machine to ask, when this module runs on several; else whichever answers, and the answer says which",
};
}
const out: Record<string, unknown> = { ...schema, type: "object", properties };
if (required) {
const have = Array.isArray(schema.required) ? (schema.required as string[]) : [];
out.required = have.includes("node") ? have : [...have, "node"];
}
return out;
}
+278
View File
@@ -0,0 +1,278 @@
#!/usr/bin/env node
/**
* `mesh` — the mesh's tools, for whoever is on a machine (novox/hq design 25 §7, design 34).
*
* Four verbs and nothing else. What tools are there, call one, serve the same two to an agent over
* stdio, and serve them on a machine's loopback as the console the mesh assigns. Deliberately thin:
* everything that could be a decision is one the mesh already made, and a client that grew opinions
* would be a second place the mesh's behaviour is defined.
*
* mesh tools what the running modules answer
* mesh call <module>.<tool> [json] call one, arguments as JSON on the command line or on stdin
* mesh mcp the same two, as an MCP server over stdio
* mesh serve [--listen host:port] the console: MCP over HTTP on this machine's loopback
*
* Who it speaks as, in order of preference:
* MESH_BROKER_FILE the module credential the mesh delivered — the console's (ADR 0152)
* MESH_CREDENTIAL / --credential <file> a person's, as `operator issue` printed it (design 25 §7)
* MESH_CONSOLE / --console <url> no credential: `tools` and `call` go through a console
* already running on this machine, over loopback HTTP
*/
import { readFile } from "node:fs/promises";
import type { Broker } from "@novox/mesh-sdk/messaging";
import {
callTool,
connectAs,
connectAsTheConsole,
credentialFrom,
seatsIn,
toolsOn,
whyItFailed,
type Listing,
} from "./client.js";
import { serveMcpHttp } from "./http.js";
import { serveMcp } from "./mcp.js";
const usage = `mesh tools
mesh call <module>.<tool> [json]
mesh mcp
mesh serve [--listen host:port]
--credential <file> a person's credential, the JSON \`operator issue\` printed; default $MESH_CREDENTIAL
--console <url> a console on this machine to ask through instead; default $MESH_CONSOLE
--listen <host:port> where \`serve\` listens; loopback only; default $MESH_CONSOLE_LISTEN or 127.0.0.1:4270
MESH_BROKER_FILE the module credential the mesh delivered, which \`serve\` runs on`;
/** What the console listens on when nothing says otherwise. */
const DEFAULT_LISTEN = "127.0.0.1:4270";
async function main(argv: string[]): Promise<number> {
const args = [...argv];
let credentialPath = process.env.MESH_CREDENTIAL ?? "";
let consoleUrl = process.env.MESH_CONSOLE ?? "";
let listen = process.env.MESH_CONSOLE_LISTEN ?? DEFAULT_LISTEN;
for (let i = 0; i < args.length; i++) {
const take = () => {
const v = args[i + 1] ?? "";
args.splice(i, 2);
i--;
return v;
};
if (args[i] === "--credential") credentialPath = take();
else if (args[i] === "--console") consoleUrl = take();
else if (args[i] === "--listen") listen = take();
}
const verb = args.shift();
if (!verb || verb === "help" || verb === "--help") {
console.log(usage);
return verb ? 0 : 1;
}
// Through a console already on this machine: no credential to hold, which is the point of one.
if (consoleUrl && (verb === "tools" || verb === "call")) {
return verb === "tools" ? listingVia(consoleUrl) : callingVia(consoleUrl, args);
}
const { bus, who } = await connecting(credentialPath, verb);
try {
switch (verb) {
case "tools":
return await listing(bus, who);
case "call":
return await calling(bus, args);
case "mcp":
// Serves until stdin closes, which is how an MCP host ends a session.
await serveMcp(bus, who);
return 0;
case "serve": {
const up = await serveMcpHttp(bus, who, listen);
console.log(`mesh console listening on http://${up.address}/mcp as ${who}`);
await new Promise<void>((resolve) => {
process.once("SIGTERM", () => resolve());
process.once("SIGINT", () => resolve());
});
await up.close();
return 0;
}
default:
console.error(`mesh has no "${verb}".\n\n${usage}`);
return 1;
}
} finally {
await bus.close();
}
}
/**
* Who this process is on the bus. The module credential first: a console is started by the mesh with
* MESH_BROKER_FILE and nothing else, and must not fall back to a person's file lying around.
*/
async function connecting(credentialPath: string, verb: string): Promise<{ bus: Broker; who: string }> {
const delivered = process.env.MESH_BROKER_FILE;
if (delivered) {
return connectAsTheConsole(delivered);
}
if (!credentialPath) {
throw new Error(
verb === "serve"
? "no credential: the console runs on MESH_BROKER_FILE, the module credential the mesh " +
"delivered; to run it by hand, pass --credential <file> with a person's credential"
: "no credential: set MESH_CREDENTIAL or pass --credential <file> (the JSON `operator issue` " +
"printed, saved verbatim), or --console <url> to ask through a console on this machine",
);
}
const held = await credentialFrom(credentialPath);
return { bus: await connectAs(held), who: held.person ?? held.user ?? "somebody" };
}
function printListing(have: Listing, who?: string): void {
if (have.tools.length === 0) {
console.log("no running module answered with any tool");
}
// **What the modules answered, not what this account may call.** The two differ and the
// difference is the point: an account seeing only its own tools cannot tell "not installed" from
// "not yours", and those need different people to fix them.
for (const t of have.tools) {
const name = `${t.module}.${t.name}`;
const line = t.description ? `${name.padEnd(36)} ${t.description}` : name;
console.log(t.seat ? `${line} (a role's tool: answered by whoever holds the ${t.module} seat)` : line);
}
if (have.notAnswering.length > 0) {
console.log(
`\nheld by the mesh and not answering: ${have.notAnswering.join(", ")} — not assigned, not up, ` +
"or built before the runtime answered `tools`; each can still be called by name",
);
}
if (who) {
console.log(`\nasked as ${who}; what ${who} may call was fixed when the account was issued.`);
}
}
async function listing(bus: Broker, who?: string): Promise<number> {
let have: Listing;
try {
have = await toolsOn(bus);
} catch (e) {
console.error(whyItFailed("mesh-catalog.catalog_modules", e));
return 1;
}
printListing(have, who);
return 0;
}
async function calling(bus: Broker, args: string[]): Promise<number> {
const key = args.shift();
if (!key) {
console.error("mesh call <module>.<tool> [json]");
return 1;
}
const parsed = await argumentsFrom(args);
if (parsed === undefined) return 1;
try {
// `<seat>.<verb>` reaches the role when the mesh lists that verb for the seat; `seat:` says so
// outright and asks nothing first.
const have = key.startsWith("seat:") ? undefined : await toolsOn(bus).catch(() => undefined);
const { result, node } = await callTool(bus, key, parsed, have && seatsIn(have), have);
console.log(JSON.stringify(result, null, 2));
if (node) console.error(`answered by ${node}`);
return 0;
} catch (e) {
console.error(whyItFailed(key, e));
return 1;
}
}
/** The console's answer to one MCP request, over loopback HTTP. */
async function viaConsole(consoleUrl: string, method: string, params?: unknown): Promise<any> {
const endpoint = consoleUrl.endsWith("/mcp") ? consoleUrl : `${consoleUrl.replace(/\/$/, "")}/mcp`;
const res = await fetch(endpoint, {
method: "POST",
headers: { "content-type": "application/json", accept: "application/json" },
body: JSON.stringify({ jsonrpc: "2.0", id: 1, method, params }),
});
if (!res.ok) {
throw new Error(`the console at ${endpoint} answered ${res.status}`);
}
const reply = (await res.json()) as { result?: any; error?: { message: string } };
if (reply.error) throw new Error(reply.error.message);
return reply.result;
}
async function listingVia(consoleUrl: string): Promise<number> {
try {
const result = await viaConsole(consoleUrl, "tools/list");
const have: Listing = {
tools: (result.tools ?? []).map((t: { name: string; description?: string; inputSchema?: unknown }) => {
const at = t.name.indexOf(".");
return { module: t.name.slice(0, at), name: t.name.slice(at + 1), description: t.description, input: t.inputSchema };
}),
notAnswering: result._meta?.notAnswering ?? [],
};
printListing(have);
return 0;
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
return 1;
}
}
async function callingVia(consoleUrl: string, args: string[]): Promise<number> {
const key = args.shift();
if (!key) {
console.error("mesh call <module>.<tool> [json]");
return 1;
}
const parsed = await argumentsFrom(args);
if (parsed === undefined) return 1;
try {
const result = await viaConsole(consoleUrl, "tools/call", { name: key, arguments: parsed });
const text = result?.content?.[0]?.text ?? JSON.stringify(result);
if (result?.isError) {
console.error(text);
return 1;
}
console.log(text);
return 0;
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
return 1;
}
}
/** A call's arguments: JSON on the command line, else on stdin, else nothing. Undefined when what
* was given is not JSON, after saying so. */
async function argumentsFrom(args: string[]): Promise<unknown> {
const raw = args.length > 0 ? args.join(" ") : await maybeStdin();
if (raw.trim() === "") return {};
try {
return JSON.parse(raw);
} catch (e) {
console.error(`the arguments are not JSON: ${(e as Error).message}`);
return undefined;
}
}
/** Arguments on stdin, for a call whose JSON is too long or too quoted to type. Empty when stdin is a
* terminal, so `mesh call x.y` with no arguments does not hang waiting for something nobody is
* typing. */
async function maybeStdin(): Promise<string> {
if (process.stdin.isTTY) return "";
const chunks: Buffer[] = [];
for await (const chunk of process.stdin) chunks.push(chunk as Buffer);
return Buffer.concat(chunks).toString("utf8");
}
// Only when run, so a test can import the pieces.
if (process.argv[1] && import.meta.url === new URL(`file://${process.argv[1]}`).href) {
main(process.argv.slice(2))
.then((code) => process.exit(code))
.catch((e) => {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
});
}
export { main, usage };
export const _readFile = readFile;
+384
View File
@@ -0,0 +1,384 @@
// The tool runtime — the per-node process that makes the mesh's tools actually serve (novox/hq
// ADR 0175). It binds the mesh broker, loads the served modules' tool bundles (each of which calls
// registerModuleTools as it imports), and serves every module's tools on that module's subjects and
// every held seat's verbs on the seat's. Everything hard — dispatch, collection, duplicate-name
// safety — is the sdk's; this is the wrapper.
//
// One runtime, many modules. It was written for one module per process and ran that way in a
// container per module; it now serves a list, as the one process per node the host supervises,
// and the per-module shape is the list with one entry. A bundle that fails to import is named —
// in the log and in what `tools` answers for it — and the others serve.
import { pathToFileURL } from "node:url";
import { resolve } from "node:path";
import { useBroker } from "@novox/mesh-sdk/messaging";
import { collectTools, toolKey, type ToolDefinition } from "@novox/mesh-sdk/tools";
import type { Broker } from "@novox/mesh-sdk/messaging";
import { atWork, seatToolSubject, type Credential, type RuntimeBroker } from "./broker-nats.js";
import { launch, launches } from "./launch.js";
import { announce, endpointsOf, type ServedSeatVerb } from "./announce.js";
/**
* The one verb every module's runtime answers for it (novox/hq ADR 0152, design 34 §3): the
* module's tool names, descriptions and argument schemas, from the code that answers them and
* from nowhere else. Discovery asks the module, because a copy kept anywhere else drifts.
*/
export const TOOLS_VERB = "tools";
/** What `tools` answers for one module. */
export interface ToolsAnswer {
module: string;
tools: {
name: string;
description: string;
input: Readonly<Record<string, unknown>>;
/** Where this tool is answered, as the mesh issued it (ADR 0160): the module's plain subject
* first when there is one, then this machine's. A caller composes nothing. */
subjects?: string[];
}[];
/** Why this module serves nothing here, when its bundle failed to load (ADR 0175): said where
* discovery looks, so a module that is silent and one that is broken are told apart. */
failed?: string;
}
/** One module this runtime serves: its name and its compiled tool entrypoints. */
export interface ServedModule {
module: string;
/** Absolute paths to the module's compiled tool entrypoints (e.g. .../umami/tools/index.js). */
entrypoints: string[];
}
export interface RuntimeOptions {
/** The mesh broker to serve over. */
broker: Broker;
/** The modules to serve, each with its entrypoints. */
serves?: ServedModule[];
/** The credential's own module's entrypoints — the one-module form, which the per-module
* containers still use; the same as naming the credential's module in `serves`. */
moduleEntrypoints?: string[];
/** The credential the mesh delivered, for what it says about the seats this module claims
* (novox/hq ADR 0159). Absent for a runtime started by hand, which then serves no seat its
* memberships do not name. */
credential?: Credential;
/** What each served module's bundles are given (novox/hq ADR 0192): module → words, composed by
* the mesh per machine. A module absent here is given the runtime's own words and nothing more. */
envs?: ReadonlyMap<string, Readonly<Record<string, string>>>;
}
/** The variable the mesh composes every served module's environment into, as JSON (ADR 0192). Read
* once at start and removed from the process's environment, so no bundle finds another's there. */
export const TOOL_ENV = "MESH_TOOL_ENV";
/** Read and remove the composed environments from an environment (the process's, by default). */
export function takeToolEnvs(env: NodeJS.ProcessEnv = process.env): Map<string, Record<string, string>> {
const raw = env[TOOL_ENV];
delete env[TOOL_ENV];
const out = new Map<string, Record<string, string>>();
if (!raw) return out;
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
throw new Error(`${TOOL_ENV} is not JSON; the mesh composes it as {"<module>": {"<word>": "<value>"}}`);
}
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
throw new Error(`${TOOL_ENV} is not an object of modules`);
}
for (const [module, words] of Object.entries(parsed as Record<string, unknown>)) {
if (!words || typeof words !== "object" || Array.isArray(words)) {
throw new Error(`${TOOL_ENV}: ${module}'s environment is not an object of words`);
}
const own: Record<string, string> = {};
for (const [k, v] of Object.entries(words as Record<string, unknown>)) own[k] = String(v);
out.set(module, own);
}
return out;
}
/** Two environment words the mesh sets for the node's runtime and every tool reads from its
* environment: whose machine this is (novox/hq to-be 37 §3, ADR 0175). */
export const OPERATOR_ACCOUNT = "MESH_OPERATOR_ACCOUNT";
export const OPERATOR_HOME = "MESH_OPERATOR_HOME";
/** Load the modules, bind the broker, and serve. Returns a stop function that unhooks serving. */
export async function runTools(opts: RuntimeOptions): Promise<() => void> {
useBroker(() => opts.broker);
const runtime = opts.broker as RuntimeBroker;
// Whose runtime this is: the credential's module, or the connection's own when a runtime is
// started by hand without one — the broker was told its module when it connected.
const self = opts.credential?.module ?? (typeof runtime.module === "string" ? runtime.module : undefined);
// What to serve: the list, with the one-module form folded in as the credential's own entry.
const served = new Map<string, string[]>();
for (const s of opts.serves ?? []) {
served.set(s.module, [...(served.get(s.module) ?? []), ...s.entrypoints]);
}
const ownEntrypoints = opts.moduleEntrypoints ?? [];
if (opts.moduleEntrypoints?.length) {
if (!self) {
throw new Error(
"entrypoints were given with no module to serve them as: name the module (MESH_TOOL_MODULES " +
"as <module>=<entrypoint>) or connect on a credential that names one",
);
}
served.set(self, [...(served.get(self) ?? []), ...opts.moduleEntrypoints]);
}
// The operator's machine, said once so a tool's behaviour under it can be read back from the
// log. Tools read the two words from their own environment, which is this process's.
const account = process.env[OPERATOR_ACCOUNT];
if (account) {
console.log(`[mesh-tools] the operator's account here is ${account}` +
(process.env[OPERATOR_HOME] ? ` (home ${process.env[OPERATOR_HOME]})` : ""));
}
// Follow every served module's membership before loading anything, so what each is issued is
// known when its tools are bound. A module's own runtime already follows its own.
if (typeof runtime.follow === "function") {
for (const module of served.keys()) await runtime.follow(module);
}
// What each module's bundles are given: the runtime's own words, and over them the module's own.
const envs = opts.envs ?? new Map<string, Record<string, string>>();
const envFor = (module: string): NodeJS.ProcessEnv => ({ ...process.env, ...(envs.get(module) ?? {}) });
// Every bundle this runtime serves is launched as a process and spoken to over MCP on stdio (ADR
// 0188, ADR 0193): given the runtime's words and its module's own, and told the module it serves
// it as. The runtime knows no language; an entrypoint that is not executable was not built to be
// served, and is refused by name. A bundle that fails to start is named, and the others serve
// (ADR 0175: one faulty bundle must not take the node's tools down).
//
// The one-module form — the credential's own module's entrypoints, which the per-module containers
// still use for their event handlers and provisioners until they move (to-be 38 WP4c) — is imported
// into this process as before: one module, one SDK, its container's own environment.
// The machine this runtime serves, which an event a launched tool emits is stamped with.
const node = opts.credential?.node ?? (typeof (runtime as unknown as { node?: unknown }).node === "string" ? (runtime as unknown as { node: string }).node : undefined);
const failed = new Map<string, string>();
const launched: { module: string; owner: string; tools: ToolDefinition[] }[] = [];
const children: Array<() => void> = [];
for (const [module, entrypoints] of served) {
const imported = module === self && ownEntrypoints.length > 0;
for (const entry of entrypoints) {
const path = resolve(entry);
try {
if (imported) {
await import(pathToFileURL(path).href);
continue;
}
if (!launches(path)) {
throw new Error(`${path} is not executable; a bundle the runtime serves is started, never imported, and its build makes it executable (novox/hq ADR 0193)`);
}
const child = await launch(module, path, {
...envFor(module), MESH_SERVED_MODULE: module, MESH_MODULE: module,
...(node ? { MESH_NODE: node } : {}),
});
children.push(child.stop);
for (const r of child.registrations) launched.push({ ...r, owner: module });
} catch (err) {
const why = err instanceof Error ? err.message : String(err);
failed.set(module, why);
console.log(`[mesh-tools] ${module}'s bundle ${entry} failed to load: ${why}; its tools are not served here`);
}
}
}
// A registration under a served module's name is that module's tools, served on its subjects.
// One under a seat's name is the module's implementation of that seat's verbs (ADR 0159, 0160):
// served on the seat's subjects by serveClaimedSeats where some served module claims the seat,
// never as a module's tools and never listed among them. A module named like its seat (the
// catalogue is the mesh-catalog seat) registers once and is both. Anything else is said and left
// out rather than fatal — on 2026-10-01 the credential of a module that had just learned to
// implement a seat did not yet name the claim, and the whole runtime restarted for it.
const claimed = seatsClaimed(served.keys(), self, opts.credential, runtime);
const registrations = [
...collectTools().map((r) => ({ ...r, owner: self ?? r.module })),
...launched,
];
const ownRegistrations = registrations.filter(({ module, owner: by }) => {
if (served.has(module)) return true;
if (claimed.has(module)) return false;
console.log(`[mesh-tools] ${by} registers tools under "${module}", which is neither a module served here nor a seat one of them claims; not served until the mesh issues the claim`);
return false;
});
const stops: Array<() => void> = [...children];
const stop = (): void => stops.splice(0).forEach((s) => s());
// Refused before anything is bound if a module named a tool of its own `tools`: one name
// answering two things is the fault nobody can diagnose afterwards, and the runtime is the only
// place that sees both. Likewise two tools of one module under one name.
for (const { module, tools: own } of ownRegistrations) {
if (own.some((t) => t.name === TOOLS_VERB)) {
throw new Error(
`${module} names a tool "${TOOLS_VERB}", which is the verb the runtime answers for every ` +
"module with what it serves (novox/hq ADR 0152) — refused, rename it",
);
}
const seen = new Set<string>();
for (const t of own) {
if (seen.has(t.name)) throw new Error(`${module} exposes two tools named ${t.name} — refused`);
seen.add(t.name);
}
}
// Each tool on its own key, namespaced by its module (ADR 0047); where that key is answered is
// the broker's to know from the module's membership (ADR 0160). A tool runs attributed to its
// module, so what it emits lands on the module's subject and not the runtime's.
const names: string[] = [];
for (const { module, tools: own } of ownRegistrations) {
for (const t of own) {
names.push(toolKey(module, t.name));
stops.push(await opts.broker.handle(toolKey(module, t.name), (args: Record<string, unknown> | undefined) =>
atWork.run({ module }, () => t.run(args ?? {}))));
}
}
// And, for every served module, the verb that says what it serves — nothing, and why, for a
// module whose bundle failed. A module that registered nothing and did not fail is a pure-events
// module (the audit logger), whose scoped account may not declare the serve queue; it is left
// silent as it always was.
const byModule = new Map<string, ToolDefinition[]>();
for (const { module, tools: own } of ownRegistrations) {
byModule.set(module, [...(byModule.get(module) ?? []), ...own]);
}
for (const module of served.keys()) {
const own = byModule.get(module) ?? [];
const why = failed.get(module);
if (own.length === 0 && !why) continue;
const subjectsOf = (tool: string): string[] | undefined => {
const m = typeof runtime.membership === "function" ? runtime.membership(module) : undefined;
if (!m) return undefined;
const plain = m.serves.filter((s) => s.queue).map((s) => s.subject.replace("{tool}", tool));
const mine = m.serves.filter((s) => !s.queue).map((s) => s.subject.replace("{tool}", tool));
return [...plain, ...mine];
};
stops.push(await opts.broker.handle(toolKey(module, TOOLS_VERB), async (): Promise<ToolsAnswer> => ({
module,
tools: own.map((t) => ({ name: t.name, description: t.description, input: t.input, subjects: subjectsOf(t.name) })),
...(why ? { failed: why } : {}),
})));
}
console.log(`[mesh-tools] serving ${names.length} tool(s) for ${served.size} module(s): ${names.join(", ") || "(none)"}` +
(failed.size ? `; not serving ${[...failed.keys()].join(", ")}, whose bundle(s) failed to load` : ""));
const seats = await serveClaimedSeats(runtime, [...served.keys()], self, opts.credential, registrations);
stops.push(seats.stop);
// **What it serves, it announces** (novox/hq ADR 0197), asked at the moment of the request.
if (typeof runtime.raw === "function") {
stops.push(announce(runtime, {
name: self ?? "runtime", id: runtime.node ?? self ?? "runtime",
description: `the tool runtime of ${self ?? "a module"}${runtime.node ? ` on ${runtime.node}` : ""}`,
metadata: runtime.node ? { node: runtime.node } : {},
}, () => endpointsOf(runtime, ownRegistrations, seats.serving())));
}
return () => stop();
}
/** The seats some served module claims: from the credential for its own module, and from every
* served module's membership (ADR 0160) — the node's runtime holds no claims of its own. */
function seatsClaimed(
modules: Iterable<string>,
self: string | undefined,
credential: Credential | undefined,
runtime: RuntimeBroker,
): Set<string> {
const out = new Set<string>();
for (const c of credential?.claims ?? []) if (credential?.module === self) out.add(c.seat);
for (const module of modules) {
const m = typeof runtime.membership === "function" ? runtime.membership(module) : undefined;
for (const s of m?.seats ?? []) out.add(s.seat);
}
return out;
}
/** One seat's verb, where its callers ask, and which served module holds the seat. */
interface SeatVerb {
seat: string;
verb: string;
subject: string;
holder: string;
}
/**
* Holding a seat means serving its tools (design 33 §3, novox/hq ADR 0159). What a served module
* claims and promises comes from its membership (ADR 0160) — and, for a module's own runtime, from
* its credential, which named the claims before memberships did. Each verb is served on the seat's
* own subject by the tool of the same name registered under the seat's name. Whether this instance
* *holds* the seat is the bus's to decide: only the holder's account may subscribe the seat's
* subjects, so a claimant that does not hold it here is refused the subscription and serves nothing
* — never a failure of its own tools.
*/
async function serveClaimedSeats(
broker: RuntimeBroker,
served: string[],
self: string | undefined,
credential: Credential | undefined,
registrations: { module: string; owner: string; tools: ToolDefinition[] }[],
): Promise<{ stop: () => void; serving: () => ServedSeatVerb[] }> {
if (typeof broker.handleSubject !== "function") return { stop: () => {}, serving: () => [] };
// A seat's verbs are the role's, not the software's (ADR 0159): implemented under the seat's
// name — `registerModuleTools("mesh-store", …)` — and never confused with the module's own tools.
const implementations = new Map<string, Map<string, (args: Record<string, unknown>) => Promise<unknown>>>();
const definitions = new Map<string, Map<string, ToolDefinition>>();
for (const { module, tools } of registrations) {
const defs = definitions.get(module) ?? new Map<string, ToolDefinition>();
for (const t of tools) defs.set(t.name, t);
definitions.set(module, defs);
}
for (const { module, owner, tools } of registrations) {
const verbs = implementations.get(module) ?? new Map<string, (args: Record<string, unknown>) => Promise<unknown>>();
for (const t of tools) verbs.set(t.name, (args) => atWork.run({ module: owner }, () => t.run(args)));
implementations.set(module, verbs);
}
/** Every verb of every seat a served module claims, where the mesh issued it. */
const wanted = (): SeatVerb[] => {
const out: SeatVerb[] = [];
const have = new Set<string>();
const add = (v: SeatVerb): void => {
if (have.has(v.subject)) return;
have.add(v.subject);
out.push(v);
};
for (const module of served) {
const m = typeof broker.membership === "function" ? broker.membership(module) : undefined;
for (const s of m?.seats ?? []) add({ seat: s.seat, verb: s.verb, subject: s.subject, holder: module });
// The credential's claims, for the module's own runtime, ONLY until the mesh issues a
// membership (novox/hq issue 218). The credential names what the module claims; the membership
// names what it holds on this machine. A seat held once for the mesh is claimed by every
// machine running the module and held by one, so once a membership exists it decides: a claim
// it leaves out is not held here, and serving it anyway announced the seat from a machine the
// bus then refused it on.
if (module !== self || m) continue;
for (const claim of credential?.claims ?? []) {
for (const verb of claim.serves ?? []) {
const subject = seatToolSubject(claim.seat, verb, claim.scope, credential?.node);
add({ seat: claim.seat, verb, subject, holder: module });
}
}
}
return out;
};
let stops: (() => void)[] = [];
let servingNow: ServedSeatVerb[] = [];
const serve = async (): Promise<void> => {
stops.forEach((s) => s());
stops = [];
servingNow = [];
for (const v of wanted()) {
const run = implementations.get(v.seat)?.get(v.verb);
const tool = definitions.get(v.seat)?.get(v.verb);
if (!run) {
console.log(`[mesh-tools] ${v.holder} claims ${v.seat} and implements no ${v.verb}, which that seat promises; not served`);
continue;
}
stops.push(await broker.handleSubject(v.subject, run));
if (tool) servingNow.push({ ...v, tool });
console.log(`[mesh-tools] serving ${v.seat}'s ${v.verb} on ${v.subject}, admitted where ${v.holder} holds the seat`);
}
};
await serve();
// A membership issued to any served module may add, move or withdraw a seat's verbs.
if (typeof broker.onMembership === "function") broker.onMembership(() => void serve());
return { stop: () => stops.forEach((s) => s()), serving: () => servingNow };
}
+69
View File
@@ -0,0 +1,69 @@
/**
* What a runtime serves, it announces (novox/hq ADR 0197): the TypeScript runtime the per-module
* containers still run answers the NATS services protocol's discovery in the same shape as the Go
* tool runtime — its module's tools on every subject issued, and the seat verbs it serves.
*
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/announce.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { fileURLToPath } from "node:url";
import { connect, StringCodec } from "nats";
import { resetTools } from "@novox/mesh-sdk/tools";
import { connectNats, membershipSubject } from "../dist/broker-nats.js";
import { runTools } from "../dist/runtime.js";
const url = process.env.MESH_TEST_NATS;
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
const sc = StringCodec();
test("the runtime answers $SRV.INFO with what it serves, in the services protocol's format", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const nc = await connect({ servers: url });
const jsm = await nc.jetstreamManager();
try {
await jsm.streams.delete("ASSIGNMENTS");
} catch {
// none yet
}
await jsm.streams.add({ name: "ASSIGNMENTS", subjects: ["mesh.assignment.>"], max_msgs_per_subject: 1, allow_direct: true } as never);
await nc.jetstream().publish(membershipSubject("anchor", "shop"), sc.encode(JSON.stringify({
node: "anchor", module: "shop",
serves: [{ subject: "mesh.mod.shop.tool.{tool}.anchor" }, { subject: "mesh.mod.shop.tool.{tool}", queue: "serve.shop" }],
emits: "mesh.mod.shop.event.{event}", tools: "mesh.mod.shop.tool.tools",
})));
const shop = await connectNats({ url, node: "anchor", module: "shop" });
let stop = () => {};
try {
stop = await runTools({ broker: shop, moduleEntrypoints: [fixture("shop-tools.mjs")] });
const msg = await nc.request("$SRV.INFO.shop.anchor", sc.encode(""), { timeout: 2000 });
const info = JSON.parse(sc.decode(msg.data)) as {
type: string; name: string; id: string; version: string;
endpoints: { name: string; subject: string; queue_group: string; metadata: Record<string, string> }[];
};
assert.equal(info.type, "io.nats.micro.v1.info_response");
assert.equal(info.name, "shop");
assert.equal(info.id, "anchor");
assert.ok(info.version);
const price = info.endpoints.filter((e) => e.metadata.tool === "price").map((e) => `${e.subject}|${e.queue_group}`).sort();
assert.deepEqual(price, ["mesh.mod.shop.tool.price.anchor|", "mesh.mod.shop.tool.price|serve.shop"]);
const one = info.endpoints.find((e) => e.metadata.tool === "price")!;
assert.equal(one.metadata.kind, "tool");
assert.equal(one.metadata.module, "shop");
assert.equal(one.metadata.node, "anchor");
assert.equal(one.metadata.interchangeable, "true");
assert.ok(JSON.parse(one.metadata.schema).type === "object");
// Ping answers with the same identity; another service's request is not answered.
const ping = JSON.parse(sc.decode((await nc.request("$SRV.PING", sc.encode(""), { timeout: 2000 })).data));
assert.equal(ping.type, "io.nats.micro.v1.ping_response");
await assert.rejects(nc.request("$SRV.INFO.somebody-else", sc.encode(""), { timeout: 300 }));
} finally {
stop();
await shop.close();
await nc.close();
resetTools();
}
});
+188
View File
@@ -0,0 +1,188 @@
/**
* A person's client, against a real bus.
*
* What is worth checking is not that a request/reply works — the runtime's own tests cover that — but
* that the two surfaces are the same thing. An agent and a person must see the same tools and get the
* same answers, or the MCP surface becomes a second definition of what a tool is.
*
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/client.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
// The built output, not the source: the client imports its siblings as `.js`, which is what ships and
// what every other file here does, and cannot be loaded as TypeScript directly. `pretest` builds.
import { connectNats } from "../dist/broker-nats.js";
import { callTool, seatsIn, toolKey, toolsOn, whyItFailed } from "../dist/client.js";
const url = process.env.MESH_TEST_NATS;
/** A mesh as discovery sees it (design 34 §3): the catalogue holding three modules, two of them up
* and answering `tools`, one held and not running. Two connections, because a person and a module
* are different users even in a test. */
async function aMeshWithTools() {
const catalogue = await connectNats({ url: url!, module: "mesh-catalog" });
const shop = await connectNats({ url: url!, module: "shop" });
await catalogue.handle("catalog_modules", async () => ({
modules: [{ module: "shop" }, { module: "mesh-catalog" }, { module: "ghost" }],
}));
await catalogue.handle("tools", async () => ({
module: "mesh-catalog",
tools: [{ name: "catalog_modules", description: "what modules the mesh has", input: {} }],
}));
await shop.handle("tools", async () => ({
module: "shop",
tools: [{ name: "price", description: "what something costs", input: { type: "object" } }],
}));
await shop.handle("price", async (body: { of?: string }) => ({ of: body.of ?? "nothing", cost: 12 }));
// And the mesh's own records, served by the holder of the mesh-controller seat (ADR 0154): one
// role's tool, and the verb that lists every role's.
const controller = await connectNats({ url: url!, module: "mesh-controller" });
await controller.handle("seat:mesh-controller.tools", async () => ({
seats: [
{ seat: "mesh-controller", scope: "mesh", tools: [{ name: "status", description: "what is wrong", input: {} }] },
{ seat: "node-dns-resolver", scope: "node", tools: [{ name: "lookup", description: "one machine's", input: {} }] },
],
}));
await controller.handle("seat:mesh-controller.status", async () => ({ output: "all quiet", ok: true }));
return {
async close() {
await catalogue.close();
await shop.close();
await controller.close();
},
};
}
test("a person sees what the running modules answer, sorted, and who did not answer", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const mesh = await aMeshWithTools();
const person = await connectNats({ url, module: "person.ada" });
try {
const began = Date.now();
const have = await toolsOn(person);
assert.deepEqual(
have.tools.map((x) => `${x.module}.${x.name}`),
["mesh-catalog.catalog_modules", "mesh-controller.status", "node-dns-resolver.lookup", "shop.price"],
"the list is what the modules answered plus every role's tools, in a stable order",
);
// A role's tool is marked as one; a node-scoped seat's carries its scope, so a caller names
// the machine and the verb resolves as the seat's (design 33 §4, ADR 0170).
assert.ok(have.tools.find((x) => x.module === "mesh-controller")!.seat);
const lookup = have.tools.find((x) => x.module === "node-dns-resolver")!;
assert.ok(lookup.seat && lookup.scope === "node");
assert.equal(toolKey("node-dns-resolver.lookup", seatsIn(have)), "seat:node-dns-resolver.lookup");
// Silence is named, never dropped: a module the catalogue holds and nothing answered for.
assert.deepEqual(have.notAnswering, ["ghost"]);
// And at once: a module that is not running costs nothing, or the list is unusable.
assert.ok(Date.now() - began < 5_000, "an absent module waited out the timeout");
} finally {
await person.close();
await mesh.close();
}
});
test("a person calls a tool and gets the module's own answer, unshaped", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const mesh = await aMeshWithTools();
const person = await connectNats({ url, module: "person.ada" });
try {
const answer = await callTool(person, "shop.price", { of: "a hat" });
assert.deepEqual(answer.result, { of: "a hat", cost: 12 });
} finally {
await person.close();
await mesh.close();
}
});
test("a tool nobody serves says so at once, and says what to do about it", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const person = await connectNats({ url: url!, module: "person.ada" });
try {
const began = Date.now();
await assert.rejects(() => callTool(person, "ghost.missing", {}));
// At once, not after the whole wait: "that module is down" and "that tool is slow" need
// different things done, and a timeout cannot tell them apart.
assert.ok(Date.now() - began < 5_000, "a tool nobody serves waited out the timeout");
} finally {
await person.close();
}
});
test("a name that is not <module>.<tool> is refused before anything is sent", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const person = await connectNats({ url: url!, module: "person.ada" });
try {
await assert.rejects(() => callTool(person, "price", {}), /does not name a tool/);
} finally {
await person.close();
}
});
test("each way a call fails says what to do about it", () => {
// The three answers a person actually gets. Without this they are one timeout and a stack trace,
// and the remedies are in three different places.
assert.match(whyItFailed("shop.price", new Error("no responders")), /nothing serves shop\.price/);
assert.match(
whyItFailed("shop.price", new Error("Permissions Violation for Publish")),
/may not call shop\.price/,
);
assert.match(whyItFailed("shop.price", new Error("timeout")), /did not answer in time/);
assert.match(whyItFailed("shop.price", new Error("something else")), /something else/);
});
test("a role's tool is reached through the seat, and a module's own name is never shadowed", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const { seatsIn, toolKey } = await import("../dist/client.js");
const mesh = await aMeshWithTools();
const person = await connectNats({ url, module: "person.ada" });
try {
const roles = seatsIn(await toolsOn(person));
assert.equal(toolKey("mesh-controller.status", roles), "seat:mesh-controller.status");
assert.equal(toolKey("mesh-controller.other", roles), "mesh-controller.other", "a verb the seat does not declare is a module's");
assert.equal(toolKey("shop.price", roles), "shop.price");
const answer = await callTool(person, "mesh-controller.status", {}, roles);
assert.deepEqual(answer.result, { output: "all quiet", ok: true });
const direct = await callTool(person, "seat:mesh-controller.status", {});
assert.deepEqual(direct.result, { output: "all quiet", ok: true });
} finally {
await person.close();
await mesh.close();
}
});
// A module on two machines (novox/hq ADR 0159): a call names the machine and reaches that instance
// and no other; a call that names none reaches one of them and says which; a seat's verb is served
// by the claimant's tool of the same name on the seat's own subject.
test("a tool call names the machine it is for, and every answer says which machine answered", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const onAnchor = await connectNats({ url: url!, module: "store", node: "anchor" });
const onHome = await connectNats({ url: url!, module: "store", node: "home-server" });
const asker = await connectNats({ url: url!, module: "console", node: "workstation" });
try {
await onAnchor.handle("databases", async () => ({ at: "anchor" }));
await onHome.handle("databases", async () => ({ at: "home-server" }));
const home = await callTool(asker, "store.databases@home-server", {});
assert.deepEqual(home.result, { at: "home-server" });
assert.equal(home.node, "home-server");
const anchor = await callTool(asker, "store.databases@anchor", {});
assert.deepEqual(anchor.result, { at: "anchor" });
assert.equal(anchor.node, "anchor");
const whichever = await callTool(asker, "store.databases", {});
assert.ok(["anchor", "home-server"].includes(whichever.node ?? ""), `an unnamed call still says who answered: ${whichever.node}`);
assert.deepEqual(whichever.result, { at: whichever.node });
// A seat's verb, served by the claimant on the seat's subject; a mesh seat's is flat.
await onAnchor.handleSubject("mesh.seat.mesh-store.tool.databases", async () => ({ seat: "mesh-store", at: "anchor" }));
const viaSeat = await callTool(asker, "seat:mesh-store.databases", {});
assert.deepEqual(viaSeat.result, { seat: "mesh-store", at: "anchor" });
assert.equal(viaSeat.node, "anchor");
} finally {
await onAnchor.close();
await onHome.close();
await asker.close();
}
});
+54
View File
@@ -0,0 +1,54 @@
// The TypeScript implementation, held to the shared fixtures (novox/hq ADR 0074, design 19).
//
// Run against a NATS server, because the question is what actually reaches the wire:
//
// docker run -d --rm --name c -p 14222:4222 nats:2.10-alpine -js
// node test/conformance.mjs
//
// **The runner lives with the implementation it exercises; the fixture does not.** It is read
// from the sdk's conformance directory by sibling path — the same file the Go suite reads. A
// fixture copied into each implementation is two fixtures, and two fixtures drift, which is the
// failure the suite exists to prevent.
import { readFileSync } from "node:fs";
import { connect } from "nats";
const clientPath = process.argv[2] ?? "../dist/broker-nats.js";
const { connectNats } = await import(clientPath);
const f = JSON.parse(readFileSync(new URL("../../mesh-sdk/conformance/events/module-event.json", import.meta.url)));
const URL_ = process.env.MESH_TEST_NATS ?? "nats://127.0.0.1:14222";
let failed = 0;
const check = (ok, what) => { console.log(` ${ok ? "ok " : "FAIL"} ${what}`); if (!ok) failed++; };
const admin = await connect({ servers: URL_ });
const jsm = await admin.jetstreamManager();
await jsm.streams.add({ name: "EVENTS", subjects: ["mesh.mod.*.event.>", "mesh.seat.*.event.>"] });
const shop = await connectNats({ url: URL_, node: f.given.node, module: f.given.module });
await shop.publish({ key: f.given.key, node: f.given.node, body: f.given.body, headers: f.given.headers });
// What actually landed, read back from the stream rather than from the client that wrote it.
const msg = await jsm.streams.getMessage("EVENTS", { last_by_subj: f.wire.subject });
check(!!msg, `it lands on ${f.wire.subject}`);
if (msg) {
const got = {};
if (msg.header) for (const k of msg.header.keys()) got[k] = msg.header.get(k);
for (const h of f.wire.requiredHeaders) {
check(got[h] !== undefined && got[h] !== "", `${h} is set`);
}
check(got["content-type"] === f.wire.headerFormats["content-type"], "content-type is as pinned");
check(new RegExp(f.wire.headerFormats["x-event-id"]).test(got["x-event-id"]), "x-event-id is as pinned");
check(!Number.isNaN(Date.parse(got["x-time"])), "x-time parses as a date");
check(got["x-source"] === f.given.module, "x-source agrees with the subject's module");
const payload = JSON.parse(new TextDecoder().decode(msg.data));
check(payload.envelope === undefined && payload.key === undefined,
"the payload is the body alone, not the envelope (the fixture refuses nesting)");
check(JSON.stringify(payload) === JSON.stringify(f.given.body),
"the body round-trips — semantic, not byte-exact, per the README");
}
await shop.close(); await admin.close();
console.log(failed ? `\n${failed} failed` : "\nall passed");
process.exit(failed ? 1 : 0);
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env python3
# A tools bundle that says who the runtime said called it (novox/hq issue 365): MCP over stdio, no SDK. Its tool
# and its seat verb answer the caller the call's `_meta` named — "" when it named none — and the arguments as given.
import json, sys
def say(m):
m["jsonrpc"] = "2.0"; sys.stdout.write(json.dumps(m) + "\n"); sys.stdout.flush()
TOOLS = [
{"name": "who", "description": "who called", "inputSchema": {"type": "object", "properties": {}}},
{"name": "node-desk.who", "description": "who called the seat's verb", "inputSchema": {"type": "object", "properties": {}}},
]
for line in sys.stdin:
req = json.loads(line); rid = req.get("id"); m = req.get("method"); p = req.get("params") or {}
if m == "initialize":
say({"id": rid, "result": {"protocolVersion": "2025-03-26", "capabilities": {"tools": {}}, "serverInfo": {"name": "ledger", "version": "1"}}})
elif m == "tools/list":
say({"id": rid, "result": {"tools": TOOLS}})
elif m == "tools/call":
meta = p.get("_meta") or {}
said = {"caller": meta.get("mesh/caller", ""), "arguments": p.get("arguments") or {}}
say({"id": rid, "result": {"content": [{"type": "text", "text": json.dumps(said)}]}})
elif rid is not None:
say({"id": rid, "result": {}})
+6
View File
@@ -0,0 +1,6 @@
// A module naming a tool after the verb the runtime answers for every module — refused at load.
import { registerModuleTools } from "@novox/mesh-sdk/tools";
registerModuleTools("clash", () => [
{ name: "tools", description: "mine, not the runtime's", input: {}, run: async () => ({}) },
]);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for clash-tools.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./clash-tools.mjs");
await serveRegisteredOverStdio();
+4
View File
@@ -0,0 +1,4 @@
import { registerModuleTools } from "@novox/mesh-sdk/tools";
registerModuleTools("delta", (env) => [
{ name: "given", description: "what delta was given", input: {}, run: async () => ({ mine: env.DELTA_TOKEN_FILE ?? null, theirs: env.GAMMA_CONFIG_FILE ?? null }) },
]);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for env-delta.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./env-delta.mjs");
await serveRegisteredOverStdio();
+5
View File
@@ -0,0 +1,5 @@
// A bundle that reads what it was given (novox/hq ADR 0192): its contributor's environment.
import { registerModuleTools } from "@novox/mesh-sdk/tools";
registerModuleTools("gamma", (env) => [
{ name: "given", description: "what gamma was given", input: {}, run: async () => ({ mine: env.GAMMA_CONFIG_FILE ?? null, theirs: env.DELTA_TOKEN_FILE ?? null, runtime: env.MESH_OPERATOR_ACCOUNT ?? null, composed: env.MESH_TOOL_ENV ?? null }) },
]);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for env-gamma.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./env-gamma.mjs");
await serveRegisteredOverStdio();
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env node
// Launched (ADR 0188): its environment is the child's own.
import { serveStdio } from "@novox/mesh-sdk/stdio";
await serveStdio("zeta", [
{ name: "given", description: "what zeta was given", input: {}, run: async () => ({ mine: process.env.ZETA_URL ?? null, theirs: process.env.GAMMA_CONFIG_FILE ?? null, composed: process.env.MESH_TOOL_ENV ?? null }) },
]);
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env node
// A long-running bundle that keeps exiting (novox/hq ADR 0198): the runtime starts it again each time.
import { appendFileSync } from "node:fs";
import { serveStdio } from "@novox/mesh-sdk/stdio";
appendFileSync(process.env.FLAKY_LOG, "started\n");
setTimeout(() => process.exit(3), 300);
await serveStdio("flaky", []);
+9
View File
@@ -0,0 +1,9 @@
// One of several bundles the node's runtime loads (novox/hq ADR 0175): a module with two tools.
import { registerModuleTools } from "@novox/mesh-sdk/tools";
import { emit } from "@novox/mesh-sdk/events";
registerModuleTools("alpha", () => [
{ name: "one", description: "alpha's first", input: {}, run: async () => ({ alpha: 1 }) },
// A tool that emits: the event must land on alpha's subject, not the runtime's.
{ name: "two", description: "alpha's second, which emits", input: {}, run: async () => { await emit("happened", { by: "alpha" }); return { alpha: 2 }; } },
]);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for many-alpha.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./many-alpha.mjs");
await serveRegisteredOverStdio();
+14
View File
@@ -0,0 +1,14 @@
// One of several bundles the node's runtime loads (novox/hq ADR 0175): a module with three tools of
// its own and the implementation of a node seat's two verbs under the seat's name (ADR 0159).
import { registerModuleTools } from "@novox/mesh-sdk/tools";
registerModuleTools("beta", () => [
{ name: "three", description: "beta's", input: { verbose: { type: "boolean", description: "say more" } }, run: async () => ({ beta: 3 }) },
{ name: "four", description: "beta's", input: {}, run: async () => ({ beta: 4 }) },
{ name: "five", description: "beta's", input: {}, run: async () => ({ beta: 5 }) },
]);
registerModuleTools("node-shelf", () => [
{ name: "list", description: "what is on the shelf", input: {}, run: async () => ({ shelf: ["a", "b"] }) },
{ name: "clear", description: "take it all off", input: {}, run: async () => ({ cleared: true }) },
]);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for many-beta.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./many-beta.mjs");
await serveRegisteredOverStdio();
+3
View File
@@ -0,0 +1,3 @@
// A bundle that throws on import — the fault ADR 0175 names as what got harder: one module's bundle
// must not take the node's other tools down.
throw new Error("gamma's bundle cannot find its client");
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for many-broken.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./many-broken.mjs");
await serveRegisteredOverStdio();
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env python3
# A tools bundle in a second language (novox/hq ADR 0188): MCP over stdio, no SDK, no dependencies.
# One tool of its own, one seat verb, and one that exits the process mid-call.
import json, sys
def say(m):
m["jsonrpc"] = "2.0"; sys.stdout.write(json.dumps(m) + "\n"); sys.stdout.flush()
TOOLS = [
{"name": "greet", "description": "say hello", "inputSchema": {"type": "object", "properties": {"who": {"type": "string"}}}},
{"name": "node-lamp.on", "description": "the seat's verb", "inputSchema": {"type": "object", "properties": {}}},
{"name": "die", "description": "exit without answering", "inputSchema": {"type": "object", "properties": {}}},
]
for line in sys.stdin:
req = json.loads(line); rid = req.get("id"); m = req.get("method"); p = req.get("params") or {}
if m == "initialize":
say({"id": rid, "result": {"protocolVersion": "2025-03-26", "capabilities": {"tools": {}}, "serverInfo": {"name": "delta", "version": "1"}}})
elif m == "tools/list":
say({"id": rid, "result": {"tools": TOOLS}})
elif m == "tools/call":
name = p.get("name"); args = p.get("arguments") or {}
if name == "greet":
say({"id": rid, "result": {"content": [{"type": "text", "text": json.dumps({"greeting": "hello " + args.get("who", "world"), "language": "python"})}]}})
elif name == "node-lamp.on":
say({"id": rid, "result": {"content": [{"type": "text", "text": json.dumps({"on": True, "language": "python"})}]}})
elif name == "die":
print("delta: told to die", file=sys.stderr); sys.exit(3)
else:
say({"id": rid, "error": {"code": -32602, "message": "no such tool"}})
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env node
// A TypeScript bundle written against the protocol and marked executable: served through the
// launcher like any other language, with the in-process shortcut off (novox/hq ADR 0188).
import { serveStdio } from "@novox/mesh-sdk/stdio";
await serveStdio("epsilon", [
{ name: "seven", description: "epsilon's", input: {}, run: async () => ({ epsilon: 7, via: "stdio" }) },
]);
+10
View File
@@ -0,0 +1,10 @@
# A bus that refuses one subject: what the mesh's grants do to a subject they do not name (issue 217).
authorization {
users = [
{ user: "runtime", password: "runtime", permissions: {
publish: { allow: [">"] }
subscribe: { allow: [">"], deny: ["$SRV.PING.>", "mesh.seat.held-elsewhere.>"] }
} }
]
}
jetstream: enabled
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env node
// A launched bundle registering its seat before its own tools, served through the SDK's loop
// (novox/hq ADR 0193): which is the seat's and which the module's comes from the name it is given.
import { registerModuleTools } from "@novox/mesh-sdk/tools";
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
registerModuleTools("node-shelf", () => [{ name: "list", description: "the shelf", input: {}, run: async () => ({ shelf: ["x"] }) }]);
registerModuleTools("theta", (env) => [{ name: "own", description: "theta's", input: {}, run: async () => ({ theta: env.THETA_WORD ?? null }) }]);
await serveRegisteredOverStdio();
+12
View File
@@ -0,0 +1,12 @@
// The shop again, in a file of its own: a module imported once stays imported, so a second test needs a second entrypoint.
import { registerModuleTools } from "@novox/mesh-sdk/tools";
registerModuleTools("shop", () => [
{
name: "price",
description: "what something costs",
input: { of: { type: "string", description: "the thing" } },
run: async (args) => ({ of: args.of ?? "nothing", cost: 12 }),
},
{ name: "refund", description: "give it back", input: {}, run: async () => ({ done: true }) },
]);
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env node
// The launcher the builder writes beside an entrypoint (novox/hq ADR 0193), for shop-seat.mjs.
import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";
await import("./shop-seat.mjs");
await serveRegisteredOverStdio();

Some files were not shown because too many files have changed in this diff Show More