Author SHA1 Message Date
mesh-tools 84ede95c4a Merge pull request 'L3 final: tool-runner back in node-tools' manifest (issue 774)' (#60) from feat/l3-final into main 2026-11-04 00:24:03 +00:00
jschoubben 7fd9d893c0 Give node-tools new manifest bytes (version 2), the recovery of the moment for a stale verdict
mesh/delivery merged: source.main moved to novox/mesh-tools@84ede95c4af7
mesh/delivery-group group feat/l3-final merged: every member is merged
mesh/check checking 7fd9d893c0a5 on 84ede95c4af7 | novox/mesh-tools#60 in group feat/l3-final
The running indexer holds a refusal from 21:14 for the exact bytes of this manifest, which says tool-runner again. A new digest is read afresh; the thorough fix of stale verdicts is in the indexer in this group (issue 774).

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 00:44:17 +01:00
jschoubben 0fb4591c9d Pin mesh-sdk 85ba0a9a70d8, the group's head of feat/l3-final (issue 774)
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/l3-final checking: 3 of 5 member(s) ready
mesh/check failed (2): unit: mesh-provisioner-client/cmd/mesh-provisioner-client.TestNatsANodeComingBackFil…; and 1 more | 0fb4591c9da9 on b4819dc1fb73
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:43:11 +01:00
jschoubben 7da4c55752 Put tool-runner back in node-tools' manifest (L3 final, issue 774)
The judge that knows the word runs now.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:09:34 +01:00
mesh-tools b4819dc1fb Merge pull request 'L3 recovery: take tool-runner out of node-tools' manifest until the judge that knows it runs (issue 774, issue 1585)' (#59) from fix/l3-words-after-judge into main 2026-11-03 21:09:06 +00:00
jschoubben 40e0e8282b Take tool-runner out of node-tools' manifest until the judge that knows it runs
mesh/delivery merged: source.main moved to novox/mesh-tools@b4819dc1fb73
mesh/delivery-group group fix/l3-words-after-judge merged: every member is merged
mesh/check being carried to 40e0e8282b38 on b4819dc1fb73 | from 01ee2ef39488, mesh-catalog e6de42630355, mesh-delivery 4154c3040cee
The running indexer's judge refuses the word, a refused manifest holds the bus's user list, and so the indexer build
that knows the word is never granted. The node-engine on main and node-tools find the runner by its name; the grants
part keeps the runner's users as they run while no manifest says it; the word comes back once the new judge runs
(issue 774).

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 21:28:29 +01:00
mesh-tools 01ee2ef394 Merge pull request 'L3: the tool runner says it is the tool runner, and the dead TypeScript callers of the old catalog module are gone (ADR 0604)' (#58) from feat/l3b-bus into main 2026-11-03 20:13:02 +00:00
jschoubben 8b8a760047 Pin mesh-sdk b7abe203a929, the group's head of feat/l3b-bus: it keeps, deprecated, the exported names the running parts still use
mesh/delivery merged: source.main moved to novox/mesh-tools@01ee2ef39488
mesh/delivery-group group feat/l3b-bus merged: every member is merged
mesh/check checking 8b8a7600479e on 01ee2ef39488 | novox/mesh-tools#58 in group feat/l3b-bus
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 18:50:12 +01:00
jschoubben db3dd04be7 Pin mesh-sdk cf5812101b0b, the group's final head of feat/l3b-bus (the module graph left mesh-sdk)
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/l3b-bus checking: 2 of 4 member(s) ready
mesh/check checking db3dd04be7b6 on 8c1e04dd3709 | novox/mesh-tools#58 in group feat/l3b-bus
2026-11-03 17:15:17 +01:00
jschoubben d13890b944 Pin mesh-sdk 65b7adc8bdd9, the group's head of feat/l3b-bus 2026-11-03 17:15:17 +01:00
jschoubben b3903446f9 Remove the TypeScript runtime code that called the old catalog module's tools; a test finds no file naming one (ADR 0604 section 4, issue 774) 2026-11-03 17:15:17 +01:00
jschoubben e550c87a29 Pin mesh-sdk e7ce86ff62b5, the group's head of feat/l3b-bus (review round 2) 2026-11-03 17:15:16 +01:00
jschoubben ff7661a136 Pin mesh-sdk 825483043549, the group's head of feat/l3b-bus (review round 1) 2026-11-03 17:15:16 +01:00
jschoubben 84533edf2e Pin mesh-sdk 09ec82cb3551, the group's head of feat/l3b-bus (ADR 0601) 2026-11-03 17:15:16 +01:00
jschoubben ea516ece5a node-tools says it is the tool runner (tool-runner), so mesh-sdk and its readers never know it by its name; pinned at mesh-sdk a973bfa (issue 774, the operator 2026-11-03) 2026-11-03 17:15:16 +01:00
mesh-tools 8c1e04dd37 Merge pull request 'Bus state by template: the tool runner's objects, stream, work-queue and ledger calls with the proven author (issue 1966, B1)' (#57) from feat/bus-state into main 2026-11-03 15:24:40 +00:00
jschoubben 43d3fba501 Pin the group's mesh-sdk head f8bd12e in place of the local replace (issue 1966)
mesh/delivery merged: source.main moved to novox/mesh-tools@8c1e04dd3709
mesh/delivery-group group feat/bus-state merged: every member is merged
mesh/check pass 43d3fba501d3 on b8e42cf54cf6 | unit 6460 run, 262 kept, 27 skipped | integration 6639 run, 264 kept, 35 skipped | over bound: unit, in…
2026-11-03 15:35:30 +01:00
jschoubben df929b48b5 Leave the own secret ledgers out of node-tools' manifest until the vault holds it on every machine, reading a missing file as no ledger credentials; grant the test's ledger account as the grants part does (issue 1966 T6) 2026-11-03 15:29:47 +01:00
jschoubben 961ec0ef96 Append and snapshot a ledger on mesh-sdk's ledger subjects, the ones its account is granted, refusing a binding naming others (issue 1966 T6) 2026-11-03 15:17:12 +01:00
jschoubben 5a9e997349 Check a ledger operation against the types its binding carries, refusing a ledger bound with none (issue 1966 T2) 2026-11-03 15:15:58 +01:00
jschoubben fadc73f73b Key and answer a ledger index by mesh-sdk's IndexKey and IndexEntry, the one shape the owner's watcher also writes (issue 1966 T3) 2026-11-03 15:15:00 +01:00
jschoubben d206b0dde1 Prove against the real bus that the ledger keeps its promises: one writer wins, the state never goes back, a cut or full state catches up, the snapshot rebuilds, the repairs repair, a forged author is refused (issue 1966 T7) 2026-11-03 13:52:58 +01:00
jschoubben 2bb0a6461d Publish a module's ledger history and snapshots only on its own ledger connection, read from the runner's own secret ledgers and followed without a restart (issue 1966 T6) 2026-11-03 13:51:47 +01:00
jschoubben 38a634723b Stamp a ledger operation with the caller the runner proved for the call in flight it names, refusing another module's call or one not in flight; serve the ledger calls on the pipe (issue 1966 T5) 2026-11-03 13:49:15 +01:00
jschoubben 8fe7ef3b82 Serve a ledger owner its repairs rebuild, reindex, snapshot and lag as tools, its own ledgers only, marked acts or reads (issue 1966 T4) 2026-11-03 13:46:54 +01:00
jschoubben 63c1e3faa2 Read a ledger: an instance's state, its history oldest first with sequence and author, a watch of its states, a declared index (issue 1966 T3) 2026-11-03 13:42:33 +01:00
jschoubben ad22c5e9fe Apply a ledger operation: check it against its type, append it with the last sequence expected, derive the state at the revision read, write the snapshot (issue 1966 T2) 2026-11-03 13:40:45 +01:00
jschoubben d6c516ab44 Serve a bundle its own bound objects, streams and work queues on its pipe (issue 1966 T1) 2026-11-03 13:34:32 +01:00
mesh-tools b8e42cf54c Merge pull request 'sdk1: node-tools reads per-node states through mesh-sdk's pernode' (#56) from feat/sdk1 into main 2026-11-02 20:43:23 +00:00
jschoubben 24c0ec0d32 Pin mesh-sdk at f02f142, the sdk1 head whose tests no longer leak package globals, so the group's check builds against it
mesh/delivery merged: source.main moved to novox/mesh-tools@b8e42cf54cf6
mesh/delivery-group group feat/sdk1 merged: every member is merged
mesh/check checking 24c0ec0d324a on d7d908cf67ae | novox/mesh-tools#56 in group feat/sdk1
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:14:16 +01:00
jschoubben bd24ceafa9 Pin mesh-sdk at the sdk1 group's head 1c7356c, so the group's check builds every module against the narrowed SDK it was written for
mesh/delivery-group group feat/sdk1 checking: 7 of 7 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:29:53 +01:00
jschoubben 3f538027e9 Read a per-node state through mesh-sdk's pernode package, so the runtime and a process share one copy of its rules
Replaces the runtime's own bucket listing, advisory hearing and per-bucket bookkeeping; the bound on progress and the runtime's shapes stay here. ADR 0593 §6.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:31:18 +01:00
jschoubben 8114b45884 Serve a per-node state to a bundle as one state: get by the key's node, keys and watch over every node's bucket, a later bucket too
The provisioner, the assigner and the view read every node's composition (ADR 0593 §6); the runtime followed mesh-sdk's membership.StateIssued PerNode and process.NodeFamily on its own connection.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:27:34 +01:00
mesh-tools d7d908cf67 Merge pull request 'node-tools without the controller's paging and handover reads (issue 1126 group B)' (#55) from feat/1126-removal into main 2026-11-02 13:41:43 +00:00
jschoubben e475d9f849 Pin mesh-sdk to the group head of issue 1126
mesh/delivery merged: source.main moved to novox/mesh-tools@d7d908cf67ae
mesh/delivery-group group feat/1126-removal merged: every member is merged
mesh/check checking e475d9f849ca on 49f9269376e3 | novox/mesh-tools#55 in group feat/1126-removal
Every member builds against mesh-sdk abf5eac, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:27:34 +01:00
jschoubben e6f27a5f74 Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk f569924, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:20:14 +01:00
jschoubben c678511871 Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk 729a86b, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:56:55 +01:00
jschoubben 4f636b7080 Name no controller verb in hints and drop its handover and paging reads, since the controller ended (issue 1126 B19)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:23:14 +01:00
mesh-tools 49f9269376 Merge pull request 'node-tools follows memberships from the grants part's state alone (issue 774)' (#54) from feat/l3-removal into main 2026-11-02 00:35:50 +00:00
jschoubben 29fd452628 Pin mesh-sdk at c21d8fd, the removal group's head, so every member builds the same contract (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-tools@49f9269376e3
mesh/delivery-group group feat/l3-removal merged: every member is merged
mesh/check failed: mesh-build-go: the check of novox/mesh-catalog#590: the ask mvyhumdm860c7f's step on shanks did not end within 30m0s of its start |…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:53:42 +01:00
jschoubben 331fa06b54 The tool runner reads the memberships state again until it can, instead of giving a module up at its one follow
A JetStream 503 or a timeout at the follow (a bus right after its boot) left the module with no membership for good, since
Follow runs once per module and the stream fallback is gone. The read is now tried again with a growing wait (1 s doubling
to 30 s) until it watches the key or the connection closes; the failure is said once, and its reading once it succeeds.
Tests: none added; a step added to an existing unit test (TestAModuleHearsEveryStreamItsMembershipNames), which fails
without the retry.

(issue 774, 774/S32, review g1 round 1)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:51:19 +01:00
jschoubben 5b2ff5e321 Test that node-tools' manifest is not refused by the lease write's check (ADR 0571 §13)
Tests: one added, the first of its kind in this repository; the repository holds 128 unit tests, within its budget of 500.
Built against the group's mesh-sdk through a go.work outside the tree; go.mod is not re-pinned here.

(issue 774, 774/S33)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:26:04 +01:00
jschoubben 4fbfdfad13 The tool runner reads its membership from the grants part's state alone: no stream read, no subject heard (ADR 0571 §10)
Follow no longer asks the ASSIGNMENTS stream for a copy nor subscribes mesh.assignment; a module with no membership key is
said so and served in its derived shape. The test mesh issues memberships where the grants part keeps them.
Tests: none added; the stream-fallback assertions of two existing tests are replaced by their state-only counterparts, and
the test mesh no longer makes the stream. The two integration tests of the bus were run locally against the embedded bus.

(issue 774, 774/S32)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:25:19 +01:00
mesh-tools 75943ca7b0 Merge pull request 'The tool runner waits on the self-check's checked, not the controller's (design 47 L7)' (#53) from feat/l5-l7-main into main 2026-11-01 13:54:02 +00:00
jschoubben fb6631aee1 Pin group A's mesh-sdk head on main
mesh/delivery merged: source.main moved to novox/mesh-tools@75943ca7b0c3
mesh/delivery-group group feat/l5-l7-main merged: every member is merged
mesh/check checking fb6631aee1a2 on 75943ca7b0c3 | novox/mesh-tools#53 in group feat/l5-l7-main
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:57:49 +01:00
jschoubben d41bee158c node-tools waits on the self-check's checked, not the controller's (ADR 0571 clause 13, issue 1955)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:39 +01:00
mesh-tools 110e55cb48 Merge pull request 'The tool runner declares the credential form of its mcp-endpoint (ADR 0579)' (#52) from feat/judge-once into main 2026-11-01 07:55:09 +00:00
jschoubben c494018074 Pin the group's mesh-sdk head after the gofmt fix
mesh/delivery merged: source.main moved to novox/mesh-tools@110e55cb4836
mesh/delivery-group group feat/judge-once merged: every member is merged
mesh/check checking c4940180746a on c2f01bf10b68 | novox/mesh-tools#52 in group feat/judge-once
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 03:27:07 +01:00
jschoubben 4cdf63443d Pin mesh-sdk to the group's head after review round 1
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/judge-once checking: 4 of 6 member(s) ready
mesh/check checking 4cdf63443d59 on c2f01bf10b68 | novox/mesh-tools#52 in group feat/judge-once
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 03:13:22 +01:00
jschoubben 1e9995faba Pin mesh-sdk to the delivery group's head
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:46:53 +01:00
jschoubben f6c401ecd2 Declare credential form none for the provision, which hands consumers no credential
The manifest judge refuses an offer that says no credential form (issue 1585).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:43:23 +01:00
mesh-tools c2f01bf10b Merge pull request 'Read and write placement per node and module (issue 774)' (#51) from feat/072-moves into main 2026-10-31 10:31:18 +00:00
jschoubben be2fc2e9b6 Pin mesh-sdk at the group's head acbf80a (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-tools@c2f01bf10b68
mesh/delivery-group group feat/072-moves merged: every member is merged
mesh/check failed: shanks: the check of novox/mesh-controller#487 failed: go mod download in modules/route-proxy: go: git.novox.be/novox/mesh-controll…
2026-10-31 10:25:46 +01:00
jschoubben aa05629375 Pin mesh-sdk at the group's head (issue 774) 2026-10-31 09:30:08 +01:00
jschoubben 35a7ff70fe Read the seat mesh-placement's state through the sdk's PlacementsOf, and write the test mesh's placement as the seat does, one key per module and node (issue 774, research 072 row 37) 2026-10-31 09:29:32 +01:00
jschoubben 747bfa71aa Merge pull request 'A state watch given up hands nothing more to its bundle (issue 774)' (#50) from fix/sweep-2026-10-30 into main
Reviewed-on: #50
2026-10-30 18:13:12 +00:00
jschoubben 316a5a2fc3 A watch given up or stopped hands over nothing more; arrival of current values restarts its bound (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-tools@747bfa71aa32
mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged
mesh/check checking 316a5a2fc317 on 0edcd2501dae | novox/mesh-tools#50 in group fix/sweep-2026-10-30
Quick: check the watch context before each deliver in StateWatch. Thorough (chosen, same
file kind): also LogWatch drops a message fetched as it stopped, and the bound restarts
on arrival, not only after the bundle took a value.
2026-10-30 18:22:01 +01:00
jschoubben 0edcd2501d Merge pull request 'The tool runner reads the grants part's memberships through the mesh-sdk codec (issue 774)' (#49) from fix/memberships-shape into main
Reviewed-on: #49
2026-10-30 14:42:20 +00:00
jschoubben ac645e2cae Refuse a held-back membership's old wrapper in the tool runner, pinning mesh-sdk's review-round-1 codec
mesh/delivery merged: source.main moved to novox/mesh-tools@0edcd2501dae
mesh/delivery-group group fix/memberships-shape merged: every member is merged
mesh/check checking ac645e2caedd on 3c9068f99a43 | novox/mesh-tools#49 in group fix/memberships-shape
The old grants part wrote its wrapper for a held-back membership too; the new part leaves such a key alone, and taking
it would serve what the hold kept back with the stream's copy ignored. The integration test now proves that copy is
served instead. Review round 1 of issue 774's fix.
2026-10-30 15:38:40 +01:00
jschoubben a479a90fb2 Read the grants part's memberships through mesh-sdk's codec, so the tool runner serves its node's modules
The tool runner unmarshalled the state's value straight into its membership; the grants part had written its wrapper
there, so every module decoded empty, was marked as given by the grants part, and the correct copy on the assignments
stream was passed over: novox served no tool. followGrants now reads with membership.BareGrantsMembership, which takes
the bare membership, still takes the old wrapper (what the live mesh holds until the part writes again), and refuses
a value naming no node or module, leaving the stream's copy in force.

The fixture IssueInGrants built its value by hand from this package's type and so agreed with the mistake; it now
writes through mesh-sdk's encoder, and IssueInGrantsWrapped writes the old wrapper for one integration test that
proves tools and state are served from it (it fails without this fix with the empty membership seen live).

Issue 774 (the bus login list, L3). Quick fix: unwrap here alone. Thorough, chosen: one codec in mesh-sdk the writer
and this reader share, so the two cannot drift again. Pins mesh-sdk at the head of its fix/memberships-shape.
2026-10-30 15:31:26 +01:00
mesh-tools 3c9068f99a Merge pull request 'Design 47's last milestones in the tool runner: per-key TTL, log verbs, memberships and seat holders from the bus (issues 1952, 1954, 1955)' (#48) from feat/774-l3-reland into main 2026-10-30 11:02:19 +00:00
jschoubben 250aa73a56 Pin mesh-sdk 2da0de7, the release's final head with main merged in, so mesh-tools builds with the group's one SDK (issue 1955)
mesh/delivery merged: source.main moved to novox/mesh-tools@3c9068f99a43
mesh/delivery-group group feat/774-l3-reland merged: every member is merged
mesh/check checking 250aa73a5631 on 95a12105cf7b | novox/mesh-tools#48 in group feat/774-l3-reland
2026-10-30 01:12:55 +01:00
jschoubben e32eef2379 Pin mesh-sdk 6e4a301, the release's final head, so mesh-tools builds with the group's one SDK (issue 1955) 2026-10-30 00:16:11 +01:00
jschoubben 070ae53128 Pin mesh-sdk 007b6fd, the release's final head, so mesh-tools builds with the group's one SDK (issue 1955) 2026-10-30 00:05:45 +01:00
jschoubben 8c0618cfc5 Prove a later assignments-stream membership changes nothing once the grants part's state has given one (issue 1955, ADR 0571 clause 7) 2026-10-29 23:44:19 +01:00
jschoubben 54680238a0 Read memberships from the grants part's state, what the mesh holds from the indexer and placement, and give bundles newest, since and watch on logs, so the tool runner asks the controller for nothing it keeps (issue 1955, ADR 0571 clauses 1, 4, 5, 7)
Pins mesh-sdk bb993dd. The assignments stream stays the membership's
fallback until the lease write; a log another module keeps is read as
<module>.<log>.
2026-10-29 23:42:43 +01:00
jschoubben 200249a84f Let the TypeScript mesh client take every role's tools from the console on its machine, so it asks the controller for none (issue 1955, ADR 0571 clause 5)
The Go console answers mesh/seats beside MCP's methods from mesh-sdk's seat
definitions and placement's seat-holders. The console test that runs bundles
expects the unread seat-holders line beside ghost.
2026-10-29 23:19:55 +01:00
jschoubben 67850457ef Read seat verbs' replaces from mesh-sdk's seat contract and seat holders from placement's seat-holders state, so the console asks the controller for neither (issue 1955, ADR 0571 clauses 4 and 5)
Pins mesh-sdk f9a496d. The controller's nodes and modules reads remain.
2026-10-29 23:13:42 +01:00
jschoubben 9d67b0995f Bind node-tools' consumers by mesh-provisioner's plans and provisions alone, dropping the read of the controller's assignments the sdk's helper no longer makes (issue 1954 L6-T1)
Pins mesh-sdk 692dbf9 (L6-P3).
2026-10-29 22:43:58 +01:00
jschoubben e9009b7f04 Answer mesh/state.info with a bucket's fill and caps, so a bundle's tools can say how full its state is (issue 1952) 2026-10-29 22:43:58 +01:00
jschoubben 9fc97e45c9 Let node-tools' provider helper read mesh-provisioner's provisions and plans by mesh-sdk's names, so the runtime follows ADR 0570's records off the controller (issue 1954 L6-T1)
Pins mesh-sdk 4a96793 (go/provisions). The assignments read stays while the
sdk's helper still reads plans there (until L6-P3). A watch's expire now
reads as a removed key.
2026-10-29 22:34:34 +01:00
jschoubben 1583fa85d9 Let a bundle write a state key with its own time to live and hear its end, so wakes need no in-process timer (issue 1952)
mesh/state.create and mesh/state.put take ttl_seconds (mesh-sdk 32f5ce3's
CreateTTL/RenewTTL contract); a bucket not declared key-ttl is refused,
naming it; a watch hands an expired key as op "expire". Pins mesh-sdk 32f5ce3.
2026-10-29 22:11:39 +01:00
88 changed files with 5403 additions and 4931 deletions
+4 -5
View File
@@ -23,8 +23,7 @@ marked executable — is **launched** rather than imported (novox/hq ADR 0188):
as a child with its own environment and speaks MCP over stdio to it, `tools/list` once and
`tools/call` per call. A tool it lists as `<seat>.<verb>` is the seat's implementation. A child that
exits is named in the log and started again on its next call. So a tools bundle may be written in
any language; the mesh's SDK for each is the stdio loop and nothing more (`node-tools/src/launch.ts`
is the runtime's side of it).
any language; the mesh's SDK for each is the stdio loop and nothing more (`node-tools/internal/launch` is the runtime's side of it).
Everything hard — dispatch, collection, duplicate-name safety — is the sdk's. This is the thin
wrapper that binds the bus and loads the modules. Keeping the bus client here, out of the sdk, is
@@ -45,7 +44,7 @@ MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME where thei
MESH_BROKER_URL a plain URL instead of the credential, for the bootstrap case
```
`node dist/main.js`. On a node the controller composes the variables and the host supervises the
`node dist/main.js`. On a node the mesh composes the variables and the node-engine supervises the
process like any other host-side workload (novox/hq to-be 38). As `node-tools` the same process is
the console: MCP on `127.0.0.1:4270` (or `MESH_CONSOLE_LISTEN`).
@@ -66,14 +65,14 @@ answer took, its size in bytes, how many modules and tools it announced, whether
fit the bus, when it was last heard — and who was expected and not heard).
`mesh_search` also finds a tool by the shell command it replaces (novox/hq ADR 0245): each seat verb and
module tool says what it replaces in the controller's records (`replaces`, read from the controller's
module tool says what it replaces in the indexer's records (`replaces`, read from its
`tools` and `modules` answers), so `journalctl -u x`, `systemctl status x` or `docker ps` puts the verb
for it first, and says the command it matched as `instead_of`. A word also matches through a short table
of the words agents use for the mesh's (`logs` finds the journal). Best first: a replaced command, then
words in a name or a replaced command, then words in a description; seats before modules on a tie.
Discovery asks the bus (ADR 0197): every runtime answers the NATS services protocol's `$SRV.PING` and
`$SRV.INFO` with what it serves at that moment, and the console reads where the controller's records
`$SRV.INFO` with what it serves at that moment, and the console reads where the placement seat's records
place each module. The console waits at least 750 ms, and up to 5 s for every runtime that answered
PING to send what it serves — a large runtime's answer crossing the bus to a broker on another machine
can arrive well after the rest. A runtime that said it is there and did not say what it serves, or
+1 -1
View File
@@ -3,7 +3,7 @@
The node's tool runtime as a module (novox/hq ADR 0175, to-be 38 WP3). Assigned to a machine, it is
one process the host runs from this bundle, as the operator's account: it serves every assigned
module's tools and every held seat's verbs on the bus, and answers MCP on the machine's loopback —
the console (design 34). The controller composes the process (which bundles to load, where the
the console (design 34). The mesh composes the process (which bundles to load, where the
credential is, whose machine it is); this manifest says only what the machine must have for it: the
interpreter, a place for the credential, the loopback port, and leave to call every tool.
+10 -1
View File
@@ -5,6 +5,7 @@
// wherever MESH_CONSOLE_LISTEN says — it is also the console, MCP over HTTP on loopback.
//
// MESH_BROKER_FILE the credential the mesh sealed to this machine for the runtime
// MESH_LEDGERS_FILE its own secret ledgers, each module's ledger credential; beside the broker file when unset
// MESH_TOOL_MODULES <module>=<entrypoint>,… the bundles to launch
// MESH_TOOL_ENV {"<module>": {"<word>": "<value>"}} what each is given (ADR 0192)
// MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME their home
@@ -56,7 +57,15 @@ func main() {
if err != nil {
log.Fatalf("node-tools: %v", err)
}
stop, err := runtime.Run(conn, served, envs, log.Printf)
// Its own secret ledgers: each module's ledger connection, the only one its history is published on (novox/hq ADR
// 0597 §9). Followed, so a credential minted or retired is taken with no restart.
var opts []runtime.Option
if file := runtime.LedgersFileOf(os.Getenv("MESH_BROKER_FILE")); file != "" {
ledgers := runtime.FollowLedgerCredentials(file, bus.Dial, log.Printf)
defer ledgers.Stop()
opts = append(opts, runtime.WithLedgerWriters(ledgers.Writer))
}
stop, err := runtime.Run(conn, served, envs, log.Printf, opts...)
if err != nil {
log.Fatalf("node-tools: %v", err)
}
+1 -1
View File
@@ -8,7 +8,7 @@ require (
)
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103221822-85ba0a9a70d8
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103221822-85ba0a9a70d8 h1:XEbIVImp6yGjA91rp0iZBoDdL5KfamjOuDLwWvqh0hc=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103221822-85ba0a9a70d8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+1 -1
View File
@@ -3,7 +3,7 @@
// **A machine heard and its runtime gone is a machine nobody can ask anything.** The node-engine says
// every minute that the machine is there; nothing said whether the runtime serving every module's
// tools and every held seat's verbs was. Now it says so too, on its own subject, every interval, with
// the interval — the controller's watchdog raises `tools-silent` after three missed — and its build.
// the interval — the mesh's watchdog raises `tools-silent` after three missed — and its build.
// Core NATS, kept by nobody, as the host's heartbeat: a lost one is the next one.
package alive
+398
View File
@@ -0,0 +1,398 @@
package bus
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats.go/jetstream"
)
// A module's bus state in the new form (novox/hq ADR 0595, design 25 § 12.5; issue 1966 T1): an object store, a stream,
// a work queue, a ledger, each asked for in `requires` by a local name and bound in the module's membership by the bus
// names the bus module created. **A bundle reaches only its own module's bindings**: one account per machine carries
// every module on it, so what a module may reach is decided here, from its membership, and refused with the reason
// before anything is sent, as its state is (state.go).
// Binding is the module's binding of a local name, of the storage type storage, or the reason there is none.
func (c *Conn) Binding(module, local, storage string) (membership.BusBinding, error) {
m := c.Membership(module)
if m == nil {
return membership.BusBinding{}, fmt.Errorf("%s has no membership issued on %s yet, so none of its bus state is "+
"reachable until the mesh issues one (novox/hq ADR 0595)", module, c.node)
}
var names []string
for _, b := range m.Bindings {
if b.Local != local {
names = append(names, b.Local+" ("+b.Storage+")")
continue
}
if b.Storage != storage {
return membership.BusBinding{}, fmt.Errorf("%s's %q is bound with the storage type %s, not %s (novox/hq ADR 0595)",
module, local, b.Storage, storage)
}
return b, nil
}
sort.Strings(names)
bound := "none"
if len(names) > 0 {
bound = strings.Join(names, ", ")
}
return membership.BusBinding{}, fmt.Errorf("%s was bound no %s called %q: a module reaches only the bus state its "+
"manifest asks for in `requires`, by its local name, once the mesh created it; it was bound: %s (novox/hq ADR 0595)",
module, storage, local, bound)
}
// Bindings is every binding a module's membership carries now.
func (c *Conn) Bindings(module string) []membership.BusBinding {
if m := c.Membership(module); m != nil {
return append([]membership.BusBinding(nil), m.Bindings...)
}
return nil
}
// jetStream is a JetStream handle on this connection.
func (c *Conn) jetStream() (jetstream.JetStream, error) { return jetstream.New(c.nc) }
func (c *Conn) objectStore(ctx context.Context, module, local string) (jetstream.ObjectStore, membership.BusBinding, error) {
b, err := c.Binding(module, local, catalogue.StorageObjectStore)
if err != nil {
return nil, b, err
}
js, err := c.jetStream()
if err != nil {
return nil, b, err
}
obs, err := js.ObjectStore(ctx, b.ObjectStore)
if errors.Is(err, jetstream.ErrBucketNotFound) || errors.Is(err, jetstream.ErrStreamNotFound) {
return nil, b, fmt.Errorf("%s's objects %q are bound and their store %s is not on the bus", module, local, b.ObjectStore)
}
return obs, b, err
}
func objectInfoOf(i *jetstream.ObjectInfo) buscalls.ObjectInfo {
return buscalls.ObjectInfo{Name: i.Name, Size: i.Size, Chunks: i.Chunks, Digest: i.Digest, Description: i.Description,
Modified: i.ModTime, Deleted: i.Deleted}
}
// ObjectsPut stores one object whole under its name in the module's object store.
func (c *Conn) ObjectsPut(module string, r buscalls.ObjectsPutRequest) (buscalls.ObjectInfo, error) {
if r.Name == "" {
return buscalls.ObjectInfo{}, errors.New("an object is put under a name")
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return buscalls.ObjectInfo{}, err
}
info, err := obs.PutBytes(ctx, r.Name, r.Data)
if err != nil {
return buscalls.ObjectInfo{}, err
}
if r.Description != "" {
if err := obs.UpdateMeta(ctx, r.Name, jetstream.ObjectMeta{Name: r.Name, Description: r.Description}); err != nil {
return buscalls.ObjectInfo{}, err
}
if info, err = obs.GetInfo(ctx, r.Name); err != nil {
return buscalls.ObjectInfo{}, err
}
}
return objectInfoOf(info), nil
}
// ObjectsGet is one object of the module's object store, its info and its bytes; nil when there is none.
func (c *Conn) ObjectsGet(module string, r buscalls.ObjectsNameRequest) (*buscalls.ObjectData, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return nil, err
}
res, err := obs.Get(ctx, r.Name)
if errors.Is(err, jetstream.ErrObjectNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
defer res.Close()
info, err := res.Info()
if err != nil {
return nil, err
}
data, err := obs.GetBytes(ctx, r.Name)
if err != nil {
return nil, err
}
return &buscalls.ObjectData{Info: objectInfoOf(info), Data: data}, nil
}
// ObjectsDelete deletes one object of the module's object store; one that is not there is not an error.
func (c *Conn) ObjectsDelete(module string, r buscalls.ObjectsNameRequest) error {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return err
}
if err := obs.Delete(ctx, r.Name); err != nil && !errors.Is(err, jetstream.ErrObjectNotFound) {
return err
}
return nil
}
// ObjectsList is every object the module's object store holds.
func (c *Conn) ObjectsList(module string, r buscalls.ObjectsRequest) (buscalls.ObjectsListed, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return buscalls.ObjectsListed{}, err
}
out := buscalls.ObjectsListed{Objects: []buscalls.ObjectInfo{}}
infos, err := obs.List(ctx)
if errors.Is(err, jetstream.ErrNoObjectsFound) {
return out, nil
}
if err != nil {
return out, err
}
for _, i := range infos {
out.Objects = append(out.Objects, objectInfoOf(i))
}
return out, nil
}
// ObjectsInfo is how full the module's object store is: its objects, its bytes and its cap.
func (c *Conn) ObjectsInfo(module string, r buscalls.ObjectsRequest) (buscalls.Fill, error) {
listed, err := c.ObjectsList(module, r)
if err != nil {
return buscalls.Fill{}, err
}
b, _ := c.Binding(module, r.Objects, catalogue.StorageObjectStore)
fill, err := c.streamFill("OBJ_" + b.ObjectStore)
if err != nil {
return buscalls.Fill{}, err
}
fill.Count = uint64(len(listed.Objects))
return fill.Fill, nil
}
// ObjectsWatch hands deliver every object of the module's object store as it is, then each stored or deleted.
func (c *Conn) ObjectsWatch(module string, r buscalls.ObjectsWatchRequest, deliver func(buscalls.ObjectsChange) error) (func(), error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
obs, _, err := c.objectStore(ctx, module, r.Objects)
cancel()
if err != nil {
return nil, err
}
watchCtx, stop := context.WithCancel(context.Background())
w, err := obs.Watch(watchCtx, jetstream.IncludeHistory())
if err != nil {
stop()
return nil, err
}
current := true
for current {
select {
case i, ok := <-w.Updates():
if !ok {
stop()
return nil, errors.New("the object store's watch ended before its current objects were handed over")
}
if i == nil {
current = false
continue
}
if err := deliver(buscalls.ObjectsChange{Watch: r.Watch, Object: objectInfoOf(i), Current: true}); err != nil {
c.Logf("[mesh-tools] %s did not take an object of %s: %v", module, r.Objects, err)
}
case <-time.After(StateTimeout):
_ = w.Stop()
stop()
return nil, fmt.Errorf("the object store %s did not hand its current objects in %s", r.Objects, StateTimeout)
}
}
go func() {
for i := range w.Updates() {
if i == nil {
continue
}
if err := deliver(buscalls.ObjectsChange{Watch: r.Watch, Object: objectInfoOf(i)}); err != nil {
c.Logf("[mesh-tools] %s did not take an object of %s: %v", module, r.Objects, err)
}
}
}()
return func() { _ = w.Stop(); stop() }, nil
}
// streamFill is a stream's fill and its first and last sequence, by its name on the bus.
func (c *Conn) streamFill(name string) (buscalls.StreamFilled, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := c.jetStream()
if err != nil {
return buscalls.StreamFilled{}, err
}
s, err := js.Stream(ctx, name)
if errors.Is(err, jetstream.ErrStreamNotFound) {
return buscalls.StreamFilled{}, fmt.Errorf("the stream %s is bound and not on the bus", name)
}
if err != nil {
return buscalls.StreamFilled{}, err
}
info, err := s.Info(ctx)
if err != nil {
return buscalls.StreamFilled{}, err
}
max := uint64(0)
if info.Config.MaxBytes > 0 {
max = uint64(info.Config.MaxBytes)
}
return buscalls.StreamFilled{Fill: buscalls.Fill{Count: info.State.Msgs, Bytes: info.State.Bytes, MaxBytes: max},
First: info.State.FirstSeq, Last: info.State.LastSeq}, nil
}
// streamBinding is the module's binding of a stream by its local name, a work queue among them.
func (c *Conn) streamBinding(module, local string) (membership.BusBinding, error) {
return c.Binding(module, local, catalogue.StorageStream)
}
// StreamFill is how full the module's stream is.
func (c *Conn) StreamFill(module string, r buscalls.StreamRequest) (buscalls.StreamFilled, error) {
b, err := c.streamBinding(module, r.Stream)
if err != nil {
return buscalls.StreamFilled{}, err
}
return c.streamFill(b.Stream)
}
// StreamReadMost is the most messages one read answers.
const StreamReadMost = 1000
// StreamRead is the module's stream's messages from a sequence on, Subjects narrowing them, at most Limit of them.
func (c *Conn) StreamRead(module string, r buscalls.StreamReadRequest) (buscalls.StreamPage, error) {
b, err := c.streamBinding(module, r.Stream)
if err != nil {
return buscalls.StreamPage{}, err
}
return c.readStream(b.Stream, r.Subjects, r.From, r.Limit)
}
// readStream reads a stream by its bus name from a sequence on, the messages on the subjects filter names (every
// subject when empty), at most limit of them.
func (c *Conn) readStream(name, filter string, from uint64, limit int) (buscalls.StreamPage, error) {
if limit <= 0 || limit > StreamReadMost {
limit = StreamReadMost
}
if filter == "" {
filter = ">"
}
if from == 0 {
from = 1
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := c.jetStream()
if err != nil {
return buscalls.StreamPage{}, err
}
s, err := js.Stream(ctx, name)
if errors.Is(err, jetstream.ErrStreamNotFound) {
return buscalls.StreamPage{}, fmt.Errorf("the stream %s is bound and not on the bus", name)
}
if err != nil {
return buscalls.StreamPage{}, err
}
info, err := s.Info(ctx)
if err != nil {
return buscalls.StreamPage{}, err
}
page := buscalls.StreamPage{Messages: []buscalls.StreamMessage{}, Last: info.State.LastSeq}
for seq := from; seq <= info.State.LastSeq && len(page.Messages) < limit; {
msg, err := s.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(filter))
if errors.Is(err, jetstream.ErrMsgNotFound) {
break
}
if err != nil {
return page, err
}
body := json.RawMessage(msg.Data)
if !json.Valid(msg.Data) {
body, _ = json.Marshal(string(msg.Data))
}
page.Messages = append(page.Messages, buscalls.StreamMessage{Sequence: msg.Sequence, Subject: msg.Subject,
Body: body, At: msg.Time})
seq = msg.Sequence + 1
if len(page.Messages) == limit && seq <= info.State.LastSeq {
page.Next = seq
}
}
return page, nil
}
// WorkQueueSize is how many messages wait in the module's work queue, against its cap.
func (c *Conn) WorkQueueSize(module string, r buscalls.WorkQueueRequest) (buscalls.WorkQueueSized, error) {
b, err := c.streamBinding(module, r.Queue)
if err != nil {
return buscalls.WorkQueueSized{}, err
}
fill, err := c.streamFill(b.Stream)
if err != nil {
return buscalls.WorkQueueSized{}, err
}
return buscalls.WorkQueueSized{Fill: fill.Fill}, nil
}
// WorkQueueDead is the module's work queue's dead letters: in a work queue a message is gone once settled, so one still
// kept below a consumer's acknowledgement floor was delivered as often as the consumer allows and never settled.
func (c *Conn) WorkQueueDead(module string, r buscalls.WorkQueueDeadRequest) (buscalls.WorkQueueDeadLetters, error) {
b, err := c.streamBinding(module, r.Queue)
if err != nil {
return buscalls.WorkQueueDeadLetters{}, err
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := c.jetStream()
if err != nil {
return buscalls.WorkQueueDeadLetters{}, err
}
s, err := js.Stream(ctx, b.Stream)
if err != nil {
return buscalls.WorkQueueDeadLetters{}, fmt.Errorf("the work queue %s: %w", b.Stream, err)
}
out := buscalls.WorkQueueDeadLetters{Dead: []buscalls.DeadLetter{}}
consumers := s.ListConsumers(ctx)
for ci := range consumers.Info() {
floor := ci.AckFloor.Stream
if ci.Config.MaxDeliver <= 0 || floor == 0 {
continue
}
page, err := c.readStream(b.Stream, ci.Config.FilterSubject, 1, r.Limit)
if err != nil {
return out, err
}
for _, m := range page.Messages {
if m.Sequence > floor {
break
}
out.Dead = append(out.Dead, buscalls.DeadLetter{Sequence: m.Sequence, Subject: m.Subject,
Deliveries: ci.Config.MaxDeliver, Body: m.Body, At: m.At})
}
}
if err := consumers.Err(); err != nil {
return out, err
}
if r.Limit > 0 && len(out.Dead) > r.Limit {
out.Dead = out.Dead[:r.Limit]
}
return out, nil
}
+112
View File
@@ -0,0 +1,112 @@
package bus_test
import (
"context"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// 1966/AC12: given a bundle whose membership binds objects `files`, when it puts and gets `a`, then it reads back what it
// put; a local name it was not bound is refused with the reason, and so is another module's binding and a binding of
// another storage type. A stream and a work queue bound alike answer their read-back, fill and size.
func TestABundleReachesItsOwnBoundObjectsAndStreamsOnly(t *testing.T) {
mesh := mt.New(t)
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := jetstream.New(nc)
ctx := context.Background()
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: "m_files"}); err != nil {
t.Fatal(err)
}
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "m_events", Subjects: []string{"mesh.state.m.events.>"}}); err != nil {
t.Fatal(err)
}
for _, s := range []string{"mesh.state.m.events.one", "mesh.state.m.events.two"} {
if _, err := js.Publish(ctx, s, []byte(`{"n":1}`)); err != nil {
t.Fatal(err)
}
}
m := mt.MembershipOf("m", "anchor", false, nil)
m.Bindings = []membership.BusBinding{
{Local: "files", Storage: catalogue.StorageObjectStore, Template: catalogue.TemplateObjects, ObjectStore: "m_files"},
{Local: "events", Storage: catalogue.StorageStream, Template: catalogue.TemplateEvents, Stream: "m_events",
Subject: "mesh.state.m.events"},
}
other := mt.MembershipOf("other", "anchor", false, nil)
mesh.Issue(t, m)
mesh.Issue(t, other)
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
conn.Follow("m")
conn.Follow("other")
mt.Until(t, func() error {
if conn.Membership("m") == nil || conn.Membership("other") == nil {
return errors.New("not issued yet")
}
return nil
})
if _, err := conn.ObjectsPut("m", buscalls.ObjectsPutRequest{Objects: "files", Name: "a", Data: []byte("what was put")}); err != nil {
t.Fatal(err)
}
got, err := conn.ObjectsGet("m", buscalls.ObjectsNameRequest{Objects: "files", Name: "a"})
if err != nil || got == nil || string(got.Data) != "what was put" || got.Info.Name != "a" {
t.Fatalf("read back %+v, %v", got, err)
}
if listed, err := conn.ObjectsList("m", buscalls.ObjectsRequest{Objects: "files"}); err != nil || len(listed.Objects) != 1 {
t.Fatalf("listed %+v, %v", listed, err)
}
page, err := conn.StreamRead("m", buscalls.StreamReadRequest{Stream: "events", Limit: 1})
if err != nil || len(page.Messages) != 1 || page.Next != 2 || page.Last != 2 {
t.Fatalf("read %+v, %v", page, err)
}
if fill, err := conn.StreamFill("m", buscalls.StreamRequest{Stream: "events"}); err != nil || fill.Count != 2 {
t.Fatalf("fill %+v, %v", fill, err)
}
for _, c := range []struct {
name, module, refusal string
call func(module string) error
}{
{"a local name not bound", "m", `bound no object-store called "photos"`, func(module string) error {
_, err := conn.ObjectsGet(module, buscalls.ObjectsNameRequest{Objects: "photos", Name: "a"})
return err
}},
{"another module's binding", "other", `other was bound no object-store called "files"`, func(module string) error {
_, err := conn.ObjectsGet(module, buscalls.ObjectsNameRequest{Objects: "files", Name: "a"})
return err
}},
{"a binding of another storage type", "m", `"events" is bound with the storage type stream, not object-store`, func(module string) error {
_, err := conn.ObjectsList(module, buscalls.ObjectsRequest{Objects: "events"})
return err
}},
{"a work queue not bound", "m", `bound no stream called "jobs"`, func(module string) error {
_, err := conn.WorkQueueSize(module, buscalls.WorkQueueRequest{Queue: "jobs"})
return err
}},
} {
t.Run(c.name, func(t *testing.T) {
err := c.call(c.module)
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Fatalf("refused with %v, want the reason %q", err, c.refusal)
}
})
}
}
+179 -100
View File
@@ -10,6 +10,7 @@
package bus
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
@@ -17,6 +18,8 @@ import (
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/membership"
"log"
"regexp"
"strings"
@@ -29,13 +32,9 @@ import (
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
// RequestTimeout is how long a call waits for its answer — what modules already expect. The
// controller answers its verbs well inside it (its AnswerWithin, ten seconds), with a call id while a
// verb still runs, so a caller is never left guessing past this (novox/hq issue 265).
// RequestTimeout is how long a call waits for its answer — what modules already expect.
const RequestTimeout = 30 * time.Second
const assignmentsStream = "ASSIGNMENTS"
// Credential is a broker credential as the mesh delivers it (novox/hq ADR 0120).
type Credential struct {
URL string `json:"url"`
@@ -79,6 +78,9 @@ type Membership struct {
// Waits are the event families a session may wait on through the mesh MCP server (novox/hq ADR 0367 §2):
// the tool runner's alone, composed from its manifest with the grant that lets it subscribe them.
Waits []WaitIssued `json:"waits,omitempty"`
// Bindings are the module's bus state in the new form, by local name (novox/hq ADR 0595 §1, mesh-sdk
// membership.BusBinding): its object stores, streams, work queues and ledgers, its own alone (bindings.go).
Bindings []membership.BusBinding `json:"bindings,omitempty"`
}
// Served is an address a tool is answered on; `{tool}` stands for the tool's name.
@@ -286,9 +288,20 @@ func (c *Conn) refused(err error) {
})
}
// Connect dials the bus as the credential's module. A module's subjects come from its credential,
// never from its calls (ADR 0074).
// Connect dials the bus as the credential's module and follows its membership. A module's subjects come from its
// credential, never from its calls (ADR 0074).
func Connect(cred Credential) (*Conn, error) {
c, err := Dial(cred)
if err != nil {
return nil, err
}
c.Follow(cred.Module)
return c, nil
}
// Dial dials the bus as the credential's module and follows nothing: a connection that only publishes for its module,
// as a module's ledger connection does (novox/hq ADR 0597 §9).
func Dial(cred Credential) (*Conn, error) {
if cred.Module == "" {
return nil, errors.New("a broker credential with no module: the runtime derives its subjects " +
"from the account the mesh issued, and cannot guess which module it is")
@@ -327,7 +340,6 @@ func Connect(cred Credential) (*Conn, error) {
}
c = &Conn{nc: nc, js: js, self: cred.Module, node: cred.Node, cred: cred,
issued: map[string]*Membership{}, Logf: log.Printf, answering: map[string]*answered{}}
c.Follow(cred.Module)
return c, nil
}
@@ -340,13 +352,10 @@ func (c *Conn) Node() string { return c.node }
// Credential is what this connection was opened with.
func (c *Conn) Credential() Credential { return c.cred }
// MembershipSubject is the one address a runtime derives for an assignment (ADR 0160).
func MembershipSubject(node, module string) string {
return "mesh.assignment." + node + "." + module
}
// Follow reads a module's membership on this machine once and follows it live, so its tools are
// served where the mesh issued them (ADR 0175).
// served where the mesh issued them (ADR 0175). It is read from the grants part's state memberships, its key
// `<node>.<module>` (novox/hq ADR 0571 clause 7 and 10, mesh-sdk buscompose.GrantsMembershipsBucket), and nowhere else:
// the old assignments stream is not read and its subject is not heard.
func (c *Conn) Follow(module string) {
c.mu.Lock()
if c.node == "" {
@@ -359,44 +368,150 @@ func (c *Conn) Follow(module string) {
}
c.issued[module] = nil
c.mu.Unlock()
subject := MembershipSubject(c.node, module)
// The subject-addressed direct get: the one address the mesh grants this account on the
// stream's API.
if got, err := c.nc.Request("$JS.API.DIRECT.GET."+assignmentsStream+"."+subject, nil, 5*time.Second); err == nil {
if got.Header.Get("Status") == "" && len(got.Data) > 0 {
var m Membership
if json.Unmarshal(got.Data, &m) == nil {
c.mu.Lock()
c.issued[module] = &m
c.mu.Unlock()
}
}
}
c.followGrants(module)
if c.Membership(module) == nil {
c.Logf("[mesh-tools] no membership issued for %s on %s yet; serving the derived shape until one arrives", module, c.node)
}
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
}
// GrantsPatience bounds the first read of a membership from the grants part's state.
var GrantsPatience = 2 * time.Second
// GrantsRetry is the first wait before the memberships state is read again after it could not be; it doubles up to
// GrantsRetryMost.
var (
GrantsRetry = time.Second
GrantsRetryMost = 30 * time.Second
)
// issue applies a membership that arrived for module, live, and re-serves on it.
func (c *Conn) issue(module string, m Membership) {
c.mu.Lock()
c.issued[module] = &m
handlers := make([]func(Membership), 0, len(c.onNew))
for _, h := range c.onNew {
handlers = append(handlers, *h)
}
c.mu.Unlock()
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
for _, h := range handlers {
h(m)
}
_ = c.nc.Flush()
}
// followGrants reads module's key of the grants part's state memberships and watches it: the value there now is applied
// before Follow returns, each change after it live. A state that is not there yet, or not readable (a bus answering 503
// right after its boot, a timeout), is said once and read again, with a growing wait, until it can be or the
// connection closes: Follow runs once per module, so nothing else would ever read it again.
func (c *Conn) followGrants(module string) {
if c.watchGrants(module, true) {
return
}
go func() {
wait := GrantsRetry
for !c.nc.IsClosed() {
time.Sleep(wait)
if c.nc.IsClosed() {
return
}
if c.watchGrants(module, false) {
c.Logf("[mesh-tools] the grants part's memberships are read again; %s on %s follows its key", module, c.node)
return
}
if wait *= 2; wait > GrantsRetryMost {
wait = GrantsRetryMost
}
}
}()
}
// watchGrants is one attempt of followGrants: true once the key is watched. A failure is said when say is true.
func (c *Conn) watchGrants(module string, say bool) bool {
logf := func(format string, a ...any) {
if say {
c.Logf(format, a...)
}
}
key := buscompose.MembershipKey(c.node, module)
// First the key's direct get, which the account is granted alone: a bus that refuses it, or a state not there yet,
// answers at once, where binding the bucket would wait out its bound on a refusal.
bucket := buscompose.GrantsMembershipsBucket
probe, err := c.nc.Request("$JS.API.DIRECT.GET.KV_"+bucket+".$KV."+bucket+"."+key, nil, GrantsPatience)
if err != nil || probe.Header.Get("Status") == "503" {
why := "it did not answer"
if err != nil {
why = err.Error()
}
logf("[mesh-tools] the grants part's memberships (%s) cannot be read (%s); %s has no membership "+
"until it can be", bucket, why, module)
return false
}
js, err := jetstream.New(c.nc)
if err != nil {
return false
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := js.KeyValue(ctx, bucket)
if err != nil {
logf("[mesh-tools] the grants part's memberships (%s) cannot be read (%v); %s has no membership "+
"until it can be", buscompose.GrantsMembershipsBucket, err, module)
return false
}
w, err := kv.Watch(context.Background(), key)
if err != nil {
logf("[mesh-tools] %s's key %s of the grants part's memberships cannot be watched (%v); %s has no membership "+
"until it can be", module, key, err, module)
return false
}
apply := func(e jetstream.KeyValueEntry) {
if e.Operation() != jetstream.KeyValuePut {
return // a membership withdrawn is the stream's, or the module's unassignment's, to say
}
// Read through mesh-sdk's one codec of this state, the grants part's writer's (issue 774): it takes the bare
// membership and a wrapper the part wrote before of one published or unchanged, and refuses a held-back one's
// wrapper or a value naming no node or module, which then leaves the module with no membership.
// A refused value arriving after a good one keeps the last good membership: nothing here clears it.
var m Membership
if err := json.Unmarshal(msg.Data, &m); err != nil {
c.Logf("[mesh-tools] a membership arrived that is not one: %v", err)
body, err := membership.BareGrantsMembership(e.Value())
if err == nil {
err = json.Unmarshal(body, &m)
}
if err != nil {
c.Logf("[mesh-tools] the membership at %s is not one: %v", key, err)
return
}
c.mu.Lock()
c.issued[module] = &m
handlers := make([]func(Membership), 0, len(c.onNew))
for _, h := range c.onNew {
handlers = append(handlers, *h)
}
c.mu.Unlock()
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
for _, h := range handlers {
h(m)
}
_ = c.nc.Flush()
})
if err == nil {
c.track(sub)
c.issue(module, m)
}
timer := time.NewTimer(StateTimeout)
defer timer.Stop()
for current := true; current; {
select {
case e, ok := <-w.Updates():
if !ok {
return false
}
if e == nil {
current = false
continue
}
apply(e)
case <-timer.C:
_ = w.Stop()
logf("[mesh-tools] %s's key %s of the grants part's memberships did not answer in %s; %s has no membership "+
"until it can be", module, key, StateTimeout, module)
return false
}
}
go func() {
for e := range w.Updates() {
if e != nil {
apply(e)
}
}
}()
return true
}
// Membership is what the mesh issued a module here, or nil when nothing has been issued.
@@ -686,83 +801,47 @@ func (c *Conn) Ask(key string, body any, on string) (Answered, error) {
// account publish with its own name alone; asked again on the subject that names no caller only when the
// first reached nobody — nothing serves it yet, or the bus does not grant it yet — so no call is done twice.
if called := CalledSubject(subject, c.cred.User); called != "" {
answered, handingOver, fallBack, err := c.askCalled(called, data)
answered, fallBack, err := c.askCalled(called, data)
if !fallBack {
if !handingOver {
return answered, err
}
time.Sleep(HandoverPause)
answered, again, _, err := c.askCalled(called, data)
if err != nil && again {
return Answered{}, fmt.Errorf("asked twice, %s apart, and both times refused as a handover: %w", HandoverPause, err)
}
return answered, err
}
}
answered, handingOver, err := c.askOnce(subject, data)
if !handingOver {
return answered, err
}
// **Refused because the controller is handing over: asked once more** (novox/hq issue 289). The
// controller being replaced did nothing and said so; the one after it answers the same call. Once:
// a second refusal is a controller that is not coming back, and the caller is told.
time.Sleep(HandoverPause)
answered, again, err := c.askOnce(subject, data)
if err != nil && again {
return Answered{}, fmt.Errorf("asked twice, %s apart, and both times refused as a handover: %w", HandoverPause, err)
}
return answered, err
return c.askOnce(subject, data)
}
// HandoverPause is how long a call refused as a handover waits before it is asked once more: the
// supervisor's restart of the controller, which stops the one being replaced and starts the next. A
// variable so a test need not wait.
var HandoverPause = 3 * time.Second
// RetryHandingOver is the mark the controller puts on a call it refused because it is being replaced
// (mesh-controller internal/link, `"retry": "handing-over"`).
const RetryHandingOver = "handing-over"
// ErrNoResponders is the bus's own refusal: nothing subscribes to the subject asked. It is told apart
// from a tool's own error by errors.Is, never by its text (a tool may say "503" itself, issue 1529).
var ErrNoResponders = errors.New("503 no responders")
// askOnce asks once, and says whether the answer was a refusal marked to be asked again.
func (c *Conn) askOnce(subject string, data []byte) (Answered, bool, error) {
// askOnce asks once.
func (c *Conn) askOnce(subject string, data []byte) (Answered, error) {
msg, err := c.nc.Request(subject, data, RequestTimeout)
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return Answered{}, false, ErrNoResponders
return Answered{}, ErrNoResponders
}
if errors.Is(err, nats.ErrTimeout) {
return Answered{}, false, errors.New("timeout")
return Answered{}, errors.New("timeout")
}
return Answered{}, false, err
return Answered{}, err
}
return c.answerOf(subject, msg.Data)
return answerOf(msg.Data)
}
// answerOf reads one answer, asking the rest of a paged one on the subject it was asked on, and says whether it
// was a refusal marked to be asked again.
func (c *Conn) answerOf(subject string, data []byte) (Answered, bool, error) {
// An answer larger than one message comes in parts (pages.go, novox/hq issue 314).
body, err := c.whole(subject, data)
if err != nil {
return Answered{}, false, err
}
// answerOf reads one answer.
func answerOf(data []byte) (Answered, error) {
var r struct {
Result json.RawMessage `json:"result"`
Error string `json:"error"`
Node string `json:"node"`
Retry string `json:"retry"`
}
if err := json.Unmarshal(body, &r); err != nil {
return Answered{}, false, err
if err := json.Unmarshal(data, &r); err != nil {
return Answered{}, err
}
if r.Error != "" {
return Answered{}, r.Retry == RetryHandingOver, errors.New(r.Error)
return Answered{}, errors.New(r.Error)
}
return Answered{Result: r.Result, Node: r.Node}, false, nil
return Answered{Result: r.Result, Node: r.Node}, nil
}
// PublishAs emits an event as a module: published into JetStream and awaited, de-duplicated by its
@@ -966,8 +1045,8 @@ func (c *Conn) AskAs(module, key string, body any) (Answered, error) {
}
// EventsStream is where every module's events land, and ConsumerOf the durable consumer the
// controller makes for a module on a machine: the same names the module's own runtime bound
// (node-tools/src/broker-nats.ts), so moving the module into the node's runtime neither loses an
// mesh makes for a module on a machine: the same names the module's own runtime bound
// (the retired TypeScript runtime's broker-nats.ts), so moving the module into the node's runtime neither loses an
// event nor sees one twice.
const EventsStream = "EVENTS"
@@ -981,7 +1060,7 @@ func ConsumerOf(node, module string) string {
// NakDelay is how long an event a handler failed waits before it is offered again: a transient cause
// gets another attempt, a permanent one exhausts the consumer's max-deliver rather than spinning. The
// consumer then gives the event up, and the controller keeps it in DEAD_LETTERS until a person delivers
// consumer then gives the event up, and the mesh keeps it in DEAD_LETTERS until a person delivers
// it again — under `mesh.again.<consumer>.…`, from which keyOf reads the same key — or drops it
// (novox/hq issue 330).
var NakDelay = 5 * time.Second
@@ -995,7 +1074,7 @@ var EventHeartbeat = 10 * time.Second
// ConsumeAs reads a module's durable consumer on EVENTS, and each one on a module's stream its
// membership names (novox/hq ADR 0363 §8), and hands each event to deliver one at a time (novox/hq
// ADR 0198): acknowledged when deliver returns nil, negatively acknowledged after NakDelay when it
// returns an error, terminated when it is not an event at all. The consumers are the controller's to
// returns an error, terminated when it is not an event at all. The consumers are the mesh's to
// create; this binds to them and never makes one. Runs until stopped.
func (c *Conn) ConsumeAs(module string, deliver func(Envelope) error) (func(), error) {
durable := ConsumerOf(c.node, module)
@@ -1005,7 +1084,7 @@ func (c *Conn) ConsumeAs(module string, deliver func(Envelope) error) (func(), e
}
h := newHearing(c, js, module, deliver)
// Bound by name, never created and never matched against a subject: the consumer and its
// filters are the controller's (design 29 §3), exactly as the module's own runtime bound it.
// filters are the mesh's (design 29 §3), exactly as the module's own runtime bound it.
events, _, doing, err := h.bind(EventsStream, durable, nil)
if err != nil {
return nil, fmt.Errorf("%s %s's consumer %s: %w", doing, module, durable, err)
+11 -11
View File
@@ -18,7 +18,7 @@ import (
// mesh.seat.<seat>.call.<verb>.<caller> a mesh seat's verb
// mesh.seat.<seat>.call.<verb>.<node>.<caller> a node seat's verb on one machine
//
// The last token is the bus user that published the call, and the controller grants each user the publish of
// The last token is the bus user that published the call, and the mesh grants each user the publish of
// these subjects with its own name there and no other — so the caller is a fact the bus server enforces, and
// the runtime hands it to the module from the subject a call arrived on, never from the payload.
//
@@ -32,7 +32,7 @@ import (
const CallKind = "call"
// A bus user's name carries dots (`<node>.<module>`, `person.<name>`); a subject token cannot. In the token
// each dot is written `~`, which no part of a bus user's name may hold (the controller refuses any part that
// each dot is written `~`, which no part of a bus user's name may hold (the mesh refuses any part that
// is not letters, digits, `_` or `-`), so the token is the user's name and nothing else.
const userDot = "~"
@@ -118,7 +118,7 @@ type callers struct {
pending map[string]map[uint64]context.CancelCauseFunc
}
// errCallRefused is the bus refusing a caller-named publish: this user is not granted it yet (the controller's
// errCallRefused is the bus refusing a caller-named publish: this user is not granted it yet (the mesh's
// grant not delivered to the bus), so the call is asked again on the subject that names no caller.
var errCallRefused = errors.New("the bus refused the caller-named subject")
@@ -150,7 +150,7 @@ func (c *Conn) publishRefused(err error) bool {
// askCalled asks once on a caller-named subject. fallBack says the call reached nobody — nothing serves that
// subject yet, or the bus refused this user its publish — so it may be asked on the subject that names no
// caller without being done twice.
func (c *Conn) askCalled(subject string, data []byte) (answered Answered, handingOver, fallBack bool, err error) {
func (c *Conn) askCalled(subject string, data []byte) (answered Answered, fallBack bool, err error) {
ctx, cancel := context.WithCancelCause(context.Background())
ctx, stop := context.WithTimeout(ctx, RequestTimeout)
defer stop()
@@ -180,22 +180,22 @@ func (c *Conn) askCalled(subject string, data []byte) (answered Answered, handin
if err != nil {
switch {
case errors.Is(context.Cause(ctx), errCallRefused):
return Answered{}, false, true, nil
return Answered{}, true, nil
case errors.Is(err, nats.ErrNoResponders):
c.saidFallBack(subject)
return Answered{}, false, true, nil
return Answered{}, true, nil
case errors.Is(err, context.DeadlineExceeded) || errors.Is(err, nats.ErrTimeout):
return Answered{}, false, false, errors.New("timeout")
return Answered{}, false, errors.New("timeout")
}
return Answered{}, false, false, err
return Answered{}, false, err
}
answered, handingOver, err = c.answerOf(subject, msg.Data)
return answered, handingOver, false, err
answered, err = answerOf(msg.Data)
return answered, false, err
}
// saidFallBack says, once per subject, that a caller-named call reached nobody and is asked on the subject that
// names no caller: a runtime or holder not yet serving it, or a grant that drifted, is seen in the log rather
// than read later as a quiet "unproven" (novox/hq issue 365; expected for the controller's seat until hq issue
// than read later as a quiet "unproven" (novox/hq issue 365; expected until hq issue
// 464).
func (c *Conn) saidFallBack(subject string) {
c.calls.mu.Lock()
+1 -1
View File
@@ -16,7 +16,7 @@ import (
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// A bus whose users are granted as the controller grants them (mesh-controller internal/broker): each named
// A bus whose users are granted as the controller grants them (mesh-placement internal/broker): each named
// user publishes the subjects listed for it.
func grantedBus(t *testing.T, publish map[string][]string) string {
t.Helper()
-73
View File
@@ -1,73 +0,0 @@
package bus_test
import (
"encoding/json"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// **A call the controller refused because it is handing over is asked once more** (novox/hq issue 289):
// the controller being replaced did nothing and marked its refusal, and the one after it answers. A
// refusal not so marked is final, and a second handover refusal is said, not asked a third time.
func TestAHandoverRefusalIsAskedOnceMore(t *testing.T) {
was := bus.HandoverPause
bus.HandoverPause = 10 * time.Millisecond
t.Cleanup(func() { bus.HandoverPause = was })
url := meshtest.URL(t)
seat, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer seat.Close()
c, err := bus.Connect(bus.Credential{URL: url, Module: "console", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
defer c.Close()
serve := func(t *testing.T, verb string, answers ...map[string]any) *atomic.Int32 {
t.Helper()
var asked atomic.Int32
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", verb, "", ""), func(m *nats.Msg) {
n := int(asked.Add(1)) - 1
if n >= len(answers) {
n = len(answers) - 1
}
body, _ := json.Marshal(answers[n])
_ = m.Respond(body)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
_ = seat.Flush()
return &asked
}
handingOver := map[string]any{"error": "the controller is handing over to the next one; nothing was done, ask again",
"retry": bus.RetryHandingOver}
asked := serve(t, "rotate", handingOver, map[string]any{"result": "rotated"})
got, err := c.Ask("seat:mesh-controller.rotate", map[string]any{}, "")
if err != nil || string(got.Result) != `"rotated"` || asked.Load() != 2 {
t.Fatalf("after a handover refusal: %s %v, asked %d times", got.Result, err, asked.Load())
}
asked = serve(t, "push", map[string]any{"error": "refused for its own reason"})
if _, err := c.Ask("seat:mesh-controller.push", map[string]any{}, ""); err == nil || asked.Load() != 1 {
t.Fatalf("an unmarked refusal was asked again: %v, asked %d times", err, asked.Load())
}
asked = serve(t, "status", handingOver)
_, err = c.Ask("seat:mesh-controller.status", map[string]any{}, "")
if err == nil || asked.Load() != 2 || !strings.Contains(err.Error(), "both times refused as a handover") {
t.Fatalf("two handover refusals: %v, asked %d times", err, asked.Load())
}
}
+351
View File
@@ -0,0 +1,351 @@
package bus
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/ledger"
)
// A ledger on the bus (novox/hq ADR 0597, design 25 § 12.4; issue 1966): its history a stream with a subject per
// instance under mesh-sdk's buscompose.LedgerHistorySubject, its state a key-value bucket, its snapshots a subject per
// instance under buscompose.LedgerSnapshotSubject, the latest only (the stream that holds them keeps one per subject, as the bus module
// creates it). **Only the module's ledger connection publishes on its history and its snapshots** (ADR 0597 §9): what
// is read, and the state, go on the runner's own.
// LedgerStores are the stores of a module's ledger binding: reads and the state on c, the history's appends and the
// snapshots' writes on writer, the module's ledger connection. The subjects are mesh-sdk's, the ones the grants part
// grants the module's ledger account byte for byte; a binding naming others is refused, naming both.
func (c *Conn) LedgerStores(module string, b membership.BusBinding, writer *Conn) (ledger.History, ledger.State, ledger.Snapshots, error) {
history, snapshot := buscompose.LedgerHistorySubject(module, b.Local), buscompose.LedgerSnapshotSubject(module, b.Local)
for _, named := range []struct{ what, bound, want string }{{"history", b.Subject, history}, {"snapshot", b.Snapshot, snapshot}} {
if named.bound != "" && prefixOf(named.bound) != named.want {
return nil, nil, nil, fmt.Errorf("%s's ledger %q is bound with the %s subject %s, and its ledger account is granted "+
"%s alone (mesh-sdk buscompose); the binding is refused", module, b.Local, named.what, named.bound, named.want)
}
}
return &busHistory{read: c, write: writer, stream: b.History, prefix: history},
&busState{c: c, bucket: b.State},
&busSnapshots{read: c, write: writer, prefix: snapshot}, nil
}
// prefixOf is a binding's subject as the prefix an instance's key is added to: written with or without its `.>`.
func prefixOf(subject string) string {
return strings.TrimSuffix(strings.TrimSuffix(subject, ">"), ".")
}
// publishExpecting publishes body on subject on the connection, written only when the subject's last message is still
// at expect (0: it has none); ledger.ErrRaced when it is not.
func publishExpecting(c *Conn, subject string, body []byte, expect uint64) (uint64, error) {
msg := nats.NewMsg(subject)
msg.Data = body
msg.Header.Set(jetstream.ExpectedLastSubjSeqHeader, strconv.FormatUint(expect, 10))
reply, err := c.nc.RequestMsg(msg, StateTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return 0, fmt.Errorf("no stream on the bus keeps %s", subject)
}
if err != nil {
return 0, err
}
var ack struct {
Seq uint64 `json:"seq"`
Error *struct {
ErrCode uint16 `json:"err_code"`
Description string `json:"description"`
} `json:"error"`
}
if err := json.Unmarshal(reply.Data, &ack); err != nil {
return 0, fmt.Errorf("the bus answered the write to %s with what is not an acknowledgement: %q", subject, reply.Data)
}
if ack.Error != nil {
if code := jetstream.ErrorCode(ack.Error.ErrCode); code == jetstream.JSErrCodeStreamWrongLastSequence ||
code == jetstream.JSErrCodeStreamWrongLastSequenceConstant {
return 0, ledger.ErrRaced
}
return 0, fmt.Errorf("the bus refused the write to %s: %s (%d)", subject, ack.Error.Description, ack.Error.ErrCode)
}
return ack.Seq, nil
}
// lastOn is the last message on a subject of the stream that keeps it: its bytes and sequence, nil and 0 for none.
func lastOn(c *Conn, stream, subject string) ([]byte, uint64, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := c.jetStream()
if err != nil {
return nil, 0, err
}
if stream == "" {
if stream, err = js.StreamNameBySubject(ctx, subject); err != nil {
return nil, 0, fmt.Errorf("no stream on the bus keeps %s: %w", subject, err)
}
}
s, err := js.Stream(ctx, stream)
if err != nil {
return nil, 0, fmt.Errorf("the stream %s: %w", stream, err)
}
m, err := s.GetLastMsgForSubject(ctx, subject)
if errors.Is(err, jetstream.ErrMsgNotFound) {
return nil, 0, nil
}
if err != nil {
return nil, 0, err
}
return m.Data, m.Sequence, nil
}
type busHistory struct {
read, write *Conn
stream, prefix string
}
func (h *busHistory) Last(key string) (uint64, error) {
_, seq, err := lastOn(h.read, h.stream, h.prefix+"."+key)
return seq, err
}
func (h *busHistory) Append(key string, op []byte, expect uint64) (uint64, error) {
if h.write == nil {
return 0, errors.New("the ledger's history is published only on its module's ledger connection, and none is held")
}
return publishExpecting(h.write, h.prefix+"."+key, op, expect)
}
func (h *busHistory) Read(keys string, from uint64, limit int) ([]ledger.Kept, error) {
var out []ledger.Kept
for {
want := StreamReadMost
if limit > 0 && limit-len(out) < want {
want = limit - len(out)
}
page, err := h.read.readStream(h.stream, h.prefix+"."+keys, from, want)
if err != nil {
return out, err
}
for _, m := range page.Messages {
out = append(out, ledger.Kept{Sequence: m.Sequence, Data: m.Body})
}
if page.Next == 0 || (limit > 0 && len(out) >= limit) {
return out, nil
}
from = page.Next
}
}
type busState struct {
c *Conn
bucket string
}
func (s *busState) kv(ctx context.Context) (jetstream.KeyValue, error) {
js, err := s.c.jetStream()
if err != nil {
return nil, err
}
kv, err := js.KeyValue(ctx, s.bucket)
if errors.Is(err, jetstream.ErrBucketNotFound) {
return nil, fmt.Errorf("the ledger's state %s is not on the bus", s.bucket)
}
return kv, err
}
func (s *busState) Get(key string) ([]byte, uint64, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return nil, 0, err
}
e, err := kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, 0, nil
}
if err != nil {
return nil, 0, err
}
return e.Value(), e.Revision(), nil
}
// isFull is a write a bucket or stream refused because it holds as much as it may.
func isFull(err error) bool {
msg := err.Error()
return strings.Contains(msg, "maximum bytes exceeded") || strings.Contains(msg, "maximum messages exceeded") ||
strings.Contains(msg, "insufficient resources")
}
func (s *busState) Write(key string, value []byte, revision uint64) (uint64, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return 0, err
}
var rev uint64
if revision == 0 {
rev, err = kv.Create(ctx, key, value)
} else {
rev, err = kv.Update(ctx, key, value, revision)
}
switch {
case err == nil:
return rev, nil
case errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) || isWrongSequence(err):
return 0, ledger.ErrChanged
case isFull(err):
return 0, fmt.Errorf("%w: %v", ledger.ErrFull, err)
}
return 0, err
}
func (s *busState) Keys() ([]string, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return nil, err
}
keys, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
type busSnapshots struct {
read, write *Conn
prefix string
}
func (s *busSnapshots) Get(key string) ([]byte, uint64, error) {
return lastOn(s.read, "", s.prefix+"."+key)
}
func (s *busSnapshots) Put(key string, value []byte, expect uint64) error {
if s.write == nil {
return errors.New("a ledger's snapshot is published only on its module's ledger connection, and none is held")
}
_, err := publishExpecting(s.write, s.prefix+"."+key, value, expect)
return err
}
// Watch hands deliver every instance's record under a key pattern as it is, then each change; a removed key is handed as
// nil.
func (s *busState) Watch(pattern string, deliver func(key string, value []byte, current bool) error) (func(), error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
kv, err := s.kv(ctx)
cancel()
if err != nil {
return nil, err
}
w, err := kv.Watch(context.Background(), pattern)
if err != nil {
return nil, err
}
hand := func(e jetstream.KeyValueEntry, current bool) {
var value []byte
if e.Operation() == jetstream.KeyValuePut {
value = e.Value()
}
if err := deliver(e.Key(), value, current); err != nil {
s.c.Logf("[mesh-tools] a ledger watch of %s was not taken: %v", s.bucket, err)
}
}
timer := time.NewTimer(StateTimeout)
defer timer.Stop()
for current := true; current; {
select {
case e, ok := <-w.Updates():
if !ok {
return nil, errors.New("the ledger's state watch ended before its current states were handed over")
}
if e == nil {
current = false
continue
}
hand(e, true)
case <-timer.C:
_ = w.Stop()
return nil, fmt.Errorf("the ledger's state %s did not hand its current states in %s", s.bucket, StateTimeout)
}
}
go func() {
for e := range w.Updates() {
if e != nil {
hand(e, false)
}
}
}()
return func() { _ = w.Stop() }, nil
}
// LedgerIndexes are the buckets of a ledger binding's declared indexes, by index name, on the runner's own connection.
func (c *Conn) LedgerIndexes(b membership.BusBinding) map[string]ledger.Index {
out := map[string]ledger.Index{}
for name, bucket := range b.Indexes {
out[name] = &busState{c: c, bucket: bucket}
}
return out
}
func (s *busState) Entries(pattern string, limit int) ([]ledger.Indexed, error) {
if pattern == "" {
pattern = ">"
}
keys, err := s.Keys()
if err != nil {
return nil, err
}
sort.Strings(keys)
var out []ledger.Indexed
for _, k := range keys {
if limit > 0 && len(out) >= limit {
break
}
if !SubjectMatches(pattern, k) {
continue
}
v, _, err := s.Get(k)
if err != nil {
return out, err
}
if v != nil {
out = append(out, ledger.Indexed{Key: k, Value: v})
}
}
return out, nil
}
func (s *busState) Put(key string, value []byte) error {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return err
}
_, err = kv.Put(ctx, key, value)
return err
}
func (s *busState) Delete(key string) error {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return err
}
if err := kv.Delete(ctx, key); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
return err
}
return nil
}
+172 -48
View File
@@ -1,6 +1,7 @@
package bus
import (
"context"
"encoding/json"
"errors"
"fmt"
@@ -8,6 +9,7 @@ import (
"sort"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/nats-io/nats.go"
@@ -15,7 +17,7 @@ import (
)
// A module's log on the bus (novox/hq ADR 0297): a stream of operations, one subject per key, that the
// controller creates from what the module declared under `logs` and issues to the module's own instances
// mesh creates from what the module declared under `logs` and issues to the module's own instances
// in its membership. It mirrors a bucket (ADR 0201) in how it is reached: what a module may reach is
// decided here, from its membership, and refused with the reason before anything is sent.
//
@@ -94,6 +96,10 @@ func logFilter(l LogIssued, pattern string) (string, error) {
}
parts := strings.Split(pattern, ".")
for i, p := range parts {
if p == "**" && i == len(parts)-1 {
// `**` for the rest, as a state's key pattern and mesh-sdk's log reader say it.
parts[i], p = ">", ">"
}
switch {
case p == ">" && i == len(parts)-1, p == "*", logToken.MatchString(p):
default:
@@ -104,7 +110,7 @@ func logFilter(l LogIssued, pattern string) (string, error) {
if len(pattern) > LogKeyMax {
return "", fmt.Errorf("%q is longer than a key can be (%d bytes)", pattern, LogKeyMax)
}
return l.Subject + "." + pattern, nil
return l.Subject + "." + strings.Join(parts, "."), nil
}
// issuedLog is the log a module may reach by a name, or the reason it may not.
@@ -126,13 +132,14 @@ func (c *Conn) issuedLog(module, name string) (LogIssued, error) {
if len(names) > 0 {
issued = strings.Join(names, ", ")
}
return LogIssued{}, fmt.Errorf("%s keeps no log called %q: a module declares the logs it keeps under `logs`, "+
"and a log is its owner's alone; it was issued: %s (novox/hq ADR 0297)", module, name, issued)
return LogIssued{}, fmt.Errorf("%s keeps no log called %q: a module declares the logs it keeps under `logs`, and "+
"another module's it reads under `reads` as <module>.<log>; it was issued: %s (novox/hq ADR 0297, ADR 0571)",
module, name, issued)
}
// notOnTheBus says a log is issued and its stream is not there yet.
func notOnTheBus(l LogIssued) error {
return fmt.Errorf("the log %q is issued and its stream %s is not on the bus yet: the controller creates "+
return fmt.Errorf("the log %q is issued and its stream %s is not on the bus yet: the mesh creates "+
"it from the catalogue on its next raise (novox/hq ADR 0297)", l.Name, l.Stream)
}
@@ -200,9 +207,10 @@ func (c *Conn) LogAppend(module, name, key string, value json.RawMessage, expect
return ack.Seq, nil
}
// LogRead is the entries from a sequence on, of the keys matching a pattern, at most limit of them and
// about 4 MiB, by one batch direct get.
func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int) (*LogPage, error) {
// LogRead is the entries from a sequence on, or from a time on (since, when not zero), of the keys matching a pattern, at
// most limit of them and about 4 MiB, by one batch direct get. A log another module keeps is read as its own is, by the
// name its reads declare, `<module>.<log>` (novox/hq ADR 0571).
func (c *Conn) LogRead(module, name string, from uint64, since time.Time, keys string, limit int) (*LogPage, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
@@ -218,52 +226,24 @@ func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int)
if err != nil {
return nil, err
}
if from != 0 && !since.IsZero() {
return nil, fmt.Errorf("a read of %s starts from a sequence or from a time, not both", name)
}
if from == 0 {
from = 1
}
req, _ := json.Marshal(map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes})
inbox := c.nc.NewInbox()
sub, err := c.nc.SubscribeSync(inbox)
req := map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes}
if !since.IsZero() {
delete(req, "seq")
req["start_time"] = since.UTC()
}
entries, more, err := c.logBatch(l, req)
if err != nil {
return nil, err
}
defer func() { _ = sub.Unsubscribe() }()
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+l.Stream, inbox, req); err != nil {
return nil, err
}
page := &LogPage{Entries: []LogEntry{}}
deadline := time.Now().Add(StateTimeout)
for done := false; !done; {
msg, err := sub.NextMsg(time.Until(deadline))
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return nil, notOnTheBus(l)
}
if errors.Is(err, nats.ErrTimeout) {
return nil, fmt.Errorf("the read of %s did not end in %s", name, StateTimeout)
}
return nil, err
}
switch status := msg.Header.Get("Status"); status {
case "":
e, err := logEntryOf(l, msg)
if err != nil {
return nil, err
}
page.Entries = append(page.Entries, *e)
case "204": // the end of the batch: what is still pending after the last entry sent
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
if pending > 0 && len(page.Entries) > 0 {
page.Next = page.Entries[len(page.Entries)-1].Sequence + 1
}
done = true
case "404": // nothing from that sequence on
done = true
case "503":
return nil, notOnTheBus(l)
default:
return nil, fmt.Errorf("the read of %s was refused by the bus: %s %s", name, status, msg.Header.Get("Description"))
}
page := &LogPage{Entries: entries}
if more && len(entries) > 0 {
page.Next = entries[len(entries)-1].Sequence + 1
}
info, err := c.logInfo(l)
if err != nil {
@@ -273,6 +253,150 @@ func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int)
return page, nil
}
// LogNewest is the newest entry of each key matching a pattern, in the log's order, by one batch direct get (mesh-sdk
// process.LogReader.Newest).
func (c *Conn) LogNewest(module, name, keys string) ([]LogEntry, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
filter, err := logFilter(l, keys)
if err != nil {
return nil, err
}
entries, more, err := c.logBatch(l, map[string]any{"multi_last": []string{filter}, "batch": LogReadMax})
if err != nil {
return nil, err
}
if more {
return nil, fmt.Errorf("%s holds more than %d keys matching %q: name fewer", name, LogReadMax, keys)
}
return entries, nil
}
// logBatch is one batch direct get of a log: its entries, and whether more are pending after them.
func (c *Conn) logBatch(l LogIssued, request map[string]any) ([]LogEntry, bool, error) {
req, _ := json.Marshal(request)
inbox := c.nc.NewInbox()
sub, err := c.nc.SubscribeSync(inbox)
if err != nil {
return nil, false, err
}
defer func() { _ = sub.Unsubscribe() }()
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+l.Stream, inbox, req); err != nil {
return nil, false, err
}
entries := []LogEntry{}
deadline := time.Now().Add(StateTimeout)
for {
msg, err := sub.NextMsg(time.Until(deadline))
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return nil, false, notOnTheBus(l)
}
if errors.Is(err, nats.ErrTimeout) {
return nil, false, fmt.Errorf("the read of %s did not end in %s", l.Name, StateTimeout)
}
return nil, false, err
}
switch status := msg.Header.Get("Status"); status {
case "":
e, err := logEntryOf(l, msg)
if err != nil {
return nil, false, err
}
entries = append(entries, *e)
case "204": // the end of the batch: what is still pending after the last entry sent
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
return entries, pending > 0, nil
case "404": // nothing from there on
return entries, false, nil
case "503":
return nil, false, notOnTheBus(l)
default:
return nil, false, fmt.Errorf("the read of %s was refused by the bus: %s %s", l.Name, status, msg.Header.Get("Description"))
}
}
}
// LogWatch hands deliver every entry of the keys matching a pattern from a sequence on (from), from a time on (since),
// or, with neither, each new one, in the log's order, one at a time, through an ordered consumer it lets go of when
// stopped (mesh-sdk process.LogReader.Watch). It returns once the entries there at its start are handed over; an error
// from deliver is said, and the watch goes on with the next entry.
func (c *Conn) LogWatch(module, name, keys string, from uint64, since time.Time, deliver func(LogEntry) error) (func(), error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
filter, err := logFilter(l, keys)
if err != nil {
return nil, err
}
cfg := jetstream.OrderedConsumerConfig{FilterSubjects: []string{filter}, DeliverPolicy: jetstream.DeliverNewPolicy}
switch {
case from != 0 && !since.IsZero():
return nil, fmt.Errorf("a watch of %s starts from a sequence or from a time, not both", name)
case from != 0:
cfg.DeliverPolicy, cfg.OptStartSeq = jetstream.DeliverByStartSequencePolicy, from
case !since.IsZero():
at := since.UTC()
cfg.DeliverPolicy, cfg.OptStartTime = jetstream.DeliverByStartTimePolicy, &at
}
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
cons, err := js.OrderedConsumer(ctx, l.Stream, cfg)
if errors.Is(err, jetstream.ErrStreamNotFound) {
return nil, notOnTheBus(l)
}
if err != nil {
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
}
info, err := cons.Info(ctx)
if err != nil {
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
}
it, err := cons.Messages()
if err != nil {
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
}
hand := func(m jetstream.Msg) error {
meta, err := m.Metadata()
if err != nil {
return err
}
e := LogEntry{Sequence: meta.Sequence.Stream, Key: strings.TrimPrefix(m.Subject(), l.Subject+"."),
Value: valueOf(m.Data()), At: meta.Timestamp.UTC()}
return deliver(e)
}
for pending := info.NumPending; pending > 0; pending-- {
m, err := it.Next()
if err == nil {
err = hand(m)
}
if err != nil {
it.Stop()
return nil, fmt.Errorf("the watch of %s: %w", name, err)
}
}
var stopped atomic.Bool
go func() {
for {
m, err := it.Next()
if err != nil || stopped.Load() {
return // stopped, or the connection closed; a message fetched as it stopped is not handed over
}
if err := hand(m); err != nil {
c.Logf("[mesh-tools] %s did not take entry of %s: %v; it is in the log, a read finds it", module, name, err)
}
}
}()
return func() { stopped.Store(true); it.Stop() }, nil
}
// LogLast is a key's last entry, or nil when it has none.
func (c *Conn) LogLast(module, name, key string) (*LogEntry, error) {
l, err := c.issuedLog(module, name)
+2 -97
View File
@@ -1,13 +1,7 @@
package bus
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
@@ -15,97 +9,8 @@ import (
// An answer larger than the bus carries in one message (novox/hq issue 314).
//
// The bus carries one message of at most its max_payload, and the client library refuses a larger one
// before it leaves the process that sent it. Until this, a holder whose answer was too large wrote one
// line to its own log and its caller waited out RequestTimeout and read "did not answer" — the
// controller's `conditions` and `status` for hours on 2026-10-08, and the operator's channel with them.
//
// The controller pages such an answer (mesh-controller internal/link, pages.go): it answers an error
// carrying `paged` — the call it holds the answer under, its size and its parts — and a caller asks the
// same subject again once per part with PageHeader, each part answered raw with PartHeader. Asking the
// same subject means no grant beyond the one the caller already had. A module's answer too large is
// refused here, loudly, in a message that fits: its instances may be several, and a part asked again
// could reach another.
const (
// PageHeader asks for one part of a paged answer: `<call> <part>`, from zero.
PageHeader = "Mesh-Page"
// PartHeader is on a part: `<part>/<parts>`.
PartHeader = "Mesh-Part"
// partTries is how often one part is asked: during a handover a part can reach the controller that
// did not hold it.
partTries = 3
)
// PartTimeout bounds the asking of one part.
var PartTimeout = 10 * time.Second
// paged is what a too-large answer says instead of itself.
type paged struct {
Call string `json:"call"`
Bytes int `json:"bytes"`
Parts int `json:"parts"`
}
func pagedIn(data []byte) (paged, bool) {
if !bytes.Contains(data, []byte(`"paged"`)) {
return paged{}, false
}
var r struct {
Paged *paged `json:"paged"`
}
if json.Unmarshal(data, &r) != nil || r.Paged == nil || r.Paged.Call == "" || r.Paged.Parts < 1 {
return paged{}, false
}
return *r.Paged, true
}
// whole is a reply's whole answer: the reply itself, or every part of a paged one, joined. A part that
// cannot be had is an error naming it, never a shorter answer.
func (c *Conn) whole(subject string, data []byte) ([]byte, error) {
p, ok := pagedIn(data)
if !ok {
return data, nil
}
out := make([]byte, 0, p.Bytes)
for n := 0; n < p.Parts; n++ {
var part []byte
var err error
for try := 0; try < partTries; try++ {
if part, err = c.askPart(subject, p, n); err == nil {
break
}
}
if err != nil {
return nil, fmt.Errorf("the answer was %d bytes, more than the bus carries in one message, paged as %s "+
"in %d parts, and part %d could not be had: %w", p.Bytes, p.Call, p.Parts, n, err)
}
out = append(out, part...)
}
if len(out) != p.Bytes {
return nil, fmt.Errorf("the answer paged as %s was %d bytes, and its parts joined are %d", p.Call, p.Bytes, len(out))
}
return out, nil
}
func (c *Conn) askPart(subject string, p paged, n int) ([]byte, error) {
ask := nats.NewMsg(subject)
ask.Header.Set(PageHeader, fmt.Sprintf("%s %d", p.Call, n))
ask.Data = []byte(`{}`)
reply, err := c.nc.RequestMsg(ask, PartTimeout)
if err != nil {
return nil, err
}
if got := reply.Header.Get(PartHeader); got != fmt.Sprintf("%d/%d", n, p.Parts) {
var r struct {
Error string `json:"error"`
}
if json.Unmarshal(reply.Data, &r) == nil && r.Error != "" {
return nil, errors.New(r.Error)
}
return nil, fmt.Errorf("asked part %d/%d, answered %q", n, p.Parts, got)
}
return reply.Data, nil
}
// before it leaves the process that sent it. A module's answer too large is refused here, loudly, in a
// message that fits, so its caller reads why instead of waiting out RequestTimeout.
// fits is an answer the bus can carry: the answer itself, or — when it is larger than one message —
// an error that says so, which is.
+3 -39
View File
@@ -2,7 +2,6 @@ package bus_test
import (
"encoding/json"
"fmt"
"strings"
"testing"
"time"
@@ -13,8 +12,8 @@ import (
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq issue 314: an answer larger than the bus carries in one message arrives whole when the
// controller pages it, and a module's is said to be too large — neither is a 30-second silence.
// novox/hq issue 314: a module's answer larger than the bus carries in one message is said to be too large,
// at once, not a 30-second silence. (The controller's paging ended with the controller, issue 1126.)
func TestAnAnswerLargerThanOneMessage(t *testing.T) {
url := meshtest.URL(t)
seat, err := nats.Connect(url)
@@ -27,43 +26,8 @@ func TestAnAnswerLargerThanOneMessage(t *testing.T) {
t.Fatal(err)
}
defer c.Close()
limit := int(seat.MaxPayload())
large := strings.Repeat("a condition held past its bound; ", 3*int(seat.MaxPayload())/32)
// The controller as it pages (mesh-controller internal/link, pages.go): the first answer an error
// with `paged`, each part asked again on the same subject with the page header.
large := strings.Repeat("a condition held past its bound; ", 3*limit/32)
whole, _ := json.Marshal(map[string]any{"result": large})
partSize := limit - 1024
parts := (len(whole) + partSize - 1) / partSize
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", "conditions", "", ""), func(m *nats.Msg) {
if page := m.Header.Get(bus.PageHeader); page != "" {
var id string
var n int
fmt.Sscanf(page, "%s %d", &id, &n)
end := min((n+1)*partSize, len(whole))
part := nats.NewMsg(m.Reply)
part.Header.Set(bus.PartHeader, fmt.Sprintf("%d/%d", n, parts))
part.Data = whole[n*partSize : end]
_ = m.RespondMsg(part)
return
}
body, _ := json.Marshal(map[string]any{"error": "too large for one message",
"paged": map[string]any{"call": "call-1-1", "bytes": len(whole), "parts": parts}})
_ = m.Respond(body)
})
if err != nil {
t.Fatal(err)
}
defer sub.Unsubscribe()
_ = seat.Flush()
got, err := c.Ask("seat:mesh-controller.conditions", map[string]any{}, "")
var said string
if err != nil || json.Unmarshal(got.Result, &said) != nil || said != large {
t.Fatalf("an answer of %d bytes in %d parts arrived as %d bytes: %v", len(whole), parts, len(got.Result), err)
}
// A module's answer too large is refused here in words, at once.
stop, err := c.HandleSubject("mesh.mod.journal.tool.lines", func(json.RawMessage) (any, error) { return large, nil })
if err != nil {
t.Fatal(err)
+105
View File
@@ -0,0 +1,105 @@
package bus
import (
"context"
"fmt"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/pernode"
)
// A state its owner keeps one bucket per node (novox/hq ADR 0593 §6: each node's engine writes its own `composed-<node>`,
// and the provisioner, the assigner and the view read them all) is served to a bundle as one state. The rules (a key's
// first part names its node's bucket, keys and a watch over every bucket of the prefix, a bucket created later heard by
// the bus's creation advisory) are mesh-sdk's pernode package, written once for a process and for this runtime; this
// file adapts its entries to the runtime's shapes. A reader is granted the listing of stream names
// (pernode.ListSubject) and the creation advisory (pernode.CreatedAdvisory) beside each bucket's read.
func perNodeRefusal(module, name, what string) error {
return fmt.Errorf("%s's state %s is kept one bucket per node and read only: no %s of it as one state (novox/hq ADR 0593 §6)",
module, name, what)
}
func (c *Conn) family(s StateIssued) pernode.Family {
return pernode.Family{Read: s.Name, Prefix: s.Bucket, Bus: pernode.OverNATS(c.nc)}
}
func (c *Conn) perNodeGet(s StateIssued, key string) (*StateEntry, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
e, err := c.family(s).Get(ctx, key)
if err != nil || e == nil {
return nil, err
}
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision()}, nil
}
func (c *Conn) perNodeKeys(s StateIssued) ([]string, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
return c.family(s).Keys(ctx)
}
// perNodeWatch is StateWatch over every bucket of the family, and over a bucket created later. As in a single bucket's
// watch, StateTimeout bounds the time without progress (issue 1726), and a value the bundle does not take is said and
// the watch goes on.
func (c *Conn) perNodeWatch(module string, s StateIssued, pattern string, deliver func(StateChange) error) (func(), error) {
ctx, cancel := context.WithCancel(context.Background())
progress := make(chan struct{}, 1)
handle := func(ch pernode.Change) error {
select {
case progress <- struct{}{}:
default:
}
e := ch.Entry
change := StateChange{State: s.Name, Key: e.Key(), Revision: e.Revision(), Current: ch.Current}
switch e.Operation() {
case jetstream.KeyValuePut:
change.Op = "put"
change.Value = valueOf(e.Value())
default:
if ch.Current {
return nil
}
change.Op = "delete"
if e.Operation() == jetstream.KeyValuePurge {
change.Op = "expire"
}
}
if ctx.Err() != nil {
return nil
}
if err := deliver(change); err != nil {
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
module, change.Op, change.Key, s.Name, err)
}
return nil
}
done := make(chan error, 1)
go func() { done <- c.family(s).Watch(ctx, pattern, handle) }()
bound := time.NewTimer(StateTimeout)
defer bound.Stop()
for {
select {
case err := <-done:
if err != nil {
cancel()
return nil, err
}
return cancel, nil
case <-progress:
if !bound.Stop() {
select {
case <-bound.C:
default:
}
}
bound.Reset(StateTimeout)
case <-bound.C:
cancel()
return nil, fmt.Errorf("the current values of %s did not arrive in %s: none was handed over in that time", s.Name, StateTimeout)
}
}
}
+3 -3
View File
@@ -14,16 +14,16 @@ import (
// ADR 0370 changes it, issue 685).
//
// While a module's code runs here, its asks reach the bus under this runtime's account, and the bus proves only
// that account. So the runtime asks its own machine's client on ProvisionAskSubject — which the controller grants
// that account. So the runtime asks its own machine's client on ProvisionAskSubject — which the mesh grants
// this machine's tool runner alone to publish (ADR 0370 §3) — naming the module itself, from the bundle that asks
// and never from the module's code or arguments (§4). The client answers that module's config values, for what its
// manifest declares, and no secret: the vault hands a secret only to a caller the bus proves (§2).
// ProvisionAskSubject is where this machine's tool runner asks its provisioner client (mesh-controller
// ProvisionAskSubject is where this machine's tool runner asks its provisioner client (the node-engine
// internal/broker ProvisionerAskSubject): `prov.ask.<node>`.
func ProvisionAskSubject(node string) string { return "prov.ask." + node }
// ProvisionChangedSubjects is every module's word of what moved on a machine (mesh-controller internal/broker
// ProvisionChangedSubjects is every module's word of what moved on a machine (the node-engine
// ProvisionerChangedSubject): `prov.changed.<node>.<module>`, a module's name dots and all.
func ProvisionChangedSubjects(node string) string { return "prov.changed." + node + ".>" }
+4 -4
View File
@@ -24,7 +24,7 @@ import (
// and anything else is refused here, with the reason, before it is sent. A bundle reaches this only through
// its own standard input and output: no tool call does.
// SeatTraffic is what a membership lists of a module's seat traffic (the controller's broker.SeatTraffic).
// SeatTraffic is what a membership lists of a module's seat traffic (the grants part's record).
type SeatTraffic struct {
Publish []string `json:"publish,omitempty"`
Subscribe []string `json:"subscribe,omitempty"`
@@ -32,7 +32,7 @@ type SeatTraffic struct {
Workers []SeatWorker `json:"workers,omitempty"`
Records []string `json:"records,omitempty"`
// Kinds are the holders of every kinded bench the module uses or watches, with what each promises: the
// controller's record of their claims (novox/hq ADR 0259 §5).
// record of their claims (novox/hq ADR 0259 §5).
Kinds []KindHeld `json:"kinds,omitempty"`
}
@@ -45,7 +45,7 @@ type KindHeld struct {
Capabilities []string `json:"capabilities,omitempty"`
}
// SeatKinds is the kinds the module's membership names, as the controller issued them.
// SeatKinds is the kinds the module's membership names, as the grants part issued them.
func (c *Conn) SeatKinds(module string) ([]KindHeld, error) {
t, err := c.trafficOf(module)
if err != nil {
@@ -249,7 +249,7 @@ type Work struct {
// SeatTake reads a worker the module's seat traffic lists and hands each message to deliver: acknowledged
// when deliver returns nil, offered again after NakDelay when it returns an error, ended when it is not
// JSON. The worker is the controller's to create; this binds to it by name and never makes one.
// JSON. The worker is the mesh's to create; this binds to it by name and never makes one.
func (c *Conn) SeatTake(module, consumer string, deliver func(Work) error) (func(), error) {
t, err := c.trafficOf(module)
if err != nil {
+2 -2
View File
@@ -98,7 +98,7 @@ func TestAModuleSubmitsAndSaysOnlyUnderItsOwnNameAndKind(t *testing.T) {
t.Fatalf("an asker could not ask under its own name: %v", err)
}
for module, subject := range map[string]string{
"mesh-delivery": "mesh.seat.operator-channel.accept.ask.mesh-controller",
"mesh-delivery": "mesh.seat.operator-channel.accept.ask.mesh-placement",
"telegram": "mesh.seat.intake.event.choice.desktop",
"messenger": "mesh.seat.intake.event.choice.telegram",
} {
@@ -206,7 +206,7 @@ func TestAnAskerReadsItsOwnRecordAndNoOther(t *testing.T) {
if _, err := kv.Put("mesh-delivery.a1", []byte(`{"state":"open"}`)); err != nil {
t.Fatal(err)
}
if _, err := kv.Put("mesh-controller.a1", []byte(`{"state":"open"}`)); err != nil {
if _, err := kv.Put("mesh-placement.a1", []byte(`{"state":"open"}`)); err != nil {
t.Fatal(err)
}
got, err := conn.SeatRecord("mesh-delivery", "messenger_asks", "a1")
+153 -3
View File
@@ -14,7 +14,7 @@ import (
"github.com/nats-io/nats.go/jetstream"
)
// A module's state on the bus (novox/hq ADR 0201): key-value buckets the controller creates from what
// A module's state on the bus (novox/hq ADR 0201): key-value buckets the mesh creates from what
// the module declared, and issues to each assignment in its membership by the name the module uses —
// its own state by the local name, another module's as `<module>.<name>`.
//
@@ -28,6 +28,10 @@ type StateIssued struct {
Name string `json:"name"`
Bucket string `json:"bucket"`
Writes bool `json:"writes,omitempty"`
// PerNode says the state is kept one bucket per node (novox/hq ADR 0593 §6, mesh-sdk membership.StateIssued): Bucket is
// then the buckets' prefix, each bucket the prefix and a node's name, and a key's first part names its node. It is read
// only, served as one state (per_node.go).
PerNode bool `json:"per-node,omitempty"`
}
// StateEntry is one key's current value.
@@ -37,7 +41,8 @@ type StateEntry struct {
Revision uint64 `json:"revision"`
}
// StateChange is one change a watch delivers: a key put or deleted.
// StateChange is one change a watch delivers: a key put, deleted, or expired (its own time to live, or the
// bucket's age, ended; issue 1952).
type StateChange struct {
State string `json:"state"`
Key string `json:"key"`
@@ -88,7 +93,7 @@ func (c *Conn) bucket(ctx context.Context, s StateIssued) (jetstream.KeyValue, e
kv, err := js.KeyValue(ctx, s.Bucket)
if err != nil {
if errors.Is(err, jetstream.ErrBucketNotFound) {
return nil, fmt.Errorf("the state %q is issued and its bucket is not on the bus yet: the controller "+
return nil, fmt.Errorf("the state %q is issued and its bucket is not on the bus yet: the mesh "+
"creates it from the catalogue on its next raise", s.Name)
}
return nil, err
@@ -113,6 +118,9 @@ func (c *Conn) StateGet(module, name, key string) (*StateEntry, error) {
if err := checkKey(key); err != nil {
return nil, err
}
if s.PerNode {
return c.perNodeGet(s, key)
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
@@ -148,6 +156,9 @@ func (c *Conn) StateHistory(module, name, key string) ([]StateKept, error) {
if err := checkKey(key); err != nil {
return nil, err
}
if s.PerNode {
return nil, perNodeRefusal(module, name, "history")
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
@@ -173,6 +184,54 @@ func (c *Conn) StateHistory(module, name, key string) ([]StateKept, error) {
return kept, nil
}
// StateInfo is how full a state's bucket is against its caps (tools-are-how-a-module-is-spoken-to): a bundle's tools
// say fill against cap from it.
type StateInfo struct {
State string `json:"state"`
Bucket string `json:"bucket"`
// Bytes is what the bucket holds now, every kept value and marker counted; Keys how many keys it holds, a key
// whose last value is a delete or expiry marker counted until the marker goes.
Bytes uint64 `json:"bytes"`
Keys uint64 `json:"keys"`
// MaxBytes and MaxValueBytes are the bucket's caps; -1 is none.
MaxBytes int64 `json:"max_bytes"`
MaxValueBytes int32 `json:"max_value_bytes"`
// History is how many values a key keeps; TTLSeconds the bucket's age for a value (0 none); KeyTTL whether a key may
// carry its own time to live (issue 1952).
History int64 `json:"history"`
TTLSeconds int64 `json:"ttl_seconds"`
KeyTTL bool `json:"key_ttl"`
}
// StateInfo reads a state's bucket's fill and caps. It is a read like StateGet: served to the owner and every reader.
func (c *Conn) StateInfo(module, name string) (*StateInfo, error) {
s, err := c.issuedState(module, name)
if err != nil {
return nil, err
}
if s.PerNode {
return nil, perNodeRefusal(module, name, "fill")
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return nil, err
}
status, err := kv.Status(ctx)
if err != nil {
return nil, err
}
b, ok := status.(*jetstream.KeyValueBucketStatus)
if !ok || b.StreamInfo() == nil {
return nil, fmt.Errorf("the bus answered no stream behind %s's state %s", module, name)
}
info := b.StreamInfo()
return &StateInfo{State: name, Bucket: status.Bucket(), Bytes: info.State.Bytes, Keys: uint64(info.State.NumSubjects),
MaxBytes: info.Config.MaxBytes, MaxValueBytes: info.Config.MaxMsgSize, History: status.History(),
TTLSeconds: int64(status.TTL() / time.Second), KeyTTL: info.Config.AllowMsgTTL && status.LimitMarkerTTL() > 0}, nil
}
// StatePut writes one key, as the module, where the module keeps the state. It answers the revision.
func (c *Conn) StatePut(module, name, key string, value json.RawMessage) (uint64, error) {
s, err := c.writable(module, name)
@@ -199,6 +258,61 @@ func (c *Conn) StatePut(module, name, key string, value json.RawMessage) (uint64
return kv.Put(ctx, key, value)
}
// MinKeyTTL is the shortest time to live a key may be given: the bus counts a key's age in whole seconds (mesh-sdk
// process.MinTTL).
const MinKeyTTL = time.Second
// StateCreateTTL writes one key only when it has no value, with its own time to live (novox/hq ADR 0564 clauses 4 and
// 7, issue 1952; mesh-sdk process.State.CreateTTL): the bus removes it when ttl passes and every watch hears the
// removal as an expiry, so a bundle's wake needs no timer. A key that has a value is refused as ErrStateChanged. The
// bucket must take keys with their own time to live (a state declared `key-ttl`); one that does not is refused, naming
// it, before anything is written.
func (c *Conn) StateCreateTTL(module, name, key string, value json.RawMessage, ttl time.Duration) (uint64, error) {
if ttl < MinKeyTTL {
return 0, fmt.Errorf("%s's state %s, key %s: a time to live of %s is shorter than the bus keeps (%s)",
module, name, key, ttl, MinKeyTTL)
}
return c.stateWrite(module, name, key, value, func(ctx context.Context, kv jetstream.KeyValue) (uint64, error) {
if err := takesKeyTTL(ctx, kv, module, name); err != nil {
return 0, err
}
return kv.Create(ctx, key, value, jetstream.KeyTTL(ttl))
})
}
// StateRenewTTL writes one key whole with a fresh time to live, whatever it held (mesh-sdk process.State.RenewTTL): a
// time to live is set only when a key is created, so a live key is deleted first, and its watches hear the delete and
// then the new value.
func (c *Conn) StateRenewTTL(module, name, key string, value json.RawMessage, ttl time.Duration) (uint64, error) {
rev, err := c.StateCreateTTL(module, name, key, value, ttl)
if !errors.Is(err, ErrStateChanged) {
return rev, err
}
if err := c.StateDelete(module, name, key); err != nil {
return 0, err
}
return c.StateCreateTTL(module, name, key, value, ttl)
}
// takesKeyTTL refuses a bucket that does not allow a time to live per key, or leaves no marker when one ends: there the
// server would refuse the write, or the key would vanish and no watch would hear it.
func takesKeyTTL(ctx context.Context, kv jetstream.KeyValue, module, name string) error {
status, err := kv.Status(ctx)
if err != nil {
return err
}
allows := status.LimitMarkerTTL() > 0
if b, ok := status.(*jetstream.KeyValueBucketStatus); ok && b.StreamInfo() != nil {
allows = allows && b.StreamInfo().Config.AllowMsgTTL
}
if !allows {
return fmt.Errorf("%s's state %s does not take keys with their own time to live: its bucket %s allows none, or "+
"leaves no marker a watch hears when one ends; %s declares the state `key-ttl` for it (novox/hq ADR 0564, "+
"issue 1952)", module, name, status.Bucket(), module)
}
return nil
}
// StateDelete removes one key, as the module, where the module keeps the state. A key that was not
// there is not an error: what is asked for is that it is gone.
func (c *Conn) StateDelete(module, name, key string) error {
@@ -224,12 +338,20 @@ func (c *Conn) StateKeys(module, name string) ([]string, error) {
if err != nil {
return nil, err
}
if s.PerNode {
return c.perNodeKeys(s)
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return nil, err
}
return keysOf(ctx, kv)
}
// keysOf is every key with a current value in one bucket, sorted.
func keysOf(ctx context.Context, kv jetstream.KeyValue) ([]string, error) {
lister, err := kv.ListKeys(ctx)
if err != nil {
return nil, err
@@ -291,12 +413,20 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
}
filter = strings.Join(parts, ".")
}
if s.PerNode {
return c.perNodeWatch(module, s, pattern, deliver)
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
kv, err := c.bucket(ctx, s)
cancel()
if err != nil {
return nil, err
}
return c.watchBucket(module, name, kv, filter, deliver)
}
// watchBucket is StateWatch over one bucket: the current values, then every change.
func (c *Conn) watchBucket(module, name string, kv jetstream.KeyValue, filter string, deliver func(StateChange) error) (stop func(), err error) {
watching, stopWatching := context.WithCancel(context.Background())
w, err := kv.Watch(watching, filter)
if err != nil {
@@ -322,6 +452,12 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
go func() {
initial := true
for e := range w.Updates() {
// A watch given up, or stopped, hands over nothing more, though values stay buffered in the
// watcher: they would reach a watch id the bundle no longer holds (issue 774: 1257 orphan
// changes after 51 failed watches).
if watching.Err() != nil {
break
}
if e == nil {
// The end of what was there when the watch began.
if initial {
@@ -330,6 +466,11 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
}
continue
}
if initial {
// Arrival from the bus is progress before the bundle is asked: the bound measures the current
// values coming in, and restarts when the bundle has taken one.
progress()
}
change := StateChange{State: name, Key: e.Key(), Revision: e.Revision(), Current: initial}
switch e.Operation() {
case jetstream.KeyValuePut:
@@ -343,6 +484,15 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
continue
}
change.Op = "delete"
if e.Operation() == jetstream.KeyValuePurge {
// The marker the server leaves when a key's time to live ends (Nats-Marker-Reason MaxAge,
// read by the client as a purge): nothing else purges a module's state, since neither the
// runtime nor mesh-sdk's State offers a purge (issue 1952).
change.Op = "expire"
}
}
if watching.Err() != nil {
break
}
if err := deliver(change); err != nil {
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
@@ -0,0 +1,111 @@
//go:build integration
package bus_test
import (
"errors"
"fmt"
"reflect"
"sort"
"strings"
"sync"
"testing"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// **A state its owner keeps one bucket per node is served to a bundle as one state** (novox/hq ADR 0593 §6: each node's
// engine writes its own `composed-<node>`, and the provisioner reads them all). Get goes to the bucket the key's first
// part names, Keys and Watch over every bucket of the prefix, a bucket created during the watch included, and a bucket
// that only starts like the prefix is never read.
func TestAPerNodeStateIsServedAcrossEveryNodesBucket(t *testing.T) {
mesh := mt.New(t)
a, b := mesh.Bucket(t, "composed-a"), mesh.Bucket(t, "composed-b")
other := mesh.Bucket(t, "composedx-z")
for kv, key := range map[interface {
Put(string, []byte) (uint64, error)
}]string{a: "a.m1.f1", b: "b.m2.f2", other: "z.m3.f3"} {
if _, err := kv.Put(key, []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
}
prov := mt.MembershipOf("mesh-provisioner", "anchor", false, nil)
prov.State = []bus.StateIssued{{Name: "mesh-host.composed", Bucket: "composed-", PerNode: true}}
mesh.Issue(t, prov)
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
conn.Follow("mesh-provisioner")
mt.Until(t, func() error {
if conn.Membership("mesh-provisioner") == nil {
return errors.New("not issued yet")
}
return nil
})
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "b.m2.f2"); err != nil || e == nil || e.Key != "b.m2.f2" {
t.Fatalf("get from the second node's bucket: %+v, %v", e, err)
}
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "q.m.f"); err != nil || e != nil {
t.Fatalf("a key of a node with no bucket should read as none: %+v, %v", e, err)
}
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "z.m3.f3"); err != nil || e != nil {
t.Fatalf("the look-alike bucket was read: %+v, %v", e, err)
}
if _, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "nonode"); err == nil {
t.Fatal("a key naming no node should be refused")
}
if keys, err := conn.StateKeys("mesh-provisioner", "mesh-host.composed"); err != nil || !reflect.DeepEqual(keys, []string{"a.m1.f1", "b.m2.f2"}) {
t.Fatalf("keys across the buckets: %v, %v", keys, err)
}
if _, err := conn.StatePut("mesh-provisioner", "mesh-host.composed", "a.m1.f9", []byte(`{}`)); err == nil {
t.Fatal("a per-node state is read only")
}
var mu sync.Mutex
var current, live []string
stop, err := conn.StateWatch("mesh-provisioner", "mesh-host.composed", "", func(ch bus.StateChange) error {
mu.Lock()
defer mu.Unlock()
if ch.Current {
current = append(current, ch.Key)
} else {
live = append(live, ch.Key)
}
return nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
mu.Lock()
sort.Strings(current)
if !reflect.DeepEqual(current, []string{"a.m1.f1", "b.m2.f2"}) {
t.Fatalf("current values: %v", current)
}
mu.Unlock()
// A third node's bucket, created during the watch, and a change in a first one.
c := mesh.Bucket(t, "composed-c")
if _, err := c.Put("c.m4.f4", []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
if _, err := a.Put("a.m1.f5", []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
mt.Until(t, func() error {
mu.Lock()
defer mu.Unlock()
got := strings.Join(append(append([]string(nil), current...), live...), ",")
for _, want := range []string{"a.m1.f5", "c.m4.f4"} {
if !strings.Contains(got, want) {
return fmt.Errorf("%s not heard yet: current %v, changes %v", want, current, live)
}
}
return nil
})
}
+115 -1
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -32,7 +33,7 @@ func TestAWatchHoldsWhileItsCurrentValuesKeepArriving(t *testing.T) {
for i := 0; i < older; i++ {
keys = append(keys, fmt.Sprintf("novox/mesh-catalog.%d.%040x.status=mesh/check", 400+i, i))
}
for _, member := range []string{"novox/mesh-sdk.76.34f210e5", "novox/mesh-controller.453.c06f75d0", "novox/mesh-delivery.33.28ee22ef"} {
for _, member := range []string{"novox/mesh-sdk.76.34f210e5", "novox/mesh-placement.453.c06f75d0", "novox/mesh-delivery.33.28ee22ef"} {
keys = append(keys, member+strings.Repeat("0", 32)+".status=mesh/check")
}
for _, k := range keys {
@@ -116,6 +117,50 @@ func TestAWatchIsRefusedWhenNoCurrentValueIsTakenWithinTheBound(t *testing.T) {
}
}
// **A membership the grants part wrote in its old wrapper is served whole** (issue 774). Until 2026-10-30 the part wrote
// its pass record, the membership inside, into its state memberships; the tool runner read the value as a bare
// membership, and served no tool and no state. This is what the live mesh held: read through mesh-sdk's codec, its tools
// are answered where the membership says and its state is reached. A held-back one's wrapper is refused, and the module has
// no membership then (774/S32/AC2: no stream is read in its place).
func TestAMembershipTheGrantsPartWroteWrappedIsServed(t *testing.T) {
mesh := mt.New(t)
mesh.Bucket(t, "gitea_repos")
gitea := mt.MembershipOf("gitea", "anchor", false, nil)
gitea.State = []bus.StateIssued{{Name: "repos", Bucket: "gitea_repos", Writes: true}}
mesh.IssueInGrantsWrapped(t, gitea, "published")
// The old wrapper of a membership the part held back carries its unpublished composition: refused, never served.
heldBack := mt.MembershipOf("pages", "anchor", true, nil)
mesh.IssueInGrantsWrapped(t, heldBack, "not published, a shadow difference: publishing would lose the last holder of x")
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
var logged []string
conn.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
t.Cleanup(conn.Close)
conn.Follow("gitea")
mt.Until(t, func() error {
if m := conn.Membership("gitea"); m == nil || len(m.Serves) == 0 {
return fmt.Errorf("gitea's membership is %+v, want the one inside the wrapper", m)
}
return nil
})
if got, want := fmt.Sprint(conn.ServedOn("gitea", "gitea_list")), "[{mesh.mod.gitea.tool.gitea_list.anchor }]"; got != want {
t.Errorf("gitea_list is served on %s, want %s", got, want)
}
if _, err := conn.StatePut("gitea", "repos", "novox.hq", []byte(`{"v":1}`)); err != nil {
t.Fatalf("the state the membership issues is refused: %v", err)
}
if e, err := conn.StateGet("gitea", "repos", "novox.hq"); err != nil || e == nil || string(e.Value) != `{"v":1}` {
t.Errorf("the state reads back %+v, %v", e, err)
}
conn.Follow("pages")
time.Sleep(300 * time.Millisecond)
if m := conn.Membership("pages"); m != nil || !strings.Contains(strings.Join(logged, "\n"), "no membership issued for pages") {
t.Errorf("a held-back membership's wrapper was served or not said: %+v, %q", m, logged)
}
}
// owedWatcher is the gitea module's view of the checker's owed: the bucket, and a runtime connection
// that serves gitea with the owed issued for reading.
func owedWatcher(t *testing.T) (*bus.Conn, interface {
@@ -142,3 +187,72 @@ func owedWatcher(t *testing.T) (*bus.Conn, interface {
})
return conn, kv
}
// **A watch given up, or stopped, hands over nothing more** (issue 774). The runtime's watch stopped its watcher
// when the current values did not arrive within the bound, but the watcher still held values, and its goroutine
// went on handing them over: to a watch id the bundle no longer held (on the provisioner, 51 failed watches were
// followed by 1257 orphan changes). Each row blocks the bundle on its first value, gives the watch up, lets the
// bundle go and counts what it is still handed.
func TestAWatchGivenUpHandsOverNothingMore(t *testing.T) {
for _, tc := range []struct {
name string
refused bool // the bound gives it up, or the caller stops it
}{
{"refused at the bound", true},
{"stopped by its caller", false},
} {
t.Run(tc.name, func(t *testing.T) {
was := bus.StateTimeout
bus.StateTimeout = 300 * time.Millisecond
t.Cleanup(func() { bus.StateTimeout = was })
conn, kv := owedWatcher(t)
put := func(i int) {
if _, err := kv.Put(fmt.Sprintf("novox/hq.%d.%040x.status=mesh/check", i+1, i), []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
}
if tc.refused {
for i := 0; i < 4; i++ {
put(i)
}
}
var calls atomic.Int32
first := make(chan struct{}, 1)
release := make(chan struct{})
deliver := func(ch bus.StateChange) error {
if calls.Add(1) == 1 {
first <- struct{}{}
<-release
}
return nil
}
var stop func()
if tc.refused {
if _, err := conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", deliver); err == nil {
t.Fatal("the watch whose bundle took nothing was not refused")
}
} else {
var err error
if stop, err = conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", deliver); err != nil {
t.Fatal(err)
}
for i := 0; i < 4; i++ {
put(i)
}
select {
case <-first:
case <-time.After(5 * time.Second):
t.Fatal("the bundle was handed no change")
}
stop()
}
// Let the watcher's server side deliver what it held, then free the bundle.
time.Sleep(300 * time.Millisecond)
close(release)
time.Sleep(500 * time.Millisecond)
if n := calls.Load(); n != 1 {
t.Fatalf("the bundle was handed %d changes of a watch given up, want 1 (the one in hand)", n)
}
})
}
}
+4 -4
View File
@@ -14,7 +14,7 @@ import (
// A module's own stream, as the runtime publishes to it and hears it (novox/hq ADR 0363 §8, issue 725).
//
// A module may declare a stream of its own; the controller creates it and says, in the module's
// A module may declare a stream of its own; the mesh creates it and says, in the module's
// membership, which of its events go there and on which subject (`streams`), and, in the membership of
// each module granted to hear it, which durable consumer on it to bind (`hears`). The runtime publishes and
// binds exactly as the membership says. A membership naming neither is what every membership was before:
@@ -30,7 +30,7 @@ type StreamIssued struct {
}
// HeardIssued is one module stream an assignment hears: the stream and the durable consumer on it the
// runtime binds. The consumer is the controller's to make, to filter and to delete.
// runtime binds. The consumer is the mesh's to make, to filter and to delete.
type HeardIssued struct {
Stream string `json:"stream"`
Consumer string `json:"consumer"`
@@ -73,7 +73,7 @@ func eventMatches(pattern, event string) bool {
}
// RebindEvery is how often a consumer the membership names and the bus did not let the runtime bind is
// tried again: the controller makes the consumer and sends the user list carrying its grant in the
// tried again: the mesh makes the consumer and sends the user list carrying its grant in the
// composition that issues the membership, so one may arrive before the other (ADR 0363 §6).
var RebindEvery = 30 * time.Second
@@ -294,7 +294,7 @@ func (h *hearing) reconcile() {
}
}
// **A loop the bus ended is no longer reading** (nats.go ends a pull loop on its own when its consumer is
// deleted): the controller deletes and makes again under the same name a restricted consumer it cannot
// deleted): the mesh deletes and makes again under the same name a restricted consumer it cannot
// narrow (ADR 0363 §6), and the membership, unchanged, says nothing — so it is bound again here.
// **A loop the bus did not end is asked about**: nats.go learns of a deleted consumer only from the answer
// to a pull it has pending, and a consumer deleted and made again between two pulls, or while none is
+62
View File
@@ -160,6 +160,68 @@ func TestAModuleHearsEveryStreamItsMembershipNames(t *testing.T) {
}
return nil
})
// The membership as the grants part keeps it (ADR 0571 clause 7, 10): read first and applied live, and nothing
// published on the controller's subject is heard.
granted := reader
granted.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "from-grants"}}
mesh.Issue(t, granted)
again, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
again.Logf = func(string, ...any) {}
t.Cleanup(again.Close)
again.Follow("reader")
consumerOf := func() string {
if m := again.Membership("reader"); m != nil && len(m.Hears) == 1 {
return m.Hears[0].Consumer
}
return ""
}
if got := consumerOf(); got != "from-grants" {
t.Fatalf("the membership read at the follow names %q, want the grants part's", got)
}
granted.Hears[0].Consumer = "from-grants-2"
mesh.Issue(t, granted)
mt.Until(t, func() error {
if got := consumerOf(); got != "from-grants-2" {
return fmt.Errorf("the membership names %q, want the grants part's change", got)
}
return nil
})
// 774/S32/AC1: a copy said on the controller's subject (mesh.assignment) changes nothing; there is no stream to read it from.
streamed := reader
streamed.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "from-subject"}}
mesh.Publish(t, "mesh.assignment.anchor.reader", streamed)
again.Flush()
time.Sleep(200 * time.Millisecond)
if got := consumerOf(); got != "from-grants-2" {
t.Fatalf("a membership said on the controller's subject was applied: the membership names %q", got)
}
// Review g1 round 1: memberships that cannot be read at the follow (a bus not ready after its boot) are read again until
// they can be; the follow, which runs once per module, does not give the module up for good.
was := bus.GrantsRetry
bus.GrantsRetry = 50 * time.Millisecond
t.Cleanup(func() { bus.GrantsRetry = was })
mesh.DropGrants(t)
late, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
late.Logf = func(string, ...any) {}
t.Cleanup(late.Close)
late.Follow("reader")
if m := late.Membership("reader"); m != nil {
t.Fatalf("a membership was served from a state not there: %+v", m)
}
mesh.Issue(t, granted)
mt.Until(t, func() error {
if m := late.Membership("reader"); m == nil || len(m.Hears) != 1 || m.Hears[0].Consumer != "from-grants-2" {
return fmt.Errorf("the membership is %+v, want the one written once the state is there", m)
}
return nil
})
}
// novox/hq ADR 0363 §6 (issue 725): a consumer the membership names before the controller made it is bound once
+187 -83
View File
@@ -17,6 +17,9 @@ package console
// session — so a tool that arrived a minute ago is found without the client reconnecting.
import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"context"
"encoding/json"
"fmt"
@@ -47,7 +50,7 @@ const (
// searchCap is how many matches a search answers before it says how many more there were.
const searchCap = 25
const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e.g. `mesh-controller.nodes`); " +
const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e.g. `mesh-placement.placement`); " +
"`<node>/<seat>.<verb>` for a seat every machine holds (e.g. `ace/node-packet-filter.rules`); " +
"`<node>/<module>.<tool>` for a module on one machine (e.g. `novox/postgres.postgres_list_databases`); " +
"and `<module>.<tool>` also for a module whose instances are interchangeable."
@@ -128,13 +131,17 @@ type index struct {
NotAnswering []string
Listing *Listing // the flat catalogue the call path resolves subjects with
Discovery announce.Discovery
// Recorded is where the controller's records place each module that declares tools.
// Recorded is where placement places each module that declares tools.
Recorded map[string][]string
// Unheard is every runtime whose answer this index lacks: it said it was there and did not say
// what it serves in time, or it answered before and not now. An address it might answer is never
// called missing while it is here (2026-10-05).
Unheard []unheard
// Replaces is what each verb and tool replaces, from the controller's records (novox/hq ADR 0245):
// Placed is the state seat-holders as the seat mesh-placement's holder wrote it (novox/hq ADR 0571 clause 4), by its
// keys `<seat>` and `<seat>.<node>`, read with delivery.HoldersOf; PlacedWhy says why it could not be read.
Placed map[string][]byte
PlacedWhy string
// Replaces is what each verb and tool replaces (novox/hq ADR 0245):
// `seat:<seat>.<verb>` and `<module>.<tool>`.
Replaces map[string][]string
// Earlier is every runtime not heard this round whose last answer, within RuntimeForgotten, this
@@ -224,42 +231,8 @@ func findTool(tools []Tool, name string) *Tool {
return nil
}
// controllerOutput is a controller seat verb's answer: the command's printed output.
func controllerOutput(conn *bus.Conn, verb string) (string, error) {
got, err := conn.Ask("seat:mesh-controller."+verb, map[string]any{}, "")
if err != nil {
return "", err
}
var r struct {
Output string `json:"output"`
OK *bool `json:"ok"`
}
if json.Unmarshal(got.Result, &r) != nil {
return "", fmt.Errorf("mesh-controller.%s answered something that is not its output", verb)
}
if r.OK != nil && !*r.OK {
return "", fmt.Errorf("mesh-controller.%s: %s", verb, strings.TrimSpace(r.Output))
}
return r.Output, nil
}
// jsonIn is the JSON document a command printed, after any lines it said first: a seat verb runs
// the controller's command, and a command may warn before it answers.
func jsonIn(output string) string {
t := strings.TrimSpace(output)
if strings.HasPrefix(t, "{") || strings.HasPrefix(t, "[") {
return t
}
for _, open := range []string{"\n{", "\n["} {
if i := strings.Index(output, open); i >= 0 {
return strings.TrimSpace(output[i+1:])
}
}
return ""
}
// recordedModule is a module as the controller's records hold it (`module list --json`): where the
// mesh assigned it, and whether it declares tools — what should announce itself, and where.
// recordedModule is a module as the bus holds it: where the placement seat places it, and whether its manifest declares
// tools — what should announce itself, and where (novox/hq ADR 0571).
type recordedModule struct {
Module string `json:"module"`
On []string `json:"on"`
@@ -268,13 +241,8 @@ type recordedModule struct {
Replaces map[string][]string `json:"replaces,omitempty"`
}
// recordedMachine is a machine as the controller's records hold it (`node list --json`).
type recordedMachine struct {
Name string `json:"name"`
}
// indexOn asks the mesh what it holds (novox/hq ADR 0197): what answers, from every runtime's own
// announcement on the bus — one `$SRV.INFO` request — and what should, from the controller's records
// announcement on the bus — one `$SRV.INFO` request — and what should, from the indexer's records
// read as JSON. Nothing is inferred from a roster and nothing is parsed from print.
//
// `held` is what the console holds already, so an instance whose announcement is unchanged is not
@@ -282,19 +250,17 @@ type recordedMachine struct {
// index holds instead, marked with its age (novox/hq issue 554).
func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery) []earlier) (*index, error) {
var wg sync.WaitGroup
var nodesOut, modulesOut string
var toolsAnswer json.RawMessage
wg.Add(3)
go func() { defer wg.Done(); nodesOut, _ = controllerOutput(conn, "nodes") }()
go func() { defer wg.Done(); modulesOut, _ = controllerOutput(conn, "modules") }()
// The seats' verbs as the records define them, for what each replaces (novox/hq ADR 0245): answered
// in-process from the records, never a command.
go func() {
defer wg.Done()
if got, err := conn.Ask("seat:mesh-controller.tools", map[string]any{}, ""); err == nil {
toolsAnswer = got.Result
}
}()
var placed map[string][]byte
var placedErr error
var recorded []recordedModule
var known []string
var recordsWhy []string
wg.Add(2)
// Which modules there are and where they are placed, and which machines the mesh has, from the bus (ADR 0571
// clauses 4 and 5): the indexer's modules and manifests and the placement seat's placements, never any other record.
go func() { defer wg.Done(); recorded, known, recordsWhy = recordsOn(conn) }()
// Who holds which seat on which node, from placement (ADR 0571 clause 4): no other call.
go func() { defer wg.Done(); placed, placedErr = seatHoldersOn(conn) }()
d, err := announce.Gather(conn, held)
wg.Wait()
if err != nil {
@@ -333,6 +299,10 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
l := x.Listing
x.Discovery = d
x.Stale, x.Earlier = stale, earlierAt
x.Placed = placed
if placedErr != nil {
x.PlacedWhy = placedErr.Error()
}
for _, s := range d.Silent {
x.Unheard = append(x.Unheard, unheard{Runtime: s.Name, Machine: s.Machine,
Why: fmt.Sprintf("it answered PING; what it serves did not arrive within %s", announce.Patience)})
@@ -340,25 +310,24 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
// What should have answered: every assignment of a module that declares tools. Silence is named;
// a module with no tools is never a name here.
var recorded []recordedModule
recordedOK := json.Unmarshal([]byte(jsonIn(modulesOut)), &recorded) == nil
x.Replaces = replacesOf(toolsAnswer, recorded)
if recordedOK {
for _, m := range recorded {
if !m.Tools {
continue
}
x.Recorded[m.Module] = append([]string{}, m.On...)
for _, n := range m.On {
// A holder of a seat held once for the mesh announces no machine — the seat is the mesh's,
// not a machine's — so what it announced without one answers for wherever it is assigned.
if !announced[m.Module][n] && !announced[m.Module][""] {
l.NotAnswering = append(l.NotAnswering, m.Module+" on "+n)
}
x.Replaces = replacesOf(catalogue.DefaultSeats(), recorded)
for _, m := range recorded {
if !m.Tools {
continue
}
x.Recorded[m.Module] = append([]string{}, m.On...)
for _, n := range m.On {
// A holder of a seat held once for the mesh announces no machine — the seat is the mesh's,
// not a machine's — so what it announced without one answers for wherever it is assigned.
if !announced[m.Module][n] && !announced[m.Module][""] {
l.NotAnswering = append(l.NotAnswering, m.Module+" on "+n)
}
}
} else {
l.NotAnswering = append(l.NotAnswering, "mesh-controller (its records of the modules did not answer, so what is missing cannot be said)")
}
l.NotAnswering = append(l.NotAnswering, recordsWhy...)
if x.PlacedWhy != "" {
l.NotAnswering = append(l.NotAnswering, delivery.SeatHoldersRead+" (who holds which seat by placement could not be read: "+
x.PlacedWhy+")")
}
for _, u := range x.Unheard {
l.NotAnswering = append(l.NotAnswering, "the runtime "+sayUnheard([]unheard{u}))
@@ -366,13 +335,8 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
sort.Strings(l.NotAnswering)
x.NotAnswering = l.NotAnswering
var known []recordedMachine
if json.Unmarshal([]byte(jsonIn(nodesOut)), &known) == nil {
for _, n := range known {
if n.Name != "" {
machines[n.Name] = true
}
}
for _, n := range known {
machines[n] = true
}
for n := range machines {
x.Machines = append(x.Machines, n)
@@ -381,6 +345,138 @@ func indexOn(conn *bus.Conn, held announce.Known, before func(announce.Discovery
return x, nil
}
// stateOn reads a state the runtime's membership reads, whole: its current values by key, a deleted key absent.
func stateOn(conn *bus.Conn, name string) (map[string][]byte, error) {
var mu sync.Mutex
keys := map[string][]byte{}
stop, err := conn.StateWatch(conn.Credential().Module, name, "", func(c bus.StateChange) error {
if c.Current && c.Op == "put" {
mu.Lock()
keys[c.Key] = c.Value
mu.Unlock()
}
return nil
})
if err != nil {
return nil, err
}
stop()
mu.Lock()
defer mu.Unlock()
return keys, nil
}
// seatHoldersOn reads the state seat-holders whole (novox/hq ADR 0571 clause 4), as delivery.SeatHoldersRead.
func seatHoldersOn(conn *bus.Conn) (map[string][]byte, error) {
return stateOn(conn, delivery.SeatHoldersRead)
}
// The indexer's states the console reads (novox/hq ADR 0571 clauses 4 and 5): every module, and each manifest by its
// digest, by the names a reader's manifest declares.
var (
modulesRead = delivery.IndexerModule + ".modules"
manifestsRead = delivery.IndexerModule + ".manifests"
)
// manifestFacts is what the console takes of a manifest: the tools it declares and what each replaces. A manifest's
// digest never names another, so what is read once is kept.
type manifestFacts struct {
Tools []string `json:"tools"`
Replaces map[string][]string `json:"replaces"`
}
var manifestsKept sync.Map // digest → manifestFacts
// recordsOn is every module the indexer holds, with where the placement seat places it and whether it declares tools,
// and every machine placement names (delivery.ModuleNames, delivery.MachineNames). What cannot be read is said, one line
// per state, for the listing's not answering.
func recordsOn(conn *bus.Conn) (recorded []recordedModule, machines []string, why []string) {
placements, err := stateOn(conn, delivery.PlacementRead)
if err != nil {
why = append(why, delivery.PlacementRead+" (where modules are placed could not be read, so what is missing cannot "+
"be said: "+err.Error()+")")
}
machines = delivery.MachineNames(placements)
on := map[string][]string{}
// The placement's keys in either form, a node key and its module keys or the form before (research 072 row 37).
assembled, err := delivery.PlacementsOf(placements)
if err != nil {
why = append(why, delivery.PlacementRead+" (not every key is a placement: "+err.Error()+")")
}
for _, node := range machines {
for _, m := range assembled[node].Modules {
on[m.Module] = append(on[m.Module], node)
}
}
modules, err := stateOn(conn, modulesRead)
if err != nil {
return nil, machines, append(why, modulesRead+" (which modules there are could not be read: "+err.Error()+")")
}
names, err := delivery.ModuleNames(modules)
if err != nil {
return nil, machines, append(why, modulesRead+" ("+err.Error()+")")
}
for _, name := range names {
r := recordedModule{Module: name, On: on[name]}
if m, err := delivery.DecodeModule(modules[name]); err == nil && m.Manifest != "" {
facts, err := manifestOf(conn, m.Manifest)
if err != nil {
why = append(why, name+" (its manifest could not be read, so whether it answers tools is not known: "+err.Error()+")")
} else {
r.Tools, r.Replaces = len(facts.Tools) > 0, facts.Replaces
}
}
recorded = append(recorded, r)
}
return recorded, machines, why
}
// manifestOf is what the console takes of the manifest of a digest, read from the indexer's manifests once.
func manifestOf(conn *bus.Conn, digest string) (manifestFacts, error) {
if kept, ok := manifestsKept.Load(digest); ok {
return kept.(manifestFacts), nil
}
entry, err := conn.StateGet(conn.Credential().Module, manifestsRead, digest)
if err != nil {
return manifestFacts{}, err
}
if entry == nil {
return manifestFacts{}, fmt.Errorf("the indexer holds no manifest %s", digest)
}
m, err := delivery.DecodeManifest(entry.Value)
if err != nil {
return manifestFacts{}, err
}
var facts manifestFacts
if err := json.Unmarshal([]byte(m.JSON), &facts); err != nil {
return manifestFacts{}, err
}
manifestsKept.Store(digest, facts)
return facts, nil
}
// placedHolders is who holds seat (on node, for a seat held per machine) by placement, or none.
func (x *index) placedHolders(seat, node string) []delivery.SeatHolder {
if len(x.Placed) == 0 {
return nil
}
hs, err := delivery.HoldersOf(x.Placed, seat, node)
if err != nil || len(hs) > 0 || node != "" {
return hs
}
// A seat held per machine, asked without one: every node's key of it.
var all []delivery.SeatHolder
for key := range x.Placed {
if s, n, ok := strings.Cut(key, "."); ok && s == seat {
if more, err := delivery.HoldersOf(x.Placed, seat, n); err == nil {
all = append(all, more...)
}
}
}
sort.Slice(all, func(i, j int) bool { return all[i].Node+"/"+all[i].Module < all[j].Node+"/"+all[j].Module })
return all
}
func containsHolder(hs []holder, h holder) bool {
for _, x := range hs {
if x == h {
@@ -606,6 +702,14 @@ func resolve(x *index, address string) (target, error) {
return target{}, fmt.Errorf("%s is assigned to %s in the mesh's records, but nothing that answered discovery serves it: %s. "+
"Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, strings.Join(on, ", "), why)
}
if hs := x.placedHolders(prefix, node); len(hs) > 0 {
var held []string
for _, h := range hs {
held = append(held, h.Module+" on "+h.Node)
}
return target{}, fmt.Errorf("the seat %s is held by %s by placement, but no holder of it announced itself on the bus. "+
"Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, strings.Join(held, ", "))
}
if len(x.Unheard) > 0 {
return target{}, fmt.Errorf("nothing that answered discovery is called %s, but not every runtime answered: %s — "+
"%s may be theirs. Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, sayUnheard(x.Unheard), prefix)
@@ -711,7 +815,7 @@ func firstLine(s string) string {
// the machine, `node` is taken out: the machine is in the address. Where it does not — a seat held
// once for the mesh, or an interchangeable module — `node` is the tool's own argument, and is kept.
//
// **Taking it out everywhere made the mesh's own verbs lie** (novox/hq issue 244): the controller's
// **Taking it out everywhere made the mesh's own verbs lie** (novox/hq issue 244): the old controller verbs
// `push`, `plan`, `assign`, `pin` and `settings` take the machine they act on as `node`, and were
// described without it — `push` as taking nothing at all, so a push naming one machine arrived as a
// push of every machine behind.
+33 -43
View File
@@ -70,35 +70,34 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
t.Cleanup(stop)
}
// The controller: its records as JSON, as its seat verbs answer them, and its own seat's verbs
// announced on the bus like every runtime's.
controller := connect(t, "mesh-controller", "bench")
out := func(s string) map[string]any { return map[string]any{"output": s, "ok": true} }
serve := func(verb string, answer func() any) {
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
return answer(), nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
// The controller: its own seat's verbs announced on the bus like every runtime's; its records are asked by no one.
controller := connect(t, "mesh-placement", "bench")
// Its verb nodes, called through the console as any seat's verb is.
stopNodes, err := controller.HandleSubject("mesh.seat.mesh-placement.tool.nodes", func(json.RawMessage) (any, error) {
return map[string]any{"ok": true, "output": `[{"name":"bench","mode":"converged"},{"name":"desk","mode":"converged"}]`}, nil
})
if err != nil {
t.Fatal(err)
}
serve("nodes", func() any {
return out("the bus's user list leaves out 2 user(s)\n" +
`[{"name":"bench","heard":"3m ago","mode":"converged","id":"1f2e"},{"name":"desk","heard":"here","mode":"converged","id":"9a8b"}]`)
t.Cleanup(stopNodes)
// What the mesh holds, as the indexer and placement write it (ADR 0571 clauses 4, 5): never the controller's records.
tools := map[string][]string{"alpha": {"one", "two"}, "beta": {"three"}, "gamma": {"given"}, "delta": nil,
"epsilon": {"seven"}, "mesh-placement": {"nodes"}}
mesh.Placed(t, tools, map[string][]string{"desk": {"alpha", "beta", "delta"}, "bench": {"epsilon", "mesh-placement"}})
console := mt.MembershipOf("node-tools", "desk", false, nil)
console.State = mt.ConsoleReads()
mesh.Issue(t, console)
mt.Until(t, func() error {
if m := nodeTools.Membership("node-tools"); m == nil || len(m.State) == 0 {
return fmt.Errorf("the console's reads are not issued yet")
}
return nil
})
var gammaOn atomic.Value
gammaOn.Store("[]")
serve("modules", func() any {
return out(fmt.Sprintf(`[{"module":"alpha","on":["desk"],"tools":true},{"module":"beta","on":["desk"],"tools":true},`+
`{"module":"gamma","on":%s,"tools":true},{"module":"delta","on":["desk"],"tools":false},`+
`{"module":"epsilon","on":["bench"],"tools":true},{"module":"mesh-controller","on":["bench"],"tools":true}]`, gammaOn.Load().(string)))
})
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-controller", ID: "bench"}, func() []announce.Endpoint {
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller", Scope: "mesh",
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-placement", ID: "bench"}, func() []announce.Endpoint {
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-placement", Seat: "mesh-placement", Scope: "mesh",
// As the live controller announces: a mesh seat's holder names no machine.
Tool: "nodes", Node: "", Description: "Every machine the mesh knows.", Schema: json.RawMessage(`{}`),
Subject: "mesh.seat.mesh-controller.tool.nodes"}}
Subject: "mesh.seat.mesh-placement.tool.nodes"}}
})
if err != nil {
t.Fatal(err)
@@ -132,13 +131,13 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
// The overview names the mesh's seats, the machines' seats and the machines.
overview, isErr := call(t, endpoint, "mesh_overview", nil)
if isErr || !strings.Contains(overview, "mesh-controller.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
if isErr || !strings.Contains(overview, "mesh-placement.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
!strings.Contains(overview, `"bench"`) || !strings.Contains(overview, `"desk"`) {
t.Errorf("overview: %s", overview)
}
// Silence is named only where tools should have answered: epsilon declares tools on bench and
// nothing there announced it; delta declares none and is never a name.
if strings.Contains(overview, "mesh-controller on bench") {
if strings.Contains(overview, "mesh-placement on bench") {
t.Errorf("a mesh seat's holder that announced no machine is called silent on its machine:\n%s", overview)
}
if !strings.Contains(overview, "epsilon on bench") || strings.Contains(overview, "delta") {
@@ -151,7 +150,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
}
// A mesh seat, a node seat, an assignment and an interchangeable module, each by address.
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-controller.nodes"}); isErr || !strings.Contains(got, "converged") {
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-placement.nodes"}); isErr || !strings.Contains(got, "converged") {
t.Errorf("mesh seat: %s", got)
}
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "desk/node-shelf.list"}); isErr || !strings.Contains(got, `"a"`) {
@@ -167,11 +166,11 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
// Refused, by name, where the address does not say enough or says the wrong thing.
for address, want := range map[string]string{
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
"desk/mesh-controller.nodes": "held once for the whole mesh",
"bench/beta.three": "beta does not run on bench; it runs on desk",
"nonsense": "is not an address",
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
"desk/mesh-placement.nodes": "held once for the whole mesh",
"bench/beta.three": "beta does not run on bench; it runs on desk",
"nonsense": "is not an address",
} {
got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": address})
if !isErr || !strings.Contains(got, want) {
@@ -202,7 +201,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
t.Fatalf("gamma was found before it served: %s", got)
}
mesh.Issue(t, mt.MembershipOf("gamma", "desk", false, nil))
gammaOn.Store(`["desk"]`)
mesh.Placed(t, tools, map[string][]string{"desk": {"alpha", "beta", "delta", "gamma"}, "bench": {"epsilon", "mesh-placement"}})
late := connect(t, "node-tools", "desk")
stopLate, err := runtime.Run(late, []runtime.Served{{Module: "gamma", Entrypoints: []string{mt.Fixture("env-gamma.serve.mjs")}}},
nil, (&mt.Logs{}).Logf)
@@ -233,12 +232,3 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
t.Errorf("an old name: %s", got)
}
}
func TestTheControllersRecordsAreReadAsJSON(t *testing.T) {
if got := jsonIn("a warning printed first\n[{\"name\":\"ace\"}]"); got != `[{"name":"ace"}]` {
t.Errorf("an array after a warning: %q", got)
}
if got := jsonIn(`{"seats":[]}`); got != `{"seats":[]}` {
t.Errorf("an object: %q", got)
}
}
-9
View File
@@ -118,9 +118,6 @@ var (
timedOut = regexp.MustCompile(`(?i)timeout`)
)
// controllerSeat is the seat whose holder keeps what came of every call (novox/hq issue 265).
const controllerSeat = "mesh-controller"
// whyItFailed says why a call failed, so the remedy is in the words.
func whyItFailed(key string, err error) string {
if err == nil {
@@ -145,12 +142,6 @@ func whyItFailed(key string, err error) string {
said := key + " did not answer within " + bus.RequestTimeout.String() + ". That is not a failure: " +
"something is serving it, so it may still be running and may already have done what was " +
"asked — check before repeating a call that changes the mesh."
if strings.HasPrefix(strings.TrimPrefix(key, "seat:"), controllerSeat+".") {
// The controller answers every call within seconds, with an id while it runs, so silence
// from it is an answer lost on the way — which it keeps.
said += " The controller answers every call within seconds, so its answer was lost on the " +
"way; " + controllerSeat + ".calls lists its recent calls and what came of each."
}
return said
}
return key + " failed: " + msg
+13 -7
View File
@@ -3,6 +3,7 @@ package console
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
@@ -56,14 +57,19 @@ func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
}
defer stop()
// The controller's records (ADR 0197): ghost declares tools on desk and nothing announces it.
controller := connect(t, "mesh-controller", "")
stopSeat, _ := controller.HandleSubject("mesh.seat.mesh-controller.tool.modules", func(json.RawMessage) (any, error) {
return map[string]any{"ok": true, "output": `[{"module":"alpha","on":["desk"],"tools":true},` +
`{"module":"beta","on":["desk"],"tools":true},{"module":"ghost","on":["desk"],"tools":true}]`}, nil
// What the mesh holds, as the indexer and placement write it (ADR 0571): ghost declares tools on desk and nothing
// announces it.
mesh.Placed(t, map[string][]string{"alpha": {"one", "two"}, "beta": {"three"}, "ghost": {"boo"}},
map[string][]string{"desk": {"alpha", "beta", "ghost"}})
console := mt.MembershipOf("node-tools", "desk", false, nil)
console.State = mt.ConsoleReads()
mesh.Issue(t, console)
mt.Until(t, func() error {
if m := nodeTools.Membership("node-tools"); m == nil || len(m.State) == 0 {
return fmt.Errorf("the console's reads are not issued yet")
}
return nil
})
defer stopSeat()
controller.Flush()
flat := NewSurface(nodeTools, "desk.node-tools")
flat.Flat = true // the whole catalogue, as before ADR 0195: still reachable, no longer announced
@@ -26,7 +26,7 @@ func TestTheManifestNamesTheFirstFamiliesOneByOne(t *testing.T) {
t.Fatal(err)
}
want := []string{"mesh-knowledge.reviewed", "mesh-delivery-gate.transition", "gitea.forge.status",
"mesh-controller.checked", "operator-channel.decided.claude-code"}
"mesh-self-check.checked", "operator-channel.decided.claude-code"}
if m.Module != "node-tools" || !reflect.DeepEqual(m.Waits, want) {
t.Errorf("%s waits on %v, want %v", m.Module, m.Waits, want)
}
+36 -1
View File
@@ -6,6 +6,8 @@ package console
import (
"context"
"encoding/json"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"os"
"strings"
"sync"
@@ -60,6 +62,34 @@ type Surface struct {
Flat bool
}
// MethodSeats is the console's JSON-RPC method beside MCP's that answers every seat's verbs and holders, in the shape
// the old `tools` answered: `{seats: [{seat, scope, tools: [{name, description, input, replaces}], holders:
// [{module, node}]}], holders_why}`.
const MethodSeats = "mesh/seats"
// seatsAnswer is MethodSeats' answer: mesh-sdk's seat definitions, with each seat's holders by placement.
func (s *Surface) seatsAnswer() map[string]any {
placed, err := seatHoldersOn(s.conn)
x := &index{Placed: placed}
seats := []map[string]any{}
for _, st := range catalogue.DefaultSeats() {
holders := x.placedHolders(st.Name, "")
if holders == nil {
holders = []delivery.SeatHolder{}
}
tools := st.Serves
if tools == nil {
tools = []catalogue.Verb{}
}
seats = append(seats, map[string]any{"seat": st.Name, "scope": st.Scope, "tools": tools, "holders": holders})
}
out := map[string]any{"seats": seats}
if err != nil {
out["holders_why"] = "who holds which seat by placement could not be read: " + err.Error()
}
return out
}
// NewSurface is the surface over a connection, as `who`.
func NewSurface(conn *bus.Conn, who string) *Surface {
return &Surface{conn: conn, who: who, Flat: os.Getenv("MESH_CONSOLE_FLAT") == "1"}
@@ -125,6 +155,11 @@ func (s *Surface) HandleContext(ctx context.Context, r Request) *Reply {
return nil
}
return answer(r.ID, map[string]any{})
case MethodSeats:
// The seats' contract and who holds each, for a client on this machine that has no seat definitions of its own
// (the TypeScript `mesh`): read in-process from mesh-sdk and from placement, never asked of anything else
// (novox/hq ADR 0571 clauses 4 and 5).
return answer(r.ID, s.seatsAnswer())
case "tools/list":
if !s.Flat {
return answer(r.ID, map[string]any{"tools": discovery()})
@@ -303,7 +338,7 @@ func (s *Surface) instructions() string {
"that serves it; what may be called was fixed when this account was issued, so a " +
"refusal means the account, not the tool. The list is what the running modules " +
"answered, plus every role's tools from the mesh's records — the mesh's own verbs " +
"(mesh-controller.status, .push, .assign …) among them; a module that did not answer " +
"(mesh-delivery.deliveries, mesh-placement.propose …) among them; a module that did not answer " +
"is named in the list's _meta and can still be called by <module>.<tool>."
}
return "The tools of a Novox mesh, reached as " + s.who + ", found by address rather than listed " +
@@ -11,9 +11,9 @@ import (
// 2026-10-05 the controller's push was described as taking nothing, and a push naming one machine
// ran as a push of every machine behind.
func TestAMeshSeatsNodeIsItsOwnArgument(t *testing.T) {
push := Tool{Module: "mesh-controller", Name: "push", Seat: true, Scope: "mesh",
push := Tool{Module: "mesh-placement", Name: "push", Seat: true, Scope: "mesh",
Input: json.RawMessage(`{"type":"object","properties":{"node":{"type":"string"},"behind":{"type":"string"}}}`)}
mesh := target{Address: "mesh-controller.push", Tool: push, Seat: true}
mesh := target{Address: "mesh-placement.push", Tool: push, Seat: true}
props := schemaAsPassed(mesh)["properties"].(map[string]any)
if _, has := props["node"]; !has {
+14 -7
View File
@@ -22,16 +22,23 @@ func TestARefusalIsSaidOnlyForTheBussOwn(t *testing.T) {
}
// **A timeout says it is not a failure** (novox/hq issue 265): a push that outlasted the wait had
// pushed, and the caller read "did not answer in time". And from the controller, where its answer is.
// pushed, and the caller read "did not answer in time". No hint names a seat that is gone (novox/hq issue 1126):
// the grammar's example, the handshake's instructions and a timeout's remedy named the controller's verbs.
func TestATimeoutIsNotAFailure(t *testing.T) {
got := whyItFailed("seat:mesh-controller.push", errors.New("timeout"))
for _, want := range []string{"not a failure", "may already have done", "mesh-controller.calls"} {
got := whyItFailed("seat:mesh-delivery.deliveries", errors.New("timeout"))
for _, want := range []string{"not a failure", "may already have done"} {
if !strings.Contains(got, want) {
t.Errorf("a controller timeout does not say %q: %s", want, got)
t.Errorf("a seat's timeout does not say %q: %s", want, got)
}
}
if other := whyItFailed("novox/postgres.postgres_list_databases", errors.New("timeout")); !strings.Contains(other, "not a failure") ||
strings.Contains(other, "calls") {
t.Errorf("a module's timeout: %s", other)
for name, text := range map[string]string{
"grammar": grammar,
"flat instructions": (&Surface{Flat: true, who: "x"}).instructions(),
"deep instructions": (&Surface{who: "x"}).instructions(),
"timeout": got,
} {
if strings.Contains(text, "controller") {
t.Errorf("%s names the controller: %s", name, text)
}
}
}
+48 -7
View File
@@ -2,10 +2,13 @@ package console
import (
"encoding/json"
"fmt"
"strings"
"sync/atomic"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-tools/node-tools/internal/announce"
@@ -52,7 +55,23 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
// Kept for the test: one round of discovery answers every question but mesh_runtimes, which asks anew.
IndexKept, announce.Patience = time.Minute, 2500*time.Millisecond
t.Cleanup(func() { IndexKept, announce.Patience = keptWas, patienceWas })
mt.New(t)
mesh := mt.New(t)
// What the mesh holds, as the indexer and placement write it (ADR 0571): never the controller's records.
mesh.Placed(t, map[string][]string{"slack": {"slack_check"}, "nap": {"doze"}, "nextcloud-client": {"nextcloud_check"}},
map[string][]string{"laptop": {"slack", "nextcloud-client"}, "sleeper": {"nap", "nextcloud-client"},
"desktop": {"nextcloud-client"}, "bench": nil})
// Who holds which seat, by placement (ADR 0571 clause 4), as the seat mesh-placement's holder composes it.
placedKV := mesh.Bucket(t, delivery.SeatHoldersBucket)
for key, sh := range delivery.SeatHoldersOf(map[string]delivery.Placement{"bench": {Node: "bench",
Holds: []delivery.Holding{{Claim: "mesh-wants", Scope: "mesh", Module: "mesh-delivery-assigner"}}}}) {
raw, _ := json.Marshal(sh)
if _, err := placedKV.Put(key, raw); err != nil {
t.Fatal(err)
}
}
consoleMembership := mt.MembershipOf("node-tools", "laptop-console", false, nil)
consoleMembership.State = mt.ConsoleReads()
mesh.Issue(t, consoleMembership)
laptop := connect(t, "node-tools", "laptop")
stopTool, err := laptop.HandleSubject("mesh.mod.slack.tool.slack_check.laptop", func(json.RawMessage) (any, error) {
@@ -76,14 +95,17 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
Subject: "mesh.mod.nextcloud-client.tool.nextcloud_check.desktop"},
})
controller := connect(t, "mesh-controller", "bench")
controller := connect(t, "mesh-placement", "bench")
var toolsAsked atomic.Int32
for verb, output := range map[string]string{
"nodes": `[{"name":"laptop"},{"name":"sleeper"},{"name":"desktop"},{"name":"bench"}]`,
"modules": `[{"module":"slack","on":["laptop"],"tools":true},{"module":"nap","on":["sleeper"],"tools":true},` +
`{"module":"nextcloud-client","on":["laptop","desktop","sleeper"],"tools":true}]`,
// The controller's records are served here only to see that none is asked (ADR 0571 clauses 4 and 5).
"nodes": `[{"name":"laptop"}]`,
"modules": `[]`,
"tools": `{"seats":[]}`,
} {
output := output
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
output, verb := output, verb
stop, err := controller.HandleSubject("mesh.seat.mesh-placement.tool."+verb, func(json.RawMessage) (any, error) {
toolsAsked.Add(1)
return map[string]any{"output": output, "ok": true}, nil
})
if err != nil {
@@ -94,6 +116,13 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
controller.Flush()
console := connect(t, "node-tools", "laptop-console")
console.Follow("node-tools")
mt.Until(t, func() error {
if console.Membership("node-tools") == nil {
return fmt.Errorf("the console's membership is not issued yet")
}
return nil
})
up, err := Serve(NewSurface(console, "laptop.node-tools"), "127.0.0.1:0")
if err != nil {
t.Fatal(err)
@@ -113,8 +142,20 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
t.Errorf("the late runtime's machine lists nothing: %s", machine)
}
// The seats' contract and holders, for a client on this machine with no seat definitions (the TypeScript `mesh`).
seats := post(t, endpoint, `{"jsonrpc":"2.0","id":9,"method":"mesh/seats"}`)
if raw, _ := json.Marshal(seats["result"]); !strings.Contains(string(raw), `"holders":[{"module":"mesh-delivery-assigner","node":"bench"}],"scope":"mesh","seat":"mesh-wants"`) ||
!strings.Contains(string(raw), `"replaces":["journalctl"]`) {
t.Errorf("mesh/seats answered %s", raw)
}
if n := toolsAsked.Load(); n != 0 {
t.Errorf("the console asked the controller's records %d times", n)
}
// The silent one is named, never called missing.
for address, want := range map[string]string{
// A seat whose holder announced nothing is named with its holder by placement, never called missing.
"mesh-wants.read": "the seat mesh-wants is held by mesh-delivery-assigner on bench by placement",
"sleeper/nap.doze": "nap is assigned to sleeper in the mesh's records, but nothing that answered discovery serves it: discovery did not hear in full from node-tools on sleeper (it answered PING",
"sleeper/nextcloud-client.nextcloud_check": "nextcloud-client on sleeper did not answer discovery: discovery did not hear in full from node-tools on sleeper",
} {
+11 -21
View File
@@ -19,7 +19,8 @@ package console
// Ties go to a seat — the place tools belong — then to the shorter name, then by address.
import (
"encoding/json"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"path"
"sort"
"strings"
@@ -265,27 +266,16 @@ func (x *index) search(query string) []hit {
return hits
}
// replacesOf is what every verb and tool replaces, from the controller's records (novox/hq ADR 0245):
// a seat's verbs from its `tools` answer, keyed `seat:<seat>.<verb>`, and a module's own tools from
// its `modules` answer, keyed `<module>.<tool>`. A controller older than the field answers neither,
// and search matches names and descriptions as before.
func replacesOf(toolsAnswer json.RawMessage, modules []recordedModule) map[string][]string {
// replacesOf is what every verb and tool replaces (novox/hq ADR 0245): a seat's verbs from the seat's contract in
// mesh-sdk, keyed `seat:<seat>.<verb>`, read in-process and asked of no part (ADR 0571 clause 5: no part keeps a
// central record of tools); and a module's own tools from its records, keyed `<module>.<tool>`. A verb or a tool with
// no `replaces` is matched by its name and description as before.
func replacesOf(seats []catalogue.Seat, modules []recordedModule) map[string][]string {
out := map[string][]string{}
var seats struct {
Seats []struct {
Seat string `json:"seat"`
Tools []struct {
Name string `json:"name"`
Replaces []string `json:"replaces"`
} `json:"tools"`
} `json:"seats"`
}
if json.Unmarshal(toolsAnswer, &seats) == nil {
for _, s := range seats.Seats {
for _, t := range s.Tools {
if len(t.Replaces) > 0 {
out["seat:"+s.Seat+"."+t.Name] = t.Replaces
}
for _, s := range seats {
for _, v := range s.Serves {
if len(v.Replaces) > 0 {
out["seat:"+s.Name+"."+v.Name] = append([]string(nil), v.Replaces...)
}
}
}
+24 -29
View File
@@ -1,7 +1,8 @@
package console
import (
"encoding/json"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"strings"
"testing"
@@ -19,6 +20,7 @@ func meshToSearch(t *testing.T) *index {
"status": "One unit as the service manager sees it now: its states, whether it starts at boot.",
"restart": "Restart one unit.",
"journal": "The last lines of one unit's journal (at most 2000).",
"failed": "The units that failed.",
}
for _, node := range []string{"anchor", "laptop"} {
for verb, does := range verbs {
@@ -31,8 +33,8 @@ func meshToSearch(t *testing.T) *index {
"node": "What one machine reported it can do, what it is assigned, and why.",
"nodes": "Every machine the mesh knows, with whether it is converged or adopted.",
} {
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller",
Tool: verb, Scope: "mesh", Description: does, Subject: "mesh.seat.mesh-controller.tool." + verb})
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-placement", Seat: "mesh-placement",
Tool: verb, Scope: "mesh", Description: does, Subject: "mesh.seat.mesh-placement.tool." + verb})
}
for tool, does := range map[string]string{
"docker_list": "Every container on this machine: name, image, state, ports.",
@@ -44,13 +46,7 @@ func meshToSearch(t *testing.T) *index {
Description: does, Subject: "mesh.mod.docker.tool." + tool + ".anchor"})
}
x, _, _ := indexOfEndpoints(endpoints)
tools := json.RawMessage(`{"seats":[{"seat":"node-service-manager","scope":"node","tools":[
{"name":"units","replaces":["systemctl list-units","systemctl --failed"]},
{"name":"status","replaces":["systemctl status","systemctl is-active"]},
{"name":"restart","replaces":["systemctl restart"]},
{"name":"journal","replaces":["journalctl"]}]},
{"seat":"mesh-controller","scope":"mesh","tools":[{"name":"node","replaces":["hostnamectl","uptime"]},{"name":"nodes"}]}]}`)
x.Replaces = replacesOf(tools, []recordedModule{{Module: "docker", On: []string{"anchor"}, Tools: true,
x.Replaces = replacesOf(catalogue.DefaultSeats(), []recordedModule{{Module: "docker", On: []string{"anchor"}, Tools: true,
Replaces: map[string][]string{"docker_list": {"docker ps"}, "docker_logs": {"docker logs"}, "docker_restart": {"docker restart"}}}})
return x
}
@@ -60,22 +56,21 @@ func meshToSearch(t *testing.T) *index {
func TestSearchFindsTheToolThatReplacesACommandFirst(t *testing.T) {
x := meshToSearch(t)
for query, want := range map[string]string{
"journalctl": "anchor/node-service-manager.journal",
"journalctl -u mesh-controller -n 50": "anchor/node-service-manager.journal",
"sudo journalctl -fu sshd": "anchor/node-service-manager.journal",
"/usr/bin/journalctl": "anchor/node-service-manager.journal",
"logs": "anchor/node-service-manager.journal",
"systemctl status": "anchor/node-service-manager.status",
"systemctl status nats.service": "anchor/node-service-manager.status",
"systemctl restart mesh-controller": "anchor/node-service-manager.restart",
"systemctl --failed": "anchor/node-service-manager.units",
"docker ps": "anchor/docker.docker_list",
"docker ps -a": "anchor/docker.docker_list",
"docker logs": "anchor/docker.docker_logs",
"docker logs --tail 100 nats": "anchor/docker.docker_logs",
"docker restart nats": "anchor/docker.docker_restart",
"container logs": "anchor/docker.docker_logs",
"uptime": "mesh-controller.node",
"journalctl": "anchor/node-service-manager.journal",
"journalctl -u mesh-placement -n 50": "anchor/node-service-manager.journal",
"sudo journalctl -fu sshd": "anchor/node-service-manager.journal",
"/usr/bin/journalctl": "anchor/node-service-manager.journal",
"logs": "anchor/node-service-manager.journal",
"systemctl status": "anchor/node-service-manager.status",
"systemctl status nats.service": "anchor/node-service-manager.status",
"systemctl restart mesh-placement": "anchor/node-service-manager.restart",
"systemctl --failed": "anchor/node-service-manager.failed",
"docker ps": "anchor/docker.docker_list",
"docker ps -a": "anchor/docker.docker_list",
"docker logs": "anchor/docker.docker_logs",
"docker logs --tail 100 nats": "anchor/docker.docker_logs",
"docker restart nats": "anchor/docker.docker_restart",
"container logs": "anchor/docker.docker_logs",
} {
hits := x.search(query)
if len(hits) == 0 {
@@ -116,7 +111,7 @@ func TestASearchByCommandSaysWhatItReplaces(t *testing.T) {
// A synonym is a word, never part of one: `logs` matches through `log`, and finds no `login`.
func TestSearchStillMatchesEveryWord(t *testing.T) {
x := meshToSearch(t)
if hits := x.search("machine converged"); len(hits) != 1 || hits[0].Address != "mesh-controller.nodes" {
if hits := x.search("machine converged"); len(hits) != 1 || hits[0].Address != "mesh-placement.nodes" {
t.Errorf("words in a description: %+v", hits)
}
if hits := x.search("journalctl kubernetes"); len(hits) != 1 || hits[0].Address != "anchor/node-service-manager.journal" {
@@ -134,10 +129,10 @@ func TestSearchStillMatchesEveryWord(t *testing.T) {
}
}
// A controller older than `replaces` answers without it: search matches names and descriptions as before.
// A seat's contract and a module's records without `replaces`: search matches names and descriptions as before.
func TestSearchWithoutReplacesMatchesAsBefore(t *testing.T) {
x := meshToSearch(t)
x.Replaces = replacesOf(json.RawMessage(`{"seats":[{"seat":"node-service-manager","tools":[{"name":"journal"}]}]}`),
x.Replaces = replacesOf([]catalogue.Seat{{Name: "node-service-manager", Serves: []catalogue.Verb{{Name: "journal"}}}},
[]recordedModule{{Module: "docker"}})
if len(x.Replaces) != 0 {
t.Fatalf("replaces from records that have none: %v", x.Replaces)
+1 -1
View File
@@ -3,7 +3,7 @@ package console
// mesh_wait: a session waits on the bus for one event instead of polling (novox/hq ADR 0367).
//
// **Only the families the tool runner was issued** (§2, §6): its membership names them, composed by the
// controller from the tool runner's manifest together with the grant that lets this account subscribe them.
// mesh from the tool runner's manifest together with the grant that lets this account subscribe them.
// A name not listed there is refused here, before the bus would refuse it; no argument or setting adds one.
// **An event only wakes the session** (§4): what it says is read again from the verb that owns it.
@@ -48,6 +48,11 @@ func (b *subscribing) Provision(string, json.RawMessage) (json.RawMessage, error
func (b *subscribing) ProvisionWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
func (b *subscribing) BusState(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) BusStateWatch(string, json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
func (b *subscribing) Begin(string) (string, func()) { return "", func() {} }
func (b *subscribing) Subscribe(d func(json.RawMessage) error) error {
b.mu.Lock()
defer b.mu.Unlock()
@@ -95,3 +100,7 @@ func TestAnEventIsTakenByAnyAnswerThatIsNotAnError(t *testing.T) {
t.Fatalf("an error answer was not the bundle's refusal of mesh/event: %#v", err)
}
}
func (b *subscribing) LogWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
+90 -7
View File
@@ -19,6 +19,7 @@ import (
"syscall"
"time"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"golang.org/x/sys/unix"
"github.com/novox/mesh-tools/node-tools/internal/wire"
@@ -38,6 +39,9 @@ type Tool struct {
Name string
Description string
Input json.RawMessage
// Effect is what the tool does, reads, acts or dangerous (novox/hq ADR 0387 §1), where the runtime knows it: its own
// tools say it; a bundle's tool is marked by its manifest.
Effect string
// Run calls the tool for caller: the bus user the call's subject named, or "" when it named none. A
// caller is handed to the bundle in the call's `_meta` (MetaCaller), never among its arguments.
Run func(caller string, args json.RawMessage) (json.RawMessage, error)
@@ -48,12 +52,21 @@ type Tool struct {
// stdio.CallerOf, and says Unproven when it is absent.
const MetaCaller = "mesh/caller"
// callParams is a `tools/call`'s params: the tool, its arguments as the caller sent them, and — only when the
// subject named one — the caller, in `_meta`, which the runtime writes and no caller can.
func callParams(name, caller string, args json.RawMessage) map[string]any {
// callParams is a `tools/call`'s params: the tool, its arguments as the caller sent them, and in `_meta`, which the
// runtime writes and no caller can, the caller — only when the subject named one — and the id of the call in flight
// (mesh-sdk buscalls.MetaCall), which a ledger's apply names so the runtime stamps the caller it proved (novox/hq ADR
// 0597 §7).
func callParams(name, caller, call string, args json.RawMessage) map[string]any {
params := map[string]any{"name": name, "arguments": args}
meta := map[string]any{}
if caller != "" {
params["_meta"] = map[string]any{MetaCaller: caller}
meta[MetaCaller] = caller
}
if call != "" {
meta[buscalls.MetaCall] = call
}
if len(meta) > 0 {
params["_meta"] = meta
}
return params
}
@@ -80,6 +93,8 @@ type Bus interface {
Watch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
// Log answers a bundle's `append`, `read`, `last` and `info` on its module's log (novox/hq ADR 0297).
Log(verb string, params json.RawMessage) (json.RawMessage, error)
// LogWatch hands deliver each entry of a log from where the bundle asked, one at a time (novox/hq ADR 0571).
LogWatch(params json.RawMessage, deliver func(entry json.RawMessage) error) (stop func(), err error)
// Seat answers a bundle's seat traffic (novox/hq ADR 0259 §3): `publish`, `prove` and `record` at once;
// `take` and `answer` bind once and hand each message or proof on through handOn, to whichever child
// of the module asked last — a restarted child asks again as its code runs again.
@@ -89,8 +104,29 @@ type Bus interface {
// (novox/hq ADR 0349 §14, ADR 0370 §4): its config, and its bindings' values that are not secret.
Provision(verb string, params json.RawMessage) (json.RawMessage, error)
ProvisionWatch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
// BusState answers a bundle's calls on its module's bus state in the new form (mesh-sdk buscalls; novox/hq ADR 0595
// §9, design 25 § 12.5): its objects, a stream's read-back and fill, a work queue's size and dead letters, and its
// ledgers, by the method the bundle asked, from its own module's bindings alone.
BusState(method string, params json.RawMessage) (json.RawMessage, error)
// BusStateWatch hands deliver each change of an objects or a ledger watch, the current ones first.
BusStateWatch(method string, params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
// Begin says a tool call of the module is in flight for the caller the runtime proved, and answers its id, which the
// bundle is handed in the call's `_meta`, and the end to call once the call is answered (novox/hq ADR 0597 §7).
Begin(caller string) (id string, end func())
}
// busStateCalls are the methods of mesh-sdk buscalls a bundle asks and BusState answers.
var busStateCalls = map[string]bool{
buscalls.ObjectsPut: true, buscalls.ObjectsGet: true, buscalls.ObjectsList: true, buscalls.ObjectsDelete: true,
buscalls.ObjectsInfo: true, buscalls.StreamRead: true, buscalls.StreamFill: true, buscalls.WorkQueueSize: true,
buscalls.WorkQueueDead: true, buscalls.LedgerApply: true, buscalls.LedgerGet: true, buscalls.LedgerHistory: true,
buscalls.LedgerQuery: true, buscalls.LedgerRebuild: true, buscalls.LedgerReindex: true, buscalls.LedgerSnapshot: true,
buscalls.LedgerLag: true,
}
// busStateWatches are the watches of mesh-sdk buscalls, each with the method its changes are handed to the bundle by.
var busStateWatches = map[string]string{buscalls.ObjectsWatch: buscalls.ObjectsWatched, buscalls.LedgerWatch: buscalls.LedgerWatched}
// EventTimeout bounds how long a bundle has to handle one event before it is offered again.
var EventTimeout = 2 * time.Minute
@@ -323,12 +359,12 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
result = answered
}
case "mesh/state.get", "mesh/state.put", "mesh/state.delete", "mesh/state.keys",
"mesh/state.create", "mesh/state.update", "mesh/state.history":
"mesh/state.create", "mesh/state.update", "mesh/state.history", "mesh/state.info":
var answered json.RawMessage
if answered, err = mesh.State(strings.TrimPrefix(m.Method, "mesh/state."), m.Params); err == nil {
result = answered
}
case "mesh/log.append", "mesh/log.read", "mesh/log.last", "mesh/log.info":
case "mesh/log.append", "mesh/log.read", "mesh/log.newest", "mesh/log.last", "mesh/log.info":
var answered json.RawMessage
if answered, err = mesh.Log(strings.TrimPrefix(m.Method, "mesh/log."), m.Params); err == nil {
result = answered
@@ -357,6 +393,25 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
c.mu.Unlock()
}
}
case "mesh/log.watch":
// As a state's watch: each entry asked of this child in order, `mesh/log {watch, entry}`,
// naming the watch by the id the child asked it with (novox/hq ADR 0571).
var stop func()
watch := m.ID
stop, err = mesh.LogWatch(m.Params, func(entry json.RawMessage) error {
_, err := c.ask(module, "mesh/log", map[string]any{"watch": watch, "entry": entry}, EventTimeout)
return err
})
if err == nil {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
stop()
} else {
c.watches = append(c.watches, stop)
c.mu.Unlock()
}
}
case "mesh/state.watch":
// Each change is asked of this child, in order, naming the watch it is for by
// the id the child asked it with — two watches of one state are two handlers;
@@ -378,7 +433,33 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
c.mu.Unlock()
}
}
case buscalls.ObjectsWatch, buscalls.LedgerWatch:
// As a state's watch: each change asked of this child in order, by the method of its
// family; the change names the watch by the id the bundle gave in its request.
var stop func()
told := busStateWatches[m.Method]
stop, err = mesh.BusStateWatch(m.Method, m.Params, func(change json.RawMessage) error {
_, err := c.ask(module, told, change, EventTimeout)
return err
})
if err == nil {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
stop()
} else {
c.watches = append(c.watches, stop)
c.mu.Unlock()
}
}
default:
if busStateCalls[m.Method] {
var answered json.RawMessage
if answered, err = mesh.BusState(m.Method, m.Params); err == nil {
result = answered
}
break
}
if len(m.ID) > 0 {
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID,
"error": map[string]any{"code": -32601, "message": "the runtime answers no " + m.Method + " from a bundle"}})
@@ -590,7 +671,9 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
if len(args) == 0 || string(args) == "null" {
args = json.RawMessage("{}")
}
res, err := asking("tools/call", callParams(full, caller, args), CallTimeout)
call, end := mesh.Begin(caller)
defer end()
res, err := asking("tools/call", callParams(full, caller, call, args), CallTimeout)
if err != nil {
return nil, err
}
@@ -105,3 +105,7 @@ func TestABundleAsksForItsValuesAndIsHandedItsWatchsChanges(t *testing.T) {
t.Errorf("the runtime's bus was asked %q", b.asked)
}
}
func (b *provisioning) LogWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
+497
View File
@@ -0,0 +1,497 @@
// Package ledger is the tool runner's side of a ledger (novox/hq ADR 0597, design 25 § 12.4; issue 1966): a storage type
// whose state is derived from its history. `apply` checks an operation against its type, appends it to the instance's
// history with the last sequence expected, derives the state by the operation's effect and writes it as an update of the
// revision read, and writes the instance's snapshot. What the history, the state and the snapshots are kept in is given
// (History, State, Snapshots): the bus in the runtime, fakes in a test.
package ledger
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"regexp"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// The refusals the stores answer, told by errors.Is.
var (
// ErrRaced is an append or a snapshot written with an expected sequence that is no longer the last: another writer
// wrote first.
ErrRaced = errors.New("another writer wrote the instance's history first")
// ErrChanged is a state written at a revision that is no longer the key's.
ErrChanged = errors.New("the state was written by another writer first")
// ErrFull is a state bucket that refuses a write because it is full.
ErrFull = errors.New("the state bucket is full")
)
// CodeChanged is the error code a bundle is answered a lost race with, as a lost state update (mesh-sdk stdio.CodeChanged).
const CodeChanged = -32010
// raced is an apply refused because another writer appended to the instance first (ADR 0597 §5).
type raced struct{ instance string }
func (e raced) Error() string {
return "the operation on " + e.instance + " is not taken: another writer appended to its history first; read it and apply again (novox/hq ADR 0597 §5)"
}
func (e raced) Is(target error) bool { return target == ErrRaced }
func (e raced) ErrorCode() int { return CodeChanged }
// Kept is one message a history keeps: its sequence and its bytes.
type Kept struct {
Sequence uint64
Data []byte
}
// History is a ledger's history: a stream with a subject per instance, each instance named by its key.
type History interface {
// Last is the sequence of the instance's last operation, 0 when it has none.
Last(key string) (uint64, error)
// Append writes one operation, expecting the instance's last sequence to be expect (0: it has none); ErrRaced when
// it is not.
Append(key string, op []byte, expect uint64) (uint64, error)
// Read is the operations of the instances keys names (a key, with `*` for one name and `>` for the rest) from a
// sequence on, oldest first, at most limit of them (every one when limit is 0).
Read(keys string, from uint64, limit int) ([]Kept, error)
}
// State is a ledger's state bucket: each instance's record under its key.
type State interface {
// Get is the key's value and revision; nil and 0 when it has none.
Get(key string) ([]byte, uint64, error)
// Write writes the key's value at the revision read (0: only when it has none); ErrChanged when the key is no longer
// at it, ErrFull when the bucket is full.
Write(key string, value []byte, revision uint64) (uint64, error)
// Keys is every key the bucket holds.
Keys() ([]string, error)
}
// Snapshots are a ledger's snapshots: the latest record of each instance, on a subject of its own, never expiring.
type Snapshots interface {
// Get is the key's latest snapshot and the sequence it was written at; nil and 0 when it has none.
Get(key string) ([]byte, uint64, error)
// Put writes the key's snapshot, expecting its last to be at expect (0: none); ErrRaced when it is not.
Put(key string, value []byte, expect uint64) error
}
// Ledger is one ledger of one module: its types and where it is kept.
type Ledger struct {
Module string
Name string
// Account is the owning module's authenticated account, which every operation it applies is stamped with.
Account string
Types []catalogue.LedgerType
History History
State State
Snapshots Snapshots
// Indexes are its declared indexes' buckets, by index name.
Indexes map[string]Index
// Now is the clock an operation is stamped by; time.Now when nil.
Now func() time.Time
// Logf says what an apply could not do after its operation was taken; nothing is said when nil.
Logf func(format string, args ...any)
}
func (l *Ledger) say(format string, args ...any) {
if l.Logf != nil {
l.Logf(format, args...)
}
}
// Record is what the state and a snapshot keep of an instance: its state and the sequence of the last operation in it.
type Record struct {
Sequence uint64 `json:"sequence"`
State json.RawMessage `json:"state"`
}
// segment is one part of an instance's path.
var segment = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
// KeyOf is the key an instance is kept under, in the state and as its history's and snapshot's subject token: its path's
// parts joined by dots (`decision/0290` is `decision.0290`).
func KeyOf(instance string) (string, error) {
parts := strings.Split(instance, "/")
if len(parts) < 2 {
return "", fmt.Errorf("%q is not an instance: an instance is named by a path whose first part is its type's name, "+
"such as decision/0290 (novox/hq ADR 0597 §1)", instance)
}
for _, p := range parts {
if !segment.MatchString(p) {
return "", fmt.Errorf("%q is not an instance: each part of its path holds letters, digits, - and _ alone", instance)
}
}
return strings.Join(parts, "."), nil
}
// InstanceOf is the instance a key names.
func InstanceOf(key string) string { return strings.ReplaceAll(key, ".", "/") }
func (l *Ledger) now() time.Time {
if l.Now != nil {
return l.Now().UTC()
}
return time.Now().UTC()
}
// typeOf is the instance's type, by its path's first part.
func (l *Ledger) typeOf(instance string) (catalogue.LedgerType, error) {
name, _, _ := strings.Cut(instance, "/")
for _, t := range l.Types {
if t.Name == name {
return t, nil
}
}
return catalogue.LedgerType{}, fmt.Errorf("%s's ledger %s has no type %q, which the instance %s names", l.Module, l.Name, name, instance)
}
// operationOf is the operation a type allows by its name.
func operationOf(t catalogue.LedgerType, name string) (catalogue.LedgerOperation, error) {
for _, op := range t.Operations {
if op.Name == name {
return op, nil
}
}
return catalogue.LedgerOperation{}, fmt.Errorf("the type %s allows no operation %q", t.Name, name)
}
// Digest is the digest a type pairs a text field with (ADR 0597 §2): sha256 of the text, as the knowledge base writes it.
func Digest(text string) string {
sum := sha256.Sum256([]byte(text))
return "sha256:" + hex.EncodeToString(sum[:])
}
// kinds is the JSON kind each schema type name is held to; a name not here is the judge's to know.
var kinds = map[string]func(any) bool{
"string": func(v any) bool { _, ok := v.(string); return ok },
"number": func(v any) bool { _, ok := v.(float64); return ok },
"integer": func(v any) bool { f, ok := v.(float64); return ok && f == float64(int64(f)) },
"boolean": func(v any) bool { _, ok := v.(bool); return ok },
"object": func(v any) bool { _, ok := v.(map[string]any); return ok },
"array": func(v any) bool { _, ok := v.([]any); return ok },
}
// checked is an operation as apply checked it: its instance's key, its operation and its body.
type checked struct {
key string
op catalogue.LedgerOperation
body map[string]any
}
// check holds an operation to its type (ADR 0597 §2): a type the instance names, an operation it allows, a body that is an
// object whose fields fit the schema, the key its effect names, and every digest matching its text.
func (l *Ledger) check(r buscalls.ApplyRequest) (checked, error) {
key, err := KeyOf(r.Instance)
if err != nil {
return checked{}, err
}
t, err := l.typeOf(r.Instance)
if err != nil {
return checked{}, err
}
op, err := operationOf(t, r.Operation)
if err != nil {
return checked{}, err
}
body := map[string]any{}
if len(r.Body) > 0 && string(r.Body) != "null" {
if err := json.Unmarshal(r.Body, &body); err != nil {
return checked{}, fmt.Errorf("the operation %s's body is a JSON object: %w", r.Operation, err)
}
}
for name, field := range t.Schema {
v, has := body[name]
if !has {
continue
}
if fits, known := kinds[field.Type]; known && !fits(v) {
return checked{}, fmt.Errorf("the operation %s's field %s is a %s in the type %s", r.Operation, name, field.Type, t.Name)
}
}
for name, field := range t.Schema {
if field.DigestOf == "" {
continue
}
digest, hasDigest := body[name]
text, hasText := body[field.DigestOf]
if !hasDigest && !hasText {
continue
}
s, isText := text.(string)
if !hasDigest || !hasText || !isText || digest != Digest(s) {
return checked{}, fmt.Errorf("the operation %s's %s is not the digest of its %s, so it is refused (novox/hq ADR 0597 §2)",
r.Operation, name, field.DigestOf)
}
}
switch op.Effect {
case catalogue.EffectSetFields, catalogue.EffectAppend:
case catalogue.EffectSetEntry, catalogue.EffectRemove:
if op.Effect == catalogue.EffectRemove && op.Into == "" {
break
}
if _, ok := body[op.Key].(string); !ok {
return checked{}, fmt.Errorf("the operation %s names its entry by its field %s, a text, which its body lacks", r.Operation, op.Key)
}
default:
return checked{}, fmt.Errorf("the operation %s's effect %q is not one the mesh offers (novox/hq ADR 0597 §2)", r.Operation, op.Effect)
}
return checked{key: key, op: op, body: body}, nil
}
// effect is an operation's effect on a state (ADR 0597 §2): it copies only the fields the operation names, and the
// operation's sequence and proven author where it says so; null is a removed instance.
func effect(state json.RawMessage, op catalogue.LedgerOperation, body map[string]any, seq uint64, author buscalls.Author) (json.RawMessage, error) {
s := map[string]any{}
if len(state) > 0 && string(state) != "null" {
if err := json.Unmarshal(state, &s); err != nil {
return nil, fmt.Errorf("the state held is not an object: %w", err)
}
}
copied := map[string]any{}
for _, f := range op.Fields {
if v, ok := body[f]; ok {
copied[f] = v
}
}
if op.Sequence {
copied["sequence"] = seq
}
// Written last, so a body's own `author` never stands for the proven one (ADR 0597 §7).
if op.Author {
copied["author"] = author
}
switch op.Effect {
case catalogue.EffectSetFields:
for k, v := range copied {
s[k] = v
}
case catalogue.EffectSetEntry:
m, _ := s[op.Into].(map[string]any)
if m == nil {
m = map[string]any{}
}
m[body[op.Key].(string)] = copied
s[op.Into] = m
case catalogue.EffectAppend:
list, _ := s[op.Into].([]any)
s[op.Into] = append(list, copied)
case catalogue.EffectRemove:
if op.Into == "" {
return json.RawMessage("null"), nil
}
if m, ok := s[op.Into].(map[string]any); ok {
delete(m, body[op.Key].(string))
}
}
return json.Marshal(s)
}
// fold applies one kept operation to a record, as apply derived it.
func (l *Ledger) fold(rec Record, k Kept) (Record, error) {
var o buscalls.Operation
if err := json.Unmarshal(k.Data, &o); err != nil {
return rec, fmt.Errorf("the history's operation %d is not one: %w", k.Sequence, err)
}
t, err := l.typeOf(o.Instance)
if err != nil {
return rec, err
}
op, err := operationOf(t, o.Operation)
if err != nil {
return rec, err
}
body := map[string]any{}
if len(o.Body) > 0 && string(o.Body) != "null" {
if err := json.Unmarshal(o.Body, &body); err != nil {
return rec, err
}
}
if (op.Effect == catalogue.EffectSetEntry || (op.Effect == catalogue.EffectRemove && op.Into != "")) && !isText(body[op.Key]) {
return rec, fmt.Errorf("the history's operation %d names no entry", k.Sequence)
}
state, err := effect(rec.State, op, body, k.Sequence, o.Author)
if err != nil {
return rec, err
}
return Record{Sequence: k.Sequence, State: state}, nil
}
func isText(v any) bool { _, ok := v.(string); return ok }
// replay folds the instance's operations after rec's sequence, up to upto (every one when 0), onto rec.
func (l *Ledger) replay(key string, rec Record, upto uint64) (Record, error) {
kept, err := l.History.Read(key, rec.Sequence+1, 0)
if err != nil {
return rec, err
}
for _, k := range kept {
if upto != 0 && k.Sequence > upto {
break
}
if k.Sequence <= rec.Sequence {
continue
}
if rec, err = l.fold(rec, k); err != nil {
return rec, err
}
}
return rec, nil
}
// readState is the instance's record in the state and its revision; an empty record when it has none.
func (l *Ledger) readState(key string) (Record, uint64, error) {
raw, rev, err := l.State.Get(key)
if err != nil || raw == nil {
return Record{}, rev, err
}
var rec Record
if err := json.Unmarshal(raw, &rec); err != nil {
return Record{}, rev, fmt.Errorf("the state of %s is not a record: %w", key, err)
}
return rec, rev, nil
}
// readSnapshot is the instance's snapshot and the sequence it was written at; an empty record when it has none.
func (l *Ledger) readSnapshot(key string) (Record, uint64, error) {
raw, at, err := l.Snapshots.Get(key)
if err != nil || raw == nil {
return Record{}, at, err
}
var rec Record
if err := json.Unmarshal(raw, &rec); err != nil {
return Record{}, at, fmt.Errorf("the snapshot of %s is not a record: %w", key, err)
}
return rec, at, nil
}
// derive is the instance's record caught up with its history up to last: from the state, or from the snapshot when it
// is further on (an instance whose state was lost, or whose operations expired, ADR 0597 §8).
func (l *Ledger) derive(key string, base Record, last uint64) (Record, error) {
if base.Sequence >= last {
return base, nil
}
snap, _, err := l.readSnapshot(key)
if err != nil {
return base, err
}
if snap.Sequence > base.Sequence {
base = snap
}
return l.replay(key, base, last)
}
// settle writes want into the state at the revision read (ADR 0597 §5): a lost update re-reads, and catches the state
// read up to want's sequence from the history, never appending again; a state already further on is left as it is. It
// answers what the state holds after it, and whether it is behind because the bucket is full (ADR 0597 §6).
func (l *Ledger) settle(key string, want Record, rev uint64) (Record, bool, error) {
for tries := 0; tries < 16; tries++ {
raw, err := json.Marshal(want)
if err != nil {
return want, false, err
}
_, err = l.State.Write(key, raw, rev)
switch {
case err == nil:
return want, false, nil
case errors.Is(err, ErrFull):
return want, true, nil
case !errors.Is(err, ErrChanged):
return want, false, err
}
cur, curRev, err := l.readState(key)
if err != nil {
return want, false, err
}
if cur.Sequence >= want.Sequence {
return cur, false, nil
}
if want, err = l.replay(key, cur, want.Sequence); err != nil {
return want, false, err
}
rev = curRev
}
return want, false, fmt.Errorf("the state of %s kept being written by other writers; the operation is in its history and the next apply catches it up", key)
}
// writeSnapshot writes rec as the instance's snapshot unless the snapshot is already as far on (ADR 0597 §8).
func (l *Ledger) writeSnapshot(key string, rec Record) error {
for tries := 0; tries < 16; tries++ {
snap, at, err := l.readSnapshot(key)
if err != nil {
return err
}
if at != 0 && snap.Sequence >= rec.Sequence {
return nil
}
raw, err := json.Marshal(rec)
if err != nil {
return err
}
if err = l.Snapshots.Put(key, raw, at); err == nil || !errors.Is(err, ErrRaced) {
return err
}
}
return fmt.Errorf("the snapshot of %s kept being written by other writers", key)
}
// Apply applies one operation to one instance, stamped with author (ADR 0597 §3, §5, §6, §8; design 25 § 12.4 steps 1-5):
// checked against its type, appended with the instance's last sequence expected, the state derived by its effect and
// written at the revision read, and the snapshot written. A racing writer's append is refused; a full state bucket
// answers taken with the state behind.
func (l *Ledger) Apply(r buscalls.ApplyRequest, author buscalls.Author) (buscalls.Applied, error) {
c, err := l.check(r)
if err != nil {
return buscalls.Applied{}, err
}
last, err := l.History.Last(c.key)
if err != nil {
return buscalls.Applied{}, err
}
held, rev, err := l.readState(c.key)
if err != nil {
return buscalls.Applied{}, err
}
base, err := l.derive(c.key, held, last)
if err != nil {
return buscalls.Applied{}, err
}
body := r.Body
if len(body) == 0 {
body = json.RawMessage("{}")
}
op, err := json.Marshal(buscalls.Operation{Instance: r.Instance, Operation: r.Operation, Body: body, Author: author, At: l.now()})
if err != nil {
return buscalls.Applied{}, err
}
seq, err := l.History.Append(c.key, op, last)
if errors.Is(err, ErrRaced) {
return buscalls.Applied{}, raced{r.Instance}
}
if err != nil {
return buscalls.Applied{}, err
}
state, err := effect(base.State, c.op, c.body, seq, author)
if err != nil {
return buscalls.Applied{}, err
}
// From here the operation is taken: it is in the history, and a fault below is the state's to catch up, said here
// and answered as the state behind, never as a refusal of what later takes effect (ADR 0597 §6).
now, behind, err := l.settle(c.key, Record{Sequence: seq, State: state}, rev)
if err != nil {
l.say("[mesh-tools] %s's ledger %s took operation %d on %s and its state could not follow: %v; the next apply "+
"or a rebuild catches it up", l.Module, l.Name, seq, r.Instance, err)
behind, now = true, Record{Sequence: seq, State: state}
}
if behind {
now = Record{Sequence: seq, State: state}
}
if err := l.writeSnapshot(c.key, now); err != nil {
l.say("[mesh-tools] %s's ledger %s took operation %d on %s and its snapshot could not be written: %v; the next "+
"apply or a snapshot repair writes it", l.Module, l.Name, seq, r.Instance, err)
}
return buscalls.Applied{Taken: true, Sequence: seq, StateBehind: behind}, nil
}
+102
View File
@@ -0,0 +1,102 @@
package ledger
import (
"encoding/json"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
var owner = buscalls.Author{Account: "ledger.anchor.m"}
// 1966/AC13, 1966/AC14 and the checks of an operation against its type (ADR 0597 §2, §5, §6, §8; design 25 § 12.4).
func TestApplyAppendsDerivesAndSnapshots(t *testing.T) {
cases := []struct {
name string
run func(t *testing.T, l *Ledger, f fakes)
}{
{"1966/AC13 an instance at sequence 4 is taken with 5, its state and snapshot the effect's result", func(t *testing.T, l *Ledger, f fakes) {
for i := 1; i <= 4; i++ {
applied(t, l, "decision/0290", "note", fmt.Sprintf(`{"says":"n%d"}`, i), owner)
}
if last, _ := f.history.Last("decision.0290"); last != 4 {
t.Fatalf("the instance is at %d, want 4", last)
}
got := applied(t, l, "decision/0290", "title", `{"title":"A ledger","ignored":"x"}`, owner)
if !got.Taken || got.Sequence != 5 || got.StateBehind {
t.Fatalf("answered %+v, want taken with 5", got)
}
raw, _, _ := f.state.Get("decision.0290")
state := record(t, raw)
want := `{"author":{"account":"ledger.anchor.m"},"notes":[{"says":"n1"},{"says":"n2"},{"says":"n3"},{"says":"n4"}],"sequence":5,"title":"A ledger"}`
if state.Sequence != 5 || string(state.State) != want {
t.Fatalf("the state is %d %s, want 5 %s", state.Sequence, state.State, want)
}
snap, _, _ := f.snapshots.Get("decision.0290")
if string(snap) != string(raw) {
t.Fatalf("the snapshot is %s, the state %s", snap, raw)
}
}},
{"1966/AC14 a full state bucket answers taken and state behind, and the next apply after room catches up", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/0290", "note", `{"says":"first"}`, owner)
f.state.Full = true
got := applied(t, l, "decision/0290", "note", `{"says":"while full"}`, owner)
if !got.Taken || got.Sequence != 2 || !got.StateBehind {
t.Fatalf("answered %+v, want taken with 2 and state behind", got)
}
if raw, _, _ := f.state.Get("decision.0290"); record(t, raw).Sequence != 1 {
t.Fatalf("the full state moved: %s", raw)
}
f.state.Full = false
got = applied(t, l, "decision/0290", "note", `{"says":"after room"}`, owner)
raw, _, _ := f.state.Get("decision.0290")
state := record(t, raw)
if got.StateBehind || state.Sequence != 3 || string(state.State) != `{"notes":[{"says":"first"},{"says":"while full"},{"says":"after room"}]}` {
t.Fatalf("after room: %+v, the state %d %s", got, state.Sequence, state.State)
}
}},
{"each effect: set an entry, remove it, remove the instance", func(t *testing.T, l *Ledger, f fakes) {
text := "the section's text"
applied(t, l, "decision/1", "section", fmt.Sprintf(`{"id":"a","text":%q,"digest":%q}`, text, Digest(text)), owner)
applied(t, l, "decision/1", "section", fmt.Sprintf(`{"id":"b","text":"b","digest":%q}`, Digest("b")), owner)
applied(t, l, "decision/1", "drop-section", `{"id":"a"}`, owner)
raw, _, _ := f.state.Get("decision.1")
if got := string(record(t, raw).State); got != fmt.Sprintf(`{"sections":{"b":{"digest":%q}}}`, Digest("b")) {
t.Fatalf("the state is %s", got)
}
applied(t, l, "decision/1", "withdraw", `{}`, owner)
raw, _, _ = f.state.Get("decision.1")
if got := record(t, raw); got.Sequence != 4 || string(got.State) != "null" {
t.Fatalf("the removed instance is %d %s", got.Sequence, got.State)
}
}},
{"an operation outside its type, a body off its schema and a digest off its text are refused", func(t *testing.T, l *Ledger, f fakes) {
for _, c := range []struct{ instance, operation, body, refusal string }{
{"listing/a17", "title", `{}`, `no type "listing"`},
{"decision/0290", "accept", `{}`, `no operation "accept"`},
{"decision/0290", "title", `{"title":7}`, "field title is a string"},
{"decision/0290", "section", `{"id":"a","text":"x","digest":"sha256:00"}`, "is not the digest of its text"},
{"decision/0290", "section", `{"text":"x"}`, "is not the digest"},
{"decision/0290", "drop-section", `{}`, "names its entry by its field id"},
{"decision", "title", `{}`, "is not an instance"},
{"decision/a.b", "title", `{}`, "is not an instance"},
} {
_, err := l.Apply(buscalls.ApplyRequest{Instance: c.instance, Operation: c.operation, Body: json.RawMessage(c.body)}, owner)
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Errorf("%s %s %s: %v, want %q", c.instance, c.operation, c.body, err, c.refusal)
}
}
if last, _ := f.history.Last("decision.0290"); last != 0 {
t.Fatalf("a refused operation was appended: %d", last)
}
}},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
l, f := newLedger()
c.run(t, l, f)
})
}
}
+204
View File
@@ -0,0 +1,204 @@
package ledger
import (
"sort"
"strings"
"sync"
)
// The memory stores keep what the bus keeps, as the bus answers it, for a test of a ledger without a bus: a stream whose
// sequence counts across every instance, a bucket with revisions that may be full, and snapshots written with an expected
// sequence.
// MemoryHistory is a history in memory.
type MemoryHistory struct {
mu sync.Mutex
seq uint64
kept []struct {
key string
Kept
}
}
// Matches is whether a key pattern (`*` one name, `>` the rest) names a key.
func Matches(pattern, key string) bool {
p, k := strings.Split(pattern, "."), strings.Split(key, ".")
for i, part := range p {
if part == ">" {
return len(k) > i
}
if i >= len(k) || (part != "*" && part != k[i]) {
return false
}
}
return len(p) == len(k)
}
func (h *MemoryHistory) Last(key string) (uint64, error) {
h.mu.Lock()
defer h.mu.Unlock()
last := uint64(0)
for _, k := range h.kept {
if k.key == key {
last = k.Sequence
}
}
return last, nil
}
func (h *MemoryHistory) Append(key string, op []byte, expect uint64) (uint64, error) {
last, _ := h.Last(key)
h.mu.Lock()
defer h.mu.Unlock()
if last != expect {
return 0, ErrRaced
}
h.seq++
h.kept = append(h.kept, struct {
key string
Kept
}{key, Kept{Sequence: h.seq, Data: op}})
return h.seq, nil
}
func (h *MemoryHistory) Read(keys string, from uint64, limit int) ([]Kept, error) {
h.mu.Lock()
defer h.mu.Unlock()
var out []Kept
for _, k := range h.kept {
if k.Sequence >= from && Matches(keys, k.key) && (limit == 0 || len(out) < limit) {
out = append(out, k.Kept)
}
}
return out, nil
}
// Expire drops every operation of the history, as a ledger's keeping days end them.
func (h *MemoryHistory) Expire() {
h.mu.Lock()
defer h.mu.Unlock()
h.kept = nil
}
type memoryValue struct {
value []byte
rev uint64
}
// MemoryState is a state bucket in memory.
type MemoryState struct {
mu sync.Mutex
rev uint64
// Full makes every write refused as a full bucket's.
Full bool
kv map[string]memoryValue
}
func (s *MemoryState) Get(key string) ([]byte, uint64, error) {
s.mu.Lock()
defer s.mu.Unlock()
v := s.kv[key]
return v.value, v.rev, nil
}
func (s *MemoryState) Write(key string, value []byte, revision uint64) (uint64, error) {
s.mu.Lock()
defer s.mu.Unlock()
if s.Full {
return 0, ErrFull
}
if s.kv[key].rev != revision {
return 0, ErrChanged
}
if s.kv == nil {
s.kv = map[string]memoryValue{}
}
s.rev++
s.kv[key] = memoryValue{value, s.rev}
return s.rev, nil
}
func (s *MemoryState) Keys() ([]string, error) {
s.mu.Lock()
defer s.mu.Unlock()
var out []string
for k := range s.kv {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
// MemorySnapshots are snapshots in memory.
type MemorySnapshots struct {
mu sync.Mutex
at uint64
kv map[string]memoryValue
}
func (s *MemorySnapshots) Get(key string) ([]byte, uint64, error) {
s.mu.Lock()
defer s.mu.Unlock()
v := s.kv[key]
return v.value, v.rev, nil
}
func (s *MemorySnapshots) Put(key string, value []byte, expect uint64) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.kv[key].rev != expect {
return ErrRaced
}
if s.kv == nil {
s.kv = map[string]memoryValue{}
}
s.at++
s.kv[key] = memoryValue{value, s.at}
return nil
}
// MemoryIndex is an index's bucket in memory.
type MemoryIndex struct {
mu sync.Mutex
kv map[string][]byte
}
func (x *MemoryIndex) Entries(pattern string, limit int) ([]Indexed, error) {
x.mu.Lock()
defer x.mu.Unlock()
if pattern == "" {
pattern = ">"
}
var keys []string
for k := range x.kv {
if Matches(pattern, k) {
keys = append(keys, k)
}
}
sort.Strings(keys)
var out []Indexed
for _, k := range keys {
if limit > 0 && len(out) >= limit {
break
}
out = append(out, Indexed{Key: k, Value: x.kv[k]})
}
return out, nil
}
func (x *MemoryIndex) Put(key string, value []byte) error {
x.mu.Lock()
defer x.mu.Unlock()
if x.kv == nil {
x.kv = map[string][]byte{}
}
x.kv[key] = value
return nil
}
func (x *MemoryIndex) Delete(key string) error {
x.mu.Lock()
defer x.mu.Unlock()
delete(x.kv, key)
return nil
}
+166
View File
@@ -0,0 +1,166 @@
package ledger
import (
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// Reading a ledger (ADR 0597 §10, design 25 § 12.5; issue 1966 T3): an instance's state, its history, a watch of its
// states, and a declared index. Only the owning module reads its ledger: the runtime opens a ledger for the module whose
// bundle asks, never one the request names.
// Index is one declared index's bucket, kept by a watcher in the owning module's process and rebuilt by reindex (ADR
// 0597 §4).
type Index interface {
// Entries is the entries whose key a pattern names (`*` one name, `>` the rest; empty is every key), at most limit
// (every one when 0), each as the bucket holds it.
Entries(pattern string, limit int) ([]Indexed, error)
// Put writes one entry; Delete removes one.
Put(key string, value []byte) error
Delete(key string) error
}
// Indexed is one key of an index's bucket and its value, a buscalls.IndexEntry as JSON.
type Indexed struct {
Key string
Value []byte
}
// Watcher is a state bucket that hands each instance's record as it is and as it changes, under a key pattern.
type Watcher interface {
Watch(pattern string, deliver func(key string, value []byte, current bool) error) (func(), error)
}
// HistoryMost is the most operations one history read answers.
const HistoryMost = 1000
// Get is an instance's state as the state holds it, and the sequence of its last operation in it; null and 0 for an
// instance with none.
func (l *Ledger) Get(r buscalls.InstanceRequest) (buscalls.InstanceState, error) {
key, err := KeyOf(r.Instance)
if err != nil {
return buscalls.InstanceState{}, err
}
rec, _, err := l.readState(key)
if err != nil {
return buscalls.InstanceState{}, err
}
state := rec.State
if len(state) == 0 {
state = json.RawMessage("null")
}
return buscalls.InstanceState{Instance: r.Instance, State: state, Sequence: rec.Sequence}, nil
}
// operationsOf reads kept operations as the history answers them, each with its sequence.
func operationsOf(kept []Kept) ([]buscalls.Operation, error) {
out := make([]buscalls.Operation, 0, len(kept))
for _, k := range kept {
var o buscalls.Operation
if err := json.Unmarshal(k.Data, &o); err != nil {
return nil, fmt.Errorf("the history's operation %d is not one: %w", k.Sequence, err)
}
o.Sequence = k.Sequence
out = append(out, o)
}
return out, nil
}
// ReadHistory is an instance's operations from a sequence on, oldest first, each with its sequence and its proven author.
func (l *Ledger) ReadHistory(r buscalls.HistoryRequest) (buscalls.HistoryPage, error) {
key, err := KeyOf(r.Instance)
if err != nil {
return buscalls.HistoryPage{}, err
}
limit := r.Limit
if limit <= 0 || limit > HistoryMost {
limit = HistoryMost
}
last, err := l.History.Last(key)
if err != nil {
return buscalls.HistoryPage{}, err
}
kept, err := l.History.Read(key, r.From, limit+1)
if err != nil {
return buscalls.HistoryPage{}, err
}
page := buscalls.HistoryPage{Last: last}
if len(kept) > limit {
page.Next = kept[limit].Sequence
kept = kept[:limit]
}
if page.Operations, err = operationsOf(kept); err != nil {
return buscalls.HistoryPage{}, err
}
return page, nil
}
// Watch hands deliver the state of every instance Instances names (`*` one name, `>` the rest, in an instance's path),
// the current ones first, then each change.
func (l *Ledger) Watch(r buscalls.LedgerWatchRequest, deliver func(buscalls.InstanceChange) error) (func(), error) {
w, ok := l.State.(Watcher)
if !ok {
return nil, errors.New("this ledger's state cannot be watched")
}
pattern := ">"
if r.Instances != "" {
pattern = keyPattern(r.Instances)
}
return w.Watch(pattern, func(key string, value []byte, current bool) error {
var rec Record
if value != nil {
if err := json.Unmarshal(value, &rec); err != nil {
return fmt.Errorf("the state of %s is not a record: %w", key, err)
}
}
state := rec.State
if len(state) == 0 {
state = json.RawMessage("null")
}
return deliver(buscalls.InstanceChange{Watch: r.Watch, Instance: InstanceOf(key), State: state, Sequence: rec.Sequence,
Current: current})
})
}
// keyPattern is an instance pattern as a key pattern: its path's parts joined by dots.
func keyPattern(instances string) string {
out := []byte(instances)
for i, c := range out {
if c == '/' {
out[i] = '.'
}
}
return string(out)
}
// index is a declared index by its name, or the reason there is none.
func (l *Ledger) index(name string) (Index, error) {
if ix, ok := l.Indexes[name]; ok {
return ix, nil
}
return nil, fmt.Errorf("%s's ledger %s declares no index %q", l.Module, l.Name, name)
}
// Query is a declared index's entries whose key Key names.
func (l *Ledger) Query(r buscalls.QueryRequest) (buscalls.QueryAnswer, error) {
ix, err := l.index(r.Index)
if err != nil {
return buscalls.QueryAnswer{}, err
}
held, err := ix.Entries(r.Key, r.Limit)
if err != nil {
return buscalls.QueryAnswer{}, err
}
entries := make([]buscalls.IndexEntry, 0, len(held))
for _, h := range held {
var e buscalls.IndexEntry
if err := json.Unmarshal(h.Value, &e); err != nil {
return buscalls.QueryAnswer{}, fmt.Errorf("the index %s's entry %s is not one: %w", r.Index, h.Key, err)
}
entries = append(entries, e)
}
return buscalls.QueryAnswer{Entries: entries}, nil
}
+120
View File
@@ -0,0 +1,120 @@
package ledger
import (
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// 1966/AC15, the other reads of a ledger (get, query) and its repairs (ADR 0597 §8, §10; design 25 § 12.5).
func TestReadsAndRepairsOfALedger(t *testing.T) {
cases := []struct {
name string
run func(t *testing.T, l *Ledger, f fakes)
}{
{"1966/AC15 history answers an instance's operations oldest first with sequence and author", func(t *testing.T, l *Ledger, f fakes) {
op := buscalls.Author{Account: "ledger.anchor.m", Caller: "op@shanks"}
applied(t, l, "decision/1", "note", `{"says":"one"}`, owner)
applied(t, l, "decision/2", "note", `{"says":"elsewhere"}`, owner)
applied(t, l, "decision/1", "title", `{"title":"Two"}`, op)
applied(t, l, "decision/1", "note", `{"says":"three"}`, owner)
page, err := l.ReadHistory(buscalls.HistoryRequest{Instance: "decision/1", Limit: 2})
if err != nil {
t.Fatal(err)
}
var got []string
for _, o := range page.Operations {
got = append(got, fmt.Sprintf("%s@%d %s %s", o.Operation, o.Sequence, o.Author.Account, o.Author.Caller))
}
if strings.Join(got, "; ") != "note@1 ledger.anchor.m ; title@3 ledger.anchor.m op@shanks" || page.Next != 4 || page.Last != 4 {
t.Fatalf("the history is %q, next %d, last %d", got, page.Next, page.Last)
}
rest, _ := l.ReadHistory(buscalls.HistoryRequest{Instance: "decision/1", From: page.Next})
if len(rest.Operations) != 1 || rest.Operations[0].Sequence != 4 || rest.Next != 0 {
t.Fatalf("read on: %+v", rest)
}
}},
{"get answers the state and its sequence, null for an instance with none", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
got, err := l.Get(buscalls.InstanceRequest{Instance: "decision/1"})
if err != nil || got.Sequence != 1 || !strings.Contains(string(got.State), `"title":"One"`) {
t.Fatalf("got %+v, %v", got, err)
}
if none, err := l.Get(buscalls.InstanceRequest{Instance: "decision/9"}); err != nil || string(none.State) != "null" {
t.Fatalf("none is %+v, %v", none, err)
}
}},
{"query reads a declared index by key, and an undeclared one is refused", func(t *testing.T, l *Ledger, f fakes) {
_ = f.index.Put("One.decision.1", []byte(`{"instance":"decision/1","sequence":1}`))
_ = f.index.Put("Two.decision.2", []byte(`{"instance":"decision/2","sequence":2}`))
got, err := l.Query(buscalls.QueryRequest{Index: "by-title", Key: "One.>"})
if err != nil || len(got.Entries) != 1 || got.Entries[0] != (buscalls.IndexEntry{Instance: "decision/1", Sequence: 1}) {
t.Fatalf("queried %+v, %v", got, err)
}
if _, err := l.Query(buscalls.QueryRequest{Index: "by-date"}); err == nil || !strings.Contains(err.Error(), `no index "by-date"`) {
t.Fatalf("an undeclared index: %v", err)
}
}},
{"a lost state rebuilds from the history, an expired one from its snapshot, and lag says what is behind", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
applied(t, l, "decision/1", "note", `{"says":"kept"}`, owner)
want, _, _ := f.state.Get("decision.1")
f.state.kv = nil
lag, _ := l.Lag(buscalls.LedgerRequest{})
if len(lag.Instances) != 1 || lag.Instances[0].History != 2 || lag.Instances[0].State != 0 || lag.Instances[0].Snapshot != 2 {
t.Fatalf("lag says %+v", lag)
}
if got, err := l.Rebuild(buscalls.RebuildRequest{}); err != nil || got.Written != 1 {
t.Fatalf("rebuilt %+v, %v", got, err)
}
if got, _, _ := f.state.Get("decision.1"); string(record(t, got).State) != string(record(t, want).State) {
t.Fatalf("rebuilt from the history %s, want %s", got, want)
}
f.state.kv = nil
f.history.Expire()
if got, err := l.Rebuild(buscalls.RebuildRequest{Instance: "decision/1", From: buscalls.RebuildFromSnapshot}); err != nil || got.Written != 1 {
t.Fatalf("rebuilt %+v, %v", got, err)
}
if got, _, _ := f.state.Get("decision.1"); string(got) != string(want) {
t.Fatalf("rebuilt from the snapshot %s, want %s", got, want)
}
}},
{"reindex builds a declared index from the states, and lag says an index behind until it does", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
applied(t, l, "decision/2", "note", `{"says":"untitled"}`, owner)
_ = f.index.Put("stale.decision.9", []byte(`{}`))
if lag, _ := l.Lag(buscalls.LedgerRequest{}); len(lag.Indexes) != 1 || lag.Indexes[0].Behind != 1 {
t.Fatalf("lag says %+v", lag)
}
if got, err := l.Reindex(buscalls.ReindexRequest{Index: "by-title"}); err != nil || got.Written != 1 {
t.Fatalf("reindexed %+v, %v", got, err)
}
entries, _ := f.index.Entries("", 0)
if len(entries) != 1 || entries[0].Key != buscalls.IndexKey([]string{"One"}, "decision/1") || string(entries[0].Value) != `{"instance":"decision/1","sequence":1}` {
t.Fatalf("the index holds %+v", entries)
}
if lag, _ := l.Lag(buscalls.LedgerRequest{}); len(lag.Indexes) != 0 || len(lag.Instances) != 0 {
t.Fatalf("lag after reindex says %+v", lag)
}
}},
{"snapshot rewrites a snapshot behind its state", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
f.snapshots.kv = nil
if got, err := l.Snapshot(buscalls.InstanceRequest{}); err != nil || got.Written != 1 {
t.Fatalf("snapshot %+v, %v", got, err)
}
state, _, _ := f.state.Get("decision.1")
if snap, _, _ := f.snapshots.Get("decision.1"); string(snap) != string(state) {
t.Fatalf("the snapshot %s, the state %s", snap, state)
}
}},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
l, f := newLedger()
c.run(t, l, f)
})
}
}
+304
View File
@@ -0,0 +1,304 @@
package ledger
import (
"encoding/json"
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// The ledger's repairs (ADR 0597 §8, §10; design 25 § 12.5; issue 1966 T4): rebuild an instance's state from its history
// or its snapshot, rewrite a snapshot behind its state, build a declared index again, and say what is behind its history.
// The tool runner serves them as tools of every module owning a ledger, its own ledgers only (the runtime's repairs.go).
// instances is every instance the ledger knows, by key: those its state holds and those its history names.
func (l *Ledger) instances() ([]string, error) {
seen := map[string]bool{}
keys, err := l.State.Keys()
if err != nil {
return nil, err
}
for _, k := range keys {
seen[k] = true
}
kept, err := l.History.Read(">", 1, 0)
if err != nil {
return nil, err
}
ops, err := operationsOf(kept)
if err != nil {
return nil, err
}
for _, o := range ops {
if k, err := KeyOf(o.Instance); err == nil {
seen[k] = true
}
}
out := make([]string, 0, len(seen))
for k := range seen {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
// keysOf is the one instance a request names, or every instance when it names none.
func (l *Ledger) keysOf(instance string) ([]string, error) {
if instance == "" {
return l.instances()
}
k, err := KeyOf(instance)
if err != nil {
return nil, err
}
return []string{k}, nil
}
// Rebuild derives the state of an instance (every instance when it names none) again, from its whole history or from its
// snapshot and the operations after it, and writes it, and its snapshot when that is behind. A derivation further back
// than the state holds is not written: the state never goes back (ADR 0597 §5).
func (l *Ledger) Rebuild(r buscalls.RebuildRequest) (buscalls.Repaired, error) {
from := r.From
if from == "" {
from = buscalls.RebuildFromHistory
}
if from != buscalls.RebuildFromHistory && from != buscalls.RebuildFromSnapshot {
return buscalls.Repaired{}, fmt.Errorf("a rebuild is from %q or %q, not %q", buscalls.RebuildFromHistory, buscalls.RebuildFromSnapshot, from)
}
keys, err := l.keysOf(r.Instance)
if err != nil {
return buscalls.Repaired{}, err
}
var out buscalls.Repaired
for _, key := range keys {
base := Record{}
if from == buscalls.RebuildFromSnapshot {
if base, _, err = l.readSnapshot(key); err != nil {
return out, err
}
}
derived, err := l.replay(key, base, 0)
if err != nil {
return out, err
}
held, rev, err := l.readState(key)
if err != nil {
return out, err
}
if derived.Sequence < held.Sequence {
continue
}
now, _, err := l.settle(key, derived, rev)
if err != nil {
return out, err
}
if err := l.writeSnapshot(key, now); err != nil {
return out, err
}
out.Written++
}
return out, nil
}
// Snapshot rewrites the snapshot of an instance (every instance when it names none) that is behind its state.
func (l *Ledger) Snapshot(r buscalls.InstanceRequest) (buscalls.Repaired, error) {
keys, err := l.keysOf(r.Instance)
if err != nil {
return buscalls.Repaired{}, err
}
var out buscalls.Repaired
for _, key := range keys {
held, _, err := l.readState(key)
if err != nil {
return out, err
}
snap, at, err := l.readSnapshot(key)
if err != nil {
return out, err
}
if held.Sequence == 0 || (at != 0 && snap.Sequence >= held.Sequence) {
continue
}
if err := l.writeSnapshot(key, held); err != nil {
return out, err
}
out.Written++
}
return out, nil
}
// indexEntryOf is an instance's entry in a declared index, as reindex writes it and the owner's watcher keeps it
// (mesh-sdk buscalls.IndexKey, buscalls.IndexEntry): its key the values of the fields the index is on, in their order,
// then the instance; its value the instance and the sequence of the state it was taken from. False when the state holds
// none of those fields. A value that is not text is written as its JSON.
func indexEntryOf(on []string, key string, rec Record) (string, []byte, bool) {
var state map[string]any
if json.Unmarshal(rec.State, &state) != nil || state == nil {
return "", nil, false
}
values := make([]string, 0, len(on))
found := false
for _, f := range on {
v, has := state[f]
found = found || has
s, ok := v.(string)
if !ok {
raw, _ := json.Marshal(v)
s = string(raw)
}
values = append(values, s)
}
if !found {
return "", nil, false
}
instance := InstanceOf(key)
value, _ := json.Marshal(buscalls.IndexEntry{Instance: instance, Sequence: rec.Sequence})
return buscalls.IndexKey(values, instance), value, true
}
// declared is the fields an index is on and the types declaring it.
func (l *Ledger) declared(index string) ([]string, map[string]bool, error) {
types := map[string]bool{}
var on []string
for _, t := range l.Types {
for _, ix := range t.Indexes {
if ix.Name == index {
types[t.Name] = true
on = ix.On
}
}
}
if len(types) == 0 {
return nil, nil, fmt.Errorf("%s's ledger %s declares no index %q", l.Module, l.Name, index)
}
return on, types, nil
}
// wanted is every entry an index should hold, from the states of the types declaring it.
func (l *Ledger) wanted(index string) (map[string][]byte, map[string]uint64, error) {
on, types, err := l.declared(index)
if err != nil {
return nil, nil, err
}
keys, err := l.instances()
if err != nil {
return nil, nil, err
}
want, seqs := map[string][]byte{}, map[string]uint64{}
for _, key := range keys {
name, _, _ := strings.Cut(key, ".")
if !types[name] {
continue
}
held, _, err := l.readState(key)
if err != nil {
return nil, nil, err
}
if k, v, ok := indexEntryOf(on, key, held); ok {
want[k], seqs[k] = v, held.Sequence
}
}
return want, seqs, nil
}
// Reindex builds a declared index again from the ledger's states, derived from its history: every entry it should hold
// written, every other removed.
func (l *Ledger) Reindex(r buscalls.ReindexRequest) (buscalls.Repaired, error) {
ix, err := l.index(r.Index)
if err != nil {
return buscalls.Repaired{}, err
}
want, _, err := l.wanted(r.Index)
if err != nil {
return buscalls.Repaired{}, err
}
held, err := ix.Entries("", 0)
if err != nil {
return buscalls.Repaired{}, err
}
var out buscalls.Repaired
for _, e := range held {
if _, keep := want[e.Key]; !keep {
if err := ix.Delete(e.Key); err != nil {
return out, err
}
}
}
keys := make([]string, 0, len(want))
for k := range want {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if err := ix.Put(k, want[k]); err != nil {
return out, err
}
out.Written++
}
return out, nil
}
// Lag is every instance whose state or snapshot is behind its history, and how many entries each declared index lacks
// or holds from an older state; nothing when nothing is behind.
func (l *Ledger) Lag(buscalls.LedgerRequest) (buscalls.Lagged, error) {
keys, err := l.instances()
if err != nil {
return buscalls.Lagged{}, err
}
var out buscalls.Lagged
for _, key := range keys {
last, err := l.History.Last(key)
if err != nil {
return out, err
}
held, _, err := l.readState(key)
if err != nil {
return out, err
}
snap, _, err := l.readSnapshot(key)
if err != nil {
return out, err
}
if held.Sequence < last || snap.Sequence < last {
out.Instances = append(out.Instances, buscalls.InstanceLag{Instance: InstanceOf(key), History: last,
State: held.Sequence, Snapshot: snap.Sequence})
}
}
names := make([]string, 0, len(l.Indexes))
for name := range l.Indexes {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
if _, _, err := l.declared(name); err != nil {
// A bucket bound for an index no type declares has nothing it should hold.
continue
}
want, seqs, err := l.wanted(name)
if err != nil {
return out, err
}
held, err := l.Indexes[name].Entries("", 0)
if err != nil {
return out, err
}
at := map[string]uint64{}
for _, e := range held {
var v buscalls.IndexEntry
_ = json.Unmarshal(e.Value, &v)
at[e.Key] = v.Sequence
}
behind := uint64(0)
for k := range want {
if s, has := at[k]; !has || s < seqs[k] {
behind++
}
}
if behind > 0 {
out.Indexes = append(out.Indexes, buscalls.IndexLag{Index: name, Behind: behind})
}
}
return out, nil
}
+61
View File
@@ -0,0 +1,61 @@
package ledger
import (
"encoding/json"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// records is a ledger type of each effect: a title set, a section set as an entry, a note appended, a section dropped,
// and the whole instance removed.
var records = catalogue.LedgerType{
Name: "decision",
Schema: map[string]catalogue.LedgerField{
"title": {Type: "string"}, "text": {Type: "string"}, "digest": {Type: "string", DigestOf: "text"},
},
Operations: []catalogue.LedgerOperation{
{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}, Sequence: true, Author: true},
{Name: "section", Effect: catalogue.EffectSetEntry, Into: "sections", Key: "id", Fields: []string{"digest"}},
{Name: "note", Effect: catalogue.EffectAppend, Into: "notes", Fields: []string{"says"}},
{Name: "drop-section", Effect: catalogue.EffectRemove, Into: "sections", Key: "id"},
{Name: "withdraw", Effect: catalogue.EffectRemove},
},
Indexes: []catalogue.LedgerIndex{{Name: "by-title", On: []string{"title"}}},
}
type fakes struct {
history *MemoryHistory
state *MemoryState
snapshots *MemorySnapshots
index *MemoryIndex
}
func newLedger() (*Ledger, fakes) {
f := fakes{&MemoryHistory{}, &MemoryState{}, &MemorySnapshots{}, &MemoryIndex{}}
return &Ledger{Module: "m", Name: "records", Types: []catalogue.LedgerType{records}, History: f.history, State: f.state,
Snapshots: f.snapshots, Indexes: map[string]Index{"by-title": f.index}}, f
}
// applied applies an operation and fails the test on a refusal.
func applied(t *testing.T, l *Ledger, instance, operation, body string, author buscalls.Author) buscalls.Applied {
t.Helper()
got, err := l.Apply(buscalls.ApplyRequest{Ledger: l.Name, Instance: instance, Operation: operation, Body: json.RawMessage(body)}, author)
if err != nil {
t.Fatalf("%s %s: %v", operation, body, err)
}
return got
}
// record is the instance's record a store holds, read as a Record.
func record(t *testing.T, raw []byte) Record {
t.Helper()
var r Record
if raw != nil {
if err := json.Unmarshal(raw, &r); err != nil {
t.Fatal(err)
}
}
return r
}
+155 -18
View File
@@ -1,7 +1,7 @@
// Package meshtest raises what the controller would, for tests against a real bus: the ASSIGNMENTS
// Package meshtest raises what the mesh would, for tests against a real bus: the ASSIGNMENTS
// and EVENTS streams, memberships issued by hand, and a fixture's path.
//
// **Every test has a bus of its own** (novox/hq ADR 0237, the controller's internal/testbus): a server
// **Every test has a bus of its own** (novox/hq ADR 0237, the old testbus): a server
// linked in at the release go.mod pins — the release the mesh runs, held so by a test beside this one —
// started for the one test and gone after it. The packages shared one bus and had to run one at a time:
// each test raised the streams afresh, and the console discovers every runtime that announces itself on
@@ -11,7 +11,11 @@
package meshtest
import (
"context"
"encoding/json"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/membership"
"os"
"path/filepath"
"runtime"
@@ -22,6 +26,7 @@ import (
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
@@ -67,7 +72,7 @@ func URL(t *testing.T) string {
// buses are the running tests' buses, by test.
var buses sync.Map
// Mesh is the controller's job, done by hand.
// Mesh is the mesh's job, done by hand.
type Mesh struct {
nc *nats.Conn
js nats.JetStreamContext
@@ -81,11 +86,10 @@ func New(t *testing.T) *Mesh {
t.Fatal(err)
}
js, _ := nc.JetStream()
for _, s := range []string{"ASSIGNMENTS", "EVENTS"} {
_ = js.DeleteStream(s)
}
if _, err := js.AddStream(&nats.StreamConfig{Name: "ASSIGNMENTS", Subjects: []string{"mesh.assignment.>"},
MaxMsgsPerSubject: 1, AllowDirect: true}); err != nil {
_ = js.DeleteStream("EVENTS")
// The grants part's state memberships, where a runtime reads what is issued it (ADR 0571 clause 10).
_ = js.DeleteKeyValue(buscompose.GrantsMembershipsBucket)
if _, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: buscompose.GrantsMembershipsBucket, History: 1}); err != nil {
t.Fatal(err)
}
if _, err := js.AddStream(&nats.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
@@ -95,15 +99,119 @@ func New(t *testing.T) *Mesh {
return &Mesh{nc: nc, js: js}
}
// Issue publishes a membership.
// Issue writes a membership where the grants part keeps it, its state memberships, key `<node>.<module>` (novox/hq ADR
// 0571 clause 7 and 10; mesh-sdk buscompose.GrantsMembershipsBucket), the one place a runtime reads it. It writes with
// mesh-sdk's encoder, the grants part's own, never json of this package's type: a value built here by hand agreed with
// the reader's mistake and hid that the part wrote another shape (issue 774).
func (m *Mesh) Issue(t *testing.T, mem bus.Membership) {
t.Helper()
body, _ := json.Marshal(mem)
if _, err := m.js.Publish(bus.MembershipSubject(mem.Node, mem.Module), body); err != nil {
m.putGrants(t, mem, membership.EncodeGrantsMembership(sdkMembership(t, mem)))
}
// IssueInGrantsWrapped writes a membership as the grants part wrote it before mesh-sdk's codec (until 2026-10-30, what
// the live mesh holds until the part writes again): its pass record, the membership inside, published saying what
// became of it ("published", "unchanged", or why it was held back).
func (m *Mesh) IssueInGrantsWrapped(t *testing.T, mem bus.Membership, published string) {
t.Helper()
body, _ := json.Marshal(map[string]any{"node": mem.Node, "module": mem.Module, "digest": "sha256:0",
"membership": sdkMembership(t, mem), "published": published})
m.putGrants(t, mem, body)
}
func (m *Mesh) putGrants(t *testing.T, mem bus.Membership, body []byte) {
t.Helper()
kv, err := m.js.KeyValue(buscompose.GrantsMembershipsBucket)
if err != nil {
if kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: buscompose.GrantsMembershipsBucket, History: 1}); err != nil {
t.Fatal(err)
}
}
if _, err := kv.Put(buscompose.MembershipKey(mem.Node, mem.Module), body); err != nil {
t.Fatal(err)
}
}
// DropGrants deletes the grants part's state memberships, as a bus that has not made it yet holds none; Issue makes it again.
func (m *Mesh) DropGrants(t *testing.T) {
t.Helper()
if err := m.js.DeleteKeyValue(buscompose.GrantsMembershipsBucket); err != nil {
t.Fatal(err)
}
}
// sdkMembership is a membership of this package's type as mesh-sdk's, which the grants part composes and writes.
func sdkMembership(t *testing.T, mem bus.Membership) membership.Membership {
t.Helper()
raw, _ := json.Marshal(mem)
var out membership.Membership
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatal(err)
}
return out
}
// ConsoleReads are the states a console's runtime reads to say what the mesh holds (novox/hq ADR 0571 clauses 4, 5):
// placement, the indexer's modules and manifests, and the seat holders.
func ConsoleReads() []bus.StateIssued {
return []bus.StateIssued{
{Name: delivery.PlacementRead, Bucket: delivery.PlacementBucket},
{Name: delivery.IndexerModule + ".modules", Bucket: delivery.ModulesBucket},
{Name: delivery.IndexerModule + ".manifests", Bucket: delivery.ManifestsBucket},
{Name: delivery.SeatHoldersRead, Bucket: delivery.SeatHoldersBucket},
}
}
// Placed writes what the mesh holds as its parts write it, with mesh-sdk's encoders: each module the indexer holds with
// its manifest (tools, the tools it declares; none for a module without), and each node's placement of modules.
func (m *Mesh) Placed(t *testing.T, tools map[string][]string, on map[string][]string) {
t.Helper()
kv := func(bucket string) nats.KeyValue {
_ = m.js.DeleteKeyValue(bucket)
b, err := m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: bucket, History: 1, MaxValueSize: 256 * 1024})
if err != nil {
t.Fatal(err)
}
return b
}
modules, manifests, placement := kv(delivery.ModulesBucket), kv(delivery.ManifestsBucket), kv(delivery.PlacementBucket)
kv(delivery.SeatHoldersBucket)
for module, declared := range tools {
manifest, _ := json.Marshal(map[string]any{"module": module, "tools": declared})
digest := delivery.ManifestDigest(manifest)
raw, err := delivery.EncodeManifest(delivery.Manifest{Module: module, JSON: string(manifest)})
if err != nil {
t.Fatal(err)
}
if _, err := manifests.Put(digest, raw); err != nil {
t.Fatal(err)
}
raw, err = delivery.EncodeModule(delivery.Module{None: "not built here", Manifest: digest, Repository: "novox/" + module,
Commit: strings.Repeat("a", 40)})
if err != nil {
t.Fatal(err)
}
if _, err := modules.Put(module, raw); err != nil {
t.Fatal(err)
}
}
for node, placed := range on {
p := delivery.Placement{V: delivery.Version, Node: node}
for _, module := range placed {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: module})
}
// As the seat writes it: its node key and a key per module (research 072 row 37).
values, err := delivery.PlacementValues(p)
if err != nil {
t.Fatal(err)
}
for key, raw := range values {
if _, err := placement.Put(key, raw); err != nil {
t.Fatal(err)
}
}
}
}
// NextEvent is the subject the next event under a pattern lands on.
func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
t.Helper()
@@ -122,7 +230,7 @@ func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
return ch
}
// MembershipOf is a membership as the controller issues one on a machine.
// MembershipOf is a membership as the mesh issues one on a machine.
func MembershipOf(module, node string, plain bool, seats map[string][]string) bus.Membership {
own := "mesh.mod." + module
serves := []bus.Served{{Subject: own + ".tool.{tool}." + node}}
@@ -183,7 +291,7 @@ func (l *Logs) Has(fragments ...string) bool {
// All is every line.
func (l *Logs) All() string { return strings.Join(l.lines, "\n") }
// Consumer makes a module's durable consumer on a machine as the controller does: pull, on the
// Consumer makes a module's durable consumer on a machine as the mesh does: pull, on the
// EVENTS stream, filtered to what the module consumes, with a short ack wait so a test sees a
// redelivery in seconds rather than the mesh's minutes.
func (m *Mesh) Consumer(t *testing.T, node, module string, filters []string, ackWait time.Duration) {
@@ -209,6 +317,18 @@ func (m *Mesh) Emit(t *testing.T, subject string, body any) {
}
}
// Publish says body on a plain subject, as a core publish with no stream behind it.
func (m *Mesh) Publish(t *testing.T, subject string, body any) {
t.Helper()
data, _ := json.Marshal(body)
if err := m.nc.Publish(subject, data); err != nil {
t.Fatal(err)
}
if err := m.nc.Flush(); err != nil {
t.Fatal(err)
}
}
// Pending is what a module's consumer still holds: delivered and not acknowledged, and not yet
// delivered.
func (m *Mesh) Pending(t *testing.T, node, module string) (ackPending, notDelivered uint64) {
@@ -231,7 +351,7 @@ func (m *Mesh) BucketKeeping(t *testing.T, name string, history uint8) nats.KeyV
return kv
}
// Bucket makes a module's state afresh as the controller does from the catalogue (novox/hq ADR 0201),
// Bucket makes a module's state afresh as the mesh does from the catalogue (novox/hq ADR 0201),
// and answers it for writing what is there before a bundle starts.
func (m *Mesh) Bucket(t *testing.T, name string) nats.KeyValue {
t.Helper()
@@ -243,10 +363,27 @@ func (m *Mesh) Bucket(t *testing.T, name string) nats.KeyValue {
return kv
}
// Log makes a module's log afresh as the controller does from the catalogue (novox/hq ADR 0297): a file
// BucketKeyTTL makes a module's state afresh as the mesh does for one declared `key-ttl` (mesh-sdk buscompose:
// a time to live per key allowed, its end leaving a marker kept a minute; issue 1952).
func (m *Mesh) BucketKeyTTL(t *testing.T, name string) {
t.Helper()
_ = m.js.DeleteKeyValue(name)
js, err := jetstream.New(m.nc)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: 1, MaxValueSize: 256 * 1024,
LimitMarkerTTL: time.Minute}); err != nil {
t.Fatal(err)
}
}
// Log makes a module's log afresh as the mesh does from the catalogue (novox/hq ADR 0297): a file
// stream named LOG_<module>_<name> on mesh.log.<module>.<name>.>, limits retention, no age, no cap per
// key, the declared cap, 260 KiB a message (a 256 KiB value and its headers), refusing new entries when full, direct get, no delete or
// purge. It answers the membership entry the controller issues for it.
// purge. It answers the membership entry the mesh issues for it.
func (m *Mesh) Log(t *testing.T, module, name string, maxBytes int64) bus.LogIssued {
t.Helper()
stream := "LOG_" + module + "_" + name
@@ -272,7 +409,7 @@ func (m *Mesh) Consumers(t *testing.T, stream string) int {
return info.State.Consumers
}
// ModuleStream makes a module's own stream afresh as the controller does from the catalogue (novox/hq ADR
// ModuleStream makes a module's own stream afresh as the mesh does from the catalogue (novox/hq ADR
// 0363 §1): named in capitals, taking `mesh.stream.<module>.event.>`, a subject space EVENTS does not take.
func (m *Mesh) ModuleStream(t *testing.T, module, name string) {
t.Helper()
@@ -318,7 +455,7 @@ func (m *Mesh) Messages(t *testing.T, stream string) uint64 {
return info.State.Msgs
}
// RemoveConsumer deletes a module's consumer on a machine on a stream, as the controller does when a grant
// RemoveConsumer deletes a module's consumer on a machine on a stream, as the mesh does when a grant
// it cannot narrow is withdrawn (novox/hq ADR 0363 §6).
func (m *Mesh) RemoveConsumer(t *testing.T, stream, node, module string) {
t.Helper()
@@ -110,6 +110,13 @@ func TestTheRepositoryHoldsNoCheckScriptAndNodeToolsIsItsOnlyModule(t *testing.T
if !d.IsDir() && d.Name() == "module.json" {
found = append(found, filepath.ToSlash(p))
}
// ADR 0604 section 4: the TypeScript that called the old catalog module's tools is gone, and nothing
// under node-tools names one (the needle is built so this file does not name it either).
if !d.IsDir() && strings.Contains(filepath.ToSlash(p), "/node-tools/") {
if body, rerr := os.ReadFile(p); rerr == nil && strings.Contains(string(body), "catalog"+"_modules") {
t.Errorf("%s names a catalog tool of the retired module", p)
}
}
return nil
})
if err != nil {
+10 -5
View File
@@ -1,6 +1,6 @@
// Package provide is the runtime's half of node-tools being a provider of `mcp-endpoint`: it runs the mesh SDK's
// provider helper, so a consumer handed to the provisioner (claude-code's `${bound:mcp-endpoint:port}`) is answered
// from an entry node-tools writes in its own bucket, not from the controller's binding file (novox/hq ADR 0349
// from an entry node-tools writes in its own bucket, not from the old binding file (novox/hq ADR 0349
// clause 12, issue 791).
//
// The offer carries no credential: the console's authority is the machine's login (ADR 0152, 0175), so the helper
@@ -16,6 +16,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-sdk/go/provisions"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
@@ -38,15 +39,18 @@ func Offers() map[string]provider.Offer {
}
// busOf is the helper's bus over the runtime's own state: the provider's bucket is its state `bound`, and the
// controller's assignments are the state it reads, `mesh-controller.assignments` (both declared in the manifest).
// provisions and plans of the seat mesh-provisioner are the states it reads, by mesh-sdk's names (novox/hq ADR 0570,
// issue 1954 L6-T1). The old assignments are read no more (mesh-sdk 692dbf9, L6-P3).
type busOf struct{ conn *bus.Conn }
func (b busOf) state(bucket string) (string, error) {
switch bucket {
case provider.BucketOf(Module):
return "bound", nil
case provider.AssignmentsBucket:
return "mesh-controller.assignments", nil
case provisions.ProvisionsBucket:
return provisions.ProvisionsRead, nil
case provisions.PlansBucket:
return provisions.PlansRead, nil
}
return "", fmt.Errorf("%s is not a bucket this provider keeps or reads", bucket)
}
@@ -57,7 +61,8 @@ func (b busOf) Watch(ctx context.Context, bucket string, handle func(key string,
return err
}
stop, err := b.conn.StateWatch(Module, name, "", func(ch bus.StateChange) error {
if ch.Op == "delete" {
if ch.Op != "put" {
// A delete, or an expiry (issue 1952): the key is gone.
handle(ch.Key, nil)
} else {
handle(ch.Key, ch.Value)
+74 -6
View File
@@ -7,6 +7,10 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-sdk/go/provisions"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
@@ -41,19 +45,39 @@ func fill(t *testing.T, template string, entry map[string]any) string {
func TestTheRuntimeAnswersAHandedConsumersMcpEndpointThroughTheHelper(t *testing.T) {
mesh := mt.New(t)
bound := mesh.Bucket(t, "node-tools_bound")
assignments := mesh.Bucket(t, "mesh-controller_assignments")
plans := mesh.Bucket(t, provisions.PlansBucket)
provs := mesh.Bucket(t, provisions.ProvisionsBucket)
mem := mt.MembershipOf("node-tools", "anchor", false, nil)
mem.State = []bus.StateIssued{
{Name: "bound", Bucket: "node-tools_bound", Writes: true},
{Name: "mesh-controller.assignments", Bucket: "mesh-controller_assignments"},
{Name: provisions.PlansRead, Bucket: provisions.PlansBucket},
{Name: provisions.ProvisionsRead, Bucket: provisions.ProvisionsBucket},
}
mesh.Issue(t, mem)
// claude-code, handed on this machine, is bound to node-tools here for mcp-endpoint (issue 788's plan entry).
plan := `{"node":"anchor","module":"claude-code","provisioning":{"provisions":[{"provision":"mcp-endpoint",` +
`"provider":"anchor","provider-module":"node-tools","at":"127.0.0.1","from":"anchor","as":"mesh_anchor_claude_code"}]}}`
if _, err := assignments.Put("anchor.claude-code", []byte(plan)); err != nil {
// claude-code, handed on this machine, is bound to node-tools here for mcp-endpoint: its plan and its provision
// key, written as mesh-provisioner writes them, with mesh-sdk's encoders (ADR 0570 clauses 1-3, issue 1954).
plan, err := provisions.EncodePlan(provisions.Plan{Node: "anchor", Module: "claude-code", Plan: catalogue.Plan{
Provisions: []catalogue.PlanProvision{{Provision: "mcp-endpoint", Provider: "anchor", ProviderModule: "node-tools",
At: "127.0.0.1", From: "anchor", As: "mesh_anchor_claude_code"}}}})
if err != nil {
t.Fatal(err)
}
if _, err := plans.Put("anchor.claude-code", plan); err != nil {
t.Fatal(err)
}
key, err := provisions.Key("anchor", "claude-code", "mcp-endpoint", "")
if err != nil {
t.Fatal(err)
}
asked, err := provisions.Encode(provisions.Provision{Provision: "mcp-endpoint", State: provisions.Asked,
Consumer: provisions.End{Node: "anchor", Module: "claude-code"}, Provider: provisions.End{Node: "anchor", Module: "node-tools"},
At: time.Now()})
if err != nil {
t.Fatal(err)
}
if _, err := provs.Put(key, asked); err != nil {
t.Fatal(err)
}
@@ -102,6 +126,8 @@ func TestTheOfferServesThePortTheManifestServes(t *testing.T) {
var m struct {
Serves map[string]map[string]any `json:"serves"`
Moved bool `json:"moved"`
State []string `json:"state"`
Reads []string `json:"reads"`
Provides []struct {
Name string `json:"name"`
AnsweredBy string `json:"answered-by"`
@@ -133,11 +159,32 @@ func TestTheOfferServesThePortTheManifestServes(t *testing.T) {
t.Error("the manifest still carries a consumers file the helper does not need")
}
}
for _, r := range m.Reads {
if r == "mesh-placement.assignments" {
t.Error("the manifest still reads mesh-placement.assignments, which the sdk's helper no longer reads (issue 1954)")
}
}
for _, a := range m.Build.Artifacts {
if a.Name == "files" || len(a.Env) > 0 {
t.Errorf("the build still carries %+v, which only a moved module needs", a)
}
}
// Every bucket the sdk's helper reaches is a state the manifest keeps or reads, by the sdk's names: the provider's
// own and the seat mesh-provisioner's provisions and plans (issue 1954 L6-T1); the controller's assignments no more.
declared := map[string]bool{}
for _, n := range append(m.State, m.Reads...) {
declared[n] = true
}
for bucket, want := range map[string]string{
provider.BucketOf(Module): "bound",
provisions.ProvisionsBucket: "mesh-provisioner.provisions",
provisions.PlansBucket: "mesh-provisioner.plans",
} {
got, err := (busOf{}).state(bucket)
if err != nil || got != want || !declared[got] {
t.Errorf("the helper's bucket %s reaches state %q (%v), want %q declared in the manifest", bucket, got, err, want)
}
}
}
// Only the machine's runtime is the provider: a process on another credential, or with no machine, answers nobody.
@@ -147,3 +194,24 @@ func TestOnlyTheMachinesRuntimeStartsTheHelper(t *testing.T) {
Start(nil, c.module, c.node, time.Second, said.Logf)()
}
}
// 1126/B19 (ADR 0590): this manifest is not refused by the check that refuses any manifest naming the controller:
// it reads, invokes, claims and requires nothing of it.
func TestTheManifestNamesNothingOfTheController(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.Decode(raw)
if err != nil {
t.Fatal(err)
}
// Read as text, as mesh-catalog's controllerdeps check reads its own: mesh-sdk names no module (issue 774), and the
// manifest says it is the tool runner rather than being known by its name.
if strings.Contains(string(raw), "mesh-controller") {
t.Error("node-tools' manifest names the ended controller")
}
if !catalogue.IsToolRunner(m) {
t.Error("node-tools' manifest does not say it is the tool runner (\"tool-runner\": true)")
}
}
+107
View File
@@ -0,0 +1,107 @@
package runtime
import (
"encoding/json"
"fmt"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// BusState answers a bundle's calls on its module's bus state in the new form (mesh-sdk buscalls; novox/hq ADR 0595
// §9, design 25 § 12.5; issue 1966): each as the module whose bundle asks, never a module the request names, so a
// bundle reaches its own module's bindings alone.
func (b *moduleBus) BusState(method string, params json.RawMessage) (json.RawMessage, error) {
conn := b.all.conn
var answer any
var err error
switch method {
case buscalls.ObjectsPut:
var r buscalls.ObjectsPutRequest
if err = decode(method, params, &r); err == nil {
answer, err = conn.ObjectsPut(b.module, r)
}
case buscalls.ObjectsGet:
var r buscalls.ObjectsNameRequest
if err = decode(method, params, &r); err == nil {
var got *buscalls.ObjectData
if got, err = conn.ObjectsGet(b.module, r); err == nil && got == nil {
return json.RawMessage("null"), nil
}
answer = got
}
case buscalls.ObjectsDelete:
var r buscalls.ObjectsNameRequest
if err = decode(method, params, &r); err == nil {
err = conn.ObjectsDelete(b.module, r)
answer = buscalls.Done{}
}
case buscalls.ObjectsList:
var r buscalls.ObjectsRequest
if err = decode(method, params, &r); err == nil {
answer, err = conn.ObjectsList(b.module, r)
}
case buscalls.ObjectsInfo:
var r buscalls.ObjectsRequest
if err = decode(method, params, &r); err == nil {
answer, err = conn.ObjectsInfo(b.module, r)
}
case buscalls.StreamRead:
var r buscalls.StreamReadRequest
if err = decode(method, params, &r); err == nil {
answer, err = conn.StreamRead(b.module, r)
}
case buscalls.StreamFill:
var r buscalls.StreamRequest
if err = decode(method, params, &r); err == nil {
answer, err = conn.StreamFill(b.module, r)
}
case buscalls.WorkQueueSize:
var r buscalls.WorkQueueRequest
if err = decode(method, params, &r); err == nil {
answer, err = conn.WorkQueueSize(b.module, r)
}
case buscalls.WorkQueueDead:
var r buscalls.WorkQueueDeadRequest
if err = decode(method, params, &r); err == nil {
answer, err = conn.WorkQueueDead(b.module, r)
}
case buscalls.LedgerApply, buscalls.LedgerGet, buscalls.LedgerHistory, buscalls.LedgerQuery, buscalls.LedgerRebuild,
buscalls.LedgerReindex, buscalls.LedgerSnapshot, buscalls.LedgerLag:
answer, err = b.ledgerCall(method, params)
default:
return nil, fmt.Errorf("the runtime answers no %s", method)
}
if err != nil {
return nil, err
}
return json.Marshal(answer)
}
// BusStateWatch hands a bundle each change of its objects or its ledger, the current ones first.
func (b *moduleBus) BusStateWatch(method string, params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
switch method {
case buscalls.ObjectsWatch:
var r buscalls.ObjectsWatchRequest
if err := decode(method, params, &r); err != nil {
return nil, err
}
return b.all.conn.ObjectsWatch(b.module, r, func(change buscalls.ObjectsChange) error {
raw, err := json.Marshal(change)
if err != nil {
return err
}
return deliver(raw)
})
case buscalls.LedgerWatch:
return b.ledgerWatch(params, deliver)
}
return nil, fmt.Errorf("the runtime answers no %s", method)
}
// decode reads a bus-state call's request.
func decode(method string, params json.RawMessage, into any) error {
if err := json.Unmarshal(params, into); err != nil {
return fmt.Errorf("%s's request is not of its shape (mesh-sdk buscalls): %w", method, err)
}
return nil
}
@@ -13,6 +13,7 @@ import (
type saidCaller struct {
Caller string `json:"caller"`
Call string `json:"call"`
Arguments map[string]any `json:"arguments"`
}
@@ -36,6 +37,7 @@ func TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller(t *testing.T)
asker.Logf = func(string, ...any) {}
defer asker.Close()
calls := map[string]bool{}
for _, key := range []string{"ledger.who", "ledger.who@anchor", "seat:node-desk.who@anchor"} {
var got saidCaller
mt.Until(t, func() error {
@@ -49,6 +51,11 @@ func TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller(t *testing.T)
if got.Caller != "shanks.node-tools" {
t.Errorf("%s reached the module with caller %q, want the asker's bus user", key, got.Caller)
}
// And the id of the call in flight, one per call, which a ledger's apply names (novox/hq ADR 0597 §7).
if got.Call == "" || calls[got.Call] {
t.Errorf("%s reached the module with call id %q, want one of its own", key, got.Call)
}
calls[got.Call] = true
if got.Arguments["caller"] != "novox.node-tools" {
t.Errorf("%s: the arguments did not reach the module as sent: %v", key, got.Arguments)
}
@@ -0,0 +1,145 @@
package runtime
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"reflect"
"sync"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// The ledger credentials (novox/hq ADR 0597 §9, design 25 § 12.4; issue 1966 T6): one per module assignment, held by
// the machine's tool runner in its own secret `ledgers`, a map of module to credential (`{}` when none), which the vault
// rewrites on each mint and retirement. **A module's ledger history is appended, and its snapshots written, only on
// that module's own ledger connection**; everything else stays on the runner's own.
// LedgersFile is where the runner reads its own secret `ledgers`: MESH_LEDGERS_FILE when set, else beside its broker
// credential, both in its directory mesh-state (`${dir:mesh-state}/ledgers`). Its manifest declares the secret only once
// the vault holds it on every machine (B3 of issue 1966): until then a file not there means no ledger credentials, and
// every ledger write is refused naming the missing one.
const LedgersFile = "MESH_LEDGERS_FILE"
// LedgersFileOf is the path of the secret `ledgers` for a runtime whose broker credential is at broker.
func LedgersFileOf(broker string) string {
if file := os.Getenv(LedgersFile); file != "" {
return file
}
if broker == "" {
return ""
}
return filepath.Join(filepath.Dir(broker), "ledgers")
}
// LedgerCredentials are the ledger connections of the modules on this machine, read from the secret `ledgers` and
// followed as it changes, with no restart.
type LedgerCredentials struct {
file string
dial func(bus.Credential) (*bus.Conn, error)
logf func(string, ...any)
mu sync.Mutex
creds map[string]bus.Credential
conns map[string]*bus.Conn
why string // why the file could not be read, when it could not
stop func()
}
// FollowLedgerCredentials reads the secret `ledgers` at file and follows it (WatchFile), dialling a module's connection
// with dial the first time it is asked for.
func FollowLedgerCredentials(file string, dial func(bus.Credential) (*bus.Conn, error), logf func(string, ...any)) *LedgerCredentials {
l := &LedgerCredentials{file: file, dial: dial, logf: logf, creds: map[string]bus.Credential{}, conns: map[string]*bus.Conn{}}
l.read()
stop, err := WatchFile(file, l.read, logf, "a change of the ledger credentials is taken only after a restart")
if err != nil {
logf("[mesh-tools] cannot follow the ledger credentials at %s: %v; a change is taken only after a restart", file, err)
} else {
l.stop = stop
}
return l
}
// read takes the file as it is now: a module whose credential changed or went has its connection closed, and is
// dialled again from the new one when next asked for.
func (l *LedgerCredentials) read() {
creds, why := map[string]bus.Credential{}, ""
raw, err := os.ReadFile(l.file)
switch {
case errors.Is(err, os.ErrNotExist):
why = "the file " + l.file + " is not there yet"
case err != nil:
why = err.Error()
case len(bytes.TrimSpace(raw)) == 0:
why = "the file " + l.file + " is empty"
default:
if err := json.Unmarshal(raw, &creds); err != nil {
why = "the file " + l.file + " is not a map of module to credential: " + err.Error()
creds = map[string]bus.Credential{}
}
}
l.mu.Lock()
var closing []*bus.Conn
for module, conn := range l.conns {
if now, has := creds[module]; !has || !reflect.DeepEqual(now, l.creds[module]) {
closing = append(closing, conn)
delete(l.conns, module)
}
}
saidBefore := l.why
l.creds, l.why = creds, why
l.mu.Unlock()
for _, c := range closing {
c.Close()
}
if why != "" && why != saidBefore {
l.logf("[mesh-tools] the ledger credentials cannot be read (%s); no ledger is written until they can", why)
}
}
// Writer is a module's ledger connection, dialled from its credential the first time it is asked for; refused, naming
// what is missing, when the secret holds none for it.
func (l *LedgerCredentials) Writer(module string) (*bus.Conn, error) {
l.mu.Lock()
defer l.mu.Unlock()
if c := l.conns[module]; c != nil {
return c, nil
}
cred, has := l.creds[module]
if !has {
why := ""
if l.why != "" {
why = " (" + l.why + ")"
}
return nil, fmt.Errorf("%s's ledger credential is missing from the tool runner's own secret ledgers%s: its history "+
"is published only on its own ledger connection, which the grants part issues with its membership and the "+
"vault writes there (novox/hq ADR 0597 §9)", module, why)
}
if cred.Module == "" {
cred.Module = module
}
c, err := l.dial(cred)
if err != nil {
return nil, fmt.Errorf("%s's ledger connection could not be opened: %w", module, err)
}
c.Logf = l.logf
l.conns[module] = c
return c, nil
}
// Stop ends the following and closes every ledger connection.
func (l *LedgerCredentials) Stop() {
if l.stop != nil {
l.stop()
}
l.mu.Lock()
conns := l.conns
l.conns = map[string]*bus.Conn{}
l.mu.Unlock()
for _, c := range conns {
c.Close()
}
}
@@ -0,0 +1,135 @@
package runtime
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// 1966/AC22: given the secret ledgers holding m's credential, when m applies, then the append goes on m's connection
// (the runner's own account may not publish on m's ledger, as the grants part grants it); given it missing, apply is
// refused naming it; a change of the file is taken without a restart.
func TestALedgerIsWrittenOnlyOnItsModulesLedgerConnection(t *testing.T) {
ledgerSubjects := []string{buscompose.LedgerHistorySubject("m", "records") + ".>", buscompose.LedgerSnapshotSubject("m", "records") + ".>"}
users := []*server.User{
{Username: "node-tools", Password: "pw", Permissions: &server.Permissions{
Publish: &server.SubjectPermission{Allow: []string{">"}, Deny: ledgerSubjects}}},
// As the grants part composes a ledger account (mesh-grants buscompose ledgerPermissions): publish on its
// module's ledger histories and snapshots, subscribe its own inbox, nothing else.
{Username: "ledger.anchor.m", Password: "pw", Permissions: &server.Permissions{
Publish: &server.SubjectPermission{Allow: ledgerSubjects},
Subscribe: &server.SubjectPermission{Allow: []string{"_INBOX.ledger.anchor.m.>"}}}},
{Username: "admin", Password: "pw"},
}
s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: t.TempDir(),
NoLog: true, NoSigs: true, Users: users})
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(30 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(s.Shutdown)
url := s.ClientURL()
admin, err := nats.Connect(url, nats.UserInfo("admin", "pw"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(admin.Close)
js, _ := jetstream.New(admin)
ctx := context.Background()
for _, cfg := range []jetstream.StreamConfig{
{Name: "LEDGER_m_records", Subjects: []string{"mesh.ledger.m.records.history.>"}},
{Name: "SNAPSHOT_m_records", Subjects: []string{"mesh.ledger.m.records.snapshot.>"}, MaxMsgsPerSubject: 1},
} {
if _, err := js.CreateStream(ctx, cfg); err != nil {
t.Fatal(err)
}
}
for _, b := range []string{"m_records", buscompose.GrantsMembershipsBucket} {
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b}); err != nil {
t.Fatal(err)
}
}
mem := mt.MembershipOf("m", "anchor", false, nil)
mem.Bindings = []membership.BusBinding{{Local: "records", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger,
History: "LEDGER_m_records", Subject: "mesh.ledger.m.records.history", State: "m_records", Snapshot: "mesh.ledger.m.records.snapshot",
Types: []catalogue.LedgerType{{Name: "decision", Operations: []catalogue.LedgerOperation{
{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}}}}}}}
raw, _ := json.Marshal(mem)
var sdk membership.Membership
_ = json.Unmarshal(raw, &sdk)
grants, _ := js.KeyValue(ctx, buscompose.GrantsMembershipsBucket)
if _, err := grants.Put(ctx, buscompose.MembershipKey("anchor", "m"), membership.EncodeGrantsMembership(sdk)); err != nil {
t.Fatal(err)
}
conn, err := bus.Connect(bus.Credential{URL: url, Module: "node-tools", Node: "anchor", User: "node-tools", Password: "pw"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
conn.Follow("m")
mt.Until(t, func() error {
if conn.Membership("m") == nil {
return os.ErrNotExist
}
return nil
})
// Not there yet, as on a machine whose vault has not made it: no ledger credentials, and the runner serves on.
file := filepath.Join(t.TempDir(), "ledgers")
creds := FollowLedgerCredentials(file, bus.Dial, t.Logf)
t.Cleanup(creds.Stop)
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
WithLedgerWriters(creds.Writer)(all)
m := all.forModule("m").(*moduleBus)
apply := func(instance string) error {
_, err := m.BusState(buscalls.LedgerApply, json.RawMessage(`{"ledger":"records","instance":"`+instance+`","operation":"title","body":{"title":"T"}}`))
return err
}
// Missing: refused, naming it and why.
if err := apply("decision/1"); err == nil || !strings.Contains(err.Error(), "m's ledger credential is missing from the tool runner's own secret ledgers (the file "+file+" is not there yet)") {
t.Fatalf("an apply with no credential: %v", err)
}
// Written: taken without a restart, on m's connection (the runner's own account is denied m's ledger subjects, so an
// append that lands went on m's), whose account stamps the operation.
cred, _ := json.Marshal(map[string]bus.Credential{"m": {URL: url, Node: "anchor", Module: "m", User: "ledger.anchor.m", Password: "pw"}})
if err := os.WriteFile(file, cred, 0o600); err != nil {
t.Fatal(err)
}
mt.Until(t, func() error { return apply("decision/1") })
page, err := m.BusState(buscalls.LedgerHistory, json.RawMessage(`{"ledger":"records","instance":"decision/1"}`))
if err != nil || !strings.Contains(string(page), `"author":{"account":"ledger.anchor.m"}`) {
t.Fatalf("the history holds %s, %v", page, err)
}
// Gone again: refused again, the change taken without a restart.
if err := os.WriteFile(file, []byte(`{}`), 0o600); err != nil {
t.Fatal(err)
}
mt.Until(t, func() error {
if err := apply("decision/2"); err == nil || !strings.Contains(err.Error(), "credential is missing") {
return os.ErrExist
}
return nil
})
}
@@ -0,0 +1,261 @@
//go:build integration
package runtime
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/ledger"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// racing is a history whose append lets another writer apply first, once: before its own append, or after it.
type racing struct {
ledger.History
before, after func()
}
func (r *racing) Append(key string, op []byte, expect uint64) (uint64, error) {
if f := r.before; f != nil {
r.before = nil
f()
}
seq, err := r.History.Append(key, op, expect)
if f := r.after; f != nil && err == nil {
r.after = nil
f()
}
return seq, err
}
// cut is a state whose next write is lost, as a writer stopped between its history and its state would lose it.
type cut struct {
ledger.State
once bool
}
func (c *cut) Write(key string, value []byte, revision uint64) (uint64, error) {
if c.once {
c.once = false
return 0, errors.New("the writer stopped")
}
return c.State.Write(key, value, revision)
}
// 1966/AC19: the ledger keeps its promises against the real bus (ADR 0597's How it is checked; design 25 § 12.7), one
// case each, and every apply leaves the snapshot equal to the state.
func TestTheLedgerKeepsItsPromisesOnTheBus(t *testing.T) {
mesh := mt.New(t)
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := jetstream.New(nc)
ctx := context.Background()
for _, cfg := range []jetstream.StreamConfig{
{Name: "LEDGER_m_records", Subjects: []string{"mesh.ledger.m.records.history.>"}},
{Name: "SNAPSHOT_m_records", Subjects: []string{"mesh.ledger.m.records.snapshot.>"}, MaxMsgsPerSubject: 1},
} {
if _, err := js.CreateStream(ctx, cfg); err != nil {
t.Fatal(err)
}
}
for _, b := range []string{"m_records", "m_records_by_title"} {
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b}); err != nil {
t.Fatal(err)
}
}
types := []catalogue.LedgerType{{Name: "decision", Schema: map[string]catalogue.LedgerField{"title": {Type: "string"}},
Operations: []catalogue.LedgerOperation{
{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}, Author: true},
{Name: "note", Effect: catalogue.EffectAppend, Into: "notes", Fields: []string{"says"}},
},
Indexes: []catalogue.LedgerIndex{{Name: "by-title", On: []string{"title"}}}}}
binding := membership.BusBinding{Local: "records", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger,
History: "LEDGER_m_records", Subject: "mesh.ledger.m.records.history", State: "m_records",
Indexes: map[string]string{"by-title": "m_records_by_title"}, Snapshot: "mesh.ledger.m.records.snapshot"}
mem := mt.MembershipOf("m", "anchor", false, nil)
binding.Types = types
mem.Bindings = []membership.BusBinding{binding}
mesh.Issue(t, mem)
conn := connect(t, "node-tools", "anchor")
conn.Follow("m")
mt.Until(t, func() error {
if conn.Membership("m") == nil {
return errors.New("not issued yet")
}
return nil
})
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
WithLedgerWriters(func(string) (*bus.Conn, error) { return conn, nil })(all)
m := all.forModule("m").(*moduleBus)
open := func(t *testing.T) *ledger.Ledger {
t.Helper()
l, err := all.ledger("m", "records")
if err != nil {
t.Fatal(err)
}
return l
}
apply := func(t *testing.T, l *ledger.Ledger, instance, operation, body string) (buscalls.Applied, error) {
t.Helper()
return l.Apply(buscalls.ApplyRequest{Ledger: "records", Instance: instance, Operation: operation, Body: json.RawMessage(body)},
buscalls.Author{Account: "m"})
}
must := func(t *testing.T, l *ledger.Ledger, instance, operation, body string) buscalls.Applied {
t.Helper()
got, err := apply(t, l, instance, operation, body)
if err != nil {
t.Fatal(err)
}
return got
}
held := func(t *testing.T, key string) ledger.Record {
t.Helper()
raw, _, err := open(t).State.Get(key)
if err != nil {
t.Fatal(err)
}
var r ledger.Record
if raw != nil {
_ = json.Unmarshal(raw, &r)
}
return r
}
snapshotIsState := func(t *testing.T, key string) {
t.Helper()
l := open(t)
state, _, _ := l.State.Get(key)
snap, _, _ := l.Snapshots.Get(key)
if string(snap) != string(state) {
t.Fatalf("the snapshot of %s is %s, its state %s", key, snap, state)
}
}
cases := []struct {
name, key string
run func(t *testing.T)
}{
{"two writers race and one is refused", "decision.race", func(t *testing.T) {
l := open(t)
l.History = &racing{History: l.History, before: func() { must(t, open(t), "decision/race", "note", `{"says":"first"}`) }}
_, err := apply(t, l, "decision/race", "note", `{"says":"second"}`)
if !errors.Is(err, ledger.ErrRaced) || !strings.Contains(err.Error(), "not taken") {
t.Fatalf("the racing writer was answered %v", err)
}
if got := held(t, "decision.race"); string(got.State) != `{"notes":[{"says":"first"}]}` {
t.Fatalf("the state is %s", got.State)
}
}},
{"two interleaved applies leave the state at the higher sequence", "decision.interleave", func(t *testing.T) {
var inner buscalls.Applied
l := open(t)
l.History = &racing{History: l.History, after: func() { inner = must(t, open(t), "decision/interleave", "note", `{"says":"later"}`) }}
outer := must(t, l, "decision/interleave", "note", `{"says":"earlier"}`)
got := held(t, "decision.interleave")
if inner.Sequence <= outer.Sequence || got.Sequence != inner.Sequence ||
string(got.State) != `{"notes":[{"says":"earlier"},{"says":"later"}]}` {
t.Fatalf("applied %d then %d, the state at %d: %s", outer.Sequence, inner.Sequence, got.Sequence, got.State)
}
}},
{"a write cut between history and state is repaired by the next apply", "decision.cut", func(t *testing.T) {
must(t, open(t), "decision/cut", "note", `{"says":"one"}`)
l := open(t)
l.State = &cut{State: l.State, once: true}
if got := must(t, l, "decision/cut", "note", `{"says":"cut"}`); !got.Taken || !got.StateBehind {
t.Fatalf("the cut apply was answered %+v", got)
}
must(t, open(t), "decision/cut", "note", `{"says":"three"}`)
if got := held(t, "decision.cut"); string(got.State) != `{"notes":[{"says":"one"},{"says":"cut"},{"says":"three"}]}` {
t.Fatalf("the state is %s", got.State)
}
}},
{"a full state bucket answers taken with its sequence and catches up", "decision.full", func(t *testing.T) {
must(t, open(t), "decision/full", "note", `{"says":"small"}`)
kv, _ := js.KeyValue(ctx, "m_records")
status, _ := kv.Status(ctx)
if _, err := js.UpdateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: "m_records", MaxBytes: int64(status.Bytes()) + 64}); err != nil {
t.Fatal(err)
}
got := must(t, open(t), "decision/full", "note", fmt.Sprintf(`{"says":%q}`, strings.Repeat("x", 512)))
if !got.Taken || got.Sequence == 0 || !got.StateBehind {
t.Fatalf("the full bucket's apply was answered %+v", got)
}
if _, err := js.UpdateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: "m_records", MaxBytes: -1}); err != nil {
t.Fatal(err)
}
after := must(t, open(t), "decision/full", "note", `{"says":"room"}`)
if st := held(t, "decision.full"); after.StateBehind || st.Sequence != after.Sequence || !strings.Contains(string(st.State), strings.Repeat("x", 512)) {
t.Fatalf("after room: %+v, the state at %d", after, st.Sequence)
}
}},
{"an instance whose operations expired rebuilds from its snapshot", "decision.expired", func(t *testing.T) {
must(t, open(t), "decision/expired", "title", `{"title":"Kept"}`)
want := held(t, "decision.expired")
s, _ := js.Stream(ctx, "LEDGER_m_records")
if err := s.Purge(ctx, jetstream.WithPurgeSubject("mesh.ledger.m.records.history.decision.expired")); err != nil {
t.Fatal(err)
}
kv, _ := js.KeyValue(ctx, "m_records")
_ = kv.Purge(ctx, "decision.expired")
got, err := open(t).Rebuild(buscalls.RebuildRequest{Instance: "decision/expired", From: buscalls.RebuildFromSnapshot})
if st := held(t, "decision.expired"); err != nil || got.Written != 1 || string(st.State) != string(want.State) {
t.Fatalf("rebuilt %+v, %v: the state %s, want %s", got, err, st.State, want.State)
}
}},
{"rebuild, reindex and lag repair by hand", "decision.repair", func(t *testing.T) {
must(t, open(t), "decision/repair", "title", `{"title":"Repaired"}`)
kv, _ := js.KeyValue(ctx, "m_records")
_ = kv.Purge(ctx, "decision.repair")
l := open(t)
lag, err := l.Lag(buscalls.LedgerRequest{})
if err != nil || !strings.Contains(fmt.Sprint(lag), "decision/repair") || len(lag.Indexes) == 0 {
t.Fatalf("lag says %+v, %v", lag, err)
}
if got, err := l.Rebuild(buscalls.RebuildRequest{Instance: "decision/repair"}); err != nil || got.Written != 1 {
t.Fatalf("rebuilt %+v, %v", got, err)
}
if _, err := l.Reindex(buscalls.ReindexRequest{Index: "by-title"}); err != nil {
t.Fatal(err)
}
if lag, err := l.Lag(buscalls.LedgerRequest{}); err != nil || len(lag.Instances) != 0 || len(lag.Indexes) != 0 {
t.Fatalf("lag after the repairs says %+v, %v", lag, err)
}
q, err := l.Query(buscalls.QueryRequest{Index: "by-title", Key: "Repaired.>"})
if err != nil || len(q.Entries) != 1 {
t.Fatalf("queried %+v, %v", q, err)
}
}},
{"a forged author is refused", "decision.forged", func(t *testing.T) {
_, err := m.BusState(buscalls.LedgerApply, json.RawMessage(`{"ledger":"records","instance":"decision/forged","operation":"title","body":{"title":"T","author":"op@shanks"},"call":"forged"}`))
if err == nil || !strings.Contains(err.Error(), "the call forged, which is not a tool call in flight") {
t.Fatalf("a forged call was answered %v", err)
}
if last, _ := open(t).History.Last("decision.forged"); last != 0 {
t.Fatalf("the forged apply was appended at %d", last)
}
}},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
c.run(t)
if c.key != "decision.forged" {
snapshotIsState(t, c.key)
}
})
}
}
+280
View File
@@ -0,0 +1,280 @@
package runtime
import (
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"sync"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
"github.com/novox/mesh-tools/node-tools/internal/ledger"
)
// A module's ledgers as the tool runner serves them (novox/hq ADR 0597, design 25 § 12.4, § 12.5; issue 1966): opened
// for the module whose bundle or whose repair tool asks, from its own bindings alone, with its types and the connection
// its history is published on.
// ledgerSource is how the runner opens a module's ledger beside its binding: the connection its history and snapshots
// are published on (the module's ledger credential, ADR 0597 §9).
type ledgerSource struct {
writer func(module string) (*bus.Conn, error)
}
// callsInFlight are the tool calls the runtime is serving, each with the module whose bundle serves it and the caller
// the runtime proved from the subject the call arrived on (bus.CalledSubject, novox/hq issue 365).
type callsInFlight struct {
mu sync.Mutex
of map[string]callInFlight
}
type callInFlight struct{ module, caller string }
// begin keeps a call of module in flight for caller, under an id no bundle can guess, until end.
func (f *callsInFlight) begin(module, caller string) (string, func()) {
var raw [16]byte
_, _ = rand.Read(raw[:])
id := hex.EncodeToString(raw[:])
f.mu.Lock()
if f.of == nil {
f.of = map[string]callInFlight{}
}
f.of[id] = callInFlight{module, caller}
f.mu.Unlock()
return id, func() {
f.mu.Lock()
delete(f.of, id)
f.mu.Unlock()
}
}
// authorOf is the author an apply of module naming call is stamped with (ADR 0597 §7): the module's account, and the
// caller the runtime proved for that call; naming no call, the account alone. A call not in flight, or another
// module's, is refused, naming it. Nothing the bundle writes is taken as the author.
func (f *callsInFlight) authorOf(module, account, call string) (buscalls.Author, error) {
author := buscalls.Author{Account: account}
if call == "" {
return author, nil
}
f.mu.Lock()
inFlight, ok := f.of[call]
f.mu.Unlock()
switch {
case !ok:
return author, fmt.Errorf("the apply names the call %s, which is not a tool call in flight here; an apply names the "+
"call it serves as its tools/call's _meta %s gave it, or none (novox/hq ADR 0597 §7)", call, buscalls.MetaCall)
case inFlight.module != module:
return author, fmt.Errorf("the apply names the call %s, which is another module's tool call; %s names only its own "+
"(novox/hq ADR 0597 §7)", call, module)
}
author.Caller = inFlight.caller
return author, nil
}
// Begin keeps one of the module's tool calls in flight for the caller the runtime proved (launch.Bus).
func (b *moduleBus) Begin(caller string) (string, func()) { return b.all.calls.begin(b.module, caller) }
// ledger opens one of a module's ledgers by its local name.
func (c *consumers) ledger(module, local string) (*ledger.Ledger, error) {
b, err := c.conn.Binding(module, local, catalogue.StorageLedger)
if err != nil {
return nil, err
}
// Its types travel on its binding, copied by the grants part from the entry the manifest-accepted event carries
// (mesh-sdk membership.BusBinding.Types): the runner reads no other part's state for them (ADR 0596).
ts := b.Types
if len(ts) == 0 {
return nil, fmt.Errorf("%s's ledger %q is bound with no types, so no operation can be checked against one: its "+
"binding carries the types its manifest asks for in `requires` (novox/hq ADR 0597 §1)", module, local)
}
var writer *bus.Conn
if c.ledgers.writer != nil {
if writer, err = c.ledgers.writer(module); err != nil {
return nil, err
}
}
history, state, snapshots, err := c.conn.LedgerStores(module, b, writer)
if err != nil {
return nil, err
}
// The owning module's authenticated account: the user of the connection its history is published on.
account := module
if writer != nil && writer.Credential().User != "" {
account = writer.Credential().User
}
return &ledger.Ledger{Module: module, Name: local, Account: account, Types: ts, History: history, State: state,
Snapshots: snapshots, Indexes: c.conn.LedgerIndexes(b), Logf: c.logf}, nil
}
// ledgerCall answers a bundle's ledger call, as the module whose bundle asks.
func (b *moduleBus) ledgerCall(method string, params json.RawMessage) (any, error) {
var named struct {
Ledger string `json:"ledger"`
}
if err := decode(method, params, &named); err != nil {
return nil, err
}
if method == buscalls.LedgerApply {
var r buscalls.ApplyRequest
if err := decode(method, params, &r); err != nil {
return nil, err
}
// The call is judged first: a forged one is refused before anything of the ledger is touched.
if _, err := b.all.calls.authorOf(b.module, "", r.Call); err != nil {
return nil, err
}
l, err := b.all.ledger(b.module, r.Ledger)
if err != nil {
return nil, err
}
author, err := b.all.calls.authorOf(b.module, l.Account, r.Call)
if err != nil {
return nil, err
}
return l.Apply(r, author)
}
l, err := b.all.ledger(b.module, named.Ledger)
if err != nil {
return nil, err
}
switch method {
case buscalls.LedgerGet:
var r buscalls.InstanceRequest
_ = json.Unmarshal(params, &r)
return l.Get(r)
case buscalls.LedgerHistory:
var r buscalls.HistoryRequest
_ = json.Unmarshal(params, &r)
return l.ReadHistory(r)
case buscalls.LedgerQuery:
var r buscalls.QueryRequest
_ = json.Unmarshal(params, &r)
return l.Query(r)
case buscalls.LedgerRebuild:
var r buscalls.RebuildRequest
_ = json.Unmarshal(params, &r)
return l.Rebuild(r)
case buscalls.LedgerReindex:
var r buscalls.ReindexRequest
_ = json.Unmarshal(params, &r)
return l.Reindex(r)
case buscalls.LedgerSnapshot:
var r buscalls.InstanceRequest
_ = json.Unmarshal(params, &r)
return l.Snapshot(r)
case buscalls.LedgerLag:
var r buscalls.LedgerRequest
_ = json.Unmarshal(params, &r)
return l.Lag(r)
}
return nil, fmt.Errorf("the runtime answers no %s", method)
}
// ledgerWatch hands a bundle each change of its ledger's states.
func (b *moduleBus) ledgerWatch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
var r buscalls.LedgerWatchRequest
if err := decode(buscalls.LedgerWatch, params, &r); err != nil {
return nil, err
}
l, err := b.all.ledger(b.module, r.Ledger)
if err != nil {
return nil, err
}
return l.Watch(r, func(change buscalls.InstanceChange) error {
raw, err := json.Marshal(change)
if err != nil {
return err
}
return deliver(raw)
})
}
// hasLedger is whether a module's membership binds a ledger.
func hasLedger(conn *bus.Conn, module string) bool {
for _, b := range conn.Bindings(module) {
if b.Storage == catalogue.StorageLedger {
return true
}
}
return false
}
// The repair tools the runner serves for every module owning a ledger (design 25 § 12.5), named so no bundle's tool is
// taken for one.
const (
RepairRebuild = "ledger_rebuild"
RepairReindex = "ledger_reindex"
RepairSnapshot = "ledger_snapshot"
RepairLag = "ledger_lag"
)
// The effects a tool carries (novox/hq ADR 0387 §1).
const (
EffectReads = "reads"
EffectActs = "acts"
)
// repairTools are a module's ledger repairs as tools, each limited to its own ledgers: the request's ledger is a local
// name of the module whose tool is called.
func (c *consumers) repairTools(module string) []launch.Tool {
schema := func(props string, required ...string) json.RawMessage {
req, _ := json.Marshal(append([]string{"ledger"}, required...))
return json.RawMessage(`{"type":"object","properties":{"ledger":{"type":"string","description":"the ledger's local name"}` +
props + `},"required":` + string(req) + `}`)
}
instance := `,"instance":{"type":"string","description":"one instance, such as decision/0290; every instance when empty"}`
run := func(do func(l *ledger.Ledger, args json.RawMessage) (any, error)) func(string, json.RawMessage) (json.RawMessage, error) {
return func(_ string, args json.RawMessage) (json.RawMessage, error) {
var named struct {
Ledger string `json:"ledger"`
}
if err := json.Unmarshal(args, &named); err != nil || named.Ledger == "" {
return nil, errors.New("a repair names the ledger, by its local name")
}
l, err := c.ledger(module, named.Ledger)
if err != nil {
return nil, err
}
answer, err := do(l, args)
if err != nil {
return nil, err
}
return json.Marshal(answer)
}
}
return []launch.Tool{
{Name: RepairRebuild, Effect: EffectActs, Input: schema(instance + `,"from":{"type":"string","enum":["history","snapshot"],"description":"derive from the whole history (the default) or from the snapshot and the operations after it"}`),
Description: "Derive a ledger instance's state again from its history or its snapshot, and write it and its snapshot (novox/hq ADR 0597 §10).",
Run: run(func(l *ledger.Ledger, args json.RawMessage) (any, error) {
var r buscalls.RebuildRequest
_ = json.Unmarshal(args, &r)
return l.Rebuild(r)
})},
{Name: RepairReindex, Effect: EffectActs, Input: schema(`,"index":{"type":"string","description":"the declared index"}`, "index"),
Description: "Build a ledger's declared index again from its states, derived from its history (novox/hq ADR 0597 §4, §10).",
Run: run(func(l *ledger.Ledger, args json.RawMessage) (any, error) {
var r buscalls.ReindexRequest
_ = json.Unmarshal(args, &r)
return l.Reindex(r)
})},
{Name: RepairSnapshot, Effect: EffectActs, Input: schema(instance),
Description: "Rewrite the snapshot of a ledger instance that is behind its state (novox/hq ADR 0597 §8, §10).",
Run: run(func(l *ledger.Ledger, args json.RawMessage) (any, error) {
var r buscalls.InstanceRequest
_ = json.Unmarshal(args, &r)
return l.Snapshot(r)
})},
{Name: RepairLag, Effect: EffectReads, Input: schema(""),
Description: "Say every ledger instance whose state or snapshot is behind its history, and each declared index behind it (novox/hq ADR 0597 §10).",
Run: run(func(l *ledger.Ledger, args json.RawMessage) (any, error) {
var r buscalls.LedgerRequest
_ = json.Unmarshal(args, &r)
return l.Lag(r)
})},
}
}
+181
View File
@@ -0,0 +1,181 @@
package runtime
import (
"context"
"encoding/json"
"errors"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// 1966/AC16: given a module with a ledger binding, when the runner lists its tools, then the four repair tools are
// among them, marked acts or reads; a module without one has none, and a binding written later brings them.
func TestTheLedgerRepairsAreToolsOfEveryLedgerOwner(t *testing.T) {
mesh := mt.New(t)
owner := mt.MembershipOf("owner", "anchor", true, nil)
owner.Bindings = []membership.BusBinding{{Local: "records", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger,
History: "LEDGER_owner_records", Subject: "mesh.ledger.owner.records.history", State: "owner_records",
Snapshot: "mesh.ledger.owner.records.snapshot"}}
plain := mt.MembershipOf("plain", "anchor", true, nil)
mesh.Issue(t, owner)
mesh.Issue(t, plain)
nodeTools := connect(t, "node-tools", "anchor")
stop, err := Run(nodeTools, []Served{{"owner", []string{mt.Fixture("caller-echo.py")}},
{"plain", []string{mt.Fixture("caller-echo.py")}}}, nil, func(string, ...any) {})
if err != nil {
t.Fatal(err)
}
defer stop()
asker := connect(t, "console", "workstation")
listed := func(module string) (string, error) {
answered, err := asker.Ask(module+"."+ToolsVerb, map[string]any{}, "")
if err != nil {
return "", err
}
var got ToolsAnswer
if err := json.Unmarshal(answered.Result, &got); err != nil {
return "", err
}
var names []string
for _, tool := range got.Tools {
names = append(names, tool.Name+":"+tool.Effect)
}
sort.Strings(names)
return strings.Join(names, " "), nil
}
for _, c := range []struct{ name, module, want string }{
{"a ledger owner lists the four repairs", "owner", "ledger_lag:reads ledger_rebuild:acts ledger_reindex:acts ledger_snapshot:acts who:"},
{"a module with no ledger has none", "plain", "who:"},
} {
t.Run(c.name, func(t *testing.T) {
var got string
mt.Until(t, func() (err error) { got, err = listed(c.module); return })
if got != c.want {
t.Fatalf("%s lists %q, want %q", c.module, got, c.want)
}
})
}
t.Run("a ledger bound later brings them", func(t *testing.T) {
plain.Bindings = owner.Bindings
mesh.Issue(t, plain)
mt.Until(t, func() error {
got, err := listed("plain")
if err == nil && !strings.Contains(got, "ledger_rebuild:acts") {
err = errors.New("listed " + got)
}
return err
})
})
}
// aLedger raises one ledger of module m on the bus as the bus module would create it, binds it in m's membership, and
// answers the runtime's buses for m and for other; its types travel on its binding, and a second binding carries none.
func aLedger(t *testing.T) (m, other *moduleBus, mesh *mt.Mesh) {
t.Helper()
mesh = mt.New(t)
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := jetstream.New(nc)
ctx := context.Background()
for _, cfg := range []jetstream.StreamConfig{
{Name: "LEDGER_m_records", Subjects: []string{"mesh.ledger.m.records.history.>"}},
{Name: "SNAPSHOT_m_records", Subjects: []string{"mesh.ledger.m.records.snapshot.>"}, MaxMsgsPerSubject: 1},
} {
if _, err := js.CreateStream(ctx, cfg); err != nil {
t.Fatal(err)
}
}
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: "m_records"}); err != nil {
t.Fatal(err)
}
mem := mt.MembershipOf("m", "anchor", false, nil)
mem.Bindings = []membership.BusBinding{{Local: "records", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger,
History: "LEDGER_m_records", Subject: "mesh.ledger.m.records.history", State: "m_records", Snapshot: "mesh.ledger.m.records.snapshot",
Types: []catalogue.LedgerType{{Name: "decision", Schema: map[string]catalogue.LedgerField{"title": {Type: "string"}},
Operations: []catalogue.LedgerOperation{{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}, Author: true}}}}},
{Local: "untyped", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger, History: "LEDGER_m_records",
State: "m_records"},
{Local: "elsewhere", Storage: catalogue.StorageLedger, Template: catalogue.TemplateLedger, History: "LEDGER_m_records",
Subject: "mesh.ledger.m.records.history", State: "m_records", Types: []catalogue.LedgerType{{Name: "decision"}}}}
mesh.Issue(t, mem)
mesh.Issue(t, mt.MembershipOf("other", "anchor", false, nil))
conn := connect(t, "node-tools", "anchor")
conn.Follow("m")
conn.Follow("other")
mt.Until(t, func() error {
if conn.Membership("m") == nil || conn.Membership("other") == nil {
return errors.New("not issued yet")
}
return nil
})
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}, ledgers: ledgerSource{
writer: func(string) (*bus.Conn, error) { return conn, nil },
}}
return all.forModule("m").(*moduleBus), all.forModule("other").(*moduleBus), mesh
}
// 1966/AC17, 1966/AC18: the author an apply is stamped with is the caller the runtime proved for the call in flight it
// names, whatever author the bundle wrote; another module's call, or one not in flight, is refused naming it (ADR 0597 §7).
func TestAnApplyIsStampedWithTheCallerTheRuntimeProved(t *testing.T) {
m, other, _ := aLedger(t)
call, end := m.Begin("op@shanks")
defer end()
theirs, endTheirs := other.Begin("op@shanks")
defer endTheirs()
forged := `"body":{"title":"T","author":{"account":"x","caller":"forged"}}`
for _, c := range []struct{ name, params, caller, refusal string }{
{"1966/AC17 a call in flight stamps its proven caller", `{"ledger":"records","instance":"decision/1","operation":"title",` + forged + `,"call":"` + call + `","author":"forged"}`, "op@shanks", ""},
{"naming no call stamps the account alone", `{"ledger":"records","instance":"decision/2","operation":"title",` + forged + `}`, "", ""},
{"1966/AC18 another module's call is refused, naming it", `{"ledger":"records","instance":"decision/3","operation":"title",` + forged + `,"call":"` + theirs + `"}`, "", "the call " + theirs + ", which is another module's"},
{"a ledger bound with no types is refused, naming it", `{"ledger":"untyped","instance":"decision/3","operation":"title",` + forged + `}`, "", `m's ledger "untyped" is bound with no types`},
{"a binding naming subjects its account is not granted is refused, naming both", `{"ledger":"elsewhere","instance":"decision/3","operation":"title",` + forged + `}`, "", "bound with the history subject mesh.ledger.m.records.history, and its ledger account is granted mesh.ledger.m.elsewhere.history alone"},
{"a call not in flight is refused, naming it", `{"ledger":"records","instance":"decision/3","operation":"title",` + forged + `,"call":"c0ffee"}`, "", "the call c0ffee, which is not a tool call in flight"},
} {
t.Run(c.name, func(t *testing.T) {
var req struct {
Instance string `json:"instance"`
}
_ = json.Unmarshal([]byte(c.params), &req)
_, err := m.BusState(buscalls.LedgerApply, json.RawMessage(c.params))
if c.refusal != "" {
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Fatalf("applied with %v, want refused with %q", err, c.refusal)
}
return
}
if err != nil {
t.Fatal(err)
}
raw, err := m.BusState(buscalls.LedgerHistory, json.RawMessage(`{"ledger":"records","instance":"`+req.Instance+`"}`))
if err != nil {
t.Fatal(err)
}
var page buscalls.HistoryPage
_ = json.Unmarshal(raw, &page)
if len(page.Operations) != 1 || page.Operations[0].Author != (buscalls.Author{Account: "m", Caller: c.caller}) {
t.Fatalf("the history holds %+v, want the author {m %s}", page.Operations, c.caller)
}
state, _ := m.BusState(buscalls.LedgerGet, json.RawMessage(`{"ledger":"records","instance":"`+req.Instance+`"}`))
if !strings.Contains(string(state), `"author":{"account":"m"`) || strings.Contains(string(state), "forged") {
t.Fatalf("the state took the bundle's author: %s", state)
}
})
}
end()
if _, err := m.BusState(buscalls.LedgerApply, json.RawMessage(`{"ledger":"records","instance":"decision/4","operation":"title","call":"`+call+`"}`)); err == nil {
t.Fatal("a call that ended was still taken as in flight")
}
}
+66 -4
View File
@@ -20,9 +20,13 @@ func logBuses(t *testing.T, maxBytes int64) (issues, other launch.Bus, mesh *mt.
t.Helper()
mesh = mt.New(t)
m := mt.MembershipOf("issues", "anchor", false, nil)
m.Logs = []bus.LogIssued{mesh.Log(t, "issues", "changes", maxBytes)}
changes := mesh.Log(t, "issues", "changes", maxBytes)
m.Logs = []bus.LogIssued{changes}
mesh.Issue(t, m)
mesh.Issue(t, mt.MembershipOf("other", "anchor", false, nil))
// other reads issues' log, as its manifest's reads declare it (ADR 0571): `<module>.<log>`, without writing it.
o := mt.MembershipOf("other", "anchor", false, nil)
o.Logs = []bus.LogIssued{{Name: "issues.changes", Stream: changes.Stream, Subject: changes.Subject}}
mesh.Issue(t, o)
conn := connect(t, "node-tools", "anchor")
conn.Follow("issues")
conn.Follow("other")
@@ -111,7 +115,7 @@ func infoOf(t *testing.T, b launch.Bus) map[string]uint64 {
// RFC 3339; last is a key's last entry or null; info says entries, bytes, cap, first and last — and no
// consumer is made for any of it.
func TestALogAnswersEachVerbOnTheBus(t *testing.T) {
b, _, mesh := logBuses(t, 1<<20)
b, other, mesh := logBuses(t, 1<<20)
before := time.Now().Add(-time.Second)
var seqs []uint64
@@ -189,6 +193,64 @@ func TestALogAnswersEachVerbOnTheBus(t *testing.T) {
if n := mesh.Consumers(t, "LOG_issues_changes"); n != 0 {
t.Fatalf("the runtime made %d consumers of the log; it uses none", n)
}
// From a time (ADR 0571): `since` in place of `from`, RFC 3339; both at once are refused.
cut := all.Entries[2].At
if p := readOf(t, b, fmt.Sprintf(`{"log":"changes","since":%q}`, cut)); len(p.Entries) == 0 || p.Entries[0].Sequence > seqs[2] ||
p.Entries[len(p.Entries)-1].Sequence != seqs[4] {
t.Fatalf("a read since %s: %+v", cut, p)
}
if p := readOf(t, b, fmt.Sprintf(`{"log":"changes","since":%q}`, time.Now().Add(time.Hour).Format(time.RFC3339))); len(p.Entries) != 0 {
t.Fatalf("a read since an hour ahead: %+v", p)
}
if _, err := logAsk(b, "read", `{"log":"changes","from":2,"since":"2026-10-29T00:00:00Z"}`); err == nil {
t.Fatal("a read from a sequence and a time was not refused")
}
// The newest of each key, in the log's order; `**` for the rest, as mesh-sdk says it.
raw, err = logAsk(b, "newest", `{"log":"changes"}`)
var newest []struct {
Sequence uint64 `json:"sequence"`
Key string `json:"key"`
}
if err != nil || json.Unmarshal(raw, &newest) != nil || len(newest) != 3 || newest[0].Key != "13" || newest[1].Key != "14.notes" ||
newest[2].Sequence != seqs[4] {
t.Fatalf("newest: %s %v", raw, err)
}
if raw, err := logAsk(b, "newest", `{"log":"changes","keys":"14.**"}`); err != nil || !strings.Contains(string(raw), `"14.notes"`) ||
strings.Contains(string(raw), `"12"`) {
t.Fatalf("newest of 14.**: %s %v", raw, err)
}
// Another module's log, read by the name its reads declare; never written.
if p := readOf(t, other, `{"log":"issues.changes","from":1}`); len(p.Entries) != 5 {
t.Fatalf("other's read of issues.changes: %+v", p)
}
if _, err := logAsk(other, "append", `{"log":"issues.changes","key":"1","value":{}}`); err == nil ||
!strings.Contains(err.Error(), "without writing it") {
t.Fatalf("other's append to the log it reads: %v", err)
}
// A watch hands each entry from where it is asked, then each new one, one at a time.
heard := make(chan bus.LogEntry, 16)
stop, err := b.LogWatch(json.RawMessage(fmt.Sprintf(`{"log":"changes","keys":"12","from":%d}`, seqs[2])), func(raw json.RawMessage) error {
var e bus.LogEntry
_ = json.Unmarshal(raw, &e)
heard <- e
return nil
})
if err != nil {
t.Fatal(err)
}
defer stop()
next := appendOf(t, b, `{"log":"changes","key":"12","value":{"op":"late"}}`)
for _, want := range []uint64{seqs[2], seqs[4], next} {
select {
case e := <-heard:
if e.Sequence != want || e.Key != "12" {
t.Fatalf("the watch handed %d %s, want %d", e.Sequence, e.Key, want)
}
case <-time.After(5 * time.Second):
t.Fatalf("the watch did not hand entry %d", want)
}
}
}
// A page holds about 4 MiB at most, whatever its limit, and says where to continue.
@@ -267,7 +329,7 @@ func TestALogNotIssuedIsRefusedWithTheReason(t *testing.T) {
} {
_, err := logAsk(other, verb, params)
if err == nil || !strings.Contains(err.Error(), `other keeps no log called "changes"`) ||
!strings.Contains(err.Error(), "issued: none") {
!strings.Contains(err.Error(), "issued: issues.changes") {
t.Errorf("%s of a log not issued: %v", verb, err)
}
}
+157 -9
View File
@@ -52,6 +52,8 @@ type ListedTool struct {
Description string `json:"description"`
Input json.RawMessage `json:"input"`
Subjects []string `json:"subjects,omitempty"`
// Effect is the tool's mark where the runtime knows it (novox/hq ADR 0387 §1): its own tools, the ledger repairs.
Effect string `json:"effect,omitempty"`
}
// NodeRuntimeModule is the module that is a machine's runtime, serving every module on it.
@@ -59,7 +61,7 @@ const NodeRuntimeModule = "node-tools"
// runtimeOnly are the words of the runtime's own environment no bundle is given.
var runtimeOnly = map[string]bool{ToolEnv: true, ToolModules: true, "MESH_BROKER_FILE": true,
"MESH_BROKER_URL": true, "MESH_CONSOLE_LISTEN": true}
"MESH_BROKER_URL": true, "MESH_CONSOLE_LISTEN": true, LedgersFile: true}
// ServedModulesFrom reads MESH_TOOL_MODULES: `<module>=<entrypoint>` entries, comma-separated, several
// per module. The one-module form — a bare path, or the runtime's own module — is the per-module
@@ -133,6 +135,60 @@ type registration struct {
module string // the module's own name, or a seat's
owner string // the module whose bundle made it
tools []launch.Tool
// repairs says it is the runtime's own ledger repairs for the module, not a bundle's (ledgers.go).
repairs bool
}
// withRepairs is regs with the module's ledger repairs as its binding now says: added when it binds a ledger, removed
// when it binds none.
func withRepairs(regs []registration, repairs *registration) []registration {
out := make([]registration, 0, len(regs)+1)
for _, r := range regs {
if !r.repairs {
out = append(out, r)
}
}
if repairs != nil {
out = append(out, *repairs)
}
return out
}
// repairsOf is a module's ledger repairs as a registration, nil when its membership binds no ledger.
func (r *runner) repairsOf(module string) *registration {
if !hasLedger(r.conn, module) {
return nil
}
return &registration{module: module, owner: module, tools: r.events.repairTools(module), repairs: true}
}
// refreshRepairs serves each module's ledger repairs as its membership binds a ledger now, re-serving its tools where
// that changed (a binding is written when the bus module created the ledger, design 25 § 12.2).
func (r *runner) refreshRepairs() {
r.mu.Lock()
defer r.mu.Unlock()
changed := false
for _, m := range r.modules {
l := r.of[m]
served := false
for _, reg := range l.own {
served = served || reg.repairs
}
repairs := r.repairsOf(m)
if served == (repairs != nil) {
continue
}
for i := len(l.serving) - 1; i >= 0; i-- {
l.serving[i]()
}
l.serving, l.names = nil, nil
l.own, l.regs = withRepairs(l.own, repairs), withRepairs(l.regs, repairs)
r.serve(l)
changed = true
}
if changed {
r.bindVerbs()
}
}
// loadedModule is one served module as its bundles were last loaded: what they registered, what of it
@@ -178,9 +234,17 @@ type runner struct {
stops []func() // the watch, the announcement and the seats, in the order they began
}
// Option sets how Run serves, beside what it is given.
type Option func(*consumers)
// WithLedgerWriters gives the runtime the connection each module's ledger history is published on (ledger_credentials.go).
func WithLedgerWriters(writer func(module string) (*bus.Conn, error)) Option {
return func(c *consumers) { c.ledgers.writer = writer }
}
// Run launches, binds and serves, and serves each module's bundle again when it appears, changes or
// goes, with no restart. It answers a stop function.
func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, logf func(string, ...any)) (func(), error) {
func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, logf func(string, ...any), opts ...Option) (func(), error) {
if account := os.Getenv(OperatorAccount); account != "" {
home := ""
if h := os.Getenv(OperatorHome); h != "" {
@@ -203,6 +267,9 @@ func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, log
// The module's events, for every child of it that subscribes (ADR 0198): one consumer per module,
// bound the first time any of its children subscribes, each event handed to every child that did.
r.events = &consumers{conn: conn, logf: logf, of: map[string]*moduleEvents{}}
for _, opt := range opts {
opt(r.events)
}
for _, s := range served {
r.of[s.Module] = r.load(s)
@@ -230,6 +297,7 @@ func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, log
line += fmt.Sprintf("; not serving %s, whose bundle(s) failed to load", strings.Join(failedNames, ", "))
}
logf("%s", line)
conn.OnMembership(func(bus.Membership) { go func() { r.refreshRepairs(); conn.Flush() }() })
stopSeats, again := serveSeats(conn, r.modules, r.registrations, logf)
r.seatsAgain = again
r.stops = append(r.stops, stopSeats)
@@ -341,6 +409,9 @@ func (r *runner) load(s Served) *loadedModule {
}
}
}
if repairs := r.repairsOf(s.Module); repairs != nil {
l.regs, l.own = append(l.regs, *repairs), append(l.own, *repairs)
}
return l
}
@@ -411,7 +482,7 @@ func (r *runner) bindVerbs() {
answer := ToolsAnswer{Module: module, Tools: []ListedTool{}, Failed: why}
for _, t := range tools {
answer.Tools = append(answer.Tools, ListedTool{Name: t.Name, Description: t.Description,
Input: t.Input, Subjects: subjectsOf(r.conn, module, t.Name)})
Input: t.Input, Subjects: subjectsOf(r.conn, module, t.Name), Effect: t.Effect})
}
return answer, nil
})
@@ -682,6 +753,10 @@ type consumers struct {
seatBound map[string]func()
// provisioned is every watch a module's code holds of its values at runtime (provision.go).
provisioned provisionWatches
// ledgers is how a module's ledger is opened (ledgers.go).
ledgers ledgerSource
// calls are the tool calls in flight, by the id each bundle was handed (ledgers.go, novox/hq ADR 0597 §7).
calls callsInFlight
}
type moduleEvents struct {
@@ -868,10 +943,14 @@ type stateAsked struct {
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Revision uint64 `json:"revision"`
// TTLSeconds gives a created or put key its own time to live (issue 1952): create makes it only when it has no
// value, put renews it whole. Zero is none.
TTLSeconds int `json:"ttl_seconds"`
}
// State answers a bundle's `get`, `put`, `delete`, `keys`, `create`, `update` and `history` on its
// module's state (ADR 0201, ADR 0259, ADR 0307). The runtime refuses, with the reason, a state the module was not issued and a write to one it only reads.
// State answers a bundle's `get`, `put`, `delete`, `keys`, `create`, `update`, `history` and `info` on its
// module's state (ADR 0201, ADR 0259, ADR 0307); `create` and `put` take `ttl_seconds`, a key's own time to live,
// whose end every watch hears as an `expire` (issue 1952). The runtime refuses, with the reason, a state the module was not issued and a write to one it only reads.
func (b *moduleBus) State(verb string, params json.RawMessage) (json.RawMessage, error) {
var asked stateAsked
if err := json.Unmarshal(params, &asked); err != nil || asked.State == "" {
@@ -897,7 +976,13 @@ func (b *moduleBus) State(verb string, params json.RawMessage) (json.RawMessage,
}
answer = kept
case "put":
revision, err := conn.StatePut(b.module, asked.State, asked.Key, asked.Value)
put := conn.StatePut
if asked.TTLSeconds != 0 {
put = func(module, name, key string, value json.RawMessage) (uint64, error) {
return conn.StateRenewTTL(module, name, key, value, time.Duration(asked.TTLSeconds)*time.Second)
}
}
revision, err := put(b.module, asked.State, asked.Key, asked.Value)
if err != nil {
return nil, err
}
@@ -907,6 +992,13 @@ func (b *moduleBus) State(verb string, params json.RawMessage) (json.RawMessage,
return nil, err
}
answer = map[string]any{}
case "info":
// The bucket's fill against its caps, so a bundle's tools can say how full it is.
info, err := conn.StateInfo(b.module, asked.State)
if err != nil {
return nil, err
}
answer = info
case "keys":
keys, err := conn.StateKeys(b.module, asked.State)
if err != nil {
@@ -915,13 +1007,23 @@ func (b *moduleBus) State(verb string, params json.RawMessage) (json.RawMessage,
answer = keys
case "create":
// Only when the key has no value: of two writers making it, one wins (novox/hq ADR 0259).
revision, err := conn.StateCreate(b.module, asked.State, asked.Key, asked.Value)
create := conn.StateCreate
if asked.TTLSeconds != 0 {
create = func(module, name, key string, value json.RawMessage) (uint64, error) {
return conn.StateCreateTTL(module, name, key, value, time.Duration(asked.TTLSeconds)*time.Second)
}
}
revision, err := create(b.module, asked.State, asked.Key, asked.Value)
if err != nil {
return nil, err
}
answer = map[string]any{"revision": revision}
case "update":
// Only when the key is still at the revision read: a second answer to one ask loses (ADR 0259).
if asked.TTLSeconds != 0 {
return nil, fmt.Errorf("mesh/state.update gives no time to live: the bus sets one only when a key is " +
"created; put the key with ttl_seconds to renew it whole (issue 1952)")
}
revision, err := conn.StateUpdate(b.module, asked.State, asked.Key, asked.Value, asked.Revision)
if err != nil {
return nil, err
@@ -942,9 +1044,24 @@ type logAsked struct {
From uint64 `json:"from"`
Keys string `json:"keys"`
Limit int `json:"limit"`
// Since is a read's or a watch's start as an RFC 3339 time, in place of From (novox/hq ADR 0571).
Since string `json:"since"`
}
// Log answers a bundle's `append`, `read`, `last` and `info` on its module's log (ADR 0297). The runtime
// since is the asked start time, zero when none.
func (a logAsked) since() (time.Time, error) {
if a.Since == "" {
return time.Time{}, nil
}
at, err := time.Parse(time.RFC3339Nano, a.Since)
if err != nil {
return time.Time{}, fmt.Errorf("since is an RFC 3339 time (2026-10-29T09:30:00Z), not %q", a.Since)
}
return at, nil
}
// Log answers a bundle's `append`, `read`, `newest`, `last` and `info` on its module's log and on a log its module reads
// as `<module>.<log>` (ADR 0297, ADR 0571). The runtime
// refuses, with the reason, a log the module was not issued; an append whose `expect` is stale is
// refused with the code of a lost state update.
func (b *moduleBus) Log(verb string, params json.RawMessage) (json.RawMessage, error) {
@@ -962,11 +1079,22 @@ func (b *moduleBus) Log(verb string, params json.RawMessage) (json.RawMessage, e
}
answer = map[string]any{"sequence": seq}
case "read":
page, err := conn.LogRead(b.module, asked.Log, asked.From, asked.Keys, asked.Limit)
since, err := asked.since()
if err != nil {
return nil, err
}
page, err := conn.LogRead(b.module, asked.Log, asked.From, since, asked.Keys, asked.Limit)
if err != nil {
return nil, err
}
answer = page
case "newest":
// The newest entry of each key the pattern names (mesh-sdk process.LogReader.Newest).
entries, err := conn.LogNewest(b.module, asked.Log, asked.Keys)
if err != nil {
return nil, err
}
answer = entries
case "last":
entry, err := conn.LogLast(b.module, asked.Log, asked.Key)
if err != nil {
@@ -988,6 +1116,26 @@ func (b *moduleBus) Log(verb string, params json.RawMessage) (json.RawMessage, e
return json.Marshal(answer)
}
// LogWatch hands a bundle each entry of a log, `{log, keys, from, since}`: from a sequence, from a time, or with neither
// each new one, one at a time (novox/hq ADR 0571; mesh-sdk process.LogReader.Watch).
func (b *moduleBus) LogWatch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
var asked logAsked
if err := json.Unmarshal(params, &asked); err != nil || asked.Log == "" {
return nil, fmt.Errorf("mesh/log.watch names no log: {log, keys, from, since}")
}
since, err := asked.since()
if err != nil {
return nil, err
}
return b.all.conn.LogWatch(b.module, asked.Log, asked.Keys, asked.From, since, func(e bus.LogEntry) error {
raw, err := json.Marshal(e)
if err != nil {
return err
}
return deliver(raw)
})
}
// Watch hands a bundle its module's state as it is and as it changes (ADR 0201): `{state, key}`, the
// key a pattern with `*` and `**`, empty for every key.
func (b *moduleBus) Watch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
@@ -70,7 +70,7 @@ func TestABundlesSeatTrafficGoesThroughItsModulesBus(t *testing.T) {
if !strings.Contains(string(first), `"duplicate":false`) || !strings.Contains(string(again), `"duplicate":true`) {
t.Errorf("the same id twice: %s, then %s", first, again)
}
if _, err := asking.Seat("publish", json.RawMessage(`{"subject":"mesh.seat.operator-channel.accept.ask.mesh-controller","body":{}}`), nil); err == nil {
if _, err := asking.Seat("publish", json.RawMessage(`{"subject":"mesh.seat.operator-channel.accept.ask.mesh-placement","body":{}}`), nil); err == nil {
t.Error("a module submitted under another's name")
}
got, err := asking.Seat("record", json.RawMessage(`{"key":"d-1"}`), nil)
@@ -4,8 +4,10 @@ import (
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/launch"
@@ -18,8 +20,10 @@ func counterBus(t *testing.T) launch.Bus {
t.Helper()
mesh := mt.New(t)
mesh.Bucket(t, "issues_numbers")
mesh.BucketKeyTTL(t, "issues_wakes")
m := mt.MembershipOf("issues", "anchor", false, nil)
m.State = []bus.StateIssued{{Name: "numbers", Bucket: "issues_numbers", Writes: true}}
m.State = []bus.StateIssued{{Name: "numbers", Bucket: "issues_numbers", Writes: true},
{Name: "wakes", Bucket: "issues_wakes", Writes: true}}
mesh.Issue(t, m)
conn := connect(t, "node-tools", "anchor")
conn.Follow("issues")
@@ -131,6 +135,9 @@ func TestTwoWritersIncrementingOneCounterOnTheBusLoseNoUpdate(t *testing.T) {
// A write at a revision that is no longer the key's is refused with the lost-write code and writes
// nothing; create is refused for a key with a value and taken for one deleted; an update names the
// revision it read, never zero.
// A key's own time to live (issue 1952, mesh-sdk process.State.CreateTTL and RenewTTL): created once, renewed whole by a
// put, its end heard by a watch as an expiry; refused, writing nothing, on a bucket that takes none, under a second, and
// on an update.
func TestAStaleWriteIsRefusedAndWritesNothing(t *testing.T) {
b := counterBus(t)
state := func(verb string, extra map[string]any) (uint64, error) {
@@ -200,4 +207,63 @@ func TestAStaleWriteIsRefusedAndWritesNothing(t *testing.T) {
if v := value(); v != "1" {
t.Fatalf("after delete and create: %s", v)
}
// A time to live per key: refused where it cannot be kept, writing nothing.
for _, c := range []struct {
verb, state, why string
ttl int
}{
{"create", "numbers", "does not take keys with their own time to live", 60},
{"put", "numbers", "does not take keys with their own time to live", 60},
{"create", "wakes", "shorter than the bus keeps", -1},
{"update", "wakes", "gives no time to live", 60},
} {
_, err := ask(b, c.verb, map[string]any{"state": c.state, "key": "late", "value": 1, "revision": 1, "ttl_seconds": c.ttl})
if err == nil || !strings.Contains(err.Error(), c.why) {
t.Errorf("%s with ttl %d on %s: want refused for %q, got %v", c.verb, c.ttl, c.state, c.why, err)
}
if raw, _ := ask(b, "get", map[string]any{"state": c.state, "key": "late"}); string(raw) != "null" {
t.Errorf("%s with ttl %d on %s wrote %s", c.verb, c.ttl, c.state, raw)
}
}
// The bucket's fill against its caps, for a bundle's tools.
if raw, err := ask(b, "info", map[string]any{"state": "wakes"}); err != nil || !strings.Contains(string(raw),
`"bucket":"issues_wakes","bytes":0,"keys":0,"max_bytes":-1,"max_value_bytes":262144,"history":1,"ttl_seconds":0,"key_ttl":true`) {
t.Errorf("info of a key-ttl bucket answered %s, %v", raw, err)
}
// Created once, renewed whole, and its end heard: a wake with no timer.
heard := make(chan bus.StateChange, 8)
stop, err := b.Watch(json.RawMessage(`{"state":"wakes"}`), func(raw json.RawMessage) error {
var c bus.StateChange
_ = json.Unmarshal(raw, &c)
heard <- c
return nil
})
if err != nil {
t.Fatal(err)
}
defer stop()
if _, err := ask(b, "create", map[string]any{"state": "wakes", "key": "settle", "value": 1, "ttl_seconds": 60}); err != nil {
t.Fatalf("create with a time to live: %v", err)
}
if _, err := ask(b, "create", map[string]any{"state": "wakes", "key": "settle", "value": 2, "ttl_seconds": 60}); !isLost(err) {
t.Fatalf("a second create with a time to live: %v", err)
}
if _, err := ask(b, "put", map[string]any{"state": "wakes", "key": "settle", "value": 3, "ttl_seconds": 1}); err != nil {
t.Fatalf("a put renewing the time to live: %v", err)
}
var ops []string
for want := "put delete put expire"; strings.Join(ops, " ") != want; {
select {
case c := <-heard:
ops = append(ops, c.Op)
if c.Op == "put" && len(ops) == 3 && string(c.Value) != "3" {
t.Fatalf("the renewed key holds %s", c.Value)
}
case <-time.After(5 * time.Second):
t.Fatalf("the watch heard %q, want %q", strings.Join(ops, " "), want)
}
}
}
+12 -3
View File
@@ -1,7 +1,8 @@
{
"module": "node-tools",
"version": "1",
"version": "2",
"slug": "node-tools",
"tool-runner": true,
"invokes": [
"*"
],
@@ -9,13 +10,16 @@
"mesh-knowledge.reviewed",
"mesh-delivery-gate.transition",
"gitea.forge.status",
"mesh-controller.checked",
"mesh-self-check.checked",
"operator-channel.decided.claude-code"
],
"provides": [
{
"name": "mcp-endpoint",
"scope": "node",
"credential": {
"form": "none"
},
"answered-by": "helper"
}
],
@@ -65,6 +69,11 @@
"bound"
],
"reads": [
"mesh-controller.assignments"
"mesh-provisioner.provisions",
"mesh-provisioner.plans",
"mesh-delivery-assigner.seat-holders",
"mesh-delivery-assigner.placement",
"mesh-delivery-indexer.modules",
"mesh-delivery-indexer.manifests"
]
}
-4
View File
@@ -11,10 +11,6 @@
"@novox/mesh-sdk": "^0.1.6",
"nats": "^2.29.0"
},
"bin": {
"mesh": "dist/mesh.js",
"mesh-tools": "dist/main.js"
},
"devDependencies": {
"@types/node": "^22.20.1",
"esbuild": "^0.25.0",
+1 -10
View File
@@ -1,17 +1,8 @@
{
"name": "@novox/mesh-tools",
"version": "0.1.0",
"description": "The Novox Mesh tool runtime \u2014 binds the mesh broker and serves the assigned modules' tools.",
"description": "The Novox Mesh tool runtime — binds the mesh broker and serves the assigned modules' tools.",
"type": "module",
"bin": {
"mesh-tools": "./dist/main.js",
"mesh": "./dist/mesh.js"
},
"scripts": {
"build": "tsc",
"pretest": "tsc",
"test": "node --test --test-concurrency=1 --experimental-strip-types 'test/*.test.ts'"
},
"dependencies": {
"@novox/mesh-sdk": "^0.1.6",
"nats": "^2.29.0"
-139
View File
@@ -1,139 +0,0 @@
// What a runtime serves, it announces (novox/hq ADR 0197): the NATS services protocol's discovery —
// `$SRV.PING`, `$SRV.INFO`, `$SRV.STATS`, and the same followed by the service's name and its id —
// answered in the io.nats.micro.v1 format with what is served at the moment of the request. Serving is
// unchanged; this only says what is served. One service per runtime process, because the bus admits
// one reply per request from each responder: one endpoint per tool per subject, its metadata saying
// which module, seat, scope and machine it is. The same shape the Go runtime answers.
import { StringCodec } from "nats";
import { asSchema } from "@novox/mesh-sdk/stdio";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import type { RuntimeBroker } from "./broker-nats.js";
const sc = StringCodec();
export const VERSION = "0.1.0";
export const INFO_RESPONSE = "io.nats.micro.v1.info_response";
export const PING_RESPONSE = "io.nats.micro.v1.ping_response";
export const STATS_RESPONSE = "io.nats.micro.v1.stats_response";
/** One tool served on one subject, as announced. */
export interface Endpoint {
kind: "tool" | "seat";
module: string;
tool: string;
seat?: string;
scope?: "mesh" | "node";
node: string;
description: string;
schema: unknown;
interchangeable: boolean;
subject: string;
queue?: string;
}
/** A seat's verb as the runtime serves it, with the definition that answers it. */
export interface ServedSeatVerb {
seat: string;
verb: string;
subject: string;
holder: string;
tool: ToolDefinition;
}
export interface Service {
name: string;
id: string;
description: string;
metadata: Record<string, string>;
}
/** The endpoint's name as the protocol allows it; the metadata, not the name, identifies it. */
function nameOf(e: Endpoint): string {
const clean = (s: string) => s.replace(/[^A-Za-z0-9_-]/g, "_");
return `${clean(e.kind === "seat" ? e.seat ?? "" : e.module)}__${clean(e.tool)}`;
}
/** The info_response for these endpoints. */
export function info(s: Service, endpoints: Endpoint[]): Record<string, unknown> {
return {
name: s.name, id: s.id, version: VERSION, metadata: s.metadata, type: INFO_RESPONSE, description: s.description,
endpoints: endpoints.map((e) => {
const metadata: Record<string, string> = {
kind: e.kind, module: e.module, tool: e.tool, node: e.node, description: e.description,
schema: JSON.stringify(e.schema ?? {}), interchangeable: e.interchangeable ? "true" : "false",
};
if (e.kind === "seat") {
metadata.seat = e.seat ?? "";
metadata.scope = e.scope ?? "mesh";
}
return { name: nameOf(e), subject: e.subject, queue_group: e.queue ?? "", metadata };
}),
};
}
/** Everything served now: each module's tools on every subject issued for them, and each held
* seat's verbs on the seat's subject. */
export function endpointsOf(
broker: RuntimeBroker,
own: { module: string; tools: ToolDefinition[] }[],
seats: ServedSeatVerb[],
): Endpoint[] {
const node = broker.node ?? "";
const out: Endpoint[] = [];
for (const { module, tools } of own) {
for (const t of tools) {
const served = broker.servedOn ? broker.servedOn(module, t.name) : [];
const interchangeable = served.some((s) => s.subject === `mesh.mod.${module}.tool.${t.name}`);
for (const s of served) {
out.push({ kind: "tool", module, tool: t.name, node, description: t.description, schema: asSchema(t.input),
interchangeable, subject: s.subject, queue: s.queue });
}
}
}
for (const v of seats) {
out.push({ kind: "seat", module: v.holder, tool: v.verb, seat: v.seat,
scope: node && v.subject.endsWith(`.${node}`) ? "node" : "mesh", node, description: v.tool.description,
schema: asSchema(v.tool.input), interchangeable: false, subject: v.subject });
}
return out;
}
/** Answer discovery for one service until stopped. */
export function announce(broker: RuntimeBroker, s: Service, current: () => Endpoint[]): () => void {
const started = new Date().toISOString();
const identity = { name: s.name, id: s.id, version: VERSION, metadata: s.metadata };
const answer = (subject: string): Uint8Array | undefined => {
const parts = subject.split(".");
if (parts[0] !== "$SRV" || parts.length < 2) return undefined;
if (parts.length >= 3 && parts[2] !== s.name) return undefined; // another service's
if (parts.length >= 4 && parts[3] !== s.id) return undefined; // another instance's
let v: unknown;
switch (parts[1]) {
case "PING":
v = { ...identity, type: PING_RESPONSE };
break;
case "INFO":
v = info(s, current());
break;
case "STATS":
v = { ...identity, type: STATS_RESPONSE, started, endpoints: current().map((e) => ({
name: nameOf(e), subject: e.subject, queue_group: e.queue ?? "", num_requests: 0, num_errors: 0,
last_error: "", processing_time: 0, average_processing_time: 0 })) };
break;
default:
return undefined;
}
return sc.encode(JSON.stringify(v));
};
const stops: (() => void)[] = [];
// Exactly the questions asked of every service and of this one by name and instance — what the
// grants allow (novox/hq ADR 0197). A wildcard is refused by the bus, and a refused subscription
// ends a runtime: on 2026-10-03 it crash-looped every container that announced itself.
for (const verb of ["PING", "INFO", "STATS"]) {
for (const subject of [`$SRV.${verb}`, `$SRV.${verb}.${s.name}`, `$SRV.${verb}.${s.name}.${s.id}`]) {
stops.push(broker.raw!(subject, (subj) => answer(subj)));
}
}
return () => stops.forEach((stop) => stop());
}
-680
View File
@@ -1,680 +0,0 @@
// The tool runtime's broker client, on NATS.
//
// **The sdk's contract does not change** (novox/hq ADR 0106, ADR 0039): a module is written
// against `request`, `handle`, `publish`, `subscribe`, `close`, and the runtime implements them.
// That is why a module built before any of this runs on the new runtime without a rebuild, and
// why the sdk's own diff for the whole bus change is three comments.
//
// Underneath, everything is a subject and durability is JetStream (novox/hq design 25,
// design 29).
//
// mesh.mod.<module>.event.<type> an event this module emits
// mesh.mod.<module>.tool.<tool> a tool this module serves
// mesh.seat.<seat>.accept.<verb> work submitted to a role
//
// The module never writes one of those: it names its events and tools locally and the mesh
// derives the subject (design 29 §1), so reorganising the subject space leaves every module
// correct.
import { AsyncLocalStorage } from "node:async_hooks";
import { createHash } from "node:crypto";
import net from "node:net";
import tls from "node:tls";
import { connect as natsConnect, headers as natsHeaders, StringCodec, type JsMsg, type Subscription, type TlsOptions } from "nats";
import type { Broker, Envelope, EventHeaders } from "@novox/mesh-sdk/messaging";
const sc = StringCodec();
/** Requests wait this long for an answer before failing. Unchanged from what modules already
* expect, so a module's timeout handling is not something the bus quietly redefines. */
const REQUEST_TIMEOUT_MS = 30_000;
export class PinMismatchError extends Error {}
/** A broker credential as the mesh delivers it (novox/hq ADR 0120): the bus's address, the
* fingerprint of the certificate it must present, and the node and module the account is scoped
* to — the runtime derives its subjects from those rather than being told them. */
export interface Credential {
url: string;
fingerprint?: string;
node?: string;
module?: string;
user?: string;
password?: string;
/** The seats this module claims, with the verbs each promises (novox/hq ADR 0159). The runtime
* serves each claimed seat's verbs with its tools of the same name; the bus admits only the
* holder's subscription, so claiming and not holding costs a refused subscription and nothing else. */
claims?: { seat: string; scope?: string; serves?: string[] }[];
}
/**
* What the mesh issued this assignment (novox/hq ADR 0160): where its tools are served, in which
* queue, the verbs of the seats it holds, where its events land, what it may reach. Read from the
* ASSIGNMENTS stream at `mesh.assignment.<node>.<module>` — the one subject a runtime derives for
* itself — and followed live. Absent for a mesh older than the membership, and then the runtime
* serves the shape it always derived, and says so.
*/
export interface Membership {
node: string;
module: string;
/** Addresses a tool is answered on; `{tool}` stands for the tool's name. */
serves: { subject: string; queue?: string }[];
seats?: { seat: string; verb: string; subject: string }[];
emits: string;
reaches?: Record<string, string[]>;
tools: string;
}
/** What a tool call answers: the module's own result, and which machine answered it
* (novox/hq ADR 0159) — a module on several machines is otherwise an answer from nowhere. */
export interface Answered<Res> {
result: Res;
node?: string;
}
/** The bus as the runtime sees it: the sdk's contract, and the few things only the runtime needs —
* an answer that says which machine gave it, serving a subject that is not a module's own tool
* (a seat's verb), and following the memberships of the modules it serves beside its own. */
export interface RuntimeBroker extends Broker {
/** Call a tool by key, or — when `on` names a subject the mesh listed for it (ADR 0160) — there. */
ask<Req, Res>(key: string, body: Req, on?: string): Promise<Answered<Res>>;
handleSubject<Req, Res>(subject: string, handler: (body: Req) => Promise<Res>): Promise<() => void>;
/**
* Serve another module's tools from this connection: read its membership on this machine and
* follow it, so `handle("<module>.<tool>")` is served where the mesh issued that module (novox/hq
* ADR 0175: one runtime per node, every assigned module's tools). The account must be allowed
* to read that membership and to subscribe its subjects — the node's is; a module's own is not,
* and is refused by the bus, not here.
*/
follow(module: string): Promise<void>;
/** What the mesh issued this assignment — this module's when none is named — or undefined when
* nothing has been issued yet. */
membership(module?: string): Membership | undefined;
/** Called when the mesh issues a new membership to any module this connection follows; the
* runtime re-serves on it. The membership says which module it is for. */
onMembership(handler: (m: Membership) => void): void;
/** The modules whose memberships this connection follows: its own and every one `follow` added. */
serving(): string[];
/** The module this connection is: what its credential named, and what a bare key serves as. */
readonly module: string;
/** Where a served module's tool is answered right now (ADR 0197: what it serves, it announces). */
servedOn?(module: string, tool: string): { subject: string; queue?: string }[];
/** Answer a subject in a format of its own, not a tool's reply envelope — the NATS services
* protocol's discovery (novox/hq ADR 0197). An undefined answer is no reply. */
raw?(subject: string, answer: (subject: string, data: Uint8Array) => Uint8Array | undefined): () => void;
/** The machine this connection serves on, when its credential names one. */
readonly node?: string;
}
/**
* Which module's tool is at work on this connection, when one is (novox/hq ADR 0175). One runtime
* carries many modules' tools, and an event a tool emits must land on the emitting *module's*
* subject, not the runtime's — so the runtime runs each tool inside this store, and `publish`
* reads the module from it. Outside a tool the connection's own module stands.
*/
export const atWork = new AsyncLocalStorage<{ module: string }>();
const ASSIGNMENTS_STREAM = "ASSIGNMENTS";
/** The one address a runtime derives for itself (ADR 0160). */
export function membershipSubject(node: string, module: string): string {
return `mesh.assignment.${node}.${module}`;
}
/** Whether a connection failure is worth retrying, or is a fact about this configuration that
* retrying cannot change. The runtime's supervisor asks this and does not need to know what it
* is connected to. */
export function fatalBrokerReason(err: unknown): string | null {
if (err instanceof PinMismatchError) return "the bus's certificate does not match the pin";
const e = err as { code?: string; message?: string };
const message = typeof e?.message === "string" ? e.message : String(err);
if (e?.code === "ERR_INVALID_URL" || /invalid url/i.test(message)) {
return "the bus address is not a usable URL";
}
if (/authorization violation|user authentication expired|permissions violation/i.test(message)) {
return "the bus refused this account";
}
return null;
}
/**
* Connect to the mesh bus and return a Broker.
*
* **A module's subjects come from its credential, not from its calls.** `node` and `module` name
* the account the mesh issued, and every subject this client publishes or subscribes is derived
* from them — so a module cannot name another's namespace even by mistake, and what it emits
* matches what the mesh authorised (ADR 0074's identity rule).
*/
export async function connectNats(
target: string | Credential,
opts: { module?: string } = {},
): Promise<RuntimeBroker> {
const cred: Credential = typeof target === "string" ? { url: target } : target;
const self = cred.module ?? opts.module;
if (!self) {
throw new Error(
"a broker credential with no module: the runtime derives its subjects from the account " +
"the mesh issued, and cannot guess which module it is",
);
}
const conn = await natsConnect({
servers: cred.url,
user: cred.user,
pass: cred.password,
name: `${cred.node ?? "?"}.${self}`,
tls: cred.fingerprint ? await pinnedTls(cred.url, cred.fingerprint) : undefined,
// Its own inbox, not a random one: every user's inbox is private to it (design 25 §4), and the
// grant names `_INBOX.<user>.>` — a reply space the client invented would be refused, and with
// it every pull for the next message and every answer to a tool call.
inboxPrefix: cred.user ? `_INBOX.${cred.user}` : undefined,
// Reconnect forever: the bus being restarted is an upgrade, not a reason for every module on
// the mesh to exit. `close()` stays the only thing that ends the connection.
maxReconnectAttempts: -1,
});
const js = conn.jetstream();
const subs: Subscription[] = [];
// Every registration, and the one reader that dispatches to them. A module has one durable
// consumer; the loop belongs to the connection rather than to a subscription.
const listeners: { pattern: string; handler: (env: Envelope<unknown>) => Promise<void> }[] = [];
let reading: Awaited<ReturnType<Awaited<ReturnType<typeof js.consumers.get>>["consume"]>> | undefined;
let closed = false;
const node = cred.node;
// The memberships, one per module this connection serves, each read once and followed. A
// module's own runtime follows one, its own; the node's runtime follows one per assigned module
// (novox/hq ADR 0175) — the same subject shape, the same stream, read as many times as there are
// modules, and nothing on the bus learns a new shape for it. A direct get is one request on the
// stream's API, which is the whole of what an account may ask JetStream for a subject it is
// granted; a 404 is a mesh that has not issued one, which is a fact to say and not an error to
// retry.
const issued = new Map<string, Membership | undefined>();
const issuedHandlers: ((m: Membership) => void)[] = [];
const follow = async (module: string): Promise<void> => {
if (!node || issued.has(module)) return;
issued.set(module, undefined);
const subjectOfTheirs = membershipSubject(node, module);
try {
// The subject-addressed form of a direct get — the stream, then the subject, nothing in
// the body — because that is the one address the mesh grants this account on the
// stream's API; the body form asks the stream's root, which it may not.
const got = await conn.request(`$JS.API.DIRECT.GET.${ASSIGNMENTS_STREAM}.${subjectOfTheirs}`,
new Uint8Array(0), { timeout: 5_000 });
const status = got.headers?.code ?? 0;
if (status === 0 && got.data.length > 0) {
issued.set(module, JSON.parse(sc.decode(got.data)) as Membership);
}
} catch {
// Not readable here: an older mesh, a stream not yet asserted, or no grant. Said below.
}
if (!issued.get(module)) {
console.log(`[mesh-tools] no membership issued for ${module} on ${node} yet; serving the derived shape until one arrives`);
}
try {
const live = conn.subscribe(subjectOfTheirs);
subs.push(live);
void (async () => {
for await (const msg of live) {
try {
const m = JSON.parse(sc.decode(msg.data)) as Membership;
issued.set(module, m);
console.log(`[mesh-tools] ${module} on ${node} was issued a new membership; re-serving on it`);
for (const h of issuedHandlers) h(m);
} catch (err) {
console.log(`[mesh-tools] a membership arrived that is not one: ${err}`);
}
}
})();
} catch {
// A subscription this account may not make is a mesh older than the membership.
}
};
await follow(self);
/** The subjects a tool of a served module is served on: from that module's membership when
* issued, derived otherwise (the shape the mesh issues on day one, so the two agree). */
const servedOn = (module: string, tool: string): { subject: string; queue?: string }[] => {
const m = issued.get(module);
if (m) {
const out = m.serves.map((s) => ({ subject: s.subject.replace("{tool}", tool), queue: s.queue }));
// The verb that lists what this module serves is answered on the mesh's plain address for it
// whatever the placement — one answer suffices, so a queue — and on this machine's beside it.
if (tool === "tools" && m.tools && !out.some((s) => s.subject === m.tools)) {
out.unshift({ subject: m.tools, queue: `serve.${module}` });
}
return out;
}
const base = `mesh.mod.${module}.tool.${tool}`;
const out: { subject: string; queue?: string }[] = [{ subject: base, queue: `serve.${module}` }];
if (node) out.push({ subject: `${base}.${node}` });
return out;
};
/** Where a call by key goes: a subject this connection's own membership says it reaches, when it
* says one — the machine's when named — else the derived shape. */
const reachedAt = (key: string): string => {
const [name, wanted] = key.split("@", 2);
const reach = issued.get(self)?.reaches?.[name];
if (reach && reach.length > 0) {
if (wanted) {
const at = reach.find((s) => s.endsWith(`.${wanted}`));
if (at) return at;
} else {
return reach[0];
}
}
return toolSubject(key, self);
};
/** Answer one subject with one handler, and say which machine answered (novox/hq ADR 0159). */
const answerOn = <Req, Res>(
subject: string,
queue: string | undefined,
handler: (body: Req) => Promise<Res>,
): (() => void) => {
const sub = conn.subscribe(subject, queue ? { queue } : {});
subs.push(sub);
void (async () => {
// **A refused subscription is said, never fatal** (novox/hq 04-ISSUES/218, as 217 for the
// announcements). The grants are the mesh's word on what this account may answer; a subject
// they leave out — a seat claimed here and held elsewhere — costs that subject, never the
// module's other tools, its handlers and its provisioning. Unhandled, the refusal ended the
// process and a module's runtime crash-looped on 2026-10-04.
try {
for await (const msg of sub) {
let reply: { result?: Res; error?: string; node?: string };
try {
reply = { result: await handler(JSON.parse(sc.decode(msg.data)) as Req) };
} catch (err) {
// The caller is told, rather than left to time out: a handler that threw is a
// different failure from a tool nobody serves, and only one of them is worth retrying.
reply = { error: err instanceof Error ? err.message : String(err) };
}
if (node) reply.node = node;
msg.respond(sc.encode(JSON.stringify(reply)));
}
} catch (err) {
console.log(`[mesh-tools] the bus refused ${subject}: ${err instanceof Error ? err.message : String(err)}; ` +
"not served here, and the rest serves on");
}
})();
return () => sub.unsubscribe();
};
/**
* Ask one question and await one answer, with the machine that gave it.
*
* Core NATS request/reply, not JetStream: a tool call must never be persisted (design 25 §3),
* and a lost one is a timeout the caller already handles. The reply travels on the inbox the
* request carries, which the responder may answer because its account has `allow_responses`
* — one reply to a message it actually received, and nothing wider.
*/
const ask = async <Req, Res>(key: string, body: Req, on?: string): Promise<Answered<Res>> => {
const msg = await conn.request(on ?? reachedAt(key), sc.encode(JSON.stringify(body)), {
timeout: REQUEST_TIMEOUT_MS,
});
const reply = JSON.parse(sc.decode(msg.data)) as { result?: Res; error?: string; node?: string };
if (reply.error) throw new Error(reply.error);
return { result: reply.result as Res, node: reply.node };
};
return {
async request<Req, Res>(key: string, body: Req): Promise<Res> {
return (await ask<Req, Res>(key, body)).result;
},
ask,
/**
* Answer a question, two ways (novox/hq ADR 0159): on the module's subject in a queue group,
* so several machines may serve one tool and exactly one of them answers each call; and on the
* same subject with this machine as its last token, so a caller that names the machine reaches
* this instance and no other. A runtime that does not know its machine serves only the first,
* which is how it always behaved.
*/
async handle<Req, Res>(key: string, handler: (body: Req) => Promise<Res>): Promise<() => void> {
// A seat's verb named outright is served on the seat's subject as given, for a holder that
// knows its role without a membership; everything else is a served module's own tool, served
// where the mesh issued that module (ADR 0160). A key naming a module this connection does
// not follow is not served here at all: a module serves its own tools, and the node's
// runtime those of the modules it was given (ADR 0175) — never a stranger's.
if (key.startsWith("seat:")) {
const stop = answerOn(toolSubject(key, self), undefined, handler);
return () => stop();
}
const dot = key.indexOf(".");
const module = dot < 0 ? self : key.slice(0, dot);
if (!issued.has(module) && module !== self) {
throw new Error(
`${self} cannot serve ${key}: a module serves its own tools, and a runtime those of the ` +
"modules it follows",
);
}
const tool = dot < 0 ? key : key.slice(dot + 1);
let stops = servedOn(module, tool).map((s) => answerOn(s.subject, s.queue, handler));
// When that module's new membership arrives, serve where it now says and stop serving where
// it no longer does.
issuedHandlers.push((m) => {
if (m.module !== module) return;
stops.forEach((stop) => stop());
stops = servedOn(module, tool).map((s) => answerOn(s.subject, s.queue, handler));
});
return () => stops.forEach((stop) => stop());
},
module: self,
node,
servedOn,
raw(subject: string, answer: (subject: string, data: Uint8Array) => Uint8Array | undefined): () => void {
const sub = conn.subscribe(subject);
subs.push(sub);
void (async () => {
// **A refusal here is said, never fatal** (novox/hq 04-ISSUES/217). This serves discovery —
// what the runtime says about itself — not the work; a bus that refuses it costs the mesh
// seeing this runtime, not the runtime's tools and handlers. Unhandled, the refusal ended the
// process and every per-module container crash-looped on 2026-10-03.
try {
for await (const msg of sub) {
const body = answer(msg.subject, msg.data);
if (body) msg.respond(body);
}
} catch (err) {
console.log(`[mesh-tools] the bus refused ${subject}: ${err instanceof Error ? err.message : String(err)}; ` +
"discovery will not see this runtime there, and it serves on");
}
})();
return () => sub.unsubscribe();
},
follow,
serving: () => [...issued.keys()],
membership: (module?: string) => issued.get(module ?? self),
onMembership: (handler: (m: Membership) => void) => {
issuedHandlers.push(handler);
},
async handleSubject<Req, Res>(subject: string, handler: (body: Req) => Promise<Res>): Promise<() => void> {
return answerOn(subject, undefined, handler);
},
/**
* Emit an event.
*
* Published into JetStream and awaited, so a publish the bus never accepted fails the emit
* rather than vanishing — at-least-once starts at the emitter, not only the consumer
* (ADR 0042).
*
* `msgID` is the event's own id, so a redelivery after a crash between publishing and
* acknowledging is de-duplicated by the server inside its window rather than seen twice.
*/
async publish<T>(env: Envelope<T>): Promise<void> {
// **The body is the payload and the metadata rides as headers** (ADR 0042). That shape
// is what the conformance suite pins: an implementation that nested the whole envelope in
// the body would pass every one of its own tests and agree with nobody.
const meta = (env.headers ?? {}) as Record<string, string>;
const h = natsHeaders();
for (const [k, v] of Object.entries(meta)) {
if (v != null) h.set(k, String(v));
}
if (!meta["content-type"]) h.set("content-type", "application/json");
if (env.node) h.set("x-node", env.node);
// The emitting module's subject: the tool at work's when a served module's tool emits from
// the node's runtime (ADR 0175), this connection's own otherwise.
await js.publish(eventSubject(env.key, atWork.getStore()?.module ?? self), sc.encode(JSON.stringify(env.body)), {
headers: h,
// De-duplicated by the server inside its window, so a redelivery after a crash between
// publishing and acknowledging is not seen twice. Only the emitter can make this id.
msgID: meta["x-event-id"],
});
},
/**
* React to events.
*
* The durable consumer is the **controller's** to create, from what this module declared it
* consumes (design 29 §3) — this binds to it and never creates one. A runtime that created
* its own would be a module deciding its own delivery semantics, and its account cannot
* reach the JetStream API to do it anyway.
*
* **One consumer, one loop, however many patterns a module registers.** A module has exactly one
* durable consumer, so two loops reading it would each take half the messages — and a loop that
* received one its own pattern does not match acknowledges it, which is the right answer for a
* filter wider than anything registered and silent loss when it is another handler's. Every
* registration is therefore dispatched from one reader, and a message is acknowledged once every
* handler it is for has taken it.
*/
async subscribe<T>(
pattern: string,
handler: (env: Envelope<T>) => Promise<void>,
): Promise<() => void> {
const listener = { pattern, handler: handler as (env: Envelope<unknown>) => Promise<void> };
listeners.push(listener);
if (!reading) {
const durable = `${cred.node ?? "?"}_${self}`;
const consumer = await js.consumers.get("EVENTS", durable);
const messages = await consumer.consume();
reading = messages;
void (async () => {
for await (const msg of messages) {
await deliver(msg, listeners);
}
})();
}
return () => {
const at = listeners.indexOf(listener);
if (at >= 0) listeners.splice(at, 1);
if (listeners.length === 0 && reading) {
void reading.close();
reading = undefined;
}
};
},
async close(): Promise<void> {
if (closed) return;
closed = true;
for (const sub of subs) sub.unsubscribe();
// Drain rather than close: an in-flight reply is finished instead of dropped, which for a
// tool call is the difference between an answer and an unexplained timeout at the caller.
await conn.drain();
},
};
}
/**
* Deliver one event to every handler it is for, acknowledging only once each has taken it.
*
* Several registrations share one durable consumer, so matching happens here rather than by having
* each registration read the stream: two readers of one consumer would split it between them, and a
* message that reached the wrong one would be acknowledged as not-for-me and lost.
*/
async function deliver(
msg: JsMsg,
listeners: { pattern: string; handler: (env: Envelope<unknown>) => Promise<void> }[],
): Promise<void> {
let env: Envelope<unknown>;
try {
env = toEnvelope<unknown>(msg);
} catch {
// Unparseable: terminate it. Redelivering a message no version of this code can read is
// an infinite loop, and what the controller keeps of a message given up on (DEAD_LETTERS,
// novox/hq issue 330) is for handlers that fail, not for bytes that were never an envelope.
msg.term();
return;
}
const forThis = listeners.filter((l) => topicMatches(l.pattern, env.key));
if (forThis.length === 0) {
// The consumer's filters are the controller's, derived from what the module declared it
// consumes, and may be wider than anything it registered a handler for. Acknowledge it, or it
// would be redelivered until it expired.
msg.ack();
return;
}
try {
for (const l of forThis) await l.handler(env);
msg.ack();
} catch {
// Negative-acknowledge with a delay, so a handler failing on a transient cause gets another
// attempt, and one failing permanently exhausts max-deliver rather than spinning: the consumer
// gives the event up, and the controller keeps it in DEAD_LETTERS until a person delivers it
// again or drops it (novox/hq issue 330). Delivered again, it arrives under
// `mesh.again.<consumer>.…`, and keyFromSubject reads the same key from it. The consumer's
// limits are the controller's; this only says "not done".
msg.nak(5_000);
}
}
/** Rebuild the envelope a module sees, from the subject, the headers and the payload — the
* mirror of publish, and the reason both live beside each other. */
function toEnvelope<T>(msg: JsMsg): Envelope<T> {
const headers: Record<string, string> = {};
if (msg.headers) {
for (const k of msg.headers.keys()) headers[k] = msg.headers.get(k);
}
return {
// The event's own key, recovered from the subject: `mesh.mod.<module>.event.<key>`. The
// module never sees the subject, only the key it declared.
key: keyFromSubject(msg.subject),
node: headers["x-node"] ?? "",
body: JSON.parse(sc.decode(msg.data)) as T,
headers: headers as EventHeaders,
};
}
/** The event key a module sees: the emitter and the event, which is exactly how its manifest names
* what it consumes (novox/hq design 29 §1, 04-ISSUES/127).
*
* **One vocabulary for the declaration and the handler.** This returned the event name alone, so a
* manifest declaring `consumes: builder.built` produced a handler pattern that could never match
* the key it was compared against — and a module consuming the same event from two emitters could
* not tell them apart except by reading a header. The subject already carries the emitter; naming it
* here makes a mismatch between manifest and code a typo rather than a category error. */
function keyFromSubject(subject: string): string {
const marker = ".event.";
const at = subject.indexOf(marker);
if (at < 0) return subject;
const event = subject.slice(at + marker.length);
// `mesh.mod.<emitter>.event.…` — the emitter is the token before the marker.
const before = subject.slice(0, at).split(".");
const emitter = before[before.length - 1];
return emitter ? `${emitter}.${event}` : event;
}
/** A module's own event subject. Derived, never taken from the caller: the module names its
* event and the mesh decides where it lands (design 29 §1). */
function eventSubject(type: string, self: string): string {
return `mesh.mod.${self}.event.${type}`;
}
/** A tool's subject. A bare name is this module's own tool; `<module>.<tool>` addresses
* another's, which is how a request reaches a module that is not this one; `seat:<seat>.<verb>`
* addresses a role's tool, answered by whoever holds the seat (novox/hq ADR 0132) — with
* `seat:<seat>.<verb>@<node>` for a node-scoped seat, whose tool carries the machine (design 33 §4). */
function toolSubject(key: string, self: string): string {
if (key.startsWith("seat:")) {
const rest = key.slice("seat:".length);
const dot = rest.indexOf(".");
if (dot < 0) throw new Error(`"${key}" names a seat and no verb: seat:<seat>.<verb>`);
const seat = rest.slice(0, dot);
const [verb, node] = rest.slice(dot + 1).split("@", 2);
return node ? `mesh.seat.${seat}.tool.${verb}.${node}` : `mesh.seat.${seat}.tool.${verb}`;
}
// `<module>.<tool>@<node>` names the machine (novox/hq ADR 0159): the same subject with the
// machine as its last token, which is what that instance serves beside the queue.
const [name, node] = key.split("@", 2);
const dot = name.indexOf(".");
const base = dot < 0
? `mesh.mod.${self}.tool.${name}`
: `mesh.mod.${name.slice(0, dot)}.tool.${name.slice(dot + 1)}`;
return node ? `${base}.${node}` : base;
}
/** A seat's verb, as its holder serves it: flat for a mesh seat, carrying the machine for a
* node-scoped one (design 33 §4) — the same shape the controller grants. */
export function seatToolSubject(seat: string, verb: string, scope: string | undefined, node: string | undefined): string {
const base = `mesh.seat.${seat}.tool.${verb}`;
return scope === "node" && node ? `${base}.${node}` : base;
}
function normalizeFingerprint(fingerprint: string): string {
return fingerprint.replace(/^sha256:/i, "").replace(/:/g, "").toLowerCase();
}
/**
* Dial once to see the certificate, and refuse unless it is exactly the one the mesh pinned.
* A certificate authority is not consulted: the mesh issued this and knows its fingerprint,
* which is stronger than trusting whoever a machine's trust store happens to contain.
*
* **The pin is the only check.** What comes back is handed to the client as its TLS options, and
* the client's transport spreads them into Node's own `tls.connect` — so the pinned certificate
* is the one authority the handshake accepts, and the hostname check beside it is replaced with
* one that always passes. Pinning the exact certificate makes verifying its name redundant, and
* the bus's certificate names the seat (`mesh-broker`), not the address a machine happens to
* dial it by: every module on the mesh met "does not match certificate's altnames" the first time
* it reached the handshake (2026-09-28).
*/
async function pinnedTls(rawUrl: string, fingerprint: string): Promise<TlsOptions> {
const url = new URL(rawUrl.includes("://") ? rawUrl : `nats://${rawUrl}`);
const port = url.port ? Number(url.port) : 4222;
// **The bus speaks first, in the clear.** A NATS server sends its INFO line before TLS begins,
// and only then expects the client to start the handshake; a raw TLS connect to that port reads
// the INFO line as a TLS record and fails with "wrong version number" — which is what every
// module met the first time it dialled the bus being built (2026-09-28). So: connect, wait for
// INFO, then start TLS on the same socket, and read the certificate the server presents.
const certificate = await new Promise<tls.DetailedPeerCertificate>((resolve, reject) => {
const plain = net.connect({ host: url.hostname, port }, () => {});
let seenInfo = false;
let buffered = "";
plain.on("error", reject);
plain.on("data", (chunk: Buffer) => {
if (seenInfo) return;
buffered += chunk.toString("utf8");
if (!buffered.includes("\r\n")) return;
seenInfo = true;
plain.removeAllListeners("data");
const secure = tls.connect(
{ socket: plain, rejectUnauthorized: false, servername: url.hostname },
() => {
const peer = secure.getPeerCertificate(true);
secure.end();
resolve(peer);
},
);
secure.on("error", reject);
});
});
const seen = createHash("sha256").update(certificate.raw).digest("hex");
if (seen !== normalizeFingerprint(fingerprint)) {
throw new PinMismatchError(
`the bus at ${url.hostname}:${port} presented ${seen}, not the pinned ${normalizeFingerprint(fingerprint)}`,
);
}
const pem = `-----BEGIN CERTIFICATE-----\n${certificate.raw.toString("base64").replace(/(.{64})/g, "$1\n")}\n-----END CERTIFICATE-----\n`;
// Node's option, not the client's: the transport passes the whole object on. `undefined` from
// checkServerIdentity is "the name is fine"; the pin above already decided the rest.
return { ca: pem, checkServerIdentity: () => undefined } as TlsOptions;
}
/** The mesh's topic matching: `*` is one token, `#` the rest. This is the module's vocabulary —
* a module's `consumes` pattern is matched here, and the subject it becomes is the mesh's
* business, not the module's. */
export function topicMatches(pattern: string, key: string): boolean {
return matchFrom(pattern.split("."), 0, key.split("."), 0);
}
function matchFrom(p: string[], pi: number, k: string[], ki: number): boolean {
if (pi === p.length) return ki === k.length;
// `**` is the mesh's wildcard for the rest of a name; `#` is the old bus's, accepted so a pattern
// written either way behaves the same while both buses ship (novox/hq design 29 §1).
if (p[pi] === "#" || p[pi] === "**") {
for (let skip = ki; skip <= k.length; skip++) {
if (matchFrom(p, pi + 1, k, skip)) return true;
}
return false;
}
if (ki === k.length) return false;
if (p[pi] !== "*" && p[pi] !== k[ki]) return false;
return matchFrom(p, pi + 1, k, ki + 1);
}
-280
View File
@@ -1,280 +0,0 @@
/**
* The mesh's tools, for whoever is on a machine (novox/hq design 25 §7, design 34).
*
* Two surfaces over one thing. A command line, for somebody at a terminal; an MCP server, for an
* agent. Both are adapters over the same three calls — what tools are there, what does this one take,
* call it — because a second way of reaching a tool is a second thing to keep correct.
*
* **It uses the same client a module's runtime uses.** Not a second protocol and not a bridge: the
* caller connects as its own bus user — a person's, or the console's — publishes on the tool subjects
* that account permits, and the server refuses anything else. So "what may this ask" is answered by the
* same permission list that answers it for a module, and there is nothing here for an audit to read
* separately.
*
* What the caller may NOT do is the more interesting half, and none of it is enforced here — it is the
* account (design 25 §4): it cannot publish an event, so it cannot claim a module said something; it
* has no consumer, so there is no delivery to acknowledge; and it cannot answer a request, so it cannot
* impersonate a module on a bus where anyone may serve a tool.
*/
import { readFile } from "node:fs/promises";
import type { Broker } from "@novox/mesh-sdk/messaging";
import { connectNats, type Answered, type Credential } from "./broker-nats.js";
import { TOOLS_VERB, type ToolsAnswer } from "./runtime.js";
/** Where the catalogue answers which modules the mesh holds. */
const CATALOGUE_MODULES = "mesh-catalog.catalog_modules";
/** Where the mesh answers every role's tools, from its records: the mesh-controller seat's own
* `tools` verb (novox/hq ADR 0154, design 33 §5). */
const SEAT_TOOLS = "seat:mesh-controller.tools";
/** A tool as its module describes it. */
export interface Tool {
/** The module that serves it — or, for a role's tool, the seat. */
module: string;
name: string;
description?: string;
/** The JSON schema of what it takes, as the module declared it. */
input?: unknown;
/** True for a role's tool: addressed to the seat, answered by whoever holds it (ADR 0132). */
seat?: boolean;
/** A role's scope: `node` for a seat held once per machine, whose verb is asked of one machine
* (design 33 §4) and takes `node` for it; `mesh` or absent otherwise. */
scope?: string;
/** Where the tool is answered, as the mesh issued it (ADR 0160): the plain subject first when the
* module answers for itself anywhere, then one per machine. Absent for a runtime older than this. */
subjects?: string[];
}
/**
* What the mesh could say about its tools when asked (design 34 §3).
*
* **Silence is named, never dropped.** A module the catalogue holds and nothing answered for is in
* `notAnswering`, because a tool that is not offered looks exactly like a tool that does not exist,
* and those need different people to fix them.
*/
export interface Listing {
tools: Tool[];
/** Modules the catalogue holds whose runtime did not answer `tools`: not assigned, not up, or built
* before the runtime answered it. Each may still be called by name. The mesh's own records are
* listed here as `mesh-controller (seat)` when the control plane did not answer. */
notAnswering: string[];
}
/** The seats and the verbs each declares, from the last listing, so a call can tell a role's tool
* from a module's when the two share a prefix (a module and a seat may share a name). */
export type Seats = Map<string, Set<string>>;
/** The roles' tools, keyed the way `toolKey` names them. */
export function seatsIn(have: Listing): Seats {
const seats: Seats = new Map();
for (const t of have.tools) {
if (!t.seat) continue;
if (!seats.has(t.module)) seats.set(t.module, new Set());
seats.get(t.module)!.add(t.name);
}
return seats;
}
/** The key a call uses for `<prefix>.<name>`: a role's when the prefix is a seat declaring that
* verb, a module's otherwise. Both names for one capability are deliberate and bounded (ADR 0132);
* the seat wins only for a verb it actually declares, so a module's own tool is never shadowed. */
export function toolKey(name: string, seats?: Seats): string {
if (name.startsWith("seat:")) return name;
const dot = name.indexOf(".");
if (dot < 0) return name;
const prefix = name.slice(0, dot);
const verb = name.slice(dot + 1);
if (seats?.get(prefix)?.has(verb)) return `seat:${prefix}.${verb}`;
return name;
}
/**
* A person's credential, as `operator issue` prints it.
*
* The same shape a module is handed, minus the parts a module needs and a person does not: no node,
* because a person is not on a machine, and no module, because they are not one.
*/
export interface PersonCredential extends Credential {
person?: string;
invokes?: string[];
}
/** Read the credential from the file `operator issue` produced. */
export async function credentialFrom(path: string): Promise<PersonCredential> {
const raw = await readFile(path, "utf8");
let held: PersonCredential;
try {
held = JSON.parse(raw) as PersonCredential;
} catch (e) {
throw new Error(
`${path} is not a credential this mesh issued: ${(e as Error).message}. ` +
"It is the JSON `operator issue` printed, saved verbatim.",
);
}
if (!held.url || !held.user || !held.password) {
throw new Error(
`${path} names no bus, user or password. It is the JSON \`operator issue\` printed, saved ` +
"verbatim — not an edited copy of it.",
);
}
return held;
}
/** Connect as this person. The module name the runtime wants is their own user, because every subject
* it derives is for a tool somebody else serves. */
export async function connectAs(held: PersonCredential): Promise<Broker> {
return connectNats({ ...held, module: held.user });
}
/**
* Connect as the console: the module credential the mesh delivered (novox/hq ADR 0152), read from
* the same variable every runtime reads. It names the node and the module, so the account's inbox
* and subjects derive from what the mesh authorised and from nothing in this process's environment.
*/
export async function connectAsTheConsole(path: string): Promise<{ bus: Broker; who: string }> {
const raw = await readFile(path, "utf8");
let held: Credential;
try {
held = JSON.parse(raw) as Credential;
} catch (e) {
throw new Error(`${path} is not a broker credential: ${(e as Error).message}`);
}
if (!held.url || !held.module || !held.user) {
throw new Error(
`${path} names no bus, module or user: the console runs on the credential the mesh sealed to ` +
"this machine for it, and nothing else",
);
}
return { bus: await connectNats(held), who: `${held.node ?? "?"}.${held.module}` };
}
/**
* What tools the mesh has, asked of the modules (design 34 §3).
*
* The catalogue says which modules the mesh holds; each module says what it serves, through the one
* verb its runtime answers for it. **Asked, not configured**: a client carrying its own list would be a
* list that goes stale the first time a module is assigned. Every module is asked at once, and the bus
* refuses at once a request nothing serves, so the cost is bounded by the modules that are up.
*/
export async function toolsOn(bus: Broker): Promise<Listing> {
const [answered, roles] = await Promise.all([
bus.request<Record<string, never>, { modules?: { module: string }[] }>(CATALOGUE_MODULES, {}),
// The roles' tools, from the mesh's records (design 33 §5). Asked beside the modules rather
// than first: a control plane that is restarting must not hide every module's tools with it.
bus
.request<Record<string, never>, { seats?: { seat: string; scope?: string; tools?: ToolsAnswer["tools"] }[] }>(
SEAT_TOOLS,
{},
)
.catch(() => undefined),
]);
const names = (answered.modules ?? []).map((m) => m.module).filter((m) => typeof m === "string");
const asked = await Promise.allSettled(
names.map((module) => bus.request<Record<string, never>, ToolsAnswer>(`${module}.${TOOLS_VERB}`, {})),
);
const tools: Tool[] = [];
const notAnswering: string[] = [];
if (roles) {
for (const s of roles.seats ?? []) {
// A node-scoped seat's tool is asked of one machine (design 33 §4): listed with its scope, so
// a caller names the machine and the call carries it — `seat:<seat>.<verb>@<node>`. Left out
// of the listing, the verb never resolved as a seat's and nothing served it (ADR 0170).
for (const t of s.tools ?? []) {
tools.push({ module: s.seat, name: t.name, description: t.description, input: t.input, seat: true, scope: s.scope });
}
}
} else {
notAnswering.push("mesh-controller (seat)");
}
asked.forEach((outcome, i) => {
const module = names[i]!;
if (outcome.status === "fulfilled" && typeof outcome.value?.failed === "string") {
// The runtime answered for it and serves nothing: the bundle failed to load (ADR 0175). Said
// with the reason, because "not answering" would send somebody to check an assignment that
// is fine.
notAnswering.push(`${module} (its tools bundle failed to load: ${outcome.value.failed})`);
} else if (outcome.status === "fulfilled" && Array.isArray(outcome.value?.tools)) {
for (const t of outcome.value.tools) {
tools.push({ module, name: t.name, description: t.description, input: t.input, subjects: t.subjects });
}
} else {
notAnswering.push(module);
}
});
tools.sort((a, b) => `${a.module}.${a.name}`.localeCompare(`${b.module}.${b.name}`));
notAnswering.sort();
return { tools, notAnswering };
}
/** Call one tool. The key is `<module>.<tool>`, which is what a person types and what the account
* permits — one vocabulary, so a refusal names the thing they asked for. */
/** Call a tool and learn which machine answered (novox/hq ADR 0159). `<module>.<tool>@<node>` asks
* the instance on one machine; without it, whichever instance answers first does, and the answer
* says which. */
export async function callTool(
bus: Broker,
key: string,
args: unknown,
seats?: Seats,
listing?: Listing,
): Promise<Answered<unknown>> {
const at = key.indexOf("@");
const name = at < 0 ? key : key.slice(0, at);
const node = at < 0 ? "" : key.slice(at + 1);
if (!name.includes(".")) {
throw new Error(
`"${key}" does not name a tool: write <module>.<tool>, as \`mesh tools\` lists them, ` +
"or <module>.<tool>@<node> for the instance on one machine",
);
}
const resolved = toolKey(name, seats) + (node ? `@${node}` : "");
// Where the tool is answered is the module's to say and the mesh's to issue (ADR 0160): when the
// listing carried subjects for it, the call goes to one of those and composes nothing.
const on = subjectListed(name, node, listing);
const asking = bus as Broker & { ask?: <Req, Res>(k: string, b: Req, on?: string) => Promise<Answered<Res>> };
if (typeof asking.ask === "function") return asking.ask<unknown, unknown>(resolved, args ?? {}, on);
return { result: await bus.request<unknown, unknown>(resolved, args ?? {}) };
}
/** The subject the listing says answers `<module>.<tool>` — the machine's when one is named, else
* the plain one — or undefined when the listing said none, and the key is composed as before. */
export function subjectListed(name: string, node: string, listing?: Listing): string | undefined {
if (!listing) return undefined;
const dot = name.indexOf(".");
const module = name.slice(0, dot);
const tool = name.slice(dot + 1);
const found = listing.tools.find((t) => t.module === module && t.name === tool && !t.seat);
const subjects = found?.subjects ?? [];
if (subjects.length === 0) return undefined;
if (node) return subjects.find((s) => s.endsWith(`.${node}`));
return subjects[0];
}
/**
* Why a call failed, said so that the remedy is in the words.
*
* Three answers a person actually gets, and they need different things done: nobody serves that tool,
* the mesh refused this account, or the tool itself failed. Without this they are one timeout and a
* stack trace.
*/
export function whyItFailed(key: string, err: unknown): string {
const message = err instanceof Error ? err.message : String(err);
if (/no responders|503/i.test(message)) {
return `nothing serves ${key}. The module may not be assigned to any machine, or it is down` +
(key.startsWith("seat:") ? ", or nothing holds that seat" : "") +
" — `mesh tools` lists what answered.";
}
if (/permissions violation|authorization/i.test(message)) {
return `this account may not call ${key}. What it may call was fixed when it was issued — a ` +
"person's by `operator issue`, the console's by its manifest.";
}
if (/timeout/i.test(message)) {
return `${key} did not answer in time. Something is serving it, so this is the tool being slow ` +
"rather than absent.";
}
return `${key} failed: ${message}`;
}
-165
View File
@@ -1,165 +0,0 @@
/**
* The console's endpoint: MCP over HTTP, on a machine's loopback (novox/hq ADR 0152, design 34 §2).
*
* **Loopback is the authority boundary.** Whoever can connect is on the machine, and whoever is on the
* machine is the account that owns the mesh there (ADR 0034, ADR 0144). So there is no token and no
* login here, and the one thing this file enforces is that it binds nothing else: a console reachable
* from another machine would be authority over the mesh handed to whoever finds the port.
*
* The transport is the streamable-HTTP shape an agent host speaks: `POST /mcp` with one JSON-RPC
* message, answered with one JSON body. No session, because the surface holds nothing per caller; no
* event stream, because nothing here has anything to say unasked.
*/
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import type { Broker } from "@novox/mesh-sdk/messaging";
import { mcpSurface, type Reply, type Request } from "./mcp.js";
/** The most a request body may be. A tool's arguments are small; a megabyte is somebody else's file. */
const BODY_LIMIT = 1 << 20;
export interface Listening {
/** Where it listens, as `host:port`, with the port the machine actually gave. */
address: string;
close(): Promise<void>;
}
/** Hosts that are this machine and no other. */
const loopback = new Set(["127.0.0.1", "::1", "localhost", "[::1]"]);
/**
* Listen on `host:port`. Refused unless the host is loopback — said before binding, so a manifest or
* a flag that would open the console to a network is a startup failure rather than something
* discovered by whoever finds it.
*/
export async function serveMcpHttp(bus: Broker, who: string, listen: string): Promise<Listening> {
const at = listen.lastIndexOf(":");
if (at < 0) {
throw new Error(`"${listen}" is not host:port`);
}
const host = listen.slice(0, at);
const port = Number(listen.slice(at + 1));
if (!loopback.has(host)) {
throw new Error(
`the console listens on loopback and nowhere else (novox/hq ADR 0152): "${host}" is not this ` +
"machine's own address — whoever is on the machine owns the mesh there, and nobody else may reach this",
);
}
if (!Number.isInteger(port) || port < 0 || port > 65535) {
throw new Error(`"${listen.slice(at + 1)}" is not a port`);
}
const surface = mcpSurface(bus, who);
const server = createServer((req, res) => {
void route(req, res, surface.handle).catch((e) => {
json(res, 500, { jsonrpc: "2.0", id: null, error: { code: -32603, message: String(e) } });
});
});
await new Promise<void>((resolve, reject) => {
server.once("error", reject);
server.listen(port, host.replace(/^\[|\]$/g, ""), () => resolve());
});
const bound = server.address();
const address = typeof bound === "object" && bound ? `${host}:${bound.port}` : listen;
return {
address,
close: () =>
new Promise<void>((resolve) => {
server.close(() => resolve());
}),
};
}
async function route(
req: IncomingMessage,
res: ServerResponse,
handle: (r: Request) => Promise<Reply | undefined>,
): Promise<void> {
const path = (req.url ?? "/").split("?")[0];
if (path === "/") {
res.writeHead(200, { "content-type": "text/plain; charset=utf-8" });
res.end("the mesh's console: MCP over HTTP at POST /mcp (novox/hq design 34)\n");
return;
}
if (path !== "/mcp") {
json(res, 404, { error: "the console serves /mcp and nothing else" });
return;
}
switch (req.method) {
case "POST":
break;
case "DELETE":
// A host ending a session. There is no session to end; saying so is the truthful answer.
res.writeHead(204).end();
return;
case "GET":
// A host opening an event stream. The console has nothing to say unasked.
res.writeHead(405, { allow: "POST, DELETE" }).end();
return;
default:
res.writeHead(405, { allow: "POST, DELETE" }).end();
return;
}
let body: string;
try {
body = await read(req);
} catch (e) {
json(res, 413, { jsonrpc: "2.0", id: null, error: { code: -32600, message: String(e) } });
return;
}
let parsed: unknown;
try {
parsed = JSON.parse(body);
} catch {
json(res, 400, { jsonrpc: "2.0", id: null, error: { code: -32700, message: "the body is not JSON" } });
return;
}
// One message, or a batch of them; a batch is answered as a batch. A notification gets no reply
// and, alone, no body: 202 is how the transport says "heard".
if (Array.isArray(parsed)) {
const replies = (await Promise.all(parsed.map((r) => handle(r as Request)))).filter(Boolean);
if (replies.length === 0) {
res.writeHead(202).end();
} else {
json(res, 200, replies);
}
return;
}
const reply = await handle(parsed as Request);
if (!reply) {
res.writeHead(202).end();
return;
}
json(res, 200, reply);
}
function read(req: IncomingMessage): Promise<string> {
return new Promise((resolve, reject) => {
let size = 0;
const chunks: Buffer[] = [];
req.on("data", (chunk: Buffer) => {
size += chunk.length;
if (size > BODY_LIMIT) {
reject(new Error(`the request is larger than ${BODY_LIMIT} bytes`));
req.destroy();
return;
}
chunks.push(chunk);
});
req.on("end", () => resolve(Buffer.concat(chunks).toString("utf8")));
req.on("error", reject);
});
}
function json(res: ServerResponse, status: number, body: unknown): void {
const text = JSON.stringify(body);
res.writeHead(status, {
"content-type": "application/json; charset=utf-8",
"content-length": Buffer.byteLength(text),
});
res.end(text);
}
-192
View File
@@ -1,192 +0,0 @@
// A tools bundle as a process the runtime launches (novox/hq ADR 0188).
//
// The runtime does not run a tool's code itself when the bundle is not JavaScript: it starts the
// bundle's executable as a child with the runtime's environment and speaks MCP over stdio to it —
// `initialize`, `tools/list` once, `tools/call` per call. A Rust binary, a Go binary, a Python
// script and a Node script are the same thing from here: a process that answers those. Everything
// the mesh adds — the subjects from the membership, the held seats, the `tools` answer, a bundle
// that failed named and the others serving — is the runtime's, outside this file.
//
// A tool the child lists as `<seat>.<verb>` is the module's implementation of that seat's verb;
// any other name is the module's own tool. The same rule the in-process registration follows.
import { spawn, type ChildProcess } from "node:child_process";
import { accessSync, constants } from "node:fs";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import { broker, type Envelope } from "@novox/mesh-sdk/messaging";
import { atWork } from "./broker-nats.js";
/** The protocol version this speaks; a bundle says the same. */
export const PROTOCOL = "2025-03-26";
/** How long a child has to answer `initialize` and `tools/list` before it is a failed bundle, and
* how long a call may take before the caller is told the tool is slow rather than absent. */
const HANDSHAKE_MS = 10_000;
const CALL_MS = 30_000;
/** Whether an entrypoint is launched as a process rather than imported: anything that is not a
* plain JavaScript file, and a JavaScript file marked executable — a bundle written against the
* protocol in TypeScript, served the same way as any other language. */
export function launches(entry: string): boolean {
const javascript = /\.(m|c)?js$/.test(entry);
let executable = false;
try {
accessSync(entry, constants.X_OK);
executable = true;
} catch {
// not executable, or not there — importing will say which
}
return !javascript || executable;
}
/** What a launched bundle registers: the groups the in-process path would have, by name. */
export interface Launched {
registrations: { module: string; tools: ToolDefinition[] }[];
stop(): void;
}
interface Pending {
resolve(v: any): void;
reject(e: Error): void;
timer: NodeJS.Timeout;
}
/**
* Launch a bundle and learn its tools. Rejects when the child cannot be started or does not complete
* the handshake, which the runtime records as the bundle having failed. A child that exits later is
* started again on the next call, once; a call in flight when it died is told so.
*/
export async function launch(module: string, entry: string, env: NodeJS.ProcessEnv = process.env): Promise<Launched> {
let child: ChildProcess | undefined;
let nextId = 1;
const pending = new Map<number, Pending>();
let stopped = false;
const start = async (): Promise<void> => {
const proc = spawn(entry, [], { stdio: ["pipe", "pipe", "pipe"], env });
child = proc;
let buffered = "";
proc.stdout!.on("data", (chunk: Buffer) => {
buffered += chunk.toString("utf8");
let at: number;
while ((at = buffered.indexOf("\n")) >= 0) {
const line = buffered.slice(0, at).trim();
buffered = buffered.slice(at + 1);
if (!line) continue;
let reply: { id?: number | string; method?: string; params?: unknown; result?: unknown; error?: { message?: string } };
try {
reply = JSON.parse(line);
} catch {
console.log(`[mesh-tools] ${module}'s bundle said something that is not a reply: ${line.slice(0, 120)}`);
continue;
}
// **The bundle asks the runtime to emit** (novox/hq ADR 0193): published on the bus as this
// module, and answered once the bus has accepted it, so the tool's emit means what it means
// in-process. Nothing else a bundle may ask.
if (typeof reply.method === "string") {
const id = reply.id;
const answer = (m: Record<string, unknown>) => proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", id, ...m }) + "\n");
if (reply.method !== "mesh/publish") {
if (id !== undefined) answer({ error: { code: -32601, message: `the runtime answers no ${reply.method} from a bundle` } });
continue;
}
atWork.run({ module }, () => broker().publish(reply.params as Envelope<unknown>))
.then(() => { if (id !== undefined) answer({ result: {} }); })
.catch((err: unknown) => { if (id !== undefined) answer({ error: { code: -32000, message: err instanceof Error ? err.message : String(err) } }); });
continue;
}
const waiting = typeof reply.id === "number" ? pending.get(reply.id) : undefined;
if (!waiting) continue;
pending.delete(reply.id as number);
clearTimeout(waiting.timer);
if (reply.error) waiting.reject(new Error(reply.error.message ?? "the bundle refused the request"));
else waiting.resolve(reply.result);
}
});
// stderr is the bundle's log; kept under the module's name so a fault reads where it belongs.
// The last thing it said is kept, so a bundle that dies says why in its own words, not by code.
let lastSaid = "";
proc.stderr!.on("data", (chunk: Buffer) => {
for (const line of chunk.toString("utf8").split("\n")) {
if (!line.trim()) continue;
console.log(`[${module}] ${line}`);
if (/\S/.test(line) && !/^\s+at\s/.test(line) && !/^Node\.js v/.test(line)) lastSaid = line.trim();
}
});
const exited = new Promise<never>((_, reject) => {
proc.once("error", (err) => reject(err));
proc.once("exit", (code, signal) => {
const why = `${module}'s bundle exited (${signal ?? code})` + (lastSaid ? `: ${lastSaid}` : "");
for (const [id, p] of pending) {
pending.delete(id);
clearTimeout(p.timer);
p.reject(new Error(why));
}
if (child === proc) child = undefined;
if (!stopped) console.log(`[mesh-tools] ${why}; started again on its next call`);
reject(new Error(why));
});
});
const ask = (method: string, params: unknown, ms: number): Promise<any> =>
Promise.race([
new Promise<any>((resolve, reject) => {
const id = nextId++;
const timer = setTimeout(() => {
pending.delete(id);
reject(new Error(`${module}'s bundle did not answer ${method} in ${ms / 1000}s`));
}, ms);
pending.set(id, { resolve, reject, timer });
proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n");
}),
exited,
]);
exited.catch(() => {}); // observed through the race; never unhandled
(proc as ChildProcess & { ask?: typeof ask }).ask = ask;
await ask("initialize", { protocolVersion: PROTOCOL, capabilities: {}, clientInfo: { name: "node-tools", version: "1" } }, HANDSHAKE_MS);
proc.stdin!.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");
};
const asking = async (method: string, params: unknown, ms: number): Promise<any> => {
if (!child) await start();
return (child as ChildProcess & { ask: (m: string, p: unknown, ms: number) => Promise<any> }).ask(method, params, ms);
};
await start();
const listed = (await asking("tools/list", {}, HANDSHAKE_MS)) as { tools?: { name: string; description?: string; inputSchema?: unknown }[] };
const groups = new Map<string, ToolDefinition[]>();
for (const t of listed.tools ?? []) {
const dot = t.name.indexOf(".");
const under = dot < 0 ? module : t.name.slice(0, dot);
const name = dot < 0 ? t.name : t.name.slice(dot + 1);
const tools = groups.get(under) ?? [];
tools.push({
name,
description: t.description ?? "",
input: (t.inputSchema as Record<string, unknown> | undefined) ?? {},
run: async (args) => {
const result = (await asking("tools/call", { name: t.name, arguments: args ?? {} }, CALL_MS)) as {
content?: { type: string; text?: string }[];
isError?: boolean;
};
const text = result?.content?.find((c) => c.type === "text")?.text ?? "";
if (result?.isError) throw new Error(text || `${module}.${t.name} failed`);
// The bundle's answer is JSON as text (that is what every MCP host renders); handed back as
// the value it encodes so a caller on the bus sees what an in-process tool would return.
try {
return JSON.parse(text);
} catch {
return text;
}
},
});
groups.set(under, tools);
}
return {
registrations: [...groups].map(([under, tools]) => ({ module: under, tools })),
stop: () => {
stopped = true;
child?.kill("SIGTERM");
child = undefined;
},
};
}
-309
View File
@@ -1,309 +0,0 @@
// The runnable entrypoint. Three modes:
//
// mesh-tools serve — bind the broker and serve the assigned modules until
// stopped; as the node-tools module, also the console on loopback. A module entrypoint that subscribes to events (on("#"))
// starts consuming as it is imported, so this also runs consumers.
// mesh-tools emit TYPE [JSON] emit one event onto the mesh and exit — an operable primitive,
// and what an events test uses to put a message on the wire.
// mesh-tools prepare bring this module's state to the shape this version needs and exit
// — the runtime's answer to the word the mesh asks every module
// (novox/hq ADR 0135). The entrypoints come from MESH_PREPARE, which
// the module's own image names beside MESH_TOOL_MODULES.
// mesh-tools run ENTRYPOINT run one compiled module entrypoint to completion and exit — the
// runtime side of a run-once step (novox/hq ADR 0052). It imports
// the given entrypoint, whose top-level code does its work — seed a
// store, migrate, health-gate — and awaits it. It does NOT connect
// to the broker: a first-boot step runs offline, before the module
// has anything to talk to, and the host gates the container that
// depends on it on this process exiting 0.
//
// The broker, in order of preference:
// MESH_BROKER_FILE a sealed {url, fingerprint} the mesh delivered (novox/hq ADR 0043) — an
// amqps account scoped to this module. Preferred: a module holds its own.
// MESH_BROKER_URL a plain URL, for the bootstrap/admin case before a module has an account.
// MESH_TOOL_MODULES <module>=/path/a,… the modules to serve and their compiled entrypoints (serve
// mode; ADR 0175); a bare path is an entrypoint of the credential's own module
// MESH_PREPARE /path/a,/path/b,… compiled entrypoints that prepare this module's state
// MESH_MODULE / MESH_NODE the identity stamped onto emitted events (ADR 0042)
import { readFileSync } from "node:fs";
import { pathToFileURL } from "node:url";
import { connectNats, fatalBrokerReason as fatalNatsReason, type Credential } from "./broker-nats.js";
import { takeToolEnvs, runTools, type ServedModule } from "./runtime.js";
import { serveMcpHttp, type Listening } from "./http.js";
/** The credential this process connected with, for what it says beyond the connection (ADR 0159). */
let lastCredential: Credential | undefined;
import { invokeTool } from "@novox/mesh-sdk/tools";
import { useBroker } from "@novox/mesh-sdk/messaging";
import type { Broker } from "@novox/mesh-sdk/messaging";
import { emit } from "@novox/mesh-sdk/events";
/**
* Connect the way this process is meant to: with its sealed credential if the mesh gave it one, and
* over the plain bootstrap URL otherwise. A scoped module assumes the foundation's exchanges exist —
* its account may not declare them (ADR 0043).
*/
// fatalBrokerReasonFor is the reason a connection failure is final rather than "not yet", for
// whichever bus this runtime is on — each transport knows its own refusals.
function fatalBrokerReasonFor(err: unknown): string | null {
return fatalNatsReason(err);
}
async function connectBroker(): Promise<Broker> {
const file = process.env.MESH_BROKER_FILE;
if (file) {
let credential: Credential;
try {
credential = JSON.parse(readFileSync(file, "utf8")) as Credential;
lastCredential = credential;
} catch (err) {
console.error(`mesh-tools: cannot read the broker credential at ${file}: ${err}`);
process.exit(1);
}
if (!credential.url) {
console.error(`mesh-tools: ${file} carries no url — it is not a broker credential`);
process.exit(1);
}
// The mesh scoped this account to a node and module; take the runtime's identity from the
// credential so its queue and the events it emits match what the mesh authorised, no matter
// what the environment says.
if (credential.node) process.env.MESH_NODE = credential.node;
if (credential.module) process.env.MESH_MODULE = credential.module;
// **The credential names the bus.** A module moved to the bus being built was handed a
// credential for it — `nats://…` with user, password and fingerprint beside the address — and
// nothing else in its environment changed (design 25; novox/hq design 28 task 5.2). The scheme
// is enough to know which bus to speak; a runtime that always dialled the old one would keep
// serving and answer nobody.
// One bus (novox/hq ADR 0131, design 28 task 5.5): the credential names it, and it is this.
return connectNats(credential);
}
const url = process.env.MESH_BROKER_URL;
if (!url) {
console.error(
"mesh-tools: set MESH_BROKER_FILE (a sealed credential) or MESH_BROKER_URL — there is no broker to reach",
);
process.exit(1);
}
return connectNats({ url });
}
/**
* Connect for serve mode, retrying while the broker is merely not reachable yet. At startup that
* is the NORMAL case, not a failure: a container comes up in seconds and the overlay tunnel a
* moment later (novox/hq issue 058). Exiting instead delegated the retry to the container
* runtime, which read as a crash-loop to every restart-counting health check and every person
* watching. Retried indefinitely, aloud: the dependency appears or somebody reads why not.
*
* A failure that waiting cannot fix (see fatalBrokerReason) is thrown at once rather than retried —
* a permanent fault masquerading as "not reachable yet" is the silent non-progress this whole
* change exists to remove. Missing-file and empty-URL configuration errors exit inside
* connectBroker before they reach here; a malformed URL and a refused login are caught here.
*/
async function connectBrokerPatiently(): Promise<Broker> {
for (let delay = 2_000; ; delay = Math.min(delay * 2, 30_000)) {
try {
return await connectBroker();
} catch (err) {
const fatal = fatalBrokerReasonFor(err);
if (fatal !== null) {
console.error(`mesh-tools: ${fatal} — waiting will not fix this; giving up`);
throw err;
}
const why = err instanceof Error ? err.message : String(err);
// A little jitter so every module that was up when the broker bounced does not retry in
// lockstep and stampede it as it recovers.
const wait = delay + Math.floor(Math.random() * 1_000);
console.error(`mesh-tools: the broker is not reachable yet (${why}); retrying in ${Math.round(wait / 1000)}s`);
await new Promise((r) => setTimeout(r, wait));
}
}
}
/**
* What MESH_TOOL_MODULES names (novox/hq ADR 0175, to-be 38 WP1): `<module>=<entrypoint>` entries,
* comma-separated, several per module allowed — the node's runtime serving every assigned module's
* bundle. A bare path is the one-module form the per-module containers still set: an entrypoint of
* the credential's own module. Both may appear; the result is one list of modules.
*/
export function servedModulesFrom(spec: string, own: string | undefined): { serves: ServedModule[]; moduleEntrypoints: string[] } {
const serves = new Map<string, string[]>();
const moduleEntrypoints: string[] = [];
for (const raw of spec.split(",")) {
const entry = raw.trim();
if (!entry) continue;
const eq = entry.indexOf("=");
if (eq < 0) {
moduleEntrypoints.push(entry);
continue;
}
const module = entry.slice(0, eq).trim();
const path = entry.slice(eq + 1).trim();
if (!module || !path) {
throw new Error(`MESH_TOOL_MODULES: "${entry}" is neither <module>=<entrypoint> nor an entrypoint of this module`);
}
if (module === own) {
moduleEntrypoints.push(path);
continue;
}
serves.set(module, [...(serves.get(module) ?? []), path]);
}
return { serves: [...serves].map(([module, entrypoints]) => ({ module, entrypoints })), moduleEntrypoints };
}
/** The module that is the node's tool runtime (novox/hq ADR 0175, to-be 38 WP3): on its credential,
* `serve` is also the console — MCP on the machine's loopback (design 34). */
export const RUNTIME_MODULE = "node-tools";
/** Where the console listens when the runtime is node-tools and nothing says otherwise: the port
* the module's manifest declares `from: machine`. MESH_CONSOLE_LISTEN overrides it either way. */
const CONSOLE_LISTEN = "127.0.0.1:4270";
async function serve(): Promise<void> {
const broker = await connectBrokerPatiently();
// Parsed after connecting: a bare entrypoint belongs to the module the credential names.
const { serves, moduleEntrypoints } = servedModulesFrom(process.env.MESH_TOOL_MODULES ?? "", lastCredential?.module);
// Each module's environment, composed by the mesh (ADR 0192): taken before any bundle is imported.
const envs = takeToolEnvs();
const stop = await runTools({ broker, serves, moduleEntrypoints, credential: lastCredential, envs });
// The console is this runtime's serving mode (ADR 0175 §6): as node-tools, or wherever the
// listen address is given, the same process answers MCP on loopback for whoever is on the
// machine. A module's own runtime in a container on the machine's network does not — two of
// them on one port would be the fault, and the console is one per machine.
const listen = process.env.MESH_CONSOLE_LISTEN ?? (lastCredential?.module === RUNTIME_MODULE ? CONSOLE_LISTEN : "");
let consoleUp: Listening | undefined;
if (listen) {
const who = `${lastCredential?.node ?? "?"}.${lastCredential?.module ?? RUNTIME_MODULE}`;
consoleUp = await serveMcpHttp(broker, who, listen);
console.log(`mesh console listening on http://${consoleUp.address}/mcp as ${who}`);
}
const shutdown = async (): Promise<void> => {
stop();
await consoleUp?.close();
await broker.close();
process.exit(0);
};
process.on("SIGTERM", () => void shutdown());
process.on("SIGINT", () => void shutdown());
}
async function emitOnce(type: string, bodyJson: string): Promise<void> {
let body: unknown = {};
if (bodyJson) {
try {
body = JSON.parse(bodyJson);
} catch {
console.error(`mesh-tools emit: body is not JSON: ${bodyJson}`);
process.exit(1);
}
}
const broker = await connectBroker();
useBroker(() => broker);
// emit awaits the broker's publish confirm (ADR 0042), so the event is accepted before we close.
await emit(type, body);
await broker.close();
}
async function invokeOnce(module: string, tool: string, argsJson: string): Promise<void> {
let args: Record<string, unknown> = {};
if (argsJson) {
try {
args = JSON.parse(argsJson) as Record<string, unknown>;
} catch {
console.error(`mesh-tools invoke: args are not JSON: ${argsJson}`);
process.exit(1);
}
}
const broker = await connectBroker();
const result = await invokeTool(broker, module, tool, args);
process.stdout.write(JSON.stringify(result) + "\n");
await broker.close();
}
/**
* Run one compiled module entrypoint to completion — the runtime side of a run-once step
* (novox/hq ADR 0052). Importing it runs its top-level code and awaits any top-level await, so this
* returns only once the step's own code has finished; a step that throws rejects here and the
* process exits non-zero, which is how the host knows the step did not complete and must not start
* the container it gates. No broker is connected — a first-boot seed or migration runs offline.
*/
async function runEntry(entrypoint: string): Promise<void> {
if (!entrypoint) {
console.error("mesh-tools run <entrypoint> — a compiled module entrypoint path is required");
process.exit(1);
}
// A file URL, not a bare path: dynamic import of an absolute path is not portable, and the
// entrypoint the manifest names is an absolute path inside the image.
await import(pathToFileURL(entrypoint).href);
}
/**
* Bring this module's state to the shape this version needs, and exit — the runtime's answer to the
* one word the mesh asks every module (novox/hq ADR 0135).
*
* The entrypoints come from `MESH_PREPARE`, which a module's own image sets beside the entrypoints it
* already lists there: the module knows which of its files prepares its state, and nothing else could.
* Each is imported in the order given, to completion, with no broker — preparation runs before the
* version that would use it, so there is nothing yet to talk to.
*
* **An empty list is a failure, not a no-op.** The mesh only asks this of a module whose manifest says
* it prepares something; a module that says so and names nothing has been built wrong, and exiting 0
* would let that version serve against a state nobody shaped.
*/
async function prepareState(): Promise<void> {
const named = (process.env.MESH_PREPARE ?? "")
.split(",")
.map((entry) => entry.trim())
.filter((entry) => entry !== "");
if (named.length === 0) {
console.error(
"mesh-tools prepare: this module was asked to prepare its state and its image names nothing " +
"to do it with — set MESH_PREPARE to the compiled entrypoint(s) that prepare it, the way " +
"MESH_TOOL_MODULES names the ones it serves",
);
process.exit(1);
}
for (const entrypoint of named) {
console.log(`[mesh-tools] preparing with ${entrypoint}`);
await runEntry(entrypoint);
}
console.log(`[mesh-tools] prepared: ${named.length} entrypoint(s) ran to completion`);
}
async function main(): Promise<void> {
const [command, ...rest] = process.argv.slice(2);
if (command === "run") {
await runEntry(rest[0] ?? "");
return;
}
if (command === "prepare") {
await prepareState();
return;
}
if (command === "invoke") {
const [module, tool] = rest;
if (!module || !tool) {
console.error("mesh-tools invoke <module> <tool> [json-args] — a module and tool are required");
process.exit(1);
}
await invokeOnce(module, tool, rest[2] ?? "");
return;
}
if (command === "emit") {
const type = rest[0];
if (!type) {
console.error("mesh-tools emit <type> [json-body] — a routing key is required");
process.exit(1);
}
await emitOnce(type, rest[1] ?? "");
return;
}
await serve();
}
// Only when run, so a test can import the pieces.
if (process.argv[1] && import.meta.url === new URL(`file://${process.argv[1]}`).href) {
void main();
}
-258
View File
@@ -1,258 +0,0 @@
/**
* The mesh's tools as an MCP server (novox/hq design 25 §7, design 34).
*
* **A thin adapter and nothing more.** Every tool an agent sees is one a module answered for and one
* this account may call; the schema is the module's own; the answer is the module's own. Nothing here
* decides anything, which is why it is short — an MCP surface that reshaped arguments or summarised
* answers would be a second definition of what a tool is, and the module's code is the first.
*
* Implemented against the protocol directly rather than through a library: the surface is three
* methods and one framing, and a dependency here would be a dependency on every machine.
*
* One handler, two transports. Over stdio for a program a person starts (`mesh mcp`), over HTTP on a
* machine's loopback for the console the mesh assigns there (`mesh serve`, http.ts). The handler does
* not know which asked.
*/
import type { Broker } from "@novox/mesh-sdk/messaging";
import { callTool, seatsIn, toolKey, toolsOn, whyItFailed, type Listing, type Seats } from "./client.js";
/** The protocol version this speaks. Stated, because a host that wants another should be told so
* rather than discovering it through a shape it did not expect. */
export const PROTOCOL = "2025-03-26";
export interface Request {
jsonrpc: string;
id?: number | string | null;
method: string;
params?: Record<string, unknown>;
}
export interface Reply {
jsonrpc: "2.0";
id: Request["id"];
result?: unknown;
error?: { code: number; message: string };
}
/** How long a fetched tool list is kept before the modules are asked again. An agent asks on every
* turn; the mesh changes on the order of minutes. */
export const LISTING_KEPT_MS = 30_000;
export interface Surface {
/** Answer one request; undefined for a notification, which expects none. */
handle(request: Request): Promise<Reply | undefined>;
}
/**
* The surface over one bus connection, as one account.
*
* The tool list is fetched when first asked and kept for a short while (design 34 §3): asking every
* module on every `tools/list` would fan out on every agent turn for something nobody changed, and
* never refreshing would hide a module assigned a moment ago.
*/
export function mcpSurface(bus: Broker, who: string): Surface {
let known: { listing: Listing; at: number } | undefined;
const answer = (id: Request["id"], result: unknown): Reply => ({ jsonrpc: "2.0", id, result });
const refuse = (id: Request["id"], code: number, message: string): Reply => ({
jsonrpc: "2.0",
id,
error: { code, message },
});
const listing = async (): Promise<Listing> => {
if (!known || Date.now() - known.at > LISTING_KEPT_MS) {
known = { listing: await toolsOn(bus), at: Date.now() };
}
return known.listing;
};
// The roles the last listing knew, so `<seat>.<verb>` resolves to the seat. Fetched once if a
// call arrives before any list did; a listing that failed leaves no roles, and the name is then
// a module's, which is the right fallback for a mesh whose control plane is away.
const roles = async (): Promise<Seats | undefined> => {
try {
return seatsIn(await listing());
} catch {
return undefined;
}
};
return {
async handle(request) {
// A notification has no id and expects no answer; `initialized` is the one every host sends.
const notification = request.id === undefined || request.id === null;
switch (request.method) {
case "initialize":
return answer(request.id, {
protocolVersion: PROTOCOL,
capabilities: { tools: {} },
serverInfo: { name: "mesh", version: "1" },
// Said in the handshake, because an agent that knows whose authority it is acting under
// can say so when a call is refused — and a refusal is the one thing here that is not
// the mesh's fault or the tool's.
instructions:
`These are the tools of a Novox mesh, reached as ${who}. Every call goes to the module ` +
`that serves it; what may be called was fixed when this account was issued, so a ` +
`refusal means the account, not the tool. The list is what the running modules ` +
`answered, plus every role's tools from the mesh's records — the mesh's own verbs ` +
`(mesh-controller.status, .push, .assign …) among them; a module that did not answer ` +
`is named in the list's _meta and can still be called by <module>.<tool>.`,
});
case "notifications/initialized":
return undefined;
case "ping":
return notification ? undefined : answer(request.id, {});
case "tools/list": {
let have: Listing;
try {
have = await listing();
} catch (e) {
return refuse(request.id, -32603, whyItFailed("mesh-catalog.catalog_modules", e));
}
return answer(request.id, {
tools: have.tools.map((t) => ({
name: `${t.module}.${t.name}`,
description: t.description ?? `${t.name}, served by ${t.module}`,
// The module's own schema, passed through — with `node`, the machine to ask when
// the module runs on several (novox/hq ADR 0159); a seat's verb takes none, the
// seat's scope decides. An empty object is a tool that takes nothing, which is a
// real answer and not a missing one.
inputSchema: t.seat && t.scope !== "node" ? asSchema(t.input)
: t.seat ? withNode(asSchema(t.input), "the machine whose seat answers; required, the seat is held once per machine", true)
: withNode(asSchema(t.input)),
})),
// Silence, named (design 34 §3): the modules the catalogue holds and nothing answered
// for. Not a tool, so not in `tools`; not dropped either.
_meta: { notAnswering: have.notAnswering },
});
}
case "tools/call": {
const given = String(request.params?.name ?? "");
const args = { ...((request.params?.arguments as Record<string, unknown> | undefined) ?? {}) };
// The machine, when the caller names one, travels in the subject and never reaches the
// module's arguments (novox/hq ADR 0159) — for a module's tool. A seat's verb takes no
// machine from the console (the seat's scope decides), so a `node` among its arguments
// is the verb's own, as `push` and `assign` take one, and is handed through untouched.
const have = await listing().catch(() => undefined);
const roles = have && seatsIn(have);
const bare = given.split("@", 1)[0];
const isSeatVerb = roles ? toolKey(bare, roles).startsWith("seat:") : false;
// A node-scoped seat's verb is asked of one machine (design 33 §4, ADR 0170): `node`
// names it and travels in the subject, as for a module's tool.
const nodeScoped = isSeatVerb && (have?.tools.some((t) => t.seat && t.scope === "node" &&
`${t.module}.${t.name}` === bare) ?? false);
const takesNode = !isSeatVerb || nodeScoped;
const node = takesNode && typeof args.node === "string" && args.node !== "" ? args.node : "";
if (takesNode) delete args.node;
if (nodeScoped && !node && !given.includes("@")) {
return refuse(request.id, -32602, `${given} is a machine's seat's verb: name the machine with \`node\``);
}
const name = node && !given.includes("@") ? `${given}@${node}` : given;
try {
const { result, node: answeredBy } = await callTool(bus, name, args, roles, have);
// Text, because that is what every host renders. The content is the module's answer
// as JSON, unshaped: an adapter that flattened it would be deciding what matters in
// somebody else's answer. Which machine answered follows it as its own line.
const content: { type: string; text: string }[] = [
{ type: "text", text: JSON.stringify(result, null, 2) },
];
if (answeredBy) content.push({ type: "text", text: `answered by ${answeredBy}` });
return answer(request.id, { content });
} catch (e) {
// **An error the agent can act on, not a stack.** isError rather than a protocol
// failure, because the call was well-formed and the mesh answered it — with a refusal,
// an absence or a fault, and the words say which.
return answer(request.id, {
content: [{ type: "text", text: whyItFailed(name, e) }],
isError: true,
});
}
}
default:
return notification ? undefined : refuse(request.id, -32601, `mesh's MCP surface has no ${request.method}`);
}
},
};
}
/**
* A module's declared input as a JSON schema an agent can read.
*
* The sdk keeps a tool's input opaque, and the catalogue's modules write it as a bare map of
* property to description — `{ module: { type, description } }` — which is the `properties` of a
* schema rather than a schema. Wrapped here when that is what arrived; passed through when a module
* already wrote a schema; an empty object when it declared nothing. The module's words are kept
* either way.
*/
export function asSchema(input: unknown): Record<string, unknown> {
if (!input || typeof input !== "object" || Array.isArray(input)) {
return { type: "object", properties: {} };
}
const given = input as Record<string, unknown>;
if (given.type === "object" || "properties" in given) return given;
if (Object.keys(given).length === 0) return { type: "object", properties: {} };
return { type: "object", properties: given };
}
/**
* Serve over stdio until stdin closes, which is how a host ends a session.
*/
export async function serveMcp(bus: Broker, who: string): Promise<void> {
const surface = mcpSurface(bus, who);
const say = (message: unknown) => {
process.stdout.write(`${JSON.stringify(message)}\n`);
};
for await (const line of lines()) {
let request: Request;
try {
request = JSON.parse(line) as Request;
} catch {
// Unparseable, and with no id there is nobody to tell. Skipped rather than answered, because a
// reply to a request that was never framed is noise on the same channel.
continue;
}
const reply = await surface.handle(request);
if (reply) say(reply);
}
}
/** stdin as newline-framed messages, which is what MCP over stdio is. */
async function* lines(): AsyncGenerator<string> {
let buffered = "";
for await (const chunk of process.stdin) {
buffered += (chunk as Buffer).toString("utf8");
let at: number;
while ((at = buffered.indexOf("\n")) >= 0) {
const line = buffered.slice(0, at).trim();
buffered = buffered.slice(at + 1);
if (line !== "") yield line;
}
}
if (buffered.trim() !== "") yield buffered.trim();
}
/** Every module tool takes an optional `node`: the machine to ask when the module runs on several
* (novox/hq ADR 0159). Added to the listing, stripped before the call, never seen by the module. */
function withNode(schema: Record<string, unknown>, description?: string, required = false): Record<string, unknown> {
const properties = { ...((schema.properties as Record<string, unknown> | undefined) ?? {}) };
if (!("node" in properties)) {
properties.node = {
type: "string",
description: description ??
"the machine to ask, when this module runs on several; else whichever answers, and the answer says which",
};
}
const out: Record<string, unknown> = { ...schema, type: "object", properties };
if (required) {
const have = Array.isArray(schema.required) ? (schema.required as string[]) : [];
out.required = have.includes("node") ? have : [...have, "node"];
}
return out;
}
-278
View File
@@ -1,278 +0,0 @@
#!/usr/bin/env node
/**
* `mesh` — the mesh's tools, for whoever is on a machine (novox/hq design 25 §7, design 34).
*
* Four verbs and nothing else. What tools are there, call one, serve the same two to an agent over
* stdio, and serve them on a machine's loopback as the console the mesh assigns. Deliberately thin:
* everything that could be a decision is one the mesh already made, and a client that grew opinions
* would be a second place the mesh's behaviour is defined.
*
* mesh tools what the running modules answer
* mesh call <module>.<tool> [json] call one, arguments as JSON on the command line or on stdin
* mesh mcp the same two, as an MCP server over stdio
* mesh serve [--listen host:port] the console: MCP over HTTP on this machine's loopback
*
* Who it speaks as, in order of preference:
* MESH_BROKER_FILE the module credential the mesh delivered — the console's (ADR 0152)
* MESH_CREDENTIAL / --credential <file> a person's, as `operator issue` printed it (design 25 §7)
* MESH_CONSOLE / --console <url> no credential: `tools` and `call` go through a console
* already running on this machine, over loopback HTTP
*/
import { readFile } from "node:fs/promises";
import type { Broker } from "@novox/mesh-sdk/messaging";
import {
callTool,
connectAs,
connectAsTheConsole,
credentialFrom,
seatsIn,
toolsOn,
whyItFailed,
type Listing,
} from "./client.js";
import { serveMcpHttp } from "./http.js";
import { serveMcp } from "./mcp.js";
const usage = `mesh tools
mesh call <module>.<tool> [json]
mesh mcp
mesh serve [--listen host:port]
--credential <file> a person's credential, the JSON \`operator issue\` printed; default $MESH_CREDENTIAL
--console <url> a console on this machine to ask through instead; default $MESH_CONSOLE
--listen <host:port> where \`serve\` listens; loopback only; default $MESH_CONSOLE_LISTEN or 127.0.0.1:4270
MESH_BROKER_FILE the module credential the mesh delivered, which \`serve\` runs on`;
/** What the console listens on when nothing says otherwise. */
const DEFAULT_LISTEN = "127.0.0.1:4270";
async function main(argv: string[]): Promise<number> {
const args = [...argv];
let credentialPath = process.env.MESH_CREDENTIAL ?? "";
let consoleUrl = process.env.MESH_CONSOLE ?? "";
let listen = process.env.MESH_CONSOLE_LISTEN ?? DEFAULT_LISTEN;
for (let i = 0; i < args.length; i++) {
const take = () => {
const v = args[i + 1] ?? "";
args.splice(i, 2);
i--;
return v;
};
if (args[i] === "--credential") credentialPath = take();
else if (args[i] === "--console") consoleUrl = take();
else if (args[i] === "--listen") listen = take();
}
const verb = args.shift();
if (!verb || verb === "help" || verb === "--help") {
console.log(usage);
return verb ? 0 : 1;
}
// Through a console already on this machine: no credential to hold, which is the point of one.
if (consoleUrl && (verb === "tools" || verb === "call")) {
return verb === "tools" ? listingVia(consoleUrl) : callingVia(consoleUrl, args);
}
const { bus, who } = await connecting(credentialPath, verb);
try {
switch (verb) {
case "tools":
return await listing(bus, who);
case "call":
return await calling(bus, args);
case "mcp":
// Serves until stdin closes, which is how an MCP host ends a session.
await serveMcp(bus, who);
return 0;
case "serve": {
const up = await serveMcpHttp(bus, who, listen);
console.log(`mesh console listening on http://${up.address}/mcp as ${who}`);
await new Promise<void>((resolve) => {
process.once("SIGTERM", () => resolve());
process.once("SIGINT", () => resolve());
});
await up.close();
return 0;
}
default:
console.error(`mesh has no "${verb}".\n\n${usage}`);
return 1;
}
} finally {
await bus.close();
}
}
/**
* Who this process is on the bus. The module credential first: a console is started by the mesh with
* MESH_BROKER_FILE and nothing else, and must not fall back to a person's file lying around.
*/
async function connecting(credentialPath: string, verb: string): Promise<{ bus: Broker; who: string }> {
const delivered = process.env.MESH_BROKER_FILE;
if (delivered) {
return connectAsTheConsole(delivered);
}
if (!credentialPath) {
throw new Error(
verb === "serve"
? "no credential: the console runs on MESH_BROKER_FILE, the module credential the mesh " +
"delivered; to run it by hand, pass --credential <file> with a person's credential"
: "no credential: set MESH_CREDENTIAL or pass --credential <file> (the JSON `operator issue` " +
"printed, saved verbatim), or --console <url> to ask through a console on this machine",
);
}
const held = await credentialFrom(credentialPath);
return { bus: await connectAs(held), who: held.person ?? held.user ?? "somebody" };
}
function printListing(have: Listing, who?: string): void {
if (have.tools.length === 0) {
console.log("no running module answered with any tool");
}
// **What the modules answered, not what this account may call.** The two differ and the
// difference is the point: an account seeing only its own tools cannot tell "not installed" from
// "not yours", and those need different people to fix them.
for (const t of have.tools) {
const name = `${t.module}.${t.name}`;
const line = t.description ? `${name.padEnd(36)} ${t.description}` : name;
console.log(t.seat ? `${line} (a role's tool: answered by whoever holds the ${t.module} seat)` : line);
}
if (have.notAnswering.length > 0) {
console.log(
`\nheld by the mesh and not answering: ${have.notAnswering.join(", ")} — not assigned, not up, ` +
"or built before the runtime answered `tools`; each can still be called by name",
);
}
if (who) {
console.log(`\nasked as ${who}; what ${who} may call was fixed when the account was issued.`);
}
}
async function listing(bus: Broker, who?: string): Promise<number> {
let have: Listing;
try {
have = await toolsOn(bus);
} catch (e) {
console.error(whyItFailed("mesh-catalog.catalog_modules", e));
return 1;
}
printListing(have, who);
return 0;
}
async function calling(bus: Broker, args: string[]): Promise<number> {
const key = args.shift();
if (!key) {
console.error("mesh call <module>.<tool> [json]");
return 1;
}
const parsed = await argumentsFrom(args);
if (parsed === undefined) return 1;
try {
// `<seat>.<verb>` reaches the role when the mesh lists that verb for the seat; `seat:` says so
// outright and asks nothing first.
const have = key.startsWith("seat:") ? undefined : await toolsOn(bus).catch(() => undefined);
const { result, node } = await callTool(bus, key, parsed, have && seatsIn(have), have);
console.log(JSON.stringify(result, null, 2));
if (node) console.error(`answered by ${node}`);
return 0;
} catch (e) {
console.error(whyItFailed(key, e));
return 1;
}
}
/** The console's answer to one MCP request, over loopback HTTP. */
async function viaConsole(consoleUrl: string, method: string, params?: unknown): Promise<any> {
const endpoint = consoleUrl.endsWith("/mcp") ? consoleUrl : `${consoleUrl.replace(/\/$/, "")}/mcp`;
const res = await fetch(endpoint, {
method: "POST",
headers: { "content-type": "application/json", accept: "application/json" },
body: JSON.stringify({ jsonrpc: "2.0", id: 1, method, params }),
});
if (!res.ok) {
throw new Error(`the console at ${endpoint} answered ${res.status}`);
}
const reply = (await res.json()) as { result?: any; error?: { message: string } };
if (reply.error) throw new Error(reply.error.message);
return reply.result;
}
async function listingVia(consoleUrl: string): Promise<number> {
try {
const result = await viaConsole(consoleUrl, "tools/list");
const have: Listing = {
tools: (result.tools ?? []).map((t: { name: string; description?: string; inputSchema?: unknown }) => {
const at = t.name.indexOf(".");
return { module: t.name.slice(0, at), name: t.name.slice(at + 1), description: t.description, input: t.inputSchema };
}),
notAnswering: result._meta?.notAnswering ?? [],
};
printListing(have);
return 0;
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
return 1;
}
}
async function callingVia(consoleUrl: string, args: string[]): Promise<number> {
const key = args.shift();
if (!key) {
console.error("mesh call <module>.<tool> [json]");
return 1;
}
const parsed = await argumentsFrom(args);
if (parsed === undefined) return 1;
try {
const result = await viaConsole(consoleUrl, "tools/call", { name: key, arguments: parsed });
const text = result?.content?.[0]?.text ?? JSON.stringify(result);
if (result?.isError) {
console.error(text);
return 1;
}
console.log(text);
return 0;
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
return 1;
}
}
/** A call's arguments: JSON on the command line, else on stdin, else nothing. Undefined when what
* was given is not JSON, after saying so. */
async function argumentsFrom(args: string[]): Promise<unknown> {
const raw = args.length > 0 ? args.join(" ") : await maybeStdin();
if (raw.trim() === "") return {};
try {
return JSON.parse(raw);
} catch (e) {
console.error(`the arguments are not JSON: ${(e as Error).message}`);
return undefined;
}
}
/** Arguments on stdin, for a call whose JSON is too long or too quoted to type. Empty when stdin is a
* terminal, so `mesh call x.y` with no arguments does not hang waiting for something nobody is
* typing. */
async function maybeStdin(): Promise<string> {
if (process.stdin.isTTY) return "";
const chunks: Buffer[] = [];
for await (const chunk of process.stdin) chunks.push(chunk as Buffer);
return Buffer.concat(chunks).toString("utf8");
}
// Only when run, so a test can import the pieces.
if (process.argv[1] && import.meta.url === new URL(`file://${process.argv[1]}`).href) {
main(process.argv.slice(2))
.then((code) => process.exit(code))
.catch((e) => {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
});
}
export { main, usage };
export const _readFile = readFile;
-384
View File
@@ -1,384 +0,0 @@
// The tool runtime — the per-node process that makes the mesh's tools actually serve (novox/hq
// ADR 0175). It binds the mesh broker, loads the served modules' tool bundles (each of which calls
// registerModuleTools as it imports), and serves every module's tools on that module's subjects and
// every held seat's verbs on the seat's. Everything hard — dispatch, collection, duplicate-name
// safety — is the sdk's; this is the wrapper.
//
// One runtime, many modules. It was written for one module per process and ran that way in a
// container per module; it now serves a list, as the one process per node the host supervises,
// and the per-module shape is the list with one entry. A bundle that fails to import is named —
// in the log and in what `tools` answers for it — and the others serve.
import { pathToFileURL } from "node:url";
import { resolve } from "node:path";
import { useBroker } from "@novox/mesh-sdk/messaging";
import { collectTools, toolKey, type ToolDefinition } from "@novox/mesh-sdk/tools";
import type { Broker } from "@novox/mesh-sdk/messaging";
import { atWork, seatToolSubject, type Credential, type RuntimeBroker } from "./broker-nats.js";
import { launch, launches } from "./launch.js";
import { announce, endpointsOf, type ServedSeatVerb } from "./announce.js";
/**
* The one verb every module's runtime answers for it (novox/hq ADR 0152, design 34 §3): the
* module's tool names, descriptions and argument schemas, from the code that answers them and
* from nowhere else. Discovery asks the module, because a copy kept anywhere else drifts.
*/
export const TOOLS_VERB = "tools";
/** What `tools` answers for one module. */
export interface ToolsAnswer {
module: string;
tools: {
name: string;
description: string;
input: Readonly<Record<string, unknown>>;
/** Where this tool is answered, as the mesh issued it (ADR 0160): the module's plain subject
* first when there is one, then this machine's. A caller composes nothing. */
subjects?: string[];
}[];
/** Why this module serves nothing here, when its bundle failed to load (ADR 0175): said where
* discovery looks, so a module that is silent and one that is broken are told apart. */
failed?: string;
}
/** One module this runtime serves: its name and its compiled tool entrypoints. */
export interface ServedModule {
module: string;
/** Absolute paths to the module's compiled tool entrypoints (e.g. .../umami/tools/index.js). */
entrypoints: string[];
}
export interface RuntimeOptions {
/** The mesh broker to serve over. */
broker: Broker;
/** The modules to serve, each with its entrypoints. */
serves?: ServedModule[];
/** The credential's own module's entrypoints — the one-module form, which the per-module
* containers still use; the same as naming the credential's module in `serves`. */
moduleEntrypoints?: string[];
/** The credential the mesh delivered, for what it says about the seats this module claims
* (novox/hq ADR 0159). Absent for a runtime started by hand, which then serves no seat its
* memberships do not name. */
credential?: Credential;
/** What each served module's bundles are given (novox/hq ADR 0192): module → words, composed by
* the mesh per machine. A module absent here is given the runtime's own words and nothing more. */
envs?: ReadonlyMap<string, Readonly<Record<string, string>>>;
}
/** The variable the mesh composes every served module's environment into, as JSON (ADR 0192). Read
* once at start and removed from the process's environment, so no bundle finds another's there. */
export const TOOL_ENV = "MESH_TOOL_ENV";
/** Read and remove the composed environments from an environment (the process's, by default). */
export function takeToolEnvs(env: NodeJS.ProcessEnv = process.env): Map<string, Record<string, string>> {
const raw = env[TOOL_ENV];
delete env[TOOL_ENV];
const out = new Map<string, Record<string, string>>();
if (!raw) return out;
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
throw new Error(`${TOOL_ENV} is not JSON; the mesh composes it as {"<module>": {"<word>": "<value>"}}`);
}
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
throw new Error(`${TOOL_ENV} is not an object of modules`);
}
for (const [module, words] of Object.entries(parsed as Record<string, unknown>)) {
if (!words || typeof words !== "object" || Array.isArray(words)) {
throw new Error(`${TOOL_ENV}: ${module}'s environment is not an object of words`);
}
const own: Record<string, string> = {};
for (const [k, v] of Object.entries(words as Record<string, unknown>)) own[k] = String(v);
out.set(module, own);
}
return out;
}
/** Two environment words the mesh sets for the node's runtime and every tool reads from its
* environment: whose machine this is (novox/hq to-be 37 §3, ADR 0175). */
export const OPERATOR_ACCOUNT = "MESH_OPERATOR_ACCOUNT";
export const OPERATOR_HOME = "MESH_OPERATOR_HOME";
/** Load the modules, bind the broker, and serve. Returns a stop function that unhooks serving. */
export async function runTools(opts: RuntimeOptions): Promise<() => void> {
useBroker(() => opts.broker);
const runtime = opts.broker as RuntimeBroker;
// Whose runtime this is: the credential's module, or the connection's own when a runtime is
// started by hand without one — the broker was told its module when it connected.
const self = opts.credential?.module ?? (typeof runtime.module === "string" ? runtime.module : undefined);
// What to serve: the list, with the one-module form folded in as the credential's own entry.
const served = new Map<string, string[]>();
for (const s of opts.serves ?? []) {
served.set(s.module, [...(served.get(s.module) ?? []), ...s.entrypoints]);
}
const ownEntrypoints = opts.moduleEntrypoints ?? [];
if (opts.moduleEntrypoints?.length) {
if (!self) {
throw new Error(
"entrypoints were given with no module to serve them as: name the module (MESH_TOOL_MODULES " +
"as <module>=<entrypoint>) or connect on a credential that names one",
);
}
served.set(self, [...(served.get(self) ?? []), ...opts.moduleEntrypoints]);
}
// The operator's machine, said once so a tool's behaviour under it can be read back from the
// log. Tools read the two words from their own environment, which is this process's.
const account = process.env[OPERATOR_ACCOUNT];
if (account) {
console.log(`[mesh-tools] the operator's account here is ${account}` +
(process.env[OPERATOR_HOME] ? ` (home ${process.env[OPERATOR_HOME]})` : ""));
}
// Follow every served module's membership before loading anything, so what each is issued is
// known when its tools are bound. A module's own runtime already follows its own.
if (typeof runtime.follow === "function") {
for (const module of served.keys()) await runtime.follow(module);
}
// What each module's bundles are given: the runtime's own words, and over them the module's own.
const envs = opts.envs ?? new Map<string, Record<string, string>>();
const envFor = (module: string): NodeJS.ProcessEnv => ({ ...process.env, ...(envs.get(module) ?? {}) });
// Every bundle this runtime serves is launched as a process and spoken to over MCP on stdio (ADR
// 0188, ADR 0193): given the runtime's words and its module's own, and told the module it serves
// it as. The runtime knows no language; an entrypoint that is not executable was not built to be
// served, and is refused by name. A bundle that fails to start is named, and the others serve
// (ADR 0175: one faulty bundle must not take the node's tools down).
//
// The one-module form — the credential's own module's entrypoints, which the per-module containers
// still use for their event handlers and provisioners until they move (to-be 38 WP4c) — is imported
// into this process as before: one module, one SDK, its container's own environment.
// The machine this runtime serves, which an event a launched tool emits is stamped with.
const node = opts.credential?.node ?? (typeof (runtime as unknown as { node?: unknown }).node === "string" ? (runtime as unknown as { node: string }).node : undefined);
const failed = new Map<string, string>();
const launched: { module: string; owner: string; tools: ToolDefinition[] }[] = [];
const children: Array<() => void> = [];
for (const [module, entrypoints] of served) {
const imported = module === self && ownEntrypoints.length > 0;
for (const entry of entrypoints) {
const path = resolve(entry);
try {
if (imported) {
await import(pathToFileURL(path).href);
continue;
}
if (!launches(path)) {
throw new Error(`${path} is not executable; a bundle the runtime serves is started, never imported, and its build makes it executable (novox/hq ADR 0193)`);
}
const child = await launch(module, path, {
...envFor(module), MESH_SERVED_MODULE: module, MESH_MODULE: module,
...(node ? { MESH_NODE: node } : {}),
});
children.push(child.stop);
for (const r of child.registrations) launched.push({ ...r, owner: module });
} catch (err) {
const why = err instanceof Error ? err.message : String(err);
failed.set(module, why);
console.log(`[mesh-tools] ${module}'s bundle ${entry} failed to load: ${why}; its tools are not served here`);
}
}
}
// A registration under a served module's name is that module's tools, served on its subjects.
// One under a seat's name is the module's implementation of that seat's verbs (ADR 0159, 0160):
// served on the seat's subjects by serveClaimedSeats where some served module claims the seat,
// never as a module's tools and never listed among them. A module named like its seat (the
// catalogue is the mesh-catalog seat) registers once and is both. Anything else is said and left
// out rather than fatal — on 2026-10-01 the credential of a module that had just learned to
// implement a seat did not yet name the claim, and the whole runtime restarted for it.
const claimed = seatsClaimed(served.keys(), self, opts.credential, runtime);
const registrations = [
...collectTools().map((r) => ({ ...r, owner: self ?? r.module })),
...launched,
];
const ownRegistrations = registrations.filter(({ module, owner: by }) => {
if (served.has(module)) return true;
if (claimed.has(module)) return false;
console.log(`[mesh-tools] ${by} registers tools under "${module}", which is neither a module served here nor a seat one of them claims; not served until the mesh issues the claim`);
return false;
});
const stops: Array<() => void> = [...children];
const stop = (): void => stops.splice(0).forEach((s) => s());
// Refused before anything is bound if a module named a tool of its own `tools`: one name
// answering two things is the fault nobody can diagnose afterwards, and the runtime is the only
// place that sees both. Likewise two tools of one module under one name.
for (const { module, tools: own } of ownRegistrations) {
if (own.some((t) => t.name === TOOLS_VERB)) {
throw new Error(
`${module} names a tool "${TOOLS_VERB}", which is the verb the runtime answers for every ` +
"module with what it serves (novox/hq ADR 0152) — refused, rename it",
);
}
const seen = new Set<string>();
for (const t of own) {
if (seen.has(t.name)) throw new Error(`${module} exposes two tools named ${t.name} — refused`);
seen.add(t.name);
}
}
// Each tool on its own key, namespaced by its module (ADR 0047); where that key is answered is
// the broker's to know from the module's membership (ADR 0160). A tool runs attributed to its
// module, so what it emits lands on the module's subject and not the runtime's.
const names: string[] = [];
for (const { module, tools: own } of ownRegistrations) {
for (const t of own) {
names.push(toolKey(module, t.name));
stops.push(await opts.broker.handle(toolKey(module, t.name), (args: Record<string, unknown> | undefined) =>
atWork.run({ module }, () => t.run(args ?? {}))));
}
}
// And, for every served module, the verb that says what it serves — nothing, and why, for a
// module whose bundle failed. A module that registered nothing and did not fail is a pure-events
// module (the audit logger), whose scoped account may not declare the serve queue; it is left
// silent as it always was.
const byModule = new Map<string, ToolDefinition[]>();
for (const { module, tools: own } of ownRegistrations) {
byModule.set(module, [...(byModule.get(module) ?? []), ...own]);
}
for (const module of served.keys()) {
const own = byModule.get(module) ?? [];
const why = failed.get(module);
if (own.length === 0 && !why) continue;
const subjectsOf = (tool: string): string[] | undefined => {
const m = typeof runtime.membership === "function" ? runtime.membership(module) : undefined;
if (!m) return undefined;
const plain = m.serves.filter((s) => s.queue).map((s) => s.subject.replace("{tool}", tool));
const mine = m.serves.filter((s) => !s.queue).map((s) => s.subject.replace("{tool}", tool));
return [...plain, ...mine];
};
stops.push(await opts.broker.handle(toolKey(module, TOOLS_VERB), async (): Promise<ToolsAnswer> => ({
module,
tools: own.map((t) => ({ name: t.name, description: t.description, input: t.input, subjects: subjectsOf(t.name) })),
...(why ? { failed: why } : {}),
})));
}
console.log(`[mesh-tools] serving ${names.length} tool(s) for ${served.size} module(s): ${names.join(", ") || "(none)"}` +
(failed.size ? `; not serving ${[...failed.keys()].join(", ")}, whose bundle(s) failed to load` : ""));
const seats = await serveClaimedSeats(runtime, [...served.keys()], self, opts.credential, registrations);
stops.push(seats.stop);
// **What it serves, it announces** (novox/hq ADR 0197), asked at the moment of the request.
if (typeof runtime.raw === "function") {
stops.push(announce(runtime, {
name: self ?? "runtime", id: runtime.node ?? self ?? "runtime",
description: `the tool runtime of ${self ?? "a module"}${runtime.node ? ` on ${runtime.node}` : ""}`,
metadata: runtime.node ? { node: runtime.node } : {},
}, () => endpointsOf(runtime, ownRegistrations, seats.serving())));
}
return () => stop();
}
/** The seats some served module claims: from the credential for its own module, and from every
* served module's membership (ADR 0160) — the node's runtime holds no claims of its own. */
function seatsClaimed(
modules: Iterable<string>,
self: string | undefined,
credential: Credential | undefined,
runtime: RuntimeBroker,
): Set<string> {
const out = new Set<string>();
for (const c of credential?.claims ?? []) if (credential?.module === self) out.add(c.seat);
for (const module of modules) {
const m = typeof runtime.membership === "function" ? runtime.membership(module) : undefined;
for (const s of m?.seats ?? []) out.add(s.seat);
}
return out;
}
/** One seat's verb, where its callers ask, and which served module holds the seat. */
interface SeatVerb {
seat: string;
verb: string;
subject: string;
holder: string;
}
/**
* Holding a seat means serving its tools (design 33 §3, novox/hq ADR 0159). What a served module
* claims and promises comes from its membership (ADR 0160) — and, for a module's own runtime, from
* its credential, which named the claims before memberships did. Each verb is served on the seat's
* own subject by the tool of the same name registered under the seat's name. Whether this instance
* *holds* the seat is the bus's to decide: only the holder's account may subscribe the seat's
* subjects, so a claimant that does not hold it here is refused the subscription and serves nothing
* — never a failure of its own tools.
*/
async function serveClaimedSeats(
broker: RuntimeBroker,
served: string[],
self: string | undefined,
credential: Credential | undefined,
registrations: { module: string; owner: string; tools: ToolDefinition[] }[],
): Promise<{ stop: () => void; serving: () => ServedSeatVerb[] }> {
if (typeof broker.handleSubject !== "function") return { stop: () => {}, serving: () => [] };
// A seat's verbs are the role's, not the software's (ADR 0159): implemented under the seat's
// name — `registerModuleTools("mesh-store", …)` — and never confused with the module's own tools.
const implementations = new Map<string, Map<string, (args: Record<string, unknown>) => Promise<unknown>>>();
const definitions = new Map<string, Map<string, ToolDefinition>>();
for (const { module, tools } of registrations) {
const defs = definitions.get(module) ?? new Map<string, ToolDefinition>();
for (const t of tools) defs.set(t.name, t);
definitions.set(module, defs);
}
for (const { module, owner, tools } of registrations) {
const verbs = implementations.get(module) ?? new Map<string, (args: Record<string, unknown>) => Promise<unknown>>();
for (const t of tools) verbs.set(t.name, (args) => atWork.run({ module: owner }, () => t.run(args)));
implementations.set(module, verbs);
}
/** Every verb of every seat a served module claims, where the mesh issued it. */
const wanted = (): SeatVerb[] => {
const out: SeatVerb[] = [];
const have = new Set<string>();
const add = (v: SeatVerb): void => {
if (have.has(v.subject)) return;
have.add(v.subject);
out.push(v);
};
for (const module of served) {
const m = typeof broker.membership === "function" ? broker.membership(module) : undefined;
for (const s of m?.seats ?? []) add({ seat: s.seat, verb: s.verb, subject: s.subject, holder: module });
// The credential's claims, for the module's own runtime, ONLY until the mesh issues a
// membership (novox/hq issue 218). The credential names what the module claims; the membership
// names what it holds on this machine. A seat held once for the mesh is claimed by every
// machine running the module and held by one, so once a membership exists it decides: a claim
// it leaves out is not held here, and serving it anyway announced the seat from a machine the
// bus then refused it on.
if (module !== self || m) continue;
for (const claim of credential?.claims ?? []) {
for (const verb of claim.serves ?? []) {
const subject = seatToolSubject(claim.seat, verb, claim.scope, credential?.node);
add({ seat: claim.seat, verb, subject, holder: module });
}
}
}
return out;
};
let stops: (() => void)[] = [];
let servingNow: ServedSeatVerb[] = [];
const serve = async (): Promise<void> => {
stops.forEach((s) => s());
stops = [];
servingNow = [];
for (const v of wanted()) {
const run = implementations.get(v.seat)?.get(v.verb);
const tool = definitions.get(v.seat)?.get(v.verb);
if (!run) {
console.log(`[mesh-tools] ${v.holder} claims ${v.seat} and implements no ${v.verb}, which that seat promises; not served`);
continue;
}
stops.push(await broker.handleSubject(v.subject, run));
if (tool) servingNow.push({ ...v, tool });
console.log(`[mesh-tools] serving ${v.seat}'s ${v.verb} on ${v.subject}, admitted where ${v.holder} holds the seat`);
}
};
await serve();
// A membership issued to any served module may add, move or withdraw a seat's verbs.
if (typeof broker.onMembership === "function") broker.onMembership(() => void serve());
return { stop: () => stops.forEach((s) => s()), serving: () => servingNow };
}
-69
View File
@@ -1,69 +0,0 @@
/**
* What a runtime serves, it announces (novox/hq ADR 0197): the TypeScript runtime the per-module
* containers still run answers the NATS services protocol's discovery in the same shape as the Go
* tool runtime — its module's tools on every subject issued, and the seat verbs it serves.
*
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/announce.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { fileURLToPath } from "node:url";
import { connect, StringCodec } from "nats";
import { resetTools } from "@novox/mesh-sdk/tools";
import { connectNats, membershipSubject } from "../dist/broker-nats.js";
import { runTools } from "../dist/runtime.js";
const url = process.env.MESH_TEST_NATS;
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
const sc = StringCodec();
test("the runtime answers $SRV.INFO with what it serves, in the services protocol's format", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const nc = await connect({ servers: url });
const jsm = await nc.jetstreamManager();
try {
await jsm.streams.delete("ASSIGNMENTS");
} catch {
// none yet
}
await jsm.streams.add({ name: "ASSIGNMENTS", subjects: ["mesh.assignment.>"], max_msgs_per_subject: 1, allow_direct: true } as never);
await nc.jetstream().publish(membershipSubject("anchor", "shop"), sc.encode(JSON.stringify({
node: "anchor", module: "shop",
serves: [{ subject: "mesh.mod.shop.tool.{tool}.anchor" }, { subject: "mesh.mod.shop.tool.{tool}", queue: "serve.shop" }],
emits: "mesh.mod.shop.event.{event}", tools: "mesh.mod.shop.tool.tools",
})));
const shop = await connectNats({ url, node: "anchor", module: "shop" });
let stop = () => {};
try {
stop = await runTools({ broker: shop, moduleEntrypoints: [fixture("shop-tools.mjs")] });
const msg = await nc.request("$SRV.INFO.shop.anchor", sc.encode(""), { timeout: 2000 });
const info = JSON.parse(sc.decode(msg.data)) as {
type: string; name: string; id: string; version: string;
endpoints: { name: string; subject: string; queue_group: string; metadata: Record<string, string> }[];
};
assert.equal(info.type, "io.nats.micro.v1.info_response");
assert.equal(info.name, "shop");
assert.equal(info.id, "anchor");
assert.ok(info.version);
const price = info.endpoints.filter((e) => e.metadata.tool === "price").map((e) => `${e.subject}|${e.queue_group}`).sort();
assert.deepEqual(price, ["mesh.mod.shop.tool.price.anchor|", "mesh.mod.shop.tool.price|serve.shop"]);
const one = info.endpoints.find((e) => e.metadata.tool === "price")!;
assert.equal(one.metadata.kind, "tool");
assert.equal(one.metadata.module, "shop");
assert.equal(one.metadata.node, "anchor");
assert.equal(one.metadata.interchangeable, "true");
assert.ok(JSON.parse(one.metadata.schema).type === "object");
// Ping answers with the same identity; another service's request is not answered.
const ping = JSON.parse(sc.decode((await nc.request("$SRV.PING", sc.encode(""), { timeout: 2000 })).data));
assert.equal(ping.type, "io.nats.micro.v1.ping_response");
await assert.rejects(nc.request("$SRV.INFO.somebody-else", sc.encode(""), { timeout: 300 }));
} finally {
stop();
await shop.close();
await nc.close();
resetTools();
}
});
-188
View File
@@ -1,188 +0,0 @@
/**
* A person's client, against a real bus.
*
* What is worth checking is not that a request/reply works — the runtime's own tests cover that — but
* that the two surfaces are the same thing. An agent and a person must see the same tools and get the
* same answers, or the MCP surface becomes a second definition of what a tool is.
*
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/client.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
// The built output, not the source: the client imports its siblings as `.js`, which is what ships and
// what every other file here does, and cannot be loaded as TypeScript directly. `pretest` builds.
import { connectNats } from "../dist/broker-nats.js";
import { callTool, seatsIn, toolKey, toolsOn, whyItFailed } from "../dist/client.js";
const url = process.env.MESH_TEST_NATS;
/** A mesh as discovery sees it (design 34 §3): the catalogue holding three modules, two of them up
* and answering `tools`, one held and not running. Two connections, because a person and a module
* are different users even in a test. */
async function aMeshWithTools() {
const catalogue = await connectNats({ url: url!, module: "mesh-catalog" });
const shop = await connectNats({ url: url!, module: "shop" });
await catalogue.handle("catalog_modules", async () => ({
modules: [{ module: "shop" }, { module: "mesh-catalog" }, { module: "ghost" }],
}));
await catalogue.handle("tools", async () => ({
module: "mesh-catalog",
tools: [{ name: "catalog_modules", description: "what modules the mesh has", input: {} }],
}));
await shop.handle("tools", async () => ({
module: "shop",
tools: [{ name: "price", description: "what something costs", input: { type: "object" } }],
}));
await shop.handle("price", async (body: { of?: string }) => ({ of: body.of ?? "nothing", cost: 12 }));
// And the mesh's own records, served by the holder of the mesh-controller seat (ADR 0154): one
// role's tool, and the verb that lists every role's.
const controller = await connectNats({ url: url!, module: "mesh-controller" });
await controller.handle("seat:mesh-controller.tools", async () => ({
seats: [
{ seat: "mesh-controller", scope: "mesh", tools: [{ name: "status", description: "what is wrong", input: {} }] },
{ seat: "node-dns-resolver", scope: "node", tools: [{ name: "lookup", description: "one machine's", input: {} }] },
],
}));
await controller.handle("seat:mesh-controller.status", async () => ({ output: "all quiet", ok: true }));
return {
async close() {
await catalogue.close();
await shop.close();
await controller.close();
},
};
}
test("a person sees what the running modules answer, sorted, and who did not answer", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const mesh = await aMeshWithTools();
const person = await connectNats({ url, module: "person.ada" });
try {
const began = Date.now();
const have = await toolsOn(person);
assert.deepEqual(
have.tools.map((x) => `${x.module}.${x.name}`),
["mesh-catalog.catalog_modules", "mesh-controller.status", "node-dns-resolver.lookup", "shop.price"],
"the list is what the modules answered plus every role's tools, in a stable order",
);
// A role's tool is marked as one; a node-scoped seat's carries its scope, so a caller names
// the machine and the verb resolves as the seat's (design 33 §4, ADR 0170).
assert.ok(have.tools.find((x) => x.module === "mesh-controller")!.seat);
const lookup = have.tools.find((x) => x.module === "node-dns-resolver")!;
assert.ok(lookup.seat && lookup.scope === "node");
assert.equal(toolKey("node-dns-resolver.lookup", seatsIn(have)), "seat:node-dns-resolver.lookup");
// Silence is named, never dropped: a module the catalogue holds and nothing answered for.
assert.deepEqual(have.notAnswering, ["ghost"]);
// And at once: a module that is not running costs nothing, or the list is unusable.
assert.ok(Date.now() - began < 5_000, "an absent module waited out the timeout");
} finally {
await person.close();
await mesh.close();
}
});
test("a person calls a tool and gets the module's own answer, unshaped", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const mesh = await aMeshWithTools();
const person = await connectNats({ url, module: "person.ada" });
try {
const answer = await callTool(person, "shop.price", { of: "a hat" });
assert.deepEqual(answer.result, { of: "a hat", cost: 12 });
} finally {
await person.close();
await mesh.close();
}
});
test("a tool nobody serves says so at once, and says what to do about it", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const person = await connectNats({ url: url!, module: "person.ada" });
try {
const began = Date.now();
await assert.rejects(() => callTool(person, "ghost.missing", {}));
// At once, not after the whole wait: "that module is down" and "that tool is slow" need
// different things done, and a timeout cannot tell them apart.
assert.ok(Date.now() - began < 5_000, "a tool nobody serves waited out the timeout");
} finally {
await person.close();
}
});
test("a name that is not <module>.<tool> is refused before anything is sent", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const person = await connectNats({ url: url!, module: "person.ada" });
try {
await assert.rejects(() => callTool(person, "price", {}), /does not name a tool/);
} finally {
await person.close();
}
});
test("each way a call fails says what to do about it", () => {
// The three answers a person actually gets. Without this they are one timeout and a stack trace,
// and the remedies are in three different places.
assert.match(whyItFailed("shop.price", new Error("no responders")), /nothing serves shop\.price/);
assert.match(
whyItFailed("shop.price", new Error("Permissions Violation for Publish")),
/may not call shop\.price/,
);
assert.match(whyItFailed("shop.price", new Error("timeout")), /did not answer in time/);
assert.match(whyItFailed("shop.price", new Error("something else")), /something else/);
});
test("a role's tool is reached through the seat, and a module's own name is never shadowed", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const { seatsIn, toolKey } = await import("../dist/client.js");
const mesh = await aMeshWithTools();
const person = await connectNats({ url, module: "person.ada" });
try {
const roles = seatsIn(await toolsOn(person));
assert.equal(toolKey("mesh-controller.status", roles), "seat:mesh-controller.status");
assert.equal(toolKey("mesh-controller.other", roles), "mesh-controller.other", "a verb the seat does not declare is a module's");
assert.equal(toolKey("shop.price", roles), "shop.price");
const answer = await callTool(person, "mesh-controller.status", {}, roles);
assert.deepEqual(answer.result, { output: "all quiet", ok: true });
const direct = await callTool(person, "seat:mesh-controller.status", {});
assert.deepEqual(direct.result, { output: "all quiet", ok: true });
} finally {
await person.close();
await mesh.close();
}
});
// A module on two machines (novox/hq ADR 0159): a call names the machine and reaches that instance
// and no other; a call that names none reaches one of them and says which; a seat's verb is served
// by the claimant's tool of the same name on the seat's own subject.
test("a tool call names the machine it is for, and every answer says which machine answered", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const onAnchor = await connectNats({ url: url!, module: "store", node: "anchor" });
const onHome = await connectNats({ url: url!, module: "store", node: "home-server" });
const asker = await connectNats({ url: url!, module: "console", node: "workstation" });
try {
await onAnchor.handle("databases", async () => ({ at: "anchor" }));
await onHome.handle("databases", async () => ({ at: "home-server" }));
const home = await callTool(asker, "store.databases@home-server", {});
assert.deepEqual(home.result, { at: "home-server" });
assert.equal(home.node, "home-server");
const anchor = await callTool(asker, "store.databases@anchor", {});
assert.deepEqual(anchor.result, { at: "anchor" });
assert.equal(anchor.node, "anchor");
const whichever = await callTool(asker, "store.databases", {});
assert.ok(["anchor", "home-server"].includes(whichever.node ?? ""), `an unnamed call still says who answered: ${whichever.node}`);
assert.deepEqual(whichever.result, { at: whichever.node });
// A seat's verb, served by the claimant on the seat's subject; a mesh seat's is flat.
await onAnchor.handleSubject("mesh.seat.mesh-store.tool.databases", async () => ({ seat: "mesh-store", at: "anchor" }));
const viaSeat = await callTool(asker, "seat:mesh-store.databases", {});
assert.deepEqual(viaSeat.result, { seat: "mesh-store", at: "anchor" });
assert.equal(viaSeat.node, "anchor");
} finally {
await onAnchor.close();
await onHome.close();
await asker.close();
}
});
-54
View File
@@ -1,54 +0,0 @@
// The TypeScript implementation, held to the shared fixtures (novox/hq ADR 0074, design 19).
//
// Run against a NATS server, because the question is what actually reaches the wire:
//
// docker run -d --rm --name c -p 14222:4222 nats:2.10-alpine -js
// node test/conformance.mjs
//
// **The runner lives with the implementation it exercises; the fixture does not.** It is read
// from the sdk's conformance directory by sibling path — the same file the Go suite reads. A
// fixture copied into each implementation is two fixtures, and two fixtures drift, which is the
// failure the suite exists to prevent.
import { readFileSync } from "node:fs";
import { connect } from "nats";
const clientPath = process.argv[2] ?? "../dist/broker-nats.js";
const { connectNats } = await import(clientPath);
const f = JSON.parse(readFileSync(new URL("../../mesh-sdk/conformance/events/module-event.json", import.meta.url)));
const URL_ = process.env.MESH_TEST_NATS ?? "nats://127.0.0.1:14222";
let failed = 0;
const check = (ok, what) => { console.log(` ${ok ? "ok " : "FAIL"} ${what}`); if (!ok) failed++; };
const admin = await connect({ servers: URL_ });
const jsm = await admin.jetstreamManager();
await jsm.streams.add({ name: "EVENTS", subjects: ["mesh.mod.*.event.>", "mesh.seat.*.event.>"] });
const shop = await connectNats({ url: URL_, node: f.given.node, module: f.given.module });
await shop.publish({ key: f.given.key, node: f.given.node, body: f.given.body, headers: f.given.headers });
// What actually landed, read back from the stream rather than from the client that wrote it.
const msg = await jsm.streams.getMessage("EVENTS", { last_by_subj: f.wire.subject });
check(!!msg, `it lands on ${f.wire.subject}`);
if (msg) {
const got = {};
if (msg.header) for (const k of msg.header.keys()) got[k] = msg.header.get(k);
for (const h of f.wire.requiredHeaders) {
check(got[h] !== undefined && got[h] !== "", `${h} is set`);
}
check(got["content-type"] === f.wire.headerFormats["content-type"], "content-type is as pinned");
check(new RegExp(f.wire.headerFormats["x-event-id"]).test(got["x-event-id"]), "x-event-id is as pinned");
check(!Number.isNaN(Date.parse(got["x-time"])), "x-time parses as a date");
check(got["x-source"] === f.given.module, "x-source agrees with the subject's module");
const payload = JSON.parse(new TextDecoder().decode(msg.data));
check(payload.envelope === undefined && payload.key === undefined,
"the payload is the body alone, not the envelope (the fixture refuses nesting)");
check(JSON.stringify(payload) === JSON.stringify(f.given.body),
"the body round-trips — semantic, not byte-exact, per the README");
}
await shop.close(); await admin.close();
console.log(failed ? `\n${failed} failed` : "\nall passed");
process.exit(failed ? 1 : 0);
+3 -2
View File
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
# A tools bundle that says who the runtime said called it (novox/hq issue 365): MCP over stdio, no SDK. Its tool
# and its seat verb answer the caller the call's `_meta` named — "" when it named none — and the arguments as given.
# and its seat verb answer the caller the call's `_meta` named — "" when it named none —, the id of the call in flight
# it named (novox/hq ADR 0597 §7), and the arguments as given.
import json, sys
def say(m):
@@ -18,7 +19,7 @@ for line in sys.stdin:
say({"id": rid, "result": {"tools": TOOLS}})
elif m == "tools/call":
meta = p.get("_meta") or {}
said = {"caller": meta.get("mesh/caller", ""), "arguments": p.get("arguments") or {}}
said = {"caller": meta.get("mesh/caller", ""), "call": meta.get("mesh/call", ""), "arguments": p.get("arguments") or {}}
say({"id": rid, "result": {"content": [{"type": "text", "text": json.dumps(said)}]}})
elif rid is not None:
say({"id": rid, "result": {}})
-113
View File
@@ -1,113 +0,0 @@
/**
* The console: the same surface over HTTP on loopback, started the way the mesh starts it — on the
* module credential in MESH_BROKER_FILE (novox/hq ADR 0152, design 34 §2).
*
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/http.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { spawn, type ChildProcess } from "node:child_process";
import { mkdtemp, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { connectNats } from "../dist/broker-nats.js";
import { serveMcpHttp } from "../dist/http.js";
const url = process.env.MESH_TEST_NATS;
async function aMesh(t: { after: (fn: () => Promise<void> | void) => void }) {
const catalogue = await connectNats({ url: url!, module: "mesh-catalog" });
const shop = await connectNats({ url: url!, module: "shop" });
await catalogue.handle("catalog_modules", async () => ({ modules: [{ module: "shop" }] }));
await shop.handle("tools", async () => ({
module: "shop",
tools: [{ name: "price", description: "what something costs", input: {} }],
}));
await shop.handle("price", async (body: { of?: string }) => ({ of: body.of ?? "nothing", cost: 12 }));
t.after(async () => {
await catalogue.close();
await shop.close();
});
}
/** `mesh serve` as the mesh runs it: MESH_BROKER_FILE, a listen address, nothing else. */
async function aConsole(t: { after: (fn: () => Promise<void> | void) => void }): Promise<string> {
const dir = await mkdtemp("/tmp/mesh-console-");
const credential = join(dir, "broker");
await writeFile(credential, JSON.stringify({ url, node: "desk", module: "mesh-console", user: "desk.mesh-console", password: "x" }));
const child: ChildProcess = spawn(process.execPath, ["dist/mesh.js", "serve", "--listen", "127.0.0.1:0"], {
env: { ...process.env, MESH_BROKER_FILE: credential, MESH_CREDENTIAL: "" },
stdio: ["ignore", "pipe", "pipe"],
});
t.after(() => {
child.kill("SIGTERM");
});
return new Promise((resolve, reject) => {
let out = "";
let err = "";
child.stdout!.on("data", (d) => {
out += d.toString();
const m = /listening on (http:\/\/[^/]+\/mcp)/.exec(out);
if (m) resolve(m[1]!);
});
child.stderr!.on("data", (d) => (err += d.toString()));
child.on("exit", (code) => reject(new Error(`serve exited ${code}: ${err}`)));
});
}
async function post(endpoint: string, body: unknown): Promise<{ status: number; json?: any }> {
const res = await fetch(endpoint, {
method: "POST",
headers: { "content-type": "application/json", accept: "application/json" },
body: JSON.stringify(body),
});
const text = await res.text();
return { status: res.status, json: text ? JSON.parse(text) : undefined };
}
test("the console answers a host on loopback, as the account the mesh gave it", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
await aMesh(t);
const endpoint = await aConsole(t);
const hello = await post(endpoint, { jsonrpc: "2.0", id: 1, method: "initialize", params: {} });
assert.equal(hello.status, 200);
assert.match(hello.json.result.instructions, /desk\.mesh-console/, "the handshake names the console's account");
const heard = await post(endpoint, { jsonrpc: "2.0", method: "notifications/initialized" });
assert.equal(heard.status, 202, "a notification is heard and not answered");
const listed = await post(endpoint, { jsonrpc: "2.0", id: 2, method: "tools/list" });
assert.deepEqual(listed.json.result.tools.map((x: { name: string }) => x.name), ["shop.price"]);
const called = await post(endpoint, {
jsonrpc: "2.0", id: 3, method: "tools/call", params: { name: "shop.price", arguments: { of: "a hat" } },
});
assert.deepEqual(JSON.parse(called.json.result.content[0].text), { of: "a hat", cost: 12 });
// A person's client through the same endpoint, with no credential of its own.
const { main } = await import("../dist/mesh.js");
const logged: string[] = [];
const was = console.log;
console.log = (line: string) => logged.push(String(line));
try {
assert.equal(await main(["tools", "--console", endpoint]), 0);
} finally {
console.log = was;
}
assert.ok(logged.some((l) => l.startsWith("shop.price")), `the client did not list through the console: ${logged}`);
});
test("the console binds loopback and nowhere else", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const bus = await connectNats({ url, module: "mesh-console", node: "desk" });
try {
await assert.rejects(() => serveMcpHttp(bus, "desk.mesh-console", "0.0.0.0:0"), /loopback and nowhere else/);
const up = await serveMcpHttp(bus, "desk.mesh-console", "127.0.0.1:0");
assert.match(up.address, /^127\.0\.0\.1:\d+$/);
await up.close();
} finally {
await bus.close();
}
});
-196
View File
@@ -1,196 +0,0 @@
/**
* The MCP surface, driven the way a host drives it.
*
* **The claim worth checking is that it is the same thing the command line is.** An agent and a
* person must see the same tools and get the same answers, or this becomes a second definition of what
* a tool is — which is exactly what a thin adapter is supposed to avoid.
*
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/mcp.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { spawn } from "node:child_process";
import { connectNats } from "../dist/broker-nats.js";
const url = process.env.MESH_TEST_NATS;
/** A module answering the catalogue's list and one tool, plus a credential file the client reads. */
async function aMeshAndACredential(t: { after: (fn: () => Promise<void> | void) => void }) {
const catalogue = await connectNats({ url: url!, module: "mesh-catalog" });
const shop = await connectNats({ url: url!, module: "shop" });
await catalogue.handle("catalog_modules", async () => ({
modules: [{ module: "shop" }, { module: "ghost" }],
}));
await shop.handle("tools", async () => ({
module: "shop",
tools: [{ name: "price", description: "what something costs", input: { of: { type: "string" } } }],
}));
await shop.handle("price", async (body: { of?: string }) => ({ of: body.of ?? "nothing", cost: 12 }));
const controller = await connectNats({ url: url!, module: "mesh-controller" });
await controller.handle("seat:mesh-controller.tools", async () => ({
seats: [
{ seat: "mesh-controller", scope: "mesh", tools: [
{ name: "status", description: "what is wrong", input: {} },
{ name: "push", description: "tell a machine", input: { node: { type: "string" } } },
] },
// A seat held once per machine (design 33 §4, ADR 0170): its verb is asked of one.
{ seat: "node-dns-resolver", scope: "node", tools: [{ name: "lookup", description: "one machine's", input: {} }] },
],
}));
await controller.handle("seat:node-dns-resolver.lookup@anchor", async () => ({ machine: "anchor", answered: true }));
await controller.handle("seat:mesh-controller.status", async () => ({ output: "all quiet", ok: true }));
await controller.handle("seat:mesh-controller.push", async (body: { node?: string }) => ({ told: body.node ?? "nobody" }));
t.after(async () => {
await catalogue.close();
await shop.close();
await controller.close();
});
const { mkdtemp, writeFile } = await import("node:fs/promises");
const { join } = await import("node:path");
const dir = await mkdtemp("/tmp/mesh-client-");
const path = join(dir, "credential.json");
await writeFile(
path,
JSON.stringify({ url, user: "person.ada", password: "x", person: "ada", invokes: ["shop.price"] }),
);
return path;
}
/** Drive `mesh mcp` over stdio and collect the replies, as a host would. */
function driving(credential: string, requests: unknown[]): Promise<Record<string, any>[]> {
return new Promise((resolve, reject) => {
const child = spawn(process.execPath, ["dist/mesh.js", "mcp", "--credential", credential], {
stdio: ["pipe", "pipe", "pipe"],
});
let out = "";
let err = "";
child.stdout.on("data", (d) => (out += d.toString()));
child.stderr.on("data", (d) => (err += d.toString()));
child.on("error", reject);
child.on("close", () => {
const replies = out
.split("\n")
.filter((l) => l.trim() !== "")
.map((l) => JSON.parse(l) as Record<string, any>);
if (replies.length === 0 && err !== "") reject(new Error(err));
else resolve(replies);
});
for (const r of requests) child.stdin.write(`${JSON.stringify(r)}\n`);
child.stdin.end();
});
}
test("a host initialises, lists the mesh's tools and calls one", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const credential = await aMeshAndACredential(t);
const replies = await driving(credential, [
{ jsonrpc: "2.0", id: 1, method: "initialize", params: {} },
{ jsonrpc: "2.0", method: "notifications/initialized" },
{ jsonrpc: "2.0", id: 2, method: "tools/list" },
{ jsonrpc: "2.0", id: 3, method: "tools/call", params: { name: "shop.price", arguments: { of: "a hat" } } },
]);
const byId = new Map(replies.map((r) => [r.id, r]));
// A notification is answered with nothing, or a host waiting on ids sees a reply it cannot match.
assert.equal(replies.length, 3, `expected three replies, got ${JSON.stringify(replies)}`);
const hello = byId.get(1)!.result;
assert.equal(hello.protocolVersion, "2025-03-26");
assert.ok(hello.capabilities.tools, "a server offering no tools is not this one");
assert.match(hello.instructions, /ada/, "the handshake says whose authority a call is made under");
const listed = byId.get(2)!.result.tools;
assert.deepEqual(listed.map((x: { name: string }) => x.name),
["mesh-controller.push", "mesh-controller.status", "node-dns-resolver.lookup", "shop.price"],
"the modules' tools and the roles', named the way a person names them");
// A node-scoped seat's verb takes the machine, and requires it (ADR 0170).
const lookup = listed[2];
assert.equal(lookup.inputSchema.properties.node.type, "string");
assert.deepEqual(lookup.inputSchema.required, ["node"]);
const price = listed[3];
assert.ok(price.inputSchema, "a tool with no schema is one an agent cannot call");
// A module's bare property map arrives as a schema an agent can read, its words kept — and
// `node`, the machine to ask when the module runs on several (novox/hq ADR 0159), beside them.
assert.deepEqual(price.inputSchema.properties.of, { type: "string" });
assert.equal(price.inputSchema.properties.node.type, "string", "a module's tool takes the machine to ask");
assert.ok(!listed[1].inputSchema.properties?.node, "a seat's verb takes no machine; the seat's scope decides");
assert.equal(listed[0].inputSchema.properties?.node?.type, "string", "a seat's verb that takes a node of its own keeps it");
// Silence is named: the module the catalogue holds and nothing answered for.
assert.deepEqual(byId.get(2)!.result._meta.notAnswering, ["ghost"]);
const called = byId.get(3)!.result;
assert.ok(!called.isError, `the call failed: ${JSON.stringify(called)}`);
// The module's own answer, unshaped. An adapter that summarised it would be deciding what matters
// in somebody else's answer.
assert.deepEqual(JSON.parse(called.content[0].text), { of: "a hat", cost: 12 });
});
test("a tool nobody serves comes back as an error the agent can act on", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const credential = await aMeshAndACredential(t);
const replies = await driving(credential, [
{ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "ghost.missing", arguments: {} } },
]);
const result = replies[0].result;
// isError, not a protocol failure: the call was well-formed and the mesh answered it — with an
// absence. A JSON-RPC error would tell the agent its request was malformed, which it was not.
assert.ok(result?.isError, `expected a tool error, got ${JSON.stringify(replies[0])}`);
assert.match(result.content[0].text, /nothing serves ghost\.missing/);
});
test("a method this surface does not have is refused, and a notification is not", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const credential = await aMeshAndACredential(t);
const replies = await driving(credential, [
{ jsonrpc: "2.0", id: 1, method: "resources/list" },
{ jsonrpc: "2.0", method: "notifications/cancelled" },
]);
assert.equal(replies.length, 1, "a notification was answered");
assert.equal(replies[0].error.code, -32601);
assert.match(replies[0].error.message, /resources\/list/);
});
// A seat's verb that takes a machine as its own argument — `push <node>` — keeps it: the console
// moves `node` into the subject for a module's tool only (ADR 0159), never for a role's verb.
test("a seat's verb keeps a node of its own; only a module's tool gives it to the subject", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const credential = await aMeshAndACredential(t);
const replies = await driving(credential, [
{ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "mesh-controller.push", arguments: { node: "anchor" } } },
]);
const result = replies[0].result;
assert.ok(!result.isError, JSON.stringify(replies[0]));
assert.deepEqual(JSON.parse(result.content[0].text), { told: "anchor" });
});
test("a host calls the mesh's own verb through the seat", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const credential = await aMeshAndACredential(t);
const replies = await driving(credential, [
{ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "mesh-controller.status", arguments: {} } },
]);
const result = replies[0].result;
assert.ok(!result.isError, JSON.stringify(replies[0]));
assert.deepEqual(JSON.parse(result.content[0].text), { output: "all quiet", ok: true });
});
test("a node-scoped seat's verb is asked of the machine named, and refused without one", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const credential = await aMeshAndACredential(t);
const replies = await driving(credential, [
{ jsonrpc: "2.0", id: 1, method: "initialize", params: {} },
{ jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "node-dns-resolver.lookup", arguments: { node: "anchor" } } },
{ jsonrpc: "2.0", id: 3, method: "tools/call", params: { name: "node-dns-resolver.lookup", arguments: {} } },
]);
const byId = new Map(replies.map((r) => [r.id, r]));
const answered = byId.get(2)!.result;
assert.ok(!answered.isError, JSON.stringify(answered));
assert.match(answered.content[0].text, /"machine": "anchor"/, "the machine's holder answered");
assert.match(byId.get(3)!.error?.message ?? JSON.stringify(byId.get(3)), /name the machine/);
});
-255
View File
@@ -1,255 +0,0 @@
/**
* The mesh issues an assignment's subjects, and a runtime serves what it is issued (novox/hq ADR
* 0160). A runtime derives one address for itself — `mesh.assignment.<node>.<module>` — reads the
* membership there, serves exactly what it says, and re-serves when a new one arrives. Against a
* real bus with JetStream, because the membership is a direct get on a stream.
*
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/membership.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { fileURLToPath } from "node:url";
import { connect, StringCodec } from "nats";
import { resetTools } from "@novox/mesh-sdk/tools";
import { connectNats, membershipSubject } from "../dist/broker-nats.js";
import { callTool, subjectListed } from "../dist/client.js";
import { runTools } from "../dist/runtime.js";
const url = process.env.MESH_TEST_NATS;
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
const sc = StringCodec();
/** The ASSIGNMENTS stream as the controller asserts it: last-per-subject, readable by direct get. */
async function anAssignmentsStream() {
const nc = await connect({ servers: url! });
const jsm = await nc.jetstreamManager();
try {
await jsm.streams.delete("ASSIGNMENTS");
} catch {
// none yet
}
await jsm.streams.add({
name: "ASSIGNMENTS",
subjects: ["mesh.assignment.>"],
max_msgs_per_subject: 1,
allow_direct: true,
} as never);
return {
async issue(m: object & { node: string; module: string }) {
await nc.jetstream().publish(membershipSubject(m.node, m.module), sc.encode(JSON.stringify(m)));
},
async close() {
await nc.close();
},
};
}
test("a runtime serves exactly the subjects it is issued, and the listing carries them", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const stream = await anAssignmentsStream();
// The mesh placed shop on two machines that are not interchangeable: each is served by name only.
await stream.issue({
node: "anchor",
module: "shop",
serves: [{ subject: "mesh.mod.shop.tool.{tool}.anchor" }],
emits: "mesh.mod.shop.event.{event}",
tools: "mesh.mod.shop.tool.tools",
});
const shop = await connectNats({ url, node: "anchor", module: "shop" });
const asker = await connectNats({ url, module: "console", node: "workstation" });
let stop = () => {};
try {
stop = await runTools({ broker: shop, moduleEntrypoints: [fixture("shop-tools.mjs")] });
assert.equal(shop.membership()?.module, "shop", "the runtime read what the mesh issued");
// By name it answers; the plain subject was not issued, so nothing serves it.
const named = await callTool(asker, "shop.price@anchor", { of: "a hat" });
assert.deepEqual(named.result, { of: "a hat", cost: 12 });
assert.equal(named.node, "anchor");
await assert.rejects(callTool(asker, "shop.price", {}), /no responders|503/i, "a subject not issued is not served");
// The tools answer says where each is served, and the client composes nothing.
const tools = await asker.request<Record<string, never>, { tools: { name: string; subjects?: string[] }[] }>(
"shop.tools",
{},
);
assert.deepEqual(tools.tools.find((x) => x.name === "price")?.subjects, ["mesh.mod.shop.tool.price.anchor"]);
const listing = { tools: [{ module: "shop", name: "price", subjects: ["mesh.mod.shop.tool.price.anchor"] }], notAnswering: [] };
assert.equal(subjectListed("shop.price", "anchor", listing as never), "mesh.mod.shop.tool.price.anchor");
assert.equal(subjectListed("shop.price", "elsewhere", listing as never), undefined);
// The mesh re-issues the membership with the plain subject too (the modules became
// interchangeable); the runtime re-serves on it without a restart.
await stream.issue({
node: "anchor",
module: "shop",
serves: [{ subject: "mesh.mod.shop.tool.{tool}", queue: "serve.shop" }, { subject: "mesh.mod.shop.tool.{tool}.anchor" }],
emits: "mesh.mod.shop.event.{event}",
tools: "mesh.mod.shop.tool.tools",
});
await new Promise((r) => setTimeout(r, 300));
const plain = await callTool(asker, "shop.price", { of: "a coat" });
assert.deepEqual(plain.result, { of: "a coat", cost: 12 });
assert.equal(plain.node, "anchor");
} finally {
stop();
await asker.close();
await shop.close();
await stream.close();
}
});
test("a seat's verbs are implemented under the seat's name, served where issued, never listed as the module's", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const stream = await anAssignmentsStream();
await stream.issue({
node: "anchor",
module: "postgres",
serves: [{ subject: "mesh.mod.postgres.tool.{tool}", queue: "serve.postgres" }, { subject: "mesh.mod.postgres.tool.{tool}.anchor" }],
seats: [{ seat: "mesh-store", verb: "databases", subject: "mesh.seat.mesh-store.tool.databases" }],
emits: "mesh.mod.postgres.event.{event}",
tools: "mesh.mod.postgres.tool.tools",
});
const credential = { url, node: "anchor", module: "postgres", claims: [{ seat: "mesh-store", scope: "mesh", serves: ["databases", "query"] }] };
const pg = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
let stop = () => {};
try {
stop = await runTools({ broker: pg, credential, moduleEntrypoints: [fixture("store-seat.mjs")] });
// The module's own `databases` and the seat's are two tools: postgres's on its subject, the
// store's on the seat's, each answering as itself.
const own = await callTool(asker, "postgres.databases", {});
assert.deepEqual(own.result, { software: "postgres" });
const seat = await callTool(asker, "seat:mesh-store.databases", {});
assert.deepEqual(seat.result, { seat: "mesh-store" });
assert.equal(seat.node, "anchor");
// What the seat does not promise — creating a database — is postgres's tool and not the store's.
await assert.rejects(callTool(asker, "seat:mesh-store.postgres_create_database", {}), /no responders|503/i);
// And the module's `tools` lists only postgres's own, never the seat's implementation.
const tools = await asker.request<Record<string, never>, { module: string; tools: { name: string }[] }>("postgres.tools", {});
assert.deepEqual(tools.tools.map((x) => x.name).sort(), ["databases", "postgres_create_database"]);
await assert.rejects(asker.request("mesh-store.tools", {}), /no responders|503/i, "a seat is not a module with a tools verb");
} finally {
stop();
await asker.close();
await pg.close();
await stream.close();
}
});
// novox/hq issue 218: the store seat is claimed by every machine running postgres and held by one.
// Where the mesh issued a membership without the seat, the claim in the credential serves nothing:
// the module's own tools answer, the seat's verbs do not, and the runtime does not announce them.
test("a claimant the membership does not make the holder serves none of the seat's verbs", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const stream = await anAssignmentsStream();
await stream.issue({
node: "elsewhere",
module: "postgres",
serves: [{ subject: "mesh.mod.postgres.tool.{tool}.elsewhere" }],
emits: "mesh.mod.postgres.event.{event}",
tools: "mesh.mod.postgres.tool.tools",
});
const credential = { url, node: "elsewhere", module: "postgres", claims: [{ seat: "mesh-store", scope: "mesh", serves: ["databases", "query"] }] };
const pg = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
let stop = () => {};
try {
stop = await runTools({ broker: pg, credential, moduleEntrypoints: [fixture("store-claimant.mjs")] });
assert.deepEqual((await callTool(asker, "postgres.databases@elsewhere", {})).result, { software: "postgres" });
await assert.rejects(callTool(asker, "seat:mesh-store.databases", {}), /no responders|503/i, "the store is not held here");
} finally {
stop();
await asker.close();
await pg.close();
await stream.close();
}
});
test("a module named like its seat registers once, and answers as the module and as the seat", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const stream = await anAssignmentsStream();
await stream.issue({
node: "anchor",
module: "shop",
serves: [{ subject: "mesh.mod.shop.tool.{tool}", queue: "serve.shop" }, { subject: "mesh.mod.shop.tool.{tool}.anchor" }],
seats: [{ seat: "shop", verb: "price", subject: "mesh.seat.shop.tool.price" }],
emits: "mesh.mod.shop.event.{event}",
tools: "mesh.mod.shop.tool.tools",
});
const credential = { url, node: "anchor", module: "shop", claims: [{ seat: "shop", scope: "mesh", serves: ["price"] }] };
const shop = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
let stop = () => {};
try {
stop = await runTools({ broker: shop, credential, moduleEntrypoints: [fixture("shop-seat.mjs")] });
assert.deepEqual((await callTool(asker, "shop.price", { of: "a hat" })).result, { of: "a hat", cost: 12 });
assert.deepEqual((await callTool(asker, "seat:shop.price", { of: "a hat" })).result, { of: "a hat", cost: 12 });
const tools = await asker.request<Record<string, never>, { tools: { name: string }[] }>("shop.tools", {});
assert.deepEqual(tools.tools.map((x) => x.name), ["price", "refund"]);
} finally {
stop();
await asker.close();
await shop.close();
await stream.close();
}
});
test("a mesh that has issued nothing yet gets the derived shape, and says so", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
const stream = await anAssignmentsStream();
const said: string[] = [];
const log = console.log;
console.log = (...a: unknown[]) => said.push(a.join(" "));
let lone;
try {
lone = await connectNats({ url, node: "home-server", module: "lone" });
} finally {
console.log = log;
}
const asker = await connectNats({ url, module: "console", node: "workstation" });
try {
assert.equal(lone.membership(), undefined);
assert.ok(said.some((s) => /no membership issued for lone on home-server/.test(s)), said.join("\n"));
await lone.handle("ping", async () => ({ pong: true }));
assert.deepEqual((await callTool(asker, "lone.ping", {})).result, { pong: true });
assert.deepEqual((await callTool(asker, "lone.ping@home-server", {})).result, { pong: true });
} finally {
await asker.close();
await lone.close();
await stream.close();
}
});
// A module that implements a seat its credential does not (yet) claim is not served for it and does
// not fall over either: the runtime says so and serves the module's own tools.
test("a registration under a seat the credential does not claim is said and skipped, not fatal", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const stream = await anAssignmentsStream();
const credential = { url, node: "anchor", module: "postgres" };
const pg = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
const said: string[] = [];
const log = console.log;
console.log = (...a: unknown[]) => said.push(a.join(" "));
let stop = () => {};
try {
stop = await runTools({ broker: pg, credential, moduleEntrypoints: [fixture("store-seat-unclaimed.mjs")] });
console.log = log;
assert.ok(said.some((s) => /registers tools under "mesh-store".*not served/.test(s)), said.join("\n"));
assert.deepEqual((await callTool(asker, "postgres.databases", {})).result, { software: "postgres" });
await assert.rejects(callTool(asker, "seat:mesh-store.databases", {}), /no responders|503/i);
} finally {
console.log = log;
stop();
await asker.close();
await pg.close();
await stream.close();
}
});
-350
View File
@@ -1,350 +0,0 @@
/**
* One runtime per node serves every assigned module's tools (novox/hq ADR 0175, to-be 38 WP1). The
* node's runtime is handed a list of modules and their bundles on the node's credential; it reads
* one membership per module, serves each module's tools on that module's subjects and each held
* seat's verbs on the seat's, names a bundle that fails to load without dropping the others, and
* re-serves a module whose membership is re-issued mid-run. Against a real bus with JetStream.
*
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/node-runtime.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { fileURLToPath } from "node:url";
import { cpSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { connect, StringCodec } from "nats";
import { resetTools } from "@novox/mesh-sdk/tools";
import { connectNats, membershipSubject } from "../dist/broker-nats.js";
import { callTool, toolsOn } from "../dist/client.js";
import { servedModulesFrom } from "../dist/main.js";
import { runTools, takeToolEnvs } from "../dist/runtime.js";
const url = process.env.MESH_TEST_NATS;
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
const sc = StringCodec();
/** The controller's job, done by hand: the ASSIGNMENTS stream (last-per-subject, direct get) and an
* EVENTS stream for what a tool emits. */
async function aMesh() {
const nc = await connect({ servers: url! });
const jsm = await nc.jetstreamManager();
for (const name of ["ASSIGNMENTS", "EVENTS"]) {
try {
await jsm.streams.delete(name);
} catch {
// none yet
}
}
await jsm.streams.add({ name: "ASSIGNMENTS", subjects: ["mesh.assignment.>"], max_msgs_per_subject: 1, allow_direct: true } as never);
await jsm.streams.add({ name: "EVENTS", subjects: ["mesh.mod.*.event.>"] });
return {
async issue(m: object & { node: string; module: string }) {
await nc.jetstream().publish(membershipSubject(m.node, m.module), sc.encode(JSON.stringify(m)));
},
/** The next subject an event lands on, under a pattern. */
nextEvent(pattern: string): Promise<string> {
const sub = nc.subscribe(pattern, { max: 1 });
return (async () => {
for await (const m of sub) return m.subject;
throw new Error("no event");
})();
},
async close() {
await nc.close();
},
};
}
/** A membership as the controller issues one on a machine, with the module's own subject when it
* answers for the module anywhere, and the verbs of the node seats it holds. */
function membershipOf(module: string, node: string, opts: { plain?: boolean; seats?: Record<string, string[]> } = {}) {
const own = `mesh.mod.${module}`;
const serves: { subject: string; queue?: string }[] = [{ subject: `${own}.tool.{tool}.${node}` }];
if (opts.plain) serves.push({ subject: `${own}.tool.{tool}`, queue: `serve.${module}` });
const seats = Object.entries(opts.seats ?? {}).flatMap(([seat, verbs]) =>
verbs.map((verb) => ({ seat, verb, subject: `mesh.seat.${seat}.tool.${verb}.${node}` })));
return { node, module, serves, seats, emits: `${own}.event.{event}`, tools: `${own}.tool.tools` };
}
/** Wait for something to be served: the bus answers "no responders" at once until it is. */
async function until<T>(attempt: () => Promise<T>, tries = 50): Promise<T> {
for (let i = 0; ; i++) {
try {
return await attempt();
} catch (e) {
if (i >= tries) throw e;
await new Promise((r) => setTimeout(r, 100));
}
}
}
test("MESH_TOOL_MODULES names modules and their entrypoints; a bare path is the credential's own module's", () => {
const have = servedModulesFrom(" alpha=/a/tools/index.js, beta=/b/one.js ,beta=/b/two.js, /mine/index.js ,node-tools=/own/x.js", "node-tools");
assert.deepEqual(have.serves, [
{ module: "alpha", entrypoints: ["/a/tools/index.js"] },
{ module: "beta", entrypoints: ["/b/one.js", "/b/two.js"] },
]);
// The credential's own module, named or bare, is the one-module form either way.
assert.deepEqual(have.moduleEntrypoints, ["/mine/index.js", "/own/x.js"]);
assert.throws(() => servedModulesFrom("=/nothing.js", undefined), /neither <module>=<entrypoint>/);
assert.deepEqual(servedModulesFrom("", "x"), { serves: [], moduleEntrypoints: [] });
});
test("the node's runtime serves five modules' bundles on one credential — two of them launched, one broken — and follows a re-issued membership", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const mesh = await aMesh();
// Three modules assigned to the machine: alpha answers for itself anywhere, beta only here and
// holds the node-shelf seat, gamma's bundle is broken.
await mesh.issue(membershipOf("alpha", "anchor", { plain: true }));
await mesh.issue(membershipOf("beta", "anchor", { seats: { "node-shelf": ["list", "clear"] } }));
await mesh.issue(membershipOf("gamma", "anchor"));
// Two more, launched rather than loaded (ADR 0188): delta is Python and holds the node-lamp seat;
// epsilon is TypeScript written against the protocol and marked executable.
await mesh.issue(membershipOf("delta", "anchor", { seats: { "node-lamp": ["on"] } }));
await mesh.issue(membershipOf("epsilon", "anchor"));
// The node's credential: the runtime module's name, no claims (seats come from the memberships).
const credential = { url, node: "anchor", module: "node-tools" };
const nodeTools = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
const said: string[] = [];
const log = console.log;
console.log = (...a: unknown[]) => said.push(a.join(" "));
let stop = () => {};
try {
process.env.MESH_OPERATOR_ACCOUNT = "somebody";
process.env.MESH_OPERATOR_HOME = "/home/somebody";
stop = await runTools({
broker: nodeTools,
credential,
serves: [
{ module: "alpha", entrypoints: [fixture("many-alpha.serve.mjs")] },
{ module: "beta", entrypoints: [fixture("many-beta.serve.mjs")] },
{ module: "gamma", entrypoints: [fixture("many-broken.serve.mjs")] },
{ module: "delta", entrypoints: [fixture("many-delta.py")] },
{ module: "epsilon", entrypoints: [fixture("many-epsilon.mjs")] },
],
});
console.log = log;
assert.deepEqual(nodeTools.serving().sort(), ["alpha", "beta", "delta", "epsilon", "gamma", "node-tools"]);
assert.ok(said.some((s) => /the operator's account here is somebody \(home \/home\/somebody\)/.test(s)), said.join("\n"));
assert.ok(said.some((s) => /gamma's bundle .*many-broken\.serve\.mjs failed to load: gamma's bundle exited \(1\): Error: gamma's bundle cannot find its client; its tools are not served here/.test(s)), said.join("\n"));
assert.ok(said.some((s) => /serving 8 tool\(s\) for 5 module\(s\): alpha\.one, alpha\.two, beta\.three, beta\.four, beta\.five, delta\.greet, delta\.die, epsilon\.seven; not serving gamma/.test(s)), said.join("\n"));
// Five tools answer, each where its module's membership says: alpha anywhere and here, beta here only.
assert.deepEqual((await callTool(asker, "alpha.one", {})).result, { alpha: 1 });
assert.deepEqual((await callTool(asker, "alpha.one@anchor", {})).result, { alpha: 1 });
assert.deepEqual((await callTool(asker, "beta.three@anchor", {})).result, { beta: 3 });
assert.deepEqual((await callTool(asker, "beta.four@anchor", {})).result, { beta: 4 });
assert.deepEqual((await callTool(asker, "beta.five@anchor", {})).result, { beta: 5 });
await assert.rejects(callTool(asker, "beta.three", {}), /no responders|503/i, "beta was not issued the module's plain subject");
// Two seat verbs answer on the seat's subjects, held by beta.
assert.deepEqual((await callTool(asker, "seat:node-shelf.list@anchor", {})).result, { shelf: ["a", "b"] });
assert.deepEqual((await callTool(asker, "seat:node-shelf.clear@anchor", {})).result, { cleared: true });
// A bundle in another language answers the same way, its seat verb among them; so does a
// TypeScript bundle served through the protocol rather than imported.
assert.deepEqual((await callTool(asker, "delta.greet@anchor", { who: "mesh" })).result, { greeting: "hello mesh", language: "python" });
assert.deepEqual((await callTool(asker, "seat:node-lamp.on@anchor", {})).result, { on: true, language: "python" });
assert.deepEqual((await callTool(asker, "epsilon.seven@anchor", {})).result, { epsilon: 7, via: "stdio" });
// A child that exits mid-call tells the caller so and is started again on the next call.
await assert.rejects(callTool(asker, "delta.die@anchor", {}), /delta's bundle exited \(3\)/);
assert.deepEqual((await callTool(asker, "delta.greet@anchor", {})).result, { greeting: "hello world", language: "python" });
// A tool that emits does so as its module, not as the runtime.
const landed = mesh.nextEvent("mesh.mod.*.event.>");
assert.deepEqual((await callTool(asker, "alpha.two", {})).result, { alpha: 2 });
assert.equal(await landed, "mesh.mod.alpha.event.happened");
// `tools` answers for each: what alpha and beta serve, and why gamma serves nothing.
const gamma = await asker.request<Record<string, never>, { module: string; tools: unknown[]; failed?: string }>("gamma.tools@anchor", {});
assert.deepEqual(gamma, { module: "gamma", tools: [], failed: "gamma's bundle exited (1): Error: gamma's bundle cannot find its client" });
const beta = await asker.request<Record<string, never>, { tools: { name: string; subjects?: string[] }[] }>("beta.tools@anchor", {});
assert.deepEqual(beta.tools.map((x) => x.name), ["three", "four", "five"]);
assert.deepEqual(beta.tools[0]!.subjects, ["mesh.mod.beta.tool.three.anchor"]);
// And discovery says so, with the reason, beside the modules that answered.
const catalogue = await connectNats({ url, module: "mesh-catalog" });
await catalogue.handle("catalog_modules", async () => ({ modules: [{ module: "alpha" }, { module: "beta" }, { module: "gamma" }, { module: "delta" }, { module: "epsilon" }] }));
try {
const have = await toolsOn(asker);
assert.deepEqual(have.tools.map((x) => `${x.module}.${x.name}`), ["alpha.one", "alpha.two", "beta.five", "beta.four", "beta.three", "delta.die", "delta.greet", "epsilon.seven"]);
assert.deepEqual(have.notAnswering, ["gamma (its tools bundle failed to load: gamma's bundle exited (1): Error: gamma's bundle cannot find its client)", "mesh-controller (seat)"]);
} finally {
await catalogue.close();
}
// The mesh re-issues beta's membership mid-run — now answering for the module anywhere — and
// the runtime serves the new subject without a restart.
await mesh.issue(membershipOf("beta", "anchor", { plain: true, seats: { "node-shelf": ["list", "clear"] } }));
assert.deepEqual((await until(() => callTool(asker, "beta.three", {}))).result, { beta: 3 });
assert.deepEqual((await callTool(asker, "seat:node-shelf.list@anchor", {})).result, { shelf: ["a", "b"] });
} finally {
console.log = log;
delete process.env.MESH_OPERATOR_ACCOUNT;
delete process.env.MESH_OPERATOR_HOME;
stop();
await asker.close();
await nodeTools.close();
await mesh.close();
resetTools();
}
});
test("a bundle carrying its own copy of the SDK registers into the runtime's registry, and its tools are served (issue 209)", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
let dir = "";
let stop = () => {};
const closing: Array<() => Promise<void>> = [];
const said: string[] = [];
const log = console.log;
try {
// A bundle as the toolchain packs one: its compiled entrypoint, a package.json saying ES modules,
// and its dependencies copied in — the SDK among them, a second copy beside the runtime's own,
// and a dependency of the bundle's own that the runtime does not carry.
dir = mkdtempSync(join(tmpdir(), "mesh-bundle-"));
const sdk = realpathSync(fileURLToPath(new URL("../node_modules/@novox/mesh-sdk/", import.meta.url)));
cpSync(sdk, join(dir, "node_modules", "@novox", "mesh-sdk"), { recursive: true });
mkdirSync(join(dir, "node_modules", "zeta-flavour"), { recursive: true });
writeFileSync(join(dir, "node_modules", "zeta-flavour", "package.json"), '{"name":"zeta-flavour","type":"module","main":"index.js"}\n');
writeFileSync(join(dir, "node_modules", "zeta-flavour", "index.js"), 'export const flavour = "the bundle\'s own";\n');
writeFileSync(join(dir, "package.json"), '{"type":"module","private":true}\n');
writeFileSync(join(dir, "index.serve.mjs"),
'#!/usr/bin/env node\nimport { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";\nawait import("./index.js");\nawait serveRegisteredOverStdio();\n', { mode: 0o755 });
writeFileSync(join(dir, "index.js"),
'import { registerModuleTools } from "@novox/mesh-sdk/tools";\n' +
'import { flavour } from "zeta-flavour";\n' +
'registerModuleTools("zeta", () => [{ name: "probe", description: "answers", input: {}, run: async () => ({ zeta: true, flavour }) }]);\n');
const mesh = await aMesh();
closing.push(() => mesh.close());
await mesh.issue(membershipOf("zeta", "anchor"));
const credential = { url, node: "anchor", module: "node-tools" };
const nodeTools = await connectNats(credential);
closing.push(() => nodeTools.close());
const asker = await connectNats({ url, module: "console", node: "workstation" });
closing.push(() => asker.close());
console.log = (...a: unknown[]) => said.push(a.join(" "));
stop = await runTools({ broker: nodeTools, credential, serves: [{ module: "zeta", entrypoints: [join(dir, "index.serve.mjs")] }] });
console.log = log;
assert.ok(said.some((s) => /serving 1 tool\(s\) for 1 module\(s\): zeta\.probe/.test(s)), said.join("\n"));
// The SDK is the runtime's (the registration arrived); the bundle's other dependency is its own.
assert.deepEqual((await callTool(asker, "zeta.probe@anchor", {})).result, { zeta: true, flavour: "the bundle's own" });
} finally {
console.log = log;
stop();
for (const close of closing.reverse()) await close();
if (dir) rmSync(dir, { recursive: true, force: true });
resetTools();
}
});
test("each bundle is given its own environment and none of another's, imported or launched (ADR 0192)", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const mesh = await aMesh();
for (const m of ["gamma", "delta", "zeta"]) await mesh.issue(membershipOf(m, "anchor"));
const credential = { url, node: "anchor", module: "node-tools" };
const nodeTools = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
const log = console.log;
let stop = () => {};
const before = process.env.MESH_TOOL_ENV;
try {
process.env.MESH_OPERATOR_ACCOUNT = "somebody";
process.env.MESH_TOOL_ENV = JSON.stringify({
gamma: { GAMMA_CONFIG_FILE: "/var/lib/mesh/gamma/config.json" },
delta: { DELTA_TOKEN_FILE: "/var/lib/mesh/delta/token" },
zeta: { ZETA_URL: "http://127.0.0.1:3000" },
});
const envs = takeToolEnvs();
assert.equal(process.env.MESH_TOOL_ENV, undefined, "the composed environments were left in the process's");
console.log = () => {};
stop = await runTools({
broker: nodeTools, credential, envs,
serves: [
{ module: "gamma", entrypoints: [fixture("env-gamma.serve.mjs")] },
{ module: "delta", entrypoints: [fixture("env-delta.serve.mjs")] },
{ module: "zeta", entrypoints: [fixture("env-zeta.mjs")] },
],
});
console.log = log;
assert.deepEqual((await callTool(asker, "gamma.given@anchor", {})).result,
{ mine: "/var/lib/mesh/gamma/config.json", theirs: null, runtime: "somebody", composed: null });
assert.deepEqual((await callTool(asker, "delta.given@anchor", {})).result,
{ mine: "/var/lib/mesh/delta/token", theirs: null });
assert.deepEqual((await callTool(asker, "zeta.given@anchor", {})).result,
{ mine: "http://127.0.0.1:3000", theirs: null, composed: null });
} finally {
console.log = log;
if (before === undefined) delete process.env.MESH_TOOL_ENV; else process.env.MESH_TOOL_ENV = before;
delete process.env.MESH_OPERATOR_ACCOUNT;
stop();
await asker.close();
await nodeTools.close();
await mesh.close();
resetTools();
}
});
test("a launched bundle is told the module it serves, so its seat's verbs stay the seat's (ADR 0193)", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const mesh = await aMesh();
await mesh.issue(membershipOf("theta", "anchor", { seats: { "node-shelf": ["list"] } }));
const credential = { url, node: "anchor", module: "node-tools" };
const nodeTools = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
const log = console.log;
let stop = () => {};
try {
console.log = () => {};
stop = await runTools({ broker: nodeTools, credential, envs: new Map([["theta", { THETA_WORD: "given" }]]),
serves: [{ module: "theta", entrypoints: [fixture("served-seat-first.mjs")] }] });
console.log = log;
assert.deepEqual((await callTool(asker, "theta.own@anchor", {})).result, { theta: "given" });
assert.deepEqual((await callTool(asker, "seat:node-shelf.list@anchor", {})).result, { shelf: ["x"] });
} finally {
console.log = log;
stop();
await asker.close();
await nodeTools.close();
await mesh.close();
resetTools();
}
});
test("an entrypoint that is not executable is refused by name, and the others serve (ADR 0193)", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const mesh = await aMesh();
for (const m of ["alpha", "plain"]) await mesh.issue(membershipOf(m, "anchor"));
const credential = { url, node: "anchor", module: "node-tools" };
const nodeTools = await connectNats(credential);
const asker = await connectNats({ url, module: "console", node: "workstation" });
const said: string[] = [];
const log = console.log;
let stop = () => {};
try {
console.log = (...a: unknown[]) => said.push(a.join(" "));
stop = await runTools({ broker: nodeTools, credential, serves: [
{ module: "alpha", entrypoints: [fixture("many-alpha.serve.mjs")] },
{ module: "plain", entrypoints: [fixture("many-alpha.mjs")] },
] });
console.log = log;
assert.ok(said.some((s) => /plain's bundle .*many-alpha\.mjs failed to load: .* is not executable; a bundle the runtime serves is started, never imported/.test(s)), said.join("\n"));
assert.deepEqual((await callTool(asker, "alpha.one@anchor", {})).result, { alpha: 1 });
} finally {
console.log = log;
stop();
await asker.close();
await nodeTools.close();
await mesh.close();
resetTools();
}
});
-63
View File
@@ -1,63 +0,0 @@
/**
* The runtime as the node-tools module (novox/hq ADR 0175 §6, to-be 38 WP3): started the way the
* host starts it — `main.js` with the node's credential and MESH_TOOL_MODULES — it serves the bundles
* AND answers MCP on loopback as the console, through which a tool it serves can be called.
*
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/node-tools-serve.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { spawn, type ChildProcess } from "node:child_process";
import { mkdtemp, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { connectNats } from "../dist/broker-nats.js";
const url = process.env.MESH_TEST_NATS;
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
async function post(endpoint: string, body: unknown): Promise<any> {
const res = await fetch(endpoint, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body) });
return res.json();
}
test("as node-tools, serve loads the bundles and is the console on loopback", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
// The catalogue, for discovery; the node credential names the runtime module and no claims.
const catalogue = await connectNats({ url, module: "mesh-catalog" });
await catalogue.handle("catalog_modules", async () => ({ modules: [{ module: "alpha" }] }));
t.after(() => catalogue.close());
const dir = await mkdtemp("/tmp/node-tools-");
const credential = join(dir, "broker");
await writeFile(credential, JSON.stringify({ url, node: "desk", module: "node-tools", user: "desk.node-tools", password: "x" }));
const child: ChildProcess = spawn(process.execPath, ["dist/main.js"], {
env: {
...process.env,
MESH_BROKER_FILE: credential,
MESH_TOOL_MODULES: `alpha=${fixture("many-alpha.serve.mjs")}`,
MESH_CONSOLE_LISTEN: "127.0.0.1:0",
},
stdio: ["ignore", "pipe", "pipe"],
});
t.after(() => {
child.kill("SIGTERM");
});
const endpoint = await new Promise<string>((resolve, reject) => {
let out = "";
let err = "";
child.stdout!.on("data", (d) => {
out += d.toString();
const m = /listening on (http:\/\/[^/]+\/mcp) as desk\.node-tools/.exec(out);
if (m) resolve(m[1]!);
});
child.stderr!.on("data", (d) => (err += d.toString()));
child.on("exit", (code) => reject(new Error(`serve exited ${code}: ${err}`)));
});
const listed = await post(endpoint, { jsonrpc: "2.0", id: 1, method: "tools/list" });
assert.deepEqual(listed.result.tools.map((x: any) => x.name).filter((n: string) => n.startsWith("alpha.")), ["alpha.one", "alpha.two"]);
// A tool the same process serves on the bus, called through the console it also is.
const called = await post(endpoint, { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "alpha.one", arguments: { node: "desk" } } });
assert.deepEqual(JSON.parse(called.result.content[0].text), { alpha: 1 });
});
-80
View File
@@ -1,80 +0,0 @@
import { spawn } from "node:child_process";
import { test } from "node:test";
import assert from "node:assert/strict";
import { fatalBrokerReason, PinMismatchError, topicMatches } from "../src/broker-nats.ts";
// novox/hq issue 058 (and its review): serve mode retries a broker that is not up yet, but must
// give up at once on a failure waiting cannot fix — otherwise a permanent fault loops for ever
// disguised as "not reachable". This is the classifier that draws the line; the bed cannot test it
// (it starts the consumer only after the broker is up), so it is proven here.
test("a broker that is not up yet is retryable, not fatal", () => {
for (const err of [
Object.assign(new Error("connect ECONNREFUSED 10.42.0.1:5671"), { code: "ECONNREFUSED" }),
Object.assign(new Error("connect ETIMEDOUT"), { code: "ETIMEDOUT" }),
Object.assign(new Error("getaddrinfo EAI_AGAIN anchor.internal"), { code: "EAI_AGAIN" }),
new Error("timed out fetching the broker's certificate"),
]) {
assert.equal(fatalBrokerReason(err), null, `should retry: ${(err as Error).message}`);
}
});
test("a certificate that does not match the pin is fatal, by type not by message", () => {
// Typed, so rewording the message cannot turn an impostor back into an infinite retry.
assert.notEqual(fatalBrokerReason(new PinMismatchError("anything at all")), null);
// A plain Error with pin-ish words is NOT treated as the pin case — only the type is.
assert.equal(fatalBrokerReason(new Error("the pinned value was fine")), null);
});
test("a malformed broker URL is fatal — it never parses on the next try", () => {
assert.notEqual(fatalBrokerReason(Object.assign(new Error("Invalid URL"), { code: "ERR_INVALID_URL" })), null);
assert.notEqual(fatalBrokerReason(new Error("Invalid URL: not-a-url")), null);
});
test("a refused login is fatal — a wrong or revoked credential, not an absent broker", () => {
// The bus refuses a login in its own words; each is final, because the next try says the same.
for (const msg of ["Authorization Violation", "nats: user authentication expired", "Permissions Violation for Subscription to \"x\""]) {
assert.notEqual(fatalBrokerReason(new Error(msg)), null, `should be fatal: ${msg}`);
}
});
test("a non-Error value does not crash the classifier", () => {
assert.equal(fatalBrokerReason("just a string"), null);
assert.equal(fatalBrokerReason(undefined), null);
});
// **A module asked to prepare its state and naming nothing is a failure, not a no-op** (novox/hq
// ADR 0135). The mesh asks this only of a module whose manifest says it prepares something, so an
// image that names nothing was built wrong, and exiting 0 would let that version serve against a
// state nobody shaped.
test("preparing with nothing named fails rather than passing quietly", async () => {
const runtime = new URL("../dist/main.js", import.meta.url).pathname;
const ran = await new Promise<{ code: number | null; said: string }>((resolve) => {
const child = spawn(process.execPath, [runtime, "prepare"], {
env: { ...process.env, MESH_PREPARE: "" },
});
let said = "";
child.stderr.on("data", (chunk) => (said += String(chunk)));
child.on("close", (code) => resolve({ code, said }));
});
assert.notEqual(ran.code, 0, "a module that prepares nothing exited 0, so its version would serve");
assert.match(ran.said, /MESH_PREPARE/);
});
// **One durable consumer feeds one reader, however many patterns a module registers.**
//
// A module has exactly one consumer, so two readers of it would each take half the messages — and a
// reader that received one its own pattern does not match acknowledges it, which is right for a
// filter wider than anything registered and silent loss when it is another handler's. The matching is
// therefore pure and tested as such: what a message is for is decided by the patterns registered, not
// by which reader happened to fetch it.
test("a message is for every pattern that matches it, and nothing else", () => {
const registered = ["mesh-build-machine.built", "mesh-controller.built-before"];
const matched = (key: string) => registered.filter((p) => topicMatches(p, key));
assert.deepEqual(matched("mesh-build-machine.built"), ["mesh-build-machine.built"]);
assert.deepEqual(matched("mesh-controller.built-before"), ["mesh-controller.built-before"]);
// Nothing registered for it: the consumer's filter is the controller's and may be wider.
assert.deepEqual(matched("mesh-catalog.upgraded"), []);
// And a handler that asked for everything gets both, which is what the audit logger does.
assert.deepEqual(["#"].filter((p) => topicMatches(p, "mesh-controller.built-before")), ["#"]);
});
-68
View File
@@ -1,68 +0,0 @@
/**
* A refused announcement is said and never fatal (novox/hq 04-ISSUES/217): against a bus whose
* permissions refuse one discovery subject, the runtime's raw subscription is refused, logged, and
* the process keeps serving — its tools still answer.
*
* docker run -d --rm --name t -p 14233:4222 -v $PWD/test/fixtures/refusing-nats.conf:/c.conf nats:2.10-alpine -c /c.conf
* MESH_TEST_REFUSING_NATS=nats://127.0.0.1:14233 node --test --experimental-strip-types test/refused.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { connectNats } from "../dist/broker-nats.js";
import { callTool } from "../dist/client.js";
const url = process.env.MESH_TEST_REFUSING_NATS;
test("a refused discovery subscription is logged and the runtime serves on", async (t) => {
if (!url) return t.skip("MESH_TEST_REFUSING_NATS unset");
const bus = await connectNats({ url, user: "runtime", password: "runtime", module: "alpha", node: "anchor" });
const said: string[] = [];
const log = console.log;
console.log = (...a: unknown[]) => said.push(a.join(" "));
const crashed: unknown[] = [];
const onRejection = (e: unknown) => crashed.push(e);
process.on("unhandledRejection", onRejection);
try {
(bus as unknown as { raw: (s: string, f: () => Uint8Array | undefined) => () => void }).raw("$SRV.PING.>", () => undefined);
const stop = await bus.handle("alpha.ping", async () => ({ pong: true }));
for (let i = 0; i < 50 && !said.some((s) => s.includes("the bus refused $SRV.PING.>")); i++) await new Promise((r) => setTimeout(r, 50));
console.log = log;
assert.ok(said.some((s) => /the bus refused \$SRV\.PING\.>.*serves on/.test(s)), said.join("\n"));
assert.equal(crashed.length, 0, `the refusal escaped: ${String(crashed[0])}`);
stop();
} finally {
console.log = log;
process.off("unhandledRejection", onRejection);
await bus.close();
}
});
// novox/hq issue 218: a tool subject the grants leave out — a seat claimed here and held elsewhere —
// is refused, said, and the module's other tools still answer.
test("a refused tool subscription is logged and the module's other tools answer", async (t) => {
if (!url) return t.skip("MESH_TEST_REFUSING_NATS unset");
const bus = await connectNats({ url, user: "runtime", password: "runtime", module: "alpha", node: "anchor" });
const asker = await connectNats({ url, user: "runtime", password: "runtime", module: "console", node: "workstation" });
const said: string[] = [];
const log = console.log;
console.log = (...a: unknown[]) => said.push(a.join(" "));
const crashed: unknown[] = [];
const onRejection = (e: unknown) => crashed.push(e);
process.on("unhandledRejection", onRejection);
try {
const refused = await bus.handleSubject!("mesh.seat.held-elsewhere.tool.databases", async () => ({ seat: true }));
const stop = await bus.handle("alpha.ping", async () => ({ pong: true }));
for (let i = 0; i < 50 && !said.some((s) => s.includes("the bus refused mesh.seat.held-elsewhere")); i++) await new Promise((r) => setTimeout(r, 50));
console.log = log;
assert.ok(said.some((s) => /the bus refused mesh\.seat\.held-elsewhere\.tool\.databases.*serves on/.test(s)), said.join("\n"));
assert.equal(crashed.length, 0, `the refusal escaped: ${String(crashed[0])}`);
assert.deepEqual((await callTool(asker, "alpha.ping@anchor", {})).result, { pong: true });
stop();
refused();
} finally {
console.log = log;
process.off("unhandledRejection", onRejection);
await asker.close();
await bus.close();
}
});
-57
View File
@@ -1,57 +0,0 @@
// Round-trip the runtime's NATS client against a real server: a tool call answered, and an
// event emitted and received with its envelope intact.
import { connect } from "nats";
import { connectNats } from "../dist/broker-nats.js";
const URL = "nats://127.0.0.1:14222";
// The controller's job, done by hand here: the stream and the module's durable consumer.
const admin = await connect({ servers: URL });
const jsm = await admin.jetstreamManager();
await jsm.streams.add({ name: "EVENTS", subjects: ["mesh.mod.*.event.>", "mesh.seat.*.event.>"] });
await jsm.consumers.add("EVENTS", {
durable_name: "one_audit", ack_policy: "explicit",
filter_subjects: ["mesh.mod.shop.event.order.placed"],
});
const shop = await connectNats({ url: URL, node: "one", module: "shop" });
const audit = await connectNats({ url: URL, node: "one", module: "audit" });
let failures = 0;
const check = (ok, what) => { console.log(` ${ok ? "ok " : "FAIL"} ${what}`); if (!ok) failures++; };
// A tool, served and called.
await shop.handle("price", async (body) => ({ total: body.qty * 3 }));
const answer = await audit.request("shop.price", { qty: 4 });
check(answer.total === 12, "a tool call is answered across two connections");
// A handler that throws reaches the caller as an error, not a timeout.
await shop.handle("boom", async () => { throw new Error("no"); });
let threw = null;
try { await audit.request("shop.boom", {}); } catch (e) { threw = e.message; }
check(threw === "no", "a handler that throws answers the caller instead of timing out");
// An event, emitted and received with its envelope intact.
const seen = [];
await audit.subscribe("order.placed", async (env) => { seen.push(env); });
await shop.publish({
key: "order.placed", node: "one", body: { id: "a1" },
headers: { "x-event-id": "e1", "x-node": "one", "content-type": "application/json" },
});
await new Promise((r) => setTimeout(r, 800));
check(seen.length === 1, `exactly one delivery (saw ${seen.length})`);
if (seen[0]) {
check(seen[0].key === "order.placed", "the key survives the subject round trip");
check(seen[0].body?.id === "a1", "the body is the payload, not the whole envelope");
check(seen[0].node === "one", "the node comes back from the headers");
check(seen[0].headers?.["x-event-id"] === "e1", "the event id survives as a header");
}
// A module cannot reach into another's namespace by naming its own event oddly.
await shop.publish({ key: "other", node: "one", body: {}, headers: { "x-event-id": "e2" } });
const msg = await jsm.streams.getMessage("EVENTS", { last_by_subj: "mesh.mod.shop.event.other" });
check(!!msg, "an event lands under the emitting module's own namespace");
await shop.close(); await audit.close(); await admin.close();
console.log(failures ? `\n${failures} failed` : "\nall passed");
process.exit(failures ? 1 : 0);
-60
View File
@@ -1,60 +0,0 @@
/**
* Every module's runtime answers `tools` for it (novox/hq ADR 0152, design 34 §3): the names,
* descriptions and schemas from the code that answers them. Against a real bus, because the claim is
* what a second connection gets back.
*
* docker run -d --rm --name t -p 14232:4222 nats:2.10-alpine -js
* MESH_TEST_NATS=nats://127.0.0.1:14232 node --test --experimental-strip-types test/runtime-tools.test.ts
*/
import assert from "node:assert/strict";
import { test } from "node:test";
import { fileURLToPath } from "node:url";
import { resetTools } from "@novox/mesh-sdk/tools";
import { connectNats } from "../dist/broker-nats.js";
import { runTools } from "../dist/runtime.js";
const url = process.env.MESH_TEST_NATS;
const fixture = (name: string) => fileURLToPath(new URL(`./fixtures/${name}`, import.meta.url));
test("a module registering two tools answers three names, the third being what it serves", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const shop = await connectNats({ url, node: "one", module: "shop" });
const asker = await connectNats({ url, module: "person.ada" });
const stop = await runTools({ broker: shop, moduleEntrypoints: [fixture("shop-tools.mjs")] });
try {
const answer = await asker.request<Record<string, never>, { module: string; tools: { name: string; input: unknown }[] }>(
"shop.tools",
{},
);
assert.equal(answer.module, "shop");
assert.deepEqual(answer.tools.map((x) => x.name), ["price", "refund"]);
// The schema travels with the name: a name alone is not callable by something that has never
// seen the mesh before.
assert.deepEqual(answer.tools[0].input, { of: { type: "string", description: "the thing" } });
// And the tools themselves still answer beside it.
const priced = await asker.request<{ of: string }, { cost: number }>("shop.price", { of: "a hat" });
assert.equal(priced.cost, 12);
} finally {
stop();
await asker.close();
await shop.close();
}
});
test("a module naming a tool of its own `tools` is refused at load", async (t) => {
if (!url) return t.skip("MESH_TEST_NATS unset");
resetTools();
const clash = await connectNats({ url, node: "one", module: "clash" });
try {
await assert.rejects(
() => runTools({ broker: clash, moduleEntrypoints: [fixture("clash-tools.mjs")] }),
/names a tool "tools"/,
);
} finally {
resetTools();
await clash.close();
}
});
-15
View File
@@ -1,15 +0,0 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"declaration": true,
"outDir": "dist",
"rootDir": "src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true
},
"include": ["src/**/*.ts"]
}