Author SHA1 Message Date
jschoubben b1337bbbf2 Pin the group's mesh-sdk head, whose step log goes quiet after one failed publish 2026-10-30 11:10:43 +01:00
jschoubben ade07bfdba Pin the group's mesh-sdk head 2026-10-30 10:58:30 +01:00
jschoubben 07ab8069a4 Pin the group's mesh-sdk head, whose stdio reads a module's own stream through this runtime 2026-10-30 10:54:32 +01:00
jschoubben b90376ba59 Let a module read its own streams back by a batch direct get, as it reads its log, so its tools can answer what the stream holds (a build step's whole failure output) 2026-10-30 10:54:05 +01:00
mesh-tools 95a12105cf Merge pull request 'The bus state guard lets a provider reference under a provision named like a credential pass (L2's seed, issue 1126)' (#47) from fix/1126-state-guard into main 2026-10-28 21:01:24 +00:00
jschoubben 2966c1e32c Runtime state guard: hold a provider reference's node and module to the mesh's name pattern, so no credential rides in one (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-tools@95a12105cf7b
mesh/check being carried to 2966c1e32c3d on 95a12105cf7b | from 27266f99a3c3
2026-10-28 21:49:52 +01:00
jschoubben 1a0424117a Runtime state guard: a credential-shaped name passes only holding an exact provider reference, node and module, so a placement's binding is kept and nothing else gets through (issue 1126) 2026-10-28 21:48:27 +01:00
jschoubben 23f5babe33 Runtime state guard: a name holding an object is data, not a credential, so a placement's binding of the provision secret is kept (issue 1126) 2026-10-28 21:47:06 +01:00
mesh-tools 27266f99a3 Merge pull request 'node-tools starts no bundle again once it is stopping (group fix/shutdown-bound, issue 1126)' (#46) from fix/shutdown-bound into main 2026-10-28 15:05:13 +00:00
jschoubben a9b02bc676 node-tools: keep the stopping-runtime test under a second, and start no first bundle once stopping
mesh/delivery merged: source.main moved to novox/mesh-tools@27266f99a3c3
mesh/delivery-group group fix/shutdown-bound merged: every member is merged
mesh/check pass a9b02bc67604 on 6cc33988801f | unit 2192 run, 3794 kept, 57 skipped | integration 2208 run, 3829 kept, 57 skipped | over bound: unit, …
The test watched two seconds per row; a 10 ms restart pause and a 250 ms window
prove the same. launch.Start itself now refuses while the runtime stops, so a
reload's first start is held too (issue 1126).
2026-10-28 15:40:39 +01:00
jschoubben f016c67a72 node-tools: start no bundle again once the runtime itself is stopping (issue 1126)
The service manager stops node-tools by signalling its whole group, so every
bundle gets SIGTERM with it; the launcher took each exit for a fault and started
the bundle again, for the 15 s the runtime took to stop. On ace's reboot of
2026-10-28 those restarted bundles dialled the system bus once polkit and the bus
had stopped, the bus and polkit came back during the shutdown, nothing stopped
them a second time, and the reboot waited 29 minutes on its job timeout.

ShutDown, called first on SIGTERM, keeps any bundle from starting again, by a
restart or by a call; a running one still answers.
2026-10-28 15:30:12 +01:00
mesh-tools 6cc3398880 Merge pull request 'node-tools waits for its broker credential instead of exiting, and retries a refusal within that bound (issue 1496)' (#45) from fix/1496-start-health into main 2026-10-26 21:38:04 +00:00
jschoubben 8cdb7c3032 The credential wait's test stays under a second, and a failed watch says the re-read, not a restart (issue 1496)
mesh/delivery merged: source.main moved to novox/mesh-tools@6cc33988801f
mesh/delivery-group group fix/1496-start-health merged: every member is merged
mesh/check checking 8cdb7c303210 on a5097dc69ab2 | novox/mesh-tools#45 in group fix/1496-start-health
Review of 3baeaec: TestAwaitCredential took 1.9 s, over the unit limit; it now
sets the watch's settle to 10 ms, writes at 20 ms and gives up at 100 ms (0.2 s).
The watch's journal line for a failure named a bundle and a restart; the
credential wait now says it is read again every 30 s instead.
2026-10-26 22:18:12 +01:00
jschoubben 3baeaece61 A module's runtime waits for its broker credential and its account at a start, bounded, instead of exiting (issue 1496)
The node-engine starts a module's runtime before the provisioner's client fills
its broker file, and the bus carries a new account only after it reloads its
users. The runtime exited on both and was restarted until they were there (ace,
2026-10-26 21:19: mesh-build-node's and mesh-build-image's runtimes exited twice).
It now waits up to five minutes from its start, said once: the file is read again
each time its directory changes (the bundles' inotify watch, a 30 s backstop),
and an authorization refusal is asked again with the backoff. Past the bound it
gives up loudly; a file with no url, a pin mismatch and a bad URL stay final.
2026-10-26 22:06:39 +01:00
mesh-tools a5097dc69a Merge pull request 'mesh_search leaves out filler words, so a question in plain words finds its tool' (#44) from fix/search-ignores-filler into main 2026-10-26 13:39:24 +00:00
jschoubben 9f25e5c9f6 mesh_search: filler words of a question (did, our, on, all) need not match, so the operator's own words find a tool; a query of filler finds nothing
mesh/delivery merged: source.main moved to novox/mesh-tools@a5097dc69ab2
mesh/check pass 9f25e5c9f6c1 on a5097dc69ab2 | carried from bb6a0e3192db | unit 116 run, 11 kept, 11 skipped | integration 118 run, 11 kept, 10 skippe…
2026-10-26 14:03:49 +01:00
mesh-tools bb6a0e3192 Merge pull request 'node-tools: a consumer deleted or made again while no pull waits is noticed at the next rebind, so a module is not deaf for a heartbeat (issue 1945)' (#43) from fix/bus-consumer-remade-flake into main 2026-10-25 15:22:24 +00:00
jschoubben 3331719407 Ask the bus whether a bound consumer still exists, so a deleted and remade one is rebound within a rebind tick, not after nats.go's heartbeat (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-tools@bb6a0e3192db
mesh/check pass 33317194073a on bb6a0e3192db | unit 115 run, 11 kept, 11 skipped | integration 117 run, 11 kept, 10 skipped | over bound: unit (slowes…
nats.go ends a pull loop on a deleted consumer only when a pull is pending at that moment; otherwise the loop waits out its heartbeat. The rebind tick now looks each bound consumer up by name and compares its creation time.
2026-10-25 16:02:27 +01:00
mesh-tools 5d6cee9d84 Merge pull request 'node-tools serves a tools bundle as soon as it is placed, changed or removed, with no restart, and says why a bundle cannot start (issue 1945)' (#42) from fix/r6-reload-bundles into main 2026-10-25 14:56:22 +00:00
jschoubben 2cd5550dc7 A stopped tool drops its membership handler and a stopped bundle starts no copy on a call, so a replaced bundle's old tools stay gone (issue 1945, review round 2)
mesh/delivery merged: source.main moved to novox/mesh-tools@5d6cee9d844e
mesh/check pass 2cd5550dc70f on 77b05089c8df | unit 0 run 126 kept | int 0 run 128 kept | clone 9.5s | composition passed on 4 machines
Its events are refused before the old bundle stops, the bundle watch follows a directory made between
its stat and its watch, and a bundle's reason is read without node's colour codes.
2026-10-25 15:44:10 +01:00
jschoubben d8ac69904d Refuse an event heard during a reload instead of panicking, and judge a bundle by its entrypoint alone (issue 1945, review)
An event a module's consumer heard after the module was dropped for a
reload read a deleted entry and panicked node-tools, taking every tool on
the machine down. The handler now refuses an event of a consumer that is no
longer the module's, and a stopped hearing hands nothing on and offers the
event again at once.

A bundle that writes beside itself (a cache, a socket) loaded itself again
after every quiet spell, because the signature took in every file in its
directory. A delivery puts every file in place anew, so the entrypoint alone
tells a new build. The watch also looks once as it begins, so a bundle that
came between its load and the watch is not missed.

Tests: 1945/AC23 and AC24 in internal/runtime/reload_test.go, and a stopped
hearing in internal/bus/hearing_stopped_test.go.
2026-10-25 15:26:02 +01:00
jschoubben b0ded57294 Serve a tools bundle as soon as it is in place, changed or gone, with no restart; say why a bundle cannot start (issue 1945)
node-tools loaded every bundle once at start, so a bundle delivered after
it started (mesh-files on shanks, 2026-10-25 12:08) went unserved until a
hand restart, and a missing file was said as "not executable". The
runtime now watches the entrypoints' directories with inotify, loads a
module again once a burst of changes settles, and discovery answers with
what is served now. A bundle that cannot start says it does not exist, is
not executable with its mode, or the exec error.

Tests: 1945/AC19-AC22 in internal/runtime/reload_test.go.
2026-10-25 15:09:49 +01:00
jschoubben 77b05089c8 Merge pull request 'Bound a state watch's hand-over by its progress, not its whole set, so the gitea module binds the checker's owed statuses again (issue 1726)' (#41) from fix/1726-group-check-status into main
Reviewed-on: #41
2026-10-24 10:35:43 +00:00
jschoubben 0211844e6a Bound a state watch's hand-over by its progress, not its whole set, so a large state's newest keys are handed over (issue 1726)
mesh/delivery delivering: 0 machine step(s) passed
mesh/check pass 0211844e6ad3 on 0e985f9bf186 | unit 0 run 117 kept | int 0 run 119 kept | clone 5.4s | composition passed on 4 machines
The gitea module's watch of mesh-delivery-checker.owed failed at every attempt
since 03:05 on novox: about 200 owed keys, each a round trip to the bundle,
outlast a 10 s bound for the whole set. The bundle dropped its handler, the
oldest keys were written again every 40 s and the newest, handed over last,
never were: group fix/1937-done-key-blobs's pass never reached mesh/check.
StateWatch now fails only when no current value is handed over within
StateTimeout.
2026-10-24 07:42:08 +02:00
jschoubben 0e985f9bf1 Merge pull request 'R4g (mesh-tools): seat.pull and seat.settle for the build modules, merge-check.sh and the root image module go (design 47)' (#40) from feat/r4g into main
Reviewed-on: #40
2026-10-23 17:51:33 +00:00
jschoubben 860043fd7f Pin mesh-sdk at feat/r4g's head a2c813a6, its test fixture fix, with go.sum tidied
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
mesh/delivery-group group feat/r4g merged: every member is merged
mesh/check checking 860043fd7fa0 on e0fadc43d9dd | novox/mesh-tools#40 in group feat/r4g
2026-10-23 17:06:12 +02:00
jschoubben 589bfc0bc5 Record the checksums of mesh-sdk feat/r4g 9205222a for the group's modules (R4g)
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: g14: novox/mesh-catalog#464: checks/health/go.mod names mesh-sdk a9560890ee26, which is neither on mesh-sdk's trunk nor the group's…
2026-10-23 16:26:11 +02:00
jschoubben 3b5a38e4a4 Pin mesh-sdk at feat/r4g's head 9205222a, the merge of lanes A, B, S8 and F1, so every module r4g touches builds against one SDK
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:22:02 +02:00
jschoubben 755f77ca45 Merge branch 'feat/r4g-asks' into feat/r4g 2026-10-23 16:20:32 +02:00
jschoubben 9153a280f8 Let a bundle pull one message from its listed worker and settle it, so a build module takes an ask only when it has a free slot, in its workers' order (issue 1887 A11, ADR 0525 §4) 2026-10-23 15:28:06 +02:00
jschoubben 8ee6db7d9a Remove the toolchain image module; node-tools is the repository's only module (issue 979, S1; ADR 0524 §5)
Only build-agent built inside that image, and it goes in R4g.
2026-10-23 15:25:33 +02:00
jschoubben 5dffb920ff Delete merge-check.sh; tests that launch bundles skip with their reason (issue 1295, S2; ADR 0529)
The checker runs node-tools' Go level and its npm test; what the script said
was NOT TESTED HERE is a skip naming the missing node or @novox/mesh-sdk.
2026-10-23 15:25:10 +02:00
mesh-tools e0fadc43d9 Merge pull request 'Say "nothing serves" only when the bus finds no subscriber, never for a tool's own error that mentions 503 (issue 1529)' (#39) from fix/1529-nothing-serves-only-for-no-responders into main 2026-10-20 22:15:35 +00:00
jschoubben d295fd4250 Say nothing serves a call only for the bus's own no-responders, never for a tool's error that mentions 503 (issue 1529)
mesh/merge-gate head d295fd4250a7 merged into trunk 18260ce9beca — pass, with a note: builds mesh-tools, node-tools (+40 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head d295fd4250a7 merged into trunk 18260ce9beca — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792534538429799487 (novox/mesh-tools at e0fadc43 into main, d…
2026-10-21 00:02:34 +02:00
mesh-tools 18260ce9be Merge pull request 'The tool runner stops waiting on the removed delivery module's transition (issue 1406)' (#38) from fix/1406-drop-the-old-transition-name into main 2026-10-20 14:36:22 +00:00
mesh-tools 4165ad59b5 Wait on the gate's transition alone: today's module is forgotten (issue 1406)
mesh/merge-gate head 4165ad59b541 merged into trunk f183b86fa0b2 — pass, with a note: builds mesh-tools, node-tools (+40 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head 4165ad59b541 merged into trunk f183b86fa0b2 — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792506983719904657 (novox/mesh-tools at 18260ce9 into main, d…
2026-10-20 12:59:33 +00:00
mesh-tools f183b86fa0 Merge pull request 'The tool runner waits on the gate's transitions too (design 47 R3e, issue 1350)' (#37) from feat/r3e-the-gate into main 2026-10-20 06:49:21 +00:00
mesh-tools 06e6721ed6 Wait on the gate's transitions beside today's delivery module's (issue 1350)
mesh/repo-check head 06e6721ed677 merged into trunk 7412c5b6c9c5 beside mesh-catalog#380@cf5c0273 — pass: its merge-check.sh passed
mesh/merge-gate head 06e6721ed677 merged into trunk 7412c5b6c9c5 — pass: group feat/r3e-the-gate composed: every machine composes with the change as it did…
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792478961597980735 (novox/mesh-tools at f183b86f into main, d…
mesh/delivery-group group feat/r3e-the-gate merged: every member is merged
2026-10-20 00:57:44 +00:00
mesh-tools 7412c5b6c9 Merge pull request 'Feature 57 (config and provisioning): release mesh-tools' (#36) from release/57-config-and-provisioning into main 2026-10-16 12:42:28 +00:00
mesh-tools 33d136e702 Feature 57 (config and provisioning): release mesh-tools (the provider's StateBus, the SDK pin, templates), reviewed
mesh/merge-gate head 33d136e70227 merged into trunk 6b3e15042fee — pass, with a note: builds mesh-tools, node-tools (+40 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head 33d136e70227 merged into trunk 6b3e15042fee — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
2026-10-16 09:41:17 +00:00
mesh-tools 6b3e15042f Merge pull request 'Give a session mesh_wait: wait on the bus for an event instead of polling (issue 745, ADR 0367)' (#35) from 745-mesh-wait into main 2026-10-15 00:52:34 +00:00
mesh-toolsandClaude Opus 5.5 21d038d339 End every wait in flight when the tool runner stops (issue 745)
mesh/merge-gate head 21d038d3397c merged into trunk 5d70eb3e35a9 — pass, with a note: builds mesh-tools, node-tools (+39 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head 21d038d3397c merged into trunk 5d70eb3e35a9 — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
Closing the console shut its server down with a context that never ends, so a stop or restart of
the tool runner waited on any mesh_wait in flight for up to ten minutes. The server's base context
is now cancelled first, ending every wait at once, with a five-second shutdown as a backstop
(second review of issue 745).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:47:59 +00:00
mesh-admin 5d70eb3e35 Merge pull request 'The runtime publishes a module's stream events on its stream and binds the stream consumers its membership names (hq issue 725, ADR 0363)' (#34) from fix-725-runtime-streams into main 2026-10-14 20:13:45 +00:00
jochen d3b0330ed1 Bind a stream consumer again when the bus ends its loop, so a module is not left deaf when the controller deletes and makes again a consumer it cannot narrow (hq issue 725)
mesh/merge-gate head d3b0330ed1cb merged into trunk d0dcae53a8c0 — pass, with a note: builds mesh-tools, node-tools (+39 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head d3b0330ed1cb merged into trunk d0dcae53a8c0 — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792008849869252364 (novox/mesh-tools at 5d70eb3e into main, d…
2026-10-14 22:10:20 +02:00
jochen 9921486fd5 Publish a module's stream events on its stream's subject and bind the stream consumers its membership names, so ADR 0363's own streams work in the runtime (hq issue 725)
mesh/merge-gate head 9921486fd53e merged into trunk d0dcae53a8c0 — pass, with a note: builds mesh-tools, node-tools (+39 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head 9921486fd53e merged into trunk d0dcae53a8c0 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 22:04:41 +02:00
57 changed files with 5088 additions and 488 deletions
-85
View File
@@ -1,85 +0,0 @@
ARG NODE_BASE=node:22-bookworm-slim
# The mesh-tools module: the two images every TypeScript module is COMPILED in and may RUN in. The
# runtime itself ships as the node-tools module's bundle (node-tools/, novox/hq ADR 0175, to-be 38
# WP3); these images are the toolchain for TypeScript bundles and the base a module's own service
# may still be built on. They are no longer how tools reach a node.
#
# **They were the same image, and that was a mistake.** A module's recipe starts from this and
# invokes the compiler out of it, so the compiler had to be here — and because the same image was
# also what every module ran in, every running container on every machine carried a TypeScript
# compiler it would never invoke. The answer is separate stages rather than one image bad at both
# jobs. `build.artifacts` in module.json names the published stage each — `toolchain` and `runtime`.
#
# The `deps` stage is neither published nor named there: it is where the mesh's package registry is
# reached, so it is where — and only where — the credential to reach it exists. The toolchain copies
# resolved node_modules out of it, so the credential is in no image any machine ever holds
# (novox/hq ADR 0076). This is the buildkit-secret's job done without buildkit, because a machine's
# docker may carry no buildx.
# ---- deps: node_modules resolved from the mesh's registry, credential and all ----------------
FROM ${NODE_BASE} AS deps
RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY node-tools/package.json ./
# The builder writes .npmrc into the build context; it authenticates to the mesh's package registry
# for the @novox scope, which is where @novox/mesh-sdk resolves. This stage is not published, so the
# credential travels no further than here. Development dependencies included: the compiler is one.
COPY .npmrc ./.npmrc
RUN npm install --no-audit --no-fund
# **The SDK this image carries is the one the mesh last published** (novox/hq issue 212). The range in
# package.json is resolved once and the layer above is cached, so a release reached no toolchain until
# that file changed. The exact version arrives as a build argument from the SDK module's published
# package (module.json `build.on`), so a new release is a new argument, this layer runs again — and
# the planner orders this module after the SDK, so a release rebuilds the toolchain and, after it,
# every bundle compiled in it (issue 211).
ARG MESH_SDK=@novox/mesh-sdk@latest
RUN npm install --no-audit --no-fund "${MESH_SDK}"
# ---- compiling: the runtime's own code built, WITHOUT the credential -------------------------
FROM ${NODE_BASE} AS compiling
WORKDIR /app
COPY node-tools/package.json ./
# The resolved libraries, but not the .npmrc that resolved them.
COPY --from=deps /app/node_modules ./node_modules
COPY node-tools/tsconfig.json ./
COPY node-tools/src ./src
RUN npm run build
# ---- what a running bundle needs, and nothing else -------------------------------------------
# Its own stage so the toolchain image keeps its build tools while the runtime image does not.
FROM compiling AS lean
RUN npm prune --omit=dev
# ---- toolchain: what a TypeScript bundle is compiled in ---------------------------------------
# The compiler, and esbuild (a development dependency) at /app/node_modules/esbuild: the builder
# bundles every entrypoint and launcher into one file with it (novox/hq ADR 0193), so a bundle
# carries what it imports and not this image's node_modules.
# Beside the compiler, at /app/runtime, what every TypeScript bundle runs with: the production
# dependencies the SDK and the runtime need, and a package.json saying the compiled files are ES
# modules. The builder copies this directory whole into a compiled bundle (novox/hq ADR 0188 §5),
# so a bundle unpacked on a machine starts — a `.js` without that package.json is read as
# CommonJS, and an import of `nats` without node_modules beside it resolves to nothing.
#
# **The client libraries a catalogue module depends on are here too** (novox/hq issue 385): pg, mongodb and
# mssql, development dependencies of node-tools, so the runtime stage prunes them. A catalogue's
# merge-check-typescript.sh fetches nothing and copies this image's node_modules beside a module; a module
# depending on a package that is not here cannot be type-checked or tested, and fails its check.
FROM compiling AS toolchain
COPY --from=lean /app/node_modules /app/runtime/node_modules
RUN printf '{"type":"module","private":true}\n' > /app/runtime/package.json
# **git, because a pull request's merge check runs in this image too** (novox/hq ADR 0237): a repository's
# own merge-check.sh that declares this toolchain runs here, and a suite that reads a repository's history
# (the lab's last-run record) ran into "spawnSync git ENOENT". Nothing compiled here calls it.
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/*
# ---- runtime: what a module's own service may run in ------------------------------------------
FROM ${NODE_BASE} AS runtime
WORKDIR /app
COPY node-tools/package.json ./
COPY --from=lean /app/node_modules ./node_modules
COPY --from=compiling /app/dist ./dist
ENTRYPOINT ["node", "dist/main.js"]
+2 -5
View File
@@ -1,13 +1,11 @@
# mesh-tools
Two modules in one repository (novox/hq ADR 0069), one piece of software:
One module, `node-tools`; the toolchain image module that stood beside it is retired (novox/hq ADR 0524 §5):
- **`node-tools`** (`node-tools/`) — the node's **tool runtime** as a module (ADR 0175, to-be 38
WP3): one process per machine the host runs from this bundle, serving every assigned module's tools
and every held seat's verbs on the bus, and answering MCP on the machine's loopback — the console
(design 34). The code, its tests and the `mesh` client all live there.
- **`mesh-tools`** (this directory) — the two images TypeScript bundles are compiled in and a module's
own *service* may still run in. Built from the same code; no longer how tools reach a node.
The runtime:
@@ -49,8 +47,7 @@ MESH_BROKER_URL a plain URL instead of the credential, for the bootstrap
`node dist/main.js`. On a node the controller composes the variables and the host supervises the
process like any other host-side workload (novox/hq to-be 38). As `node-tools` the same process is
the console: MCP on `127.0.0.1:4270` (or `MESH_CONSOLE_LISTEN`). The container (`Dockerfile`) is how
a module's own *service* may still be built; it is no longer how tools reach a node.
the console: MCP on `127.0.0.1:4270` (or `MESH_CONSOLE_LISTEN`).
## `mesh` — the tools for whoever is on a machine
-55
View File
@@ -1,55 +0,0 @@
#!/bin/sh
# mesh-check-toolchain: go
#
# The tool runtime's own check (novox/hq ADR 0237 as amended): the second layer of a pull request's merge
# check, `mesh/repo-check`, run by the build seat in the mesh's Go toolchain — and by hand.
#
# The gate — every machine of the facts snapshot composed with this change — is the build seat's first
# layer (`mesh/merge-gate`), run because the mesh's module graph builds node-tools and mesh-tools from
# here. This is the code's own: node-tools (Go) formatted, vetted and its suite, every test on a bus of its
# own at the release the mesh runs (internal/meshtest), packages in parallel, under the race detector when
# the toolchain has a C compiler.
#
# **Said, never passed silently**: the runtime's and the console's tests launch TypeScript and Python
# bundles that import the mesh's own package (@novox/mesh-sdk), which comes from the package registry — and
# a check of code nobody has approved is given no credential for it. Where node and that package are not
# present those two packages are not tested here, and the TypeScript in node-tools/src neither.
set -eu
cd node-tools
unformatted=$(gofmt -l cmd internal)
if [ -n "$unformatted" ]; then
echo "not gofmt'd:"
echo "$unformatted"
exit 1
fi
CGO_ENABLED=0 go vet ./...
packages=$(go list ./...)
bundleless=""
if ! command -v node >/dev/null 2>&1 || [ ! -d node_modules/@novox/mesh-sdk ]; then
echo "NOT TESTED HERE: internal/runtime and internal/console launch bundles that need node and @novox/mesh-sdk"
packages=$(printf '%s\n' "$packages" | grep -v -e '/internal/runtime$' -e '/internal/console$')
# What of the console launches no bundle runs anyway: how an address resolves (novox/hq issue 287),
# and how a search ranks (novox/hq ADR 0245), and what discovery heard and how a round that heard nothing is
# answered (novox/hq issue 554).
bundleless='^(TestASeatAndAModuleOfOneNameAreEachReached|TestSearch.*|TestASearchByCommandSaysWhatItReplaces|TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed|TestARuntimeHeardBeforeAndNotNowIsNamed|TestACallAfterARoundThatHeardNothingUsesWhatWasHeardBefore|TestAnAddressKnownOnlyFromAnEarlierAnswerSaysHowOld)$'
# And what of the runtime launches no bundle: how a bundle's seat traffic reaches the bus, what its
# environment holds, what a runtime on a module's own account serves (novox/hq ADR 0259), and that a write
# by compare-and-set loses no update under contention (novox/hq research 042), and a module's log verbs
# (novox/hq ADR 0297), and a key's history (novox/hq ADR 0307).
runtimeless='^(TestALogAnswersEachVerbOnTheBus|TestALogPageIsBoundedByItsBytes|TestAnAppendAtAStaleExpectIsRefusedAsALostUpdate|TestALogNotIssuedIsRefusedWithTheReason|TestWhatALogCannotHoldIsRefusedWithTheReason|TestALogRefusesAValueNamingACredential|TestAKeysHistoryIsEveryValueKeptOldestFirst|TestAKeysHistoryIsRefusedToAModuleNotIssuedTheState|TestARuntimeOnAModulesOwnAccountServesThatModuleAlone|TestNoBusWordReachesABundle|TestAToolCallNeverReachesASeatsEventAcceptOrProof|TestABundlesSeatTrafficGoesThroughItsModulesBus|TestALostCompareAndSetCarriesItsCode|TestTwoWritersIncrementingOneCounterOnTheBusLoseNoUpdate|TestAStaleWriteIsRefusedAndWritesNothing|TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller|TestTheSubjectThatNamesNoCallerStillWorksAndNamesNone)$'
fi
if command -v gcc >/dev/null 2>&1; then
CGO_ENABLED=1 go test -race -count=1 $packages
else
echo "NOT RACE-CHECKED: the toolchain holds no C compiler; the suite runs without the race detector"
CGO_ENABLED=0 go test -count=1 $packages
fi
if [ -n "$bundleless" ]; then
CGO_ENABLED=0 go test -count=1 -run "$bundleless" ./internal/console
fi
if [ -n "${runtimeless:-}" ]; then
CGO_ENABLED=0 go test -count=1 -run "$runtimeless" ./internal/runtime
fi
echo "NOT TESTED HERE: node-tools/src (TypeScript) needs @novox/mesh-sdk from the package registry"
-33
View File
@@ -1,33 +0,0 @@
{
"module": "mesh-tools",
"version": "1",
"slug": "tools",
"build": {
"artifacts": [
{
"name": "build",
"kind": "image",
"from": "Dockerfile",
"target": "toolchain"
},
{
"name": "runtime",
"kind": "image",
"from": "Dockerfile",
"target": "runtime"
}
],
"on": [
{
"arg": "NODE_BASE",
"image": "node@sha256:48e4b67d85f87bd551df43704e24d252f56cc5f8e9718841aace50f19948f0f9"
},
{
"arg": "MESH_SDK",
"module": "mesh-sdk",
"artifact": "lib"
}
]
},
"resources": []
}
+143 -29
View File
@@ -12,7 +12,9 @@
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"log"
"math/rand/v2"
@@ -24,6 +26,8 @@ import (
"github.com/novox/mesh-tools/node-tools/internal/alive"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/console"
"github.com/novox/mesh-tools/node-tools/internal/launch"
"github.com/novox/mesh-tools/node-tools/internal/provide"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
@@ -40,8 +44,9 @@ func main() {
fmt.Fprintf(os.Stderr, "node-tools: %q is not a mode; this runtime serves (novox/hq ADR 0193)\n", os.Args[1])
os.Exit(2)
}
cred := credential()
conn := connectPatiently(cred)
until := time.Now().Add(startWait)
cred := credential(until)
conn := connectPatiently(cred, until)
served, err := runtime.ServedModulesFrom(os.Getenv(runtime.ToolModules), cred.Module)
if err != nil {
@@ -62,6 +67,10 @@ func main() {
stopSaying = alive.Keep(conn, cred.Node, alive.Every, log.Printf)
}
// As the machine's runtime it is also the provider of `mcp-endpoint`: it answers its consumers through the
// SDK's helper wherever it runs, whether or not the module is handed (novox/hq ADR 0349 clause 12, issue 791).
stopProviding := provide.Start(conn, cred.Module, cred.Node, time.Minute, log.Printf)
listen := os.Getenv("MESH_CONSOLE_LISTEN")
if listen == "" && cred.Module == runtimeModule {
listen = consoleListen
@@ -83,7 +92,11 @@ func main() {
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGTERM, syscall.SIGINT)
<-signals
// First of all: the service manager signalled every bundle with this process, and none is started again
// (issue 1126).
launch.ShutDown()
stopSaying()
stopProviding()
stop()
if up != nil {
_ = up.Close()
@@ -91,48 +104,149 @@ func main() {
conn.Close()
}
// credential reads the credential the mesh delivered; a missing or unreadable one is a fault of
// configuration, said and final.
func credential() bus.Credential {
// startWait is how long a runtime waits, from its start, for its broker credential to be on the machine and for the bus
// to accept the account it names (issue 1496): the node-engine starts a module's runtime before the provisioner's client
// fills the credential, and the bus carries a new account only after it reloads its users. Past it the runtime gives up,
// loudly; inside it the wait is said once.
var startWait = 5 * time.Minute
// say is a line in the runtime's journal.
func say(format string, args ...any) { fmt.Fprintf(os.Stderr, "node-tools: "+format+"\n", args...) }
// credential reads the credential the mesh delivered, waiting for it within the start's bound (issue 1496); one that can
// never be a credential (no url) is a fault of configuration, said and final.
func credential(until time.Time) bus.Credential {
file := os.Getenv("MESH_BROKER_FILE")
if file == "" {
if url := os.Getenv("MESH_BROKER_URL"); url != "" {
return bus.Credential{URL: url, Module: runtimeModule}
}
fmt.Fprintln(os.Stderr, "node-tools: set MESH_BROKER_FILE (a sealed credential) or MESH_BROKER_URL — there is no broker to reach")
say("set MESH_BROKER_FILE (a sealed credential) or MESH_BROKER_URL — there is no broker to reach")
os.Exit(1)
}
raw, err := os.ReadFile(file)
cred, err := awaitCredential(file, until, say)
if err != nil {
fmt.Fprintf(os.Stderr, "node-tools: cannot read the broker credential at %s: %v\n", file, err)
os.Exit(1)
}
var cred bus.Credential
if err := json.Unmarshal(raw, &cred); err != nil {
fmt.Fprintf(os.Stderr, "node-tools: cannot read the broker credential at %s: %v\n", file, err)
os.Exit(1)
}
if cred.URL == "" {
fmt.Fprintf(os.Stderr, "node-tools: %s carries no url — it is not a broker credential\n", file)
say("%v", err)
os.Exit(1)
}
return cred
}
// connectPatiently retries while the bus is merely not reachable yet — the normal case at startup —
// and gives up at once on what waiting cannot fix (issue 058).
func connectPatiently(cred bus.Credential) *bus.Conn {
for delay := 2 * time.Second; ; delay = min(delay*2, 30*time.Second) {
conn, err := bus.Connect(cred)
if err == nil {
return conn
// errNotCredential is a file that is there and readable and still no broker credential: waiting will not fix it.
var errNotCredential = errors.New("it carries no url — it is not a broker credential")
// readCredential reads the credential at file. A file not there yet, empty, or not yet whole JSON is one the node-engine
// has still to write: the error says so and waiting may fix it. errNotCredential is final.
func readCredential(file string) (bus.Credential, error) {
raw, err := os.ReadFile(file)
if err != nil {
return bus.Credential{}, err
}
if len(bytes.TrimSpace(raw)) == 0 {
return bus.Credential{}, errors.New("the file is empty")
}
var cred bus.Credential
if err := json.Unmarshal(raw, &cred); err != nil {
return bus.Credential{}, err
}
if cred.URL == "" {
return bus.Credential{}, errNotCredential
}
return cred, nil
}
// awaitCredential is the credential at file, read at once and, while it is not there or not whole, again each time its
// directory changes (runtime.WatchFile, inotify), until the time until. The wait is said once; at until it ends with an
// error naming the file, how long it waited and what the last read found. A file that is not a credential ends it at once.
func awaitCredential(file string, until time.Time, say func(string, ...any)) (bus.Credential, error) {
cred, err := readCredential(file)
if err == nil {
return cred, nil
}
if errors.Is(err, errNotCredential) {
return bus.Credential{}, fmt.Errorf("%s: %w", file, err)
}
began := time.Now()
say("waits for its broker credential at %s (%v): the node-engine writes it once the provisioner's client fills it; "+
"it gives up at %s", file, err, until.Format("15:04:05"))
changed := make(chan struct{}, 1)
stop, werr := runtime.WatchFile(file, func() {
select {
case changed <- struct{}{}:
default:
}
if fatal := bus.Fatal(err); fatal != "" {
fmt.Fprintf(os.Stderr, "node-tools: %s — waiting will not fix this; giving up\n", fatal)
os.Exit(1)
}, func(format string, args ...any) { say(format, args...) },
fmt.Sprintf("the credential is read again every %s instead", rereadEvery))
if werr != nil {
say("cannot watch %s (%v): it is read again every %s instead", file, werr, rereadEvery)
} else {
defer stop()
}
for {
left := time.Until(until)
if left <= 0 {
return bus.Credential{}, fmt.Errorf("cannot read the broker credential at %s: waited %s for it, and the last read "+
"found: %v", file, time.Since(began).Round(time.Second), err)
}
// The watch wakes the read; the clock is only the backstop for a watch that could not be made or missed a change.
select {
case <-changed:
case <-time.After(min(left, rereadEvery)):
}
if cred, err = readCredential(file); err == nil {
say("read its broker credential at %s after %s", file, time.Since(began).Round(time.Second))
return cred, nil
}
if errors.Is(err, errNotCredential) {
return bus.Credential{}, fmt.Errorf("%s: %w", file, err)
}
}
}
// rereadEvery is the backstop of the wait for the credential, beside the watch of its directory.
var rereadEvery = 30 * time.Second
// connectPatiently retries while the bus is merely not reachable yet — the normal case at startup — and, until the start's
// bound, while the bus refuses the account: the bus carries a new account only after it reloads its users, which may come
// after the credential is on the machine (issue 1496, issue 1423). What waiting cannot fix (a pin that does not match, an
// address that is no URL) gives up at once (issue 058), and so does a refusal past the bound.
func connectPatiently(cred bus.Credential, until time.Time) *bus.Conn {
conn, err := connectWithin(cred, until, bus.Connect, time.Sleep, say)
if err != nil {
say("%v", err)
os.Exit(1)
}
return conn
}
// connectWithin is connectPatiently with its dial, its sleep and its words given, as a test gives them.
func connectWithin(cred bus.Credential, until time.Time, dial func(bus.Credential) (*bus.Conn, error), sleep func(time.Duration),
say func(string, ...any)) (*bus.Conn, error) {
refusedSaid := false
for delay := 2 * time.Second; ; delay = min(delay*2, 30*time.Second) {
conn, err := dial(cred)
if err == nil {
if refusedSaid {
say("the bus accepts the account now")
}
return conn, nil
}
wait := delay + time.Duration(rand.IntN(1000))*time.Millisecond
fmt.Fprintf(os.Stderr, "node-tools: the broker is not reachable yet (%v); retrying in %ds\n", err, int(wait.Round(time.Second)/time.Second))
time.Sleep(wait)
switch fatal := bus.Fatal(err); {
case bus.Refused(err) && time.Now().Add(wait).Before(until):
if !refusedSaid {
say("the bus refuses this account (%v): it may not carry it yet, since it loads a new account by a reload; "+
"retrying until %s", err, until.Format("15:04:05"))
refusedSaid = true
}
case bus.Refused(err) && refusedSaid:
return nil, fmt.Errorf("the bus still refuses this account (%v) at the end of the start's bound, %s; giving up", err,
until.Format("15:04:05"))
case fatal != "":
return nil, fmt.Errorf("%s — waiting will not fix this; giving up", fatal)
default:
say("the broker is not reachable yet (%v); retrying in %ds", err, int(wait.Round(time.Second)/time.Second))
}
sleep(wait)
}
}
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
// A runtime started before its credential is on the machine waits for it, woken by the watch of its directory, and gives
// up loudly at its bound; a file that is no credential ends the wait at once (issue 1496: on ace mesh-build-node's and
// mesh-build-image's runtimes exited twice on "cannot read the broker credential ... no such file").
func TestAwaitCredential(t *testing.T) {
const good = `{"url":"tls://bus.example:4222","module":"m"}`
rereadEvery = time.Hour // only the watch can wake the read in time
settle := runtime.Settle
runtime.Settle = 10 * time.Millisecond // a unit test stays well under a second
t.Cleanup(func() { rereadEvery, runtime.Settle = 30*time.Second, settle })
place := func(path, body string) {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
tmp := path + ".tmp"
if err := os.WriteFile(tmp, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
if err := os.Rename(tmp, path); err != nil {
t.Fatal(err)
}
}
for _, c := range []struct {
name string
before string // written before the wait, "" none
later string // written 20ms into the wait, "" none
deeper bool // the file's directory is not there at the start
bound time.Duration
want string // "" a credential; else a word of the error
said int
}{
{name: "there at the start", before: good, bound: time.Second, said: 0},
{name: "written during the wait", later: good, bound: 5 * time.Second, said: 2},
{name: "empty, then filled", before: " ", later: good, bound: 5 * time.Second, said: 2},
{name: "its directory made during the wait", later: good, deeper: true, bound: 5 * time.Second, said: 2},
{name: "never written", bound: 100 * time.Millisecond, want: "waited", said: 1},
{name: "no url", before: `{"module":"m"}`, bound: 5 * time.Second, want: "not a broker credential", said: 0},
} {
t.Run(c.name, func(t *testing.T) {
dir := t.TempDir()
file := filepath.Join(dir, "broker")
if c.deeper {
file = filepath.Join(dir, "var", "lib", "m", "broker")
}
if c.before != "" {
place(file, c.before)
}
if c.later != "" {
go func() { time.Sleep(20 * time.Millisecond); place(file, c.later) }()
}
var said []string
began := time.Now()
cred, err := awaitCredential(file, time.Now().Add(c.bound), func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
switch {
case c.want == "" && (err != nil || cred.URL == ""):
t.Fatalf("no credential: %v", err)
case c.want != "" && (err == nil || !strings.Contains(err.Error(), c.want)):
t.Fatalf("error %v, want one saying %q", err, c.want)
case c.want == "" && c.later != "" && time.Since(began) > 3*time.Second:
t.Fatalf("read after %s: the watch did not wake it", time.Since(began))
}
if len(said) != c.said {
t.Fatalf("said %d lines, want %d: %q", len(said), c.said, said)
}
})
}
}
// A bus that refuses the account at a start is asked again until the start's bound, said once; a refusal past it, a pin
// that does not match and an address that is no URL give up (issue 1496, issue 058).
func TestConnectWithin(t *testing.T) {
refused := errors.New("nats: Authorization Violation")
unreachable := errors.New("nats: no servers available for connection")
for _, c := range []struct {
name string
errs []error // what each dial answers, then a connection
bound time.Duration
fails string
saidAt int // lines said
}{
{name: "refused twice, then accepted", errs: []error{refused, refused}, bound: time.Hour, saidAt: 2},
{name: "unreachable, then accepted", errs: []error{unreachable}, bound: time.Hour, saidAt: 1},
{name: "refused past the bound", errs: []error{refused, refused}, bound: 0, fails: "refused this account"},
{name: "refused, then the bound ends", errs: []error{refused, refused, refused, refused, refused, refused}, bound: 5 * time.Second,
fails: "still refuses"},
{name: "pin", errs: []error{fmt.Errorf("tls: %w", bus.ErrPin)}, bound: time.Hour, fails: "pin"},
} {
t.Run(c.name, func(t *testing.T) {
dials := 0
dial := func(bus.Credential) (*bus.Conn, error) {
dials++
if dials <= len(c.errs) {
return nil, c.errs[dials-1]
}
return nil, nil
}
until := time.Now().Add(c.bound)
var said []string
_, err := connectWithin(bus.Credential{URL: "tls://bus.example:4222"}, until, dial, func(time.Duration) {},
func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
switch {
case c.fails == "" && err != nil:
t.Fatalf("gave up: %v", err)
case c.fails != "" && (err == nil || !strings.Contains(err.Error(), c.fails)):
t.Fatalf("error %v, want one saying %q", err, c.fails)
case c.fails == "" && len(said) != c.saidAt:
t.Fatalf("said %d lines, want %d: %q", len(said), c.saidAt, said)
}
})
}
}
+2 -2
View File
@@ -8,15 +8,15 @@ require (
)
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030100935-c2ec8b6b2fde
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nats-server/v2 v2.11.17 // indirect
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
go.uber.org/automaxprocs v1.6.0 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+8 -2
View File
@@ -1,3 +1,11 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030095131-16bc0dcf3e9d h1:jFOz6wUfZvIayd4DctwlaFKKZHQNME6NLs2OS1rNv30=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030095131-16bc0dcf3e9d/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030095816-530d8d3007dc h1:Wdjl2gomooHswJ5qpkwZPq/ureZgXVw6FMCsgzPJhPo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030095816-530d8d3007dc/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030100935-c2ec8b6b2fde h1:eKXSllHCS31GgMazfvZ7BeiLoXIUibnUmzxWVkj+Vcw=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030100935-c2ec8b6b2fde/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -16,8 +24,6 @@ github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
go.uber.org/automaxprocs v1.6.0 h1:O3y2/QNTOdbF+e/dpXNNW7Rx2hZ4sTIPyybbxyNqTUs=
go.uber.org/automaxprocs v1.6.0/go.mod h1:ifeIMSnPZuznNm6jmdzmU3/bfk01Fe2fotchwEFJ8r8=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+80 -55
View File
@@ -10,7 +10,6 @@
package bus
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
@@ -71,6 +70,15 @@ type Membership struct {
// SeatTraffic is what the module's code may submit, say, take, ask, answer and read on seats that name
// their caller or their kind (novox/hq ADR 0259): the runtime does for it only what is listed here.
SeatTraffic *SeatTraffic `json:"seat-traffic,omitempty"`
// Streams are the module's own streams (novox/hq ADR 0363 §2): an event one of them carries is published
// on that stream's subject, never on EVENTS (streams.go).
Streams []StreamIssued `json:"streams,omitempty"`
// Hears are the module streams the module hears, each through a durable consumer on that stream the
// runtime binds beside the one on EVENTS (ADR 0363 §8).
Hears []HeardIssued `json:"hears,omitempty"`
// Waits are the event families a session may wait on through the mesh MCP server (novox/hq ADR 0367 §2):
// the tool runner's alone, composed from its manifest with the grant that lets it subscribe them.
Waits []WaitIssued `json:"waits,omitempty"`
}
// Served is an address a tool is answered on; `{tool}` stands for the tool's name.
@@ -121,12 +129,19 @@ func Fatal(err error) string {
strings.Contains(msg, "nats: invalid url") {
return "the bus address is not a usable URL"
}
if regexp.MustCompile(`(?i)authorization violation|user authentication expired|permissions violation`).MatchString(msg) {
if Refused(err) || regexp.MustCompile(`(?i)user authentication expired|permissions violation`).MatchString(msg) {
return "the bus refused this account"
}
return ""
}
// Refused says whether the bus refused the account a connection named (an authorization violation): final for a running
// runtime, but at a start possibly only "not yet", since the bus carries a new account after it reloads its users (issue
// 1496). Fatal still names it; the caller decides how long a start waits.
func Refused(err error) bool {
return err != nil && regexp.MustCompile(`(?i)authorization violation`).MatchString(err.Error())
}
func normalizeFingerprint(f string) string {
f = strings.TrimSpace(f)
if len(f) > 7 && strings.EqualFold(f[:7], "sha256:") {
@@ -165,7 +180,7 @@ type Conn struct {
cred Credential
mu sync.Mutex
issued map[string]*Membership // module → membership; present with nil = followed, none issued
onNew []func(Membership)
onNew []*func(Membership)
subs []*nats.Subscription
Logf func(format string, args ...any)
// answering is every subject this runtime answers on, so a subscription the bus refused can be
@@ -176,6 +191,8 @@ type Conn struct {
RetryAfter []time.Duration
// calls are the caller-named calls in flight (caller.go, novox/hq issue 365).
calls callers
// waits are the waits in flight by subject, each told when the bus refuses its subscription (wait.go).
waits map[string]map[chan string]bool
}
// answered is one subject the runtime answers on: what to subscribe again with, and how often it was
@@ -213,6 +230,9 @@ func (c *Conn) refused(err error) {
if c.publishRefused(err) {
return
}
if subject := subjectRefused(err); subject != "" && c.waitRefused(subject) {
return
}
m := refusedSubject.FindStringSubmatch(err.Error())
if len(m) < 2 {
return
@@ -363,7 +383,10 @@ func (c *Conn) Follow(module string) {
}
c.mu.Lock()
c.issued[module] = &m
handlers := append([]func(Membership){}, c.onNew...)
handlers := make([]func(Membership), 0, len(c.onNew))
for _, h := range c.onNew {
handlers = append(handlers, *h)
}
c.mu.Unlock()
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
for _, h := range handlers {
@@ -405,10 +428,28 @@ func (c *Conn) Serving() []string {
// OnMembership is called with every new membership any followed module is issued.
func (c *Conn) OnMembership(h func(Membership)) {
c.mu.Lock()
c.onNew = append(c.onNew, h)
c.onNew = append(c.onNew, &h)
c.mu.Unlock()
}
// onMembership is OnMembership until the answered function is called, which drops the handler.
func (c *Conn) onMembership(h func(Membership)) func() {
mine := &h
c.mu.Lock()
c.onNew = append(c.onNew, mine)
c.mu.Unlock()
return func() {
c.mu.Lock()
defer c.mu.Unlock()
for i, had := range c.onNew {
if had == mine {
c.onNew = append(c.onNew[:i:i], c.onNew[i+1:]...)
return
}
}
}
}
func (c *Conn) track(s *nats.Subscription) {
c.mu.Lock()
c.subs = append(c.subs, s)
@@ -568,9 +609,13 @@ func (c *Conn) HandleCalled(key string, h CalledHandler) (func(), error) {
}
var mu sync.Mutex
var stops []func()
stopped := false
serve := func() {
mu.Lock()
defer mu.Unlock()
if stopped {
return
}
for _, s := range stops {
s()
}
@@ -584,14 +629,18 @@ func (c *Conn) HandleCalled(key string, h CalledHandler) (func(), error) {
}
}
serve()
c.OnMembership(func(m Membership) {
// Its stop drops the membership handler too: a handler left behind would bind the stopped tool again
// on the module's next membership, and call a handler whose bundle was replaced (issue 1945).
unfollow := c.onMembership(func(m Membership) {
if m.Module == module {
serve()
}
})
return func() {
unfollow()
mu.Lock()
defer mu.Unlock()
stopped = true
for _, s := range stops {
s()
}
@@ -674,12 +723,16 @@ var HandoverPause = 3 * time.Second
// (mesh-controller internal/link, `"retry": "handing-over"`).
const RetryHandingOver = "handing-over"
// ErrNoResponders is the bus's own refusal: nothing subscribes to the subject asked. It is told apart
// from a tool's own error by errors.Is, never by its text (a tool may say "503" itself, issue 1529).
var ErrNoResponders = errors.New("503 no responders")
// askOnce asks once, and says whether the answer was a refusal marked to be asked again.
func (c *Conn) askOnce(subject string, data []byte) (Answered, bool, error) {
msg, err := c.nc.Request(subject, data, RequestTimeout)
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return Answered{}, false, errors.New("503 no responders")
return Answered{}, false, ErrNoResponders
}
if errors.Is(err, nats.ErrTimeout) {
return Answered{}, false, errors.New("timeout")
@@ -715,7 +768,9 @@ func (c *Conn) answerOf(subject string, data []byte) (Answered, bool, error) {
// PublishAs emits an event as a module: published into JetStream and awaited, de-duplicated by its
// own id (ADR 0042). The body is the payload; the metadata rides as headers.
func (c *Conn) PublishAs(module string, env Envelope) error {
msg := nats.NewMsg("mesh.mod." + module + ".event." + env.Key)
// On the module's own stream when its membership names one carrying the event (ADR 0363 §2), on EVENTS
// otherwise — and only there, as before, for a membership that names no stream.
msg := nats.NewMsg(eventSubject(c.Membership(module), module, env.Key))
for k, v := range env.Headers {
msg.Header.Set(k, v)
}
@@ -937,64 +992,34 @@ var NakDelay = 5 * time.Second
// handed over, and a burst would end in max-deliver with nothing failed (novox/hq issue 387).
var EventHeartbeat = 10 * time.Second
// ConsumeAs reads a module's durable consumer and hands each event to deliver (novox/hq ADR 0198):
// acknowledged when deliver returns nil, negatively acknowledged after NakDelay when it returns an
// error, terminated when it is not an event at all. The consumer is the controller's to create; this
// binds to it and never makes one. Runs until stopped.
// ConsumeAs reads a module's durable consumer on EVENTS, and each one on a module's stream its
// membership names (novox/hq ADR 0363 §8), and hands each event to deliver one at a time (novox/hq
// ADR 0198): acknowledged when deliver returns nil, negatively acknowledged after NakDelay when it
// returns an error, terminated when it is not an event at all. The consumers are the controller's to
// create; this binds to them and never makes one. Runs until stopped.
func (c *Conn) ConsumeAs(module string, deliver func(Envelope) error) (func(), error) {
durable := ConsumerOf(c.node, module)
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
h := newHearing(c, js, module, deliver)
// Bound by name, never created and never matched against a subject: the consumer and its
// filters are the controller's (design 29 §3), exactly as the module's own runtime bound it.
consumer, err := js.Consumer(ctx, EventsStream, durable)
events, _, doing, err := h.bind(EventsStream, durable, nil)
if err != nil {
return nil, fmt.Errorf("binding %s's consumer %s: %w", module, durable, err)
}
reading, err := consumer.Consume(func(msg jetstream.Msg) {
env, ok := envelopeOf(msg.Subject(), msg.Headers(), msg.Data())
if !ok {
// Unparseable: redelivering bytes no version of a handler can read is a loop.
_ = msg.Term()
return
}
// Kept alive while the bundle works on it, as a worker's work is (seat.go): the consumer's ack wait
// (30 seconds) is for a runtime that died, not a handler that is slow — the runtime gives a handler
// launch.EventTimeout. Without it an event handled longer than the ack wait was offered again while
// it was handled, and counted toward max-deliver (novox/hq issue 387).
done, stopped := make(chan struct{}), make(chan struct{})
go func() {
defer close(stopped)
tick := time.NewTicker(EventHeartbeat)
defer tick.Stop()
for {
select {
case <-done:
return
case <-tick.C:
_ = msg.InProgress()
}
}
}()
err := deliver(env)
close(done)
// No in-progress after the answer: one after a Nak would stretch its delay to the whole ack wait.
<-stopped
if err != nil {
_ = msg.NakWithDelay(NakDelay)
return
}
_ = msg.Ack()
}, jetstream.PullMaxMessages(1))
if err != nil {
return nil, fmt.Errorf("reading %s's consumer %s: %w", module, durable, err)
return nil, fmt.Errorf("%s %s's consumer %s: %w", doing, module, durable, err)
}
// And every consumer on a module's stream its membership names (novox/hq ADR 0363 §8), followed as the
// membership changes. A membership naming none binds nothing more: the module hears EVENTS alone.
h.start()
var once sync.Once
return func() { once.Do(reading.Stop) }, nil
return func() {
once.Do(func() {
h.stop()
events.Stop()
})
}, nil
}
// envelopeOf rebuilds the envelope a module sees from a delivered event: its key is the emitter and
@@ -0,0 +1,48 @@
package bus
import (
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// heardMsg is a delivered event as a consumer's loop hands it, saying how it was settled.
type heardMsg struct {
jetstream.Msg
mu sync.Mutex
settled []string
}
func (m *heardMsg) Subject() string { return "mesh.mod.alpha.event.happened" }
func (m *heardMsg) Headers() nats.Header { return nats.Header{} }
func (m *heardMsg) Data() []byte { return []byte(`{"n":1}`) }
func (m *heardMsg) InProgress() error { return nil }
func (m *heardMsg) Ack() error { return m.say("ack") }
func (m *heardMsg) Nak() error { return m.say("nak") }
func (m *heardMsg) NakWithDelay(time.Duration) error { return m.say("nak-delayed") }
func (m *heardMsg) Term() error { return m.say("term") }
func (m *heardMsg) say(how string) error {
m.mu.Lock()
defer m.mu.Unlock()
m.settled = append(m.settled, how)
return nil
}
// An event the bus hands a hearing after it was stopped (a module's bundle loaded again, issue 1945) is
// offered again at once and never handed to the module: the module's handler that heard it may be gone.
func TestAHearingStoppedHandsNoEventOnAndOffersItAgain(t *testing.T) {
delivered := 0
h := newHearing(nil, nil, "alpha", func(Envelope) error { delivered++; return nil })
h.stop()
msg := &heardMsg{}
h.handle(msg)
if delivered != 0 {
t.Errorf("a stopped hearing handed the event to the module")
}
if len(msg.settled) != 1 || msg.settled[0] != "nak" {
t.Errorf("a stopped hearing settled the event as %v, want it offered again at once (nak)", msg.settled)
}
}
+57 -37
View File
@@ -221,49 +221,17 @@ func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int)
if from == 0 {
from = 1
}
req, _ := json.Marshal(map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes})
inbox := c.nc.NewInbox()
sub, err := c.nc.SubscribeSync(inbox)
msgs, next, err := c.directBatch(l.Stream, filter, from, limit, name, func() error { return notOnTheBus(l) })
if err != nil {
return nil, err
}
defer func() { _ = sub.Unsubscribe() }()
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+l.Stream, inbox, req); err != nil {
return nil, err
}
page := &LogPage{Entries: []LogEntry{}}
deadline := time.Now().Add(StateTimeout)
for done := false; !done; {
msg, err := sub.NextMsg(time.Until(deadline))
page := &LogPage{Entries: []LogEntry{}, Next: next}
for _, msg := range msgs {
e, err := logEntryOf(l, msg)
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return nil, notOnTheBus(l)
}
if errors.Is(err, nats.ErrTimeout) {
return nil, fmt.Errorf("the read of %s did not end in %s", name, StateTimeout)
}
return nil, err
}
switch status := msg.Header.Get("Status"); status {
case "":
e, err := logEntryOf(l, msg)
if err != nil {
return nil, err
}
page.Entries = append(page.Entries, *e)
case "204": // the end of the batch: what is still pending after the last entry sent
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
if pending > 0 && len(page.Entries) > 0 {
page.Next = page.Entries[len(page.Entries)-1].Sequence + 1
}
done = true
case "404": // nothing from that sequence on
done = true
case "503":
return nil, notOnTheBus(l)
default:
return nil, fmt.Errorf("the read of %s was refused by the bus: %s %s", name, status, msg.Header.Get("Description"))
}
page.Entries = append(page.Entries, *e)
}
info, err := c.logInfo(l)
if err != nil {
@@ -273,6 +241,58 @@ func (c *Conn) LogRead(module, name string, from uint64, keys string, limit int)
return page, nil
}
// directBatch is the messages of a stream from a sequence on, on the subjects matching filter, at most limit and about
// LogPageBytes, by one batch direct get (NATS 2.11), and the sequence to read on from, 0 when nothing follows. name is
// what is read, for its errors; missing is the error of a stream that is not on the bus.
func (c *Conn) directBatch(stream, filter string, from uint64, limit int, name string, missing func() error) ([]*nats.Msg, uint64, error) {
if from == 0 {
from = 1
}
req, _ := json.Marshal(map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes})
inbox := c.nc.NewInbox()
sub, err := c.nc.SubscribeSync(inbox)
if err != nil {
return nil, 0, err
}
defer func() { _ = sub.Unsubscribe() }()
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+stream, inbox, req); err != nil {
return nil, 0, err
}
var msgs []*nats.Msg
var next uint64
deadline := time.Now().Add(StateTimeout)
for done := false; !done; {
msg, err := sub.NextMsg(time.Until(deadline))
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return nil, 0, missing()
}
if errors.Is(err, nats.ErrTimeout) {
return nil, 0, fmt.Errorf("the read of %s did not end in %s", name, StateTimeout)
}
return nil, 0, err
}
switch status := msg.Header.Get("Status"); status {
case "":
msgs = append(msgs, msg)
case "204": // the end of the batch: what is still pending after the last message sent
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
if pending > 0 && len(msgs) > 0 {
last, _ := strconv.ParseUint(msgs[len(msgs)-1].Header.Get("Nats-Sequence"), 10, 64)
next = last + 1
}
done = true
case "404": // nothing from that sequence on
done = true
case "503":
return nil, 0, missing()
default:
return nil, 0, fmt.Errorf("the read of %s was refused by the bus: %s %s", name, status, msg.Header.Get("Description"))
}
}
return msgs, next, nil
}
// LogLast is a key's last entry, or nil when it has none.
func (c *Conn) LogLast(module, name, key string) (*LogEntry, error) {
l, err := c.issuedLog(module, name)
+98
View File
@@ -0,0 +1,98 @@
package bus
import (
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
)
// A module's code asks its machine's provisioner client for its values at runtime (novox/hq ADR 0349 clause 14 as
// ADR 0370 changes it, issue 685).
//
// While a module's code runs here, its asks reach the bus under this runtime's account, and the bus proves only
// that account. So the runtime asks its own machine's client on ProvisionAskSubject — which the controller grants
// this machine's tool runner alone to publish (ADR 0370 §3) — naming the module itself, from the bundle that asks
// and never from the module's code or arguments (§4). The client answers that module's config values, for what its
// manifest declares, and no secret: the vault hands a secret only to a caller the bus proves (§2).
// ProvisionAskSubject is where this machine's tool runner asks its provisioner client (mesh-controller
// internal/broker ProvisionerAskSubject): `prov.ask.<node>`.
func ProvisionAskSubject(node string) string { return "prov.ask." + node }
// ProvisionChangedSubjects is every module's word of what moved on a machine (mesh-controller internal/broker
// ProvisionerChangedSubject): `prov.changed.<node>.<module>`, a module's name dots and all.
func ProvisionChangedSubjects(node string) string { return "prov.changed." + node + ".>" }
// ProvisionTimeout bounds one ask of the client; a variable so a test need not wait.
var ProvisionTimeout = 10 * time.Second
// ProvisionChanged is what the client says moved of one module: the kind, and the key, or every key of the kind when
// empty. Never the value.
type ProvisionChanged struct {
Kind string `json:"kind"`
Key string `json:"key,omitempty"`
}
// ProvisionRefused is the client's refusal, in its own words: a key the module does not declare, a module not placed
// on the machine, a secret, a kind it does not answer yet.
type ProvisionRefused struct{ Reason string }
func (r *ProvisionRefused) Error() string { return r.Reason }
// AskProvisioner asks this machine's provisioner client for one value of module: kind `config` or `bound`, and the
// key. The module is the caller's to name from the bundle it runs; this never takes it from what a bundle sent.
func (c *Conn) AskProvisioner(module, kind, key string) (json.RawMessage, error) {
if c.node == "" {
return nil, errors.New("this runtime names no machine, so it has no provisioner client to ask")
}
body, err := json.Marshal(map[string]string{"module": module, "kind": kind, "key": key})
if err != nil {
return nil, err
}
msg, err := c.nc.Request(ProvisionAskSubject(c.node), body, ProvisionTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return nil, fmt.Errorf("%s's provisioner client does not answer: it is not running there, or the bus does "+
"not let %s ask it", c.node, c.cred.User)
}
if err != nil {
return nil, fmt.Errorf("asking %s's provisioner client for %s's %s %s: %w", c.node, module, kind, key, err)
}
var answered struct {
Value json.RawMessage `json:"value"`
Error string `json:"error"`
}
if err := json.Unmarshal(msg.Data, &answered); err != nil {
return nil, fmt.Errorf("%s's provisioner client answered something that is not its answer: %w", c.node, err)
}
if answered.Error != "" {
return nil, &ProvisionRefused{Reason: answered.Error}
}
if len(answered.Value) == 0 {
return nil, fmt.Errorf("%s's provisioner client answered no value and no reason", c.node)
}
return answered.Value, nil
}
// OnProvisionChanged hands told every word of this machine's provisioner client of what moved, by module, until
// stopped.
func (c *Conn) OnProvisionChanged(told func(module string, changed ProvisionChanged)) (func(), error) {
if c.node == "" {
return nil, errors.New("this runtime names no machine, so it hears no provisioner client")
}
prefix := "prov.changed." + c.node + "."
sub, err := c.nc.Subscribe(ProvisionChangedSubjects(c.node), func(m *nats.Msg) {
var changed ProvisionChanged
if json.Unmarshal(m.Data, &changed) != nil || changed.Kind == "" {
return
}
told(strings.TrimPrefix(m.Subject, prefix), changed)
})
if err != nil {
return nil, err
}
return func() { _ = sub.Unsubscribe() }, c.nc.Flush()
}
+181
View File
@@ -0,0 +1,181 @@
package bus
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"fmt"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
)
// **A holder that pulls one piece of work when it has room** (novox/hq ADR 0525 §4, §6; issue 1887 A11). SeatTake
// hands a bundle every piece a worker holds, one at a time and prefetched; a build module must instead take one ask
// only when it has a free slot, from its workers in their order (delivery, then check, then recheck), and decide each
// ask's fate when its step ends. SeatPull is one fetch of at most one message from a worker the module's membership
// lists, never a prefetch; SeatSettle gives a pulled message its fate. A message the bundle never settles (its process
// died) is delivered again by the worker's own acknowledgement wait.
// MaxPullWait bounds how long one pull waits for a message: under the bundle's 30 s wait for its answer.
var MaxPullWait = 25 * time.Second
// Pulled is one message a module pulled from a seat's worker, named by an id the runtime gave it.
type Pulled struct {
ID string `json:"id"`
Worker string `json:"worker"`
Subject string `json:"subject"`
Body json.RawMessage `json:"body"`
Headers map[string]string `json:"headers,omitempty"`
Delivered uint64 `json:"delivered"`
}
// pulls are the messages modules pulled and have not settled, by id; each kept with the module it was pulled for, so
// one module never settles another's.
type pulls struct {
mu sync.Mutex
held map[string]pulledMsg
cons map[string]jetstream.Consumer // by module and consumer
}
type pulledMsg struct {
module string
msg jetstream.Msg
}
var seatPulls = &pulls{held: map[string]pulledMsg{}, cons: map[string]jetstream.Consumer{}}
func pullID() string {
var b [8]byte
_, _ = rand.Read(b[:])
return hex.EncodeToString(b[:])
}
// SeatPull fetches at most one message from consumer for module, waiting at most wait (none when 0): nil when none
// came. The consumer must be one the module's membership lists as its worker (novox/hq ADR 0259); the runtime binds to
// it by name and never makes one.
func (c *Conn) SeatPull(module, consumer string, wait time.Duration) (*Pulled, error) {
t, err := c.trafficOf(module)
if err != nil {
return nil, err
}
var w *SeatWorker
var names []string
for i := range t.Workers {
names = append(names, t.Workers[i].Consumer)
if t.Workers[i].Consumer == consumer {
w = &t.Workers[i]
}
}
if w == nil {
return nil, fmt.Errorf("%s pulls no work from %s: its membership lists %s (novox/hq ADR 0259)", module,
consumer, strings.Join(names, ", "))
}
if wait > MaxPullWait {
wait = MaxPullWait
}
bound, err := c.pullConsumer(module, *w)
if err != nil {
return nil, err
}
var batch jetstream.MessageBatch
if wait <= 0 {
batch, err = bound.FetchNoWait(1)
} else {
batch, err = bound.Fetch(1, jetstream.FetchMaxWait(wait))
}
if err != nil {
return nil, fmt.Errorf("pulling %s's worker %s: %w", module, w.Consumer, err)
}
msg, ok := <-batch.Messages()
if !ok {
if err := batch.Error(); err != nil {
return nil, fmt.Errorf("pulling %s's worker %s: %w", module, w.Consumer, err)
}
return nil, nil
}
if !json.Valid(msg.Data()) {
_ = msg.Term()
return nil, nil
}
headers := map[string]string{}
for k := range msg.Headers() {
headers[k] = msg.Headers().Get(k)
}
delivered := uint64(1)
if meta, merr := msg.Metadata(); merr == nil && meta != nil {
delivered = meta.NumDelivered
}
p := &Pulled{ID: pullID(), Worker: w.Consumer, Subject: msg.Subject(), Body: json.RawMessage(msg.Data()),
Headers: headers, Delivered: delivered}
seatPulls.mu.Lock()
seatPulls.held[p.ID] = pulledMsg{module: module, msg: msg}
seatPulls.mu.Unlock()
return p, nil
}
func (c *Conn) pullConsumer(module string, w SeatWorker) (jetstream.Consumer, error) {
key := module + " " + w.Consumer
seatPulls.mu.Lock()
bound, ok := seatPulls.cons[key]
seatPulls.mu.Unlock()
if ok {
return bound, nil
}
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
bound, err = js.Consumer(ctx, w.Stream, w.Consumer)
if err != nil {
return nil, fmt.Errorf("binding %s's worker %s on %s: %w", module, w.Consumer, w.Stream, err)
}
seatPulls.mu.Lock()
seatPulls.cons[key] = bound
seatPulls.mu.Unlock()
return bound, nil
}
// The fates SeatSettle gives a pulled message.
const (
SettleAck = "ack" // done: never delivered again
SettleNak = "nak" // handed back, to be delivered again at once
SettleTerm = "term" // dropped
SettleProgress = "progress" // still being worked: its acknowledgement wait starts again
)
// SeatSettle gives the message module pulled under id its fate. A message pulled for another module, or settled
// already, is refused.
func (c *Conn) SeatSettle(module, id, fate string) error {
seatPulls.mu.Lock()
p, ok := seatPulls.held[id]
if ok && p.module != module {
ok = false
}
if ok && fate != SettleProgress {
delete(seatPulls.held, id)
}
seatPulls.mu.Unlock()
if !ok {
return fmt.Errorf("%s holds no pulled message %s", module, id)
}
switch fate {
case SettleAck:
return p.msg.Ack()
case SettleNak:
return p.msg.Nak()
case SettleTerm:
return p.msg.Term()
case SettleProgress:
return p.msg.InProgress()
}
seatPulls.mu.Lock()
seatPulls.held[id] = p // an unknown fate settles nothing
seatPulls.mu.Unlock()
return fmt.Errorf("a pulled message's fate is ack, nak, term or progress, not %q", fate)
}
+64
View File
@@ -0,0 +1,64 @@
package bus_test
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// 1887/A11: a holder pulls one message at a time, only from a worker its membership lists, never a prefetch; it settles
// only what it pulled, and a message handed back comes again.
func TestAHolderPullsOneMessageWhenItHasRoom(t *testing.T) {
_, conn, js := seatMesh(t)
for _, id := range []string{"a1", "a2"} {
if _, err := conn.SeatPublish("mesh-delivery", "mesh.seat.operator-channel.accept.ask.mesh-delivery",
json.RawMessage(`{"id":"`+id+`"}`), id); err != nil {
t.Fatal(err)
}
}
first, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 0)
if err != nil || first == nil || string(first.Body) != `{"id":"a1"}` || first.Delivered != 1 {
t.Fatalf("the first pull = %+v, %v", first, err)
}
// One pulled, one still waiting on the stream: nothing was fetched ahead.
info, err := js.ConsumerInfo("SEAT_OPERATOR_CHANNEL", "SEAT_OPERATOR_CHANNEL_worker")
if err != nil || info.NumPending != 1 || info.NumAckPending != 1 {
t.Fatalf("after one pull: pending %d, in hand %d (%v); want 1 and 1", info.NumPending, info.NumAckPending, err)
}
second, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 0)
if err != nil || second == nil || string(second.Body) != `{"id":"a2"}` {
t.Fatalf("the second pull = %+v, %v", second, err)
}
if none, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 0); err != nil || none != nil {
t.Fatalf("an empty worker answered %+v, %v", none, err)
}
began := time.Now()
if none, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 200*time.Millisecond); err != nil || none != nil || time.Since(began) < 150*time.Millisecond {
t.Fatalf("a waiting pull answered %+v, %v after %v", none, err, time.Since(began))
}
if err := conn.SeatSettle("telegram", first.ID, bus.SettleAck); err == nil {
t.Error("a module settled another's message")
}
if err := conn.SeatSettle("messenger", first.ID, bus.SettleProgress); err != nil {
t.Fatal(err)
}
if err := conn.SeatSettle("messenger", first.ID, bus.SettleAck); err != nil {
t.Fatal(err)
}
if err := conn.SeatSettle("messenger", first.ID, bus.SettleAck); err == nil {
t.Error("a message was settled twice")
}
if err := conn.SeatSettle("messenger", second.ID, bus.SettleNak); err != nil {
t.Fatal(err)
}
again, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", time.Second)
if err != nil || again == nil || string(again.Body) != `{"id":"a2"}` || again.Delivered != 2 {
t.Fatalf("the message handed back = %+v, %v", again, err)
}
if _, err := conn.SeatPull("telegram", "SEAT_OPERATOR_CHANNEL_worker", 0); err == nil || !strings.Contains(err.Error(), "membership lists") {
t.Errorf("a channel pulled the router's work: %v", err)
}
}
+55 -9
View File
@@ -261,7 +261,12 @@ func (c *Conn) writable(module, name string) (StateIssued, error) {
// StateWatch hands deliver the current value of every key matching the pattern — none that is
// deleted — and then every change, in order (ADR 0201). It returns once the current values are
// delivered; deliver is called from one goroutine, one change at a time, and an error from it is said
// delivered. **StateTimeout bounds the time without progress, not the whole set** (novox/hq issue
// 1726): each current value is a round trip to the bundle, so a state of a few hundred keys outlasts
// any fixed bound for all of them; the watch then failed at every attempt, the bundle dropped its
// handler, and the newest keys, handed over last, were never taken (the checker's owed, whose newest
// verdicts never reached mesh/check). It fails only when no current value arrives, or one is not
// taken, within StateTimeout; deliver is called from one goroutine, one change at a time, and an error from it is said
// and the watch goes on: state is not a queue, and the next change, or the next start, reads it again.
// The pattern is a key, with `*` for one name and `**` for the rest; empty is every key.
func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange) error) (stop func(), err error) {
@@ -306,6 +311,14 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
})
}
current := make(chan struct{})
// handed says a current value was handed over or passed by: progress, which restarts the bound.
handed := make(chan struct{}, 1)
progress := func() {
select {
case handed <- struct{}{}:
default:
}
}
go func() {
initial := true
for e := range w.Updates() {
@@ -326,6 +339,7 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
// A deletion among the current values is a key that is not there: not handed over
// (measured, research 024 — the server sends its marker among the initial values).
if initial {
progress()
continue
}
change.Op = "delete"
@@ -334,17 +348,32 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
module, change.Op, change.Key, name, err)
}
if initial {
progress()
}
}
if initial {
close(current)
}
}()
select {
case <-current:
return stop, nil
case <-time.After(StateTimeout):
stop()
return nil, fmt.Errorf("the current values of %s did not arrive in %s", name, StateTimeout)
bound := time.NewTimer(StateTimeout)
defer bound.Stop()
for {
select {
case <-current:
return stop, nil
case <-handed:
if !bound.Stop() {
select {
case <-bound.C:
default:
}
}
bound.Reset(StateTimeout)
case <-bound.C:
stop()
return nil, fmt.Errorf("the current values of %s did not arrive in %s: none was handed over in that time", name, StateTimeout)
}
}
}
@@ -362,7 +391,10 @@ func valueOf(raw []byte) json.RawMessage {
var credentialEndings = []string{"password", "passwd", "secret", "token", "credential", "credentials",
"authorization", "apikey", "privatekey", "accesskey", "cookie"}
// credentialField is the first field anywhere in a JSON value whose name says it is a credential, or "".
// credentialField is the first field anywhere in a JSON value whose name says it is a credential, or "". The one shape a
// credential-shaped name may hold is a provider reference, an object of exactly `node` and `module`, both strings: a
// placement's binding of the provision "secret" to the vault (issue 1126) names who provides it, never its value. Any
// other shape under such a name is refused.
func credentialField(value json.RawMessage) string {
var v any
if json.Unmarshal(value, &v) != nil {
@@ -380,7 +412,7 @@ func credentialField(value json.RawMessage) string {
for _, k := range keys {
norm := strings.NewReplacer("-", "", "_", "", " ", "").Replace(strings.ToLower(k))
for _, end := range credentialEndings {
if strings.HasSuffix(norm, end) {
if strings.HasSuffix(norm, end) && !providerReference(t[k]) {
return k
}
}
@@ -399,3 +431,17 @@ func credentialField(value json.RawMessage) string {
}
return walk(v)
}
// providerReference says whether v is exactly {"node": <name>, "module": <name>}, each a name as mesh-sdk names them.
func providerReference(v any) bool {
m, ok := v.(map[string]any)
if !ok || len(m) != 2 {
return false
}
node, okN := m["node"].(string)
module, okM := m["module"].(string)
return okN && okM && referenceName.MatchString(node) && referenceName.MatchString(module)
}
// referenceName is a node's or a module's name as mesh-sdk names them: lower case, digits and dashes.
var referenceName = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
+8
View File
@@ -15,6 +15,14 @@ func TestACredentialNamedFieldIsFoundAnywhereInAValue(t *testing.T) {
`{"clientSecret":"s"}`: "clientSecret",
`{"maxTokens":4096,"name":"a","generation":3}`: "",
`"just a string"`: "",
// Issue 1126: a placement binds the provision "secret" to the vault; only an exact provider reference passes.
`{"bindings":{"secret":{"node":"anchor","module":"mesh-vault"}}}`: "",
`{"secret":{"value":"s"}}`: "secret",
`{"secret":{"node":"anchor","module":"mesh-vault","value":"s"}}`: "secret",
`{"secret":{"node":"anchor","module":7}}`: "secret",
`{"secret":{"node":"anchor","module":"hunter2 Bearer s3cr3t"}}`: "secret",
`{"secret":["s"]}`: "secret",
`{"bindings":{"secret":{"token":"s"}}}`: "secret",
} {
if got := credentialField(json.RawMessage(value)); got != want {
t.Errorf("%s: found %q, want %q", value, got, want)
+144
View File
@@ -0,0 +1,144 @@
//go:build integration
package bus_test
import (
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// **A watch whose current values take longer than the bound to hand over still holds** (novox/hq issue
// 1726). The gitea module watches the checker's owed: one key per pull request head, each a round trip
// to the bundle. With a few hundred keys the hand-over outlasted StateTimeout at every attempt, the
// watch was refused, and the newest keys, handed over last, never reached mesh/check: a group's passing
// verdict was never written on its members' heads. The bound is on progress: a watch fails only when
// no current value is handed over within it.
func TestAWatchHoldsWhileItsCurrentValuesKeepArriving(t *testing.T) {
was := bus.StateTimeout
bus.StateTimeout = 400 * time.Millisecond
t.Cleanup(func() { bus.StateTimeout = was })
conn, kv := owedWatcher(t)
// A group of three pull requests judged last, after many older heads: their keys come last.
const older = 30
var keys []string
for i := 0; i < older; i++ {
keys = append(keys, fmt.Sprintf("novox/mesh-catalog.%d.%040x.status=mesh/check", 400+i, i))
}
for _, member := range []string{"novox/mesh-sdk.76.34f210e5", "novox/mesh-controller.453.c06f75d0", "novox/mesh-delivery.33.28ee22ef"} {
keys = append(keys, member+strings.Repeat("0", 32)+".status=mesh/check")
}
for _, k := range keys {
if _, err := kv.Put(k, []byte(`{"v":1,"act":"status","context":"mesh/check","state":"success"}`)); err != nil {
t.Fatal(err)
}
}
var mu sync.Mutex
var current, live []string
// Each value takes 40 ms to be taken: 33 values take about 1.3 s, three times the bound.
stop, err := conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", func(ch bus.StateChange) error {
if ch.Current {
time.Sleep(40 * time.Millisecond)
}
mu.Lock()
defer mu.Unlock()
if ch.Current {
current = append(current, ch.Key)
} else {
live = append(live, ch.Key)
}
return nil
})
if err != nil {
t.Fatalf("a watch whose current values kept arriving was refused: %v", err)
}
t.Cleanup(stop)
mu.Lock()
got := append([]string(nil), current...)
mu.Unlock()
if len(got) != len(keys) {
t.Fatalf("the watch returned after %d of %d current values", len(got), len(keys))
}
for i, k := range keys {
if got[i] != k {
t.Fatalf("current value %d is %s, want %s", i, got[i], k)
}
}
// And it holds: a verdict written afterwards reaches it as a change.
next := "novox/mesh-sdk.77." + strings.Repeat("a", 40) + ".status=mesh/check"
if _, err := kv.Put(next, []byte(`{"v":1,"act":"status","context":"mesh/check","state":"pending"}`)); err != nil {
t.Fatal(err)
}
mt.Until(t, func() error {
mu.Lock()
defer mu.Unlock()
if len(live) != 1 || live[0] != next {
return fmt.Errorf("live changes so far: %v", live)
}
return nil
})
}
// A watch whose bundle takes nothing for longer than the bound is still refused, so a bundle that hangs
// is said, not waited on for ever.
func TestAWatchIsRefusedWhenNoCurrentValueIsTakenWithinTheBound(t *testing.T) {
was := bus.StateTimeout
bus.StateTimeout = 300 * time.Millisecond
t.Cleanup(func() { bus.StateTimeout = was })
conn, kv := owedWatcher(t)
for i := 0; i < 3; i++ {
if _, err := kv.Put(fmt.Sprintf("novox/hq.%d.%040x.status=mesh/check", i+1, i), []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
}
release := make(chan struct{})
t.Cleanup(func() { close(release) })
began := time.Now()
_, err := conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", func(bus.StateChange) error {
<-release
return nil
})
if err == nil || !strings.Contains(err.Error(), "did not arrive in") {
t.Fatalf("a watch whose first value was never taken: want refused, got %v", err)
}
if took := time.Since(began); took > 3*time.Second {
t.Fatalf("the refusal took %s, far past the bound", took)
}
}
// owedWatcher is the gitea module's view of the checker's owed: the bucket, and a runtime connection
// that serves gitea with the owed issued for reading.
func owedWatcher(t *testing.T) (*bus.Conn, interface {
Put(string, []byte) (uint64, error)
}) {
t.Helper()
mesh := mt.New(t)
kv := mesh.Bucket(t, "mesh-delivery-checker_owed")
gitea := mt.MembershipOf("gitea", "anchor", false, nil)
gitea.State = []bus.StateIssued{{Name: "mesh-delivery-checker.owed", Bucket: "mesh-delivery-checker_owed"}}
mesh.Issue(t, gitea)
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
conn.Follow("gitea")
mt.Until(t, func() error {
if conn.Membership("gitea") == nil {
return errors.New("gitea not issued yet")
}
return nil
})
return conn, kv
}
+126
View File
@@ -0,0 +1,126 @@
package bus
import (
"encoding/json"
"fmt"
"sort"
"strconv"
"strings"
"time"
"github.com/nats-io/nats.go"
)
// A module reading its own stream back (novox/hq ADR 0363; the operator, 2026-10-30: a build step's whole failure output
// is said on its build module's stream, and the module's tools read it there). As a log's read: one batch direct get,
// no consumer, and only a stream the module's own membership names (`streams`); the bus grants its owner that stream's
// info and direct get and no other's (mesh-sdk buscompose ownStreamGrants).
// StreamEntry is one event of a module's own stream: its sequence, its event (the subject after the stream's
// `…event.`), its body and when the bus took it.
type StreamEntry struct {
Sequence uint64 `json:"sequence"`
Event string `json:"event"`
Body json.RawMessage `json:"body"`
At time.Time `json:"at"`
}
// StreamPage is one page of a stream's read: Next is the sequence to continue from, 0 when nothing follows; Last the
// stream's last sequence.
type StreamPage struct {
Entries []StreamEntry `json:"entries"`
Next uint64 `json:"next"`
Last uint64 `json:"last"`
}
// streamBusName is a stream's local name as the bus holds it (mesh-sdk catalogue.StreamBusName): capitals, a hyphen an
// underscore; a bus name is itself.
func streamBusName(name string) string { return strings.ToUpper(strings.ReplaceAll(name, "-", "_")) }
// issuedStream is the stream of its own a module may read by a name, local or the bus's, or the reason it may not.
func (c *Conn) issuedStream(module, name string) (StreamIssued, error) {
m := c.Membership(module)
if m == nil {
return StreamIssued{}, fmt.Errorf("%s has no membership issued on %s yet, so no stream of it is readable "+
"until the mesh issues one (novox/hq ADR 0363)", module, c.node)
}
var names []string
for _, s := range m.Streams {
if s.Stream == streamBusName(name) && strings.Contains(s.Subject, "{event}") {
return s, nil
}
names = append(names, s.Stream)
}
sort.Strings(names)
issued := "none"
if len(names) > 0 {
issued = strings.Join(names, ", ")
}
return StreamIssued{}, fmt.Errorf("%s keeps no stream called %q: a module reads the streams it declares under `streams`, "+
"and no other's; it was issued: %s (novox/hq ADR 0363)", module, name, issued)
}
// StreamRead is a module's own stream's events from a sequence on, whose event names match a pattern (`*` one name, `>`
// the rest, last; empty is every event), at most limit (LogReadDefault when 0, at most LogReadMax) and about LogPageBytes.
func (c *Conn) StreamRead(module, name, events string, from uint64, limit int) (*StreamPage, error) {
s, err := c.issuedStream(module, name)
if err != nil {
return nil, err
}
switch {
case limit == 0:
limit = LogReadDefault
case limit < 0 || limit > LogReadMax:
return nil, fmt.Errorf("a read's limit is 1 to %d events, %d when none is given, not %d", LogReadMax, LogReadDefault, limit)
}
if events == "" {
events = ">"
}
parts := strings.Split(events, ".")
for i, p := range parts {
if !(p == ">" && i == len(parts)-1) && p != "*" && !logToken.MatchString(p) {
return nil, fmt.Errorf("%q is not an event pattern: names, `*` for one and `>` for the rest, last", events)
}
}
prefix, _, _ := strings.Cut(s.Subject, "{event}")
missing := func() error {
return fmt.Errorf("the stream %s is issued and not on the bus yet: the controller creates it from the catalogue "+
"on its next raise, and reads it directly once it allows that (novox/hq ADR 0363)", s.Stream)
}
msgs, next, err := c.directBatch(s.Stream, prefix+events, from, limit, s.Stream, missing)
if err != nil {
return nil, err
}
page := &StreamPage{Entries: []StreamEntry{}, Next: next}
for _, msg := range msgs {
e, err := streamEntryOf(s.Stream, prefix, msg)
if err != nil {
return nil, err
}
page.Entries = append(page.Entries, *e)
}
info, err := c.logInfo(LogIssued{Name: s.Stream, Stream: s.Stream})
if err != nil {
return nil, err
}
page.Last = info.Last
return page, nil
}
// streamEntryOf is an event a direct get answered: its event is its subject after prefix.
func streamEntryOf(stream, prefix string, msg *nats.Msg) (*StreamEntry, error) {
subject := msg.Header.Get("Nats-Subject")
event, ok := strings.CutPrefix(subject, prefix)
if !ok {
return nil, fmt.Errorf("the bus answered an event of %s on %q, which is not under %s", stream, subject, prefix)
}
seq, err := strconv.ParseUint(msg.Header.Get("Nats-Sequence"), 10, 64)
if err != nil {
return nil, fmt.Errorf("the bus answered an event of %s with no sequence", stream)
}
at, err := time.Parse(time.RFC3339Nano, msg.Header.Get("Nats-Time-Stamp"))
if err != nil {
return nil, fmt.Errorf("the bus answered event %d of %s with no time: %v", seq, stream, err)
}
return &StreamEntry{Sequence: seq, Event: event, Body: valueOf(msg.Data), At: at.UTC()}, nil
}
+367
View File
@@ -0,0 +1,367 @@
package bus
import (
"context"
"errors"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/nats-io/nats.go/jetstream"
)
// A module's own stream, as the runtime publishes to it and hears it (novox/hq ADR 0363 §8, issue 725).
//
// A module may declare a stream of its own; the controller creates it and says, in the module's
// membership, which of its events go there and on which subject (`streams`), and, in the membership of
// each module granted to hear it, which durable consumer on it to bind (`hears`). The runtime publishes and
// binds exactly as the membership says. A membership naming neither is what every membership was before:
// every event on EVENTS, and only the consumer on EVENTS bound.
// StreamIssued is one of a module's own streams as its membership names it: the events it carries, as the
// bus spells them (`*` one token, `>` the rest), and the subject an event of them is published on, `{event}`
// standing for the event.
type StreamIssued struct {
Stream string `json:"stream"`
Events []string `json:"events"`
Subject string `json:"subject"`
}
// HeardIssued is one module stream an assignment hears: the stream and the durable consumer on it the
// runtime binds. The consumer is the controller's to make, to filter and to delete.
type HeardIssued struct {
Stream string `json:"stream"`
Consumer string `json:"consumer"`
}
// eventSubject is where a module's event is published: on the subject of the first of its membership's
// streams that carries it, else on EVENTS — `mesh.mod.<module>.event.<event>`, as every event was before.
func eventSubject(m *Membership, module, event string) string {
if m != nil {
for _, s := range m.Streams {
if s.Subject == "" || !strings.Contains(s.Subject, "{event}") {
continue
}
for _, pattern := range s.Events {
if eventMatches(pattern, event) {
return strings.ReplaceAll(s.Subject, "{event}", event)
}
}
}
}
return "mesh.mod." + module + ".event." + event
}
// eventMatches says whether an event's name falls under a pattern as the bus matches subjects: `*` is one
// token, `>` (or `**`, the manifest's spelling) the one or more tokens left.
func eventMatches(pattern, event string) bool {
if pattern == "" || event == "" {
return false
}
ps, es := strings.Split(pattern, "."), strings.Split(event, ".")
for i, p := range ps {
if p == ">" || p == "**" {
return i == len(ps)-1 && len(es) > i
}
if i >= len(es) || (p != "*" && p != es[i]) {
return false
}
}
return len(ps) == len(es)
}
// RebindEvery is how often a consumer the membership names and the bus did not let the runtime bind is
// tried again: the controller makes the consumer and sends the user list carrying its grant in the
// composition that issues the membership, so one may arrive before the other (ADR 0363 §6).
var RebindEvery = 30 * time.Second
// hearing is one module's consumers: the one on EVENTS, and each on a module stream its membership names.
// Every event of all of them is handed to the module one at a time, as the consumer on EVENTS alone did.
type hearing struct {
c *Conn
js jetstream.JetStream
module string
deliver func(Envelope) error
// serial hands the module one event at a time across its consumers (novox/hq issue 387).
serial sync.Mutex
// halted is stopped, read by handle without mu: the bus's loops hand events while mu is held.
halted atomic.Bool
mu sync.Mutex
stopped bool
bound map[HeardIssued]jetstream.ConsumeContext
// made is when the bus made each bound consumer: a consumer deleted and made again under its name has
// another time, and a loop bound to the old one is not reading the new one.
made map[HeardIssued]time.Time
said map[HeardIssued]bool
// why is the last error the bus gave a consumer's loop, said when the loop ends on it.
// Guarded by whyMu, not mu: the bus hands errors from its own goroutines, possibly while mu is held.
whyMu sync.Mutex
why map[HeardIssued]error
kick chan struct{}
done chan struct{}
unhear func()
}
func newHearing(c *Conn, js jetstream.JetStream, module string, deliver func(Envelope) error) *hearing {
return &hearing{c: c, js: js, module: module, deliver: deliver,
bound: map[HeardIssued]jetstream.ConsumeContext{}, made: map[HeardIssued]time.Time{}, said: map[HeardIssued]bool{}, why: map[HeardIssued]error{},
kick: make(chan struct{}, 1), done: make(chan struct{})}
}
// bind reads one durable consumer by name, and on failure says what it was doing. Errors the bus gives
// the loop afterwards are handed to onErr, when given.
func (h *hearing) bind(stream, durable string, onErr func(error)) (jetstream.ConsumeContext, time.Time, string, error) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
consumer, err := h.js.Consumer(ctx, stream, durable)
if err != nil {
return nil, time.Time{}, "binding", err
}
var created time.Time
if info := consumer.CachedInfo(); info != nil {
created = info.Created
}
opts := []jetstream.PullConsumeOpt{jetstream.PullMaxMessages(1)}
if onErr != nil {
opts = append(opts, jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { onErr(err) }))
}
reading, err := consumer.Consume(h.handle, opts...)
if err != nil {
return nil, time.Time{}, "reading", err
}
return reading, created, "", nil
}
// handle hands one delivered event to the module: acknowledged when it took it, offered again after NakDelay
// when it did not, terminated when it is not an event at all.
func (h *hearing) handle(msg jetstream.Msg) {
env, ok := envelopeOf(msg.Subject(), msg.Headers(), msg.Data())
if !ok {
// Unparseable: redelivering bytes no version of a handler can read is a loop.
_ = msg.Term()
return
}
// Kept alive while the bundle works on it, as a worker's work is (seat.go): the consumer's ack wait
// (30 seconds) is for a runtime that died, not a handler that is slow — the runtime gives a handler
// launch.EventTimeout. Without it an event handled longer than the ack wait was offered again while
// it was handled, and counted toward max-deliver (novox/hq issue 387). Kept alive too while it waits
// for an event from another of the module's consumers to be handled.
done, stopped := make(chan struct{}), make(chan struct{})
go func() {
defer close(stopped)
tick := time.NewTicker(EventHeartbeat)
defer tick.Stop()
for {
select {
case <-done:
return
case <-tick.C:
_ = msg.InProgress()
}
}
}()
h.serial.Lock()
halted := h.halted.Load()
var err error
if !halted {
err = h.deliver(env)
}
h.serial.Unlock()
close(done)
// No in-progress after the answer: one after a Nak would stretch its delay to the whole ack wait.
<-stopped
if halted {
// **A hearing stopped hands nothing on** (issue 1945): the module's bundle that subscribed may be
// gone, loaded again in its place; offered again at once, to the consumer bound after it.
_ = msg.Nak()
return
}
if err != nil {
_ = msg.NakWithDelay(NakDelay)
return
}
_ = msg.Ack()
}
// start binds what the membership names now, and follows it: a new membership binds what it adds and stops
// what it no longer names, and a consumer not bound yet is tried again every RebindEvery.
func (h *hearing) start() {
h.reconcile()
h.unhear = h.c.onMembership(func(m Membership) {
if m.Module != h.module {
return
}
h.nudge()
})
go func() {
tick := time.NewTicker(RebindEvery)
defer tick.Stop()
for {
select {
case <-h.done:
return
case <-h.kick:
case <-tick.C:
}
h.reconcile()
}
}()
}
// forget drops the last error a consumer's loop was given.
func (h *hearing) forget(heard HeardIssued) {
h.whyMu.Lock()
delete(h.why, heard)
h.whyMu.Unlock()
}
// nudge asks the loop to reconcile now.
func (h *hearing) nudge() {
select {
case h.kick <- struct{}{}:
default:
}
}
// stop stops every consumer on a module stream this module reads; the one on EVENTS is its caller's.
func (h *hearing) stop() {
h.mu.Lock()
defer h.mu.Unlock()
if h.stopped {
return
}
h.stopped = true
h.halted.Store(true)
close(h.done)
if h.unhear != nil {
h.unhear()
}
for heard, reading := range h.bound {
reading.Stop()
delete(h.bound, heard)
delete(h.made, heard)
}
}
// replaced says whether the consumer a loop is bound to is gone from the bus or was made again since. A bus
// that cannot be asked right now says no: the loop is left as it is.
func (h *hearing) replaced(heard HeardIssued) bool {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
consumer, err := h.js.Consumer(ctx, heard.Stream, heard.Consumer)
if errors.Is(err, jetstream.ErrConsumerNotFound) {
return true
}
if err != nil || consumer.CachedInfo() == nil {
return false
}
created := h.made[heard]
return !created.IsZero() && !consumer.CachedInfo().Created.Equal(created)
}
// reconcile makes what is bound what the module's membership names now.
func (h *hearing) reconcile() {
want := map[HeardIssued]bool{}
if m := h.c.Membership(h.module); m != nil {
for _, heard := range m.Hears {
if heard.Stream != "" && heard.Consumer != "" {
want[heard] = true
}
}
}
h.mu.Lock()
defer h.mu.Unlock()
if h.stopped {
return
}
for heard, reading := range h.bound {
if !want[heard] {
reading.Stop()
delete(h.bound, heard)
delete(h.made, heard)
h.c.Logf("[mesh-tools] %s no longer hears %s: its membership stopped naming its consumer %s",
h.module, heard.Stream, heard.Consumer)
}
}
for heard := range h.said {
if !want[heard] {
delete(h.said, heard)
h.forget(heard)
}
}
// **A loop the bus ended is no longer reading** (nats.go ends a pull loop on its own when its consumer is
// deleted): the controller deletes and makes again under the same name a restricted consumer it cannot
// narrow (ADR 0363 §6), and the membership, unchanged, says nothing — so it is bound again here.
// **A loop the bus did not end is asked about**: nats.go learns of a deleted consumer only from the answer
// to a pull it has pending, and a consumer deleted and made again between two pulls, or while none is
// pending, leaves the loop waiting out its heartbeat (30 seconds) on a consumer that is gone. So each
// bound consumer is looked up by name: gone, or made at another time, ends its loop here.
for heard, reading := range h.bound {
if h.replaced(heard) {
h.whyMu.Lock()
if h.why[heard] == nil {
h.why[heard] = errors.New("the consumer was deleted or made again")
}
h.whyMu.Unlock()
reading.Stop()
}
}
for heard, reading := range h.bound {
select {
case <-reading.Closed():
delete(h.bound, heard)
delete(h.made, heard)
h.whyMu.Lock()
why := h.why[heard]
h.whyMu.Unlock()
if why == nil {
why = errors.New("no reason given")
}
h.c.Logf("[mesh-tools] %s stopped hearing %s: the bus ended reading its consumer %s (%v); bound again as it exists",
h.module, heard.Stream, heard.Consumer, why)
h.said[heard] = true
default:
}
}
missing := make([]HeardIssued, 0, len(want))
for heard := range want {
if h.bound[heard] == nil {
missing = append(missing, heard)
}
}
sort.Slice(missing, func(i, j int) bool { return missing[i].Stream < missing[j].Stream })
for _, heard := range missing {
heard := heard
h.forget(heard)
reading, created, doing, err := h.bind(heard.Stream, heard.Consumer, func(err error) {
h.whyMu.Lock()
h.why[heard] = err
h.whyMu.Unlock()
})
if err != nil {
// Said once, not at every attempt; said again when it is bound.
if !h.said[heard] {
h.c.Logf("[mesh-tools] %s does not hear %s yet: %s its consumer %s failed: %v; tried again every %s",
h.module, heard.Stream, doing, heard.Consumer, err, RebindEvery)
h.said[heard] = true
}
continue
}
h.bound[heard] = reading
h.made[heard] = created
delete(h.said, heard)
h.c.Logf("[mesh-tools] %s hears %s on its consumer %s", h.module, heard.Stream, heard.Consumer)
// When the bus ends the loop, look again at once rather than at the next tick.
go func() {
select {
case <-reading.Closed():
h.nudge()
case <-h.done:
}
}()
}
}
@@ -0,0 +1,48 @@
package bus
import "testing"
// novox/hq ADR 0363 §2, §3: an event goes to the stream whose pattern carries it, as the bus matches subjects.
func TestAnEventFallsUnderAStreamsPatternAsTheBusMatchesIt(t *testing.T) {
for _, c := range []struct {
pattern, event string
want bool
}{
{"mail.*.message.added", "mail.home.message.added", true},
{"mail.*.message.added", "mail.home.message.removed", false},
{"mail.*.message.added", "mail.home.work.message.added", false},
{"mail.>", "mail.home.message.added", true},
{"mail.**", "mail.home", true},
{"mail.>", "mail", false},
{"status", "status", true},
{"status", "status.changed", false},
{"", "status", false},
} {
if got := eventMatches(c.pattern, c.event); got != c.want {
t.Errorf("%q under %q: %v, want %v", c.event, c.pattern, got, c.want)
}
}
}
// A membership naming no stream publishes every event on EVENTS, exactly as before (issue 725).
func TestAnEventNoStreamCarriesIsPublishedOnEvents(t *testing.T) {
mail := &Membership{Module: "mail", Streams: []StreamIssued{{Stream: "MAIL", Events: []string{"mail.*.message.added"},
Subject: "mesh.stream.mail.event.{event}"}}}
for _, c := range []struct {
m *Membership
event string
want string
}{
{nil, "mail.home.message.added", "mesh.mod.mail.event.mail.home.message.added"},
{&Membership{Module: "mail"}, "mail.home.message.added", "mesh.mod.mail.event.mail.home.message.added"},
{mail, "mail.home.message.added", "mesh.stream.mail.event.mail.home.message.added"},
{mail, "status", "mesh.mod.mail.event.status"},
} {
if got := eventSubject(c.m, "mail", c.event); got != c.want {
t.Errorf("%s: %s, want %s", c.event, got, c.want)
}
}
if got := keyOf("mesh.stream.mail.event.mail.home.message.added"); got != "mail.mail.home.message.added" {
t.Errorf("a stream's event reaches its consumer as %q, not as on EVENTS", got)
}
}
+280
View File
@@ -0,0 +1,280 @@
package bus_test
import (
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// streamMesh is a mesh with the mail module's own stream MAIL, and a connection on anchor following the
// memberships of the modules named.
func streamMesh(t *testing.T, memberships ...bus.Membership) (*mt.Mesh, *bus.Conn) {
t.Helper()
mesh := mt.New(t)
mesh.ModuleStream(t, "mail", "MAIL")
for _, m := range memberships {
mesh.Issue(t, m)
}
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
for _, m := range memberships {
conn.Follow(m.Module)
}
mt.Until(t, func() error {
for _, m := range memberships {
if conn.Membership(m.Module) == nil {
return errors.New(m.Module + " not issued yet")
}
}
return nil
})
return mesh, conn
}
// novox/hq ADR 0363 §2, §8 (issue 725): an event one of the module's streams carries is published on that
// stream's subject and kept by that stream alone; any other event of it goes to EVENTS as before.
func TestAnEventADeclaredStreamCarriesIsPublishedOnItsSubjectAlone(t *testing.T) {
mail := mt.MembershipOf("mail", "anchor", true, nil)
mail.Streams = []bus.StreamIssued{{Stream: "MAIL", Events: []string{"mail.*.message.added"},
Subject: "mesh.stream.mail.event.{event}"}}
mesh, conn := streamMesh(t, mail)
landed := mesh.NextEvent(t, "mesh.stream.mail.event.>")
if err := conn.PublishAs("mail", bus.Envelope{Key: "mail.home.message.added", Body: json.RawMessage(`{"id":1}`)}); err != nil {
t.Fatal(err)
}
select {
case got := <-landed:
if got != "mesh.stream.mail.event.mail.home.message.added" {
t.Errorf("published on %s", got)
}
case <-time.After(5 * time.Second):
t.Fatal("the event did not land on its stream's subject")
}
if err := conn.PublishAs("mail", bus.Envelope{Key: "status", Body: json.RawMessage(`{}`)}); err != nil {
t.Fatal(err)
}
if n := mesh.Messages(t, "MAIL"); n != 1 {
t.Errorf("MAIL holds %d messages, want the one event it carries", n)
}
if n := mesh.Messages(t, "EVENTS"); n != 1 {
t.Errorf("EVENTS holds %d messages, want only the event no stream carries", n)
}
}
// novox/hq ADR 0363 §8 (issue 725): a module hears, beside EVENTS, every module stream its membership names,
// one event at a time, with the key it reads on EVENTS; a membership naming none hears EVENTS alone.
func TestAModuleHearsEveryStreamItsMembershipNames(t *testing.T) {
reader := mt.MembershipOf("reader", "anchor", false, nil)
mesh, conn := streamMesh(t, reader)
mesh.Consumer(t, "anchor", "reader", []string{"mesh.mod.alpha.event.>"}, 5*time.Second)
mesh.ConsumerOn(t, "MAIL", "anchor", "reader", []string{"mesh.stream.mail.event.mail.home.>"}, 5*time.Second)
var mu sync.Mutex
heard := map[string]int{}
handling := 0
stop, err := conn.ConsumeAs("reader", func(env bus.Envelope) error {
mu.Lock()
handling++
if handling > 1 {
t.Error("two events were handed to the module at once")
}
mu.Unlock()
time.Sleep(50 * time.Millisecond)
mu.Lock()
handling--
heard[env.Key]++
mu.Unlock()
return nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
count := func(key string) int {
mu.Lock()
defer mu.Unlock()
return heard[key]
}
// Its membership names no stream: EVENTS alone, as before.
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 1})
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": 1})
mt.Until(t, func() error {
if count("alpha.happened") != 1 {
return errors.New("the event on EVENTS was not heard")
}
return nil
})
if _, notYet := mesh.PendingOn(t, "MAIL", "anchor", "reader"); notYet != 1 || count("mail.mail.home.message.added") != 0 {
t.Fatalf("a module whose membership names no stream heard MAIL (%d not yet delivered)", notYet)
}
// Its membership now names its consumer on MAIL: bound, and what waited there is heard.
reader.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "anchor_reader"}}
mesh.Issue(t, reader)
mt.Until(t, func() error {
if count("mail.mail.home.message.added") != 1 {
return errors.New("the event on MAIL was not heard")
}
return nil
})
for n := 2; n <= 4; n++ {
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": n})
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": n})
}
mt.Until(t, func() error {
if a, m := count("alpha.happened"), count("mail.mail.home.message.added"); a != 4 || m != 4 {
return fmt.Errorf("heard %d on EVENTS and %d on MAIL, want 4 and 4", a, m)
}
for _, stream := range []string{"EVENTS", "MAIL"} {
if ack, notYet := mesh.PendingOn(t, stream, "anchor", "reader"); ack != 0 || notYet != 0 {
return fmt.Errorf("%s still holds %d unacknowledged, %d undelivered", stream, ack, notYet)
}
}
return nil
})
// A membership that stops naming it stops what it reads there.
reader.Hears = nil
mesh.Issue(t, reader)
mt.Until(t, func() error {
if mesh.Consumers(t, "MAIL") != 1 {
return errors.New("MAIL's consumer is gone")
}
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": 0})
time.Sleep(300 * time.Millisecond)
if _, notYet := mesh.PendingOn(t, "MAIL", "anchor", "reader"); notYet == 0 {
return errors.New("MAIL is still read")
}
return nil
})
}
// novox/hq ADR 0363 §6 (issue 725): a consumer the membership names before the controller made it is bound once
// it exists, without the module's events on EVENTS waiting for it.
func TestAConsumerNamedBeforeItExistsIsBoundOnceItDoes(t *testing.T) {
restore := bus.RebindEvery
bus.RebindEvery = 200 * time.Millisecond
t.Cleanup(func() { bus.RebindEvery = restore })
reader := mt.MembershipOf("reader", "anchor", false, nil)
reader.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "anchor_reader"}}
mesh, conn := streamMesh(t, reader)
mesh.Consumer(t, "anchor", "reader", []string{"mesh.mod.alpha.event.>"}, 5*time.Second)
var mu sync.Mutex
heard := map[string]int{}
stop, err := conn.ConsumeAs("reader", func(env bus.Envelope) error {
mu.Lock()
heard[env.Key]++
mu.Unlock()
return nil
})
if err != nil {
t.Fatalf("a consumer on a module stream not made yet kept the module from hearing EVENTS: %v", err)
}
t.Cleanup(stop)
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{})
mesh.ConsumerOn(t, "MAIL", "anchor", "reader", []string{"mesh.stream.mail.event.>"}, 5*time.Second)
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{})
mt.Until(t, func() error {
mu.Lock()
defer mu.Unlock()
if heard["alpha.happened"] != 1 || heard["mail.mail.home.message.added"] != 1 {
return fmt.Errorf("heard %v", heard)
}
return nil
})
}
// novox/hq ADR 0363 §6 (issue 725): a consumer the controller deletes and makes again under the same name, its
// membership unchanged, is bound again, and what is published after is heard — the module does not go deaf.
func TestAConsumerDeletedAndMadeAgainIsHeardAgain(t *testing.T) {
restore := bus.RebindEvery
bus.RebindEvery = 200 * time.Millisecond
t.Cleanup(func() { bus.RebindEvery = restore })
reader := mt.MembershipOf("reader", "anchor", false, nil)
reader.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "anchor_reader"}}
mesh, conn := streamMesh(t, reader)
mesh.Consumer(t, "anchor", "reader", []string{"mesh.mod.alpha.event.>"}, 5*time.Second)
mesh.ConsumerOn(t, "MAIL", "anchor", "reader", []string{"mesh.stream.mail.event.>"}, 5*time.Second)
logs := &syncLogs{}
conn.Logf = logs.Logf
var mu sync.Mutex
heard := 0
stop, err := conn.ConsumeAs("reader", func(env bus.Envelope) error {
if env.Key == "mail.mail.home.message.added" {
mu.Lock()
heard++
mu.Unlock()
}
return nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
count := func() int {
mu.Lock()
defer mu.Unlock()
return heard
}
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": 1})
mt.Until(t, func() error {
if count() != 1 {
return errors.New("the first event on MAIL was not heard")
}
return nil
})
mesh.RemoveConsumer(t, "MAIL", "anchor", "reader")
mt.Until(t, func() error {
if !logs.Has("stopped hearing MAIL") {
return errors.New("the end of the loop was not said")
}
return nil
})
mesh.ConsumerOn(t, "MAIL", "anchor", "reader", []string{"mesh.stream.mail.event.>"}, 5*time.Second)
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": 2})
mt.Until(t, func() error {
if count() != 2 {
return fmt.Errorf("heard %d events on MAIL after its consumer was made again, want 2", count())
}
return nil
})
}
// syncLogs keeps what the runtime says, from any goroutine.
type syncLogs struct {
mu sync.Mutex
lines []string
}
func (l *syncLogs) Logf(format string, args ...any) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, fmt.Sprintf(format, args...))
}
func (l *syncLogs) Has(fragment string) bool {
l.mu.Lock()
defer l.mu.Unlock()
for _, line := range l.lines {
if strings.Contains(line, fragment) {
return true
}
}
return false
}
+132
View File
@@ -0,0 +1,132 @@
package bus
import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
)
// Waiting on the bus for one event (novox/hq ADR 0367): what the mesh MCP server's `mesh_wait` does.
//
// **A subscription for the call, and nothing kept** (§2): no durable consumer, no queue, no stream. It
// subscribes the subjects when called and unsubscribes them before it answers, so an event published
// between two calls is not heard — a session reads the state from its owner after every wake (§4).
// WaitIssued is one event family the mesh issued this account to wait on (ADR 0367 §2): the name a session
// asks for, and the subject it lands on. Only the tool runner's membership carries any.
type WaitIssued struct {
Event string `json:"event"`
Subject string `json:"subject"`
}
// Heard is one event a wait heard: the subject it arrived on, its body and when it arrived.
type Heard struct {
Subject string
Body json.RawMessage
At time.Time
}
// HearFirst subscribes the subjects for this call alone and answers the first message whose body match
// accepts (every one, when match is nil), or ok false once `within` has passed. Every subscription is
// closed before it returns. A subscription the bus refuses is said as an error naming the subject, at once;
// a caller gone — ctx done — ends the wait at once with ctx's error, so nothing waits on for nobody.
func (c *Conn) HearFirst(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (heard Heard, ok bool, err error) {
if len(subjects) == 0 {
return Heard{}, false, fmt.Errorf("a wait names at least one subject")
}
got := make(chan Heard, 1)
refused := make(chan string, len(subjects))
var subs []*nats.Subscription
defer func() {
for _, s := range subs {
_ = s.Unsubscribe()
}
c.mu.Lock()
for _, subject := range subjects {
delete(c.waits[subject], refused)
if len(c.waits[subject]) == 0 {
delete(c.waits, subject)
}
}
c.mu.Unlock()
}()
c.mu.Lock()
if c.waits == nil {
c.waits = map[string]map[chan string]bool{}
}
for _, subject := range subjects {
if c.waits[subject] == nil {
c.waits[subject] = map[chan string]bool{}
}
c.waits[subject][refused] = true
}
c.mu.Unlock()
for _, subject := range subjects {
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
body := json.RawMessage(msg.Data)
if !json.Valid(body) {
quoted, _ := json.Marshal(string(msg.Data))
body = quoted
}
if match != nil && !match(body) {
return
}
select {
case got <- Heard{Subject: msg.Subject, Body: body, At: time.Now().UTC()}:
default:
}
})
if err != nil {
return Heard{}, false, fmt.Errorf("could not wait on %s: %w", subject, err)
}
subs = append(subs, sub)
}
// The bus has every subscription — or has said which it refused — once this returns.
if err := c.nc.FlushTimeout(5 * time.Second); err != nil {
return Heard{}, false, fmt.Errorf("the bus did not take the wait: %w", err)
}
timer := time.NewTimer(within)
defer timer.Stop()
select {
case h := <-got:
return h, true, nil
case subject := <-refused:
return Heard{}, false, fmt.Errorf("the bus refused this account a subscription on %s: it is not in the "+
"account's wait grant (novox/hq ADR 0367 §2)", subject)
case <-timer.C:
return Heard{}, false, nil
case <-ctx.Done():
return Heard{}, false, ctx.Err()
}
}
// waitRefused tells every wait on a subject the bus refused that it was refused; true when one was waiting.
func (c *Conn) waitRefused(subject string) bool {
c.mu.Lock()
defer c.mu.Unlock()
told := false
for ch := range c.waits[subject] {
select {
case ch <- subject:
default:
}
told = true
}
return told
}
// Subscriptions is how many subscriptions this connection holds now: what a wait leaves behind is none.
func (c *Conn) Subscriptions() int { return c.nc.NumSubscriptions() }
// subjectRefused is the subject a subscription refusal names, or "".
func subjectRefused(err error) string {
m := refusedSubject.FindStringSubmatch(err.Error())
if len(m) < 2 {
return ""
}
return strings.Trim(m[1], `"`)
}
+83
View File
@@ -0,0 +1,83 @@
package bus_test
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq ADR 0367 §2, §5: the bus refuses the tool runner's account any subject its wait grant does not
// name — another asker's `decided` among them — and a wait on one says the bus refused it, at once.
func TestTheBusRefusesAWaitOutsideTheGrant(t *testing.T) {
granted := "mesh.seat.operator-channel.event.decided.claude-code"
s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, NoLog: true, NoSigs: true,
Users: []*server.User{{Username: "desk.node-tools", Password: "pw", Permissions: &server.Permissions{
Publish: &server.SubjectPermission{Allow: []string{"$JS.API.DIRECT.GET.>", "mesh.>"}},
Subscribe: &server.SubjectPermission{Allow: []string{granted, "_INBOX.desk.node-tools.>", "mesh.assignment.desk.*"}},
}}}})
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(s.Shutdown)
c := dial(t, s.ClientURL(), "node-tools", "desk", "desk.node-tools")
start := time.Now()
_, ok, err := c.HearFirst(context.Background(), []string{"mesh.seat.operator-channel.event.decided.shop"}, 5*time.Second, nil)
if ok || err == nil || !strings.Contains(err.Error(), "refused") {
t.Errorf("a wait on another asker's decided: heard %v, %v", ok, err)
}
if time.Since(start) > 3*time.Second {
t.Errorf("the refusal took %s to be said", time.Since(start))
}
done := make(chan bus.Heard, 1)
go func() {
h, ok, err := c.HearFirst(context.Background(), []string{granted}, 5*time.Second, func(body json.RawMessage) bool { return true })
if ok && err == nil {
done <- h
}
}()
time.Sleep(300 * time.Millisecond)
if err := c.Say(granted, []byte(`{"ask":"a1"}`)); err != nil {
t.Fatal(err)
}
select {
case h := <-done:
if h.Subject != granted || string(h.Body) != `{"ask":"a1"}` {
t.Errorf("heard %+v", h)
}
case <-time.After(5 * time.Second):
t.Fatal("a wait on the granted subject heard nothing")
}
}
// A wait whose caller is gone ends at once, its subscription closed (the review of issue 745).
func TestAWaitEndsWhenItsCallerIsGone(t *testing.T) {
c := dial(t, mt.URL(t), "node-tools", "desk", "")
before := c.Subscriptions()
ctx, cancel := context.WithCancel(context.Background())
time.AfterFunc(200*time.Millisecond, cancel)
start := time.Now()
_, ok, err := c.HearFirst(ctx, []string{"mesh.mod.mesh-knowledge.event.reviewed"}, time.Minute, nil)
if ok || !errors.Is(err, context.Canceled) {
t.Errorf("a cancelled wait answered %v, %v", ok, err)
}
if took := time.Since(start); took > 2*time.Second {
t.Errorf("a cancelled wait ended after %s", took)
}
if after := c.Subscriptions(); after != before {
t.Errorf("%d subscriptions before, %d after", before, after)
}
}
+12 -6
View File
@@ -2,8 +2,8 @@ package console
// The mesh's tools found by address, not announced whole (novox/hq ADR 0195, to-be 34 §3a).
//
// The console announces six tools — five to find and call, and mesh_runtimes to say which runtimes
// discovery heard. Everything the mesh answers is reached through them by one address per layer:
// The console announces seven tools — five to find and call, mesh_runtimes to say which runtimes
// discovery heard, and mesh_wait to wait on the bus for an event (novox/hq ADR 0367). Everything the mesh answers is reached through them by one address per layer:
//
// <seat>.<verb> a seat held once for the mesh — its holder answers
// <node>/<seat>.<verb> a seat held once per machine — that machine's holder answers
@@ -17,6 +17,7 @@ package console
// session — so a tool that arrived a minute ago is found without the client reconnecting.
import (
"context"
"encoding/json"
"fmt"
"sort"
@@ -32,7 +33,7 @@ import (
// one agent turn's search, describe and call ask once, short enough that nothing goes stale.
var IndexKept = 5 * time.Second
// The six tools the console announces. Names of the API's kind — letters, digits, `_`, `-`.
// The seven tools the console announces. Names of the API's kind — letters, digits, `_`, `-`.
const (
verbOverview = "mesh_overview"
verbMachine = "mesh_machine"
@@ -40,6 +41,7 @@ const (
verbDescribe = "mesh_describe"
verbCall = "mesh_call"
verbRuntimes = "mesh_runtimes"
verbWait = "mesh_wait"
)
// searchCap is how many matches a search answers before it says how many more there were.
@@ -50,7 +52,7 @@ const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e
"`<node>/<module>.<tool>` for a module on one machine (e.g. `novox/postgres.postgres_list_databases`); " +
"and `<module>.<tool>` also for a module whose instances are interchangeable."
// discovery is the six tools as tools/list announces them.
// discovery is the seven tools as tools/list announces them.
func discovery() []map[string]any {
str := func(desc string) map[string]any { return map[string]any{"type": "string", "description": desc} }
obj := func(props map[string]any, required ...string) map[string]any {
@@ -84,12 +86,13 @@ func discovery() []map[string]any {
"arrive, its size in bytes, how many modules and tools it announced, whether it was shortened to fit the " +
"bus, and when it was last heard — and every runtime or machine expected that was not heard. Read-only; " +
"for when an address is said to be missing or on another machine."},
waitTool(),
}
}
func isDiscovery(name string) bool {
switch name {
case verbOverview, verbMachine, verbSearch, verbDescribe, verbCall, verbRuntimes:
case verbOverview, verbMachine, verbSearch, verbDescribe, verbCall, verbRuntimes, verbWait:
return true
}
return false
@@ -815,10 +818,13 @@ func failure(text string) map[string]any {
}
// discover answers one of the discovery verbs.
func (s *Surface) discover(name string, args map[string]any) map[string]any {
func (s *Surface) discover(ctx context.Context, name string, args map[string]any) map[string]any {
if name == verbRuntimes {
return s.runtimesAnswer()
}
if name == verbWait {
return s.waitAnswer(ctx, args)
}
x, err := s.index()
if err != nil {
return failure("the mesh's discovery failed: " + err.Error())
+4 -3
View File
@@ -36,9 +36,10 @@ func call(t *testing.T, endpoint, tool string, args map[string]any) (string, boo
return text(t, post(t, endpoint, string(body)))
}
// novox/hq ADR 0195: the console announces six tools, and everything the mesh answers is reached
// novox/hq ADR 0195 and ADR 0367: the console announces seven tools, and everything the mesh answers is reached
// through them by one address per layer.
func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
mt.NeedBundles(t)
was := IndexKept
IndexKept = 0 // every discovery asks the mesh, so a module arriving mid-test is found
t.Cleanup(func() { IndexKept = was })
@@ -113,7 +114,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
defer up.Close()
endpoint := "http://" + up.Address + "/mcp"
// Six tools, nothing else.
// Seven tools, nothing else (ADR 0367).
listed := post(t, endpoint, `{"jsonrpc":"2.0","id":2,"method":"tools/list"}`)["result"].(map[string]any)
var names []string
for _, x := range listed["tools"].([]any) {
@@ -125,7 +126,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
}
}
}
if got := strings.Join(names, ","); got != "mesh_overview,mesh_machine,mesh_search,mesh_describe,mesh_call,mesh_runtimes" {
if got := strings.Join(names, ","); got != "mesh_overview,mesh_machine,mesh_search,mesh_describe,mesh_call,mesh_runtimes,mesh_wait" {
t.Errorf("announced %s", got)
}
+4 -2
View File
@@ -7,6 +7,8 @@ import (
"regexp"
"strings"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
@@ -109,7 +111,6 @@ func subjectListed(name, node string, l *Listing) string {
}
var (
noResponders = regexp.MustCompile(`(?i)no responders|503`)
// The bus's own two refusals, by their whole phrase. "authorization" alone also matched a tool's own
// answer that merely mentions the word — the runtime refusing a state value with an Authorization
// header (novox/hq ADR 0201) read as "this account may not call", which sent the reader the wrong way.
@@ -127,7 +128,8 @@ func whyItFailed(key string, err error) string {
}
msg := err.Error()
switch {
case noResponders.MatchString(msg):
// By type, not text: a tool's own error that mentions 503 is the tool's answer (issue 1529).
case errors.Is(err, bus.ErrNoResponders) || errors.Is(err, nats.ErrNoResponders):
extra := ""
if strings.HasPrefix(key, "seat:") {
extra = ", or nothing holds that seat"
@@ -42,6 +42,7 @@ func post(t *testing.T, endpoint string, body string) map[string]any {
// As node-tools, the runtime serves the bundles and is the console on loopback: the listing is what
// the modules and the mesh's records answered, and a call reaches the module on the machine named.
func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "desk", true, nil))
mesh.Issue(t, mt.MembershipOf("beta", "desk", false, map[string][]string{"node-shelf": {"list", "clear"}}))
+23 -4
View File
@@ -10,6 +10,7 @@ import (
"net/http"
"strconv"
"strings"
"time"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
@@ -45,12 +46,30 @@ func Serve(s *Surface, listen string) (*Listening, error) {
if err != nil {
return nil, err
}
server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { route(w, r, s) })}
// Every request's context ends when the console is closed, so a mesh_wait in flight ends at once rather
// than holding a stop or a restart for up to its ten minutes (the review of issue 745).
base, cancel := context.WithCancel(context.Background())
server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { route(w, r, s) }),
BaseContext: func(net.Listener) context.Context { return base }}
go func() { _ = server.Serve(ln) }()
bound := ln.Addr().(*net.TCPAddr).Port
return &Listening{Address: host + ":" + strconv.Itoa(bound), Close: func() error { return server.Shutdown(context.Background()) }}, nil
return &Listening{Address: host + ":" + strconv.Itoa(bound), Close: func() error {
cancel()
// A backstop: what is still answering after the cancel is cut off, never waited on for long.
ctx, done := context.WithTimeout(context.Background(), ShutdownWithin)
defer done()
err := server.Shutdown(ctx)
if errors.Is(err, context.DeadlineExceeded) {
return server.Close()
}
return err
}}, nil
}
// ShutdownWithin is how long closing the console waits for the requests still being answered after their
// context was cancelled.
var ShutdownWithin = 5 * time.Second
func route(w http.ResponseWriter, r *http.Request, s *Surface) {
switch r.URL.Path {
case "/":
@@ -89,7 +108,7 @@ func route(w http.ResponseWriter, r *http.Request, s *Surface) {
}
replies := []*Reply{}
for _, req := range batch {
if reply := s.Handle(req); reply != nil {
if reply := s.HandleContext(r.Context(), req); reply != nil {
replies = append(replies, reply)
}
}
@@ -105,7 +124,7 @@ func route(w http.ResponseWriter, r *http.Request, s *Surface) {
writeJSON(w, 400, map[string]any{"jsonrpc": "2.0", "id": nil, "error": map[string]any{"code": -32700, "message": "the body is not JSON"}})
return
}
reply := s.Handle(req)
reply := s.HandleContext(r.Context(), req)
if reply == nil {
w.WriteHeader(202)
return
@@ -0,0 +1,41 @@
package console
import (
"encoding/json"
"os"
"reflect"
"strings"
"testing"
)
// novox/hq ADR 0367 §5: the tool runner's manifest names the families a session waits on today, one by one,
// and the asker's `decided` for the claude-code module alone. Adding one is a reviewed change to this file
// (§6), so this list changes with it. The delivery's transitions are waited on under the gate's name alone (ADR 0450 §8, issues 1350
// and 1406): today's module is forgotten.
func TestTheManifestNamesTheFirstFamiliesOneByOne(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Module string `json:"module"`
Waits []string `json:"waits"`
Consumes []string `json:"consumes"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
want := []string{"mesh-knowledge.reviewed", "mesh-delivery-gate.transition", "gitea.forge.status",
"mesh-controller.checked", "operator-channel.decided.claude-code"}
if m.Module != "node-tools" || !reflect.DeepEqual(m.Waits, want) {
t.Errorf("%s waits on %v, want %v", m.Module, m.Waits, want)
}
for _, w := range m.Waits {
if strings.ContainsAny(w, "*>") {
t.Errorf("a family named by a wildcard: %s", w)
}
}
if len(m.Consumes) > 0 {
t.Errorf("the tool runner consumes %v: a wait is no consumer (ADR 0367 §2)", m.Consumes)
}
}
+9 -3
View File
@@ -4,6 +4,7 @@
package console
import (
"context"
"encoding/json"
"os"
"strings"
@@ -103,7 +104,11 @@ func idOrNull(id json.RawMessage) json.RawMessage {
}
// Handle answers one request; nil for a notification, which expects none.
func (s *Surface) Handle(r Request) *Reply {
func (s *Surface) Handle(r Request) *Reply { return s.HandleContext(context.Background(), r) }
// HandleContext answers one request for as long as ctx lasts: a wait whose client hung up or cancelled
// ends at once, its subscription closed (the review of issue 745).
func (s *Surface) HandleContext(ctx context.Context, r Request) *Reply {
notification := isNotification(r.ID)
switch r.Method {
case "initialize":
@@ -164,7 +169,7 @@ func (s *Surface) Handle(r Request) *Reply {
if p.Arguments == nil {
p.Arguments = map[string]any{}
}
return answer(r.ID, s.discover(p.Name, p.Arguments))
return answer(r.ID, s.discover(ctx, p.Name, p.Arguments))
}
args := map[string]any{}
for k, v := range p.Arguments {
@@ -305,7 +310,8 @@ func (s *Surface) instructions() string {
"whole (novox/hq ADR 0195). mesh_overview shows the mesh's seats and machines; mesh_machine one " +
"machine's seats and modules; mesh_search finds a tool by words; mesh_describe gives one tool's " +
"arguments; mesh_call calls it; mesh_runtimes says which machines' runtimes answered discovery, how fast " +
"and how much — ask it when an address is said to be missing or on another machine. " + grammar + " What may be called was fixed when this account " +
"and how much — ask it when an address is said to be missing or on another machine; mesh_wait waits on the " +
"bus for an event instead of polling (novox/hq ADR 0367). " + grammar + " What may be called was fixed when this account " +
"was issued, so a refusal means the account, not the tool."
}
@@ -0,0 +1,28 @@
package console
import (
"errors"
"fmt"
"strings"
"testing"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// A tool's own error that mentions 503 is the tool's answer, not a routing fault (issue 1529).
func TestAToolsOwn503IsNotNothingServes(t *testing.T) {
got := whyItFailed("scanner.scan", errors.New("the scanner refused the scan job: 503 Service Unavailable"))
if strings.Contains(got, "nothing serves") || !strings.HasPrefix(got, "scanner.scan failed: ") || !strings.Contains(got, "503") {
t.Fatalf("got %q", got)
}
}
func TestARealNoRespondersStillSaysNothingServes(t *testing.T) {
for _, err := range []error{bus.ErrNoResponders, fmt.Errorf("ask: %w", bus.ErrNoResponders), nats.ErrNoResponders} {
if got := whyItFailed("a.b", err); !strings.HasPrefix(got, "nothing serves a.b") {
t.Fatalf("%v: got %q", err, got)
}
}
}
+29 -1
View File
@@ -44,6 +44,31 @@ var synonyms = map[string][]string{
"hosts": {"hostname"},
}
// filler are the function words a question carries and no tool is named for: a query's words must all
// match, so "did our weekly upgrade run on all our nodes" would otherwise find nothing for "did", "our",
// "on" and "all". Kept short and here only; a word that names something (`every`, `list`, `not`) is not
// in it. A query of nothing but these matches nothing.
var filler = map[string]bool{}
func init() {
for _, w := range strings.Fields("a an the did do does our my your we i is are was on in at of for to all any and or with it its there what which how has have been") {
filler[w] = true
}
}
// queryWords are a query's words to match: split at spaces, the punctuation of a sentence trimmed from
// each end (`nodes?` is `nodes`), filler left out.
func queryWords(query string) []string {
var out []string
for _, w := range strings.Fields(query) {
w = strings.Trim(w, "?!.,;:\"'()")
if w != "" && !filler[w] {
out = append(out, w)
}
}
return out
}
// hit is one match as mesh_search answers it.
type hit struct {
Address string `json:"address"`
@@ -169,8 +194,11 @@ func scoreOf(query string, qwords []string, name, description string, replaces [
// search is mesh_search's answer: every match, best first.
func (x *index) search(query string) []hit {
query = strings.ToLower(strings.TrimSpace(query))
qwords := strings.Fields(query)
qwords := queryWords(query)
var hits []hit
if len(qwords) == 0 {
return nil
}
for _, st := range x.Seats {
for _, v := range st.Verbs {
replaces := x.Replaces["seat:"+st.Seat+"."+v.Name]
@@ -149,3 +149,31 @@ func TestSearchWithoutReplacesMatchesAsBefore(t *testing.T) {
t.Errorf("a tool's own name: %+v", hits)
}
}
// A question's filler words do not have to match (every other word still does): the operator's own words
// find the tool whose description says it in other words; a query of filler alone finds nothing.
func TestSearchIgnoresFillerWords(t *testing.T) {
x := meshToSearch(t)
x.Modules["pacman"] = &moduleInfo{Module: "pacman", On: []string{"anchor"}, Tools: []Tool{{Module: "pacman", Name: "pacman_round",
Description: "Whether this week's system upgrade ran on every node (the weekly upgrade round, last run of all nodes)."}}}
for _, c := range []struct {
query string
want string // the first hit's address, "" for none
}{
{"did our weekly system upgrade run on all our nodes?", "anchor/pacman.pacman_round"},
{"the weekly upgrade", "anchor/pacman.pacman_round"},
{"did our", ""},
{"what is the", ""},
{"", ""},
{"did our weekly kubernetes upgrade run?", ""},
} {
hits := x.search(c.query)
got := ""
if len(hits) > 0 {
got = hits[0].Address
}
if got != c.want || (c.want == "" && len(hits) != 0) {
t.Errorf("%q: first hit %q, want %q (%+v)", c.query, got, c.want, hits)
}
}
}
+177
View File
@@ -0,0 +1,177 @@
package console
// mesh_wait: a session waits on the bus for one event instead of polling (novox/hq ADR 0367).
//
// **Only the families the tool runner was issued** (§2, §6): its membership names them, composed by the
// controller from the tool runner's manifest together with the grant that lets this account subscribe them.
// A name not listed there is refused here, before the bus would refuse it; no argument or setting adds one.
// **An event only wakes the session** (§4): what it says is read again from the verb that owns it.
import (
"bytes"
"context"
"encoding/json"
"fmt"
"math"
"sort"
"strings"
"time"
)
// WaitDefault is how long a wait lasts when no timeout is given, and WaitLongest the most it may last
// (ADR 0367 §1): four minutes and ten.
var (
WaitDefault = 4 * time.Minute
WaitLongest = 10 * time.Minute
)
func waitTool() map[string]any {
return map[string]any{"name": verbWait, "inputSchema": map[string]any{"type": "object", "properties": map[string]any{
"subjects": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": "one or more event families to wait on, by name, e.g. `mesh-knowledge.reviewed`; only " +
"those the mesh granted this machine's tool runner — a refusal names them"},
"filter": map[string]any{"type": "object",
"description": "fields of the event's body that must equal the given values, e.g. {\"kind\": \"decision\", \"number\": 367} (optional)"},
"timeout": map[string]any{"type": "number",
"description": fmt.Sprintf("seconds to wait: %d when absent, %d at most", int(WaitDefault.Seconds()), int(WaitLongest.Seconds()))},
}, "required": []string{"subjects"}},
"description": "Wait for the first event of the named families that matches the filter, and answer it with its " +
"family, body and time — or say it timed out. Call it in a loop from a background command, and after every " +
"answer read the state from the verb that owns it: an event only wakes you, and one published between two " +
"waits is not heard (novox/hq ADR 0367)."}
}
// waitAnswer answers one mesh_wait call.
func (s *Surface) waitAnswer(ctx context.Context, args map[string]any) map[string]any {
names, err := waitNames(args["subjects"])
if err != nil {
return failure(err.Error())
}
timeout, err := waitTimeout(args["timeout"])
if err != nil {
return failure(err.Error())
}
filter, _ := args["filter"].(map[string]any)
if args["filter"] != nil && filter == nil {
return failure("mesh_wait's filter is an object of body fields to the values they must equal")
}
issued := map[string]string{} // name → subject
if m := s.conn.Membership(s.conn.Module()); m != nil {
for _, w := range m.Waits {
issued[w.Event] = w.Subject
}
}
if len(issued) == 0 {
return failure("this machine's tool runner was issued no event families to wait on: the mesh grants them from " +
"the tool runner's manifest (novox/hq ADR 0367 §2), and its membership names none yet")
}
byName := make([]string, 0, len(issued))
for name := range issued {
byName = append(byName, name)
}
sort.Strings(byName)
family := map[string]string{} // subject → name
var subjects []string
for _, name := range names {
subject, granted := issued[name]
if !granted {
return failure(fmt.Sprintf("mesh_wait may not wait on %q: this machine's tool runner is granted %s alone. "+
"A family is added only by a reviewed change to the tool runner's manifest (node-tools' module.json, "+
"novox/hq ADR 0367 §6); no argument, call or setting adds one", name, strings.Join(byName, ", ")))
}
if _, twice := family[subject]; !twice {
subjects = append(subjects, subject)
}
family[subject] = name
}
heard, ok, err := s.conn.HearFirst(ctx, subjects, timeout, func(body json.RawMessage) bool { return bodyMatches(body, filter) })
if err != nil {
return failure("mesh_wait could not wait: " + err.Error())
}
if !ok {
return answerText(map[string]any{"timed-out": true, "waited": timeout.String(), "subjects": names,
"read": "nothing matching was heard; read the state once from the verb that owns it, then wait again"})
}
return answerText(map[string]any{
"heard": map[string]any{"subject": family[heard.Subject], "body": heard.Body, "at": heard.At.Format(time.RFC3339Nano)},
"read": "an event only wakes you: read the state from the verb that owns it before acting (novox/hq ADR 0367 §4)",
})
}
// waitNames is the families a call names: a list of names, or one name.
func waitNames(given any) ([]string, error) {
var names []string
switch v := given.(type) {
case string:
names = []string{v}
case []any:
for _, x := range v {
name, ok := x.(string)
if !ok {
return nil, fmt.Errorf("mesh_wait's subjects are event names, such as mesh-knowledge.reviewed; %v is not one", x)
}
names = append(names, name)
}
}
for i := range names {
names[i] = strings.TrimSpace(names[i])
}
if len(names) == 0 {
return nil, fmt.Errorf("mesh_wait names at least one event family in `subjects`, such as mesh-knowledge.reviewed")
}
return names, nil
}
// waitTimeout is the call's timeout: WaitDefault when absent, refused beyond WaitLongest or below a second.
func waitTimeout(given any) (time.Duration, error) {
if given == nil {
return WaitDefault, nil
}
longest := int(WaitLongest.Seconds())
seconds, ok := given.(float64)
if !ok {
if n, isInt := given.(int); isInt {
seconds, ok = float64(n), true
}
}
if !ok || math.IsNaN(seconds) || seconds < 1 || seconds > float64(longest) {
return 0, fmt.Errorf("mesh_wait's timeout is a number of seconds from 1 to %d (novox/hq ADR 0367 §1), not %v", longest, given)
}
return time.Duration(seconds * float64(time.Second)), nil
}
// bodyMatches says whether every field the filter names equals, in the event's body, the value given.
func bodyMatches(body json.RawMessage, filter map[string]any) bool {
if len(filter) == 0 {
return true
}
var fields map[string]json.RawMessage
if json.Unmarshal(body, &fields) != nil {
return false
}
for k, want := range filter {
got, has := fields[k]
if !has {
return false
}
wanted, _ := json.Marshal(want)
if !sameJSON(got, wanted) {
return false
}
}
return true
}
// sameJSON compares two JSON values as values: 367 and 367.0 are one number, key order does not matter.
func sameJSON(a, b []byte) bool {
var x, y any
da, db := json.NewDecoder(bytes.NewReader(a)), json.NewDecoder(bytes.NewReader(b))
if da.Decode(&x) != nil || db.Decode(&y) != nil {
return false
}
ja, _ := json.Marshal(x)
jb, _ := json.Marshal(y)
return bytes.Equal(ja, jb)
}
+279
View File
@@ -0,0 +1,279 @@
package console
import (
"context"
"encoding/json"
"net/http"
"os"
"reflect"
"sort"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// waitingRuntime is the tool runner on a machine, issued the families ADR 0367 §5 names, as the controller
// composes them from its manifest: the surface over its connection, and the mesh to publish on.
func waitingRuntime(t *testing.T) (*Surface, *mt.Mesh) {
t.Helper()
mesh := mt.New(t)
m := mt.MembershipOf("node-tools", "desk", false, nil)
m.Waits = []bus.WaitIssued{
{Event: "mesh-knowledge.reviewed", Subject: "mesh.mod.mesh-knowledge.event.reviewed"},
{Event: "operator-channel.decided.claude-code", Subject: "mesh.seat.operator-channel.event.decided.claude-code"},
}
mesh.Issue(t, m)
conn := connect(t, "node-tools", "desk")
mt.Until(t, func() error {
if got := conn.Membership("node-tools"); got == nil || len(got.Waits) == 0 {
return os.ErrNotExist
}
return nil
})
return NewSurface(conn, "desk.node-tools"), mesh
}
// callWait is one tools/call of mesh_wait, answered as an agent host reads it.
func callWait(s *Surface, args map[string]any) (text string, isError bool) {
raw, _ := json.Marshal(map[string]any{"name": verbWait, "arguments": args})
r := s.Handle(Request{JSONRPC: "2.0", ID: json.RawMessage("1"), Method: "tools/call", Params: raw})
result, _ := r.Result.(map[string]any)
content, _ := result["content"].([]map[string]any)
var parts []string
for _, c := range content {
parts = append(parts, c["text"].(string))
}
isErr, _ := result["isError"].(bool)
return strings.Join(parts, "\n"), isErr
}
// novox/hq ADR 0367 §1: mesh_wait answers the first event that matches — by its family and its filter — with
// its subject, body and time, within a second of its publication, and ignores the others.
func TestMeshWaitAnswersTheFirstMatchingEvent(t *testing.T) {
s, mesh := waitingRuntime(t)
type answer struct {
text string
isErr bool
at time.Time
}
done := make(chan answer, 1)
go func() {
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-knowledge.reviewed"},
"filter": map[string]any{"kind": "decision", "number": 367}, "timeout": 20})
done <- answer{text, isErr, time.Now()}
}()
// Once it listens: another record's verdict, another family's event, then the one it waits for.
time.Sleep(300 * time.Millisecond)
mesh.Emit(t, "mesh.mod.mesh-knowledge.event.reviewed", map[string]any{"kind": "decision", "number": 366, "what": "approved"})
mesh.Emit(t, "mesh.mod.mesh-knowledge.event.proposed", map[string]any{"kind": "decision", "number": 367})
published := time.Now()
mesh.Emit(t, "mesh.mod.mesh-knowledge.event.reviewed", map[string]any{"kind": "decision", "number": 367, "what": "approved"})
var got answer
select {
case got = <-done:
case <-time.After(10 * time.Second):
t.Fatal("mesh_wait did not answer the matching event")
}
if got.isErr {
t.Fatalf("mesh_wait failed: %s", got.text)
}
if got.at.Sub(published) > time.Second {
t.Errorf("answered %s after the event was published, want within a second", got.at.Sub(published))
}
var heard struct {
Heard struct {
Subject string `json:"subject"`
Body map[string]any `json:"body"`
At time.Time `json:"at"`
} `json:"heard"`
}
if err := json.Unmarshal([]byte(got.text), &heard); err != nil {
t.Fatalf("not an answer: %s", got.text)
}
if heard.Heard.Subject != "mesh-knowledge.reviewed" || heard.Heard.Body["number"] != float64(367) || heard.Heard.At.IsZero() {
t.Errorf("answered %+v, want record 367's verdict with its time", heard.Heard)
}
// And the subject it is granted on the asker's own `decided`, by the name a session asks for.
go func() {
text, isErr := callWait(s, map[string]any{"subjects": "operator-channel.decided.claude-code", "timeout": 20})
done <- answer{text, isErr, time.Now()}
}()
time.Sleep(300 * time.Millisecond)
// A seat's event, said by the router on core NATS here: no stream of the test's bus carries the seat.
if err := connect(t, "messenger", "").Say("mesh.seat.operator-channel.event.decided.claude-code", []byte(`{"ask":"a1"}`)); err != nil {
t.Fatal(err)
}
select {
case got = <-done:
if got.isErr || !strings.Contains(got.text, `"subject": "operator-channel.decided.claude-code"`) {
t.Errorf("the asker's decided: %s", got.text)
}
case <-time.After(10 * time.Second):
t.Fatal("mesh_wait did not answer the asker's decided")
}
}
// It says it timed out at its timeout, and leaves no subscription behind (ADR 0367 §1, §2).
func TestMeshWaitSaysItTimedOutAndKeepsNothing(t *testing.T) {
s, _ := waitingRuntime(t)
before := s.conn.Subscriptions()
start := time.Now()
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-knowledge.reviewed"}, "timeout": 1})
if took := time.Since(start); took < time.Second || took > 3*time.Second {
t.Errorf("a one-second wait took %s", took)
}
if isErr || !strings.Contains(text, `"timed-out": true`) {
t.Errorf("a wait that heard nothing: %s", text)
}
if after := s.conn.Subscriptions(); after != before {
t.Errorf("%d subscriptions before the wait, %d after it", before, after)
}
}
// Its timeout is four minutes when none is given and ten at most: a longer one, or none at all, is refused
// with the bound, never shortened in silence.
func TestMeshWaitsTimeoutIsBounded(t *testing.T) {
s, _ := waitingRuntime(t)
if WaitDefault != 4*time.Minute || WaitLongest != 10*time.Minute {
t.Errorf("the bounds are %s and %s, want 4m and 10m (ADR 0367 §1)", WaitDefault, WaitLongest)
}
for _, timeout := range []any{601, 0, -5, "soon"} {
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-knowledge.reviewed"}, "timeout": timeout})
if !isErr || !strings.Contains(text, "600") {
t.Errorf("a timeout of %v: %s", timeout, text)
}
}
}
// novox/hq ADR 0367 §2 and §6: an event outside the granted families is refused, naming the families and how
// one is added; no argument — a wildcard, another asker's `decided`, a subject spelled out — widens it.
func TestMeshWaitRefusesAnEventOutsideTheGrantedFamilies(t *testing.T) {
s, _ := waitingRuntime(t)
for _, subjects := range []any{
[]any{"mesh-knowledge.proposed"},
[]any{"mesh-knowledge.reviewed", "gitea.forge.push"},
[]any{"mesh-knowledge.*"},
[]any{"operator-channel.decided.shop"},
[]any{"operator-channel.decided.*"},
[]any{"mesh.mod.mesh-knowledge.event.reviewed"},
[]any{">"},
[]any{},
} {
text, isErr := callWait(s, map[string]any{"subjects": subjects, "timeout": 1})
if !isErr {
t.Errorf("%v was waited on: %s", subjects, text)
continue
}
if len(subjects.([]any)) > 0 && (!strings.Contains(text, "mesh-knowledge.reviewed") || !strings.Contains(text, "manifest")) {
t.Errorf("%v refused without naming the families and how one is added: %s", subjects, text)
}
}
}
// A tool runner issued no families waits on nothing, and says so.
func TestMeshWaitWithNoFamiliesIssuedWaitsOnNothing(t *testing.T) {
mt.New(t)
s := NewSurface(connect(t, "node-tools", "desk"), "desk.node-tools")
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-knowledge.reviewed"}, "timeout": 1})
if !isErr || !strings.Contains(text, "no event families") {
t.Errorf("a runtime issued no families: %s", text)
}
}
// The mesh MCP server announces seven tools (ADR 0367 v13): ADR 0195's five, mesh_runtimes and mesh_wait, its timeout bounded.
func TestMeshWaitIsAnnouncedAmongSeven(t *testing.T) {
var names []string
var wait map[string]any
for _, tool := range discovery() {
names = append(names, tool["name"].(string))
if tool["name"] == verbWait {
wait = tool
}
}
sort.Strings(names)
want := []string{"mesh_call", "mesh_describe", "mesh_machine", "mesh_overview", "mesh_runtimes", "mesh_search", "mesh_wait"}
if !reflect.DeepEqual(names, want) {
t.Errorf("announced %v, want %v", names, want)
}
if wait == nil || !isDiscovery(verbWait) {
t.Fatal("mesh_wait is not announced")
}
schema, _ := json.Marshal(wait["inputSchema"])
for _, word := range []string{`"subjects"`, `"filter"`, `"timeout"`} {
if !strings.Contains(string(schema), word) {
t.Errorf("mesh_wait's schema has no %s: %s", word, schema)
}
}
}
// A client that hangs up or cancels ends its wait at once: the subscription is closed and nothing is left
// waiting for the timeout (the review of issue 745). Over the console's own HTTP, as an agent host calls it.
func TestMeshWaitEndsWhenItsClientHangsUp(t *testing.T) {
s, _ := waitingRuntime(t)
up, err := Serve(s, "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = up.Close() })
before := s.conn.Subscriptions()
ctx, cancel := context.WithCancel(context.Background())
body := `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"mesh_wait",` +
`"arguments":{"subjects":["mesh-knowledge.reviewed"],"timeout":600}}}`
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, "http://"+up.Address+"/mcp", strings.NewReader(body))
req.Header.Set("content-type", "application/json")
go func() { _, _ = http.DefaultClient.Do(req) }()
mt.Until(t, func() error {
if s.conn.Subscriptions() <= before {
return os.ErrNotExist
}
return nil
})
cancel()
start := time.Now()
for s.conn.Subscriptions() > before {
if time.Since(start) > 3*time.Second {
t.Fatalf("%d subscriptions 3s after the client hung up, %d before the wait", s.conn.Subscriptions(), before)
}
time.Sleep(20 * time.Millisecond)
}
}
// The tool runner stopping ends every wait in flight at once (the second review of issue 745): Close cancels
// the requests' context rather than waiting up to ten minutes for a wait to time out.
func TestMeshWaitEndsWhenTheConsoleCloses(t *testing.T) {
s, _ := waitingRuntime(t)
up, err := Serve(s, "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
before := s.conn.Subscriptions()
body := `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"mesh_wait",` +
`"arguments":{"subjects":["mesh-knowledge.reviewed"],"timeout":600}}}`
go func() { _, _ = http.Post("http://"+up.Address+"/mcp", "application/json", strings.NewReader(body)) }()
mt.Until(t, func() error {
if s.conn.Subscriptions() <= before {
return os.ErrNotExist
}
return nil
})
closed := make(chan error, 1)
start := time.Now()
go func() { closed <- up.Close() }()
select {
case <-closed:
case <-time.After(5 * time.Second):
t.Fatal("closing the console waited on a mesh_wait in flight")
}
if took := time.Since(start); took > 3*time.Second {
t.Errorf("closing took %s", took)
}
mt.Until(t, func() error {
if s.conn.Subscriptions() > before {
return os.ErrExist
}
return nil
})
}
@@ -3,7 +3,12 @@ package launch
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
"time"
)
type lostWrite struct{}
@@ -28,3 +33,84 @@ func TestALostWriteIsAnsweredWithItsOwnCodeAndNothingElseIs(t *testing.T) {
t.Errorf("a plain refusal is answered %d", got)
}
}
// A bundle that answers the handshake and an empty tool list, and notes each start and its pid in $OUT.
const notingBundle = `#!/bin/sh
printf 'started %s\n' "$$" >> "$OUT"
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id" ;;
esac
done
`
// The service manager stops the runtime by signalling its whole group, every bundle with it (issue 1126, ace's
// reboot of 2026-10-28: every bundle "exited (SIGTERM); started again in 500ms" for the 15 s the runtime took
// to stop, the power bundle among them dialling the system bus again, which restarted the bus during the
// shutdown and held it 29 minutes). A bundle that exits while it serves is started again; once the runtime
// itself is stopping, none is, by a restart or by a call.
func TestABundleIsStartedAgainUnlessTheRuntimeIsStopping(t *testing.T) {
saved := RestartFirst
RestartFirst = 10 * time.Millisecond
t.Cleanup(func() { RestartFirst = saved; shuttingDown.Store(false) })
for _, row := range []struct {
name string
stopping bool
starts int
}{
{"a bundle that dies while the runtime serves", false, 2},
{"a bundle signalled with the stopping runtime", true, 1},
} {
t.Run(row.name, func(t *testing.T) {
shuttingDown.Store(false)
dir := t.TempDir()
entry, out := filepath.Join(dir, "bundle"), filepath.Join(dir, "out")
if err := os.WriteFile(entry, []byte(notingBundle), 0o755); err != nil {
t.Fatal(err)
}
l, err := Start("noting", entry, append(os.Environ(), "OUT="+out), &subscribing{}, t.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
said, _ := os.ReadFile(out)
var pid int
if _, err := fmt.Sscanf(string(said), "started %d", &pid); err != nil {
t.Fatalf("the bundle noted %q", said)
}
if row.stopping {
ShutDown()
}
if err := syscall.Kill(pid, syscall.SIGTERM); err != nil {
t.Fatal(err)
}
// The restart row ends at its second start; the stopping row watches a window many restart pauses long.
deadline := time.Now().Add(250 * time.Millisecond)
for {
said, _ = os.ReadFile(out)
n := strings.Count(string(said), "started")
if n > row.starts {
t.Fatalf("started %d times, want %d", n, row.starts)
}
if n == row.starts && !row.stopping {
break
}
if time.Now().After(deadline) {
if n != row.starts {
t.Fatalf("started %d times, want %d", n, row.starts)
}
break
}
time.Sleep(5 * time.Millisecond)
}
if row.stopping {
// And a bundle first loaded now (a reload) is not started either.
if _, err := Start("late", entry, append(os.Environ(), "OUT="+out), &subscribing{}, t.Logf); err == nil {
t.Error("a bundle was started after the runtime began stopping")
}
}
})
}
}
@@ -38,12 +38,17 @@ func (b *subscribing) Publish(json.RawMessage) error {
func (b *subscribing) Ask(json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) State(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) Log(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) StreamRead(json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) Watch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
func (b *subscribing) Seat(string, json.RawMessage, func(string, any) (json.RawMessage, error)) (json.RawMessage, error) {
return nil, nil
}
func (b *subscribing) Provision(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) ProvisionWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
func (b *subscribing) Subscribe(d func(json.RawMessage) error) error {
b.mu.Lock()
defer b.mu.Unlock()
+81 -7
View File
@@ -15,6 +15,7 @@ import (
"regexp"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
@@ -79,10 +80,18 @@ type Bus interface {
Watch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
// Log answers a bundle's `append`, `read`, `last` and `info` on its module's log (novox/hq ADR 0297).
Log(verb string, params json.RawMessage) (json.RawMessage, error)
// StreamRead answers a bundle's `read` of one of its module's own streams (novox/hq ADR 0363): {stream, subject,
// from, limit}, as a log's read.
StreamRead(params json.RawMessage) (json.RawMessage, error)
// Seat answers a bundle's seat traffic (novox/hq ADR 0259 §3): `publish`, `prove` and `record` at once;
// `take` and `answer` bind once and hand each message or proof on through handOn, to whichever child
// of the module asked last — a restarted child asks again as its code runs again.
Seat(verb string, params json.RawMessage, handOn func(method string, params any) (json.RawMessage, error)) (json.RawMessage, error)
// Provision answers a bundle's `get` of one of its module's values at runtime, and ProvisionWatch hands deliver
// that value now and each change after, as the module whose bundle asks and never a module the bundle names
// (novox/hq ADR 0349 §14, ADR 0370 §4): its config, and its bindings' values that are not secret.
Provision(verb string, params json.RawMessage) (json.RawMessage, error)
ProvisionWatch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
}
// EventTimeout bounds how long a bundle has to handle one event before it is offered again.
@@ -107,6 +116,17 @@ var (
RestartSettle = time.Minute
)
// shuttingDown is set once the runtime itself is stopping (ShutDown): from then on no bundle is started again.
var shuttingDown atomic.Bool
// ShutDown says the runtime itself is stopping, before it stops its bundles one by one. The service manager stops
// the runtime by signalling its whole group, so every bundle receives SIGTERM with it, and a bundle's exit must not
// be taken for a fault: started again, it would run on through the machine's shutdown with nothing left to stop it
// (issue 1126: on ace's reboot of 2026-10-28 the restarted bundles dialled the system bus again and started it
// during the shutdown, which then waited 29 minutes). After ShutDown no bundle is started again, by a restart or a
// call.
func ShutDown() { shuttingDown.Store(true) }
// StopWait bounds how long Launched.Stop waits for its bundle's exit to be handled; a bundle that
// ignores SIGTERM longer than this is left to exit on its own (novox/hq issue 476).
var StopWait = 5 * time.Second
@@ -169,9 +189,17 @@ func (c *child) write(m any) error {
var stackLine = regexp.MustCompile(`^\s+at\s`)
// colour is a terminal's colour or style sequence, which node writes into a stack trace when its
// environment asks for colour (FORCE_COLOR): taken out before a line is judged or kept as a bundle's
// reason, so the reason reads the same in every environment.
var colour = regexp.MustCompile("\x1b\\[[0-9;]*m")
// Start launches one bundle and learns its tools. It fails when the child cannot be started or does
// not complete the handshake. A child that exits later is started again on its next call.
func Start(module, entry string, env []string, mesh Bus, logf func(string, ...any)) (*Launched, error) {
if shuttingDown.Load() {
return nil, fmt.Errorf("%s's bundle is not started: the runtime is stopping", module)
}
var mu sync.Mutex
var current *child
// The child that last subscribed is the one events are handed to: a restarted child subscribes
@@ -240,9 +268,10 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
continue
}
logf("[%s] %s", module, line)
if !stackLine.MatchString(line) && !strings.HasPrefix(line, "Node.js v") {
plain := colour.ReplaceAllString(line, "")
if !stackLine.MatchString(plain) && !strings.HasPrefix(plain, "Node.js v") {
saidMu.Lock()
lastSaid = strings.TrimSpace(line)
lastSaid = strings.TrimSpace(plain)
saidMu.Unlock()
}
}
@@ -283,7 +312,7 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
subscriber = c
mu.Unlock()
err = mesh.Subscribe(deliver)
case "mesh/seat.publish", "mesh/seat.prove", "mesh/seat.record", "mesh/seat.kinds":
case "mesh/seat.publish", "mesh/seat.prove", "mesh/seat.record", "mesh/seat.kinds", "mesh/seat.pull", "mesh/seat.settle":
var answered json.RawMessage
if answered, err = mesh.Seat(strings.TrimPrefix(m.Method, "mesh/seat."), m.Params, nil); err == nil {
result = answered
@@ -307,6 +336,35 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
if answered, err = mesh.Log(strings.TrimPrefix(m.Method, "mesh/log."), m.Params); err == nil {
result = answered
}
case "mesh/stream.read":
var answered json.RawMessage
if answered, err = mesh.StreamRead(m.Params); err == nil {
result = answered
}
case "mesh/provision.get":
var answered json.RawMessage
if answered, err = mesh.Provision("get", m.Params); err == nil {
result = answered
}
case "mesh/provision.watch":
// As a state's watch: each change asked of this child in order, naming the watch by the
// id the child asked it with, and answered once the current value is handed over.
var stop func()
watch := m.ID
stop, err = mesh.ProvisionWatch(m.Params, func(change json.RawMessage) error {
_, err := c.ask(module, "mesh/provision", map[string]any{"watch": watch, "change": change}, EventTimeout)
return err
})
if err == nil {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
stop()
} else {
c.watches = append(c.watches, stop)
c.mu.Unlock()
}
}
case "mesh/state.watch":
// Each change is asked of this child, in order, naming the watch it is for by
// the id the child asked it with — two watches of one state are two handlers;
@@ -409,7 +467,10 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
pause = pauseFirst
}
mu.Unlock()
if !wasStopped && wasServing {
if wasServing && !wasStopped && shuttingDown.Load() {
logf("[mesh-tools] %s; not started again: the runtime is stopping", why)
}
if !wasStopped && wasServing && !shuttingDown.Load() {
// Every bundle stays up (ADR 0198): code that runs long — a handler, a provisioner —
// is not waiting for a call to bring it back, and a tool bundle back early costs nothing.
logf("[mesh-tools] %s; started again in %s", why, wait.Round(100*time.Millisecond))
@@ -427,14 +488,27 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
asking := func(method string, params any, timeout time.Duration) (json.RawMessage, error) {
mu.Lock()
c := current
c, gone := current, stopped
mu.Unlock()
if gone {
// A stopped bundle is never started again by a call: a process started now would be one
// nothing stops (issue 1945).
return nil, fmt.Errorf("%s's bundle is stopped; it answers no call", module)
}
if c == nil {
if shuttingDown.Load() {
return nil, fmt.Errorf("%s's bundle is not started again: the runtime is stopping", module)
}
fresh, err := start()
if err != nil {
return nil, err
}
mu.Lock()
if stopped {
mu.Unlock()
_ = fresh.cmd.Process.Signal(syscall.SIGTERM)
return nil, fmt.Errorf("%s's bundle is stopped; it answers no call", module)
}
if current == nil {
current = fresh
startedAt = time.Now()
@@ -452,7 +526,7 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
go func() {
time.Sleep(after)
mu.Lock()
if stopped || current != nil {
if stopped || current != nil || shuttingDown.Load() {
mu.Unlock()
return
}
@@ -461,7 +535,7 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
mu.Lock()
defer mu.Unlock()
if err != nil {
if !stopped {
if !stopped && !shuttingDown.Load() {
wait := pause
if pause *= 2; pause > pauseMost {
pause = pauseMost
@@ -0,0 +1,107 @@
package launch
import (
"encoding/json"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
)
// A bundle that asks for one of its values and watches it, and writes each change it is handed to $OUT.
const provisionBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*)
printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"g1","method":"mesh/provision.get","params":{"kind":"config","key":"role"}}\n'
printf '{"jsonrpc":"2.0","id":"w1","method":"mesh/provision.watch","params":{"kind":"config","key":"role"}}\n' ;;
*'"method":"mesh/provision"'*)
printf '%s\n' "$line" >> "$OUT"
printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"id":"g1"'*) printf '%s\n' "$line" >> "$OUT" ;;
esac
done
`
type provisioning struct {
subscribing
mu sync.Mutex
asked []string
}
func (b *provisioning) Provision(verb string, params json.RawMessage) (json.RawMessage, error) {
b.mu.Lock()
defer b.mu.Unlock()
b.asked = append(b.asked, verb+" "+string(params))
return json.RawMessage(`"the build machine"`), nil
}
func (b *provisioning) ProvisionWatch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
b.mu.Lock()
b.asked = append(b.asked, "watch "+string(params))
b.mu.Unlock()
return func() {}, deliver(json.RawMessage(`{"kind":"config","key":"role","value":"the build machine"}`))
}
// novox/hq ADR 0349 §14, ADR 0370 §4: a bundle's ask for its values reaches the runtime's bus from its own channel,
// answered there, and a watch's change is handed to it naming the watch it asked.
func TestABundleAsksForItsValuesAndIsHandedItsWatchsChanges(t *testing.T) {
entry := filepath.Join(t.TempDir(), "bundle")
if err := os.WriteFile(entry, []byte(provisionBundle), 0o755); err != nil {
t.Fatal(err)
}
out := filepath.Join(t.TempDir(), "out")
b := &provisioning{}
l, err := Start("claude-code", entry, append(os.Environ(), "OUT="+out), b, t.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
deadline := time.Now().Add(10 * time.Second)
for {
said, _ := os.ReadFile(out)
if strings.Count(string(said), "\n") >= 2 {
// Each line as the bundle read it, by what it is: the get's answer, and the watch's change.
got := map[string]string{}
for _, line := range strings.Split(strings.TrimSpace(string(said)), "\n") {
var m struct {
ID any `json:"id"`
Method string `json:"method"`
Result json.RawMessage `json:"result"`
Params json.RawMessage `json:"params"`
}
if err := json.Unmarshal([]byte(line), &m); err != nil {
t.Fatalf("the bundle read %s", line)
}
if m.Method != "" {
got[m.Method] = string(m.Params)
} else {
got["answer"] = string(m.Result)
}
}
if got["answer"] != `"the build machine"` {
t.Errorf("the get was answered %s", got["answer"])
}
if got["mesh/provision"] != `{"change":{"kind":"config","key":"role","value":"the build machine"},"watch":"w1"}` {
t.Errorf("the watch was handed %s", got["mesh/provision"])
}
break
}
if time.Now().After(deadline) {
t.Fatalf("the bundle was handed %q", said)
}
time.Sleep(20 * time.Millisecond)
}
b.mu.Lock()
defer b.mu.Unlock()
slices.Sort(b.asked)
if len(b.asked) != 2 || b.asked[0] != `get {"kind":"config","key":"role"}` || b.asked[1] != `watch {"kind":"config","key":"role"}` {
t.Errorf("the runtime's bus was asked %q", b.asked)
}
}
+42
View File
@@ -69,3 +69,45 @@ func TestABundleTakesASeatsWorkThroughItsOwnChannel(t *testing.T) {
t.Fatalf("the work was not handed to the bundle: %s %v", got, err)
}
}
// A bundle that pulls one ask from its worker when it is listed.
const pullingBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*)
printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"p1","method":"mesh/seat.pull","params":{"worker":"SEAT_NODE_BUILD_GO_DELIVERY_worker","wait_ms":0}}\n' ;;
esac
done
`
// 1887/A11: a bundle's pull reaches the runtime's bus from its own channel, as its other seat traffic does.
func TestABundlePullsASeatsWorkThroughItsOwnChannel(t *testing.T) {
entry := filepath.Join(t.TempDir(), "bundle")
if err := os.WriteFile(entry, []byte(pullingBundle), 0o755); err != nil {
t.Fatal(err)
}
b := &seating{}
l, err := Start("mesh-build-go", entry, os.Environ(), b, t.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
for i := 0; ; i++ {
b.mu.Lock()
verb, params := b.verb, b.params
b.mu.Unlock()
if verb == "pull" {
if params != `{"worker":"SEAT_NODE_BUILD_GO_DELIVERY_worker","wait_ms":0}` {
t.Fatalf("pulled with %s", params)
}
return
}
if i > 100 {
t.Fatal("the bundle's pull never reached the runtime")
}
time.Sleep(20 * time.Millisecond)
}
}
+38
View File
@@ -0,0 +1,38 @@
package meshtest
import (
"os"
"os/exec"
"path/filepath"
"runtime"
"testing"
)
// bundlesWhy returns the reason bundles cannot be launched from a checkout rooted at root, or "" when they can: the
// TypeScript and Python bundles the runtime and the console launch need node and the package
// @novox/mesh-sdk, which comes from the package registry and is installed beside node-tools/package.json
// (novox/hq ADR 0529 clause 1: what merge-check.sh said "NOT TESTED HERE" about is a skip with its reason).
func bundlesWhy(root string, haveNode bool) string {
if !haveNode {
return "node is not installed"
}
if _, err := os.Stat(filepath.Join(root, "node_modules", "@novox", "mesh-sdk")); err != nil {
return "@novox/mesh-sdk is not installed in " + filepath.Join(root, "node_modules")
}
return ""
}
// NeedBundles skips the test, saying why, where the bundles it launches cannot run here.
func NeedBundles(t *testing.T) {
t.Helper()
_, err := exec.LookPath("node")
if why := bundlesWhy(moduleRoot(), err == nil); why != "" {
t.Skip("NOT TESTED HERE: this test launches bundles, and " + why)
}
}
// moduleRoot is node-tools/, where package.json and node_modules are.
func moduleRoot() string {
_, here, _, _ := runtime.Caller(0)
return filepath.Join(filepath.Dir(here), "..", "..")
}
+55
View File
@@ -271,3 +271,58 @@ func (m *Mesh) Consumers(t *testing.T, stream string) int {
}
return info.State.Consumers
}
// ModuleStream makes a module's own stream afresh as the controller does from the catalogue (novox/hq ADR
// 0363 §1): named in capitals, taking `mesh.stream.<module>.event.>`, a subject space EVENTS does not take.
func (m *Mesh) ModuleStream(t *testing.T, module, name string) {
t.Helper()
_ = m.js.DeleteStream(name)
if _, err := m.js.AddStream(&nats.StreamConfig{Name: name, Subjects: []string{"mesh.stream." + module + ".event.>"},
Retention: nats.LimitsPolicy, Discard: nats.DiscardOld, AllowDirect: true}); err != nil {
t.Fatal(err)
}
}
// ConsumerOn makes a module's durable consumer on a machine on any stream, as Consumer does on EVENTS.
func (m *Mesh) ConsumerOn(t *testing.T, stream, node, module string, filters []string, ackWait time.Duration) {
t.Helper()
cfg := &nats.ConsumerConfig{Durable: node + "_" + module, AckPolicy: nats.AckExplicitPolicy,
AckWait: ackWait, MaxDeliver: 10, DeliverPolicy: nats.DeliverNewPolicy}
if len(filters) == 1 {
cfg.FilterSubject = filters[0]
} else {
cfg.FilterSubjects = filters
}
if _, err := m.js.AddConsumer(stream, cfg); err != nil {
t.Fatal(err)
}
}
// PendingOn is what a module's consumer on a stream still holds, as Pending is on EVENTS.
func (m *Mesh) PendingOn(t *testing.T, stream, node, module string) (ackPending, notDelivered uint64) {
t.Helper()
info, err := m.js.ConsumerInfo(stream, node+"_"+module)
if err != nil {
t.Fatal(err)
}
return uint64(info.NumAckPending), info.NumPending
}
// Messages is how many messages a stream holds.
func (m *Mesh) Messages(t *testing.T, stream string) uint64 {
t.Helper()
info, err := m.js.StreamInfo(stream)
if err != nil {
t.Fatal(err)
}
return info.State.Msgs
}
// RemoveConsumer deletes a module's consumer on a machine on a stream, as the controller does when a grant
// it cannot narrow is withdrawn (novox/hq ADR 0363 §6).
func (m *Mesh) RemoveConsumer(t *testing.T, stream, node, module string) {
t.Helper()
if err := m.js.DeleteConsumer(stream, node+"_"+module); err != nil {
t.Fatal(err)
}
}
@@ -73,3 +73,49 @@ func TestEachTestHasABusOfItsOwn(t *testing.T) {
}
})
}
// What merge-check.sh said was "NOT TESTED HERE" is a skip with its reason: no node, or no package, each says so.
func TestBundlesCannotBeLaunchedWithoutNodeOrTheSDKAndTheSkipSaysWhy(t *testing.T) {
root := t.TempDir()
if why := bundlesWhy(root, false); !strings.Contains(why, "node is not installed") {
t.Errorf("no node said %q", why)
}
if why := bundlesWhy(root, true); !strings.Contains(why, "@novox/mesh-sdk") {
t.Errorf("no package said %q", why)
}
if err := os.MkdirAll(filepath.Join(root, "node_modules", "@novox", "mesh-sdk"), 0o755); err != nil {
t.Fatal(err)
}
if why := bundlesWhy(root, true); why != "" {
t.Errorf("node and the package present said %q", why)
}
}
// The checker runs node-tools' Go level by Go's convention and its Node level by package.json's test script
// (novox/hq ADR 0529): no script of the repository's own runs beside them, and no module but node-tools is built from
// here (ADR 0524 §5, the toolchain image retired).
func TestTheRepositoryHoldsNoCheckScriptAndNodeToolsIsItsOnlyModule(t *testing.T) {
root := filepath.Join("..", "..", "..")
if _, err := os.Stat(filepath.Join(root, "merge-check.sh")); err == nil {
t.Errorf("merge-check.sh is back: the checker's levels run the tests")
}
var found []string
err := filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() && (d.Name() == "node_modules" || d.Name() == ".git") {
return filepath.SkipDir
}
if !d.IsDir() && d.Name() == "module.json" {
found = append(found, filepath.ToSlash(p))
}
return nil
})
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || !strings.HasSuffix(found[0], "/node-tools/module.json") {
t.Errorf("the module manifests are %v, want node-tools' alone", found)
}
}
+132
View File
@@ -0,0 +1,132 @@
// Package provide is the runtime's half of node-tools being a provider of `mcp-endpoint`: it runs the mesh SDK's
// provider helper, so a consumer handed to the provisioner (claude-code's `${bound:mcp-endpoint:port}`) is answered
// from an entry node-tools writes in its own bucket, not from the controller's binding file (novox/hq ADR 0349
// clause 12, issue 791).
//
// The offer carries no credential: the console's authority is the machine's login (ADR 0152, 0175), so the helper
// reads nothing from the vault, needs no consumers' file and the entry names none. node-tools is not marked moved:
// its manifest says `"answered-by": "helper"` for the provision, and the helper runs whenever the runtime does
// (issue 791), as route-proxy's does (issue 792).
package provide
import (
"context"
"encoding/json"
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/provider"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
const (
// Module is the provider: the runtime's own module.
Module = "node-tools"
// Provision is what node-tools provides, as its manifest says.
Provision = "mcp-endpoint"
// ServedPort is the port its manifest serves (`serves.mcp-endpoint.port`): where the console listens.
ServedPort = 4270
)
// Offers is what node-tools serves, per provision.
func Offers() map[string]provider.Offer {
return map[string]provider.Offer{Provision: {
Credential: provider.NoCredential,
Serves: map[string]any{"port": ServedPort},
}}
}
// busOf is the helper's bus over the runtime's own state: the provider's bucket is its state `bound`, and the
// controller's assignments are the state it reads, `mesh-controller.assignments` (both declared in the manifest).
type busOf struct{ conn *bus.Conn }
func (b busOf) state(bucket string) (string, error) {
switch bucket {
case provider.BucketOf(Module):
return "bound", nil
case provider.AssignmentsBucket:
return "mesh-controller.assignments", nil
}
return "", fmt.Errorf("%s is not a bucket this provider keeps or reads", bucket)
}
func (b busOf) Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error {
name, err := b.state(bucket)
if err != nil {
return err
}
stop, err := b.conn.StateWatch(Module, name, "", func(ch bus.StateChange) error {
if ch.Op == "delete" {
handle(ch.Key, nil)
} else {
handle(ch.Key, ch.Value)
}
return nil
})
if err != nil {
return err
}
go func() { <-ctx.Done(); stop() }()
return nil
}
func (b busOf) Put(_ context.Context, bucket, key string, value []byte) error {
name, err := b.state(bucket)
if err != nil {
return err
}
_, err = b.conn.StatePut(Module, name, key, json.RawMessage(value))
return err
}
func (b busOf) Delete(_ context.Context, bucket, key string) error {
name, err := b.state(bucket)
if err != nil {
return err
}
return b.conn.StateDelete(Module, name, key)
}
// New is the helper answering node-tools's consumers on node.
func New(conn *bus.Conn, node string, every time.Duration, logf func(string, ...any)) *provider.Provider {
return provider.New(provider.Options{
Node: node,
Module: Module,
Offers: Offers(),
Bus: busOf{conn: conn},
Every: every,
Logf: logf,
})
}
// Run answers the consumers until ctx ends. The state is not reachable until the mesh has issued the runtime its
// membership and made the buckets, so a start that cannot watch is tried again, not given up.
func Run(ctx context.Context, conn *bus.Conn, node string, every time.Duration, logf func(string, ...any)) {
for delay := 2 * time.Second; ctx.Err() == nil; delay = min(delay*2, time.Minute) {
attempt, cancel := context.WithCancel(ctx)
err := New(conn, node, every, logf).Run(attempt)
cancel()
if ctx.Err() != nil {
return
}
logf("[mesh-tools] the provider helper stopped: %v; trying again in %s", err, delay)
select {
case <-ctx.Done():
case <-time.After(delay):
}
}
}
// Start runs the helper when this process is the machine's runtime (the node-tools credential, on a machine's
// name), with no condition on the environment, and returns what ends it. A runtime on another credential, a module's
// own in a test, is not the provider and answers nobody.
func Start(conn *bus.Conn, module, node string, every time.Duration, logf func(string, ...any)) (stop func()) {
if module != Module || node == "" {
return func() {}
}
ctx, cancel := context.WithCancel(context.Background())
logf("[mesh-tools] answering %s's consumers of %s through the provider helper", Module, Provision)
go Run(ctx, conn, node, every, logf)
return cancel
}
+149
View File
@@ -0,0 +1,149 @@
package provide
import (
"encoding/json"
"os"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// claude-code's facts.json template, as the catalogue keeps it (modules/claude-code/files/facts.json).
const factsTemplate = `{
"node": "${machine:name}",
"console": "http://127.0.0.1:${bound:mcp-endpoint:port}/mcp"
}
`
// factsToday is that file as the controller composes it today, from the port node-tools serves.
const factsToday = `{
"node": "anchor",
"console": "http://127.0.0.1:4270/mcp"
}
`
// fill is what the consumer's client does with the template: the machine's name, and each ${bound:<provision>:<key>}
// from the provider's entry.
func fill(t *testing.T, template string, entry map[string]any) string {
t.Helper()
values, _ := entry["values"].(map[string]any)
out := strings.ReplaceAll(template, "${machine:name}", "anchor")
for key, v := range values {
raw, _ := json.Marshal(v)
out = strings.ReplaceAll(out, "${bound:mcp-endpoint:"+key+"}", string(raw))
}
return out
}
func TestTheRuntimeAnswersAHandedConsumersMcpEndpointThroughTheHelper(t *testing.T) {
mesh := mt.New(t)
bound := mesh.Bucket(t, "node-tools_bound")
assignments := mesh.Bucket(t, "mesh-controller_assignments")
mem := mt.MembershipOf("node-tools", "anchor", false, nil)
mem.State = []bus.StateIssued{
{Name: "bound", Bucket: "node-tools_bound", Writes: true},
{Name: "mesh-controller.assignments", Bucket: "mesh-controller_assignments"},
}
mesh.Issue(t, mem)
// claude-code, handed on this machine, is bound to node-tools here for mcp-endpoint (issue 788's plan entry).
plan := `{"node":"anchor","module":"claude-code","provisioning":{"provisions":[{"provision":"mcp-endpoint",` +
`"provider":"anchor","provider-module":"node-tools","at":"127.0.0.1","from":"anchor","as":"mesh_anchor_claude_code"}]}}`
if _, err := assignments.Put("anchor.claude-code", []byte(plan)); err != nil {
t.Fatal(err)
}
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
conn.Follow("node-tools")
said := &mt.Logs{}
t.Cleanup(Start(conn, "node-tools", "anchor", 20*time.Millisecond, said.Logf))
var entry map[string]any
mt.Until(t, func() error {
kept, err := bound.Get("anchor.claude-code.mcp-endpoint")
if err != nil {
return err
}
return json.Unmarshal(kept.Value(), &entry)
})
values, _ := entry["values"].(map[string]any)
for key, want := range map[string]any{"port": float64(4270), "at": "127.0.0.1", "from": "anchor"} {
if values[key] != want {
t.Errorf("the entry's %s is %v, want %v (entry %v)", key, values[key], want, entry)
}
}
if entry["as"] != "mesh_anchor_claude_code" {
t.Errorf("the entry's as is %v", entry["as"])
}
if c, has := entry["credential"]; has {
t.Errorf("an offer with no credential names one: %v", c)
}
if got := fill(t, factsTemplate, entry); got != factsToday {
t.Errorf("claude-code's facts.json fills to\n%s\nwant today's\n%s", got, factsToday)
}
}
// The port the offer serves is the port the manifest serves and the console listens on.
func TestTheOfferServesThePortTheManifestServes(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Serves map[string]map[string]any `json:"serves"`
Moved bool `json:"moved"`
Provides []struct {
Name string `json:"name"`
AnsweredBy string `json:"answered-by"`
} `json:"provides"`
Resources []struct {
ID string `json:"id"`
} `json:"resources"`
Build struct {
Artifacts []struct {
Name string `json:"name"`
Env map[string]string `json:"env"`
} `json:"artifacts"`
} `json:"build"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
if got := m.Serves[Provision]["port"]; got != float64(ServedPort) {
t.Errorf("the manifest serves %v for %s, the offer %d", got, Provision, ServedPort)
}
if m.Moved {
t.Error("node-tools is marked moved: that also hands the runtime's bus account to the vault (issue 791 round 1)")
}
if len(m.Provides) != 1 || m.Provides[0].Name != Provision || m.Provides[0].AnsweredBy != "helper" {
t.Errorf("node-tools does not say its %s is answered by the helper: %+v", Provision, m.Provides)
}
for _, r := range m.Resources {
if r.ID == "consumers" {
t.Error("the manifest still carries a consumers file the helper does not need")
}
}
for _, a := range m.Build.Artifacts {
if a.Name == "files" || len(a.Env) > 0 {
t.Errorf("the build still carries %+v, which only a moved module needs", a)
}
}
}
// Only the machine's runtime is the provider: a process on another credential, or with no machine, answers nobody.
func TestOnlyTheMachinesRuntimeStartsTheHelper(t *testing.T) {
said := &mt.Logs{}
for _, c := range []struct{ module, node string }{{"claude-code", "anchor"}, {"node-tools", ""}} {
Start(nil, c.module, c.node, time.Second, said.Logf)()
}
}
@@ -18,6 +18,7 @@ import (
// read here with NATS's own types, each endpoint a subject actually served — and the announcement
// follows a re-issued membership.
func TestTheRuntimeAnnouncesWhatItServesInTheServicesProtocol(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("beta", "anchor", false, map[string][]string{"node-shelf": {"list", "clear"}}))
@@ -50,6 +50,7 @@ func countLines(t *testing.T, path, want string) int {
// is its bus — it binds the module's own consumer, hands each event to the bundle, and acknowledges it
// only when the bundle has taken it; one the bundle failed or died on is offered again.
func TestTheRuntimeHandsAModulesEventsToItsBundleAndAcknowledgesThemOnlyWhenTaken(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
restore := bus.NakDelay
bus.NakDelay = 300 * time.Millisecond
@@ -128,6 +129,7 @@ func TestTheRuntimeHandsAModulesEventsToItsBundleAndAcknowledgesThemOnlyWhenTake
// A long-running bundle that exits is started again, without waiting for a call (ADR 0198).
func TestALongRunningBundleThatExitsIsStartedAgain(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
firstPause := launch.RestartFirst
launch.RestartFirst = 100 * time.Millisecond
+42 -1
View File
@@ -21,6 +21,8 @@ func logBuses(t *testing.T, maxBytes int64) (issues, other launch.Bus, mesh *mt.
mesh = mt.New(t)
m := mt.MembershipOf("issues", "anchor", false, nil)
m.Logs = []bus.LogIssued{mesh.Log(t, "issues", "changes", maxBytes)}
mesh.ModuleStream(t, "issues", "ISSUE_MOVES")
m.Streams = []bus.StreamIssued{{Stream: "ISSUE_MOVES", Events: []string{"moved.>"}, Subject: "mesh.stream.issues.event.{event}"}}
mesh.Issue(t, m)
mesh.Issue(t, mt.MembershipOf("other", "anchor", false, nil))
conn := connect(t, "node-tools", "anchor")
@@ -111,7 +113,7 @@ func infoOf(t *testing.T, b launch.Bus) map[string]uint64 {
// RFC 3339; last is a key's last entry or null; info says entries, bytes, cap, first and last — and no
// consumer is made for any of it.
func TestALogAnswersEachVerbOnTheBus(t *testing.T) {
b, _, mesh := logBuses(t, 1<<20)
b, other, mesh := logBuses(t, 1<<20)
before := time.Now().Add(-time.Second)
var seqs []uint64
@@ -189,6 +191,45 @@ func TestALogAnswersEachVerbOnTheBus(t *testing.T) {
if n := mesh.Consumers(t, "LOG_issues_changes"); n != 0 {
t.Fatalf("the runtime made %d consumers of the log; it uses none", n)
}
// A module reads its own stream back as it reads its log (mesh/stream.read): by its local name, an event pattern,
// in pages; another module's stream is refused with the reason.
for i, ev := range []string{"moved.a.1", "moved.b.1", "moved.a.1"} {
if err := b.Publish(json.RawMessage(fmt.Sprintf(`{"key":%q,"body":{"n":%d}}`, ev, i))); err != nil {
t.Fatal(err)
}
}
streamOf := func(b launch.Bus, params string) (wireStream, error) {
var p wireStream
raw, err := b.StreamRead(json.RawMessage(params))
if err == nil {
err = json.Unmarshal(raw, &p)
}
return p, err
}
page, err := streamOf(b, `{"stream":"issue-moves","subject":"moved.a.>","limit":1}`)
if err != nil || len(page.Entries) != 1 || page.Entries[0].Event != "moved.a.1" || string(page.Entries[0].Body) != `{"n":0}` || page.Next == 0 || page.Last != 3 {
t.Fatalf("the first page of moved.a: %+v, %v", page, err)
}
page, err = streamOf(b, fmt.Sprintf(`{"stream":"ISSUE_MOVES","subject":"moved.a.>","from":%d}`, page.Next))
if err != nil || len(page.Entries) != 1 || string(page.Entries[0].Body) != `{"n":2}` || page.Next != 0 {
t.Fatalf("the second page of moved.a: %+v, %v", page, err)
}
if _, err := streamOf(other, `{"stream":"issue-moves"}`); err == nil || !strings.Contains(err.Error(), "keeps no stream") {
t.Fatalf("another module's stream: %v", err)
}
}
// wireStream is a stream read's answer with exactly the contract's field names (mesh-sdk stdio.StreamPage).
type wireStream struct {
Entries []struct {
Sequence uint64 `json:"sequence"`
Event string `json:"event"`
Body json.RawMessage `json:"body"`
At string `json:"at"`
} `json:"entries"`
Next uint64 `json:"next"`
Last uint64 `json:"last"`
}
// A page holds about 4 MiB at most, whatever its limit, and says where to continue.
+228
View File
@@ -0,0 +1,228 @@
package runtime
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"sync"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// A module's code reads its values at runtime (novox/hq ADR 0349 clause 14 as ADR 0370 changes it, issue 685).
//
// A bundle asks `mesh/provision.get` and `mesh/provision.watch` with `{kind, key}`; the runtime asks its machine's
// provisioner client **as the module whose bundle asks** — the module this bus was made for, from the bundle it
// launched, never a name the bundle sent (ADR 0370 §4) — and hands the answer back. A watch hands the current value
// first and then each change: the client says which value moved, and the runtime asks again. A secret is refused
// here, with the reason, and never asked: the client serves no secret on the tool runner's word (§2), and a module
// under an account of its own asks the vault itself, which waits on issue 569.
// The kinds a bundle may ask.
const (
provisionConfig = "config"
provisionBound = "bound"
provisionSecret = "secret"
)
// provisionAsked is what a bundle names. Anything else it sends, a module's name among it, is not read.
type provisionAsked struct {
Kind string `json:"kind"`
Key string `json:"key"`
}
// ProvisionChange is what a watch hands a bundle: the value, or, when it is no longer answered, why.
type ProvisionChange struct {
Kind string `json:"kind"`
Key string `json:"key"`
Value json.RawMessage `json:"value,omitempty"`
// Refused is why the value is no longer answered: the module taken off the machine, the key no longer declared.
Refused string `json:"refused,omitempty"`
}
func readProvisionAsked(verb string, params json.RawMessage) (provisionAsked, error) {
var asked provisionAsked
if err := json.Unmarshal(params, &asked); err != nil || asked.Kind == "" || asked.Key == "" {
return asked, fmt.Errorf("mesh/provision.%s names no value: {kind, key}", verb)
}
switch asked.Kind {
case provisionConfig, provisionBound:
return asked, nil
case provisionSecret:
return asked, fmt.Errorf("the secret %s is not asked for at runtime while the module's code runs in the "+
"tool runner: the vault hands a secret only to a caller the bus proves, and the bus proves only the tool "+
"runner here (novox/hq ADR 0370 §2, ADR 0349 §11); the module's secrets reach it through its files, "+
"until it runs under a bus account of its own (issue 569)", asked.Key)
}
return asked, fmt.Errorf("mesh/provision.%s: %q is not a kind of value a module reads at runtime: config or bound",
verb, asked.Kind)
}
// Provision answers a bundle's `get` of one of its module's values.
func (b *moduleBus) Provision(verb string, params json.RawMessage) (json.RawMessage, error) {
if verb != "get" {
return nil, fmt.Errorf("the runtime answers no mesh/provision.%s", verb)
}
asked, err := readProvisionAsked(verb, params)
if err != nil {
return nil, err
}
return b.all.conn.AskProvisioner(b.module, asked.Kind, asked.Key)
}
// ProvisionWatch hands deliver the value now and each change after, until stopped. A value not answered now is
// refused now, with the client's reason.
func (b *moduleBus) ProvisionWatch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
asked, err := readProvisionAsked("watch", params)
if err != nil {
return nil, err
}
value, err := b.all.conn.AskProvisioner(b.module, asked.Kind, asked.Key)
if err != nil {
return nil, err
}
w := &provisionWatch{bus: b, asked: asked, deliver: deliver, last: value}
if err := w.hand(ProvisionChange{Kind: asked.Kind, Key: asked.Key, Value: value}); err != nil {
return nil, err
}
stop, err := b.all.provisioned.add(b.all, b.module, w)
if err != nil {
return nil, err
}
return stop, nil
}
// provisionWatch is one watch a bundle holds: what it asked, and what it was handed last.
type provisionWatch struct {
bus *moduleBus
asked provisionAsked
deliver func(json.RawMessage) error
mu sync.Mutex // one change at a time, in order
last json.RawMessage
refused string
stopped bool
}
func (w *provisionWatch) hand(change ProvisionChange) error {
raw, err := json.Marshal(change)
if err != nil {
return err
}
return w.deliver(raw)
}
// again asks once more after the client said something moved, and hands the bundle what changed.
func (w *provisionWatch) again() {
w.mu.Lock()
defer w.mu.Unlock()
if w.stopped {
return
}
value, err := w.bus.all.conn.AskProvisioner(w.bus.module, w.asked.Kind, w.asked.Key)
change := ProvisionChange{Kind: w.asked.Kind, Key: w.asked.Key}
var refused *bus.ProvisionRefused
switch {
case errors.As(err, &refused):
if w.refused == refused.Reason {
return
}
change.Refused = refused.Reason
case err != nil:
// The client not reached: asked again on its next word, which it says whenever it comes up.
w.bus.all.logf("[mesh-tools] %s's watch of its %s %s could not ask again: %v", w.bus.module, w.asked.Kind,
w.asked.Key, err)
return
default:
if w.refused == "" && bytes.Equal(value, w.last) {
return
}
change.Value = value
}
if err := w.hand(change); err != nil {
w.bus.all.logf("[mesh-tools] %s did not take a change of its %s %s: %v", w.bus.module, w.asked.Kind,
w.asked.Key, err)
return
}
w.last, w.refused = value, change.Refused
}
// provisionWatches is every watch on this runtime, by module, and the one subscription that hears the client.
type provisionWatches struct {
mu sync.Mutex
of map[string]map[*provisionWatch]bool
stop func()
}
func (p *provisionWatches) add(all *consumers, module string, w *provisionWatch) (func(), error) {
p.mu.Lock()
defer p.mu.Unlock()
if p.stop == nil {
stop, err := all.conn.OnProvisionChanged(func(module string, changed bus.ProvisionChanged) {
for _, w := range p.told(module, changed) {
go w.again()
}
})
if err != nil {
return nil, fmt.Errorf("the runtime cannot hear its machine's provisioner client: %w", err)
}
p.stop = stop
}
if p.of == nil {
p.of = map[string]map[*provisionWatch]bool{}
}
if p.of[module] == nil {
p.of[module] = map[*provisionWatch]bool{}
}
p.of[module][w] = true
return func() {
w.mu.Lock()
w.stopped = true
w.mu.Unlock()
p.mu.Lock()
defer p.mu.Unlock()
delete(p.of[module], w)
}, nil
}
// told is the watches of module that what moved concerns.
func (p *provisionWatches) told(module string, changed bus.ProvisionChanged) []*provisionWatch {
p.mu.Lock()
defer p.mu.Unlock()
var out []*provisionWatch
for w := range p.of[module] {
if w.asked.Kind == changed.Kind && (changed.Key == "" || changed.Key == w.asked.Key) {
out = append(out, w)
}
}
return out
}
// stopModule stops every watch of one module, whose bundle is loaded again.
func (p *provisionWatches) stopModule(module string) {
p.mu.Lock()
defer p.mu.Unlock()
for w := range p.of[module] {
w.mu.Lock()
w.stopped = true
w.mu.Unlock()
}
delete(p.of, module)
}
func (p *provisionWatches) stopAll() {
p.mu.Lock()
defer p.mu.Unlock()
for _, ws := range p.of {
for w := range ws {
w.mu.Lock()
w.stopped = true
w.mu.Unlock()
}
}
p.of = nil
if p.stop != nil {
p.stop()
p.stop = nil
}
}
@@ -0,0 +1,203 @@
package runtime
import (
"encoding/json"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// A module's code reads its values at runtime through moduleBus.Provision and ProvisionWatch (novox/hq ADR 0349
// §14, ADR 0370, issue 685), against a real bus with a stand-in for the machine's provisioner client: the runtime
// asks as the module whose bundle asks, whatever the bundle names; a watch hands the current value, then each change
// the client says moved, and a refusal once; a secret is refused without asking.
// standInClient answers prov.ask.<node> from its values, keyed `<module>.<kind>.<key>`, and keeps every ask.
type standInClient struct {
mu sync.Mutex
values map[string]string
asks []string
nc *nats.Conn
}
func aStandInClient(t *testing.T, node string, values map[string]string) *standInClient {
t.Helper()
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
c := &standInClient{values: values, nc: nc}
if _, err := nc.Subscribe("prov.ask."+node, func(m *nats.Msg) {
var a struct{ Module, Kind, Key string }
_ = json.Unmarshal(m.Data, &a)
c.mu.Lock()
c.asks = append(c.asks, string(m.Data))
v, ok := c.values[a.Module+"."+a.Kind+"."+a.Key]
c.mu.Unlock()
if !ok {
_ = m.Respond([]byte(`{"error":"` + a.Module + ` declares no config value ` + a.Key + `"}`))
return
}
_ = m.Respond([]byte(`{"value":` + v + `}`))
}); err != nil {
t.Fatal(err)
}
_ = nc.Flush()
return c
}
func (c *standInClient) set(key, value string) {
c.mu.Lock()
defer c.mu.Unlock()
if value == "" {
delete(c.values, key)
return
}
c.values[key] = value
}
func (c *standInClient) asked() []string {
c.mu.Lock()
defer c.mu.Unlock()
return append([]string(nil), c.asks...)
}
func (c *standInClient) say(t *testing.T, node, module, body string) {
t.Helper()
if err := c.nc.Publish("prov.changed."+node+"."+module, []byte(body)); err != nil {
t.Fatal(err)
}
}
func TestABundleAsksForItsValuesAsItsOwnModule(t *testing.T) {
client := aStandInClient(t, "anchor", map[string]string{
"claude-code.config.managed_settings": `{"spinnerTipsEnabled":false}`,
"dunst.config.managed_settings": `"dunst's"`,
})
conn := connect(t, "node-tools", "anchor")
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
b := all.forModule("claude-code").(*moduleBus)
// Its own value, though the bundle names another module: the runtime names the module from the bundle.
got, err := b.Provision("get", json.RawMessage(`{"kind":"config","key":"managed_settings","module":"dunst"}`))
if err != nil || string(got) != `{"spinnerTipsEnabled":false}` {
t.Fatalf("answered %s, %v", got, err)
}
if asks := client.asked(); len(asks) != 1 || asks[0] != `{"key":"managed_settings","kind":"config","module":"claude-code"}` {
t.Fatalf("the client was asked %v", asks)
}
// The client's refusal, in its words.
if _, err := b.Provision("get", json.RawMessage(`{"kind":"config","key":"nothing"}`)); err == nil ||
!strings.Contains(err.Error(), "declares no config value nothing") {
t.Errorf("an undeclared key: %v", err)
}
// A secret, refused here and never asked; and what is not an ask.
before := len(client.asked())
for params, want := range map[string]string{
`{"kind":"secret","key":"gitea-read"}`: "the vault hands a secret only to a caller the bus proves",
`{"kind":"setting","key":"role"}`: "not a kind of value",
`{"key":"role"}`: "names no value",
} {
if _, err := b.Provision("get", json.RawMessage(params)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%s: %v, want %q", params, err, want)
}
}
if _, err := b.ProvisionWatch(json.RawMessage(`{"kind":"secret","key":"gitea-read"}`), func(json.RawMessage) error {
return nil
}); err == nil {
t.Error("a secret was watched")
}
if n := len(client.asked()); n != before {
t.Errorf("the client was asked for what the runtime refuses: %v", client.asked()[before:])
}
}
func TestABundlesWatchIsHandedTheValueAndEachChange(t *testing.T) {
client := aStandInClient(t, "anchor", map[string]string{"claude-code.config.managed_settings": `{"a":1}`})
conn := connect(t, "node-tools", "anchor")
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
t.Cleanup(all.stopAll)
b := all.forModule("claude-code").(*moduleBus)
handed := make(chan ProvisionChange, 16)
stop, err := b.ProvisionWatch(json.RawMessage(`{"kind":"config","key":"managed_settings","module":"dunst"}`),
func(raw json.RawMessage) error {
var c ProvisionChange
if err := json.Unmarshal(raw, &c); err != nil {
t.Errorf("handed %s", raw)
}
handed <- c
return nil
})
if err != nil {
t.Fatal(err)
}
next := func(what string) ProvisionChange {
t.Helper()
select {
case c := <-handed:
return c
case <-time.After(5 * time.Second):
t.Fatalf("nothing handed: %s", what)
}
return ProvisionChange{}
}
nothing := func(what string) {
t.Helper()
select {
case c := <-handed:
t.Errorf("%s: handed %+v", what, c)
case <-time.After(300 * time.Millisecond):
}
}
if c := next("the current value"); string(c.Value) != `{"a":1}` || c.Kind != "config" || c.Key != "managed_settings" {
t.Fatalf("first handed %+v", c)
}
// A change, said by the client: asked again, handed.
client.set("claude-code.config.managed_settings", `{"a":2}`)
client.say(t, "anchor", "claude-code", `{"kind":"config","key":"managed_settings"}`)
if c := next("the change"); string(c.Value) != `{"a":2}` {
t.Fatalf("after the change, handed %+v", c)
}
// Every key of the kind said to have moved, the value the same: nothing handed. Another module's or another
// key's word: not even asked.
client.say(t, "anchor", "claude-code", `{"kind":"config"}`)
nothing("the same value again")
before := len(client.asked())
client.say(t, "anchor", "dunst", `{"kind":"config","key":"managed_settings"}`)
client.say(t, "anchor", "claude-code", `{"kind":"config","key":"role"}`)
nothing("another's word")
if n := len(client.asked()); n != before {
t.Errorf("asked again for another's word: %v", client.asked()[before:])
}
// No longer answered: handed the refusal, once.
client.set("claude-code.config.managed_settings", "")
client.say(t, "anchor", "claude-code", `{"kind":"config"}`)
if c := next("the refusal"); c.Refused == "" || len(c.Value) != 0 {
t.Fatalf("handed %+v, want the refusal", c)
}
client.say(t, "anchor", "claude-code", `{"kind":"config"}`)
nothing("the same refusal again")
// Answered again, and then stopped: nothing after.
client.set("claude-code.config.managed_settings", `{"a":3}`)
client.say(t, "anchor", "claude-code", `{"kind":"config"}`)
if c := next("answered again"); string(c.Value) != `{"a":3}` {
t.Fatalf("handed %+v", c)
}
stop()
client.set("claude-code.config.managed_settings", `{"a":4}`)
client.say(t, "anchor", "claude-code", `{"kind":"config"}`)
nothing("a stopped watch")
for _, a := range client.asked() {
if !strings.Contains(a, `"module":"claude-code"`) {
t.Errorf("asked as another module: %s", a)
}
}
}
+442
View File
@@ -0,0 +1,442 @@
package runtime
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// A runtime serves a module's bundle as soon as it is in place, again when it is replaced, and no
// more once it is gone, with no restart (issue 1945, note 23: mesh-files' bundle arrived on a machine
// a quarter of an hour after its runtime started, and its tools were missing until a hand restart).
// syncLogs is a logger that keeps the lines, safe to read while the runtime's watcher writes.
type syncLogs struct {
mu sync.Mutex
lines []string
}
func (l *syncLogs) Logf(format string, args ...any) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, fmt.Sprintf(format, args...))
}
func (l *syncLogs) Has(fragments ...string) bool {
l.mu.Lock()
defer l.mu.Unlock()
for _, line := range l.lines {
all := true
for _, f := range fragments {
all = all && strings.Contains(line, f)
}
if all {
return true
}
}
return false
}
func (l *syncLogs) All() string {
l.mu.Lock()
defer l.mu.Unlock()
return strings.Join(l.lines, "\n")
}
// bundleServing is a tools bundle, MCP over stdio with no SDK, whose one tool is named tool.
func bundleServing(tool string) string { return bundleScript(tool, false, false) }
// bundleScript is a tools bundle whose one tool is named tool. With subscribe it asks the runtime for its
// module's events and writes "<tool> subscribed" and "<tool> handled <n>" to $EVENT_LOG; with writeBeside it
// writes a file beside itself as it starts, as a bundle keeping a cache or a socket there does.
func bundleScript(tool string, subscribe, writeBeside bool) string {
py := func(b bool) string {
if b {
return "True"
}
return "False"
}
return `#!/usr/bin/env python3
import json, os, sys
TOOL = "` + tool + `"
SUBSCRIBE = ` + py(subscribe) + `
if ` + py(writeBeside) + `:
with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "cache"), "w") as f:
f.write(TOOL + " " + str(os.getpid()))
def say(m):
m["jsonrpc"] = "2.0"; sys.stdout.write(json.dumps(m) + "\n"); sys.stdout.flush()
def log(line):
with open(os.environ["EVENT_LOG"], "a") as f:
f.write(line + "\n")
TOOLS = [{"name": TOOL, "description": "answers its name", "inputSchema": {"type": "object", "properties": {}}}]
for line in sys.stdin:
req = json.loads(line); rid = req.get("id"); m = req.get("method"); p = req.get("params") or {}
if m is None:
if rid == "subscribe":
log(TOOL + " subscribed")
continue
if m == "initialize":
say({"id": rid, "result": {"protocolVersion": "2025-03-26", "capabilities": {"tools": {}}, "serverInfo": {"name": "ledger", "version": "1"}}})
elif m == "tools/list":
say({"id": rid, "result": {"tools": TOOLS}})
if SUBSCRIBE:
say({"id": "subscribe", "method": "mesh/subscribe", "params": {}})
elif m == "tools/call":
say({"id": rid, "result": {"content": [{"type": "text", "text": json.dumps({"tool": TOOL})}]}})
elif m == "mesh/event":
body = (p.get("envelope") or {}).get("body") or {}
log(TOOL + " handled " + str(body.get("n")))
say({"id": rid, "result": {}})
elif rid is not None:
say({"id": rid, "result": {}})
`
}
// place puts a bundle serving tool at path the way a delivery does: written beside it, made
// executable, and renamed into place, its directories made as needed.
func place(t *testing.T, path, tool string) {
t.Helper()
placeScript(t, path, bundleServing(tool))
}
func placeScript(t *testing.T, path, script string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
tmp := path + ".new"
if err := os.WriteFile(tmp, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Rename(tmp, path); err != nil {
t.Fatal(err)
}
}
// fastSettle makes the watcher act on a burst of changes quickly in a test.
func fastSettle(t *testing.T) {
was := Settle
Settle = 50 * time.Millisecond
t.Cleanup(func() { Settle = was })
}
type reloadBed struct {
mesh *mt.Mesh
nodeTools *bus.Conn
asker *bus.Conn
logs *syncLogs
entry string
}
func newReloadBed(t *testing.T) *reloadBed {
t.Helper()
fastSettle(t)
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("ledger", "anchor", false, nil))
return &reloadBed{
mesh: mesh,
nodeTools: connect(t, "node-tools", "anchor"),
asker: connect(t, "console", "workstation"),
logs: &syncLogs{},
// Below directories that do not exist yet, as on a machine before the module's first delivery.
entry: filepath.Join(t.TempDir(), "bundles", "ledger", "tools", "serve.py"),
}
}
func (b *reloadBed) run(t *testing.T) {
t.Helper()
stop, err := Run(b.nodeTools, []Served{{"ledger", []string{b.entry}}}, nil, b.logs.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
func (b *reloadBed) answers(t *testing.T, tool string) {
t.Helper()
mt.Until(t, func() error {
got, err := call(t, b.asker, "ledger."+tool+"@anchor", map[string]any{})
if err != nil {
return fmt.Errorf("ledger.%s is not served: %v\n%s", tool, err, b.logs.All())
}
if !strings.Contains(got, `"tool":"`+tool+`"`) {
return fmt.Errorf("ledger.%s answered %s", tool, got)
}
return nil
})
}
func (b *reloadBed) silent(t *testing.T, tool string) {
t.Helper()
mt.Until(t, func() error {
if _, err := call(t, b.asker, "ledger."+tool+"@anchor", map[string]any{}); err == nil {
return fmt.Errorf("ledger.%s is still served\n%s", tool, b.logs.All())
}
return nil
})
}
func announced(t *testing.T) string {
t.Helper()
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
msg, err := nc.Request("$SRV.INFO.node-tools.anchor", nil, 2*time.Second)
if err != nil {
t.Fatal(err)
}
var i micro.Info
if err := json.Unmarshal(msg.Data, &i); err != nil {
t.Fatal(err)
}
var out []string
for _, e := range i.Endpoints {
out = append(out, e.Subject)
}
return strings.Join(out, "\n")
}
// 1945/AC19: a bundle placed after the runtime started is served, and announced, with no restart.
func TestABundlePlacedAfterStartIsServed(t *testing.T) {
b := newReloadBed(t)
b.run(t)
if !b.logs.Has("ledger's bundle", "does not exist") {
t.Errorf("the missing bundle was not said as missing:\n%s", b.logs.All())
}
place(t, b.entry, "first")
b.answers(t, "first")
mt.Until(t, func() error {
if a := announced(t); !strings.Contains(a, "mesh.mod.ledger.tool.first.anchor") {
return fmt.Errorf("discovery does not announce the placed bundle's tool:\n%s", a)
}
return nil
})
}
// 1945/AC20: a bundle replaced while served is loaded again: its new tool is served, its old one no more.
func TestAReplacedBundleIsReloaded(t *testing.T) {
b := newReloadBed(t)
place(t, b.entry, "first")
b.run(t)
b.answers(t, "first")
place(t, b.entry, "second")
b.answers(t, "second")
b.silent(t, "first")
mt.Until(t, func() error {
a := announced(t)
if !strings.Contains(a, "mesh.mod.ledger.tool.second.anchor") || strings.Contains(a, "mesh.mod.ledger.tool.first.anchor") {
return fmt.Errorf("discovery does not follow the replaced bundle:\n%s", a)
}
return nil
})
// The bundle's whole directory swapped for a new one, as an archive unpacked beside it and renamed in.
dir := filepath.Dir(b.entry)
fresh := dir + ".unpacked"
place(t, filepath.Join(fresh, filepath.Base(b.entry)), "third")
if err := os.Rename(dir, dir+".old"); err != nil {
t.Fatal(err)
}
if err := os.Rename(fresh, dir); err != nil {
t.Fatal(err)
}
b.answers(t, "third")
b.silent(t, "second")
// And changed again inside the directory swapped in: the watch follows the new directory.
place(t, b.entry, "fourth")
b.answers(t, "fourth")
}
// A replaced bundle's old tool stays gone when the module's membership is issued again: a stopped tool
// leaves no membership handler behind to bind it once more, and a call on it starts no unmanaged second
// copy of the bundle (issue 1945, review round 2).
func TestAReplacedBundlesOldToolStaysGoneOnTheNextMembership(t *testing.T) {
b := newReloadBed(t)
place(t, b.entry, "first")
b.run(t)
b.answers(t, "first")
place(t, b.entry, "second")
b.answers(t, "second")
b.silent(t, "first")
b.mesh.Issue(t, mt.MembershipOf("ledger", "anchor", false, nil))
b.answers(t, "second")
// Long enough for the reissued membership to have been heard, before the old tool is asked.
time.Sleep(300 * time.Millisecond)
if got, err := call(t, b.asker, "ledger.first@anchor", map[string]any{}); err == nil {
t.Fatalf("ledger.first is served again after the membership was reissued, and answered %s\n%s", got, b.logs.All())
}
}
// 1945/AC21: a bundle removed while served is no longer served, and its module says why.
func TestARemovedBundleIsNoLongerServed(t *testing.T) {
b := newReloadBed(t)
place(t, b.entry, "first")
b.run(t)
b.answers(t, "first")
if err := os.RemoveAll(filepath.Dir(b.entry)); err != nil {
t.Fatal(err)
}
b.silent(t, "first")
mt.Until(t, func() error {
got, err := call(t, b.asker, "ledger.tools@anchor", map[string]any{})
if err != nil {
return err
}
var answer ToolsAnswer
if err := json.Unmarshal([]byte(got), &answer); err != nil {
return err
}
if len(answer.Tools) != 0 || !strings.Contains(answer.Failed, "does not exist") {
return fmt.Errorf("ledger's tools after its bundle was removed: %s", got)
}
return nil
})
if a := announced(t); strings.Contains(a, "mesh.mod.ledger.tool.first") {
t.Errorf("discovery still announces the removed bundle's tool:\n%s", a)
}
}
// 1945/AC22: a bundle that cannot be started is said with its real reason: a missing file does not
// exist, and a file without an execute bit is not executable with its mode.
func TestABundleThatCannotStartSaysTheRealReason(t *testing.T) {
dir := t.TempDir()
missing := filepath.Join(dir, "nowhere", "serve.py")
if why := notStartable(missing); !strings.Contains(why, missing+" does not exist") || strings.Contains(why, "not executable") {
t.Errorf("a missing bundle said: %q", why)
}
plain := filepath.Join(dir, "serve.py")
if err := os.WriteFile(plain, []byte(bundleServing("first")), 0o644); err != nil {
t.Fatal(err)
}
if why := notStartable(plain); !strings.Contains(why, plain+" is not executable (mode -rw-r--r--)") {
t.Errorf("a bundle without an execute bit said: %q", why)
}
if why := notStartable(dir); !strings.Contains(why, "is a directory") {
t.Errorf("a directory said: %q", why)
}
if err := os.Chmod(plain, 0o755); err != nil {
t.Fatal(err)
}
if why := notStartable(plain); why != "" {
t.Errorf("a startable bundle said: %q", why)
}
}
// 1945/AC23: an event a module's consumer hears while the module is loaded again is offered again, never
// a panic of the runtime, and the bundle loaded in its place is handed the next offer.
func TestAnEventHeardDuringAReloadIsOfferedAgainNotAPanic(t *testing.T) {
c := &consumers{logf: func(string, ...any) {}, of: map[string]*moduleEvents{}}
var got []string
before := func(json.RawMessage) error { got = append(got, "before"); return nil }
old := &moduleEvents{delivers: []*func(json.RawMessage) error{&before}}
c.of["ledger"] = old
c.dropModule("ledger")
env := bus.Envelope{Key: "alpha.happened", Body: json.RawMessage(`{"n":1}`)}
hand := func(m *moduleEvents) (err error) {
defer func() {
if r := recover(); r != nil {
t.Fatalf("an event heard during a reload panicked the runtime: %v", r)
}
}()
return c.handOut("ledger", m, env)
}
if err := hand(old); err == nil {
t.Error("an event heard while the module is loaded again was taken, not offered again")
}
after := func(json.RawMessage) error { got = append(got, "after"); return nil }
fresh := &moduleEvents{delivers: []*func(json.RawMessage) error{&after}}
c.of["ledger"] = fresh
if err := hand(fresh); err != nil {
t.Errorf("the bundle loaded again was not handed the event: %v", err)
}
if err := hand(old); err == nil {
t.Error("the stopped consumer's handler handed an event on after the reload")
}
if strings.Join(got, ",") != "after" {
t.Errorf("handed to %v, want the bundle loaded again alone", got)
}
}
// 1945/AC23, whole: a module's events reach its bundle before a reload, around it, and after it.
func TestAModulesEventsReachItsBundleLoadedAgain(t *testing.T) {
fastSettle(t)
restore := bus.NakDelay
bus.NakDelay = 200 * time.Millisecond
t.Cleanup(func() { bus.NakDelay = restore })
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("ledger", "anchor", false, nil))
mesh.Consumer(t, "anchor", "ledger", []string{"mesh.mod.alpha.event.>"}, 2*time.Second)
nodeTools := connect(t, "node-tools", "anchor")
logs := &syncLogs{}
dir := t.TempDir()
log := filepath.Join(dir, "events.log")
entry := filepath.Join(dir, "bundles", "ledger", "tools", "serve.py")
has := func(line string) error {
b, _ := os.ReadFile(log)
for _, l := range strings.Split(string(b), "\n") {
if l == line {
return nil
}
}
return fmt.Errorf("no %q yet in:\n%s\nruntime:\n%s", line, b, logs.All())
}
placeScript(t, entry, bundleScript("first", true, false))
stop, err := Run(nodeTools, []Served{{"ledger", []string{entry}}}, map[string]map[string]string{"ledger": {"EVENT_LOG": log}}, logs.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
mt.Until(t, func() error { return has("first subscribed") })
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 1})
mt.Until(t, func() error { return has("first handled 1") })
placeScript(t, entry, bundleScript("second", true, false))
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 2})
mt.Until(t, func() error {
if has("first handled 2") != nil && has("second handled 2") != nil {
return has("second handled 2")
}
return nil
})
mt.Until(t, func() error { return has("second subscribed") })
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 3})
mt.Until(t, func() error { return has("second handled 3") })
}
// 1945/AC24: a bundle that writes a file beside itself as it starts is loaded once at start and once
// when it is replaced, never again by its own writing.
func TestABundleWritingBesideItselfIsNotLoadedAgainByIt(t *testing.T) {
b := newReloadBed(t)
t.Setenv("EVENT_LOG", filepath.Join(t.TempDir(), "unused.log"))
placeScript(t, b.entry, bundleScript("first", false, true))
b.run(t)
b.answers(t, "first")
reloads := func() int {
return strings.Count(b.logs.All(), "loading it again")
}
time.Sleep(20 * Settle)
if n := reloads(); n != 0 {
t.Fatalf("the bundle was loaded again %d time(s) by its own writing:\n%s", n, b.logs.All())
}
placeScript(t, b.entry, bundleScript("second", false, true))
b.answers(t, "second")
time.Sleep(20 * Settle)
if n := reloads(); n != 1 {
t.Errorf("replaced once, loaded again %d time(s):\n%s", n, b.logs.All())
}
}
+438 -146
View File
@@ -14,6 +14,7 @@ import (
"sort"
"strings"
"sync"
"time"
"github.com/novox/mesh-tools/node-tools/internal/announce"
"github.com/novox/mesh-tools/node-tools/internal/bus"
@@ -134,7 +135,51 @@ type registration struct {
tools []launch.Tool
}
// Run launches, binds and serves. It answers a stop function.
// loadedModule is one served module as its bundles were last loaded: what they registered, what of it
// is served, and each entrypoint's signature at that load, which tells the watch that it changed.
type loadedModule struct {
served Served
seen map[string]string // entrypoint → its signature when it was loaded
children []func()
regs []registration // every registration its bundles made
own []registration // those under the name of a module served here
failed string
names []string // the tools it serves, as module.tool
serving []func() // its tools' subscriptions
}
// unload stops serving the module's tools and stops its bundles.
func (l *loadedModule) unload() {
for i := len(l.serving) - 1; i >= 0; i-- {
l.serving[i]()
}
l.serving, l.names = nil, nil
for i := len(l.children) - 1; i >= 0; i-- {
l.children[i]()
}
l.children = nil
}
// runner is the runtime at work: every served module as last loaded, loaded again when its bundle
// appears, changes or goes (issue 1945, note 23).
type runner struct {
conn *bus.Conn
logf func(string, ...any)
envFor func(module string) []string
events *consumers
modules []string
isServed map[string]bool
mu sync.Mutex
of map[string]*loadedModule
verbs map[string]func() // each module's `tools` verb, where it is answered
seatsAgain func()
stops []func() // the watch, the announcement and the seats, in the order they began
}
// Run launches, binds and serves, and serves each module's bundle again when it appears, changes or
// goes, with no restart. It answers a stop function.
func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, logf func(string, ...any)) (func(), error) {
if account := os.Getenv(OperatorAccount); account != "" {
home := ""
@@ -143,153 +188,324 @@ func Run(conn *bus.Conn, served []Served, envs map[string]map[string]string, log
}
logf("[mesh-tools] the operator's account here is %s%s", account, home)
}
modules := make([]string, 0, len(served))
isServed := map[string]bool{}
r := &runner{conn: conn, logf: logf, isServed: map[string]bool{}, of: map[string]*loadedModule{},
verbs: map[string]func(){}}
for _, s := range served {
modules = append(modules, s.Module)
isServed[s.Module] = true
r.modules = append(r.modules, s.Module)
r.isServed[s.Module] = true
conn.Follow(s.Module)
}
node := conn.Node()
base := os.Environ()
envFor := func(module string) []string { return BundleEnv(base, envs[module], module, node) }
r.envFor = func(module string) []string { return BundleEnv(base, envs[module], module, node) }
// The module's events, for every child of it that subscribes (ADR 0198): one consumer per module,
// bound the first time any of its children subscribes, each event handed to every child that did.
events := &consumers{conn: conn, logf: logf, of: map[string]*moduleEvents{}}
r.events = &consumers{conn: conn, logf: logf, of: map[string]*moduleEvents{}}
failed := map[string]string{}
var registrations []registration
var stops []func()
for _, s := range served {
for _, entry := range s.Entrypoints {
path, _ := filepath.Abs(entry)
module := s.Module
fail := func(why string) {
failed[module] = why
logf("[mesh-tools] %s's bundle %s failed to load: %s; its tools are not served here", module, entry, why)
}
if !launch.Executable(path) {
fail(path + " is not executable; a bundle the runtime serves is started, never imported, and its build makes it executable (novox/hq ADR 0193)")
continue
}
child, err := launch.Start(module, path, envFor(module), events.forModule(module), logf)
if err != nil {
fail(err.Error())
continue
}
stops = append(stops, child.Stop)
for _, r := range child.Registrations {
registrations = append(registrations, registration{module: r.Module, owner: module, tools: r.Tools})
}
r.of[s.Module] = r.load(s)
}
for _, m := range r.modules {
if err := refusal(r.of[m].own); err != nil {
r.stop()
return nil, err
}
}
claimed := map[string]bool{}
for _, m := range modules {
if mem := conn.Membership(m); mem != nil {
for _, s := range mem.Seats {
claimed[s.Seat] = true
}
}
}
var own []registration
for _, r := range registrations {
switch {
case isServed[r.module]:
own = append(own, r)
case claimed[r.module]:
default:
logf(`[mesh-tools] %s registers tools under "%s", which is neither a module served here nor a seat one of them claims; not served until the mesh issues the claim`, r.owner, r.module)
}
}
stopAll := func() {
events.stopAll()
for i := len(stops) - 1; i >= 0; i-- {
stops[i]()
}
}
for _, r := range own {
seen := map[string]bool{}
for _, t := range r.tools {
if t.Name == ToolsVerb {
stopAll()
return nil, fmt.Errorf(`%s names a tool "%s", which is the verb the runtime answers for every module with what it serves (novox/hq ADR 0152) — refused, rename it`, r.module, ToolsVerb)
}
if seen[t.Name] {
stopAll()
return nil, fmt.Errorf("%s exposes two tools named %s — refused", r.module, t.Name)
}
seen[t.Name] = true
}
}
var names []string
byModule := map[string][]launch.Tool{}
for _, r := range own {
for _, t := range r.tools {
t := t
// Told who called, from the subject the call arrived on, never from its body (novox/hq issue 365).
stop, err := conn.HandleCalled(r.module+"."+t.Name, func(caller string, body json.RawMessage) (any, error) {
return t.Run(caller, argsOf(body))
})
if err != nil {
logf("[mesh-tools] cannot serve %s.%s: %v", r.module, t.Name, err)
continue
}
names = append(names, r.module+"."+t.Name)
stops = append(stops, stop)
}
byModule[r.module] = append(byModule[r.module], r.tools...)
}
for _, module := range modules {
module := module
tools := byModule[module]
why := failed[module]
if len(tools) == 0 && why == "" {
continue // a pure-events module: silent, as it always was
}
stop, err := conn.Handle(module+"."+ToolsVerb, func(json.RawMessage) (any, error) {
answer := ToolsAnswer{Module: module, Tools: []ListedTool{}, Failed: why}
for _, t := range tools {
answer.Tools = append(answer.Tools, ListedTool{Name: t.Name, Description: t.Description,
Input: t.Input, Subjects: subjectsOf(conn, module, t.Name)})
}
return answer, nil
})
if err == nil {
stops = append(stops, stop)
}
}
failedNames := make([]string, 0, len(failed))
for _, m := range modules {
if _, f := failed[m]; f {
r.mu.Lock()
var names, failedNames []string
for _, m := range r.modules {
r.serve(r.of[m])
names = append(names, r.of[m].names...)
if r.of[m].failed != "" {
failedNames = append(failedNames, m)
}
}
line := fmt.Sprintf("[mesh-tools] serving %d tool(s) for %d module(s): %s", len(names), len(modules), orNone(names))
r.bindVerbs()
r.mu.Unlock()
line := fmt.Sprintf("[mesh-tools] serving %d tool(s) for %d module(s): %s", len(names), len(r.modules), orNone(names))
if len(failedNames) > 0 {
line += fmt.Sprintf("; not serving %s, whose bundle(s) failed to load", strings.Join(failedNames, ", "))
}
logf("%s", line)
stops = append(stops, serveSeats(conn, modules, registrations, logf))
stopSeats, again := serveSeats(conn, r.modules, r.registrations, logf)
r.seatsAgain = again
r.stops = append(r.stops, stopSeats)
// **What it serves, it announces** (novox/hq ADR 0197): the NATS services protocol's discovery,
// answered with what is served at the moment it is asked — re-served memberships included.
// answered with what is served at the moment it is asked — re-served memberships and bundles loaded
// again included, so a bundle served after start is announced as one served at start.
announced, err := announce.Serve(conn, announce.Service{
Name: conn.Module(), ID: instanceOf(conn),
Description: "the mesh's tool runtime on " + node + ": every assigned module's tools and the seats they hold",
Metadata: map[string]string{"node": node},
}, func() []announce.Endpoint { return endpointsOf(conn, own, registrations, modules) })
}, func() []announce.Endpoint { return endpointsOf(conn, r.own(), r.registrations(), r.modules) })
if err != nil {
logf("[mesh-tools] cannot announce what it serves: %v", err)
} else {
stops = append(stops, announced)
r.stops = append(r.stops, announced)
}
// **A bundle is served as soon as it is in place** (issue 1945, note 23): the entrypoints'
// directories are watched, and a module whose bundle appears, changes or goes is loaded again.
var paths []string
for _, s := range served {
for _, entry := range s.Entrypoints {
path, _ := filepath.Abs(entry)
paths = append(paths, path)
}
}
if len(paths) > 0 {
stopWatch, err := watchBundles(paths, r.reconcile, logf)
if err != nil {
logf("[mesh-tools] cannot watch the bundles: %v; a bundle placed or changed later is served only after a restart", err)
} else {
r.stops = append(r.stops, stopWatch)
}
}
conn.Flush()
return stopAll, nil
return r.stop, nil
}
// stop ends the watch, the announcement and the seats, unbinds the modules' consumers, and stops every
// module's tools and bundles.
func (r *runner) stop() {
for i := len(r.stops) - 1; i >= 0; i-- {
r.stops[i]()
}
r.stops = nil
r.events.stopAll()
r.mu.Lock()
for _, stop := range r.verbs {
stop()
}
r.verbs = map[string]func(){}
loaded := make([]*loadedModule, 0, len(r.modules))
for _, m := range r.modules {
if l := r.of[m]; l != nil {
loaded = append(loaded, l)
}
}
r.mu.Unlock()
for i := len(loaded) - 1; i >= 0; i-- {
loaded[i].unload()
}
}
// claimed is every seat a served module's membership names.
func (r *runner) claimed() map[string]bool {
out := map[string]bool{}
for _, m := range r.modules {
if mem := r.conn.Membership(m); mem != nil {
for _, s := range mem.Seats {
out[s.Seat] = true
}
}
}
return out
}
// load launches one module's bundles and learns what they register; it serves nothing yet.
func (r *runner) load(s Served) *loadedModule {
l := &loadedModule{served: s, seen: map[string]string{}}
claimed := r.claimed()
for _, entry := range s.Entrypoints {
path, _ := filepath.Abs(entry)
module := s.Module
l.seen[path] = signatureOf(path)
fail := func(why string) {
l.failed = why
r.logf("[mesh-tools] %s's bundle %s failed to load: %s; its tools are not served here", module, entry, why)
}
if why := notStartable(path); why != "" {
fail(why)
continue
}
child, err := launch.Start(module, path, r.envFor(module), r.events.forModule(module), r.logf)
if err != nil {
fail(err.Error())
continue
}
l.children = append(l.children, child.Stop)
for _, reg := range child.Registrations {
got := registration{module: reg.Module, owner: module, tools: reg.Tools}
l.regs = append(l.regs, got)
switch {
case r.isServed[got.module]:
l.own = append(l.own, got)
case claimed[got.module]:
default:
r.logf(`[mesh-tools] %s registers tools under "%s", which is neither a module served here nor a seat one of them claims; not served until the mesh issues the claim`, got.owner, got.module)
}
}
}
return l
}
// refusal is why a module's registrations cannot be served: a tool named as the verb the runtime
// answers for every module, or two tools of one name.
func refusal(own []registration) error {
for _, r := range own {
seen := map[string]bool{}
for _, t := range r.tools {
if t.Name == ToolsVerb {
return fmt.Errorf(`%s names a tool "%s", which is the verb the runtime answers for every module with what it serves (novox/hq ADR 0152) — refused, rename it`, r.module, ToolsVerb)
}
if seen[t.Name] {
return fmt.Errorf("%s exposes two tools named %s — refused", r.module, t.Name)
}
seen[t.Name] = true
}
}
return nil
}
// serve binds a loaded module's tools. r.mu is held.
func (r *runner) serve(l *loadedModule) {
for _, reg := range l.own {
for _, t := range reg.tools {
t := t
// Told who called, from the subject the call arrived on, never from its body (novox/hq issue 365).
stop, err := r.conn.HandleCalled(reg.module+"."+t.Name, func(caller string, body json.RawMessage) (any, error) {
return t.Run(caller, argsOf(body))
})
if err != nil {
r.logf("[mesh-tools] cannot serve %s.%s: %v", reg.module, t.Name, err)
continue
}
l.names = append(l.names, reg.module+"."+t.Name)
l.serving = append(l.serving, stop)
}
}
}
// toolsOf is every tool served under a module's name, and why its bundle failed, if it did. r.mu is held.
func (r *runner) toolsOf(module string) ([]launch.Tool, string) {
var tools []launch.Tool
for _, m := range r.modules {
for _, reg := range r.of[m].own {
if reg.module == module {
tools = append(tools, reg.tools...)
}
}
}
return tools, r.of[module].failed
}
// bindVerbs answers each module's `tools` verb where it serves a tool or its bundle failed, and not
// where neither is so (a pure-events module is silent, as it always was). r.mu is held.
func (r *runner) bindVerbs() {
for _, module := range r.modules {
module := module
tools, why := r.toolsOf(module)
want := len(tools) > 0 || why != ""
stop, bound := r.verbs[module]
switch {
case want && !bound:
stop, err := r.conn.Handle(module+"."+ToolsVerb, func(json.RawMessage) (any, error) {
r.mu.Lock()
tools, why := r.toolsOf(module)
r.mu.Unlock()
answer := ToolsAnswer{Module: module, Tools: []ListedTool{}, Failed: why}
for _, t := range tools {
answer.Tools = append(answer.Tools, ListedTool{Name: t.Name, Description: t.Description,
Input: t.Input, Subjects: subjectsOf(r.conn, module, t.Name)})
}
return answer, nil
})
if err == nil {
r.verbs[module] = stop
}
case !want && bound:
stop()
delete(r.verbs, module)
}
}
}
// own is every registration served under a served module's name, now.
func (r *runner) own() []registration {
r.mu.Lock()
defer r.mu.Unlock()
var out []registration
for _, m := range r.modules {
out = append(out, r.of[m].own...)
}
return out
}
// registrations is every registration the bundles made, now.
func (r *runner) registrations() []registration {
r.mu.Lock()
defer r.mu.Unlock()
var out []registration
for _, m := range r.modules {
out = append(out, r.of[m].regs...)
}
return out
}
// reconcile loads again every module whose entrypoint appeared, changed or went since it was loaded.
// The watch calls it once a burst of changes settles; it runs on the watch's one goroutine.
func (r *runner) reconcile() {
for _, m := range r.modules {
r.mu.Lock()
l := r.of[m]
r.mu.Unlock()
for _, entry := range l.served.Entrypoints {
path, _ := filepath.Abs(entry)
was, now := l.seen[path], signatureOf(path)
if was == now {
continue
}
what := "changed"
switch {
case was == "":
what = "appeared"
case now == "":
what = "is gone"
}
r.reload(m, entry+" "+what)
break
}
}
}
// reload stops a module's tools and bundles, loads them again, serves what they register now, and
// serves its seats' verbs again; discovery answers with it from then on.
func (r *runner) reload(module, what string) {
r.mu.Lock()
old := r.of[module]
r.of[module] = &loadedModule{served: old.served, seen: old.seen}
r.mu.Unlock()
// Its events are refused first, so one heard while the old bundle stops is refused at once.
r.events.dropModule(module)
old.unload()
r.logf("[mesh-tools] %s's bundle %s; loading it again", module, what)
fresh := r.load(old.served)
if err := refusal(fresh.own); err != nil {
fresh.unload()
fresh.regs, fresh.own, fresh.failed = nil, nil, err.Error()
r.events.dropModule(module)
r.logf("[mesh-tools] %s's bundle is refused: %v; its tools are not served here", module, err)
}
r.mu.Lock()
r.of[module] = fresh
r.serve(fresh)
r.bindVerbs()
r.mu.Unlock()
if fresh.failed != "" {
r.logf("[mesh-tools] not serving %s, whose bundle failed to load", module)
} else {
r.logf("[mesh-tools] serving %d tool(s) for %s: %s", len(fresh.names), module, orNone(fresh.names))
}
if r.seatsAgain != nil {
r.seatsAgain()
}
r.conn.Flush()
}
// instanceOf is this runtime's instance on the bus: its machine, which is what tells two instances of
@@ -389,26 +605,31 @@ func subjectsOf(conn *bus.Conn, module, tool string) []string {
// serveSeats serves every verb of every seat a served module holds, where the mesh issued it, by
// the tool of the same name registered under the seat's name (ADR 0159, 0160) — and serves again
// whenever a membership changes. Whether this machine holds the seat is the bus's to decide.
func serveSeats(conn *bus.Conn, modules []string, registrations []registration, logf func(string, ...any)) func() {
impl := map[string]map[string]launch.Tool{}
for _, r := range registrations {
if impl[r.module] == nil {
impl[r.module] = map[string]launch.Tool{}
}
for _, t := range r.tools {
impl[r.module][t.Name] = t
}
}
// whenever a membership changes, and when again is called because a bundle was loaded again. Whether
// this machine holds the seat is the bus's to decide. It answers its stop and again.
func serveSeats(conn *bus.Conn, modules []string, current func() []registration, logf func(string, ...any)) (func(), func()) {
var mu sync.Mutex
var stops []func()
stopped := false
serve := func() {
mu.Lock()
defer mu.Unlock()
if stopped {
return
}
for _, s := range stops {
s()
}
stops = nil
impl := map[string]map[string]launch.Tool{}
for _, r := range current() {
if impl[r.module] == nil {
impl[r.module] = map[string]launch.Tool{}
}
for _, t := range r.tools {
impl[r.module][t.Name] = t
}
}
have := map[string]bool{}
for _, module := range modules {
m := conn.Membership(module)
@@ -446,7 +667,8 @@ func serveSeats(conn *bus.Conn, modules []string, registrations []registration,
s()
}
stops = nil
}
stopped = true
}, serve
}
// consumers holds, per module the runtime serves, the one durable consumer its events arrive on and
@@ -458,6 +680,8 @@ type consumers struct {
of map[string]*moduleEvents
// seatBound is each module's seat traffic bound once: a worker taken, a proof subject answered.
seatBound map[string]func()
// provisioned is every watch a module's code holds of its values at runtime (provision.go).
provisioned provisionWatches
}
type moduleEvents struct {
@@ -479,6 +703,25 @@ func (c *consumers) stopAll() {
stop()
}
c.seatBound = map[string]func(){}
c.provisioned.stopAll()
}
// dropModule unbinds one module's consumer, its seat traffic and its provision watches, so the bundle
// loaded again in its place binds them anew and no event is handed to a child that was stopped.
func (c *consumers) dropModule(module string) {
c.mu.Lock()
defer c.mu.Unlock()
if m := c.of[module]; m != nil && m.stop != nil {
m.stop()
}
delete(c.of, module)
for key, stop := range c.seatBound {
if strings.HasPrefix(key, module+" ") {
stop()
delete(c.seatBound, key)
}
}
c.provisioned.stopModule(module)
}
// forModule is the bus one launched bundle of a module reaches the mesh through.
@@ -560,23 +803,7 @@ func (b *moduleBus) Subscribe(deliver func(json.RawMessage) error) error {
return nil
}
module := b.module
stop, err := c.conn.ConsumeAs(module, func(env bus.Envelope) error {
raw, err := json.Marshal(env)
if err != nil {
return err
}
c.mu.Lock()
targets := append([]*func(json.RawMessage) error(nil), c.of[module].delivers...)
c.mu.Unlock()
for _, d := range targets {
if err := (*d)(raw); err != nil {
c.logf("[mesh-tools] %s did not take %s%s: %s; offered again in %s", module, env.Key,
eventRef(env), whyNotTaken(err), bus.NakDelay)
return err
}
}
return nil
})
stop, err := c.conn.ConsumeAs(module, func(env bus.Envelope) error { return c.handOut(module, m, env) })
if err != nil {
return err
}
@@ -585,6 +812,35 @@ func (b *moduleBus) Subscribe(deliver func(json.RawMessage) error) error {
return nil
}
// errModuleReloading refuses an event heard by a module's consumer that was stopped for a reload.
var errModuleReloading = errors.New("the module's bundle is being loaded again; offered again")
// handOut hands one event of a module's consumer m to every child of the module that subscribed; an
// event of a consumer that is no longer the module's is refused.
func (c *consumers) handOut(module string, m *moduleEvents, env bus.Envelope) error {
raw, err := json.Marshal(env)
if err != nil {
return err
}
c.mu.Lock()
if c.of[module] != m {
// Heard by a consumer stopped while the module is loaded again (issue 1945): refused, so the bus
// offers it again, to the bundle loaded in its place once it subscribes.
c.mu.Unlock()
return errModuleReloading
}
targets := append([]*func(json.RawMessage) error(nil), m.delivers...)
c.mu.Unlock()
for _, d := range targets {
if err := (*d)(raw); err != nil {
c.logf("[mesh-tools] %s did not take %s%s: %s; offered again in %s", module, env.Key,
eventRef(env), whyNotTaken(err), bus.NakDelay)
return err
}
}
return nil
}
// whyNotTaken says why a bundle did not take an event, so the line names who failed: the module's
// handler, in its own words — an error it answered — or the runtime not reaching it. A bundle's
// handler error read bare ("Unexpected end of JSON input") looked like the runtime's (issue 276).
@@ -732,6 +988,26 @@ func (b *moduleBus) Log(verb string, params json.RawMessage) (json.RawMessage, e
return json.Marshal(answer)
}
// StreamRead answers a bundle's read of one of its module's own streams (ADR 0363): {stream, subject, from, limit}, the
// stream by its local name, the subject an event pattern. The runtime refuses, with the reason, a stream the module's
// membership does not name.
func (b *moduleBus) StreamRead(params json.RawMessage) (json.RawMessage, error) {
var asked struct {
Stream string `json:"stream"`
Subject string `json:"subject"`
From uint64 `json:"from"`
Limit int `json:"limit"`
}
if err := json.Unmarshal(params, &asked); err != nil || asked.Stream == "" {
return nil, fmt.Errorf("mesh/stream.read names no stream: {stream, subject, from, limit}")
}
page, err := b.all.conn.StreamRead(b.module, asked.Stream, asked.Subject, asked.From, asked.Limit)
if err != nil {
return nil, err
}
return json.Marshal(page)
}
// Watch hands a bundle its module's state as it is and as it changes (ADR 0201): `{state, key}`, the
// key a pattern with `*` and `**`, empty for every key.
func (b *moduleBus) Watch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
@@ -756,6 +1032,9 @@ type seatAsked struct {
Worker string `json:"worker"`
Bucket string `json:"bucket"`
Key string `json:"key"`
// WaitMS is how long a pull waits for a message (none when 0); Fate what a settle gives one (issue 1887 A11).
WaitMS int `json:"wait_ms"`
Fate string `json:"fate"`
}
// Seat answers a bundle's seat traffic, each as the module and only as far as its membership lists:
@@ -789,6 +1068,19 @@ func (b *moduleBus) Seat(verb string, params json.RawMessage, handOn func(string
return json.RawMessage("null"), err
}
return json.Marshal(entry)
case "pull":
// One message from a worker the membership lists, never a prefetch: a build module pulls an ask only when it
// has room, from its workers in their order (novox/hq ADR 0525 §4; issue 1887 A11).
p, err := conn.SeatPull(b.module, asked.Worker, time.Duration(asked.WaitMS)*time.Millisecond)
if err != nil || p == nil {
return json.RawMessage("null"), err
}
return json.Marshal(p)
case "settle":
if err := conn.SeatSettle(b.module, asked.ID, asked.Fate); err != nil {
return nil, err
}
return json.RawMessage(`{}`), nil
case "take":
key := "take " + asked.Worker
if b.bound(key) {
+5 -1
View File
@@ -46,6 +46,7 @@ func same(t *testing.T, got, want string) {
// The node's runtime serves five modules' bundles on one credential — two TypeScript, one broken,
// one Python, one written against the protocol — and follows a re-issued membership (ADR 0175, 0193).
func TestTheNodesRuntimeServesFiveModulesAndFollowsAReissuedMembership(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", true, nil))
mesh.Issue(t, mt.MembershipOf("beta", "anchor", false, map[string][]string{"node-shelf": {"list", "clear"}}))
@@ -152,6 +153,7 @@ func TestTheNodesRuntimeServesFiveModulesAndFollowsAReissuedMembership(t *testin
// Each bundle is given its own environment and none of another's (ADR 0192), and the composed
// environments are not left in the runtime's.
func TestEachBundleIsGivenItsOwnEnvironment(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
for _, m := range []string{"gamma", "delta", "zeta"} {
mesh.Issue(t, mt.MembershipOf(m, "anchor", false, nil))
@@ -192,6 +194,7 @@ func TestEachBundleIsGivenItsOwnEnvironment(t *testing.T) {
// An entrypoint that is not executable is refused by name, and the others serve (ADR 0193).
func TestAnEntrypointThatIsNotExecutableIsRefused(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "anchor", false, nil))
mesh.Issue(t, mt.MembershipOf("plain", "anchor", false, nil))
@@ -206,7 +209,7 @@ func TestAnEntrypointThatIsNotExecutableIsRefused(t *testing.T) {
t.Fatal(err)
}
defer stop()
if !logs.Has("plain's bundle", "many-alpha.mjs failed to load:", "is not executable; a bundle the runtime serves is started, never imported") {
if !logs.Has("plain's bundle", "many-alpha.mjs failed to load:", "is not executable (mode -", "); a bundle the runtime serves is started, never imported") {
t.Errorf("the non-executable entrypoint was not refused by name:\n%s", logs.All())
}
got, err := call(t, asker, "alpha.one@anchor", map[string]any{})
@@ -218,6 +221,7 @@ func TestAnEntrypointThatIsNotExecutableIsRefused(t *testing.T) {
// A launched bundle is told the module it serves, so its seat's verbs stay the seat's (ADR 0193).
func TestALaunchedBundleRegisteringItsSeatFirstServesTheSeat(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("theta", "anchor", false, map[string][]string{"node-shelf": {"list"}}))
nodeTools := connect(t, "node-tools", "anchor")
@@ -147,3 +147,65 @@ func TestALostCompareAndSetCarriesItsCode(t *testing.T) {
t.Error("an update of revision zero was taken")
}
}
// 1887/A11: a bundle pulls one message from its own worker and settles it through its module's bus; another module may
// neither pull that worker nor settle what it did not pull.
func TestABundlePullsAndSettlesThroughItsModulesBus(t *testing.T) {
mesh := mt.New(t)
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := nc.JetStream()
if _, err := js.AddStream(&nats.StreamConfig{Name: "SEAT_NODE_BUILD_GO", Retention: nats.WorkQueuePolicy,
Subjects: []string{"mesh.seat.node-build-go.accept.>"}}); err != nil {
t.Fatal(err)
}
if _, err := js.AddConsumer("SEAT_NODE_BUILD_GO", &nats.ConsumerConfig{Durable: "SEAT_NODE_BUILD_GO_DELIVERY_worker",
AckPolicy: nats.AckExplicitPolicy, FilterSubject: "mesh.seat.node-build-go.accept.delivery"}); err != nil {
t.Fatal(err)
}
if _, err := js.Publish("mesh.seat.node-build-go.accept.delivery", []byte(`{"id":"ask1"}`)); err != nil {
t.Fatal(err)
}
holder := mt.MembershipOf("mesh-build-go", "anchor", false, nil)
holder.SeatTraffic = &bus.SeatTraffic{Workers: []bus.SeatWorker{{Stream: "SEAT_NODE_BUILD_GO",
Consumer: "SEAT_NODE_BUILD_GO_DELIVERY_worker", Filter: "mesh.seat.node-build-go.accept.delivery"}}}
other := mt.MembershipOf("mesh-build-node", "anchor", false, nil)
other.SeatTraffic = &bus.SeatTraffic{}
mesh.Issue(t, holder)
mesh.Issue(t, other)
conn := connect(t, "node-tools", "anchor")
conn.Follow("mesh-build-go")
conn.Follow("mesh-build-node")
mt.Until(t, func() error {
if conn.Membership("mesh-build-go") == nil || conn.Membership("mesh-build-node") == nil {
return errors.New("not issued yet")
}
return nil
})
all := &consumers{conn: conn, logf: t.Logf, of: map[string]*moduleEvents{}}
building := all.forModule("mesh-build-go").(*moduleBus)
elsewhere := all.forModule("mesh-build-node").(*moduleBus)
if _, err := elsewhere.Seat("pull", json.RawMessage(`{"worker":"SEAT_NODE_BUILD_GO_DELIVERY_worker"}`), nil); err == nil {
t.Fatal("another module pulled the Go build seat's worker")
}
raw, err := building.Seat("pull", json.RawMessage(`{"worker":"SEAT_NODE_BUILD_GO_DELIVERY_worker","wait_ms":1000}`), nil)
if err != nil {
t.Fatal(err)
}
var p bus.Pulled
if err := json.Unmarshal(raw, &p); err != nil || string(p.Body) != `{"id":"ask1"}` || p.ID == "" {
t.Fatalf("pulled %s (%v)", raw, err)
}
if _, err := elsewhere.Seat("settle", json.RawMessage(`{"id":"`+p.ID+`","fate":"ack"}`), nil); err == nil {
t.Fatal("another module settled the pulled ask")
}
if _, err := building.Seat("settle", json.RawMessage(`{"id":"`+p.ID+`","fate":"ack"}`), nil); err != nil {
t.Fatal(err)
}
if none, err := building.Seat("pull", json.RawMessage(`{"worker":"SEAT_NODE_BUILD_GO_DELIVERY_worker"}`), nil); err != nil || string(none) != "null" {
t.Fatalf("an empty worker answered %s, %v", none, err)
}
}
@@ -139,6 +139,7 @@ func waitFor(t *testing.T, path, line string) {
// The TypeScript SDK against the runtime (ADR 0201): two watches of one state, each handed only its
// own changes, the current values handled before the module's import goes on.
func TestTheSDKsStateReachesTheRuntime(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
kv := mesh.Bucket(t, "sdkkeeper_servers")
if _, err := kv.Put("all.one", []byte(`{"url":"http://one"}`)); err != nil {
+221
View File
@@ -0,0 +1,221 @@
package runtime
import (
"encoding/binary"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"syscall"
"time"
"golang.org/x/sys/unix"
)
// Settle is how long the bundles' directories stay quiet before the runtime acts on what changed there:
// a delivery unpacks a bundle in many writes, and the runtime loads it once, when they are done
// (issue 1945, note 23).
var Settle = 300 * time.Millisecond
// watchMask is what the runtime hears of a watched directory: a file or directory made, written,
// renamed in or out, removed or given another mode, and the directory itself removed or renamed.
const watchMask = unix.IN_CREATE | unix.IN_CLOSE_WRITE | unix.IN_MOVED_TO | unix.IN_MOVED_FROM | unix.IN_DELETE |
unix.IN_ATTRIB | unix.IN_DELETE_SELF | unix.IN_MOVE_SELF | unix.IN_ONLYDIR
// bundleWatch hears, through inotify, the directory of every entrypoint the runtime serves, or for one
// not there yet the nearest directory above it that is, and calls changed once a burst of changes has
// settled. Nothing is looked at on a timer: between bursts the watch sleeps in poll(2).
type bundleWatch struct {
fd int
wake [2]int // a pipe: a byte written to wake[1] ends the loop
paths []string
byWD map[int]string
settle time.Duration
changed func()
logf func(string, ...any)
done chan struct{}
// what names what is watched, and lost what follows when the watch fails, in the words a journal line carries.
what, lost string
}
// WatchFile hears, as the bundles are heard, the directory of the file at path, or the nearest directory above it that
// exists, and calls changed once at its start and once after each burst of changes there has settled: a runtime waiting
// for its broker credential reads it again when the node-engine writes it, never on a clock (issue 1496). It answers its
// stop.
func WatchFile(path string, changed func(), logf func(string, ...any), lost string) (func(), error) {
return watch([]string{path}, changed, logf, path, lost)
}
// watchBundles starts the watch of the entrypoints at paths. It answers its stop.
func watchBundles(paths []string, changed func(), logf func(string, ...any)) (func(), error) {
return watch(paths, changed, logf, "the bundles", "a bundle placed or changed there is served only after a restart")
}
// watch starts the watch of the files at paths, named what in its words; lost says what follows when it fails.
func watch(paths []string, changed func(), logf func(string, ...any), what, lost string) (func(), error) {
fd, err := unix.InotifyInit1(unix.IN_CLOEXEC | unix.IN_NONBLOCK)
if err != nil {
return nil, fmt.Errorf("inotify: %w", err)
}
w := &bundleWatch{fd: fd, paths: paths, byWD: map[int]string{}, settle: Settle, changed: changed, logf: logf,
done: make(chan struct{}), what: what, lost: lost}
if err := unix.Pipe2(w.wake[:], unix.O_CLOEXEC|unix.O_NONBLOCK); err != nil {
unix.Close(fd)
return nil, fmt.Errorf("pipe: %w", err)
}
w.arm()
go w.loop()
return func() {
_, _ = unix.Write(w.wake[1], []byte{1})
<-w.done
unix.Close(w.fd)
unix.Close(w.wake[0])
unix.Close(w.wake[1])
}, nil
}
// arm watches, for every entrypoint, its directory, or the nearest directory above it that exists. A
// watch already held on the same directory is kept (inotify answers its descriptor again). A directory
// made below the watched one between its stat and its watch would be heard by nobody, so after each watch
// the path is looked at again, and a deeper directory that came to be is watched in turn.
func (w *bundleWatch) arm() {
for _, p := range w.paths {
target := filepath.Dir(p)
watched := ""
for {
dir := nearestDir(target)
if dir == watched {
break
}
wd, err := unix.InotifyAddWatch(w.fd, dir, watchMask)
if err != nil {
w.logf("[mesh-tools] cannot watch %s for %s: %v; %s", dir, p, err, w.lost)
break
}
w.byWD[wd] = dir
watched = dir
}
}
}
// nearestDir is dir when it is a directory, else the nearest directory above it.
func nearestDir(dir string) string {
for {
if info, err := os.Stat(dir); err == nil && info.IsDir() {
return dir
}
up := filepath.Dir(dir)
if up == dir {
return dir
}
dir = up
}
}
func (w *bundleWatch) loop() {
defer close(w.done)
buf := make([]byte, 64*1024)
// Looked at once as the watch begins: a bundle that came between its load and the watch is not missed.
pending := true
quietAt := time.Now()
for {
timeout := -1
if pending {
timeout = int((time.Until(quietAt) + time.Millisecond - 1) / time.Millisecond)
if timeout < 0 {
timeout = 0
}
}
fds := []unix.PollFd{{Fd: int32(w.fd), Events: unix.POLLIN}, {Fd: int32(w.wake[0]), Events: unix.POLLIN}}
n, err := unix.Poll(fds, timeout)
if errors.Is(err, unix.EINTR) {
continue
}
if err != nil {
w.logf("[mesh-tools] the watch of %s stopped: %v; %s", w.what, err, w.lost)
return
}
if fds[1].Revents != 0 {
return
}
if n == 0 {
// Quiet for the settle: watch the directories that came to be, then act on what changed.
pending = false
w.arm()
w.changed()
continue
}
if fds[0].Revents&unix.POLLIN != 0 {
w.drain(buf)
pending = true
quietAt = time.Now().Add(w.settle)
}
}
}
// drain reads every event queued, dropping the watches of directories that went away or moved: the
// next arm watches what is at their path now.
func (w *bundleWatch) drain(buf []byte) {
for {
n, err := unix.Read(w.fd, buf)
if err != nil || n <= 0 {
return
}
for off := 0; off+unix.SizeofInotifyEvent <= n; {
wd := int(int32(binary.NativeEndian.Uint32(buf[off:])))
mask := binary.NativeEndian.Uint32(buf[off+4:])
length := int(binary.NativeEndian.Uint32(buf[off+12:]))
switch {
case mask&unix.IN_IGNORED != 0:
delete(w.byWD, wd)
case mask&unix.IN_MOVE_SELF != 0:
_, _ = unix.InotifyRmWatch(w.fd, uint32(wd))
delete(w.byWD, wd)
}
off += unix.SizeofInotifyEvent + length
}
}
}
// signatureOf is what the runtime compares to tell that an entrypoint changed: the file's identity,
// mode, size and times; "" is a file that does not exist. Only the entrypoint: a delivery puts every file
// of a bundle in place anew (a directory swapped in, or each file renamed in), so the entrypoint is new
// whenever its build is, while a file beside it may be one the bundle itself writes (a cache, a socket),
// which would load it again after every quiet spell, and modules sharing a directory each other.
func signatureOf(path string) string {
info, err := os.Stat(path)
if errors.Is(err, fs.ErrNotExist) {
return ""
}
if err != nil {
return "unreadable: " + err.Error()
}
return statSignature(info)
}
func statSignature(info fs.FileInfo) string {
sig := fmt.Sprintf("%s %d %d", info.Mode(), info.Size(), info.ModTime().UnixNano())
if st, ok := info.Sys().(*syscall.Stat_t); ok {
sig += fmt.Sprintf(" %d:%d %d.%d", st.Dev, st.Ino, st.Ctim.Sec, st.Ctim.Nsec)
}
return sig
}
// notStartable says why the runtime cannot start an entrypoint, in its real reason, or "" when it can
// (issue 1945, note 23: a missing file was said as "not executable").
func notStartable(path string) string {
info, err := os.Stat(path)
switch {
case errors.Is(err, fs.ErrNotExist):
return path + " does not exist; it is served as soon as it is in place"
case err != nil:
return path + " cannot be read: " + err.Error()
case info.IsDir():
return path + " is a directory, not a bundle's entrypoint"
case info.Mode().Perm()&0o111 == 0:
return fmt.Sprintf("%s is not executable (mode %s); a bundle the runtime serves is started, never imported, "+
"and its build makes it executable (novox/hq ADR 0193)", path, info.Mode().Perm())
}
return ""
}
+16 -2
View File
@@ -5,10 +5,18 @@
"invokes": [
"*"
],
"waits": [
"mesh-knowledge.reviewed",
"mesh-delivery-gate.transition",
"gitea.forge.status",
"mesh-controller.checked",
"operator-channel.decided.claude-code"
],
"provides": [
{
"name": "mcp-endpoint",
"scope": "node"
"scope": "node",
"answered-by": "helper"
}
],
"serves": {
@@ -52,5 +60,11 @@
"binary": "node-tools"
}
]
}
},
"state": [
"bound"
],
"reads": [
"mesh-controller.assignments"
]
}